CN115049963A - Video classification method and device, processor and electronic equipment - Google Patents

Abstract

The present application discloses a video classification method, apparatus, processor and electronic device. It relates to the field of artificial intelligence, and the method includes: acquiring a video to be classified; determining key frames and sampling frames of the video to be classified; and combining the key frames and sampling frames according to the time sequence of the key frames and sampling frames in the video to be classified to obtain an image Frame sequence; determine the classification result of the video to be classified based on the image frame sequence. Through the present application, the problem of poor video classification effect in the related art is solved.

Description

Video classification method, apparatus, processor and electronic equipment

technical field

The present application relates to the field, and in particular, to a video classification method, apparatus, processor and electronic device.

Background technique

For video classification using deep neural network, the current solution is to first decompose the video frame by frame, and then use the deep neural network to classify and identify frame by frame (or randomly select frames, such as: reserve one frame every 5 frames), and finally weighted average. Calculate the final classification result.

This technology is time-consuming and labor-intensive. Generally, a video is 30 frames per second, and a 10-minute video will generate 18,000 frames, or 18,000 images. At present, the V100 graphics card can predict about 60FPS for the real-time ResNet model, that is, it takes 5 minutes to completely process a 10-minute video. Therefore, in order to accelerate the prediction, multiple GPUs are often required for parallel computing acceleration. Although random frame extraction can reduce the overhead of GPU resources, due to the randomness of extraction, it is difficult to ensure that the selected frames contain key information, and the results are often bad due to serious interference. However, even if all frame information is used, the final result may be affected when the final weighting calculation is caused by too much irrelevant information in the video.

For the problem of poor video classification effect in the related art, no effective solution has been proposed yet.

SUMMARY OF THE INVENTION

The main purpose of the present application is to provide a video classification method, apparatus, processor and electronic device to solve the problem of poor video classification effect in the related art.

In order to achieve the above object, according to an aspect of the present application, a video classification method is provided. The method includes: acquiring a video to be classified; determining key frames and sample frames of the video to be classified; The sampled frames are combined to obtain an image frame sequence; the classification result of the video to be classified is determined based on the image frame sequence.

Optionally, determining the key frame and sampling frame of the video to be classified includes: determining multiple image frames of the video to be classified; extracting from the multiple image frames to obtain the sampling frame; A plurality of image frames are clustered to determine the key frame.

Optionally, extracting from the multiple image frames to obtain the sampling frame includes: extracting image frames corresponding to a specified number of frames from the multiple image frames as the sampling frame according to a predetermined frame number interval; Or according to a predetermined time interval, an image frame corresponding to a specified time is extracted from the plurality of image frames as the sampling frame.

Optionally, according to the time sequence of the key frame and the sample frame in the video to be classified, combining the key frame and the sample frame to obtain an image frame sequence includes: determining that the key frame is in the video frame. The playing time in the video to be classified is the first playing time; it is determined that the playing time of the sampling frame in the video to be classified is the second playing time; according to the difference between the first playing time and the second playing time The time sequence is to determine the arrangement order of the key frames and the sampled frames; the image frame sequence is determined according to the arrangement order of the key frames and the sampled frames.

Optionally, determining the classification result of the video to be classified based on the image frame sequence includes: inputting a plurality of predetermined image frames in the image frame sequence into a preset convolutional neural network model, and determining each of the predetermined image frames. The feature matrix and probability of the frame, wherein the predetermined image frame includes: the key frame and the sampling frame, and the preset convolutional neural network model is trained according to the sample image of the calibrated feature; according to the image frame The feature matrix and probability of a plurality of predetermined image frames in the sequence determine the classification result of the video to be classified.

Optionally, determining the classification result of the video to be classified according to the feature matrices and probabilities of a plurality of predetermined image frames in the image frame sequence includes: determining the feature matrix and probability product of each of the predetermined image frames to obtain a feature. result; according to the sequence of each predetermined image frame in the sequence of image frames, determine a sequence of feature results, wherein the sequence of feature results includes feature results of multiple predetermined image frames in the sequence of image frames; input the sequence of feature results into A preset cyclic neural network model is used to determine the classification result of the video to be classified, wherein the preset cyclic neural network model is obtained by training according to the sample video for which the classification result has been calibrated and the feature result sequence corresponding to the sample video.

Optionally, after inputting a plurality of predetermined image frames in the image frame sequence into a preset convolutional neural network model, and determining the feature matrix and probability of each of the predetermined image frames, the method further includes: The probability of the key frame is adjusted to a preset value; the sampling frame whose probability is lower than the preset threshold is deleted from the image frame sequence.

In order to achieve the above object, according to another aspect of the present application, there is provided a video classification device, comprising: an acquisition unit for acquiring a video to be classified; a first determination unit for determining a key frame and a key frame of the to-be-classified video a sampling frame; a merging unit, configured to merge the key frame and the sampling frame according to the time sequence of the key frame and the sampling frame in the video to be classified to obtain an image frame sequence; a second determining unit , for determining the classification result of the video to be classified based on the image frame sequence.

In order to achieve the above object, according to another aspect of the present application, a processor is provided. The processor is used for running a program, wherein the above-mentioned video classification method is executed when the program is running.

In order to achieve the above object, according to another aspect of the present application, an electronic device is provided. The electronic device includes one or more processors and a memory for storing one or more programs, wherein the one or more programs, when executed by the one or more processors, cause the one or more programs to One or more processors implement the video classification method described above.

Through the present application, the following steps are adopted: obtaining the video to be classified; determining the key frame and sampling frame of the video to be classified; according to the time sequence of the key frame and the sampling frame in the video to be classified, combining the key frame and the sampling frame to obtain the image frame Sequence; the classification result of the video to be classified is determined based on the image frame sequence, which solves the problem of poor video classification effect in the related art. Thus, the effect of accurately determining the video classification result is achieved.

Description of drawings

The accompanying drawings constituting a part of the present application are used to provide further understanding of the present application, and the schematic embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation of the present application. In the attached image:

1 is a flowchart of a video classification method provided according to an embodiment of the present application;

2 is a schematic diagram of a predetermined convolutional neural network model according to an embodiment of the present application;

3 is a schematic diagram of a preset recurrent neural network according to an embodiment of the present application;

4 is a schematic diagram of a video classification apparatus provided according to an embodiment of the present application;

FIG. 5 is a schematic diagram of an electronic device provided according to an embodiment of the present application.

Detailed ways

It should be noted that the embodiments in the present application and the features of the embodiments may be combined with each other in the case of no conflict. The present application will be described in detail below with reference to the accompanying drawings and in conjunction with the embodiments.

In order to make those skilled in the art better understand the solutions of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only The embodiments are part of the present application, but not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work shall fall within the scope of protection of the present application.

It should be noted that the terms "first", "second", etc. in the description and claims of the present application and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific sequence or sequence. It is to be understood that the data so used are interchangeable under appropriate circumstances for the embodiments of the application described herein. Furthermore, the terms "comprising" and "having" and any variations thereof, are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those expressly listed Rather, those steps or units may include other steps or units not expressly listed or inherent to these processes, methods, products or devices.

The present invention will be described below with reference to the preferred implementation steps. FIG. 1 is a flowchart of a video classification method provided according to an embodiment of the present application. As shown in FIG. 1 , the method includes the following steps:

Step S101, obtaining the video to be classified;

Step S102, determining the key frame and sampling frame of the video to be classified;

Step S103, according to the time sequence of the key frame and the sample frame in the video to be classified, combine the key frame and the sample frame to obtain an image frame sequence;

Step S104, determining the classification result of the video to be classified based on the image frame sequence.

It should be noted that videos are all composed of still pictures, and these still pictures are called frames, so the video to be classified includes multiple image frames.

In the above step S101, the video to be classified may use any known non-encrypted video format (MP4, MAV, MOV, etc.), and the video time is not limited.

In the above step S102, the sampling frame may be randomly selected from the image frames of the video to be classified, and the key frame may be an image frame with special image characteristics in the video to be classified.

Optionally, the key frame and the image frame have special differences in image characteristics such as color, contrast, or brightness.

In the above step S103, the image frame represents the screen of the video to be classified at a specific playback time. Therefore, the image frame has a time attribute, and the key frame and sampling frame determined according to the image frame also have a time attribute. Therefore, when determining the key frame and After sampling the frames, the playing time of each key frame and the sampling frame in the video to be classified can be determined, and according to the time sequence of the playing time, the key frame and the sampling frame are combined to obtain an image frame sequence.

In the above step S103, the image frame sequence includes at least one key frame and at least one sample frame.

In the above-mentioned embodiment of the present invention, since the image frame sequence includes both sampling frames and key frames, the key information of the video to be classified is largely preserved in the image frame sequence, and further, according to the image frame sequence, the video to be classified can be accurately processed. Classification.

In the above step S104, the image frame sequence may be analyzed by means of machine learning to determine the classification result of the video to be classified.

Optionally, in the video classification method provided in the embodiment of the present application, determining the key frame and sampling frame of the video to be classified includes: determining multiple image frames of the video to be classified; extracting from the multiple image frames to obtain the sampling frame. ; Cluster multiple image frames to determine key frames.

In the above-mentioned embodiment of the present invention, the sampling frame is extracted from multiple image frames of the video to be classified according to the method of simple frame extraction, and the simple frame extraction refers to removing some images according to a certain rule; the key frame is extracted from multiple image frames of the video to be classified. Image frames are determined by clustering.

Optionally, the video to be classified is processed frame by frame to generate an image, and an original frame picture set (ie, multiple image frames are obtained) P={F1, F2, F3....Fn}.

Optionally, by performing simple frame extraction processing on the original frame picture set (that is, to obtain multiple image frames) P={F1, F2, F3.... sample frame) F={F1,F6,F11....Fn}.

It should be noted that simple frame extraction refers to removing some pictures according to certain rules. The common method is to reserve one frame every 5 frames or reserve 1/3 of the number of frames per second (if the video is 30FPS, reserve 10 frames).

Optionally, the key frame calculation is performed based on a collection of original frame images (ie, a plurality of image frames are obtained) P={F1,F2,F3....Fn}. The difference between keyframes and other frames is that keyframes are special for image characteristics such as color, contrast, or brightness. Therefore, it is necessary to perform cluster analysis on the original frame image set (that is, to obtain multiple image frames) P={F1, F2, F3....Fn} through the K-means clustering algorithm to obtain different clusters, and then for each The images in the cluster are subjected to histogram analysis, and those whose mean values deviate from the average are selected as key frames to obtain the key frame set K.

Optionally, in the video classification method provided in the embodiment of the present application, extracting from multiple image frames to obtain a sampling frame includes: extracting image frames corresponding to a specified number of frames from the multiple image frames according to a predetermined frame number interval. As a sampling frame; or according to a predetermined time interval, an image frame corresponding to a specified time is extracted from a plurality of image frames as a sampling frame.

In the above-mentioned embodiment of the present invention, in the process of acquiring sampling frames, the sampling may be performed at intervals of a predetermined number of frames. For example, after the video to be classified is divided into multiple image frames, one frame may be reserved for every 5 frames as the sampling frame; The sampling may also be performed at predetermined time intervals, for example, 1/3 of the frames of the multiple image frames corresponding to the video to be classified per second are reserved as sampling frames.

Optionally, in the video classification method provided in the embodiment of the present application, according to the time sequence of the key frame and the sample frame in the video to be classified, combining the key frame and the sample frame to obtain the image frame sequence includes: determining that the key frame is in the to-be-classified video frame sequence. The playback time in the classified video is the first playback time; the playback time of the sampling frame in the video to be classified is determined as the second playback time; according to the time sequence of the first playback time and the second playback time, the key frame and the sampling frame are determined. Arrangement order; determine the sequence of image frames according to the arrangement order of key frames and sampling frames.

In the above-mentioned embodiment of the present invention, the key frame and the sample frame are image frames in the video to be classified, so each key frame and sample frame has a corresponding playback time in the video to be classified, according to the key frame and sample frame in the video to be classified The playback time in the video frame is determined, and the sequence of key frames and sample frames is determined, so as to obtain the image frame sequence that retains the key information of the video to be classified.

Optionally, in the video classification method provided in the embodiment of the present application, determining the classification result of the video to be classified based on the image frame sequence includes: inputting a plurality of predetermined image frames in the image frame sequence into a preset convolutional neural network model, and determining The feature matrix and probability of each predetermined image frame, wherein the predetermined image frame includes: key frames and sample frames, the preset convolutional neural network model is trained according to the sample images of the calibrated features; The feature matrix and probability of the frame determine the classification result of the video to be classified.

In the above-mentioned embodiment of the present invention, according to the preset convolutional neural network model, the feature matrix and probability of each predetermined image frame can be quickly determined by analyzing a plurality of predetermined image frames in the image frame sequence, and then according to the determined feature matrix and probability Classify the video to be classified.

Optionally, in the video classification method provided in the embodiment of the present application, determining the classification result of the video to be classified according to the feature matrix and the probability of a plurality of predetermined image frames in the image frame sequence includes: determining according to the product of the feature matrix and the probability. The feature result of each predetermined image frame; according to the sequence of the predetermined image frame in the image frame sequence, sort the feature results of multiple predetermined image frames in the image frame sequence to obtain the feature result sequence; input the feature result sequence into the preset recurrent neural network The network model determines the classification result of the video to be classified, wherein the preset recurrent neural network model is obtained by training according to the sample video for which the classification result has been calibrated and the feature result sequence corresponding to the sample video.

In the above-mentioned embodiment of the present invention, according to the feature matrix and the probability product of the predetermined image frame, the feature result of the predetermined image frame can be determined, and then the feature result of the predetermined image frame can be analyzed according to the preset cyclic neural network model, and the feature result of the predetermined image frame can be analyzed according to the image frame sequence. The feature results of a plurality of predetermined image frames classify the video to be classified.

Optionally, in the video classification method provided in the embodiment of the present application, after inputting multiple predetermined image frames in the image frame sequence into a preset convolutional neural network model, and determining the feature matrix and probability of each predetermined image frame, The method further includes: adjusting the probability of the key frame to a preset value; and deleting the sampled frame whose probability is lower than the preset threshold in the image frame sequence.

In the above embodiment of the present invention, the probability of the key frame is adjusted to a preset value, which can highlight the key information of the video to be classified, and delete the sampling frame whose probability is lower than the preset threshold in the image frame sequence, which can reduce the interference information of the video to be classified. , and then the classification result of the video to be classified can be accurately determined based on the adjusted image frame sequence.

The present invention also provides a preferred embodiment, which provides a video classification method based on key frames.

The invention solves the problem of missing key information caused by random frame selection by pre-extracting key frames of the video in advance, and dynamically adjusts the final weighting factor while using the RNN model (ie, the preset convolutional neural network model) to solve the problem of irrelevant information in the video. impact on the overall outcome.

As an optional example, the present invention provides a key frame-based video classification method, which includes the following steps:

S201: Input the video to be classified.

Optionally, the video to be classified may be any known non-encrypted video format (MP4, MAV, MOV, etc.), and the video duration is not limited.

S202: After performing frame extraction processing on the video to be classified, a preset convolutional neural network model is input to obtain image features and probabilities.

Optionally, the video to be classified is processed frame by frame to generate multiple image frames, and then simple frame extraction processing is performed. In addition, it is also necessary to perform key frame calculation on all image frames. If the key frames that meet the requirements are filtered out after simple frame extraction, they need to be added back in chronological order.

Optionally, the structure of the preset convolutional neural network model can be Resnet or VGG, and it is necessary to output the feature matrix of the image frame and the classification probability corresponding to the image frame.

S203: Input the feature matrix and probability of the image frame into a preset recurrent neural network model to obtain a classification result

A new matrix (ie, a feature result) is generated from the feature matrix*probability of the image frame output in S202, and then stacked in time sequence and input to the preset recurrent neural network model, and then the final classification result is obtained through softmax.

As an optional example, the detailed flow of the key frame-based video classification method of the present invention includes the following steps:

S301: Input the video to be classified.

S302: Decompose the video to be classified into multiple image frames frame by frame, and calculate a key frame set (ie, multiple key frames) K. According to one sampling every N frames, a set of sampled frames (ie, a plurality of sampled frames) F is obtained from the reserved pictures. The image frame sequence FN is obtained by taking the union of the key frame set (ie multiple key frames) K and the sample frame set (ie multiple sample frames) F according to time.

Optionally, the video to be classified is processed frame by frame to generate pictures to obtain a set of original frame pictures (ie multiple image frames) P={F1, F2, F3....Fn}, and then simple frame extraction processing is performed. The so-called simple frame extraction refers to removing some pictures according to certain rules. The common method is to reserve one frame every 5 frames or reserve 1/3 of the number of frames per second (if the video is 30FPS, reserve 10 frames). Thus, a sample frame set (ie, a plurality of sample frames) F={F1, F6, F11 . . . Fn} after the simple frame sampling is obtained.

Optionally, the key frame calculation is performed based on the original frame picture set (ie, multiple image frames) P. The difference between a key frame and other frames is that the key frame has special image characteristics such as color, contrast, or brightness. Therefore, the K-means clustering algorithm is used to perform cluster analysis on the original frame picture set (that is, multiple image frames) and P to obtain different clusters, and then perform histogram analysis on the pictures in each cluster, and select those histograms. Too much deviation from the mean value is used as a key frame, so as to obtain a key frame set (ie, multiple key frames) K.

Optionally, the sampling frame set (that is, multiple sampling frames) F and the key frame set (that is, multiple key frames) K are merged in chronological order, that is, if the key frame set (that is, multiple key frames) K If the frame is already in the sampling frame set (ie multiple sampling frames) F, do nothing, if the frame in the key frame set (ie multiple key frames) K is not in the sampling frame set (ie multiple sampling frames) F , then insert this frame into the sample frame set (ie, a plurality of sample frames) F according to the time sequence, to obtain the image frame sequence FN with key frames.

For example, F={F1,F6,F11}, K={F1,F3,F9}, then FN={F1,F3,F6,F9,F11}.

S303: Input the image frame sequence FN into a preset convolutional neural network model to obtain a feature matrix of each preset image frame and a probability of belonging to a certain category.

FIG. 2 is a schematic diagram of a predetermined convolutional neural network model according to an embodiment of the present application. As shown in Figure 2, it is necessary to input the preset image frames contained in the image frame sequence FN into the pre-trained predetermined convolutional neural network model one by one. The output feature matrix is the output before the fully connected layer, and the probability is is the result output by softmax.

Optionally, when training the predetermined convolutional neural network model, the training is still performed in a traditional way, that is, CNN + fully connected layers are trained together.

S304: Multiply the feature matrix of the preset image frame by its probability, and input the results into the preset recurrent neural network model by stacking the results in the original order of time.

Optionally, both the probability and feature matrix here come from the output of S303, but it should be noted that the probability of the key frame is manually adjusted to 1, and it is not necessary to discard frames with a probability less than 0.5, because the probability is too low, which will affect the final recognition. As a result, it becomes a distractor.

FIG. 3 is a schematic diagram of a preset recurrent neural network according to an embodiment of the present application. As shown in FIG. 3 , the preset recurrent neural network model includes a predetermined convolutional neural network model RNN + fully connected layer.

S305: Output the classification result.

It should be noted that in the past, video recognition was based on fixed frame extraction, which was easy to miss some key information to interfere with the results. At the same time, the results may be biased due to the randomness of the extracted frames; and even in the case of frame extraction, It takes a lot of computing power to calculate.

The present invention introduces the pre-extraction of key frames, so that the number of extracted frames can be less and more sparse, thereby reducing the calculation time and improving the accuracy of the results. After simple image classification, the RNN network is introduced to analyze the video globally, which is more accurate than the traditional weighted average.

The present invention adopts the frame extraction technology based on key frames, which greatly retains the key information of the video in the case of more sparse frame extraction, and the introduced RNN network can also better understand an entire stream of video.

The video classification method provided in the embodiment of the present application obtains the video to be classified; determines the key frame and the sample frame of the video to be classified; according to the time sequence of the key frame and the sample frame in the video to be classified, the key frame and the sample frame are combined to obtain Image frame sequence; the classification result of the video to be classified is determined based on the image frame sequence, which solves the problem of poor video classification effect in the related art. Thus, the effect of accurately determining the video classification result is achieved.

It should be noted that the steps shown in the flowcharts of the accompanying drawings may be executed in a computer system, such as a set of computer-executable instructions, and, although a logical sequence is shown in the flowcharts, in some cases, Steps shown or described may be performed in an order different from that herein.

The embodiment of the present application further provides a video classification apparatus. It should be noted that the video classification apparatus of the embodiment of the present application may be used to execute the method for video classification provided by the embodiment of the present application. The video classification apparatus provided by the embodiments of the present application will be introduced below.

FIG. 4 is a schematic diagram of a video classification apparatus according to an embodiment of the present application. As shown in FIG. 4 , the device includes: an acquisition unit 41 for acquiring a video to be classified; a first determining unit 42 for determining key frames and sample frames of the video to be classified; a merging unit 43 for determining according to the key frame and The time sequence of the sampled frames in the video to be classified is obtained by combining the key frame and the sampled frame to obtain an image frame sequence; the second determining unit 44 is configured to determine the classification result of the video to be classified based on the image frame sequence.

It should be noted that, the obtaining unit 41 in this embodiment may be used to execute step S101 in this embodiment of the present application, and the first determining unit 42 in this embodiment may be used to execute step S102 in this embodiment of the present application, the The merging unit 43 in this embodiment may be configured to perform step S103 in this embodiment of the present application, and the second determining unit 44 in this embodiment may be configured to perform step S104 in this embodiment of the present application. The examples and application scenarios implemented by the foregoing units and corresponding steps are the same, but are not limited to the contents disclosed in the foregoing embodiments.

Optionally, in the video classification apparatus provided in the embodiment of the present application, the first determination unit includes: a first determination module, configured to determine multiple image frames of the video to be classified; Extraction is performed to obtain sampled frames; the clustering module is used to cluster multiple image frames to determine key frames.

Optionally, in the video classification apparatus provided in the embodiment of the present application, the extraction module includes: a first extraction module, configured to extract image frames corresponding to a specified number of frames from a plurality of image frames as sampling frames according to a predetermined frame number interval. ; or a second extraction module, configured to extract image frames corresponding to a specified time from a plurality of image frames as sampling frames according to a predetermined time interval.

Optionally, in the video classification apparatus provided in the embodiment of the present application, the merging unit includes: a third determination module, configured to determine the playback time of the key frame in the video to be classified as the first playback time; a fourth determination module, used In determining that the playback time of the sampling frame in the video to be classified is the second playback time; the fifth determining module is used to determine the arrangement order of the key frames and the sampling frames according to the time sequence of the first playback time and the second playback time; Six determination modules are used to determine the sequence of image frames according to the arrangement order of the key frames and the sampling frames.

Optionally, in the video classification device provided in the embodiment of the present application, the second determination unit includes: a sixth determination module, configured to input a plurality of predetermined image frames in the image frame sequence into a preset convolutional neural network model, and determine The feature matrix and probability of each predetermined image frame, wherein the predetermined image frame includes: key frames and sample frames, the preset convolutional neural network model is trained according to the sample images of the calibrated features; the seventh determination module is used for The feature matrix and probability of multiple predetermined image frames in the frame sequence determine the classification result of the video to be classified.

Optionally, in the video classification apparatus provided in the embodiment of the present application, the sixth determination module includes: a seventh determination module, configured to determine the characteristic result of each predetermined image frame according to the product of the characteristic matrix and the probability; the eighth determination The module is used for sorting the feature results of a plurality of predetermined image frames in the image frame sequence according to the sequence of the predetermined image frames in the image frame sequence to obtain the feature result sequence; the ninth determination module is used for inputting the feature result sequence into the preset The cyclic neural network model determines the classification result of the video to be classified, wherein the preset cyclic neural network model is obtained by training according to the sample video for which the classification result has been calibrated and the feature result sequence corresponding to the sample video.

Optionally, in the video classification apparatus provided in the embodiment of the present application, the apparatus further includes: an adjustment unit, configured to input a plurality of predetermined image frames in the image frame sequence into a preset convolutional neural network model, and determine each predetermined image frame. After obtaining the feature matrix and probability of the image frame, the probability of the key frame is adjusted to a preset value; the deletion unit is used to delete the sampling frame whose probability is lower than the preset threshold in the image frame sequence.

The video classification device provided in the embodiment of the present application obtains the video to be classified; determines the key frame and the sample frame of the video to be classified; Image frame sequence; the classification result of the video to be classified is determined based on the image frame sequence, which solves the problem of poor video classification effect in the related art. And then achieve the effect of accurately determining the video classification results

The video classification device includes a processor and a memory, the above-mentioned units and the like are all stored in the memory as program units, and the processor executes the above-mentioned program units stored in the memory to realize corresponding functions.

The processor includes a kernel, and the kernel calls the corresponding program unit from the memory. One or more kernels can be set, and the purpose of accurately determining the video classification result can be achieved by adjusting the kernel parameters.

Memory may include non-persistent memory in computer readable media, random access memory (RAM) and/or non-volatile memory, such as read only memory (ROM) or flash memory (flash RAM), the memory including at least one memory chip.

An embodiment of the present invention provides a computer-readable storage medium on which a program is stored, and when the program is executed by a processor, implements the video classification method.

An embodiment of the present invention provides a processor for running a program, wherein the video classification method is executed when the program is running.

FIG. 5 is a schematic diagram of an electronic device provided according to an embodiment of the present application. As shown in FIG. 5, an embodiment of the present invention provides an electronic device 50. The device includes a processor 501, a memory 502, and a program stored in the memory and running on the processor. The processor implements the following steps when executing the program: Obtain the video to be classified; determine the key frame and sample frame of the video to be classified; combine the key frame and sample frame to obtain an image frame sequence according to the time sequence of the key frame and the sample frame in the video to be classified; determine the image frame sequence based on the image frame sequence Classification results of classified videos.

Optionally, when the processor executes the program, the following steps are implemented: determining multiple image frames of the video to be classified; extracting from multiple image frames to obtain sampled frames; clustering multiple image frames to determine key frames.

Optionally, the processor implements the following steps when executing the program: according to a predetermined frame number interval, extracting image frames corresponding to a specified number of frames from a plurality of image frames as sampling frames; or according to a predetermined time interval, extracting from a plurality of image frames. The image frame corresponding to the specified time is used as the sampling frame.

Optionally, when the processor executes the program, the following steps are implemented: determining the playback time of the key frame in the video to be classified as the first playback time; determining the playback time of the sample frame in the video to be classified as the second playback time; The chronological sequence of the playing time and the second playing time determines the arrangement order of the key frames and the sampling frames; the image frame sequence is determined according to the arrangement order of the key frames and the sampling frames.

Optionally, when the processor executes the program, the following steps are implemented: inputting a plurality of predetermined image frames in the image frame sequence into a preset convolutional neural network model, and determining the feature matrix and probability of each predetermined image frame, wherein the predetermined image frame Including: key frames and sampling frames, the preset convolutional neural network model is trained according to the sample images with calibrated features; according to the feature matrix and probability of multiple predetermined image frames in the image frame sequence, the classification result of the video to be classified is determined.

Optionally, when the processor executes the program, the following steps are implemented: determining the characteristic result of each predetermined image frame according to the product of the characteristic matrix and the probability; The feature results of the image frames are sorted to obtain a sequence of feature results; the sequence of feature results is input into a preset cyclic neural network model to determine the classification result of the video to be classified, wherein the preset cyclic neural network model is based on the sample videos that have calibrated the classification results and The feature result sequence corresponding to the sample video is obtained by training.

Optionally, when the processor executes the program, the following steps are implemented: after inputting a plurality of predetermined image frames in the image frame sequence into a preset convolutional neural network model, after determining the feature matrix and probability of each predetermined image frame, the key frame The probability is adjusted to the preset value; the sampling frame whose probability is lower than the preset threshold is deleted from the image frame sequence

Optionally, the electronic device herein may be a server, a PC, a PAD, a mobile phone, and the like.

The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing a program initialized with the following method steps: acquiring a video to be classified; determining key frames and sampling frames of the video to be classified; According to the time sequence of the sampled frames in the video to be classified, the key frame and the sampled frame are combined to obtain an image frame sequence; the classification result of the to-be-classified video is determined based on the image frame sequence.

Optionally, when executed on a data processing device, it is suitable for executing a program initialized with the following method steps: determining multiple image frames of the video to be classified; extracting from multiple image frames to obtain sampled frames; Image frames are clustered to determine keyframes.

Optionally, when executed on a data processing device, it is suitable for executing a program initialized with the following method steps: extracting image frames corresponding to a specified number of frames from a plurality of image frames as sampling frames according to a predetermined frame interval; or At a predetermined time interval, an image frame corresponding to a specified time is extracted from a plurality of image frames as a sampling frame.

Optionally, when executed on a data processing device, it is suitable for executing a program initialized with the following method steps: determining the playback time of the key frame in the video to be classified as the first playback time; determining the playback time of the sample frame in the video to be classified; The play time is the second play time; according to the time sequence of the first play time and the second play time, the arrangement order of the key frames and the sampling frames is determined; the image frame sequence is determined according to the arrangement order of the key frames and the sampling frames.

Optionally, when executed on a data processing device, it is suitable for executing a program initialized with the following method steps: inputting a plurality of predetermined image frames in the image frame sequence into a preset convolutional neural network model, and determining each predetermined image frame. The feature matrix and probability of , wherein the predetermined image frame includes: key frame and sampling frame, the preset convolutional neural network model is trained according to the sample image of the calibrated feature; probability, to determine the classification result of the video to be classified.

Optionally, when executed on a data processing device, it is suitable for executing a program initialized with the following method steps: determining the characteristic result of each predetermined image frame according to the product of the characteristic matrix and the probability; order, sort the feature results of a plurality of predetermined image frames in the image frame sequence to obtain a feature result sequence; input the feature result sequence into a preset cyclic neural network model to determine the classification result of the video to be classified, wherein the preset cyclic neural network The network model is trained according to the sample videos whose classification results have been calibrated and the feature result sequences corresponding to the sample videos.

Optionally, when executed on a data processing device, it is suitable for executing a program initialized with the following method steps: inputting a plurality of predetermined image frames in a sequence of image frames into a preset convolutional neural network model, determining each predetermined image After the feature matrix and probability of the frame, the probability of the key frame is adjusted to a preset value; the sampled frame whose probability is lower than the preset threshold is deleted from the image frame sequence.

As will be appreciated by those skilled in the art, the embodiments of the present application may be provided as a method, a system, or a computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present application. It will be understood that each flow and/or block in the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to the processor of a general purpose computer, special purpose computer, embedded processor or other programmable data processing device to produce a machine such that the instructions executed by the processor of the computer or other programmable data processing device produce Means for implementing the functions specified in a flow or flow of a flowchart and/or a block or blocks of a block diagram.

These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory result in an article of manufacture comprising instruction means, the instructions The apparatus implements the functions specified in the flow or flow of the flowcharts and/or the block or blocks of the block diagrams.

These computer program instructions can also be loaded on a computer or other programmable data processing device to cause a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process such that The instructions provide steps for implementing the functions specified in the flow or blocks of the flowcharts and/or the block or blocks of the block diagrams.

In a typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.

Memory may include non-persistent memory in computer readable media, random access memory (RAM) and/or non-volatile memory in the form of, for example, read only memory (ROM) or flash memory (flash RAM). Memory is an example of a computer-readable medium.

Computer-readable media includes both persistent and non-permanent, removable and non-removable media, and storage of information may be implemented by any method or technology. Information may be computer readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), Flash Memory or other memory technology, Compact Disc Read Only Memory (CD-ROM), Digital Versatile Disc (DVD) or other optical storage, Magnetic tape cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media, such as modulated data signals and carrier waves.

It should also be noted that the terms "comprising", "comprising" or any other variation thereof are intended to encompass a non-exclusive inclusion such that a process, method, article or device comprising a series of elements includes not only those elements, but also Other elements not expressly listed, or which are inherent to such a process, method, article of manufacture, or apparatus are also included. Without further limitation, an element qualified by the phrase "comprising a..." does not preclude the presence of additional identical elements in the process, method, article of manufacture or apparatus that includes the element.

It will be appreciated by those skilled in the art that the embodiments of the present application may be provided as a method, a system or a computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.

The above are merely examples of the present application, and are not intended to limit the present application. Various modifications and variations of this application are possible for those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included within the scope of the claims of the present application.

Claims (10)

1. A method of video classification, comprising:

acquiring a video to be classified;

determining key frames and sampling frames of the video to be classified;

merging the key frames and the sampling frames according to the time sequence of the key frames and the sampling frames in the video to be classified to obtain an image frame sequence;

determining a classification result of the video to be classified based on the image frame sequence.

2. The method of claim 1, wherein determining key frames and sample frames of the video to be classified comprises:

determining a plurality of image frames of the video to be classified;

extracting the plurality of image frames to obtain the sampling frame;

clustering the plurality of image frames, and determining the key frame.

3. The method of claim 2, wherein decimating among the plurality of image frames to obtain the sampled frame comprises:

extracting image frames corresponding to the appointed number of frames from the plurality of image frames as the sampling frames according to the preset frame number interval; or

And extracting an image frame corresponding to a designated time from the plurality of image frames as the sampling frame at a predetermined time interval.

4. The method of claim 1, wherein combining the key frames and the sample frames into an image frame sequence according to the time sequence of the key frames and the sample frames in the video to be classified comprises:

determining the playing time of the key frame in the video to be classified as a first playing time;

determining the playing time of the sampling frame in the video to be classified as a second playing time;

determining the arrangement sequence of the key frames and the sampling frames according to the time sequence of the first playing time and the second playing time;

and determining the image frame sequence according to the arrangement sequence of the key frames and the sampling frames.

5. The method of claim 1, wherein determining the classification result for the video to be classified based on the sequence of image frames comprises:

inputting a plurality of predetermined image frames in the image frame sequence into a preset convolutional neural network model, and determining a feature matrix and a probability of each predetermined image frame, wherein the predetermined image frames comprise: the preset convolutional neural network model is obtained by training the key frame and the sampling frame according to a sample image with calibrated characteristics;

and determining the classification result of the video to be classified according to the feature matrix and the probability of a plurality of preset image frames in the image frame sequence.

6. The method of claim 5, wherein determining the classification result of the video to be classified according to the feature matrix and the probability of a plurality of predetermined image frames in the image frame sequence comprises:

determining a feature result of each of the predetermined image frames according to a product of the feature matrix and the probability;

sequencing the feature results of a plurality of preset image frames in the image frame sequence according to the sequence of the preset image frames in the image frame sequence to obtain a feature result sequence;

and inputting the characteristic result sequence into a preset cyclic neural network model, and determining the classification result of the video to be classified, wherein the preset cyclic neural network model is obtained by training according to the sample video with the classification result calibrated and the characteristic result sequence corresponding to the sample video.

7. The method of claim 5, wherein after inputting a plurality of predetermined image frames in the sequence of image frames into a preset convolutional neural network model, determining a feature matrix and a probability for each of the predetermined image frames, the method further comprises:

adjusting the probability of the key frame to a preset value;

and deleting the sampling frames with the probability lower than a preset threshold value in the image frame sequence.

8. A video classification apparatus, comprising:

the acquiring unit is used for acquiring a video to be classified;

the first determining unit is used for determining key frames and sampling frames of the video to be classified;

the merging unit is used for merging the key frames and the sampling frames according to the time sequence of the key frames and the sampling frames in the video to be classified to obtain an image frame sequence;

a second determining unit, configured to determine a classification result of the video to be classified based on the image frame sequence.

9. A processor, characterized in that the processor is configured to run a program, wherein the program is configured to perform the video classification method according to any one of claims 1 to 7 when running.

10. An electronic device comprising one or more processors and memory storing one or more programs, wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the video classification method of any of claims 1-7.

Images