CN106919652B - Short-sighted frequency automatic marking method and system based on multi-source various visual angles transductive learning - Google Patents

Abstract

The invention discloses a short video automatic tagging method based on multi-source multi-view transductive learning, comprising: acquiring short video data; preprocessing the short video data to generate image key frames, audio tracks, and text in a consistent format and semantic label; extract the multi-view feature vector of the image key frame, audio track and text; set up a short video annotation database, the multi-view feature vector and the semantic label are stored in the short video annotation database; calculate the The similarity between the multi-view feature vectors; the multi-view fusion subspace is established through the similarity of the multi-view feature vectors; the transductive solution to the multi-view fusion subspace is automatically marked in the semantic label to be marked on short video data. And a short video automatic labeling system based on multi-source and multi-view transductive learning. The invention fully considers the multi-source information attached to the short video data, and improves the labeling accuracy.

Description

Short video automatic labeling method and system based on multi-source and multi-view transductive learning

technical field

The invention relates to the field of short video tagging, in particular to a short video automatic tagging method and system based on multi-source and multi-view direct push learning.

Background technique

With the development of mobile communication technology and Internet technology and the popularization of various smart terminals, short videos shot through mobile phones, tablet computers and other terminals and shared in social circles have become popular social applications among users. This short video social method originated from the video sharing website Vine in 2013. Its mobile client can limit the shooting time of short videos to 6 seconds. Various short video applications, such as Instagram, Meipai, WeChat, Weibo, Tencent Weishi, Weikepai, etc., have developed rapidly in recent years, and the duration of short videos has also expanded to 60 seconds. Short videos are seamlessly connected to various social platforms on the Internet, and are directly spread to social networks after shooting. Short videos integrate dynamic perspectives such as vision, hearing, text and user comments, which can meet users' expression and communication needs more intuitively and three-dimensionally. Therefore, short videos are more in line with the trend of the mobile communication era, and are more in line with users' habits of timely recording, rapid editing and rapid sharing. The amount of information brought by dynamic short video interaction is more diversified, and it is easier to promote the dissemination of topics, and various short video sharing platforms have made more and more short video users through commenting, following and reposting mechanisms, and user stickiness has become stronger and stronger. So short video is a new type of media.

It plays a very important role in retrieving and managing a large amount of short video content. At the same time, from the perspective of network information management, the analysis, mining and screening of video content has become an urgent problem to be solved. Video itself is a comprehensive media integrating images, sounds, etc. It contains a large amount of information, and its retrieval itself is a difficult point. Many search engines can already handle text retrieval, but there is still no effective method to retrieve video data, the main reason is that there is no effective method to index this multimedia data. Video annotation is the process of giving video text tags. The given tags describe the content semantics of the video, and realize the mapping between the underlying features of the video and semantic concepts. The obtained tags can be used as metadata for efficient indexing. The corresponding video retrieval and analysis are transformed into traditional keyword and mining.

Existing patents mainly focus on traditional long-term video sequences, and are only annotated by single-view feature analysis of visual content. For example, the Chinese Invention Patent (Application No. 201511021303.2) "Video Labeling Method and Device" applies the frequency of occurrence of objects in video frames to set the identification information of each video segment. Chinese Invention Patent (Application No. 201010134073.1) "A Video Labeling Method" realizes video labeling by single-view similarity calculation of visual color features, without applying audio and text information. Chinese Invention Patent (Application No. 201511005264.7) "A Video Labeling Method and Video Labeling Device" invented a video labeling method based on image comparison, which superimposes the reference video image with better clarity and the image to be labeled in the same scene Compare and highlight sensitive target positions. Chinese Invention Patent (Application No. 201511005264.7) "A Video Search Method and Device" determines video frame tags by predicting, merging, and labeling video frame tags, but only uses visual single-view features. Chinese Invention Patent (Application No. 201210075050.7) "Video Semantic Annotation Method Based on Feature Bag Model and Supervised Learning" uses support vector machine to realize visual vocabulary annotation through vocabulary ontology library and visual vocabulary modeling, but this method only uses SIFT features .

The annotation methods proposed in the existing literature are also limited to annotation through single-view feature analysis of visual content [1]. For example, in [2], the SIFT visual word quantization feature of the video frame is input into the support vector machine to complete the classification and labeling of the video. Literature [3] uses support vector machine to classify and label visual features such as HSI color features and shape features. Literature [4] extracts the color and texture feature vectors of sample key frames, conducts meta-learning strategy training, and then forms a visual dictionary. The above methods only perform semantic annotation through visual analysis, and none of them have the ability to fuse multi-source and multi-view features, and are not suitable for short video media with multi-source and multi-view descriptions.

Different from traditional video annotation, network short video annotation has its own characteristics. For example, because of its short video time, there is no need for lens segmentation; because of its fast sharing, it does not have text subtitles; because of its social attributes, it has multi-source information such as content introduction, user comments, etc., which are very important for short video content. Annotations have a greater effect.

Literature [1] Yin Wenjie, Han Junwei, Guo Lei. The latest progress in automatic annotation of images and videos [J]. Computer Science, 2011,38(12):12-16.

Literature [2] Wang Han, Wu Xinxiao, Jia Yunde. Video Annotation Using Heterogeneous Internet Image Groups [J]. Journal of Computer Science, 2013,36(10):2062-2069.

Literature [3] Zhang Jianming, Sun Chunmei, Yan Ting. Semi-supervised active learning video annotation based on adaptive SVM [J]. Computer Engineering, 2013,39(8):190-195.

Literature [4] Cui Tong, Xu Xin. A Big Data Video Annotation Method Based on Semantic Analysis [J]. Journal of Nanjing University of Aeronautics and Astronautics, 2016, 48(5).

Contents of the invention

In view of this, the present invention provides a short video automatic labeling method and system based on multi-source and multi-view transductive learning.

In the first aspect, the present invention provides a short video automatic tagging method based on multi-source multi-view transductive learning, including:

Obtain short video data;

Preprocessing the short video data to generate image keyframes, audio tracks, text and semantic labels in a consistent format;

Extracting the multi-view feature vectors of the key frame of the image, the audio track and the text;

Set up a short video annotation database, and the multi-view feature vector and the semantic label are stored in the short video annotation database;

Calculating the similarity between the multi-view feature vectors;

Establishing a multi-view fusion subspace through the similarity of the multi-view feature vectors;

Straightforwardly solving the multi-view fusion subspace, automatically marking the semantic label on the short video data to be marked;

Wherein, the short video data to be marked is short video data without semantic tags after the short video data is preprocessed;

Wherein, the short video data is multi-source information composed of visual images, audio, content introduction, and user comments.

Preferably, the calculation of the similarity of the multi-view feature vectors includes:

Calculate the distance between the multi-view feature vectors

Calculating the mean value σ _k of each view feature vector of the multi-view feature vector;

Using the distance between the multi-view feature vectors and the mean value σ _k of the feature vectors of each view, using the Gaussian kernel function similarity to calculate the similarity between the multi-view feature vectors in the interval from 0 to 1, the similarity of the k-th view constitutes the similarity of the k-th view degree matrix S _k ;

The calculation formula of the i-th row and j-th column element of the similarity matrix S _k of the k-th viewing angle is:

Wherein, X={x ₁ ,x ₂ ,...,x _N ,x _N+1 ,...,x _N+M } is the sample set of the short video data, and the short video with the semantic label The number of data is N, the number of short video samples to be marked is M, and the sample set is described using K perspectives, is the feature vector of the kth viewing angle of the i-th video, is the feature vector of the k-th view of the j-th video, 1≤k≤K, 1≤i≤N+M, 1≤j≤N+M.

Preferably, the establishment method of the multi-view fusion subspace includes:

Calculate the normalized Laplacian similarity matrix L(S _k ) of the similarity matrix S _k of the k-th viewing angle respectively, 1≤k≤K:

Wherein, I is (N+M)×(N+M) order unit matrix, is the degree matrix of the similarity matrix S _k of the k-th viewing angle, that is, the main diagonal element is the sum of the columns of S _k , and the non-main diagonal element is 0;

Calculate the trace normalization matrix of the normalized Laplacian similarity matrix L(S _k ):

Trace normalization processing of multi-view fusion subspace similarity matrix S ₀ :

in, is the linear combination coefficient of K views initialized randomly.

Preferably, the transductive method for solving the multi-view fusion subspace includes:

Construct the objective function to obtain the predicted label matrix f:

in, Be the label set that described semantic label forms, the sample x _i of described short video data has the jth label, then Y _ij =1, otherwise Y _ij =0, C is the scale of described label set;

in, is the predicted label matrix initialized randomly;

First, the objective function is transformed into

Then, fix f, solve for β, seek the partial derivative of Γ to β, make the partial derivative of Γ to β be 0, and obtain the linear combination coefficient β of the K viewing angles;

Finally, fix β and solve for f: obtain the partial derivative of Γ to f _i (1≤i≤N) and the partial derivative of Γ to f _i (N+1≤i≤N+M) respectively, let the Γ to The partial derivative of f _i (1≤i≤N) and the partial derivative of Γ to f _i (N+1≤i≤N+M) are 0, and the predicted label matrix f is obtained;

Repeat the steps of "fix f, solve β" and "fix β, solve f" alternately until the predicted label matrix f converges, and the linear combination coefficient β of the fusion subspace and the predicted label matrix f are obtained. Annotation of video data;

in,

in,

in, express the tth column of

Among them, δ is a non-negative Lagrangian multiplier.

Preferably, the parameters λ, μ, θ of the objective function are determined using 10-fold cross-validation.

In the second aspect, the present invention provides a short video automatic tagging system based on multi-source multi-view transductive learning, including:

A short video data acquisition unit, configured to acquire short video data;

The short video data preprocessing unit is respectively connected with the short video data acquisition unit and the short video annotation database unit, and is used to preprocess the short video data to generate consistent format image key frames, audio tracks, text and semantic labels , the semantic tags are stored in the database;

Extract the multi-view feature vector unit, which is respectively connected with the short video data preprocessing unit and the short video annotation database unit, for extracting the multi-view feature vector of the image key frame, audio track and text;

The short video annotation database unit is used to store the multi-view feature vector and the semantic label;

A multi-view feature vector similarity calculation unit is connected to the short video labeling database unit for calculating the similarity between the multi-view feature vectors;

The multi-view fusion subspace transductive solving unit is connected with the multi-view feature vector similarity calculation unit, and the multi-view fusion subspace is established through the multi-view feature vector similarity, and the multi-view fusion subspace is directly solved. Subspace, the semantic label is automatically marked on the short video data to be marked;

Wherein, the short video data to be marked is short video data without semantic tags after the short video data is preprocessed;

Wherein, the short video data is multi-source information composed of visual images, audio, content introduction, and user comments.

Preferably, the multi-view feature vector similarity calculation unit includes:

The distance between multi-view feature vectors Calculation module, used to calculate the distance between the multi-view feature vectors

The mean σ _k calculation module of each view feature vector of the multi-view feature vector is used to calculate the mean σ _k of each view feature vector of the multi-view feature vector;

The multi-view feature vector similarity calculation module, the distance between the multi-view feature vector and the multi-view feature vector The calculation module is connected to the mean value σ _k calculation module of each view feature vector of the multi-view feature vector, and the similarity between the multi-view feature vectors in the 0 to 1 interval is calculated by using the Gaussian kernel function similarity, and the similarity of the kth view is degree to form the similarity matrix S _k of the kth viewing angle;

The calculation formula of the i-th row and j-th column element of the similarity matrix S _k of the k-th viewing angle is:

Wherein, X={x ₁ ,x ₂ ,...,x _N ,x _N+1 ,...,x _N+M } is the sample set of the short video data, and the short video with the semantic label The number of data is N, the number of short video samples to be marked is M, and the sample set is described using K perspectives, is the feature vector of the kth viewing angle of the i-th video, is the feature vector of the k-th view of the j-th video, 1≤k≤K, 1≤i≤N+M, 1≤j≤N+M.

Preferably, the multi-view fusion subspace direct calculation unit includes:

The multi-view fusion subspace establishment module, the multi-view fusion subspace establishment module is as follows for the establishment process of the multi-view fusion subspace:

Calculate the normalized Laplacian similarity matrix L(S _k ) of the similarity matrix S _k of the k-th viewing angle respectively, 1≤k≤K:

Wherein, I is (N+M)×(N+M) order unit matrix, is the degree matrix of the similarity matrix S _k of the k-th viewing angle, that is, the main diagonal element is the sum of the columns of S _k , and the non-main diagonal element is 0;

Calculate the trace normalization matrix of the normalized Laplacian similarity matrix L(S _k ):

Trace normalization processing of multi-view fusion subspace similarity matrix S ₀ :

in, is the linear combination coefficient of K views initialized randomly.

Preferably, the multi-view fusion subspace direct calculation unit further includes:

The multi-view fusion subspace solving module is connected with the multi-view fusion subspace module, and the multi-view fusion subspace solving process of the multi-view fusion subspace by the multi-view fusion subspace is as follows:

Construct the objective function to obtain the predicted label matrix f:

in, Be the label set that described semantic label forms, the sample x _i of described short video data has the jth label, then Y _ij =1, otherwise Y _ij =0, C is the scale of described label set;

in, is the predicted label matrix initialized randomly;

First, the objective function is transformed into

Then, fix f, solve for β, seek the partial derivative of Γ to β, make the partial derivative of Γ to β be 0, and obtain the linear combination coefficient β of the K viewing angles;

Finally, fix β and solve for f: obtain the partial derivative of Γ to f _i (1≤i≤N) and the partial derivative of Γ to f _i (N+1≤i≤N+M) respectively, let the Γ to The partial derivative of f _i (1≤i≤N) and the partial derivative of Γ to f _i (N+1≤i≤N+M) are 0, and the predicted label matrix f is obtained;

Repeat the steps of "fix f, solve β" and "fix β, solve f" alternately until the predicted label matrix f converges, and the linear combination coefficient β of the fusion subspace and the predicted label matrix f are obtained. Annotation of video data;

in,

in,

in, express the tth column of

Among them, δ is a non-negative Lagrangian multiplier.

Preferably, the parameters λ, μ, θ of the objective function are determined using 10-fold cross-validation.

The present invention has at least the following beneficial effects:

The present invention considers the multi-source information attached to the short video data, acquires short video data with and without semantic tags at the same time, preprocesses the short video data, and generates image key frames, audio tracks, text and Semantic tags, which automatically tag the short video data to be tagged without semantic tags, improving the accuracy of tagging.

Description of drawings

Through the following description of the embodiments of the present invention with reference to the accompanying drawings, the above-mentioned and other objects, features and advantages of the present invention are more clear, in the accompanying drawings:

Fig. 1 is a flow chart of a short video automatic tagging method based on multi-source multi-view transductive learning according to an embodiment of the present invention;

Fig. 2 is a schematic structural diagram of a short video annotation database according to an embodiment of the present invention;

Fig. 3 is a functional block diagram of a short video automatic tagging system based on multi-source multi-view transductive learning according to an embodiment of the present invention.

Detailed ways

The present invention will be described below based on examples, but it should be noted that the present invention is not limited to these examples. In the following detailed description of the invention, some specific details are set forth in detail. However, the present invention can be fully understood by those skilled in the art about the parts that are not described in detail.

In addition, those of ordinary skill in the art should understand that the provided drawings are only for illustrating the objects, features and advantages of the present invention, and the drawings are not actually drawn to scale.

At the same time, unless the context clearly requires, the words "include", "include" and other similar words in the entire specification and claims should be interpreted as an inclusive meaning rather than an exclusive or exhaustive meaning; that is, "include but not limited to the meaning of ".

FIG. 1 is a flow chart of a short video automatic labeling method based on multi-source and multi-view transductive learning according to an embodiment of the present invention. As shown in Figure 1, a short video automatic labeling method based on multi-source and multi-view transductive learning includes step 101 to obtain short video data, step 102 to preprocess short video data, step 103 to extract multi-view feature vectors, and step 104 to establish The short video annotation database, step 105 calculates the similarity of multi-view feature vectors and step 106 transductively solves the multi-view fusion subspace.

Step 101 Acquire short video data: download training samples (ie short video data) from the short video resource website, extract information and store it in a buffer file. The implementation process is as follows: first initialize the website link and access rights; initialize the query label queue; initialize the number of threads, generate a download queue, and download videos, comments, and introductions in a loop, and write them into buffer files in the form of streams. The above is data capture The engine adopts the specific implementation manner implemented by Python, and the present invention does not limit the method of obtaining short video data.

Among them, the captured training samples are short video files, content introduction, user comments and other multi-source information.

Step 102 Short video data preprocessing: Short video data preprocessing completes video processing, audio processing, and text processing of training samples (ie, short video data) and short video data to be labeled, and generates image key frames, audio tracks, and semantic labels respectively. Among them, image key frames and audio tracks enter step 103 for multi-view feature vector extraction, and semantic tags directly enter step 104 to establish a short video annotation database and store them in the short video annotation database.

Wherein, there is short video data without semantic labels in the short video data, by step 103 multi-view feature vector extraction, step 104 establishment of short video annotation database, step 105 calculation of multi-view feature vector similarity and step 106 transductive solution to multiple The perspective fusion subspace carries out semantic tagging on the short video data without semantic tags, and completes the automatic tagging of short video data without semantic tags.

Further, for each short video file, take every 20 frames as a unit, take the 10th frame as a key frame, convert the generated key frame sequence to a consistent size, such as 640×360, and store it in the buffer with the data structure of CV_32FC3 zone file.

Further, extract the audio track from each short video file, and use FFmpeg tool to extract the audio track with a fixed 16kHz sampling rate and 16bit quantization. The obtained single-channel audio is stored in a buffer file in a pulse code modulation (PCM, Pulse Code Modolation) format.

Further, for the video description and comment content, the NLP TOOLS of Stanford University is used to segment the video description and comment, stem the word, and filter the stop words, and use the WordNet dictionary to filter and generate the initial semantic label.

Further, the short video data is preprocessed to generate image key frames, audio tracks and semantic tags, and the semantic tags directly enter step 104 to establish a short video tagging database and store them in the short video tagging database.

Step 103 Multi-view feature vector extraction: the multi-view feature vector extraction extracts visual view features, audio view features, and text view features of the short video data respectively, and the specific extraction methods are as follows.

Step 103 Multi-view feature vector extraction: Multi-view feature vector extraction extracts the visual view features, audio view features and text view features of the short video data in the training set and the short video data to be labeled respectively. The specific extraction methods are as follows.

For each image key frame, visual features of the following perspectives are extracted: (1) 225-dimensional color moment features after 5×5 blocks; (2) 144-dimensional HSV correlation map; (3) 128-dimensional wavelet texture; (4) 64 (5) 75-dimensional edge distribution histogram; (6) 16-dimensional co-occurrence texture features; (7) using the literature "Caffe: Convolutional architecture for fastfeature embedding (Jia, Yangqing, Evan Shelhamer, Jeff Donahue, Sergey Karayev , Jonathan Long, Ross Girshick, Sergio Guadarrama, and Trevor Darrell. In Proceedings of the 22nd ACM international conference on Multimedia, pp.675-678. ACM, 2014)" method to extract 1000-dimensional softmax layer object features; (8) Using the literature "Large-scale visual sentiment ontology and detectors using adjective noun pairs. (Borth, Damian, Rongrong Ji, Tao Chen, Thomas Breuel, and Shih-Fu Chang. In Proceedings of the 21st ACM international conference on Multimedia, pp.223- 232.ACM,2013.)", using the emotional feature pre-training model on the SentiBank dataset to extract the 2089-dimensional emotional features of key frames.

The tool Sentence2vec is used to extract 100-dimensional text description features.

Extract audio perspective features: First, use the tool FFmpeg to take 16kHz as the sampling frequency, take 20 milliseconds to sample and extract the single-channel audio of the audio track, and then use the Hamming window to window the audio signal to divide the frequency band from 0 to 8kHz There are 7 sub-bands, and the literature "Design, analysis and experimental evaluation of blockbased transformation in MFCC computation for speaker recognition. (Sahidullah, Md, and Goutam Saha. Speech Communication 54, no.4, 2012:543-565.)" The method extracts 14-dimensional Mel cepstrum parameter features, and extracts 276-dimensional audio energy features, 170-dimensional audio spectral centroid features, 169-dimensional spectral time characteristic features and 80-dimensional audio signature features.

Further, the multi-view feature vector obtained in step 103 is stored in the short video annotation database.

Step 104 establishes a short video annotation database: the short video annotation database has a user information table SeedUserTable, a feature encoding table ViewCodeTable, a video file table VideoDescriptionTable, a multi-view feature table MultiviewTable, a tag vocabulary TagTable, and a video tag table VideoTagTable, a total of 6 Table, see the description in Figure 2 for details.

Further, the implementation of the database adopts MongoDB NoSQL distributed storage database.

Furthermore, the contents of SeedUserTable, KeyWordsForDownloadTable, ViewCodeTable, and TagTable need to be pre-entered with data and called by the data capture engine.

Furthermore, the semantic tags generated by the short video data preprocessing in step 102, the visual features extracted from the multi-view feature vector in step 103, the audio track view features, and the text description features are stored in the multi-view feature table MultiviewTable of the short video annotation database.

Step 105 Calculate the similarity of multi-view feature vectors: Calculate the similarity matrix of each view according to the extracted multi-view features, perform Laplacian normalization processing, determine the optimization target, and perform transductive solution.

For the convenience of description, the following notation is given:

A sample set X of short video data={x ₁ , x ₂ ,...,x _N , x _N+1 ,...,x _N+M }, wherein the semantic tags are included in the short video annotation database The number of (training set samples) in the short video labeling database is N, the number of test samples of the short video data to be labeled is M, and K perspectives are used to describe the sample set.

is a label set (training set label matrix) composed of semantic labels, if x _i has the jth label, then Y _ij =1, otherwise Y _ij =0, C is the size of the label set.

is a randomly initialized predicted label matrix.

Among them, the K views are the multiple visual features, audio features and text description features extracted in step 103 through multi-view feature vector extraction.

Calculate the similarity between the feature vectors of each view. First calculate the distance between the eigenvectors, then calculate the mean value of the eigenvectors, and then use the Gaussian kernel function similarity to calculate the similarity of the eigenvectors in the interval from 0 to 1, and construct the similarity matrix of the corresponding neighbor graph.

Calculate the distance between multi-view feature vectors

in, is the feature vector of the kth viewing angle of the i-th video, as a vector The l-th dimension vector value of ;

in, is the feature vector of the kth viewing angle of the jth video, as a vector The l-th dimension vector value of ; m _k is the length of the feature vector of the k-th view.

in, as a vector 1 norm of , as a vector 1 norm of .

Calculate the eigenvector mean σ _k of the kth view:

The calculation formula of the i-th row and j-th column element of the similarity matrix S _k of the k-th viewing angle is as follows:

Step 106 Transductively solve the multi-view fusion subspace: first calculate the normalized Laplacian similarity matrix of K views, then randomly initialize the prediction matrix f and the combination coefficient β of K views, and then solve the prediction matrix by using an alternate optimization strategy f and the combination coefficient β until the objective function converges.

The formula for calculating the normalized Laplacian similarity matrix L(S _k ) of the k-th viewing angle is as follows:

Wherein, I is (N+M)×(N+M) order unit matrix, is the degree matrix of the similarity matrix S _k of the k-th viewing angle, that is, the main diagonal elements are the sum of the columns of S _k , and the non-main diagonal elements are 0.

According to the calculated Laplacian similarity matrix, the fusion subspace is calculated, and the labels of M samples in the test set are derived. The specific steps are as follows: Calculate the trace normalization matrix of the k-th viewing angle according to the Laplacian similarity matrix, and the calculation formula is as follows:

The trace normalization expression formula of the multi-view fusion subspace similarity matrix _S0 is as follows:

in, The linear combination coefficients initialized randomly. In order to obtain the predicted label matrix f, the following objective function is calculated:

The first term of this function constrains the predicted labels of the training set videos to be consistent with their original labels. The second term of the function constrains the multi-view fusion subspace and the original space to have equidistant properties. The third item of the function constrains samples with high similarity, and their labels are similar. The fourth constraint of this function is the sparsity constraint of the combination coefficient. In order to prevent overfitting, each parameter in the objective function is determined by 10-fold cross-validation. In this implementation, the parameters are set as follows λ=1, μ=0.01, θ=100,

The prediction matrix f and the combination coefficient β are randomly initialized, and then the prediction matrix f and the combination coefficient β are solved by an alternate optimization strategy until the objective function converges. The solution steps are as follows.

First fix f and solve for β.

The objective function is transformed into:

in,

express The tth column of . Will replace with According to the Lagrange multiplier method, the following formula is obtained:

Among them, δ is a non-negative Lagrangian multiplier. Find the partial derivative of Γ with respect to β, and get the following formula:

in, is a positive definite matrix. I is the unit matrix of K×K, let the partial derivative of Γ with respect to β be 0, and get β=H ^-1 (δe-μg), put this formula into e ^T β=1, and get the solution formula of β:

Since H is positive definite, H ^-1 is also a positive definite matrix, so e ^T H ^-1 e>0.

Then, fix β and solve f, the specific steps are as follows:

Γ calculates the partial derivative with respect to f _i (1≤i≤N), and obtains:

Γ calculates the partial derivative with respect to f _i (N+1≤i≤N+M), and obtains:

Respectively let the partial derivative of Γ with respect to f _i (1≤i≤N) and the partial derivative of Γ with respect to f _i (N+1≤i≤N+M) be 0, and get

in,

in,

Repeat the steps of "fix f, solve β" and "fix β, solve f" alternately until the predicted label matrix f converges, and the linear combination coefficient β of the fusion subspace and the predicted label matrix f are obtained. Annotation of video data.

Fig. 2 is a schematic structural diagram of a short video annotation database according to an embodiment of the present invention. As shown in Figure 2, the short video annotation database has six tables: user information table SeedUserTable, feature coding table ViewCodeTable, video file table VideoDescriptionTable, multi-view feature table MultiviewTable, tag vocabulary table TagTable, and video tag table VideoTagTable. Video file table VideoDescriptionTable includes video ID VideoID, network link ID WebSiteID, video introduction VideoDescription and video storage path VideoPath, the primary key PK is VideoID. The feature coding table ViewCodeTable includes the view ID ViewID and the feature name ViewName, and the primary key PK is ViewID. The multi-view feature table MultiViewTable includes the video identifier VideoID, the view ID ViewID, and the multi-view feature vector ViewValue, and the primary key PK is VideoID. The video tag table VideoTagTable stores the tag vector of the short video, including the video identifier VideoID and the tag column TagVector, and the primary key PK is VideoID. The tag vocabulary TagTable contains the tag identifier TagID and the tag text TagText, and the primary key PK is TagID. The user information table SeedUserTable includes the network link identifier WebSiteID, the website address WebSite, the login name Username and the password Password, and the primary key PK is WebSiteID.

Fig. 2 is a schematic structural diagram of a short video annotation database according to an embodiment of the present invention. As shown in Figure 2, the short video annotation database has six tables: user information table SeedUserTable, feature coding table ViewCodeTable, video file table VideoDescriptionTable, multi-view feature table MultiviewTable, tag vocabulary table TagTable, and video tag table VideoTagTable. Video file table VideoDescriptionTable includes video ID VideoID, network link ID WebSiteID, video introduction VideoDescription and video storage path VideoPath, the primary key PK is VideoID. The feature coding table ViewCodeTable includes the view ID ViewID and the feature name ViewName, and the primary key PK is ViewID. The multi-view feature table MultiViewTable includes the video identifier VideoID, the view ID ViewID, and the multi-view feature vector ViewValue, and the primary key PK is VideoID. The video tag table VideoTagTable stores the tag vector of the short video, including the video identifier VideoID and the tag column TagVector, and the primary key PK is VideoID. The tag vocabulary TagTable contains the tag identifier TagID and the tag text TagText, and the primary key PK is TagID. The user information table SeedUserTable includes the network link identifier WebSiteID, the website address WebSite, the login name Username and the password Password, and the primary key PK is WebSiteID.

Fig. 3 is a functional block diagram of a short video automatic tagging system based on multi-source multi-view transductive learning according to an embodiment of the present invention. A short video automatic tagging system based on multi-source multi-view transductive learning includes: short video data acquisition unit 301, short video data preprocessing unit 302, multi-view feature vector extraction unit 303, short video tagging database unit 304, multiple A perspective feature vector similarity calculation unit 305 and a multi-view fusion subspace direct calculation unit 306 .

In FIG. 3 , the short video data acquisition unit 301 is used to acquire short video data. The method for acquiring video by the short video data acquisition unit 301 is detailed in the description of acquiring short video data in step 101 in FIG. 1 .

In Fig. 3, the short video data preprocessing unit 302 is connected with the short video data acquisition unit 301 and the short video annotation database unit 304 respectively, and is used to preprocess the short video data to generate consistent format image key frames, audio tracks, text and semantic tags, semantic tags are stored in the database, the short video data preprocessing unit 302 preprocessing method for short video data refer to the description of step 102 short video data preprocessing in Figure 1.

In Fig. 3, extract multi-view feature vector unit 303, connect with short video data preprocessing unit 302 and short video label database unit 304 respectively, be used to extract the multi-view feature vector of described image key frame, audio track and text, extract The multi-view feature vector extraction method of the image key frame, audio track and text of the multi-view feature vector unit 303 is detailed in the description of the multi-view feature vector extraction in step 103 in FIG. 1 .

In FIG. 3 , the short video annotation database unit 304 is used to store multi-view feature vectors and the semantic tags. The database structure of the short video annotation database unit 304 is detailed in the description in FIG. 2 .

In FIG. 3 , the multi-view feature vector similarity calculation unit 305 is connected to the short video annotation database unit 304 for calculating the similarity between the multi-view feature vectors.

In Fig. 3, the multi-view fusion subspace transductive solution unit 306 is connected with the multi-view feature vector similarity calculation unit 305, through the multi-view feature vector similarity, the multi-view fusion subspace is established, and the multi-view fusion subspace is solved directly. The perspective fusion subspace automatically annotates semantic labels on the short video data to be annotated.

Wherein, the short video data to be marked is the short video data without semantic tags after the short video data has been preprocessed.

Among them, the short video data is multi-source information composed of visual images, audio, content introduction, and user comments.

Further, the multi-view feature vector similarity calculation unit 305 includes: the distance between the multi-view feature vectors Calculation module, mean value σ _k calculation module of each view feature vector of multi-view feature vector and multi-view feature vector similarity calculation module.

The distance between multi-view feature vectors Calculation module, used to calculate the distance between multi-view feature vectors

in, is the feature vector of the kth viewing angle of the i-th video, as a vector The l-th dimension vector value of .

in, is the feature vector of the kth viewing angle of the jth video, as a vector The l-th dimension vector value of ; m _k is the length of the feature vector of the k-th view.

in, as a vector 1 norm of , as a vector 1 norm of .

The mean σ _k calculation module of each view feature vector of the multi-view feature vector is used to calculate the mean σ _k of each view feature vector of the multi-view feature vector,

Multi-view feature vector similarity calculation module, respectively, and the distance between the multi-view feature vector The calculation module is connected to the mean value σ _k calculation module of each view feature vector of the multi-view feature vector, and the similarity between the multi-view feature vectors in the 0 to 1 interval is calculated by using the Gaussian kernel function similarity, and the similarity of the kth view is degree to form the similarity matrix S _k of the kth viewing angle;

The calculation formula of the i-th row and j-th column element of the similarity matrix S _k of the k-th viewing angle is:

Among them, X={x ₁ ,x ₂ ,...,x _N ,x _N+1 ,...,x _N+M } is a sample set of short video data, and the number of short video data with semantic labels is N, the number of short video samples to be labeled is M, and K perspectives are used to describe the sample set, is the feature vector of the kth viewing angle of the i-th video, is the feature vector of the k-th view of the j-th video, 1≤k≤K, 1≤i≤N+M, 1≤j≤N+M.

Furthermore, the multi-view fusion subspace direct calculation unit 306 further includes: a multi-view fusion subspace establishment module and a multi-view fusion subspace calculation module.

Multi-view fusion subspace building module:

Calculate the normalized Laplacian similarity matrix L(S _k ) of the similarity matrix S _k of the k-th viewing angle, 1≤k≤K:

Wherein, I is (N+M)×(N+M) order unit matrix, is the degree matrix of the similarity matrix S _k of the k-th viewing angle, that is, the main diagonal elements are the sum of the columns of S _k , and the non-main diagonal elements are 0.

Compute the trace normalization matrix of the normalized Laplacian similarity matrix L(S _k ):

Trace normalization processing of multi-view fusion subspace similarity matrix S ₀ :

in, is the linear combination coefficient of K views initialized randomly.

Multi-view fusion subspace solving module, establish module connection with multi-view fusion subspace:

Construct the objective function to obtain the predicted label matrix f:

in, is a tag set composed of semantic tags, if the sample x _i of short video data has the jth tag, then Y _ij =1, otherwise Y _ij =0, C is the size of the tag set;

in, is the predicted label matrix initialized randomly;

first, express The tth column of . Will replace with According to the Lagrange multiplier method, the following formula is obtained:

Find the partial derivative of Γ with respect to β, and get the following formula:

in, is a positive definite matrix. I is the unit matrix of K×K, let the partial derivative of Γ with respect to β be 0, and get β=H ^-1 (δe-μg), put this formula into e ^T β=1, and get the solution formula of β:

Since H is positive definite, H ^-1 is also a positive definite matrix, so e ^T H ^-1 e>0.

Then, fix β and solve f, the specific steps are as follows:

Γ calculates the partial derivative with respect to f _i (1≤i≤N), and obtains:

Γ calculates the partial derivative with respect to f _i (N+1≤i≤N+M), and obtains:

Respectively let the partial derivative of Γ with respect to f _i (1≤i≤N) and the partial derivative of Γ with respect to f _i (N+1≤i≤N+M) be 0, and get

in,

in,

Repeat the steps of "fix f, solve β" and "fix β, solve f" alternately until the predicted label matrix f converges, and the linear combination coefficient β of the fusion subspace and the predicted label matrix f are obtained. Annotation of video data.

in,

in,

in, express the tth column of

Among them, δ is a non-negative Lagrangian multiplier.

Obviously, those skilled in the art should understand that each module or each step of the above-mentioned present invention can be realized by a general-purpose computing device, and they can be concentrated on a single computing device, or distributed in a network formed by multiple computing devices Optionally, they can be implemented with program codes executable by a computing device, so that they can be stored in a storage device and executed by a computing device, or they can be made into individual integrated circuit modules, or they can be integrated into Multiple modules or steps are fabricated into a single integrated circuit module to realize. As such, the present invention is not limited to any specific combination of hardware and software.

The above-mentioned embodiments are only to express the implementation of the present invention, and the descriptions thereof are more specific and detailed, but should not be construed as limiting the patent scope of the present invention. It should be noted that those skilled in the art can make several modifications, equivalent replacements, improvements, etc. without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent for the present invention should be based on the appended claims.

Claims (2)

1. A method for automatic labeling of short videos based on multi-source and multi-view transductive learning, characterized in that it includes: Obtain short video data; Preprocessing the short video data to generate image keyframes, audio tracks, text and semantic labels in a consistent format; Extracting the multi-view feature vectors of the key frame of the image, the audio track and the text; Set up a short video annotation database, and the multi-view feature vector and the semantic label are stored in the short video annotation database; Calculating the similarity between the multi-view feature vectors; Establishing a multi-view fusion subspace through the similarity of the multi-view feature vectors; Straightforwardly solving the multi-view fusion subspace, automatically marking the semantic label on the short video data to be marked; Wherein, the short video data to be marked is short video data without semantic tags after the short video data is preprocessed; Wherein, the short video data is multi-source information composed of visual images, audio, content introduction, and user comments; Further, the calculation of the similarity of the multi-view feature vectors includes: Calculate the distance between the multi-view feature vectors Calculating the mean value σ _k of each view feature vector of the multi-view feature vector; Using the distance between the multi-view feature vectors and the mean value σ _k of the feature vectors of each view, using the Gaussian kernel function similarity to calculate the similarity between the multi-view feature vectors in the interval from 0 to 1, the similarity of the k-th view constitutes the similarity of the k-th view degree matrix S _k ; The calculation formula of the i-th row and j-th column element of the similarity matrix S _k of the k-th viewing angle is: <mrow><msub><mi>S</mi><mi>k</mi></msub><mrow><mo>(</mo><mi>i</mi><mo>,</mo><mi>j</mi><mo>)</mo></mrow><mo>=</mo><mfenced open = "{" close = ""><mtable><mtr><mtd><mrow><mi>exp</mi><mrow><mo>(</mo><mo>-</mo><mfrac><mrow><mo>|</mo><mo>|</mo><mi>d</mi><mi>i</mi><mi>s</mi><mi>t</mi><mrow><mo>(</mo><msubsup><mi>x</mi><mi>k</mi><mi>i</mi></msubsup><mo>,</mo><msubsup><mi>x</mi><mi>k</mi><mi>j</mi></msubsup><mo>)</mo></mrow><mo>|</mo><msup><mo>|</mo><mn>2</mn></msup></mrow><mrow><mn>2</mn><msubsup><mi>&amp;sigma;</mi><mi>k</mi><mn>2</mn></msubsup></mrow></mfrac><mo>)</mo></mrow><mo>,</mo></mrow></mtd><mtd><mrow><mi>i</mi><mi>f</mi></mrow></mtd><mtd><mrow><mi>i</mi><mo>&amp;NotEqual;</mo><mi>j</mi></mrow></mtd></mtr><mtr><mtd><mrow><mn>0</mn><mo>,</mo></mrow></mtd><mtd><mrow><mi>i</mi><mi>f</mi></mrow></mtd><mtd><mrow><mi>i</mi><mo>=</mo><mi>j</mi></mrow></mtd></mtr></mtable></mfenced><mo>;</mo></mrow> Wherein, X={x ₁ ,x ₂ ,...,x _N ,x _N+1 ,...,x _N+M } is the sample set of the short video data, and the short video with the semantic label The number of data is N, the number of short video samples to be marked is M, and the sample set is described using K perspectives, is the feature vector of the kth viewing angle of the i-th video, is the feature vector of the kth view of the jth video, 1≤k≤K, 1≤i≤N+M, 1≤j≤N+M; Further, the establishment method of the multi-view fusion subspace includes: Calculate the normalized Laplacian similarity matrix L(S _k ) of the similarity matrix S _k of the k-th viewing angle respectively, 1≤k≤K: <mrow><mi>L</mi><mrow><mo>(</mo><msub><mi>S</mi><mi>k</mi></msub><mo>)</mo></mrow><mo>=</mo><mi>I</mi><mo>-</mo><msubsup><mi>D</mi><mi>k</mi><mrow><mo>-</mo><mfrac><mn>1</mn><mn>2</mn></mfrac></mrow></msubsup><msub><mi>S</mi><mi>k</mi></msub><msubsup><mi>D</mi><mi>k</mi><mrow><mo>-</mo><mfrac><mn>1</mn><mn>2</mn></mfrac></mrow></msubsup><mo>;</mo></mrow> Wherein, I is (N+M)×(N+M) order unit matrix, is the degree matrix of the similarity matrix S _k of the k-th viewing angle, that is, the main diagonal element is the sum of the columns of S _k , and the non-main diagonal element is 0; Calculate the trace normalization matrix of the normalized Laplacian similarity matrix L(S _k ): <mrow><mover><msub><mi>L</mi><mi>k</mi></msub><mo>~</mo></mover><mo>=</mo><mfrac><mn>1</mn><mrow><mi>t</mi><mi>r</mi><mrow><mo>(</mo><mi>L</mi><mo>(</mo><msub><mi>S</mi><mi>k</mi></msub><mo>)</mo><mo>)</mo></mrow></mrow></mfrac><mi>L</mi><mrow><mo>(</mo><msub><mi>S</mi><mi>k</mi></msub><mo>)</mo></mrow><mo>;</mo></mrow> Trace normalization processing of multi-view fusion subspace similarity matrix S ₀ : <mrow><msub><mover><mi>L</mi><mo>~</mo></mover><mn>0</mn></msub><mo>=</mo><mfrac><mn>1</mn><mrow><mi>t</mi><mi>r</mi><mrow><mo>(</mo><mi>L</mi><mo>(</mo><msub><mi>S</mi><mn>0</mn></msub><mo>)</mo><mo>)</mo></mrow></mrow></mfrac><mi>L</mi><mrow><mo>(</mo><msub><mi>S</mi><mn>0</mn></msub><mo>)</mo></mrow><mo>=</mo><munderover><mo>&amp;Sigma;</mo><mrow><mi>k</mi><mo>=</mo><mn>1</mn></mrow><mi>K</mi></munderover><msub><mi>&amp;beta;</mi><mi>k</mi></msub><mover><msub><mi>L</mi><mi>k</mi></msub><mo>~</mo></mover><mo>,</mo><munderover><mo>&amp;Sigma;</mo><mrow><mi>k</mi><mo>=</mo><mn>1</mn></mrow><mi>K</mi></munderover><msub><mi>&amp;beta;</mi><mi>k</mi></msub><mo>=</mo><mn>1</mn><mo>;</mo></mrow> in, is the linear combination coefficient of randomly initialized K viewing angles; Further, the transductive method for solving the multi-view fusion subspace includes: Construct the objective function to obtain the predicted label matrix f: <mrow><munder><mi>min</mi><mrow><mi>f</mi><mo>,</mo><mi>&amp;beta;</mi></mrow></munder><munderover><mo>&amp;Sigma;</mo><mrow><mi>i</mi><mo>=</mo><mn>1</mn></mrow><mi>N</mi></munderover><mo>|</mo><mo>|</mo><mrow><mo>(</mo><msub><mi>y</mi><mi>i</mi></msub><mo>-</mo><msub><mi>f</mi><mi>i</mi></msub><mo>)</mo></mrow><mo>|</mo><msup><mo>|</mo><mn>2</mn></msup><mo>+</mo><mi>&amp;lambda;</mi><munderover><mo>&amp;Sigma;</mo><mrow><mi>k</mi><mo>=</mo><mn>1</mn></mrow><mi>K</mi></munderover><mo>|</mo><mo>|</mo><munderover><mo>&amp;Sigma;</mo><mrow><mi>i</mi><mo>=</mo><mn>1</mn></mrow><mi>N</mi></munderover><msub><mi>&amp;beta;</mi><mi>i</mi></msub><mover><msub><mi>L</mi><mi>i</mi></msub><mo>~</mo></mover><mo>-</mo><mover><msub><mi>L</mi><mi>k</mi></msub><mo>~</mo></mover><mo>|</mo><msubsup><mo>|</mo><mi>F</mi><mn>2</mn></msubsup><mo>+</mo><mi>&amp;mu;</mi><mo>|</mo><mo>|</mo><msup><mi>f</mi><mi>T</mi></msupup><munderover><mo>&amp;Sigma;</mo><mrow><mi>k</mi><mo>=</mo><mn>1</mn></mrow><mi>K</mi></munderover><msub><mi>&amp;beta;</mi><mi>i</mi></msub><mover><msub><mi>L</mi><mi>k</mi></msub><mo>~</mo></mover><mi>f</mi><mo>|</mo><msubsup><mo>|</mo><mi>F</mi><mn>2</mn></msubsup><mo>+</mo><mi>&amp;theta;</mi><mo>|</mo><mo>|</mo><mi>&amp;beta;</mi><mo>|</mo><msup><mo>|</mo><mn>2</mn></msup><mo>,</mo><msup><mi>e</mi><mi>T</mi></msup><mi>&amp;beta;</mi><mo>=</mo><mn>1</mn><mo>;</mo></mrow> in, Be the label set that described semantic label forms, the sample x _i of described short video data has the jth label, then Y _ij =1, otherwise Y _ij =0, C is the scale of described label set; in, is the predicted label matrix initialized randomly; First, the objective function is transformed into <mrow><mi>&amp;Gamma;</mi><mo>=</mo><munder><mrow><mi>m</mi><mi>i</mi><mi>n</mi></mrow><mi>&amp;beta;</mi></munder><mo>{</mo><mi>&amp;lambda;</mi><munderover><mo>&amp;Sigma;</mo><mrow><mi>k</mi><mo>=</mo><mn>1</mn></mrow><mi>K</mi></munderover><munderover><mo>&amp;Sigma;</mo><mrow><mi>t</mi><mo>=</mo><mn>1</mn></mrow><mrow><mi>N</mi><mo>+</mo><mi>M</mi></mrow></munderover><mo>|</mo><mo>|</mo><mrow><mo>(</mo><msup><mi>M</mi><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></msup><mo>-</mo><mover><msubsup><mi>l</mi><mi>k</mi><mi>t</mi></msubsup><mo>~</mo></mover><msup><mi>e</mi><mi>T</mi></msup><mo>)</mo></mrow><mi>&amp;beta;</mi><mo>|</mo><msubsup><mo>|</mo><mi>F</mi><mn>2</mn></msubsup><mo>+</mo><msup><mi>&amp;mu;g</mi><mi>T</mi></msup><mi>&amp;beta;</mi><mo>+</mo><mi>&amp;delta;</mi><mrow><mo>(</mo><mn>1</mn><mo>-</mo><msup><mi>e</mi><mi>T</mi></msup><mi>&amp;beta;</mi><mo>)</mo></mrow><mo>+</mo><mi>&amp;theta;</mi><mo>|</mo><mo>|</mo><mi>&amp;beta;</mi><mo>|</mo><msup><mo>|</mo><mn>2</mn></msup><mo>}</mo><mo>;</mo></mrow> Then, fix f, solve for β, seek the partial derivative of Γ to β, make the partial derivative of Γ to β be 0, and obtain the linear combination coefficient β of the K viewing angles; Finally, fix β and solve for f: obtain the partial derivative of Γ to f _i (1≤i≤N) and the partial derivative of Γ to f _i (N+1≤i≤N+M) respectively, let the Γ to The partial derivative of f _i (1≤i≤N) and the partial derivative of Γ to f _i (N+1≤i≤N+M) are 0, and the predicted label matrix f is obtained; Repeat the steps of "fix f, solve β" and "fix β, solve f" alternately until the predicted label matrix f converges, and the linear combination coefficient β of the fusion subspace and the predicted label matrix f are obtained. Annotation of video data; in, in, in, express the tth column of Among them, δ is a non-negative Lagrangian multiplier; Further, the parameters λ, μ, θ of the objective function are determined by using 10-fold cross-validation. 2. A short video automatic labeling system based on multi-source and multi-view transductive learning, characterized in that it includes: A short video data acquisition unit, configured to acquire short video data; The short video data preprocessing unit is respectively connected with the short video data acquisition unit and the short video annotation database unit, and is used to preprocess the short video data to generate consistent format image key frames, audio tracks, text and semantic labels , the semantic tags are stored in the database; Extract the multi-view feature vector unit, which is respectively connected with the short video data preprocessing unit and the short video annotation database unit, for extracting the multi-view feature vector of the image key frame, audio track and text; The short video annotation database unit is used to store the multi-view feature vector and the semantic label; A multi-view feature vector similarity calculation unit is connected to the short video labeling database unit for calculating the similarity between the multi-view feature vectors; The multi-view fusion subspace transductive solving unit is connected with the multi-view feature vector similarity calculation unit, and the multi-view fusion subspace is established through the multi-view feature vector similarity, and the multi-view fusion subspace is directly solved. Subspace, the semantic label is automatically marked on the short video data to be marked; Wherein, the short video data to be marked is short video data that does not have semantic tags after the short video data is preprocessed; Wherein, the short video data is multi-source information composed of visual images, audio, content introduction, and user comments; Further, the multi-view feature vector similarity calculation unit includes: The distance between multi-view feature vectors Calculation module, used to calculate the distance between the multi-view feature vectors The mean σ _k calculation module of each view feature vector of the multi-view feature vector is used to calculate the mean σ _k of each view feature vector of the multi-view feature vector; The multi-view feature vector similarity calculation module, the distance between the multi-view feature vector and the multi-view feature vector The calculation module is connected with the mean value σ _k calculation module of each view feature vector of the multi-view feature vector, and the similarity between the multi-view feature vectors in the 0 to 1 interval is calculated by using the Gaussian kernel function similarity, and the similarity of the kth view is degree to form the similarity matrix S _k of the kth viewing angle; The calculation formula of the i-th row and j-th column element of the similarity matrix S _k of the k-th viewing angle is: <mrow><msub><mi>S</mi><mi>k</mi></msub><mrow><mo>(</mo><mi>i</mi><mo>,</mo><mi>j</mi><mo>)</mo></mrow><mo>=</mo><mfenced open = "{" close = ""><mtable><mtr><mtd><mrow><mi>exp</mi><mrow><mo>(</mo><mo>-</mo><mfrac><mrow><mo>|</mo><mo>|</mo><mi>d</mi><mi>i</mi><mi>s</mi><mi>t</mi><mrow><mo>(</mo><msub><mi>x</mi><mi>k</mi></msub><mo>,</mo><msubsup><mi>x</mi><mi>k</mi><mi>j</mi></msubsup><mo>)</mo></mrow><mo>|</mo><msup><mo>|</mo><mn>2</mn></msup></mrow><mrow><mn>2</mn><msubsup><mi>&amp;sigma;</mi><mi>k</mi><mn>2</mn></msubsup></mrow></mfrac><mo>)</mo></mrow><mo>,</mo></mrow></mtd><mtd><mrow><mi>i</mi><mi>f</mi></mrow></mtd><mtd><mrow><mi>i</mi><mo>&amp;NotEqual;</mo><mi>j</mi></mrow></mtd></mtr><mtr><mtd><mrow><mn>0</mn><mo>,</mo></mrow></mtd><mtd><mrow><mi>i</mi><mi>f</mi></mrow></mtd><mtd><mrow><mi>i</mi><mo>=</mo><mi>j</mi></mrow></mtd></mtr></mtable></mfenced><mo>;</mo></mrow> Wherein, X={x ₁ ,x ₂ ,...,x _N ,x _N+1 ,...,x _N+M } is the sample set of the short video data, and the short video with the semantic label The number of data is N, the number of short video samples to be marked is M, and the sample set is described using K perspectives, is the feature vector of the kth viewing angle of the i-th video, is the feature vector of the k-th view of the j-th video, 1≤k≤K, 1≤i≤N+M, 1≤j≤N+M; Further, the multi-view fusion subspace direct calculation unit includes: The multi-view fusion subspace establishment module, the multi-view fusion subspace establishment module is as follows for the establishment process of the multi-view fusion subspace: Calculate the normalized Laplacian similarity matrix L(S _k ) of the similarity matrix S _k of the k-th viewing angle respectively, 1≤k≤K: <mrow><mi>L</mi><mrow><mo>(</mo><msub><mi>S</mi><mi>k</mi></msub><mo>)</mo></mrow><mo>=</mo><mi>I</mi><mo>-</mo><msubsup><mi>D</mi><mi>k</mi><mrow><mo>-</mo><mfrac><mn>1</mn><mn>2</mn></mfrac></mrow></msubsup><msub><mi>S</mi><mi>k</mi></msub><msubsup><mi>D</mi><mi>k</mi><mrow><mo>-</mo><mfrac><mn>1</mn><mn>2</mn></mfrac></mrow></msubsup><mo>;</mo></mrow> Wherein, I is (N+M)×(N+M) order unit matrix, is the degree matrix of the similarity matrix S _k of the k-th viewing angle, that is, the main diagonal element is the sum of the columns of S _k , and the non-main diagonal element is 0; Calculate the trace normalization matrix of the normalized Laplacian similarity matrix L(S _k ): <mrow><mover><msub><mi>L</mi><mi>k</mi></msub><mo>~</mo></mover><mo>=</mo><mfrac><mn>1</mn><mrow><mi>t</mi><mi>r</mi><mrow><mo>(</mo><mi>L</mi><mo>(</mo><msub><mi>S</mi><mi>k</mi></msub><mo>)</mo><mo>)</mo></mrow></mrow></mfrac><mi>L</mi><mrow><mo>(</mo><msub><mi>S</mi><mi>k</mi></msub><mo>)</mo></mrow><mo>;</mo></mrow> Trace normalization processing of multi-view fusion subspace similarity matrix S ₀ : <mrow><msub><mover><mi>L</mi><mo>~</mo></mover><mn>0</mn></msub><mo>=</mo><mfrac><mn>1</mn><mrow><mi>t</mi><mi>r</mi><mrow><mo>(</mo><mi>L</mi><mo>(</mo><msub><mi>S</mi><mn>0</mn></msub><mo>)</mo><mo>)</mo></mrow></mrow></mfrac><mi>L</mi><mrow><mo>(</mo><msub><mi>S</mi><mn>0</mn></msub><mo>)</mo></mrow><mo>=</mo><munderover><mo>&amp;Sigma;</mo><mrow><mi>k</mi><mo>=</mo><mn>1</mn></mrow><mi>K</mi></munderover><msub><mi>&amp;beta;</mi><mi>k</mi></msub><mover><msub><mi>L</mi><mi>k</mi></msub><mo>~</mo></mover><mo>,</mo><munderover><mo>&amp;Sigma;</mo><mrow><mi>k</mi><mo>=</mo><mn>1</mn></mrow><mi>K</mi></munderover><msub><mi>&amp;beta;</mi><mi>k</mi></msub><mo>=</mo><mn>1</mn><mo>;</mo></mrow> in, is the linear combination coefficient of randomly initialized K viewing angles; Further, the multi-view fusion subspace direct calculation unit further includes: The multi-view fusion subspace solving module is connected with the multi-view fusion subspace module, and the multi-view fusion subspace solving process of the multi-view fusion subspace by the multi-view fusion subspace is as follows: Construct the objective function to obtain the predicted label matrix f: <mrow><munder><mi>min</mi><mrow><mi>f</mi><mo>,</mo><mi>&amp;beta;</mi></mrow></munder><munderover><mo>&amp;Sigma;</mo><mrow><mi>i</mi><mo>=</mo><mn>1</mn></mrow><mi>N</mi></munderover><mo>|</mo><mo>|</mo><mrow><mo>(</mo><msub><mi>y</mi><mi>i</mi></msub><mo>-</mo><msub><mi>f</mi><mi>i</mi></msub><mo>)</mo></mrow><mo>|</mo><msup><mo>|</mo><mn>2</mn></msup><mo>+</mo><mi>&amp;lambda;</mi><munderover><mo>&amp;Sigma;</mo><mrow><mi>k</mi><mo>=</mo><mn>1</mn></mrow><mi>K</mi></munderover><mo>|</mo><mo>|</mo><munderover><mo>&amp;Sigma;</mo><mrow><mi>i</mi><mo>=</mo><mn>1</mn></mrow><mi>N</mi></munderover><msub><mi>&amp;beta;</mi><mi>i</mi></msub><mover><msub><mi>L</mi><mi>i</mi></msub><mo>~</mo></mover><mo>-</mo><mover><msub><mi>L</mi><mi>k</mi></msub><mo>~</mo></mover><mo>|</mo><msubsup><mo>|</mo><mi>F</mi><mn>2</mn></msubsup><mo>+</mo><mi>&amp;mu;</mi><mo>|</mo><mo>|</mo><msup><mi>f</mi><mi>T</mi></msupup><munderover><mo>&amp;Sigma;</mo><mrow><mi>k</mi><mo>=</mo><mn>1</mn></mrow><mi>K</mi></munderover><msub><mi>&amp;beta;</mi><mi>i</mi></msub><mover><msub><mi>L</mi><mi>k</mi></msub><mo>~</mo></mover><mi>f</mi><mo>|</mo><msubsup><mo>|</mo><mi>F</mi><mn>2</mn></msubsup><mo>+</mo><mi>&amp;theta;</mi><mo>|</mo><mo>|</mo><mi>&amp;beta;</mi><mo>|</mo><msup><mo>|</mo><mn>2</mn></msup><mo>,</mo><msup><mi>e</mi><mi>T</mi></msup><mi>&amp;beta;</mi><mo>=</mo><mn>1</mn><mo>;</mo></mrow> in, Be the label set that described semantic label forms, the sample x _i of described short video data has the jth label, then Y _ij =1, otherwise Y _ij =0, C is the scale of described label set; in, is the predicted label matrix initialized randomly; First, the objective function is transformed into <mrow><mi>&amp;Gamma;</mi><mo>=</mo><munder><mrow><mi>m</mi><mi>i</mi><mi>n</mi></mrow><mi>&amp;beta;</mi></munder><mo>{</mo><mi>&amp;lambda;</mi><munderover><mo>&amp;Sigma;</mo><mrow><mi>k</mi><mo>=</mo><mn>1</mn></mrow><mi>K</mi></munderover><munderover><mo>&amp;Sigma;</mo><mrow><mi>t</mi><mo>=</mo><mn>1</mn></mrow><mrow><mi>N</mi><mo>+</mo><mi>M</mi></mrow></munderover><mo>|</mo><mo>|</mo><mrow><mo>(</mo><msup><mi>M</mi><mrow><mo>(</mo><mi>t</mi><mo>)</mo></mrow></msup><mo>-</mo><mover><msubsup><mi>l</mi><mi>k</mi><mi>t</mi></msubsup><mo>~</mo></mover><msup><mi>e</mi><mi>T</mi></msup><mo>)</mo></mrow><mi>&amp;beta;</mi><mo>|</mo><msubsup><mo>|</mo><mi>F</mi><mn>2</mn></msubsup><mo>+</mo><msup><mi>&amp;mu;g</mi><mi>T</mi></msup><mi>&amp;beta;</mi><mo>+</mo><mi>&amp;delta;</mi><mrow><mo>(</mo><mn>1</mn><mo>-</mo><msup><mi>e</mi><mi>T</mi></msup><mi>&amp;beta;</mi><mo>)</mo></mrow><mo>+</mo><mi>&amp;theta;</mi><mo>|</mo><mo>|</mo><mi>&amp;beta;</mi><mo>|</mo><msup><mo>|</mo><mn>2</mn></msup><mo>}</mo><mo>;</mo></mrow> Then, fix f, solve for β, seek the partial derivative of Γ to β, make the partial derivative of Γ to β be 0, and obtain the linear combination coefficient β of the K viewing angles; Finally, fix β and solve for f: obtain the partial derivative of Γ to f _i (1≤i≤N) and the partial derivative of Γ to f _i (N+1≤i≤N+M) respectively, let the Γ to The partial derivative of f _i (1≤i≤N) and the partial derivative of Γ to f _i (N+1≤i≤N+M) are 0, and the predicted label matrix f is obtained; Repeat the steps of "fix f, solve β" and "fix β, solve f" alternately until the predicted label matrix f converges, and the linear combination coefficient β of the fusion subspace and the predicted label matrix f are obtained. Annotation of video data; in, in, in, express the tth column of Among them, δ is a non-negative Lagrangian multiplier; Further, the parameters λ, μ, θ of the objective function are determined by using 10-fold cross-validation.