CN102427507A - Football video highlight automatic synthesis method based on event model - Google Patents

Abstract

The present invention is an event model-based automatic synthesis method of football video highlights, including: aiming at football match video highlights, defining football video highlight segments as football video events that can be decomposed into multi-action combinations; constructing a core-surrounding event model to represent football highlights Fragment; use the football game video and its corresponding text commentary to build a training set, select goals and red and yellow cards as two types of football highlights, and train the event model; input a football game video without commentary, and identify the football highlight clips in the input video The appearance position of the match score is given; according to the user's needs, the football highlight clips with higher scores are automatically synthesized into a football video highlight. The method for generating football video collections of the present invention can break through the limitations of factors such as the distance of the input video lens and the length of the video, and can be widely applied to the fields of personal digital entertainment, sports film and television production, and the like.

Description

A method of automatic synthesis of football video highlights based on event model

technical field

The invention relates to the fields of computer vision, video processing and augmented reality, in particular to a method for automatically synthesizing football video highlights based on an event model.

Background technique

As a kind of sports film and television program, sports video highlights are popular among audiences because they can obtain sufficient information in a relatively short period of time. Especially in football matches, it is very time-consuming to watch a 90-minute game video just to see a favorite player or a wonderful shot. Therefore, football match highlights are often used to record highlights, game summaries, and personal experiences of players. and other topics related to the competition. Traditional video highlights are edited by humans. Although the editing accuracy is high and full of emotion, it takes a lot of manpower to check the video frame by frame to find the desired highlights, and requires a high level of experience and knowledge of the editor. With the continuous progress of research in the field of video understanding and computer vision, automatic generation of highlight videos for sports event videos has gradually become a technology and research hotspot.

At present, according to different video sources, automatically generating highlight videos for sports event videos can be divided into two categories. One is automated highlights for televised video. Since the TV broadcast video has added the broadcaster's understanding of the event, the broadcasting skills can be used as the implicit clue of the video highlights when processing. For example, when broadcasting a football game, close-ups or slow-motion shots usually appear after the goal; the same event usually occurs between two camera cuts; long shot usually means the opening or the ball's large-scale trajectory, etc. This type of method detects the above clues in the football video, completes the detection of the football highlight segment and finally generates the highlight video, or directly detects the on-screen text (such as the scoreboard) in the video to determine the occurrence time of the football highlight segment. Although this type of method can obtain better collection results to a certain extent, it relies too much on TV broadcast video, which has great limitations in the scope of application.

The other category is automated highlights for non-televisioned video. Among them, the method with strong pertinence to the video topic usually uses the special prior knowledge of the video topic (such as the prior knowledge of the mesh goal in the football video, the large green lawn, the cheers of the audience, etc.) to obtain information about the video. A great shot of the subject detects clues. Its strong pertinence determines that the model of this type of method is fixed and its reusability is poor. What is more valuable for research is the collection method which has universal applicability within a certain range. At present, research in this area mainly focuses on two directions: (1) video event analysis; (2) video content summarization.

In terms of video event analysis, at the ECCV conference in 2010, Li Fei-Fei et al. of Stanford University proposed a behavior model based on the temporal relationship of human actions. The model represents actions as behavioral segmentations at different time points. This method trains two models, the discriminative model and the appearance model: the decision model is used to encode video sequences based on time decomposition, and the appearance model is used for each behavior segmentation. In the recognition process, the video is matched with the model by learning feature and behavior segmentation decomposition. By introducing temporal structure, this method can better recognize simple and complex human actions, but due to its fixed temporal structure pattern, it cannot handle complex events composed of actions. At the CVPR conference in 2009, Larry S. Davis et al. of the University of Maryland proposed a method for learning a complete visual storyline model from videos with weakly labeled data. The storyline model is expressed in the form of an AND or graph, which can simply encode the storyline changes in the video. Edges in an AND-OR graph are equivalent to causality based on spatio-temporal constraints. Using this model and the learned training data, behavior recognition and storyline extraction can be performed. Considering the relationship between human body posture and surrounding objects in the video frame, in 2010, Fowlkes et al. of the University of California, USA proposed a method for recognizing actions based on modeling of the relationship between human body posture and surrounding objects. This method mainly addresses the problem of action recognition from static images and transforms it into a latent structure labeling problem.

In terms of video content summary, the method proposed by Pritch et al. in the PAMI journal in 2008 can condense a long video into a short summary by analyzing the video, and display the motion information of multiple frames on each frame at the same time, but the method’s The limitation is that it cannot handle situations where the entire scene of the video is in motion and edited video. Hwang et al. from the University of Washington proposed a key frame extraction method based on video object segmentation and designed and implemented a corresponding system, which can perform online processing quickly and effectively. At the CVPR conference in 2005, Jojic et al. of Microsoft Research proposed a new interaction model for surveillance video to index and analyze video. In addition, Wu et al. from the University of Vermont proposed a hierarchical video summarization strategy, which provides users with multi-scale and multi-level video summaries by analyzing the video content structure.

To sum up, there are mainly two problems in the current video collection technology: (1) It depends heavily on the quality of the input video, and its application range is narrow. Although video highlights using clues rich in semantic hints such as camera cuts, whistles, and transitions can quickly detect football highlights, they cannot understand the progress of football events, so it is difficult to extract the time interval of events. (2) Less video highlights are performed in units of events. Due to the rich variety of video events, it is difficult to fully cover the changes of events by directly adopting the feature statistics method. How to make reasonable use of domain knowledge and combine the visual features of events to model events is a difficult point and a research hotspot.

Contents of the invention

According to the above-mentioned actual needs and key problems, the object of the present invention is to propose a method for automatically synthesizing football video highlights based on an event model. This method can break through the limitations of factors such as the distance of the input video, the length of the video, and the sound of the video. Especially when the input video is a non-rebroadcast video, and key clues such as close-up shots and cheers cannot be obtained from it, the event-based model proposed by the present invention The collection method of is especially applicable.

The present invention considers that the football video highlight is a synthetic video composed of several football highlight segments, and each highlight segment contains an important football event. Compared with other sports videos, football game videos have two characteristics: first, it is difficult to find the beginning and end clues of video events from the video; second, the rules of football games are complex, and the same type of important football events (such as scoring or red and yellow cards) each time it appears, its duration and course of events are often different. Through a large number of observations, it is known that important football events can usually be decomposed into a combination of several actions, including a frequently occurring important action called core action; relatively speaking, other actions are called peripheral actions. Therefore, the present invention considers that the highlight segment of football game video can be represented by a core-surrounding event model.

In order to condense a football game video into a football highlight video, it is necessary to detect and extract football highlight segments in the input video. Therefore, the present invention first constructs a core-surrounding event model to model the semantic relationship, temporal relationship and visual features between the events and the actions that make up the event.

The training process of the core-surrounding event model includes the following steps: (1) Input a series of football game videos and their corresponding text commentary, extract keywords from the commentary, and count each keyword according to the event record of the commentary The probability of occurrence of the keyword, and the probability of multiple keywords appearing at the same time; (2) select the keyword with the highest probability of occurrence as the core keyword; (3) correspond the commentary to the football game video, record the keyword appearance time, and Count the action duration and event duration represented by the keywords; (4) Calculate the gradient features and optical flow features of the spatio-temporal interest points during the keyword appearance period, and count the gradient histogram and optical flow histogram as the local visual features of the action.

In a nutshell, the modeling content of the core-surrounding event model includes: the visual statistical characteristics of each action; the sequence of actions in the event occurrence process; the ratio of action duration to event duration; the probability of each action occurring.

After the model is trained, it is used for the detection and extraction of video events. In general, inputting a football game video, the steps of synthesizing a football highlight video can be divided into: (1) extracting highlight segments. For each type of football highlight segment, firstly, according to the important football events contained in this type of highlight segment, the core action and surrounding actions that make up the event are detected on the input video, and the time period of each action is obtained; then, the core action As a benchmark, the event occurrence time period is determined in combination with the action timing relationship, which is counted as the time period of the candidate highlight segment; finally, the event model is matched with the candidate highlight segment to obtain a model matching score. (2) Synthesize video highlights. First, obtain a list of candidate highlight segments for each type of football highlight segment through step (1), and sort them according to the model matching score from high to low; then select several football highlight segments according to the highlight segment category and highlight video length required by the user clips, and arrange them according to their occurrence time; finally, select the last few frames of the previous football highlight clip and the first few frames of the next clip to make a smooth transition to make it more in line with the visual sensory effect.

Compared with other video collection methods, the present invention has the following advantages: (1) It is applicable to a wide range of video sources. Compared with other video collection methods that need to rely on clues such as close-up shots and transitions when TV stations rebroadcast videos, the present invention detects and recognizes various events in videos by analyzing the visual features of video events, thus being widely applicable to personal digital Video highlights from entertainment, sports science research, TV programming, and more. (2) The combination of collection fragments is flexible. Since the present invention uses video events as the unit of video highlights, the user can specify the type of highlights, the length of highlights, and other conditions, so that personalized video highlights that meet user needs can be synthesized.

Description of drawings:

Fig. 1 is a core-surrounding event model structural diagram of the present invention;

Fig. 2 is a schematic diagram of the model training process of the present invention;

Fig. 3 is the modeling flowchart of semantic layer event model of the present invention;

Fig. 4 is the flow chart of visual layer event model training process of the present invention;

Fig. 5 is a schematic diagram of the extraction process of football highlights fragments of the present invention;

Fig. 6 is a schematic diagram of synthesizing football highlight fragments in the present invention.

Detailed ways:

The present invention will be described in detail below in conjunction with the accompanying drawings.

The present invention defines a football video collection as a collection of important football events that take place in a football match and take video as a carrier. Football video highlights are composed of a series of football highlight clips, and each football highlight clip contains an important football event. The core-surrounding event model constructed by the present invention is used to detect and identify important football events in football game videos, and then extract football highlights. Football highlight clips have different categories depending on the type of important football events they contain. For example, goals and yellow cards belong to different important football events, therefore, football highlight clips containing goals and football highlight clips containing yellow cards belong to different categories of football highlight clips.

Referring to Fig. 1, the structure diagram of the core-surrounding event model of the present invention, the core-surrounding event model constructed by the present invention models the important football events contained in the football highlights segment both semantically and visually. The model mainly includes 3 parts: (1) Semantic relationship, which mainly models the possibility of the core action and each surrounding action at the same time, and the possibility of each action appearing in this important football event. (2) Time sequence. This part mainly models the possible time position and duration of each action during the occurrence of important football events. (3) Visual appearance, this part mainly refers to the visual feature statistics on the spatio-temporal interest points in the video of the time interval of the action. For similar important football events, the most likely action is selected as the core action, and other actions are regarded as surrounding actions supporting the event. Therefore, temporal relationship constraints between surrounding actions and core actions are implicitly built into the model, which is very helpful for localizing events in videos.

The core-surrounding event model can be trained in two layers: a semantic layer and a visual layer. For a class of event E and its action set {a _i , i=1,...,n}, the semantic layer models the occurrence probability of a _i in event E and whether a _i is the core of E. The visual layer models the visual appearance of events and introduces the semantic layer model as a prior probability. The visual layer model has three parameters: the best classifier A _i to recognize an action a _i ; the best appearance time anchor point t _i of the classifier A _i ; the time interval r _i of A _i during the occurrence of the event.

The training set of the event model includes video segments {V ¹ ,...,V ^N }, and the category labels y _i of the corresponding actions (y _i ∈ {-1, 1}, i=1,...,N) . The implicit support vector machine (LSVM) is used to learn the model. In the LSVM framework, the energy function is maximized according to the hidden variable. The hidden variable here refers to the best position of the action classifier. This position is not given precisely, but by The training samples are implicitly trained.

Referring to Fig. 2, the schematic diagram of the training process of the football highlights segment model of the present invention, the model training process of the present invention is mainly divided into three steps: (1) Semantic relationship modeling. The specific process is shown in Figure 3. First, the commentary with time and event marks is used as the training text, and after sentence component analysis, the verb and gerund keywords are extracted, and the keyword set representing the event is constructed; based on WordNet vocabulary classification, mapping keywords to different categories, and using the category label as an action category label; counting the number of occurrences and total number of occurrences of each action in this category of highlights, and calculating the identification degree of each action to the category of highlights , and select the action with the highest identification degree as the core action; record the number of times the action occurs, and calculate its occurrence probability as the prior probability. (2) Action visual feature statistics. According to the time stamp and action category label of the commentary, the video time interval in which the action occurred is obtained; the video segment in the video time interval is divided into several parts, and the gradient histogram and optical flow histogram on the spatiotemporal interest points are calculated in each part picture. (3) Time series relationship modeling. According to the time mark, event mark and action category label of the commentary, the action occurrence sequence diagram of the events contained in the similar football highlight clips is obtained. According to the event visual layer model, LSVM is used to train the best occurrence position of each action.

的时空兴趣点

其中

为视频段

中的时空兴趣点个数。(2)统计st_l的梯度直方图

和光流直方图

其中梯度直方图的横坐标是梯度向量区间，区间个数用ng表示，纵坐标表示落在每个向量区间的梯度向量个数；光流直方图的横坐标是光流向量区间，区间个数用nf表示，纵坐标表示落在每个向量区间的光流向量个数。(3)将每个视频段时空兴趣点的梯度直方图和光流直方图归一化为一个nd维向量，其中nd＝ng+nf，并利用k-means算法将

个向量聚为K类，构造出视频段视觉统计特征的编码表。(4)初始化分类器A_i的最佳出现时间锚点t_i和A_i在事件发生过程中的时间区间r_i，然后通过步骤(5)(6)训练分类器A_i。(5)根据t_i和r_i截取视频V^p的若干视频段，统计其包含的时空兴趣点向量，并映射到编码表构成一个长度为K的向量分布直方图

将该直方图归一化为K维向量加入正例集Ψ。(6)以r_i确定截取窗口大小，在视频V^p上滑动，计算在时间锚点t处所截取视频段的向量分布直方图

计算该直方图构成的K维向量与正例集中向量的距离

若

(ε为某极小量)，则将

代替加入正例集，重复本步骤；否则结束本步骤。(7)统计t在视频V^p中出现的位置，将其拟合为二次抛物曲线其中{α_i，β_i}为二次曲线参数。该二次抛物曲线横坐标表示归一化后的t的出现时间，纵坐标表示在该时间上的出现次数，作为时间惩罚函数留待识别过程使用。Referring to Fig. 4 visual layer event model training process flowchart of the present invention, the training process of event model of the present invention on the visual layer is as follows: (1) calculate feature point, each video V ^p (p ∈ {1 ,...,N}) are evenly divided into M video segments detection

spatio-temporal points of interest

in

for the video segment

The number of spatiotemporal interest points in . (2) Gradient histogram of statistical st _l

and optical flow histogram

The abscissa of the gradient histogram is the gradient vector interval, the number of intervals is represented by ng, and the ordinate indicates the number of gradient vectors falling in each vector interval; the abscissa of the optical flow histogram is the optical flow vector interval, the number of intervals Represented by nf, the ordinate indicates the number of optical flow vectors falling in each vector interval. (3) Normalize the gradient histogram and optical flow histogram of the spatiotemporal interest points of each video segment into an nd-dimensional vector, where nd=ng+nf, and use the k-means algorithm to

The vectors are clustered into K classes, and the coding table of the visual statistical features of the video segment is constructed. (4) Initialize the best appearance time anchor point t _{i of the classifier A i and} the time interval r _i of A _i in the event occurrence process, and then train the classifier A _i through steps (5) (6). (5) Intercept several video segments of video V ^p according to t _i and r _i , count the spatio-temporal interest point vectors contained in them, and map them to the encoding table to form a vector distribution histogram with a length of K

Normalize the histogram to a K-dimensional vector and add it to the positive example set Ψ. (6) Use _ri to determine the interception window size, slide on the video ^Vp , and calculate the vector distribution histogram of the intercepted video segment at the time anchor point t

Calculate the distance between the K-dimensional vector formed by the histogram and the positive example set vector

like

(ε is a very small amount), then the

replace Add positive examples and repeat this step; otherwise end this step. (7) Count the position where t appears in the video ^Vp , and fit it to a quadratic parabolic curve Where {α _i , β _i } are parameters of the quadratic curve. The abscissa of the quadratic parabolic curve represents the time of occurrence of the normalized t, and the ordinate represents the number of occurrences at this time, which is used as a time penalty function for the recognition process.

(2)取分类器A_i，根据其时间区间r_i划定滑动窗口大小，在V^F的Q段视频段上滑动，计算在时间锚点t处所截取视频段的向量分布直方图

计算该直方图构成的K维向量与正例集中向量相似度

(3)计算时间锚点t处的时间惩罚

(4)根据公式

计算分类器A_i在候选足球集锦片段V^F上的最好得分作为分类器A_i的匹配分数；(5)累加模型匹配分数，并返回步骤(2)直至所有分类器匹配完毕。Referring to Fig. 5 the schematic diagram of the extraction process of the football highlight segment of the present invention, the extraction process mainly includes the following steps: (1) for the input football game video segment, detect all possible actions; (2) take a certain type of football highlight segment as an example, Use the core action of the important football event contained in this type of football highlight segment to locate the rough time period of the football highlight segment as the candidate time period of the football highlight segment; (3) calculate the matching degree of the candidate time period and the corresponding event model, and It is represented by a score, which is called the match score of the candidate time period for the football highlight segment. Arrange all candidate time periods of the same football highlight segment in descending order of matching score. The matching process steps of the candidate football highlight segment and the event model are as follows: (1) Divide the candidate football highlight segment V ^F into video segments according to the video division scale of the training set

(2) Take the classifier A _i , define the size of the sliding window according to its time interval r _i , slide on the Q segment video segment of ^VF , and calculate the vector distribution histogram of the video segment intercepted at the time anchor point t

Calculate the similarity between the K-dimensional vector formed by the histogram and the vector in the positive example set

(3) Calculate the time penalty at the time anchor point t

(4) According to the formula

Calculate the best score of classifier A _i on the candidate football highlights segment ^VF as the matching score of classifier A _i ; (5) accumulate model matching scores, and return to step (2) until all classifiers are matched.

和B的第x帧透明度

满足 Referring to Fig. 6, a schematic diagram of synthesizing a football highlight segment of the present invention, according to the type of football highlight segment and the length of the highlight video required by the user, the video highlight is completed by editing the transition effect between every two football highlight segments. Select the last N frames of football highlight clip A and the first N frames of football highlight clip B as the transition area, adjust the transparency of each frame, and make the adjusted x-th frame of A transparent

and the x-th frame transparency of B

satisfy

The present invention can support summarizing football game videos according to user needs. (1) Given the length of the highlight video, generate a highlight video of a football match. (2) Designate a football highlight segment type, and generate a football highlight video about the specified highlight segment type. (3) Designate the length of the highlight video and the type of the highlight segment at the same time, and generate a football highlight video of a specific length for this type of highlight segment.

The above descriptions are only some basic explanations of the present invention, and any equivalent transformation made according to the technical solution of the present invention shall fall within the scope of protection of the present invention.

Claims (6)

1. A method for automatically synthesizing football video highlights based on an event model, characterized in that it comprises the following steps: (1) The definition of a football video highlight segment is an important football event that is performed by a single person or multiple people and can be decomposed into multiple action combinations; (2) Construct a core-surrounding event model. According to the probability of action occurrence, the most likely action is designated as the core action, and the rest of the actions are surrounding actions. The event model specifically includes three parts: action semantic relationship, action timing relationship and local visual features. parts; (3) Use football game videos and their corresponding text commentaries to construct a training set, select goals and red and yellow cards as two types of football highlights, and train the core from the three aspects of action semantic relationship, action timing relationship and local visual features - surrounding event model; (4) Input a football game video without commentary, use the event model trained to extract football highlight segments in the input video, and give the matching score of the candidate highlight segments and the model; (5) Sort the football highlight segments according to their matching scores, and automatically synthesize the football highlight segments with higher scores into a football video highlight. 2. the automatic synthesis method of football video highlights based on event model according to claim 1, it is characterized in that: in step (1), use video event as football highlight segment unit, carry out football video separately for a certain type of football highlight segment Highlights. 3. the automatic synthesis method of football video highlights based on event model according to claim 1, is characterized in that: the core of step (2)-surrounding event model requires event can be decomposed into a plurality of actions, and described core-surrounding event The model mainly models three parts: (2.1) Action semantic relations include the probability of occurrence of each action, and the probability of simultaneous occurrence of each surrounding action and core action; (2.2) The timing relationship of actions includes the sequence of actions in the event occurrence process, and the ratio of action duration to event duration; (2.3) Local visual features include the gradient and optical flow statistical features of each action during the motion duration. 4. the automatic synthesis method of football video highlights based on event model according to claim 1, is characterized in that: the football game video text commentary that requires input in the step (3) contains time record and event record, can be correlated with video time Correspondingly, the core-peripheral steps for a certain type of football training are as follows: (3.1) Input a series of football game videos and their corresponding text commentary, extract keywords from the commentary, and count the occurrence probability of each keyword according to the event records of the commentary, and the occurrence probability of multiple keywords at the same time probability; (3.2) select the keyword with the greatest probability of occurrence as the core keyword; (3.3) The commentary is corresponding to the football game video, the keyword occurrence time is recorded, and the action duration and event duration of the keyword representation are counted; (3.4) Calculate the gradient features and optical flow features of the spatio-temporal interest points during the keyword occurrence period, and count the gradient histogram and optical flow histogram as the local visual features of the action. 5. the automatic synthesis method of football video collection based on event model according to claim 1, is characterized in that: step (4) imports a section of football game video, and its collection segment extraction process is divided into the following steps: (4.1) Detect core actions and surrounding actions on the input video, and obtain the time periods of all actions; (4.2) Based on the core action, combined with the timing relationship of the action to determine the event occurrence time period and count it as a candidate football highlight segment; (4.3) Match the event model on the candidate segment to obtain a model matching score. 6. the automatic synthesis method of football video highlights based on event model according to claim 1, is characterized in that: when some candidate football highlight segments are combined into football video highlights in the step (5), according to the highlight types and video highlights required by users Length transitions the beginning and end of each football highlight clip.

Images