KR100642888B1 - A method of creating a video summary based on the narrative structure that can be understood and a recording medium storing a program for implementing the method. - Google Patents

Abstract

The present invention relates to a method of generating a video summary capable of understanding a story in a story-driven video such as drama, and more specifically, to the narrative video, the narrative structure (story development structure) and the degree of development (quantity) per unit are automatically determined. The present invention relates to a video summary generation method capable of automatically generating video summaries that can be extracted, and to a story understanding based on a desired time, and a recording medium storing a program for implementing the method.

According to the present invention, when providing a summary service for a video having a story, the video is analyzed according to the video analysis rules proposed in the present invention for each video to generate a story development structure and a degree of development for each unit, and based on this. Create a shortened summary video that best represents the story development for the summary time input by the user. The user can view the generated summary video and get the full story within a limited time.

Video, summary, story, understanding, story development, video analysis, narrative structure

Description

Narrative structure based video abstraction method for understanding a story and storage medium storing program for realizing the method

1 is an exemplary view showing the main scene characteristics of a soccer video according to a conventional summary method.

2 is a diagram illustrating a conventional summary method using a variable sampling rate.

3 is a view showing a detailed process of the pretreatment process according to an embodiment of the present invention.

Figure 4 is an illustration showing the type of interaction between the characters according to an embodiment of the present invention.

5 is an exemplary view showing a kind of film making technology applicable to a new book with interaction according to an embodiment of the present invention.

Figure 6 is an illustration of a narrative structure graph presented in the present invention.

FIG. 7 is a diagram illustrating a summary result subgraph according to a summary time input by a user illustrated in FIG. 6.

8 is a view briefly illustrating the entire process of the story comprehension video summary method presented in the present invention.

9 is a view showing experimental data implementing the story comprehensible video summary method proposed in the present invention.

The present invention relates to a method for generating a story comprehensible video summary. More specifically, it is possible to automatically extract the narrative structure and the amount of story development by unit for the video with the story, and based on this, the video can automatically generate a video summary that can understand the story at the desired time. A method of generating a summary and a recording medium storing a program for implementing the method.

Recently, as multimedia services are available in various media such as the Internet, mobile devices, and TVs, there is an increasing demand for a method of quickly obtaining necessary multimedia information. In particular, in the case of a video, a search for a specific unit in the file takes less time to check the results, such as searching a specific part of the video file or gathering necessary parts, rather than searching a whole video file. It takes

The summary method, which gathers the necessary parts, has the advantage that it takes less time to check the results while providing information about the entire video, while the approach requires the user to view the summary results or the content attributes of the target video to be summarized. The disadvantage is that it depends on. For example, a sports video consists of videos according to the rules of the game, and the scoring scene is the main scene of the video.

1 is an example of analyzing a rule for finding a goal scene in soccer. Referring to FIG. 1, the location of a goal scene in a typical soccer relay video is shown. Videos such as documentaries and dramas have more important parts of the video than the ones that matter. Therefore, in this case, the existing research method has been studied how to grasp the overall mood of the video by showing various scenes of the entire video by omitting similar scenes.

FIG. 2 is an example in which a user may notice a change state of the entire video earlier than the original playback time by removing overlapping portions by changing sampling rates. That is, the portion 201 having a large amount of change of the screen (for example, the change of color, the change of motion, etc.) increases the sampling rate while the portion of the small change 202 decreases the sampling rate to quickly reproduce the change state of the entire video. The method is shown.

In case of story-driven video, such as drama, a user who wants to see a summary wants to know not only the specific part but also how the entire story is developed. When the method of FIG. 1 is used, a part having a specific property is detected, or when the method of FIG. 2 is used, it is possible to know the situation as to who appeared, but it is difficult to determine what story has been developed.

Accordingly, an object of the present invention is to solve the above problems, and an object of the present invention is to extract a feature that allows a user to understand a story in a story-centric video such as drama, and generate a video summary that can be understood based on the story. The present invention provides a method for generating a video summary and a recording medium storing a program for implementing the method.

It is another object of the present invention to provide a video summary generation method and method for generating a video summary capable of understanding a story for a given summary time, even if the user sets a different summary time due to the influence of a network environment or a personal environment. A program for implementing is provided to present a stored recording medium.

It is still another object of the present invention to implement a video summary generation method and method for automating all processes to generate a video summary capable of story comprehension for a given summary time, which reduces the manpower and time required for generating the summary. The program is to present a stored recording medium.

As a technical idea for achieving the above object of the present invention, there is provided a method for generating a story understandable summary for a story-centric video, comprising: receiving a source video of a subject to be summarized from a user; Performing a predetermined preprocessing process on the source video; Calculating a narrative structure and a story development degree for each unit based on the information generated from the preprocessing step; Receiving a summary time from the user; Provided is a method for generating a story understandable summary including the step of generating a summary capable of understanding the story in accordance with the summary time using the narrative structure and the degree of story development by unit. A system, apparatus, and record carrier are provided that enable performing the method of generating a summary.

In this case, performing a predetermined preprocessing process on the source image may include extracting a shot boundary with respect to the source image, extracting a scene (unit of narrative structure) with respect to the source image, and taking a shot belonging to the scene. Extracting the number and length, recognizing the person included in each shot, extracting the position of each person in the shot, and all the frames belonging to each shot except the entire frame and the person area (background area) Extracting an average color of the < RTI ID = 0.0 >

In addition, the step of calculating and generating the narrative structure and the story development for each unit based on the information generated from the preprocessing process may include: determining whether each scene extracted from the source image includes interaction between characters; Identifying the types of interactions involved for the gods containing interactions, and identifying the application of film-making techniques for story development among all the gods for the gods containing interactions and the interaction between characters Computing the story development of the new story using the action and whether the film production technology is applied.

In addition, the step of determining whether each scene extracted from the source image includes the interaction between the characters is characterized in that it comprises at least two characters. The type of interaction included in the scene may include one or more of possible types such as 'conversation' (when the characters interact with each other) and 'action' (when there is a lot of movement between the characters). Also, the kind of film making technique applicable to story development between all the gods about the god including interaction is one of the possible types such as 'graphical relationship', 'rhythmical relationship', 'spatial relationship' and 'temporal relationship'. The above can be applied. The extent of the story development using the interaction between the characters and the application of the film creation technology depends on the importance of the characters included in the scene, the strength of the interaction included in the scene, and the strength of the film creation technology applied to the scene. It is characterized by the difference.

Finally, the step of generating a summary capable of understanding the story in accordance with the summary time using the narrative structure and the degree of story development for each unit satisfies the summary time, and the new story development degree included in the summary is maximized. It is characterized by.

Next, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

The narrative structure-oriented video summarization method according to the present invention includes an application (hereinafter, a video summary request program) installed in a user terminal (eg, a personal computer, a mobile communication terminal, a personal digital assistant, etc.). It may be performed by a separate device for performing the function, as well as). Hereinafter, assuming that the video summarization method according to the present invention is performed by an application program installed in a user terminal, the operation of the application program will be described in detail.

3 is a diagram illustrating a preprocessing process and information extracted therein according to an embodiment of the present invention.

Referring to FIG. 3, when a user inputs a target video to be summarized, shot boundary, neural boundary, and character information are extracted. At this time, the number, average length, average color, etc. of shots belonging to each scene which are necessary information are calculated. In particular, in the case of character information, a face is recognized in a frame belonging to the scene, and the position of the recognized face is also extracted.

In addition, the average color uses the RGB value of the pixel of the corresponding area. Various algorithms already available for extraction of shots, nervous system, face recognition, average color, etc. can be used. Since various algorithms are already disclosed for each, the specification does not mention them.

After that, the information generated in the preprocessing process is used to find out the type of interaction between the characters included in the scene and the type of film making technique applied to the book with the interaction.

4 and 5 are exemplary views showing the type of interaction between the characters according to an embodiment of the present invention and the type of film making technique applicable to the new book with the interaction.

4 and 5 show examples of scenes of drama videos that have already been aired. In the case of the 'conversation', 'action' interaction shown in Fig. 4, the 'conversation' is when the scene is composed of alternating shots of the person having a conversation, and the 'action' is the distance between the characters or within the shot. This can be found as the case of sudden change.

In the case of a new film production technique for displaying the story development shown in FIG. 5, 'graphic relationship' is a contrast of new graphic properties, that is, a movement of characters, contrast, and color, and a 'rhythmic relationship' is applicable. In contrasting the length of shots that make up a scene (for example, from a slow tempo god to a sudden fast tempo god), 'spatial relationship' is a contrast of space according to the change of background area, and 'temporal relationship' is a character's character. You can find out the properties of the time flow such as change. The type of interaction and the type of film making technique are extensible within the spirit of the present invention, and no change in the summary method of the present specification with extension is necessary.

In the present invention, the interacting gods become a unit of narrative structure for summarization, and the story development degree of each god is obtained from the above-described information.

That is, the degree of story development by unit occurred between specific scenes A and B is based on the importance of the characters interacting in A and B, the intensity of the interactions in A and B, and the film creation techniques to indicate story development. Calculate using strength. The greater the importance of characters, the intensity of interactions, and the intensity of film-making techniques, the more storytelling becomes, and the weight of each element can be adjusted according to the purpose. The following equation shows how to develop the story.

Story development degree between AB = (importance of characters belonging to A and B × W ₁ ) + (strength of interactions belonging to A and B × W ₂ ) + (intensity of film making techniques applied between AB × W ₃ )

However, A and B are arbitrary scenes including interactions, W ₁ , W ₂ , and W ₃ are narrative structures based on weighted scenes and story development degree is based on narrative structure, and story development degree is mapped to edge. The graph structure can be displayed briefly.

6 is an exemplary diagram of a narrative structure graph of a video consisting of four scenes.

When the preprocessing, narrative structure, and story development are analyzed for the source image, the corresponding graph is generated. Based on this graph, a video summary is created that consists of scenes that satisfy the user's summary time and maximize the storyline. In the embodiment of the present invention, a method of selecting and moving two scenes representing the maximum story development degree will be described.

FIG. 7 illustrates a method summarized in the example diagram of FIG. 6. That is, the two scenes are selected as shown in Figs. 7 (a) and 7 (b) to compare whether they are close to the video summary time of the user. Repeat for as long as you do not exceed the summary time.

8 is a conceptual view illustrating the configuration of an embodiment of the present invention.

According to an embodiment of the present invention, when inputting four drama videos that are already broadcast for about 50 minutes each, the recall and precision are greater than 80% in the 1/5 summary when compared with the summary generated by the screenwriter. (See FIG. 9). This shows that the present invention produces a story developable video summary.

As described above, the present invention is not limited to the above embodiment, and many variations are possible by those skilled in the art within the spirit of the present invention.

 As described above, according to the present invention, first, a video summarization method and method for extracting a feature that enables a user to understand a story from a story-driven video such as drama, and generating a video summarization that can be understood based on the story. It is possible to provide a recording medium storing a program for implementing the.

Second, according to the present invention, a video summarization method and a method for implementing the video summarization method capable of generating a video summarization capable of understanding a story in accordance with a given summarization time even if the user sets a different summarization time to be viewed by the influence of a network environment or a personal environment are implemented. It is possible to provide a recording medium storing a program for this purpose.

Third, according to the present invention, there is provided a video summarization method capable of generating a video summarization capable of comprehending a story that can reduce the manpower and time used to generate a summarization by automating all processes, and a recording medium storing a program for implementing the method. Can provide.

Claims (10)

A method of generating a summary of a source video by executing the program in a user terminal having a program installed therein, the method for performing a story understanding method for story-centric video is provided. A first step of receiving a source video to be summarized; A second step of performing a preprocessing process for extracting a shot boundary, extracting a new boundary, recognizing a character face, and extracting a recognized face position with respect to the source video; A third step of generating a narrative structure and a new story development degree based on the information generated from the preprocessing step; Receiving a summary time for the source video; And Using the screening times of the deities in the narrative structure and the story development between the deities in the narrative structure, the sum of the screening times of the gods among the combinations of the deities in the narrative structure does not exceed the video summarization time, but each scene A fifth step of extracting a combination of gods whose sum of the degree of story development between them is the maximum; The third step, Extracting, from each scene extracted from the source video, a scene forming a rookie narrative structure including a dialogue or action or interaction between characters including the dialogue and action; Finding out the kinds of interactions that are involved with the deities that form the narrative structure; Finding a kind of film making technique for story development between any two gods constituting the narrative structure; And And calculating a story development degree between any two gods constituting the narrative structure using the following equation. [Equation] Degree of story development between A and B = (Importance of characters in A and B × W ×) + (Intensity of interactions in A and B × W₂) + (Intensity of film-making techniques applied between A and B × W₃) Where A and B are arbitrary scenes containing interactions, and W₁, W₂, W₃ are weighted. ( delete ) ( delete ) ( delete ) The method according to claim 1, In the third step, the film making technology for developing the story between the gods Graphical relationships indicative of contrasts in graphical properties such as character movement, contrast, color, and the like; A rhythmic relationship representing the length contrast of the shots that make up the scene (eg, from a slow tempo god to a sudden fast tempo god); Spatial relationships representing contrast of space with changes in background area; And And at least one of a temporal relationship such as a change of character. ( delete ) ( delete ) In the recording medium readable by the user terminal, a program of instructions that can be executed by the user terminal is tangibly implemented in order to perform a video summary generation method capable of story understanding for the story-centric video. A first step of receiving a source video to be summarized; A second step of performing a preprocessing process for extracting a shot boundary, extracting a new boundary, recognizing a character face, and extracting a recognized face position with respect to the source video; A third step of generating a narrative structure and a new story development degree based on the information generated from the preprocessing step; Receiving a summary time for the source video; And Using the screening times of the deities in the narrative structure and the story development between the deities in the narrative structure, the sum of the screening times of the gods among the combinations of the deities in the narrative structure does not exceed the video summarization time, but each scene A fifth step of extracting a combination of gods whose sum of the degree of story development between them is the maximum; The third step, Extracting, from each scene extracted from the source video, a scene forming a rookie narrative structure including a dialogue or action or interaction between characters including the dialogue and action; Finding out the kinds of interactions that are involved with the deities that form the narrative structure; Finding a kind of film making technique for story development between any two gods constituting the narrative structure; And And a video summary program for causing a procedure to be performed by the user terminal including calculating a story development degree between all two gods constituting the narrative structure using the following equation. [Equation] Degree of story development between A and B = (Importance of characters in A and B × W ×) + (Intensity of interactions in A and B × W₂) + (Intensity of film-making techniques applied between A and B × W₃) Where A and B are arbitrary scenes containing interactions, and W₁, W₂, W₃ are weighted. The method according to claim 8, And a personal computer, a mobile communication terminal, and a personal digital assistant (PDA) in the user terminal. ( delete )

Images