KR100319159B1 - Content-based video retrieval data organization method and video retrieval method - Google Patents

Abstract

The present invention relates to a method of constructing content-based video retrieval data and a video retrieval method. In particular, the video content is expressed in a relationship between an object, a place, and an event, thereby making it easier to understand the video content. Make it searchable.

Conventional video retrieval system performs video retrieval by temporally arranging key frames, presenting a temporal connection of video shots, or defining an appearance period and an attribute of an object appearing in a video.

As shown in FIG. 2, the present invention is expressed as a relationship between an object (such as a character), a place, and an event (content description) that act as main components in the development of an event for a movie such as a drama or a movie. If you select the relationship between one place or the relationship between a plurality of objects and one place, search for the section that meets the conditions and present the contents (event) of the section together with the representative scene as text. Of course, it provides a content-based video retrieval data construction method and video retrieval method that makes it easier and easier to search even partial contents including a specific location.

Description

Content-based video retrieval data organization and video retrieval method

The present invention relates to a method of constructing content-based video retrieval data and a video retrieval method. In particular, the present invention relates to interrelationship elements that substantially reflect video content in terms of objects, places, and events over time. Defining and expressing the relation between the two objects is a structure of the content-based video retrieval data and its retrieval method, which makes it easier to retrieve a video from the interrelationship between the object, place, and event. A video story browser as a user interface that enables searching for a video.

Most of the conventional techniques for video retrieval have been one-dimensionally arranged key frames on a time axis.

Meanwhile, in the case of US Patent No. 5,708,767 (Method and apparatus for video browsing based on content and structure) as a conventional technology for video search, it simply shows a graph of temporal connection of shots of video.

As such, since the temporal relationship of shots presents only representative scenes, it is very difficult to understand the development of the contents from the representative scenes and the contents themselves.

On the other hand, in the prior art for video retrieval US Patent No. 5,655,117 (Method and apparatus for indexing multimedia information streams) by defining the objects appearing in the video, by defining the appearance period of each object and the attributes (properties) of the object Although it provides an object-based search method, this method also has a problem that it is difficult to understand the overall content, it is difficult to determine what specific objects and attributes to specify a search condition for the desired search because the content is not understood.

In other words, the general video such as drama or movie is based on the development of events in which the characters are intertwined in different places according to time, so simply extracting the characters or scenes summarizes the entire contents. It is practically impossible to understand and search.

The present invention provides a content-based video retrieval system that can summarize and search video contents more easily by expressing the contents of the video as a component of the video by the relationship of elements such as object, place, event, and time.

The present invention provides a video retrieval system that allows users to search general video contents such as movies and dramas in relation to main objects and places, and to search for content-based videos from a relationship graph constructed by reflecting the viewpoint of the development of events and time. to provide.

The present invention regards video content as a relationship between a main object and a place, and there are a plurality of appearance sections based on temporal elements in the relationship between the object and the place, and each appearance section has a content-based video in terms of describing an event. Provide a search system to search it.

In the present invention, the video content is represented by a graph of the relationship between the main object and the place, the main object is defined as a person, an animal, etc. according to the video type, the event is represented by the text, and the unit of the place is represented at different levels according to the video content. Together, the present invention provides a retrieval system for retrieving a video based on a unit of a place represented by a single level or a multilevel.

The present invention expresses the contents of the video as a main object, a place, and a relationship therebetween. One relationship is displayed in which one or more appearance sections exist, and one appearance section represents an actual section in the video and represents the section. The present invention provides a retrieval system for retrieving a video by displaying a frame and a part describing an event as text.

The present invention provides a content-based video retrieval system capable of expressing video contents in relation to main objects, places, and events, and retrieving not only the relationship between one object and one place, but also the relationship between a plurality of objects and places. to provide.

The present invention also provides an object information area indicating information on an object in a video, a place information area indicating information on a place correlated with the object, and event content information indicating a relationship between the object and a place. An event content information area includes information indicating a representative scene and text information describing the event content, and each information area provides a recording medium on which video search data, which are interconnected to each other, is recorded so as to be searchable.

1 is a flowchart illustrating a step of configuring search information by setting a relationship between an object and a place for video search in the present invention.

2 is a view showing an example of the structure of graph data for setting the relationship between an object and a place implemented in the present invention.

Figure 3 is a block diagram showing an embodiment of a video production and searcher according to the present invention

4 is a diagram showing an example of a user interface according to a graph data structure for setting a relationship between an object and a place implemented in the present invention.

5 illustrates another example of a user interface according to a graph data structure for setting a relationship between an object and a place implemented in the present invention.

FIG. 6 is a view showing an example of a user interface according to a case where a plurality of objects and places are selected in a graph data structure for setting the relationship between objects and places implemented in the present invention. FIG.

7 and 8 show examples of a user interface for explaining multiple levels of a place in the present invention.

The present invention relates to an element constituting the contents of a moving picture and includes information indicating an object and a place correlated with the object and indicating the relationship, and a moving picture section and a representative scene as an event of content development belonging to the relationship between the object and the place. A content-based video retrieval data structure comprising information indicative of a message and information indicating an event content of a corresponding video section, wherein the objects and places of sub-layers that depend on each of the objects and places with respect to the object and place. It may further include information indicating the relationship of.

The present invention also relates to a process of establishing a relation graph for selecting an object and a place as elements constituting the contents of a moving picture and associating objects and places with mutual correlation; A search information setting process of setting a video section and a representative scene as an event of content development corresponding to the relationship between the related object and a place, and adding contents of the event development; And a search process for presenting the video section, the representative scene and the content as search information corresponding to the relationship selected from the set relationship graph.

In addition, the present invention sets the relationship graph between the main object and the place as an element constituting the content of the video, and describes the event together with the position information of the appearance sections corresponding to the relationship and the representative frame position information representing the appearance section. A process of forming search data to be added as text, a process of displaying a relationship graph of an object and a place for the video, and when at least one relationship is selected from the displayed relationship graph, the selected data based on the location information; A content-based video retrieval method comprising a search result display process for displaying a representative frame and an event description of lighting sections existing in a relationship and screening a section desired by a user. The representative frame is displayed An event description or a more detailed event description is displayed, and a relationship between a plurality of objects and a place may be selected from the relationship graph, and when a relationship between the plurality of objects and a place is selected, a section in which the plurality of objects have appeared together at the corresponding place Content-based video retrieval method characterized in that the search and display only.

Hereinafter, the present invention will be described in more detail with reference to the accompanying drawings.

1 is a block diagram of a video search system for implementing a content-based video search method and a search browser according to the present invention.

First, step 101 shows setting of main objects and places.

In this step 101, a main object and a place are set for the contents of a video such as a movie or a drama. For example, as the main object, a main character necessary for the development of the event is set as the main object, and the event is accompanied by the character. Set the main place required for deployment, and the place to be set may appear completely different in the scene.

For example, even if the place is a hospital, the corridor scene, the hospital scene, the operating room scene, the lobby scene, the hospital yard scene, etc. in the hospital can be further subdivided. Can be set.

Of course, if it is conceptually known as a 'hospital' place and expressed in the same place helps to understand the entire video content, combine them into one main place.

In addition, even in the case of a character as a main object, an object of a lower hierarchy dependent on the character may be further set.

The next step 102 is to establish a relationship between the main object and the place set in the step 101.

For example, if character A, character B, and character C are set as main objects, and hospital, room, and outdoor are set as places, the place where character A appears (relevant) is hospital, room, character B. Place is related to room, outdoor, and character C is related to the place and character.

Since this relationship will actually be the section in which the object appears in that place in the video, it is indexed.

The section in which the object appears may mean only the moment when the object is actually visible on the screen, but may also mean the entire section of the scene in which the object appears.

The next step 103 is a step of setting the representative scene, and sets a scene that can represent the appearance section of the object related to the place from the relationship between the object and the place set in the step 102. It is set to a frame to display the frame position.

According to the above example, from the relationship between the character A-hospital, the character A represents a section in which the character appears in the hospital, and sets a 'surgical scene frame' and displays the position of the frame.

The next step 104 is an event description step, in which appropriate annotations are given for each emergence section obtained in the previous steps.

According to the above example, the operation scene frame is set as the representative scene from the relationship between the character A-hospital, and a description to summarize the content of the appearance section is added as text.

As described above, the relationship between the main object and the place is set based on the contents of the video, and the indexing of the appearance section, the representative frame setting of the appearance section, and the annotation are performed (events). Basic information of video search based is completed.

Such information may be included in a header of a multimedia stream as an example.

Then, the video reproducing (searching) system can detect and read the information contained in the header of the multimedia stream and perform a content-based search for the video.

FIG. 2 shows an example of a data structure for a relationship graph of an object and a place / event implemented through the steps of FIG.

Human 1 201, Human B 202, Human C 203 are set as objects, hospital 204, room 205, outdoor 206 as a place. Is set, the character 1 (201) has a relationship with the place of the hospital 204 and the outdoor 206, and the person 2 202 has a relationship with the place of the hospital 204 and the room 205 is established The person 3 203 has a relationship with a place called the room 205.

And, the relationship between each object 201, 202, 203 and each place 204, 205, 206 is graphed as a link 207.

The section in which the person appears is indexed according to the object-place setting relationship graphed as the link 207.

For example, in FIG. 2, three appearance sections 1, 2, 3 (208, 209, 210) in which the person 1 appears in the hospital correspond to each other based on the relationship between the person 1 201 and the hospital 204. The scene representing each appearance section, the start and end positions of the appearance section, and the event description for the appearance section are set.

For example, as shown in FIG. 2, among the sections in which the person 1 201 appears in the related place of the hospital 204, in the case of the appearance section 2 209, the appearance section is from frame 16 to frame 21. Among them, as the representative scene, frame 18 is used as the key frame 211 to store the positional information Frm_K, and the first frame Frame 16 of the appearance section 2 209 is the start frame 212. The position information Frm_S is recorded, the last frame Frame21 of appearance section 2 is used as the end frame 213, and the position information Frm_E is recorded, and the summary corresponding to appearance section 2 209 is summarized. (Event) is shown as event information (Event) in text.

On the other hand, the object related information 215 (Human Object Scheme) of the lower layer dependent on (dependent on) the characters 201, 202, and 203 as the object may be stored, and the hospital 204 as the place may be included. ) May include a Place Object Scheme 216 of a lower hierarchy dependent on (dependent on) the room 205 or the outdoor 206.

This means, for example, that the lower hierarchical places such as an inpatient room, a corridor, a lobby, an operating room, etc. are set in an object-place relationship even in a large bordered place called the hospital 204.

In the object-place / event relationship graph structure of FIG. 2, a plurality of appearance sections may exist for a relationship between an object and a place, a plurality of objects may have a relationship, and one object may have a plurality of places. Places may have relationships.

That is, not only a single level of object-place relationship but also multiple object-place relationships can be provided.

FIG. 3 shows one embodiment of an overall block diagram of the video search system of the present invention implemented from FIGS. 1 and 2 above, and the present invention is not limited thereto.

In FIG. 3, for convenience of description, the video provider is separated into a user's position.

The video provider provides search information as shown in FIG. 2 through the process of FIG. 1, and the user can perform video search by the search browser as shown in FIGS. 4 to 6 using the search information as shown in FIG. .

The digital video data 301 is stored in the input video buffer 302, and the digital video data stored in the video buffer 302 is controlled by the video processing unit 303 as a file of the multimedia video stream, and the control processing unit ( 304) produces each piece of information required or reads and processes the information.

The user interface and display unit 305 provides an interface for displaying the video data processed by the video processing unit 303 and the control processing unit 304 as well as inputting user information for data production of FIG. do.

The object-place / event relationship information of the structure of FIG. 2 processed by the control processing unit 304 is stored in the video information processing memory 306, and the object-place / event relationship information stored therein is stored in the video buffer 302. The video data is stored in the storage 307 as an appropriate format.

The video data stored in the storage space 307 and the retrieval related information of FIG. 2 are provided to the user in the form of a recording medium or via a wired or wireless broadcast communication network.

The user side detects and reads video data and object-place / event data provided from the storage space 307 to respond to user requests.

That is, data processing of the video processing unit 309 and the control processing unit 310 is performed on the video data stored in the video buffer 308, and the object-place / event data is stored in the video information processing memory 312. And the user interface and the display unit 311 respond.

That is, according to the content-based video retrieval system shown in FIG. 3, the provider side completes the process of FIG. 1 and the data structure of FIG. 2 through the user interface and the display unit 305, and stores the completed information in the appropriate storage space. In addition, the user side performs a video search as shown in FIGS. 4 to 6 through the user interface and the display unit 311.

FIG. 4 illustrates an example of a video search browser (user interface) provided to the user side, and corresponds to the relationship of an object (person) -place / event as shown in FIG.

That is, the content-based video retrieval browser 400 presents the relationship graph 401 of the object and the place produced as shown in FIG. 2 and a key frame 402 / event (content description) corresponding to the selected relationship.

The object-place relationship is established in relation to the hospital 406 and the open air 408 as a place, as human A 403 as the object, and the human B 404 as the hospital 406 as an object. ) And the room 407, and the character C (405) is set in the relationship with the room (407).

If the user selects a search for a single level of the person 2 404 and the hospital 406 associated with the link 409 as the content that the user wants to search, the control processing unit 310 is selected. ) Refers to the video information processing memory 312, the four appearance sections 1, 2, 3, 4 410 corresponding to the relationship 409 and the events (content description) 411 corresponding to each appearance section. To present.

Of course, if only one or more objects are selected from the relation graph, a section in which the object (s) appears together can be searched for and displayed regardless of a place, and if only a specific place is selected, the section in which the place is the background regardless of the object You can also search for and display only.

FIG. 5 illustrates a user interface in which a description of the appearance section and a representative frame are displayed when one of the plurality of appearance sections belonging to one object-place relationship is selected using the representative frame in FIG. 4.

That is, FIG. 5 selects the relationship between the person 2 404 and the hospital 406 in the search browser as shown in FIG. 4, and again shows the appearance area 2 among the 4 appearance areas corresponding to the relationship between the person 2 and the hospital. When it is selected, it is shown that the description 511 and the representative frame 510 of the appearance section 2 are displayed.

In this case, the search browser 500 presents the object-place setting relationship graph 501 and the key frame 502, and the characters 1, 2, 3 (503, 504, 505), the place (506, 507, 508), and the relationship link 509. The data structure of is presented.

Reference numeral 512 is presented as an icon for selecting when the user wants to play the corresponding section by looking at the representative frame 510 of the appearance section 2, and when using this, the frame from the start frame to the end frame of the section is displayed.

FIG. 6 shows a case where an appearance section and an event (contents description) are presented from one place correspondence relationship for a plurality of objects in the present invention.

The search browser 600 presents the object-place setting relationship graph 601 and the key frame 602, and the characters 1, 2, 3 (603, 604, 605), the place 606, 607, 608, and the relationship link 609 (610). When the data structure of the character is selected and a plurality of objects and a place 606 of the character 1 603 and the character 2 604 are selected, the character 1 603 and the character 2 604 simultaneously. The representative frames 611 and 613 and descriptions 612 and 614 of two sections appearing in the hospital 606 are searched and shown.

That is, among the appearance sections existing in the relationship between person 1 603, person 2 604, and place 606, two sections appearing together are two sections, representing frames 611 and 613 for the sections. ) And explanations (612, 614).

In addition, if only a plurality of objects are selected, only the section in which the selected objects appear together regardless of the location is searched.

In addition, if one place is selected, the section in which the selected place appears is searched regardless of the object.

7 and 8 illustrate examples of a user interface for describing multiple levels of a place in the present invention.

First, in FIG. 7, the search browser 700 has a relationship display unit 701 of an object and a place and an event description (explanation) display unit 702. The relationship display unit 701 of an object and a place shows a relationship between an object and a place. It has a portion 703 for displaying in a graph, a key frame 704 corresponding to the selected relationship, and a portion for displaying a description 705 of the event contents of the section.

In Fig. 7, three characters (Character A, B, C) are presented as objects, and a hospital (Hospital), a character's house (A's house), and a park (The park) are presented as places.

In addition, character A is related to the hospital and the park, character B is related to the hospital, and character C is related to the house of the character A.

In this case, when the relationship between the character B and the hospital (706) is selected, the representative scene of the section in which the character B appears in the hospital is presented as a key frame 704, and the event description (705) for the section is presented together. Seems to be.

Meanwhile, as described above, the display unit-extension button marker 707 indicates that there is a level corresponding to the lower hierarchy of the place called 'hospital'.

This extended button marker may be represented as a menu.

Therefore, when the user selects the extended button marker 707, the object-place relationship is presented at the level corresponding to the lower hierarchy of the place called 'hospital' as shown in FIG.

That is, in Fig. 8, the search browser 800 also includes an object-place relationship display unit 801, an event description unit 802, an object-place relationship graph display unit 803, a key frame 804 and an event corresponding to the selected relationship. With the description of the contents (805), the level of the lower class of the lobby of the hospital and the room of the hospital is presented for the place of the upper class of 'hospital'. A relationship 806a, 806b with character B is presented for the level of the lower hierarchy.

Therefore, the user can select the relationship 806a between the character B and the hospital lobby or the relationship 806b between the character B and the sickroom from the search browser of FIG. 8, and the key frame 804 corresponding to the selected relationship and the description thereof. 805 is presented as a search result.

In addition, if the button marker 807 is placed to group the hospital lobby and the hospital room as shown in the figure to return to the upper level hospital, the button marker 807 is returned to the search screen of FIG.

As described above with reference to Figs. 1 to 8, the present invention can retrieve the contents of a video from the relation between an object and a place.

1 to 8, one video file, that is, one drama or one movie has been described.

However, by extending the concept of the object-place based video retrieval system of the present invention, the video retrieval browser of the present invention; Represents a plurality of video files in one relationship graph, and has a structure in which the contents of each video file are expressed in a relationship graph. The entire content is roughly retrieved from the relationship graph, and the relationship graph of the contents of one video file is shown. It is also possible to implement a video search system that searches in detail.

For example, in the case of a series, even if it is divided into several video files, due to the nature of the series, different events are developed under a common object (for example, the same main character) and a common place (for example, the main place of event development). In this case, since the same events are often screened in several contexts, in this case, the contents of several video files are regarded as a single video file as a whole, and the object-place relationship graph structure of FIGS. By expressing the content of each video file in the object-place relationship graph structure as shown in Figs. 1 to 8, the entire contents of the plurality of video files are roughly searched and another video file is displayed. It is possible to search in detail about.

The present invention provides a video retrieval system that enables a user to easily summarize and search the entire or partial contents of a video by expressing the contents of a general video such as a drama or a movie in terms of an object, a place, and an event according to time. do.

The present invention can search the contents corresponding to the location by selecting a specific location as well as the summary of the entire content of the video.

The present invention presents a correlation graph based on three elements of characters, events, and backgrounds, which are basic and general elements of the development of the event, for a movie such as a movie or a drama, so that the content expression itself is very natural to understand, and an object Can also be applied to videos that contain animal stories rather than people.

In the present invention, the place can be adaptively responded even though the level may be changed according to the development of the event.

That is, according to the present invention, an object or a place of a lower layer that is dependent on the set object and place is set, making it easier to understand and summarize contents according to the development of an event.

In the present invention, various applications are possible because the video content is set in terms of the development of an object and an event according to a place and a time.

For example, for a video about a sport such as golf, the main object may be a golfer, and the main place may be a place grouped into a tee ground, fairway, rough, etc. according to each hole or characteristics of each hole, Since the event of each scene can be a golfer's swing type, it can be a good application for efficiently representing sports relay.

In addition, the present invention has a structure in which the contents of a plurality of video files are represented by one relationship graph and a structure in which the contents of each video file are represented by one relationship graph, and the entire contents of the plurality of video files are roughly searched from the relation graph. The detailed search can be performed from the relationship graph of the contents of one video file, thereby enabling the batch search for a plurality of video files.

As described above, the present invention is applied to various fields to enable a summary and understanding of an entire video and an efficient search of a desired section.

Claims (21)

In an element constituting the contents of a video including an object and place information, information about an object and a place where the object appears, a graph showing a relationship thereof, and contents of an event occurring at a place related to the object A method of constructing content-based video retrieval data, comprising retrieval data for retrieving video data including information indicating. The method of claim 1, wherein the content of the event occurring at a location associated with the object includes information indicating a video section and a representative scene. The content-based video retrieval data construction method according to claim 1, further comprising information indicating a relationship between an object and a place of a lower hierarchy dependent on each of the object and the place. The method of claim 1, wherein the object and the place are erased in at least one mutually correlated relationship. The method of claim 1, wherein the information representing the relationship between the object and the place is presented as a graph connecting the object and the place with each other. 3. The method of claim 2, wherein the information indicating the video section and the representative scene indicates a corresponding frame position of the video. The method of claim 1, wherein the information indicating the event content of the video section is text. The method of claim 2, further comprising information for continuously playing the video section in response to the information indicating the video section and the representative scene. A relationship graph setting process of selecting an object and a place as an element constituting the contents of a moving picture and associating a mutually correlated object and a place; A search information setting process of setting a video section and a representative scene as an event of content development corresponding to the relationship between the related object and a place, and adding contents of the event development; And a search process of presenting the video section, the representative scene and the content as search information corresponding to the relationship selected from the set relationship graph. Set up a graph of the relationship between the main object and the place as the elements that constitute the contents of the video, and add the event description as text along with the position information of the appearance sections corresponding to the relationship and the representative frame position information representing the appearance section. Forming a search data, displaying a relation graph of the object and the place with respect to the moving image, and if at least one relation is selected from the displayed relation graph, the relation data is present in the selected relation based on the position information. Content-based video retrieval method characterized by consisting of a search result display process for displaying the representative frame and the event description of the appearance section and the user desired section. 11. The method of claim 10, wherein the object-place relationship graph is set for all of the plurality of video files and for each video file, and the plurality of video files in total from the set video file and the relationship graph for each video file. Content-based video retrieval method characterized by performing a content search and detailed search for each video file. 12. The method of claim 10 or 11, wherein if a desired appearance section is selected from among the appearance sections, a corresponding representative frame is displayed and the event description or a more detailed event description is displayed. 12. The method of claim 10 or 11, wherein a relationship between a plurality of objects and a place may be selected from the relationship graph, and when the relationship between the plurality of objects and a place is selected, only sections in which the plurality of objects appear together at the corresponding place Content-based video search method characterized in that the search and display. 12. The method of claim 10 or 11, wherein if only one or more objects are selected from the relationship graph, a section in which the object (s) appears together is displayed regardless of the location. The method of claim 10, wherein if a specific place is selected from the relationship graph, only a section in which the place is a background is searched and displayed regardless of an object. In the elements constituting the contents of a video including object and place information, an event is included in a relationship graph selected by a user from the relationship graph with a graph showing information on a place related to a main object and displayed in the corresponding area. The video search browser, comprising: displaying content information and searching for information representing an event content of a video section corresponding to a place related to the selected object. 17. The moving picture section according to claim 16, wherein the relationship graph of the object-place has an object-place relationship graph as a sublayer dependent on an object or a place, and a moving picture section corresponding to the relationship selected from the relationship graph of the object-place of this sublayer. Video search browser, characterized by searching the contents of the case. 17. The relationship graph of claim 16, wherein the object-place relationship graph has a relationship between one object and at least one place or a relationship graph between at least one object and one place and the relationship graph with this object-place At least one relationship graph is selected from the video search browser, characterized in that for searching the event content of the video section. 19. The event content according to any one of claims 16, 17 and 18, wherein the event content corresponding to the relationship selected from the object-place relationship graph is retrieved as the representative scene of the video section and the description of the event content. Video search browser, characterized in that. An object information area representing information about an object in the video, A place information area indicating information about a place correlated with the object; As the event content information indicating the relationship between the object and the place, it has an event content information area including information indicating a representative scene of the event and text information describing the event content, and each information area is searchable. Recording media having recorded video retrieval data characterized in that they are interconnected 21. The recording medium of claim 20, wherein the place information area further includes information indicating a relationship between an object and a place of a lower hierarchy subordinate to each of the object and the place.