JPH07192003A - Moving image retrieval apparatus and method - Google Patents

Abstract

(57) [Abstract] [Purpose] To extract the images with subtitles from the moving images and display them in a list so that the user can easily find the desired scene. [Structure] With respect to a frame image captured as a digital image, a luminance determining unit 1800 and a frame collating unit 1804 detect a frame image in which a high-luminance pixel appears at the same position in a series of frame images. [Effect] Only scenes in which subtitles appear are selected and displayed in a list, and the information of subtitles is reflected in the list display, so that the user can quickly find a desired scene among them.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention detects a scene in which a caption appears in a moving image and displays a representative image of the scene as a highly important image in a list to search for a desired scene. The present invention relates to an image search device and method.

[0002]

2. Description of the Related Art In recent years, against the backdrop of high-speed and large-capacity computers, the construction of databases for moving image information such as movies and videos, which cannot be handled in the past, has become active. Along with this, a search technique for efficiently selecting a desired scene from a large amount of accumulated moving images is being put into practical use. There are two main approaches to moving image retrieval. (1) A method in which an index or keyword that describes the characteristics of a scene is added to a moving image in advance, the user specifies the desired characteristics of the scene with words or conditional expressions when searching, and the computer finds a scene that matches the characteristics. (2) A method of extracting a moving image according to some standard to show an outline of the video and presenting them directly to the user, so that the user can find out a desired scene while checking with his / her own eyes.

The method (1) has a problem that it is difficult for a general user who has only ambiguous memory and information about a desired scene to set appropriate conditions, which hinders free search. With the method (2), it is only necessary to identify whether or not the scene is a desired scene while looking at the outline of the presented video, so that even a beginner can easily perform a search. However, the number of images extracted for such an outline must be sufficiently small in order to perform the search efficiently, and only the important part is properly selected from the huge amount of moving image information to create the outline. There is a need to. Manual work is not practical as creating this summary is time consuming and labor intensive. A method of automatically extracting a representative image from a moving image is shown in IPSJ Journal Vol.33, No.4, "Automatic Indexing Method and Object Search Method in Color Video Images". In this method, a moving image is automatically divided for each scene, and one image or a partial moving image in each scene is reduced and displayed as a list, which serves as a search clue. However, in this method, there is a possibility that an important part of a scene is not included in the list, but there is a problem in that even if the scene itself is not important, it is registered in the list.

In order to solve the above problems, it is necessary to measure the importance of images displayed as a list by some scale and select the one with high importance. Moreover, in order to be practical, it is necessary that this work can be automatically performed by a computer or can be performed by a computer so as to minimize human work. As a method of automatically detecting a specific scene, there is a method of detecting a scene in which subtitles appear. One such method is described in Proceedings of the 44th Annual Conference of the Information Processing Society of Japan, 2-301, "Content Discrimination Method Based on Recognition from Grand Sumo Match".

[0005]

[Problems to be Solved by the Invention] The caption detection method described in the "content identification method based on recognition from the sumo wrestling match" is as follows.
It detects specific subtitles that have a fixed display format, such as match-up subtitles of sumo broadcasting. For example, in the case of match subtitles,
Focusing on the fact that white subtitles (Rikishi name) are presented in the center of the left half and right half of the screen respectively, binarize the image with respect to white, and detect the distribution status of the white pixels. Executed by. Therefore, there is an essential problem that it is only possible to detect a subtitle portion of a specific program and having a specific display form. Therefore, even if this method is applied as it is, subtitles having different display forms cannot be detected not only in another program but also in the same program, and therefore general information cannot be created for a general video.

An object of the present invention is to detect a variety of captions appearing in a moving image and automatically select a representative image of a scene in which the caption appears from the moving image to create an accurate video outline. In addition, in order to allow a user to easily find a desired scene and effectively utilize it, a moving image search device and method for presenting a representative image extracted in this way by using various subtitle characteristics are provided. It is to be.

[0007]

With respect to digital image data captured as a moving image, a brightness determining means for extracting a pixel having a brightness equal to or higher than a first threshold value for each frame image, and a series of the extracted pixels A frame collating means for extracting a frame image having pixels appearing at the same position with respect to the frame image is provided, and the extracted frame image is output as the representative image.

Further, with respect to the pixels extracted by the brightness determining means, the brightness difference determining means for extracting the pixels whose brightness difference in the vicinity is greater than or equal to the second threshold value by the brightness comparing means, and the frame comparing means. A subtitle area extracting means is provided for extracting an area in which pixels appearing at the same position densely in the frame image as an area where subtitles appear.

Further, the obtained representative image is presented to the moving image display means by using the information of the detected subtitle area. In particular, in order to present it to the user in an easy-to-understand manner, the representative image is presented based on the classification according to the display form of subtitles.
Alternatively, the upper and lower levels of the subtitles are determined according to the existence time of the subtitles and the representative image is hierarchically presented. Furthermore, character recognition of subtitles is performed, and a representative image is presented based on the classification based on the presence of the same character string. Alternatively, a dictionary for classifying the results of character recognition and a means for referring to the dictionary are provided, and the representative images are classified according to the meaning content of the captions and presented. Further, in order to display a list of images of many scenes at a time, and in the case where each of the scene images becomes small and the subtitles become difficult to read, a means for enlarging and displaying only the subtitle portion is provided.

[0010]

According to the present invention, regardless of the display form of subtitles,
Only scenes with subtitles in any form can be selected. Luminance is the most appropriate index for the presence or absence of captions for various forms. Generally, subtitles are often represented in white, but many subtitles such as news are not limited to white. As long as it has brightness, any form of subtitles can be determined as a unified index.

In the present invention, attention is paid to the fact that the subtitle is composed of pixels having high brightness, and the brightness determining means extracts high brightness pixels. Further, since the caption is presented at a certain position for a certain period of time, the pixel extracted by the brightness determining unit is collated with the position appearing on the frame image by the frame collating unit, and the same pixel appears at the same position for a certain period of time. Extract the frame image.

Further, since the character forming the subtitle has a large area in contact with the background, by providing a brightness difference determining means for determining the brightness difference between the pixel extracted by the brightness determining means and the surrounding pixels, Extract the edge part. In this case, frame matching is performed on the pixels extracted by the brightness difference determination means. Further, since the pixels forming the characters are densely present, a subtitle area extracting means is provided,
It is determined whether or not there are dense pixels for frame-matched pixels.

By presenting the frame images in which the subtitles detected in this way appear as a list, the user can quickly find a desired scene from the list. Furthermore, since subtitles correspond to the video content, using the subtitle information detected in this way, it is possible to sort representative images by various methods, rather than merely displaying a list of representative images, and to create a certain meaningful grouping. By presenting the representative image to the user, the moving image search can be used in an easy-to-understand manner and effectively. Specifically, by providing a means to classify by subtitle type,
The search range can be narrowed down, and a desired scene can be searched for more quickly. In addition, a hierarchical search can be performed by providing a means for determining the upper / lower order of subtitles. Further, by providing a means for recognizing the subtitles as characters, it is possible to perform a search using an arbitrary character string directly presented by the user as a key. By providing a dictionary for classifying the result of character recognition and a means for referring to the dictionary, subtitles can be classified according to their meaning. Also,
By providing a means for enlarging and displaying only the subtitle part, even when each of the scene images in the list is small,
Characters will not be crushed and will not be difficult for the user to read.

[0014]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of the present invention will be described in detail below.
FIG. 1 is an example of a system configuration for implementing the present invention. A display device 1 such as a CRT displays an output screen of the computer 4. Instructions to the computer 4 can be given using the pointing device 5. The moving image reproducing apparatus 10 is an optical disk, a video deck, or the like. The video signal output from the moving image reproducing apparatus is sequentially converted into digital image data by the A / D converter 3 and sent to the computer. Inside the computer, the digital image data enters the memory 9 via the interface 8 and is processed by the CPU 7 according to the program stored in the memory 9. Each frame of the moving image handled by 10 is numbered (frame number) in order from the beginning of the moving image. By sending the frame number to the moving picture reproducing apparatus via the control line 2, the moving picture of the scene is reproduced. Various types of information can be stored in the external information storage device 6 as needed for processing. The memory 9 stores various data created by the processing described below and is referred to when necessary.

FIG. 2a) is a block diagram showing the processing executed by the CPU 7. Arrows indicate the flow of signals or data. The processing operation of the CPU 7 is
It is roughly divided into five processing parts. Subtitle detection processing unit 1
Reference numeral 700 is taken in as a digital image from the moving image reproducing apparatus 10 shown in FIG. 1 through the A / D converter 3, receives the moving image information stored in the memory 9 and the external information storage device 6, and executes the caption detection processing. . Information such as the position and size of subtitles obtained by the processing, and image data presented as a list are stored in a data structure format described later (1710, 1712). Image information management unit 170
2 manages the input / output of those data. A user interface unit 1706 receives various instructions from the user. An arrow from the user interface unit 1706 to the image information management unit 1702 corresponds to an instruction of a caption detection / correction process described later. Further, an instruction for presenting a series of representative images, which is an outline of video, is issued from the user interface unit 1706 to the image presentation processing unit 17.
04 is transmitted. The image presentation processing unit 1704 receives the subtitle attribute information data and the image data through the image information management unit 1702 according to the instruction, analyzes them, and transmits a series of image information to the image display unit 1708 in a format corresponding to the instruction from the user. To do. The image display unit 1708 presents the image transmitted to the display device 1. Using the GUI, the input to the user interface unit 1706 and the output from the image display unit 1708 can be realized on the screen of the same display device 1.

First, the subtitle detection processing will be described. The method according to the present embodiment sequentially checks frames in a moving image (images of the smallest unit that constitutes the moving image) and identifies whether or not a caption appears. The features of the method of this embodiment that are considered to be universal to captions are as follows: 1) Each character of the caption is displayed in a color with high brightness, 2) There is a large gap between the character color and the background color. There are differences in brightness, 3) In the area where characters appear, the pairs of pixels with large differences in brightness as in 2) above are relatively dense, and 4) characters are not affected by background changes. Caption detection is performed using the following four features: that they remain at the same position for a certain period of time. Regarding 1) and 2), we use the fact that captions often use prominent colors compared to the background to make it easier for viewers to read the characters. Of course, when the background of the character has a high brightness, the character may be displayed in a color with a low brightness, so that the condition 1) must be replaced with a color with a low brightness. This may be performed by executing both the processing when the brightness is high and the processing when the brightness is low. In that case, one of the subtitles does not satisfy the other criteria based on the feature amount for detecting subtitles, and thus a subtitle with high brightness and a subtitle with low brightness can be automatically detected. 3) has a complicated shape in which characters are intricate, so there are relatively many areas where the boundary between the character color and the background color per unit area, and as a result, pixels with a high luminance difference We pay attention to the fact that there are many pairs of. In the present embodiment, the condition 4) exists at the same position for a certain period of time. However, there are cases where subtitles do not exist at the same position and move in a certain direction (for example, when characters flow from the bottom of the moving image from left to right), and in that case, it is assumed at a predetermined time interval. Considering the obtained moving direction, a condition for detecting whether or not a character exists may be added.

FIG. 2b) shows the details of the subtitle detection processing section.
The brightness determination unit 1800 receives the digital pixel data of the frame image and extracts high brightness pixels (condition 1)). The brightness difference determination unit 1802 extracts pixels with a high brightness difference by comparing the brightness of the extracted high brightness pixels with the brightness of neighboring pixels (condition 2). This process corresponds to extracting the pixels that form the edge of the character. Next, in order to determine that the extracted high-intensity difference pixels are in the same position for a certain period of time, the frame collating unit 1804 collates a series of frames (condition 4). In order to perform this matching, it is necessary to store information on a series of frames to be matched. This information is temporarily stored in the memory 9 and referred to in processing. The condition 3) is applied to the pixel candidates that form the edge of the subtitle obtained by these processes. The subtitle area extraction unit 1806 determines the existence distribution of the candidate pixels to determine whether the candidate pixels are subtitles, cuts out the area forming the subtitles, and stores information about the subtitles in a memory in a data structure format described later. Remember. Threshold management unit 1
808 sets and saves a threshold value for executing the above extraction and determination processing.

However, in the present embodiment, the case in which the judgment is carried out under the above-mentioned four conditions is described as the structure capable of detecting the caption most accurately, but these four conditions are different in importance. Especially important are the detection of high-luminance pixels under condition 1) and the matching of frame images under condition 4). An embodiment based on these two condition determinations will be described later as another embodiment.

The method of this embodiment will be described in detail with reference to the flow chart shown in FIG. A program for realizing the above flowchart is stored in the memory 9. CPU7
First, as initialization processing, the variable t representing time is set to 0.
(100). The variable t here has almost the same meaning as the frame number. Next, the moving image reproducing apparatus 10 is controlled to reproduce the moving image, and the frame image is taken into the memory 9 as a digital image of size w × h by using the A / D converter 3 (102). Although this size is arbitrary, in view of the nature of this algorithm, in order to improve the reliability of detection, it is desirable that the size is large enough so that the lines forming the characters in the digital image are not connected. In this case, the features required for subtitle detection are retained. Then, a three-dimensional array E (x, y, t) of size w × h is prepared at the time point of time t.

For each pixel of the captured image, the following processing is performed (106) to (12) in order to determine whether or not the pixel is one of the pixels forming a caption.
4). (1) It is determined whether the brightness (density) level is equal to or higher than the threshold value th1 (108). If it is equal to or larger than the threshold value, the process proceeds to 110. Otherwise, it is determined that the pixel is unrelated to the subtitle and 116
Is executed and 0 is assigned to the data area of the array E corresponding to the current pixel coordinate. (2) The coordinates of 8 neighboring pixels, that is, the coordinates of the current pixel are (x,
y), (x-1, y-1), (x, y-1), (x + 1, y-1), (x-1,
y), (x + 1, y), (x-1, y + 1), (x, y + 1), 8 pixels located at the coordinates (x + 1, y + 1) and the current pixel The difference in brightness between each pixel is calculated, and it is checked whether there is a difference in brightness greater than or equal to the threshold value th2 (110). If there is at least one, proceed to 112, otherwise proceed to 116.

(3) It is judged whether the brightness difference exceeds the threshold value in all the pixels in the vicinity of 8 (112). If all the pixels exceed the threshold value, the pixel is an isolated point and is likely to be noise on the contrary. Therefore, the pixel is unrelated to the subtitle, and the process proceeds to step 116. If not, 114 is executed as one of the pixels forming the subtitle, and 1 is assigned to the data area of the array E corresponding to the current pixel coordinates. 11
The process 2 may be unnecessary depending on the size of the sampled image data. When the size of the image is small, the ratio of the area occupied by the entire subtitles per pixel is high, and therefore the process of 112 is omitted.

(4) The variables x and y indicating the current coordinates are increased by 1 in correspondence with the rows and columns, and finally the processes of (1) to (3) are performed for all the pixels in the image. Do it (11
8)-(124).

The threshold th1 can be set in the system. It is desirable that the brightness setting value is high from the viewpoint of extracting only subtitles. However, because the brightness of the pixels that make up the subtitles varies, if the value is too high, many pixels that make up the subtitles are not determined to be so and inconvenience occurs when determining the brightness difference. It is necessary to set it with a certain amount of margin. For example, an appropriate result can be obtained with about 70% of the highest gradation. Further, it is desirable that the threshold value th2 be variable to some extent depending on the value of the background brightness. That is, when the background brightness is high, the threshold th2 is set small, and when the background brightness is low, the threshold th2 is set larger. An average value of the luminance values of the entire image is taken as a reference of the background luminance, and the threshold value th2 is defined as its function. The thresholds th1 and th2 and th3 described later are managed by the threshold management unit 1808 in FIG. It is preferable that the threshold values th1 and th3 can be modified. Threshold th2
With respect to, the brightness determining unit 1802 receives the brightness information of the frame image and sets the value. As a relatively simple configuration, the coefficients given to the thresholds th1, th3 and the function for determining the threshold th2 may be retained as the initial values. Furthermore, in order to detect subtitles effectively, it is necessary to tune the value. For example, if there is an inconvenient specific subtitle due to omission of detection, caption detection is performed on the frame image in which the subtitle appears, while changing the threshold. The result is fed back and subtitle detection is executed with an optimum value as a threshold value. By customizing the threshold value in this way, appropriate subtitle detection processing can be performed.

Next, it is determined that a character exists at the same position for a certain period of time. The two-dimensional array E created by the above processing is stored for the past n frames. The logical product of the two-dimensional array E for those n frames is calculated. In particular,
All the data corresponding to the same coordinates in each array are compared, and if they are all 1, 1 is substituted into the two-dimensional array E '(x, y). If there is at least one 0, then 0 is similarly assigned to the two-dimensional array E '(x, y). Then, E '(x, y) is output as the logical product calculation result (126).

Next, it is determined whether or not the pixels, which are regarded as a part of the subtitle, are actually densely present in the frame image when the above processing is completed. First, E '(x,
For each row (x direction) and column (y direction) of y),
The number of those whose value is 1 is counted. Specifically, in the x direction, x is i and the value of E '(x, y) is 1
The frequency histogram Hx (i)
To create. Similarly, in the y direction, y is i,
In addition, the number of E '(x, y) values of 1 is counted and a frequency histogram Hy (i) is created (128). These histograms indicate in which row or column the pixel like a subtitle exists. If some of the frequencies in the histogram exceed the threshold th3 (13
0), it is finally determined that the caption appears in the row or column corresponding to the frequency. Then, since E'is the result of the logical product, it is judged that the subtitle appears in the frame n frames before (132). Then, the caption detection processing 132 is executed. Here, the threshold th3 can be determined based on the size of the smallest character used as a subtitle. The subtitles utilize the fact that any video has characters of a certain size or more as long as it is presented for viewing by the viewer.

Details of the subtitle detection processing 132 are shown in FIGS. If it is determined from the frequency histogram that there is a line exceeding the threshold th3 (800), the process according to the flowchart of FIG. 4 is performed. The part where the line exceeding the threshold th3 is continuous is extracted. At this time, the threshold th
If there are multiple rows or columns with a frequency higher than 3 and there is not enough frequency in the rows or columns between the peaks, it is considered that multiple subtitles have appeared, and those subtitles are displayed. Find the number of and assign it to Ln (80
2). And for all subtitles, the following 804-8
20 processes are performed. Ln is used as a loop counter and is decremented by 1 each time one subtitle is processed (8
16), loop until 0 (820). 804 detects a portion in which lines that exceed the threshold value th3 are continuous, finds the section, and sets the y coordinate of the starting line to yo and the y coordinate of the ending line to yc.
To. Next, in E '(x, y), the number of 1s in which y is between yo and yc is counted to create a histogram H'x (i) (806), and the threshold th4 is exceeded in the histogram. Extract the part where the frequency is continuous. Then, the x coordinate of the start column of the column corresponding to the continuous portion is substituted for xo, and the x coordinate of the end column is substituted for xc (808). Thus obtained
The rectangular area indicated by (xo, yo) and (xc, yc) is the subtitle existence area. This can be regarded as a horizontal subtitle area. This takes into account the feature that subtitles are composed of multiple characters, and the area is elongated in the row or column direction. If it is determined that there is a subtitle in the rectangular area (xo, yo)-(xc, yc) from E'at time t-1, it is assumed that this is a subtitle that has appeared before, and special processing is performed. No (812). If there is no subtitle, it is determined that a new subtitle has appeared, and the time or frame number n frames before the subtitle is stored as the start time of the subtitle (814). The current time is not used because the caption appears at E ′ only when the caption appears continuously at the same position from n frames before. The subtitles detected in this way are
The time, which is constantly monitored in the program, is subtracted from the time t when it is no longer detected, or the corresponding frame number is stored as the end time of the caption. Then, all the elements of the rectangular area (xo, yo)-(xc, yc) in E '(x, y) are set to 0 (818).

On the other hand, when there is a frequency greater than the threshold value th3 for the column, the processing shown in the flowchart of FIG. 5 is performed (822) to (842). This is the same as the above process performed for rows. This is regarded as a vertically written subtitle area. If both the row and the column exceed the threshold value th3, both the vertically written and horizontally written subtitle areas can be detected by continuously performing the processing of the flowchart of FIG. 4 and the processing of the flowchart of FIG. Finally time t
One step, return to 102 and repeat the above process (1
34).

The data structure 2 shown in FIG. 6 shows each of the sections (caption sections) in which the captions are found by the above means.
It is stored in the format of 00. The beginning frame number of the subtitle section is stored in 204 as the starting point frame number, and the ending frame number is stored in 206 as the ending point frame number.
This is 810 to 81 in the flowchart of FIG. 4 (FIG. 5).
4 (832 to 836). Reference numeral 202 is a representative frame number, and the first frame is selected as a default, but the user can arbitrarily set it. The subtitle position 208 stores a pointer to a subtitle position structure that stores the subtitle position. The upper subtitle section 210 is a pointer to a subtitle section structure of a higher subtitle, which will be described later. Subtitle section 2
Conversely, 12 is a pointer to the subtitle section structure of the lower subtitle. The same subtitle section 214 is a pointer to a subtitle section structure of subtitles having a common upper subtitle, and forms a concatenation list with the subtitle section structure designated by this pointer. By tracing this same subtitle section pointer 214, all subtitle section structures of subtitles having a common upper subtitle can be listed. The corresponding scene 216 is a pointer to a scene structure described later. Reference numeral 218 is a pointer for storing a subtitle section structure for all subtitles appearing in the corresponding scene in a concatenated list format.

FIG. 7 shows the structure of the subtitle position structure 300 described above.
Shown in. 302 is a pointer to the next subtitle position. If the subtitle is displayed in a fixed position throughout the existing section, the subtitle position can be represented by one structure.
Contains NULL. If the caption is a flowing caption, which will be described later, the caption position changes, so a plurality of structures are created, the concatenation list is constructed by the pointer 302, and the caption position data is stored. One subtitle position structure is created for each section in which it is determined that the subtitle is displayed at the fixed position. This determination also changes depending on how accurate the position information of the caption is required. The start point frame number of such a section is stored in 304, and the end point frame number is similarly stored in 306. In 308, the x coordinate of one apex when the region in which the caption appears is surrounded by a rectangular region, and the y coordinate thereof is entered in 310. The width of the rectangular area is stored in 312, and the height thereof is stored in 314. The rectangular area is 804 to 808 in the flowchart of FIG. 4 (FIG. 5).
It is obtained by the processing of (826 to 830).

FIG. 8 is an example of the scene structure 500. In order to enable the subtitles appearing in the scene to be called immediately during the playback of the scene, the corresponding subtitle section is included in various attribute information such as the representative frame number 502, the start point frame number 504, and the end point frame number 506. Fifty
8 pointer storage areas are provided. Corresponding subtitle section 508
Holds information about the subtitles that appear in the scene,
Specifically, all subtitles appearing in the scene are stored in a linked list format. This corresponds to the subtitle 218 of the corresponding scene in the subtitle section structure 200.

The information on the subtitles obtained by the above processing is automatically obtained by a computer or the like, but there are cases in which erroneous determination is included. When referring to such subtitle information only once, some erroneous determination is unavoidable, but it is a great dissatisfaction for the user to repeat the same error when referring to the subtitle information twice or three times. If there is a demand to be referred to multiple times, it is effective to correct the error at the first time and memorize the error-free one to use for the next reference.
The caption detection / correction process will be described below. This process
If you notice that there is an error in the detection of subtitles during the image presentation process that will be described in detail later, the image presentation process is terminated at any time,
The caption detection / correction process can be executed.
FIG. 10 is an example of an interface for correcting the detected subtitle information.

The correction window 1000 is displayed on the computer screen and accepts various operations with the mouse cursor. Reference numeral 1002 denotes a frame image of a part of the video image which is sampled at equal time intervals and arranged in a horizontal row like a negative of a photograph. The sampling time interval can be determined by 1018 scroll bars. The time interval can be freely changed by dragging the knob 1032 in 1018 with the mouse, and the frame image appearing in 1002 also changes accordingly.
The closer 1032 is to the left edge, the shorter the time interval,
The closer to the right end, the longer the time interval. Which partial section in the video is displayed can be changed by the scroll bar and button of 1016. Knob 103 in 1016
The position of 0 indicates the position of the partial section to be displayed, and can be freely changed by dragging 1030. When it is moved to the left end, the beginning part of the image is displayed, and at the right end, the end part of the image is displayed. The width of the knob 1030 corresponds to the length of the displayed partial section. This length is 1002
Can be easily calculated from the number of frame images that can be displayed at one time, the time interval determined in 1018, and the time length of the entire video. The arrows 1004 and 1006 represent the beginning and the end of the section in which the caption appears. Reference numeral 1014 indicates the position of the detected subtitle section in the entire video. The position of the detected subtitle section 1012 in the entire image can be known by the scale display of 1014. The small arrow 1010 attached to 1012 is 1
This corresponds to the arrows 004 and 1006. 100
Reference numeral 8 denotes an enclosing frame showing the position where subtitles exist in each frame. The position of the arrow 1004 or 1006 can be changed, and the arrow disappears when the arrow portion is clicked with the mouse. Then, when you click on any frame, an arrow with the same direction as the erased arrow appears at that position and a new section is defined. As a result, the section expands and contracts, but when it expands, the subtitle position structure 300 for the expanded section
Is created and conversely becomes shorter, the subtitle position structure of the deleted section is deleted from the registration. 1004 and 10
When 06 is deleted continuously, it is determined that there is no caption in the section, and the registration of the caption section structure 200 is deleted. At the same time, the pointers to this subtitle section structure stored in the upper subtitle section, the lower subtitle section, etc. are deleted from all the subtitle section structures having the pointer to this subtitle section. Further, the subtitle position can be changed by dragging 1008. If you grab any of the four vertices of 1008 and drag it, you can scale the rectangle by fixing the vertices facing each other through the diagonal line. With this change data, the subtitle origin x and y coordinates of the subtitle position structure and the width of the subtitle size,
The height storage area is updated.

A method of presenting the information on the subtitles obtained by the subtitle detection processing described above in an easy-to-understand manner so that the user can effectively use the information will be described. FIG. 11 shows an example of presentation on the display of the computer. The window 700 extracts typical frame images showing an outline of video and displays them as a list. The representative frame image is selected by mechanically extracting the first image of the subtitle section. If the size of the representative image is the same as the image sampled at the original resolution of NTSC, it cannot be displayed on the computer display or the number of images that can be displayed is limited. The user can search for a desired scene while checking these representative images. When the scene to be viewed is found, the scene is displayed on the monitor window 702. In this case, the selection of the scene to be reproduced is performed in conjunction with the click on the scene displayed in the window 700. Monitor window 702
As a method of reproducing the video, there are a method of using the video output from the moving picture reproducing apparatus connected to the computer and a method of using the digitized video registered in the external information storage device. In the case of the moving picture reproducing apparatus, the frame number at the beginning of the scene is sent to the moving picture reproducing apparatus, and the reproduction is started from the scene corresponding to the frame number. Then, when the frame number at the end of the scene is reached, an instruction to suspend the reproduction is sent to the moving picture reproducing apparatus. In the case of a digitized video, the image data of the frame corresponding to the frame number at the beginning of the scene is extracted from the video data stored in the format of the physical video structure 400 shown in FIG. 9 and displayed as graphic data. When the display process for one frame is completed, the display process of the next frame is continuously performed to display the moving image. Adjust the number of frame images to be displayed per fixed time according to the time required for display processing so that the moving image does not tend to fast forward or slow. A button 704 is a button for controlling the reproduction state of the video image, and controls reproduction of the video image and fast forward / rewind according to the click of the mouse or the touch panel. By these operations, it is possible to easily confirm before and after the searched scene. Reference numeral 708 denotes a text input / output window, which is used for searching for character strings in subtitles, which will be described later.
The keyboard 11 is used to input text. These windows can be moved to arbitrary positions on the screen by operating a cursor 706 that can be freely moved using a mouse which is one of the pointing devices 5. Alternatively, the same thing can be done by using a touch panel which is another type of pointing device.

The window 700 for presenting the representative image is
By using the detected subtitle information, it is possible to facilitate the user's work of searching for a desired scene. Hereinafter, a method of presenting a list of representative images using subtitle information will be described. As a method of displaying the list, the method of arranging in time series in the order of appearance in the video image is basically used. The time series is determined based on the frame number of the representative image. In FIG. 12, the representative images 604 are simply arranged in time series, with the upper left being the head side of the video image and the lower right being the end side. If the data cannot be displayed in one line, the data is folded and displayed over a plurality of lines as shown in the figure. Further, if the data cannot be displayed in one window, a scroll bar 602 appears and scroll display is performed.

It is also effective to classify the representative images according to the detected captions and present the representative images using the classification. For example, 1) Presentation based on the classification of the representative screen based on the display form of the subtitles 2) Presentation based on the inclusion relation of the subtitle presentation time or the hierarchization of the representative screen depending on the display form of the subtitles 3) Based on the classification of the representative screen based on the contents of the subtitle Presentation The user can specify how to display the list in the image presentation processing mode. In these display designations, by storing display information in a temporary memory, it is possible to sequentially designate these display methods and narrow down representative images as the user's search progresses. of course,
If the specified display method is inappropriate, it is possible to cancel the specification and specify a new display method.

FIG. 13 shows an example in which the captions are categorized according to the display form and displayed based on the categorization. The classification is
For example, by using the four features of 1) the brightness of the character, 2) the color of the character, 3) the size of the character, and 4) the vertical writing / horizontal writing, they can be classified automatically or by the user. This can be done by specifying the item. That is, the origin x, y coordinates and size width and height of the subtitle are determined by the subtitle position structure for the representative image, and the subtitle portion determined by the above-mentioned data by the physical video structure depending on the required classification item of pixel data. It is possible to determine the display form of the subtitles by referring to the pixel data of, and classify and present the representative image accordingly.

1) is a classification according to whether the character part of the subtitle is expressed in a color lighter than the background or in a color darker than the background. This is judged by the brightness level of the subtitle part. In 2), when the background and the character part are separated, it is classified by what color structure the character part is represented. This takes the color histogram of the character part and judges by the similarity between the histograms. Regarding the character portion, if the luminance level of the subtitle portion is high, it is determined that the pixel with high luminance is a pixel forming a character, and if the luminance level of the subtitle portion is low, the pixel with low luminance is a pixel forming a character. can do. 3) is classified according to the length of the short side of the subtitle area. Vertical writing / horizontal writing in 4) can be judged by whether the short side is parallel to the x-axis or the y-axis. With these, subtitles of the same type can be grouped together as one group. When the subtitles are similar in shape, the contents of the scene often have a common undercurrent, so it can be expected that the classified scenes are classified with commonality in contents.

FIG. 14 shows an example in which representative images are hierarchized based on a certain standard, and the hierarchized state is visually displayed. The hierarchization criterion is, for example, the inclusion relation of the time when subtitles exist or the display form of subtitles. The inclusion relation of the time when subtitles exist means that the subtitles corresponding to the headline for a scene often appear throughout the scene, and the subtitles that occasionally appear in the meantime are more detailed within the scene. This part is used because it corresponds to subheadings, so to speak, subheadings.
The section in which a subtitle exists can be obtained by referring to the starting point frame number and the ending point frame number of the subtitle section structure for each subtitle. Therefore, by checking the mutual inclusion relation of the section, whether the subtitle is a large caption or a subtitle. You can distinguish what. FIG. 14 shows the representative image 604 of the scene in which the representative images in which the subtitles of the large headline appear are arranged in the upper layer and the representative images of the small headlines are arranged in the lower layer, and the representative images of the upper and lower layers are indicated by a line 606 so that the inclusion relation can be seen. Is tied. Such a structure tree is created for each group having a containment relationship, and is arranged in the horizontal direction. You can also increase the number of layers in the structure tree by grouping subtitles in the upper layer. When it cannot be displayed in one window, scroll bars appear in the vertical and horizontal directions to scroll the part that cannot be displayed. When the number of representative images is large, it is possible to display only the representative images of the upper layer subtitles to roughly find candidates and then display the lower layer subtitles to narrow down. In the above example, the upper layer and the lower layer are determined by the inclusion relation of the existence time, but the upper layer and the lower layer can be determined by using the result of the classification described with reference to FIG. In particular, for news programs, etc., we focus on the fact that after explaining the general description with subtitles that correspond to the main heading, we explain the detailed matters and then explain the general matter of another case. .
By the method described above, a group of major headlines is extracted, the scene in which they appear is used as a representative image in the upper layer, and the subtitle section structure of each subtitle is referenced, and the subtitle of the next major heading appears from that scene. A representative image of each scene that appears up to the current scene is displayed in the lower layer. In addition, the system holds judgment criteria automatically for upper and lower layers, or the user can arbitrarily select a group classified in the classification display according to the above-mentioned display form to select the upper layer scene. It is also possible to set and execute hierarchical display.

Consider recognizing a subtitle appearing in a representative image as a character and utilizing the information. By applying the character recognition technology that has been widely used in the past to the character portion of the caption determined by the brightness, the caption appearing in the representative image can be expressed by a character code such as JIS code. The characters of JIS code can be converted by Kana-Kanji conversion method used in Japanese word processor.
Since an arbitrary character string can be input by a human from the keyboard, it is possible to search for subtitles using the input character string as a key, that is, to search for a scene in which such subtitle appears. Further, by recognizing the characters of the subtitles, it becomes possible to group by the contents of the subtitles. Subtitles that contain the same words can also be classified as the same group. Furthermore, by creating a dictionary that associates words with their meanings in advance, it is possible to identify whether the words in the subtitle character string are, for example, a person's name or a technical term in a specific field. . A person's name can be classified into a group called subtitles related to a person's name, a political term can be classified into a group related to politics, and a medical term can be classified into a medical group. FIG. 15 shows an example of the display format. A character string 608 is displayed at the left end of the window 600. Also,
FIG. 16 shows an example of a structure of a dictionary for classifying words. 900 is an array for storing words, and 902 is a coded word, which also has a pointer to an attribute information array 904 for classification. A classification symbol 906 is coded and stored as to whether the word is a person's name or a political term. Since words have polysemy, I try to have multiple classification symbols.

In the list display described so far,
In order to increase the number of scenes that can be checked at a glance in order to improve the efficiency of search, the display size and resolution of the display are limited, so the representative image of the scene must be reduced and displayed. However, there is a problem that subtitles become unreadable when the reduction ratio is increased. Therefore, FIG.
As shown in, the subtitle portion stored in the subtitle position structure is cut out and only that portion is relatively enlarged, that is, the reduction rate is made lower than that of the representative image 604 and displayed near 604. At this time, in the case of horizontal subtitles, 604
It should be displayed below, and in the case of vertical writing, it should be placed on the right side so that there is as little space between the representative images as possible. Since the subtitle portion is a smaller area than the entire representative image, even if an image in which the subtitle portion is enlarged is added, the area required for display is much smaller than the reduction ratio of the entire representative image. This makes it possible to achieve both display of many representative images and proper reading of subtitles.
This display method is effective not only when selectively displaying a list of scenes in which subtitles have been described, but also when simply displaying a list of all scenes including scenes in which subtitles appear. is there.

Although the most effective method has been described in the above-mentioned caption detection method, another embodiment of the present invention will be described below. Although the luminance difference between the subtitle and the background is used in the above embodiment, the present invention is not limited to this.
You can also use color information. This is because the pixels forming the subtitles have a color that is more noticeable than the background, such as white. This can be realized by replacing the judgment about the brightness in the configuration of FIG. 2b) with the judgment about the color. The method of using color instead of luminance is more suitable for detecting subtitles having a characteristic display form. That is, there is an advantage that the caption can be selectively extracted by specifying a certain color and detecting the caption. Therefore, although the size and the like may change depending on the content, such as the headline of a news program, it is efficient when it is desired to always find a subtitle presented in the same style. The threshold value in this case is set as a certain range of colors centered on a specific color. Specifically, for example, it can be defined as a partial space centered on a specific color in the RGB color space. As for the setting, it is desirable to call a screen having a subtitle to be detected, and register the coordinates in the RGB color space corresponding to the subtitle color and the subspace centered on the coordinate in the threshold management unit as a table.

Although the example using the RGB color space has been shown above, the present invention is not limited to this, and for example, C
A uniform color space such as the IE 1976 Y * u * v * color space can be used. The uniform color space is a space in which the sensory difference of color change is proportional to the Euclidean distance from a point in any direction in which the color changes. Also in this case, a certain range of colors is represented as a subspace occupying the color space. In this case, there is an advantage that it is possible to define a color range more faithful to the user's vision.

FIG. 18 shows a combination of the determination based on the luminance and the determination based on the color. Brightness determination unit 180
0, a brightness difference determination unit 1802, a frame matching unit 1804,
The subtitle area extraction unit 1806 has been described with reference to FIG. The color determination unit 1810 and the color difference determination unit 1812 are blocks that perform caption detection based on the color information described above. The color determination unit 1810 detects pixels having a certain range of colors.
With respect to the pixels detected by 1810, the color difference determination unit 1812 detects pixels corresponding to the edge portion of the character by comparing them with neighboring pixels. The threshold management unit 1808 is different from that of FIG. 2B) in that the threshold management unit 1808 manages the threshold information about the color as described above because the determination based on the color is also performed. It is possible to switch between the determination based on the brightness and the determination based on the color by the selector 1814, and the efficient detection according to the purpose can be performed. In the method of specifying the detected caption, it is also possible to restrict it by the appearance position of the caption.

Further, although the subtitle is stationary for a certain period of time is used as a feature amount for detection, the present invention can be applied not only to a static subtitle but also to a scroll type subtitle. That is, for a frame at a certain time,
A candidate for a subtitle-like region is selected using the feature amount for detecting subtitles excluding stationary for a certain period of time. From the detected
In the next frame, the candidates are narrowed down to those in which regions in which the same features appear in the same sequence are in the vicinity. Then, in the next frame, the candidates are limited to regions in which the same features also appear in the same arrangement in the vicinity in the same direction and the same distance as in the previous frame. Even if this is repeated for several frames, only those that are continuously detected are determined as the subtitle area.

From the viewpoint of the display method, it has been described that the scenes in which the subtitles appear are displayed as a list, but a display method in which only those scenes are collected and sequentially reproduced on the monitor screen is also useful. is there. It can be used as a video digest.

The method of the present invention functions extremely effectively when the system has one search mode for performing video search and is used in combination with a plurality of search methods. Another method of performing video search is a method of associatively searching for video by the applicant. According to this method, a user registers in advance the objects that appear in the video, and presents a list of the screens in which the registered objects appear as representative images. The user can find a desired screen by following the presented screen. The caption detection of the present invention is suitable for cutting out a representative image corresponding to a change point of a video subject. Among them, for example, when a person A of interest is found, the person A can be moved to the associative search mode, the person A can be registered, and how the person A appears in the entire image can be examined. As described above, the user can perform a multifaceted video search by selecting various search modes and sequentially using them.

The method described so far has been described as using all of the subtitle pixel determination conditions 1) to 4) according to the first embodiment. However, as described above, the indispensable configuration of the present invention is the luminance (color) determination and the frame image comparison, and it may be desirable to reduce the load on the apparatus by limiting the determination conditions only to them. That is, this is a case where such subtitle detection processing is applied to a general video image reproducing device or a television with a video image reproducing means. In such a case, rather than accumulating subtitle information and using it many times, real-time processing is required to search the scenes in which subtitles appear one after another to find a desired scene. Ah In this case, it is effective to narrow down the determination condition from the viewpoint that the processing speed is of the highest importance.

In such a case, it is possible to omit the determination of the difference in luminance or the difference in color from the neighborhood. Further, it is not necessary to determine the subtitle area, and the frame image may be presented as it is when a different subtitle is detected, and therefore the subtitle area extraction processing can be omitted. As a method of presenting an image to the user in that case, it is effective to display the above-described images in order or to switch and display the windows 700 and 702 in FIG. That is, the video search mode and the video playback mode are provided, and in the video search mode, the images detected in the list are presented on the entire screen, and when the user designates the image to be viewed while looking at them, the screen is switched to the video playback mode screen. As with normal video playback, the image is displayed on the entire screen from the specified scene.

[0049]

According to the present invention, since only the scenes in which subtitles appear are selected and displayed in a list, the user can quickly find a desired scene from among them.

[Brief description of drawings]

FIG. 1 is a system block diagram for realizing an embodiment of the present invention.

FIG. 2 is a block diagram showing a flow of processing in a CPU.

FIG. 3 is a flowchart of an algorithm for detecting a subtitle.

FIG. 4 is a flowchart showing details of subtitle detection processing.

FIG. 5 is a flowchart showing details of subtitle detection processing.

FIG. 6 is a structure for storing a subtitle section.

FIG. 7 is a structure for storing a subtitle position.

FIG. 8 is a structure that stores a scene.

FIG. 9 is a structure that stores the image itself.

FIG. 10 is an example of a graphic interface for correcting the result of subtitle detection.

FIG. 11 is an example of a graphic interface for displaying a list of subtitles.

FIG. 12 is an example of displaying a list of scenes in which subtitles appear.

FIG. 13 is an example in which subtitles are classified and displayed according to types.

FIG. 14 is an example of hierarchical display in accordance with the inclusion relationship of subtitle existence sections.

FIG. 15 is an example in which classification is performed and displayed according to the character recognition result of subtitles.

FIG. 16 is an example of a structure of a dictionary for classifying recognized words.

FIG. 17 is an example in which only a subtitle portion is relatively enlarged and displayed.

FIG. 18 is a block diagram of processing of a caption detection processing unit.

[Explanation of symbols]

1 ... Display, 2 ... Control signal line, 3 ... A / D converter, 4 ... Computer, 5 ... Pointing device,
6 ... External information storage device, 7 ... CPU, 8 ... Connection interface, 9 ... Memory, 10 ... Moving image reproducing device, 11 ... Keyboard.

Claims (18)

[Claims] 1. A moving image search apparatus for extracting a part of a moving image captured as digital image data as a representative image and outputting the representative image on a moving image display means, in the digital image data, for each frame image. A luminance determining means for extracting a pixel having a luminance equal to or higher than a first threshold value, and a frame collating means for extracting a frame image having pixels appearing at the same position in a series of frame images for the extracted pixel, A moving image search device, wherein the extracted frame image is output as the representative image. 2. The moving image search device according to claim 1, wherein, with respect to the pixels extracted by the brightness determining means, pixels having a difference in brightness that is equal to or larger than a second threshold value are extracted as compared with the brightness of neighboring pixels. A luminance difference determination means and a subtitle area extraction means for extracting, as a subtitle appearance area, an area in which the pixels appearing at the same position are densely present in the frame image extracted by the frame matching means, Video search device. 3. The moving image search device according to claim 2, further comprising threshold management means for setting the second threshold value as a variable of the luminance of the frame image. 4. A moving image search apparatus for extracting a part of a moving image captured as digital image data as a representative image and outputting the representative image on a moving image display means, in the digital image data, for each frame image. A color determining means for extracting pixels having a first fixed range of color, and a frame matching means for extracting frame images having pixels appearing at the same position in a series of frame images for the extracted pixels, A moving image search device, wherein the extracted frame image is output as the representative image. 5. The moving image search device according to claim 4, wherein the pixels extracted by the color determining means are pixels having a color difference within a second predetermined range as compared with colors of neighboring pixels. And a subtitle area extracting means for extracting, as a subtitled area, an area in which pixels appearing at the same position are densely present in the frame image extracted by the frame matching means. Video search device. 6. The moving image search device according to claim 5, further comprising threshold management means for holding the second fixed range as a table. 7. A moving image search device for extracting a part of a moving image captured as digital image data as a representative image and outputting the representative image on a moving image display means, in the digital image data, for each frame image. A luminance determination unit that extracts a pixel having a luminance equal to or higher than a first threshold value, and a pixel that has a luminance difference that is equal to or greater than a second threshold value with respect to the pixels extracted by the luminance determination unit. A brightness difference determining means for extracting a pixel, a color determining means for extracting a pixel having a color within a first fixed range for each frame image in the digital image data, and a neighboring pixel for the pixel extracted by the color determining means. Color difference determination means for extracting pixels whose color difference is within a second fixed range as compared with the color, and the second threshold as a variable of the luminance of the frame image. Threshold management means for setting a second fixed range as a table, a selector for selectively inputting the digital image to one of the brightness determination means and the color determination means, and the brightness difference determination The pixel extracted by one of the means and the color difference determination means, a frame collating means for extracting a frame image having pixels appearing at the same position in a series of frame images, and a frame image extracted by the frame collating means. A moving image characterized by having subtitle region extracting means for extracting a region in which pixels appearing at the same position are densely present as a region in which a subtitle appears, and outputting the extracted frame image as the representative image. Search device. 8. The moving image search device according to claim 1, 4 or 7, wherein the moving image display means has at least two moving image display screens, and a video is displayed on the first moving image display screen. A moving image search apparatus, comprising: an image presentation processing unit that displays the image in a reproduced state and displays a list of the representative images on the second moving image display screen. 9. The moving image search device according to claim 8, wherein the representative images are displayed in a time series list. 10. The moving image search device according to claim 8, wherein in the detected subtitle area, all or part of the brightness, color arrangement, size, and arrangement of characters appearing in the subtitles are classified as classification conditions, and each of them is classified. A moving image search device characterized by displaying a list of the representative images on the screen. 11. The moving image search device according to claim 8, wherein in the detected subtitle area, when one subtitle has a plurality of other subtitles, the one subtitle is an upper subtitle. A moving image search device, characterized in that a plurality of other subtitles are determined as subordinate subtitles and the representative image is hierarchically displayed as a list. 12. The moving image search device according to claim 8, wherein the subtitles in the detected subtitle area are character-recognized, and subtitles at least partially including a common character string are classified into one group, and each subclass is classified. A moving image search device characterized by displaying a list of the representative images on the screen. 13. The moving image search device according to claim 8, wherein the subtitles in the detected subtitle area are character-recognized, and a dictionary in which words are associated with classification items for classifying words according to their meanings is held in advance. As a result, the moving image search device is characterized in that the representative images are displayed in a list for each classification. 14. The moving image search device according to claim 8, wherein, for the representative images displayed in a list, only the subtitle area is displayed relatively enlarged as compared with the representative image. apparatus. 15. A video image reproducing apparatus for outputting a stored video image as a video signal, the video signal being converted into digital image data, and the digital image data having a luminance of a first threshold value or more for each frame image. Brightness determining means for extracting a pixel having a pixel, and frame collating means for extracting a frame image having a pixel appearing at the same position in a series of frame images for the extracted pixel. A video image reproducing device characterized by outputting as an image. 16. A television with a video image reproducing means for reproducing the accumulated video image on a moving image display means, converting a video signal output by the video image reproducing means into digital image data, With respect to the data, a brightness determining unit that extracts a pixel having a brightness equal to or higher than a first threshold value for each frame image, and a frame that extracts a frame image having pixels that appear at the same position in a series of frame images for the extracted pixel A television with a video image reproducing means, which has a collating means and outputs the extracted frame image as the representative image. 17. A moving image search method for extracting a part of a moving image captured as digital image data as a representative image and outputting the representative image on a moving image display means, in which the digital image data is frame image by frame image. Pixels having a luminance equal to or higher than the first threshold are extracted, and the extracted pixels are subjected to frame matching to extract frame images having pixels appearing at the same position in a series of frame images. A moving image search method characterized by outputting as a representative image. 18. A moving image search method for extracting a part of a moving image captured as digital image data as a representative image and outputting the representative image on a moving image display means, wherein each frame image of the digital image data is First, a pixel having a certain range of colors is extracted, and the extracted pixels are subjected to frame matching to extract a frame image having pixels that appear in the same position in a series of frame images. A moving image search method characterized by outputting as a representative image.