CN101887439B - Method and device for generating video abstract and image processing system including device - Google Patents

Abstract

A method for generating a video summary of a video signal is provided, including: a video decoding step, decoding the video signal, so as to obtain a multi-frame video decoding image; a subtitle characteristic obtaining step, obtaining the multi-frame video decoding image contained in At least one of the following characteristics of each subtitle in all subtitles: the duration of the subtitle, the position of the subtitle in the corresponding video decoding image, the character size of the subtitle, the identification code of the subtitle image after optical character recognition (OCR) and Credibility; the video abstract generation step, according to the relationship between the content of the video signal and the characteristics of the subtitles, at least one characteristic of the obtained subtitles is processed in order to generate a content suitable for the content of the video signal Video summary. The method of the invention makes it possible to generate a video digest that accurately reflects the main content of the original video file. A device for generating a video summary of a video signal and an image processing system having the device are also provided.

Description

Method and device for generating video summaries, and image processing system including the device

technical field

The present invention generally relates to the technical field of video image processing, and more specifically, relates to a method and device for generating video summaries, and an image processing system including the device.

Background technique

Video Summary VS (Video Summarization) refers to a short summary formed for a video file, which is used to summarize the main content of the video file. Two different types of video summarization are commonly used today. The first is called "SVS (Static Video Summary, Static Video Summary)", which consists of a series of key frames extracted or synthesized from the original video file. The second is called "DVS (Dynamic Video Skimming), which is a reduced version of the original video file composed of a group of continuous video clips.

In the above-mentioned method for forming a video summary in the prior art, a video shot boundary detection technology (shot boundary detection technology) is used to determine the key frame of the SVS. A video shot is a series of undivided frames captured by a camera. A scene is defined as a collection of one or more adjacent video shots focused on an object or objects of interest. There are several different transitions between video shots, such as cut, fade, dissolve, and wipe. Algorithms in video shot boundary detection techniques include pixel difference, statistical difference, histogram, motion vector, and so on. For example, in this video shot boundary detection technology, the frame set in a certain situation is artificially set in advance as a key frame. The video shots and their related frames involved in the conversion such as , or slip change reflect the important information of the original video file, so these video shots or frames are extracted to form a video summary.

However, in most videos, the above-mentioned SVS based on video shots includes too many video shots, and the video shots involved in the above conversion may not necessarily reflect the main content of the video file. Therefore, although the final video summary includes many frame image, but still cannot provide important information about the original video file. For example, in a story-telling video, many video lens transitions may be included, and these video lens transitions are only to explain the time and place involved in the story development process to ensure the integrity of the story telling or as a foreshadowing for the development of the plot. However, the video shots and their associated frames involved in these conversions have little to do with the main content of the story itself, so if these frames are all included in the video summary, it will be impossible to accurately obtain the outline of the story from the video summary.

The above-mentioned DVS in the prior art also has a similar defect, that is, it is difficult to obtain a video summary that accurately reflects the main content of the original video file.

Contents of the invention

In order to overcome the defects in the above-mentioned prior art, the object of the present invention is to provide a method and device for generating video summaries, and an image processing system including the device, so as to generate video summaries that can accurately reflect the main content of the original video file.

According to an embodiment of the present invention, there is provided a method for generating a video summary of a video signal, comprising the steps of: a video decoding step, for decoding the video signal, so as to obtain a multi-frame video decoding image; a subtitle characteristic obtaining step, using At least one of the following characteristics of each subtitle in all subtitles contained in the multi-frame video decoding image is obtained: the duration of the subtitle, the position of the subtitle in the corresponding video decoding image, the character size of the subtitle, and the subtitle image The identification code and reliability after optical character recognition (OCR); and, a video summary generation step for at least one of the obtained subtitles according to the relationship between the content of the video signal and the characteristics of the subtitles The characteristics are processed in order to generate a video summary adapted to the content of said video signal.

According to an embodiment of the present invention, there is also provided a device for generating a video summary of a video signal, including: a video decoding unit, configured to decode the video signal, so as to obtain a multi-frame video decoded image; a subtitle characteristic obtaining unit, using At least one of the following characteristics of each subtitle in all subtitles contained in the multi-frame video decoding image is obtained: the duration of the subtitle, the position of the subtitle in the corresponding video decoding image, the character size of the subtitle, and the subtitle image The identification code and credibility after optical character recognition (OCR); and a video summary generation unit for at least one characteristic of the obtained subtitles according to the relationship between the content of the video signal and the characteristics of the subtitles Processing is performed to generate a video summary adapted to the content of said video signal.

Other embodiments of the present invention also provide a video image processing system, which has the above-mentioned device for generating a video summary of a video signal according to the present invention. Such a video image processing system is, for example, a TV capture card, a DVD player or a laptop computer.

In addition, other embodiments of the present invention also provide a program product storing machine-readable instruction codes. When the instruction codes are read and executed by a machine, the video signal generation method according to the present invention as described above can be executed. method of video summarization.

As mentioned above, in the prior art methods for obtaining video summaries, the relationship between the subtitle information existing in the video file and the content of the video file is not considered, but the method of the present invention utilizes this relationship to generate The video abstract thus improves the correlation between the obtained video abstract and the content of the original video file, so that the video abstract can accurately reflect the main content of the original video file.

Description of drawings

The above and other objects, features and advantages of the present invention will be more easily understood with reference to the following description of the embodiments of the present invention in conjunction with the accompanying drawings. The components in the figures are not drawn to scale, merely illustrating the principles of the invention. In the drawings, the same or similar technical features or components will be denoted by the same or similar reference numerals.

FIG. 1 is a simplified flowchart illustrating a method for generating a video abstract according to an embodiment of the present invention;

2A is a schematic diagram illustrating an example of an original video file involved in the process of generating a video summary by a method according to an embodiment of the present invention;

FIG. 2B is a schematic diagram showing a video abstract obtained from the raw video file of FIG. 2A by a method according to an embodiment of the present invention;

3A is a schematic diagram illustrating another example of an original video file involved in the process of generating a video summary by a method according to an embodiment of the present invention;

3B is a schematic diagram showing a video abstract obtained from the original video file of FIG. 3A by a method according to an embodiment of the present invention;

4 is a simplified flowchart illustrating an example of a method for obtaining a character size of a subtitle in the process of generating a video summary by a method according to an embodiment of the present invention;

Fig. 5 is a schematic block diagram showing an apparatus for generating a video summary according to an embodiment of the present invention.

Detailed ways

Embodiments of the present invention will be described below with reference to the drawings. Elements and features described in one drawing or one embodiment of the present invention may be combined with elements and features shown in one or more other drawings or embodiments. It should be noted that representation and description of components and processes that are not related to the present invention and known to those of ordinary skill in the art are omitted from the drawings and descriptions for the purpose of clarity.

FIG. 1 is a simplified flowchart illustrating a method for generating a video summary according to an embodiment of the present invention. As shown in the figure, the method 100 for generating a video abstract according to this embodiment of the present invention starts at step S110. In the video decoding step S120, the video signal included in the original video file is decoded so as to obtain multiple frames of video decoded images. In subtitle characteristic obtaining step S130, at least one of the following characteristics of each subtitle in all subtitles contained in the multi-frame video decoding image is obtained: the duration of the subtitle, the position of the subtitle in the corresponding video decoding image, the character of the subtitle Dimensions, ID code and confidence level of the subtitle image after Optical Character Recognition (OCR). In the video summary generation step S140, according to the relationship between the content of the video signal and the characteristics of the subtitle, at least one characteristic of the obtained subtitle is processed, so as to generate the main content of the video signal (ie the content of the original video file) Compatible video summary.

The decoding process of the video information in the video decoding step S120 can be completed by various video decoding methods in the prior art. The video decoding technology is a very mature image processing technology at present, and its details will not be repeated here. As an example, the video information may be decoded by the decoder ffdshow. Information about the decoder ffdshow is available at http://sourceforge.net/projects/ffdshow.

In the subtitle characteristic obtaining step S130, corresponding processing is performed on the subtitles in the multi-frame video decoding images obtained in the video decoding step S120. Generally, the subtitle information is additionally added to the video image captured by the camera. The subtitle information usually reflects the relevant information of the video image frame where it is located, for example, the subject involved in the frame image, the content expressed in the frame image, etc. wait. Subtitles may include, for example, the event time, location, parties, scores of sports games, weather forecasts, prices of commodities, and the like.

Various properties of subtitles are obtained by detecting, tracking and identifying all subtitles contained in multi-frame decoded video images. Wherein, the duration of subtitles can be realized by the technology of determining subtitle appearance time existing in various prior art, for example: in the U.S. patent that the inventor is Rainer Wolfgang, Lienhart, Axel Wernicke, the invention name is "Generalized text localization inimages" No. 6,470,094, and the inventor is Rainer Wolfgang Lienhart, Axel Wernicke, the invention title is "Estimating text color and segmentation of images" U.S. Patent No. 6,473,522 disclosed in the method of using signature algorithm to track the text in the video; the inventor is Lu Lie, SunYan-Feng, Li Mingjing, Hua Xian-Sheng, Zhang Hong-Jiang, U.S. Patent Application Publication No. 2004/0170392 entitled "Automatic detection and segmentation of music videos in an audio/video stream" The position information of the subtitle determines the method of the start and end time of the subtitle in the music video; the inventor is SanghoonSull, Hyeokman Kim, Min Chung, Sangwook Lee, Sangwook Oh, and the invention name is "System and method for indexing, searching, identifying, and editingmultimedia files" U.S. Patent Application Publication No.2007/0038612, which uses similar color and position information as features to track subtitles; in the author Huiping Li et al., titled "Text enhancement in digital video using multiple frame integration", The method disclosed in ACM Multimedia (pp.19-22, 1999) uses an image matching algorithm based on SSD (Sum of Square Difference) to track text regions; the author is Xiaoou Tang, etc., and the name is "A spatial-temporal approach for video caption detection and recognition", IEEE Transactions on Neural The method disclosed in Networks (Vol.13, No.4, pp.961-971, 2002) using QSDD (Quantized Spatial DifferenceDensity) to detect the position of the frame where the subtitle changes; the author is Takeshi Mita et al., and the name is " Improvement of Video Recognition by Character Selection", ICDAR (pp.1089-1093, 2001), and Japanese Patent Application Publication JP Unexamined with the title of "テロツプ information processing device and びテロツプ information display device" in which the inventor is Yushi Mita, etc. 2001-285716 discloses a method for determining the relationship between consecutive frames by using the encoding and credibility of the characters generated by the recognition engine to compare the features of the upper layer; and so on. In addition, the duration of the subtitles can also be determined by the method disclosed in the Chinese patent application with the inventor Bai Hongliang et al., the title of the invention is "Apparatus and Method for Determining Subtitle Existence Time", and the application number is 200810074125.3.

The position of the subtitle in the corresponding video decoding image can also be realized by various methods for extracting the subtitle area in the prior art, for example: the inventors are Ito Kiyomi and Niakura Yasuju, and the title of the invention is "image category determination method, Japanese Patent Publication JP 2006-53802 of Image Type Judgment Device and Image Type Judgment Program", inventors are Shinagawa Shinichi and Matsubayashi Kazuhiro, Japanese Patent Application Publication JP 9-16769 titled "Image Processing Device および Method" , and the author is Rainer Lienhart, etc., named "Localizing and Segmenting Text in Image and Videos", IEEETransactions on Circuits and System for Video Technology (Vol.12, No.4, pp.256-268, 2002) based on Different features such as connected domain features, edge features, etc. to extract subtitle regions; in the author Yu Zhong et al., the name is "Automatic Caption Localization in Compressed Video", IEEE Transaction on Pattern Analysis and Machine Intelligence (Vol.22, No. 4, pp.385-392, 2000) discloses a method for extracting subtitle regions based on texture features; the author is Xiaoou Tang, etc., and the name is "A Spatial-Temporal Approach for Video Caption Detection and Recognition", IEEE Transactions on Neural Network (Vol.13, No.4, pp.961-971, 2002), and the author is Toshio Sato et al., titled "Video OCR for DigitalNews Archive", Workshop on Content-Based Access of Image and VideoDatabases (pp52-60, 1998) disclosed in the method of identifying subtitle regions; and so on. In addition, the position of the subtitle in the corresponding video decoding image can also be determined by the method disclosed in the Chinese patent application with the inventor Bai Hongliang et al., the title of the invention is "Apparatus and Method for Extracting Subtitle Area", and the application number is 200710140327.9.

The method for determining the character size of subtitles can be implemented using various methods in the prior art, for example: in the author's article, Lyu, M.R.; Jiqiang Song; Min Cai, named "Acomprehensive method for multilingual video text detection, localization, and extraction", the method disclosed in IEEE Trans.on Circuits and Systems for videotechonology (15(2), 243-255, 2005), in which Figure 7 is a projection-based method; and, in the authors Xiaoou Tang, Xinbo Gao, Jianzhuang Liu, Hongjiang Zhang, named "A spatial-temporal approach for video caption detection and recognition", the method disclosed in IEEE Transactions on Neural Networks (13(4), 2002, 961-971), where Figure 9 is based on projection method.

As for the identification code and credibility of the subtitle image after optical character recognition (OCR), since OCR is a very mature technology in image recognition processing, the specific details will not be repeated here.

In the video abstract generating step S140, a video abstract suitable for the content of the original video file is generated according to the relationship between the content of the video signal and the characteristics of the subtitle. The so-called "adaptation", that is, "matching", means that the video summary accurately summarizes the main content of the original video file (which is represented as a video signal).

As mentioned above, the subtitle information in the video decoded image usually reflects the relevant information of the video image frame, for example, the theme involved in the frame image, the content expressed in the frame image, and so on. Therefore, if a video digest is generated based on the relationship between subtitle information and a video signal including the subtitle, a video digest that accurately reflects the main content of the video signal can be generated.

As for the relationship between the content of the video signal and the characteristics of the subtitles, it can be acquired in various ways. For example, relevant knowledge about the relationship between the content of the video signal and the characteristics of the subtitles can be obtained by learning a certain number and various types of video images with subtitles in advance. When actually performing video abstract generation processing, Determine the relationship between the content of the video signal and the characteristics of the subtitles according to the knowledge obtained through pre-learning, and then process the various characteristics of the obtained subtitles according to the determined relationship, so as to generate a Adapted video summary. Alternatively, it is also possible to obtain information on the relationship between the content of the video signal and the characteristics of the subtitle in advance. This information is then provided as input information together with the video signal for generation of a video summary, so that a video summary that accurately reflects the important content of the video signal is generated based on this relationship.

The processing of learning a certain number of various types of video images to obtain relevant information is a commonly used method in the field of image processing technology. For example, the following information can be obtained through the learning process: the larger the subtitle size in the decoded image frame, the higher the probability that it represents the main content of the video signal; the subtitle at a specific position in the decoded image frame represents the main content of the video signal more likely; and so on. The above information obtained through learning reflects the relationship between the content of the video signal and the subtitles. Of course, the information about the relationship between the content of the video signal and the characteristics of the subtitles is just an example. Depending on the specific situation, there may be various relationships between the content of the video signal and the characteristics of the subtitles, which will be described in detail later. .

In addition, as we all know, subtitles are additional information after the video file is produced. One of the main purposes of subtitles is also to help viewers understand the main content of the video file. Therefore, the relationship between the content of the video signal and the subtitles is often determined to a certain extent when the subtitles are added. Thus, information about this relationship can also be obtained before the process of generating a video summary takes place. For example, it can be obtained through operator judgment or targeted analysis of video signals with subtitles. The determined information about this relationship is then provided as an input parameter to the process for generating the video summary.

It can be seen that there is always a specific relationship between the content of the video signal and the characteristics of the subtitles, and the video abstract generation method according to the embodiment of the present invention utilizes this relationship, and based on this relationship, each of the obtained subtitles Corresponding processing is performed on at least one of these characteristics, so as to generate a video summary that can accurately reflect the main content of the video signal. Accordingly, the video abstract generation method according to the present invention can be called a "subtitle-based" video abstract extraction method.

As mentioned above, for example, the relationship between the content of the video signal and the subtitle can be obtained by learning in advance and judging in real time, or can be obtained in advance, and then the information about this relationship can be used together with the video signal to generate the video abstract as It is used according to the input parameters of the video summarization method of the present invention. Of course, in addition to the methods listed above for obtaining the relationship between the content of the video signal and the subtitles, any other suitable method may also be used, which will not be described one by one here.

In the video abstract generation step S140, according to the relationship between the content of the video signal and the characteristics of the subtitle, at least one characteristic of the obtained subtitle is processed so as to generate Adapted video summary. According to the above, depending on the content of the video signal, the relationship between the content of the video signal and the characteristics of the subtitles can be of various types. In order to help further understand the method for generating a video abstract according to this embodiment of the present invention, several specific examples will be described in detail below with reference to FIGS. 2A-2B and 3A-3B.

Fig. 2A is a schematic diagram showing an example of an original video file involved in the process of generating a video summary by a method according to an embodiment of the present invention. As shown in FIG. 2A , it is assumed that the content of the video signal to be extracted involves a talk-type or news-type program, such as an interview with a character. In this example, a talk-type program is taken as an example. Fig. 2A shows multi-frame decoded images No. 1, . . . , No. 6, . . . obtained by decoding the video signal. Taking the NO.1 image frame as an example, the subtitle A in the upper left corner represents the topic involved in the talk show, the subtitle B1 in the middle of the left and the subtitle C1 in the middle of the right represent the names of the participants in the talk show, The title, etc., the subtitle D1 below it represents the talk content of the talk show participant B1 or C1, and the time t1 in brackets represents the time point where the NO.1 image frame is located. The other decoded image frames have the same structure as the NO.1 image frame, and the details thereof will not be described one by one.

It is easy to understand that although only 6 frames of decoded images are shown in FIG. 2A , this is an illustration. In fact, depending on the size of the video signal and the decoding method used, the number of frames of the decoded image obtained It can be any number, which is indicated by an ellipsis in the figure. Moreover, although subtitles are included in each frame of image shown in this example, in actual situations, not every frame of image must include subtitles. These situations do not affect the implementation of the method for generating a video summary according to the embodiment of the present invention. For example, for an image frame that does not include subtitles, it may choose not to perform corresponding processing such as extracting subtitles when producing a video summary, or to process it according to a traditional video summary generation method.

From the multi-frame decoded images in Fig. 2A, it can be seen that in the entire duration of the video signal of the relevant talk show, the duration of subtitle A is the longest (it can also include all other decoded images not shown in the figure). or in a part); represent that the subtitle B1 of the first talking participant exists in the decoded image of NO.1-NO.6, and its duration (t6-t1) is less than the duration of subtitle A; represent the subtitle B1 of the second talking participant Subtitle C1 exists between image frames NO.1 to NO.3, and its duration is (t3-t1), and after image frame NO.4, it is replaced by subtitle C2 representing the third conversation participant, and its duration is (t6-t4) is also shorter than the duration of subtitle A; the duration of subtitles D1-D6 representing the content of the conversation is shorter than that of subtitles A, B1, C1, and C2. It can be seen that there is a "cascade" relationship between the characteristics of the video signal and the subtitles, that is, the subtitle A with the longest duration, located in the upper left part of the video decoding image, and representing the theme of the talk show is considered to represent the The most important information of the talk show; subtitles B1, C1, and C2 that are short in duration, located in the left middle or right middle of the decoded image, and represent the talk show participants are considered to represent the secondary important information of the talk show information; and the subtitles D1-D6 with the shortest duration, located in the lower part of the video decoding image, and representing the specific conversation content of each conversation participant are considered to represent less important information of the talk show. Because the topic and participants of the talk show are important information of the talk show, they should be included in the video summary generated for the video signal of the talk show, and the specific content of each participant's talk is not the focus of the video summary , does not have to be included in the video summary. Therefore, subtitle A, subtitles (B1, C1, C2), and subtitles (D1-D6) actually relate to the main content of the "cascade" of the talk show. In addition, if you want to know the specific conversation content of a talking participant, you can directly retrieve the starting and ending time of the talking participant in the video signal according to the video summary, and then watch the relevant original video (details will be described below) .

It can be seen that based on the above-mentioned cascading relationship between the content of the video signal and the characteristics of the subtitle, the duration of the subtitle of the subtitle obtained from the decoded video image, the subtitle in the corresponding video decoded image The position in the subtitle image, the identification code and reliability of the subtitle image after optical character recognition are processed to generate a "cascade" video summary of the video signal of this type of conversation. The process of generating such a cascaded video summary will be described in detail below with reference to FIG. 2B .

By processing the above-mentioned characteristics of the obtained subtitles, the subtitle A that is located in the upper left part of the video decoded image (or the upper right part depending on the specific circumstances) and whose duration is the longest first duration is determined as the first subtitle A. First-level subtitles, the first-level subtitles represent the topics involved in the video signal of the talk show, and are the most important information.

Among all video decoded images that contain first-level subtitles A, the subtitles that are located in the left middle or right middle of the video decoded image and whose duration is a second duration shorter than the first duration, that is, The subtitle B1 with a duration of (t6-t1), the subtitle C1 with a duration of (t3-t1), and the subtitle C2 with a duration of (t6-t4) serve as the second-level subtitles. Moreover, according to the surname, title (such as position) or salutation expressed by the identification code obtained after optical character recognition and reliability of these second-level subtitles, the second-level subtitles can be divided into different subtitles B1 , C1 and C2, these different subtitles are respectively used to identify different talking participants of the talk show during the duration of subtitle A, that is, they can be considered as identifiers of different talking participants.

In all video decoded images containing first-level subtitles A and second-level subtitles B1, C1, and C2, the third duration, which will be located in the lower part of the video decoded image and whose duration is shorter than the second duration The subtitle of time is determined as the third-level subtitle. Wherein, the subtitles D1-D3 in the third-level subtitles coexist with the subtitles B1 and C1 in the second-level subtitles, then the subtitles D1-D3 are determined to correspond to the subtitles B1 and C1 representing the participants of the conversation Conversation. Similarly, subtitles D4-D6 are determined to be the content of the talk corresponding to subtitles B1 and C2 representing the talking participants.

After the above-mentioned processing of the corresponding characteristics of the subtitles, at least one frame of video decoding image of the cascaded subtitles containing the first-level subtitles, the second-level subtitles and the third-level subtitles can be selected as the level of the talk show video signal syndicated video summaries. FIG. 2B is a schematic diagram of such a cascaded video summary. As shown in Figure 2B, the selection includes the decoded video image frames NO.1, NO.3, NO.4 and NO.6 representing different talking participants, which constitute the relevant subtitles B1, C1 and C2 in the video summary part of the participants. Although in Fig. 2B, the decoded image frames NO.1, NO.3, NO.4 and NO.6 corresponding to the start and end time points of joining and exiting the talk show of the talking participants represented by subtitles C1 and C2 have been chosen respectively, But in fact, since the specific conversation content of different talkers is not the focus of generating the video summary, the video summary may only include the decoded image frames corresponding to one of the time points when different talking participants join and exit the talk show . In this case, if one wants to know the specific conversation content of a certain conversation participant, such as the conversation participant represented by subtitle C1, the decoded image frame related to it can be found from the video summary (in Fig. 2B Be image frame NO.1 or NO.3), obtain its joining and/or exit the time point (t1 and/or t3) of talk show from this image frame, just can utilize this time point as index to original talk show The specific content of the conversation with the conversation participant represented by the subtitle C1 is found in the video signal for viewing or other processing. Of course, in addition to selecting decoded image frames corresponding to the start and end time points of joining and quitting of different talking participants, more decoding image frames related to corresponding talking participants may also be selected. This selection can be arbitrary, or based on a certain standard, and can be set according to actual needs, and the specific details will not be repeated here.

Optionally, any frame of image (image frame NO.1 at the position of the root node in the dendrogram in FIG. 2B ) may be selected from the video summary to represent the talk show. The frame image can be used as an index of the video summary, for example, it is used when searching for a corresponding video summary when multiple video summaries are generated.

As can be seen from the above description, the "cascaded" video summary formed according to this example of the present invention shows the cascading of the importance levels between the subtitles contained in the decoded image frame, that is, subtitle A, subtitle ( B1, C1, C2), and subtitles (D1-D6) are arranged in descending order of importance in terms of reflecting the main content of the video signal, which constitute a subtitle that can be called "cascaded". Of course, depending on the specific video signal and subtitles, the importance of such "cascaded" subtitles in reflecting the main content of the video signal can also be arranged in ascending order. It is easy to understand that although only three different conversation participants are involved in FIGS. 2A-2B , there can be any number of conversation participants. If there is only one talking participant (such as a news broadcast), at least one decoded image frame can be arbitrarily selected to form a video summary. This selection may also be based on certain criteria, for example, a frame of images may be selected at regular intervals to form a video summary. This setting can be performed according to actual needs, and the specific details will not be repeated here. If there are more than three talking participants, the cascaded video summary as shown in FIG. 2B may have more branches representing different talking participants (shown as child nodes in the tree diagram in FIG. 2B ).

In addition, although the concatenated subtitles shown in the specific examples shown in FIGS. Concatenated subtitles with more or fewer levels are formed in the video summary. That is, there may be N levels of "cascaded" subtitles, N being a positive integer. For example, if it is necessary to distinguish the utterances of different speakers, the information of other characteristics such as the color and size of the subtitles can be used for this distinction. The subtitles determined by using characteristics such as color and size and used to distinguish the conversation contents of different conversation participants can be regarded as the fourth-level subtitles in the cascaded subtitles. For example, in the video summary as shown in Figure 2B, other decoded image frames can be branched under the decoded image frame NO. decoded images of the speaker's speech.

Another specific example of generating a video digest of a video signal by processing other characteristics of subtitle characteristics is described below with reference to FIGS. 3A-3B .

Fig. 3A is a schematic diagram showing another example of an original video signal involved in the process of generating a video summary by a method according to an embodiment of the present invention. In some video signals, there is such a relationship between the subtitles included in the video image and the content of the video signal that the subtitles are located at a fixed position in the decoded video image and relate to the main content of the video signal. In this case, subtitles located at fixed positions are compared, a subtitle at a time point at which an image of the subtitle changes is determined, and a video digest of a video signal is composed from a video decoded image including the determined subtitle.

For example, it is assumed that the content of the video signal for which the video abstract is to be generated relates to a game. Fig. 3A shows multi-frame decoded video images No. 1, . . . , No. 6, . . . obtained by decoding the video signal. It is easy to understand that although only 6 frames of decoded images are shown in the figure, this is an illustration. In fact, depending on the size of the video signal and the decoding method used, the number of frames of the decoded images obtained can vary. is any number. Taking image frame No. 1 as an example, the subtitle S1 located at a fixed position, that is, the lower right subtitle S1 is a score board displaying the score information of the game. The score board usually includes information such as the type of game, the participants, and the specific score. The other decoded image frames have the same structure as the NO.1 image frame, and the details thereof will not be described one by one. In the decoded image frames shown in Figure 2A, the scores involved in the decoded image frames NO.1-NO.2 are related to subtitle S1, the scores involved in the decoded image frames No.3-NO.5 are related to subtitle S2, and the decoded image The score involved in frame No. 6 is related to subtitle S3.

In a game, the appearance of the score board and its position in each decoded image frame generally remain unchanged, while the information on specific score changes reflects the main content of the game-type video signal. Therefore, in the process of generating video summaries, all scoreboard subtitles located at a certain fixed position are extracted and compared. Determine the score board subtitle S1 at the point in time when the image of the score board subtitle changes, S2, S3, and by the video decoding image frame NO.1 (or NO.2), NO.3 ( Or NO.5), NO.6 to form the video summary of the video signal, as shown in Figure 3B.

It is easy to understand that although the subtitle of the scoreboard in Fig. 3A-3B is located at the lower right corner of each frame of the decoded image, the scoreboard may also be located at any other suitable position according to the actual situation. In addition, although in this example, the decoded image in the case where the score changes is found by extracting the subtitle at a fixed position and comparing the images of the subtitle to generate a video summary, the extracted score can also be The card subtitles are processed by OCR, and the score card subtitles with score changes are determined by comparing the OCR results to generate video summaries.

Figures 2A-2B and Figures 3A-3B above describe two specific examples of generating video summaries of video information by processing various characteristics of subtitles based on the relationship between the content of the video signal and the characteristics of the subtitles. Of course, which of the various characteristics of the subtitles to process can also be determined according to other relationships between the characteristics of the subtitles and the content of the video signal for which the video abstract is to be generated. Below are some specific examples.

In a specific example, there is a relationship between the content of the video signal and the characteristics of the subtitles: subtitles with a larger character size relate to more important content of the video signal. In many video images, it is common to use the size of subtitles to represent the importance of the content represented by the subtitles. In this case, the average character size of a part of subtitles (for example, %K) with the longest duration among all subtitles can be obtained. The subtitles whose character size is larger than the average character size among all the subtitles are determined, and the video digest of the video signal is composed from video decoded images containing the determined subtitles.

In another specific example, there is a relationship between the content of the video signal and the characteristics of the subtitles: a subtitle with a larger character size and a longer subtitle duration relates to more important content of the video signal. In this case, the size threshold H of the character size of the subtitle and the time threshold T of the duration of the subtitle can be preset, the subtitle whose character size is larger than H and the duration is larger than T is determined, and the video containing the determined subtitle The pictures are decoded to form a video digest of the video signal.

In yet another specific example, there is such a relationship between the content of the video signal and the characteristics of the subtitles: the subtitles have a longer duration and are located at a specific position in the video decoded image and relate to more important content of the video signal . In this case, the time threshold T of the duration of subtitles is set in advance, the subtitles whose duration is longer than T and located at a specific position of the video decoded image are determined, and are composed of the video decoded image containing the determined subtitle A video digest of the video signal.

In yet another specific example, there is a relationship between the content of the video signal and the characteristics of the subtitles: subtitles with larger character sizes and located at specific positions in the video decoded image relate to more important content of the video signal. In this case, the size threshold H of the character size of the subtitle is set in advance, the subtitle whose character size is larger than H and located at the specific position of the video decoded image is determined, and is composed of the video decoded image containing the determined subtitle A video digest of the video signal.

In another specific example, there is a relationship between the content of the video signal and the characteristics of the subtitles: the subtitles located at a specific position in the video decoded image and whose subtitles have a higher reliability after optical character recognition relate to the video signal more important content. In this case, the threshold L of the reliability of subtitles after optical character recognition can be preset, and the subtitles located at the specific position in the video decoding image and whose reliability after optical character recognition of subtitles is greater than L can be determined. , and form a video summary of the video signal from video decoded images containing the determined subtitles.

In another specific example, there is a relationship between the content of the video signal and the characteristics of the subtitles: subtitles with larger character sizes and higher reliability of subtitles after optical character recognition relate to more important parts of the video signal content. In this case, pre-set the size threshold H of the character size of the subtitle, and set the threshold L of the credibility of the subtitle after OCR, and determine the credibility of the character size greater than H and OCR subtitles greater than L, and a video digest of the video signal is composed of video decoded pictures containing the determined subtitles.

Although some specific examples of generating video summaries by processing the characteristics of subtitles have been listed above, those skilled in the art understand that since the relationship between video signals and subtitles is varied in actual situations, it is also possible to Corresponding to this relationship, various appropriate methods are adopted to handle the characteristics of subtitles. This can be set correspondingly by those skilled in the art according to specific situations, which will not be repeated here. For example, it is possible to determine which subtitles to select according to the specific content contained in the subtitles, and then determine which image frames to select to generate a video summary by judging the results of OCR for all subtitles.

It is easy to understand that since the subtitles are closely related to the content of the video signal, the video summary can be generated by processing the characteristics of all the subtitles included in the video decoded image, which can accurately reflect the main content of the video signal. In a sense, the subtitle-based video summary generation method according to the present invention can also be called a "content-based" video summary generation method, because it is based on the content of the video signal to some extent. The relationship between the content of video information and the characteristics of subtitles. Of course, it is not necessary to judge and process the content of the decoded video image itself.

The character size of the subtitle is included in the above-mentioned characteristics of the subtitle. As mentioned above, the size of subtitle characters can be obtained through various existing methods. As a supplement and improvement to the prior art, a preferred method for obtaining the size of subtitle characters is introduced below with reference to FIG. 4 .

FIG. 4 is a simplified flowchart illustrating an example of a method 400 for obtaining the size of subtitle characters in the process of generating a video summary by a method according to an embodiment of the present invention. In this example, the subtitle character size is the height of the subtitle character. As shown in Figure 4, in step S405, the subtitle image is processed to produce the binarized image of the subtitle and its image features, wherein the image height is H, and the width is W. For example, the value of the stroke pixel can be 255, which is not Stroke pixels have a value of 0. Various known image processing methods can be used to obtain subtitle image features. For example, the method disclosed in Canny, J., A, titled "Computational Approach To Edge Detection", IEEE Trans. Pattern Analysis and Machine Intelligence (8: 679-714, 1986) can be used to perform such subtitle image characterization Extract processing.

In step S410, it is determined whether the height H of the subtitle image is less than or equal to twice the predefined character height MIN_CS. If the determination result is "Yes", it indicates that the subtitle is a single line. The image height H is determined as the character height of the subtitle in step S4150. Otherwise, go to step S420.

In step S420, the subtitle image is projected. Specifically, carry out horizontal projection for the image of the horizontal subtitle, accumulate the stroke pixel value of the subtitle image in the horizontal direction, and the accumulated value is stored in Hist(i), wherein i is an index value, representing a segmentation point, and i=1, 2,....H; or, for the vertical projection of the image of the vertical subtitle, the stroke pixel value of the subtitle image is accumulated in the vertical direction, and the accumulated value is stored in Hist(i), where i is the index value, indicating the segmentation point , and i=1, 2, ... H. The processing of projecting images can be realized by various known methods, for example: in the author is Lyu, M.R.; Jiqiang Song; Min Cai, titled "A comprehensive method formullingual video text detection, localization, and extraction", IEEE Trans .on Circuits and Systems for video technology (15(2), 243-255, 2005), the method disclosed in the literature, wherein Figure 7 is a method based on projection; and, in the authors Xiaoou Tang, Xinbo Gao, Jianzhuang Liu, Hongjiang Zhang, named "A spatial-temporal approach for video caption detection and recognition", the method disclosed in the literature of IEEE Transactions on Neural Networks (13(4), 2002, 961-971), where Figure 9 is a projection-based method.

The flow then goes to step S425, where it is determined whether subtitles can be divided into multiple lines. If Hist(i)<=T, it means that the subtitle is separable at position i, where i is a positive integer. Record the position coordinate values i1, i2, ... ij of all possible separable positions, and calculate the height (i2-i1), (i3-i2), ... , the average value of (ij-i(j-1)) is used as the character height of the subtitle. Among them, j is a positive integer less than or equal to NC, NC is the number of segmentation points, T is a predetermined constant, the value of i and j depends on whether the subtitle can be divided, and the value of T can be properly set in advance according to the actual situation . On the other hand, when the determined result of step S425 is Hist(i)>T, it means that the subtitle is inseparable, that is, there is a case of character conglutination, and the flow then goes to step S435.

In step S435, a connected component analysis (CCA) analysis is performed on the image features of the subtitles. CCA analysis is a common method for extracting cohesive characters, and its detailed description is omitted here. Several subtitle character components (CC) are obtained after CCA analysis, and the average height of subtitle character components (CC) is estimated to be H1.

In step S440, the CC of the subtitle is aligned, and a new subtitle character component CC_New is obtained after alignment, and the average height of the new CC_New is estimated to be H2. The specific implementation of this alignment operation is as follows: firstly, the rectangle center of each CC is obtained, for example, represented by the coordinate value yn of the rectangle center. For the rectangle centers y1, y2 of any two CCs (y1, y2 are the coordinates of the center of the rectangle), if |y1-y2|<M (M=3), then the two rectangles are considered to be roughly on a straight line, that is aligned. Wherein, M is a preset threshold, and other values different from 3 can also be taken according to the actual situation. This alignment operation is to remove the noise that may exist in the character component CC, so as to avoid negative impact on the calculation of subtitle character height.

In step S445, it is determined whether the area of a certain character component in the new character component CC_new obtained after the alignment operation is greater than (α*H*W) or whether the number of the new character component CC_new is equal to 1. If any one of the above-mentioned conditions is true, the process goes to step S455, and the character height of the subtitle is determined as H1; otherwise, in step S450, the character height of the subtitle is determined as H2. Wherein, α is a predetermined constant, which can be specifically set according to the actual situation. After the alignment operation in step S440, the characters may be mixed together to a higher degree or even mixed together due to the unsatisfactory processing of removing noise, which shows that the number of CC_new obtained is called one (all character components stick together together), or the area of a certain CC_new is too large (some character components are glued together). The processing in step S445 is for judging the occurrence of this sticking situation. If conglutination occurs, the average value H1 calculated in step S435 is used as the height of subtitle characters; if conglutination does not occur, the average value H2 calculated in step S440 after noise removal can be used as the height of subtitle characters.

In the above method, the coordinate system used to calculate the position coordinates i1, i2, ... ij of possible separable positions and the rectangular center coordinates y1, y2 of any two character components (CC) can be any suitable The plane coordinate system of , for example, the Cartesian rectangular coordinate system with the plane where the subtitle image is located as the coordinate plane, and so on.

In addition, as a preferred way of determining the subtitle character size as shown in Figure 4, if the inventor is Bai Hongliang, etc., the invention name is "Apparatus and Method for Determining Subtitle Existence Time", and the application number is 200810074125.3. The stable global feature obtained in the scheme disclosed in the application is used as the subtitle image feature to calculate the character height of the subtitle, so that the subtitle can be tracked robustly and stably, and the tracking area can be continuously optimized.

As an alternative, the method for generating a video summary according to the above-mentioned embodiments of the present invention may further include a video summary outputting step for outputting the video summary generated in the video summary generating step. For example, the generated video summary can be output to a predetermined storage device.

According to other embodiments of the present invention, an apparatus for generating a video summary of a video signal is also provided. Fig. 5 is a schematic block diagram showing an apparatus 500 for realizing the method for generating a video abstract according to the embodiment of the present invention. As shown in the figure, the device 500 includes: a video decoding unit 510, which is used to decode the video signal, so as to obtain a multi-frame video decoding image; a subtitle characteristic obtaining unit 520, which is used to obtain all the subtitles contained in the multi-frame video decoding image At least one of the following characteristics of each subtitle: the duration of the subtitle, the position of the subtitle in the corresponding video decoded image, the character size of the subtitle, the identification code and reliability of the subtitle image after optical character recognition (OCR) and, the video abstract generation unit 530 is used to process at least one characteristic of the obtained subtitles according to the relationship between the content of the video signal and the characteristics of the subtitles, so as to generate a video summary adapted to the content of the video signal .

Those skilled in the art understand that the video decoding unit 510 included in the apparatus 500 shown in FIG. 5 , the subtitle characteristic obtaining unit 520 and the video summary generating unit 530 can be configured to perform 3B describes the video abstract generation method, and although not shown in the drawings, the video abstract extraction method has been fully described in the various specific examples above.

Each constituent unit in the above device can be configured by means of software, hardware or a combination thereof. Specific means or manners that can be used for configuration are well known to those skilled in the art, and will not be repeated here.

Other embodiments of the present invention also propose a video image processing system, which is equipped with the device according to the embodiment of the present invention shown in FIG. generate method.

Such a video image processing system may be, for example, a TV capture card, a DVD player, or a laptop computer, and the like.

In addition, the method for generating a video abstract according to the above-mentioned embodiments of the present invention may be implemented by a program product storing machine-readable instruction codes. When these instruction codes are read and executed by a machine such as a computer, various operation procedures and steps of the method for generating a video summary according to the above-mentioned embodiments of the present invention can be executed. The program product may have any form of expression, for example, an object program, a program executed by an interpreter, or a script program provided to an operating system.

Correspondingly, a storage medium for carrying the program product storing the above-mentioned machine-readable instruction codes is also included in the disclosure of the present invention. Such storage media include, but are not limited to, floppy disks, optical disks, magneto-optical disks, memory cards, memory sticks, and the like.

In the above description of specific embodiments of the present invention, features described and/or illustrated for one embodiment can be used in the same or similar manner in one or more other embodiments, and features in other embodiments Combination or replacement of features in other embodiments.

It should be emphasized that the term "comprising/comprising" when used herein refers to the presence of a feature, element, step or component, but does not exclude the presence or addition of one or more other features, elements, steps or components.

In addition, the method of the present invention is not limited to being executed in the chronological order described in the specification, and may also be executed in other chronological order, in parallel or independently. Therefore, the execution order of the methods described in this specification does not limit the technical scope of the present invention.

Although the present invention has been disclosed by the description of specific embodiments of the present invention above, it should be understood that those skilled in the art can design various modifications, improvements or equivalents of the present invention within the spirit and scope of the appended claims things. These modifications, improvements or equivalents should also be considered to be included in the protection scope of the present invention.

Claims (38)

1. A method for generating a video summary of a video signal, comprising the steps of: The video decoding step is used to decode the video signal so as to obtain a multi-frame video decoding image; The subtitle characteristic obtaining step is used to obtain at least one of the following characteristics of each subtitle in all the subtitles contained in the multi-frame video decoded image: the duration of the subtitle, the position of the subtitle in the corresponding video decoded image, the subtitle The character size of the subtitle image, the identification code and reliability after OCR of the subtitle image; and A video summary generation step, for processing at least one characteristic of the obtained subtitles according to the relationship between the content of the video signal and the characteristics of the subtitles, so as to generate a video summary suitable for the content of the video signal , Wherein, the relationship between the content of the video signal and the characteristics of the subtitles indicates that: the decoded video image includes N levels of cascaded subtitles, wherein the first level subtitles to the Nth level subtitles reflect the main content of the video signal The aspects of are in descending or ascending order of importance, where N is a positive integer; and The video abstract generation step generates a concatenated video abstract of the video signal, the cascaded video abstract includes: at least one of the concatenated subtitles including the first-level subtitles to the Nth-level subtitles Frame video decoded image. 2. method as claimed in claim 1, wherein, described video signal is the video signal of news type or talk type, N=3, the subtitle of the first level to the subtitle of the third level embodies the main content of the video signal Aspects are listed in descending order of importance; and The video summary generation step generates the video signal by processing the obtained subtitle duration, the position of the subtitle in the corresponding video decoded image, the identification code and reliability of the subtitle image after OCR The cascading video summary, wherein the process of generating a cascading video summary includes: Determining the subtitles located in the upper left or upper right of the video decoded image and having the longest duration as the first-level subtitles, the first-level subtitles represent the information contained in the video signal of the news type or talk type. the subject matter covered; In all video decoded images containing the first-level subtitles, it will be located in the left middle or right middle of the video decoded image, and the duration is a second duration shorter than the first duration as the second-level subtitles, and according to the surname, title or salutation expressed by the identification code and reliability of the image of the second-level subtitles after optical character recognition, the second-level subtitles are divided into different subtitles, said different subtitles respectively representing identifiers of different participants of said news or conversation during the duration of said first level subtitles; In all video decoded images containing the first-level subtitles and second-level subtitles, the subtitles will be located in the lower part of the video decoded images, and the duration is a third duration shorter than the second duration The subtitles are determined as third-level subtitles, wherein the part of the third-level subtitles that coexists with different subtitles in the second-level subtitles is determined as the content of different participants represented by the different subtitles the content of the conversation; and Selecting at least one frame of decoded video image of concatenated subtitles including first-level subtitles, second-level subtitles and third-level subtitles as the concatenated video summary of the video signal. 3. The method as claimed in claim 1 or 2, further comprising a video summary output step for outputting the video summary generated in the video summary generation step. 4. A method for generating a video summary of a video signal, comprising the steps of: The video decoding step is used to decode the video signal so as to obtain a multi-frame video decoding image; The subtitle characteristic obtaining step is used to obtain at least one of the following characteristics of each subtitle in all the subtitles contained in the multi-frame video decoded image: the duration of the subtitle, the position of the subtitle in the corresponding video decoded image, the subtitle The character size of the subtitle image, the identification code and reliability after OCR of the subtitle image; and A video summary generation step, for processing at least one characteristic of the obtained subtitles according to the relationship between the content of the video signal and the characteristics of the subtitles, so as to generate a video summary suitable for the content of the video signal , Wherein, the relationship between the content of the video signal and the characteristics of the subtitle indicates that: the subtitle is located at a fixed position in the decoded video image and relates to the main content of the video signal, and The video digest generating step includes: comparing the subtitles located at the fixed positions, determining the subtitles at the point in time when the image of the subtitles changes, and composing the video signal from a video decoded image including the determined subtitles video summary of . 5. method as claimed in claim 4, wherein, shown video signal is the video signal of match type, and the subtitle that described fixed position place is in described video decoded image, the scoreboard of the score information of display match, as well as The step of generating the video summary includes: comparing all scoreboard subtitles, determining the scoreboard subtitles at the point in time when the image of the scoreboard subtitles changes, and forming the whole scoreboard subtitles from the video decoding images that contain the determined scoreboard subtitles A video summary of the described video signal. 6. The method according to claim 4 or 5, further comprising a video abstract output step for outputting the video abstract generated in the video abstract generating step. 7. A method for generating a video summary of a video signal, comprising the steps of: The video decoding step is used to decode the video signal so as to obtain a multi-frame video decoding image; The subtitle characteristic obtaining step is used to obtain at least one of the following characteristics of each subtitle in all the subtitles contained in the multi-frame video decoded image: the duration of the subtitle, the position of the subtitle in the corresponding video decoded image, the subtitle The character size of the subtitle image, the identification code and reliability after OCR of the subtitle image; and A video summary generation step, for processing at least one characteristic of the obtained subtitles according to the relationship between the content of the video signal and the characteristics of the subtitles, so as to generate a video summary suitable for the content of the video signal , Wherein, the relationship between the content of the video signal and the characteristics of the subtitles indicates that subtitles with a larger character size and a longer subtitle duration relate to more important content of the video signal, and The video abstract generation steps include: Setting the size threshold H of the character size of the subtitle and the time threshold T of the duration of the subtitle; and A subtitle with a character size greater than H and a duration greater than T is determined, and a video digest of the video signal is composed of video decoded images containing the determined subtitle. 8. The method as claimed in claim 7, wherein the subtitle characteristic obtaining step obtains the character height of the subtitle by the following processing, as the character size of the subtitle: processing the image of the subtitle to obtain a binarized image of the image of the subtitle; If the height H of the image of the subtitle is less than or equal to 2 times of the predefined character height MIN_CS, then the image height H is determined as the character height of the subtitle, otherwise enter the next step; For the image of the horizontal subtitle, the horizontal projection is performed, and the stroke pixel value of the subtitle image is accumulated in the horizontal direction, and the accumulated value is stored in Hist(i), i=1,2,...,H, or, for the image of the vertical subtitle, the Vertical projection, accumulating the stroke pixel values of the subtitle image in the vertical direction, and the accumulated value is stored in Hist(i), i=1,2,....H, where i is an index value as a positive integer, indicating the segmentation point ; If Hist(i)<=T, it means that the subtitle is separable at the position of the split point i, record the position coordinate values i1, i2,...ij of all possible separable positions, and set (i2-i1) ,(i3-i2),...,(ij-i(j-1)) is the average value of the character height of the subtitle, and when Hist(i)>T, it means that the subtitle is inseparable and enters the next step, Wherein, j is a positive integer less than or equal to H, H is the number of segmentation points, and T is a predetermined constant; Connected domain CCA analysis is performed on the image features of subtitles, and the average value of CC height of character components of subtitles is estimated to be H1; Perform an alignment operation on the character component CC of the subtitle, and obtain a new character component CC_New after alignment, and estimate the average height of the new character component CC_New to be H2; If the area of a certain character component in the new character component CC_new is greater than (α*H*W) or the number of the new character component CC_new is equal to 1, then the character height of the subtitle is determined as H1, otherwise the character height of the subtitle is determined is H2, where α is a predetermined constant, and W is the width of the subtitle image. 9. The method according to claim 7 or 8, further comprising a video abstract output step for outputting the video abstract generated in the video abstract generation step. 10. A method for generating a video summary of a video signal, comprising the steps of: The video decoding step is used to decode the video signal so as to obtain a multi-frame video decoding image; The subtitle characteristic obtaining step is used to obtain at least one of the following characteristics of each subtitle in all the subtitles contained in the multi-frame video decoded image: the duration of the subtitle, the position of the subtitle in the corresponding video decoded image, the subtitle The character size of the subtitle image, the identification code and reliability after OCR of the subtitle image; and A video summary generation step, for processing at least one characteristic of the obtained subtitles according to the relationship between the content of the video signal and the characteristics of the subtitles, so as to generate a video summary suitable for the content of the video signal , Wherein, the relationship between the content of the video signal and the characteristics of the subtitle indicates that the subtitle has a longer duration and the subtitle located at a specific position in the video decoded image involves more important content of the video signal, and The video abstract generation steps include: setting a time threshold T for the duration of subtitles; and A subtitle whose duration is greater than T and located at a specific position of the video decoded picture is determined, and a video summary of the video signal is composed from the video decoded picture containing the determined subtitle. 11. The method according to claim 10, further comprising a video summary output step for outputting the video summary generated in the video summary generation step. 12. A method for generating a video summary of a video signal, comprising the steps of: The video decoding step is used to decode the video signal so as to obtain a multi-frame video decoding image; The subtitle characteristic obtaining step is used to obtain at least one of the following characteristics of each subtitle in all the subtitles contained in the multi-frame video decoded image: the duration of the subtitle, the position of the subtitle in the corresponding video decoded image, the subtitle The character size of the subtitle image, the identification code and reliability after OCR of the subtitle image; and A video summary generation step, for processing at least one characteristic of the obtained subtitles according to the relationship between the content of the video signal and the characteristics of the subtitles, so as to generate a video summary suitable for the content of the video signal , Wherein, the relationship between the content of the video signal and the characteristics of the subtitles indicates that the subtitles with larger character size and located at a specific position in the video decoded image relate to more important content of the video signal, and The video abstract generation steps include: setting the size threshold H of the character size of the subtitle; and A subtitle whose character size is greater than H and located at the specific position of the video decoded image is determined, and a video digest of the video signal is composed from the video decoded image including the determined subtitle. 13. The method as claimed in claim 12, wherein the subtitle characteristic obtaining step obtains the character height of the subtitle by the following processing, as the character size of the subtitle: processing the image of the subtitle to obtain a binarized image of the image of the subtitle; If the height H of the image of the subtitle is less than or equal to 2 times of the predefined character height MIN_CS, then the image height H is determined as the character height of the subtitle, otherwise enter the next step; For the image of the horizontal subtitle, the horizontal projection is performed, and the stroke pixel value of the subtitle image is accumulated in the horizontal direction, and the accumulated value is stored in Hist(i), i=1,2,...,H, or, for the image of the vertical subtitle, the Vertical projection, accumulating the stroke pixel values of the subtitle image in the vertical direction, and the accumulated value is stored in Hist(i), i=1,2,....H, where i is an index value as a positive integer, indicating the segmentation point ; If Hist(i)<=T, it means that the subtitle is separable at the position of the split point i, record the position coordinate values i1, i2,...ij of all possible separable positions, and set (i2-i1) ,(i3-i2),...,(ij-i(j-1)) is the average value of the character height of the subtitle, and when Hist(i)>T, it means that the subtitle is inseparable and enters the next step, Wherein, j is a positive integer less than or equal to H, H is the number of segmentation points, and T is a predetermined constant; Connected domain CCA analysis is performed on the image features of subtitles, and the average value of CC height of character components of subtitles is estimated to be H1; Perform an alignment operation on the character component CC of the subtitle, and obtain a new character component CC_New after alignment, and estimate the average height of the new character component CC_New to be H2; If the area of a certain character component in the new character component CC_new is greater than (α*H*W) or the number of the new character component CC_new is equal to 1, then the character height of the subtitle is determined as H1, otherwise the character height of the subtitle is determined is H2, where α is a predetermined constant, and W is the width of the subtitle image. 14. The method as claimed in claim 12 or 13, further comprising a video summary output step for outputting the video summary generated in the video summary generation step. 15. A method for generating a video summary of a video signal, comprising the steps of: The video decoding step is used to decode the video signal so as to obtain a multi-frame video decoding image; The subtitle characteristic obtaining step is used to obtain at least one of the following characteristics of each subtitle in all the subtitles contained in the multi-frame video decoded image: the duration of the subtitle, the position of the subtitle in the corresponding video decoded image, the subtitle The character size of the subtitle image, the identification code and reliability after OCR of the subtitle image; and A video summary generation step, for processing at least one characteristic of the obtained subtitles according to the relationship between the content of the video signal and the characteristics of the subtitles, so as to generate a video summary suitable for the content of the video signal , Wherein, the relationship between the content of the video signal and the characteristics of the subtitle indicates that the subtitle located at a specific position in the video decoded image and whose subtitle image has a higher reliability after optical character recognition is more important in relation to the video signal the content of , and The video abstract generation steps include: Setting the threshold L of the reliability of the subtitle image after optical character recognition; and Determining the subtitles located at the specific position in the decoded video image and having a reliability greater than L after optical character recognition of the subtitle image, and composing a video abstract of the video signal from the decoded video image containing the determined subtitles. 16. The method according to claim 15, further comprising a video summary output step for outputting the video summary generated in the video summary generation step. 17. A method for generating a video summary of a video signal, comprising the steps of: The video decoding step is used to decode the video signal so as to obtain a multi-frame video decoding image; The subtitle characteristic obtaining step is used to obtain at least one of the following characteristics of each subtitle in all the subtitles contained in the multi-frame video decoded image: the duration of the subtitle, the position of the subtitle in the corresponding video decoded image, the subtitle The character size of the subtitle image, the identification code and reliability after OCR of the subtitle image; and A video summary generation step, for processing at least one characteristic of the obtained subtitles according to the relationship between the content of the video signal and the characteristics of the subtitles, so as to generate a video summary suitable for the content of the video signal , Wherein, the relationship between the content of the video signal and the characteristics of the subtitle indicates that the subtitle with a larger character size and a higher reliability of the subtitle image after optical character recognition involves more important content of the video signal, and , The video abstract generation steps include: Setting the size threshold H of the character size of the subtitle, and setting the threshold L of the reliability of the subtitle image after optical character recognition; and Determine the subtitles whose character size is greater than H and whose reliability after optical character recognition is greater than L, and form the video summary of the video signal from the video decoding images containing the determined subtitles. 18. The method as claimed in claim 17, wherein the subtitle characteristic obtaining step obtains the character height of the subtitle by the following processing, as the character size of the subtitle: processing the image of the subtitle to obtain a binarized image of the image of the subtitle; If the height H of the image of the subtitle is less than or equal to 2 times of the predefined character height MIN_CS, then the image height H is determined as the character height of the subtitle, otherwise enter the next step; For the image of the horizontal subtitle, the horizontal projection is performed, and the stroke pixel value of the subtitle image is accumulated in the horizontal direction, and the accumulated value is stored in Hist(i), i=1,2,...,H, or, for the image of the vertical subtitle, the Vertical projection, accumulating the stroke pixel values of the subtitle image in the vertical direction, and the accumulated value is stored in Hist(i), i=1,2,....H, where i is an index value as a positive integer, indicating the segmentation point ; If Hist(i)<=T, it means that the subtitle is separable at the position of the split point i, record the position coordinate values i1, i2,...ij of all possible separable positions, and set (i2-i1) ,(i3-i2),...,(ij-i(j-1)) is the average value of the character height of the subtitle, and when Hist(i)>T, it means that the subtitle is inseparable and enters the next step, Wherein, j is a positive integer less than or equal to H, H is the number of segmentation points, and T is a predetermined constant; Connected domain CCA analysis is performed on the image features of subtitles, and the average value of CC height of character components of subtitles is estimated to be H1; Perform an alignment operation on the character component CC of the subtitle, and obtain a new character component CC_New after alignment, and estimate the average height of the new character component CC_New to be H2; If the area of a certain character component in the new character component CC_new is greater than (α*H*W) or the number of the new character component CC_new is equal to 1, then the character height of the subtitle is determined as H1, otherwise the character height of the subtitle is determined is H2, where α is a predetermined constant, and W is the width of the subtitle image. 19. The method as claimed in claim 17 or 18, further comprising a video summary output step for outputting the video summary generated in the video summary generation step. 20. An apparatus for generating a video summary of a video signal, comprising: A video decoding unit is used to decode video signals so as to obtain multi-frame video decoding images; A subtitle characteristic obtaining unit, configured to obtain at least one of the following characteristics of each subtitle in all the subtitles contained in the multi-frame video decoded image: the duration of the subtitle, the position of the subtitle in the corresponding video decoded image, the subtitle The character size of the subtitle image, the identification code and reliability of the subtitle image after OCR; and A video summary generation unit, configured to process at least one characteristic of the obtained subtitles according to the relationship between the content of the video signal and the characteristics of the subtitles, so as to generate a video summary suitable for the content of the video signal , Wherein, the relationship between the content of the video signal and the characteristics of the subtitles indicates that: the decoded video image includes N levels of cascaded subtitles, wherein the first level subtitles to the Nth level subtitles reflect the main content of the video signal The importance of aspects of is arranged in descending or ascending order, where N is a positive integer; and The video summary generating unit generates a concatenated video summary of the video signal, and the concatenated video summary includes: at least one of the concatenated subtitles including the first-level subtitles to the Nth-level subtitles Frame video decoded image. 21. The device according to claim 20, wherein the video signal is a video signal of a news type or a talk type, N=3, and the first-level subtitles to the third-level subtitles embody the main content of the video signal Aspects are listed in descending order of importance; and The video summary generation unit generates the video signal by processing the obtained subtitle duration, the position of the subtitle in the corresponding video decoded image, the identification code and reliability of the subtitle image after OCR The cascading video summary, wherein the video summary generation unit is configured to: Determining the subtitles located in the upper left or upper right of the video decoded image and having the longest duration as the first-level subtitles, the first-level subtitles represent the information contained in the video signal of the news type or talk type. the subject matter covered; In all video decoded images containing the first-level subtitles, it will be located in the left middle or right middle of the video decoded image, and the duration is a second duration shorter than the first duration as the second-level subtitles, and according to the surname, title or salutation expressed by the identification code and reliability of the image of the second-level subtitles after optical character recognition, the second-level subtitles are divided into different subtitles, said different subtitles respectively representing identifiers of different participants of said news or conversation during the duration of said first level subtitles; In all video decoded images containing the first-level subtitles and second-level subtitles, the subtitles will be located in the lower part of the video decoded images, and the duration is a third duration shorter than the second duration The subtitles are determined as third-level subtitles, wherein the part of the third-level subtitles that coexists with different subtitles in the second-level subtitles is determined as the content of different participants represented by the different subtitles the content of the conversation; and Selecting at least one frame of decoded video image of concatenated subtitles including first-level subtitles, second-level subtitles and third-level subtitles as the concatenated video summary of the video signal. 22. The device according to claim 20 or 21, further comprising a video summary output unit configured to output the video summary generated by the video summary generation unit. 23. An apparatus for generating a video summary of a video signal, comprising: A video decoding unit is used to decode video signals so as to obtain multi-frame video decoding images; A subtitle characteristic obtaining unit, configured to obtain at least one of the following characteristics of each subtitle in all the subtitles contained in the multi-frame video decoded image: the duration of the subtitle, the position of the subtitle in the corresponding video decoded image, the subtitle The character size of the subtitle image, the identification code and reliability of the subtitle image after OCR; and A video summary generation unit, configured to process at least one characteristic of the obtained subtitles according to the relationship between the content of the video signal and the characteristics of the subtitles, so as to generate a video summary suitable for the content of the video signal , Wherein, the relationship between the content of the video signal and the characteristics of the subtitle indicates that: the subtitle is located at a fixed position in the decoded video image and relates to the main content of the video signal, and The video summary generating unit is configured to compare the subtitles located at the fixed positions, determine the subtitles at the point in time when the image of the subtitles changes, and compose the subtitles from video decoded images containing the determined subtitles Video summary of the video signal. 24. The device as claimed in claim 23, wherein the shown video signal is a video signal of a game type, and the subtitles positioned at a fixed position are a scoreboard displaying score information of a game in the video decoded image, as well as The video summary generating unit is configured to: compare all scoreboard subtitles, determine scoreboard subtitles at a point in time at which an image of scoreboard subtitles changes, and compose by video decoding images containing the determined scoreboard subtitles A video digest of the video signal. 25. The device according to claim 23 or 24, further comprising a video summary output unit configured to output the video summary generated by the video summary generation unit. 26. An apparatus for generating a video summary of a video signal, comprising: A video decoding unit is used to decode video signals so as to obtain multi-frame video decoding images; A subtitle characteristic obtaining unit, configured to obtain at least one of the following characteristics of each subtitle in all the subtitles contained in the multi-frame video decoded image: the duration of the subtitle, the position of the subtitle in the corresponding video decoded image, the subtitle The character size of the subtitle image, the identification code and reliability of the subtitle image after OCR; and A video summary generation unit, configured to process at least one characteristic of the obtained subtitles according to the relationship between the content of the video signal and the characteristics of the subtitles, so as to generate a video summary suitable for the content of the video signal , Wherein, the relationship between the content of the video signal and the characteristics of the subtitles indicates that subtitles with a larger character size and a longer subtitle duration relate to more important content of the video signal, and The video summary generation unit is configured to: Setting the size threshold H of the character size of the subtitle and the time threshold T of the duration of the subtitle; and Subtitles with a character size greater than H and a duration greater than T are determined, and a video digest of the video signal is composed from video decoded images containing the determined subtitles. 27. The apparatus according to claim 26, wherein the subtitle characteristic obtaining unit is configured to obtain the character height of the subtitle as the character size of the subtitle through the following processing: processing the image of the subtitle to obtain a binarized image of the image of the subtitle; If the height H of the image of the subtitle is less than or equal to 2 times of the predefined character height MIN_CS, then the image height H is determined as the character height of the subtitle, otherwise enter the next step; For the image of the horizontal subtitle, horizontal projection is performed, and the stroke pixel value of the subtitle image is accumulated in the horizontal direction, and the accumulated value is stored in Hist(i), i=1,2,....H, or, for the image of the vertical subtitle, the Vertical projection, accumulating the stroke pixel values of the subtitle image in the vertical direction, the accumulated value is stored in Hist(i), i=1,2,....H, where i is an index value as a positive integer, representing the segmentation point; If Hist(i)<=T, it means that the subtitle is separable at the position of the split point i, record the position coordinate values i1, i2,...ij of all possible separable positions, and set (i2-i1) ,(i3-i2),...,(ij-i(i-1)) is the average value of the character height of the subtitle, and, when Hist(i)>T, it means that the subtitle is inseparable, and the next step is processed , wherein j is a positive integer less than or equal to H, H is the number of split points, and T is a predetermined constant; Connected domain CCA analysis is performed on the image features of subtitles, and the average value of CC height of character components of subtitles is estimated to be H1; Perform an alignment operation on the character component CC of the subtitle, and obtain a new character component CC_New after alignment, and estimate the average height of the new character component CC_New to be H2; If the area of a certain character component in the new character component CC_new is greater than (α*H*W) or the number of the new character component CC_new is equal to 1, then the character height of the subtitle is determined as H1, otherwise the character height of the subtitle is determined is H2, where α is a predetermined constant, and W is the width of the subtitle image. 28. The device according to claim 26 or 27, further comprising a video summary output unit configured to output the video summary generated by the video summary generation unit. 29. An apparatus for generating a video summary of a video signal, comprising: A video decoding unit is used to decode video signals so as to obtain multi-frame video decoding images; A subtitle characteristic obtaining unit, configured to obtain at least one of the following characteristics of each subtitle in all the subtitles contained in the multi-frame video decoded image: the duration of the subtitle, the position of the subtitle in the corresponding video decoded image, the subtitle The character size of the subtitle image, the identification code and reliability of the subtitle image after OCR; and A video summary generation unit, configured to process at least one characteristic of the obtained subtitles according to the relationship between the content of the video signal and the characteristics of the subtitles, so as to generate a video summary suitable for the content of the video signal , Wherein, the relationship between the content of the video signal and the characteristics of the subtitle indicates that the subtitle has a longer duration and the subtitle located at a specific position in the video decoded image involves more important content of the video signal, and The video summary generation unit is configured to: setting a time threshold T for the duration of subtitles; and A subtitle whose duration is greater than T and located at a specific position of the video decoded image is determined, and a video digest of the video signal is composed from the video decoded image containing the determined subtitle. 30. The device according to claim 29, further comprising a video summary output unit configured to output the video summary generated by the video summary generation unit. 31. An apparatus for generating a video summary of a video signal, comprising: A video decoding unit is used to decode video signals so as to obtain multi-frame video decoding images; A subtitle characteristic obtaining unit, configured to obtain at least one of the following characteristics of each subtitle in all the subtitles contained in the multi-frame video decoded image: the duration of the subtitle, the position of the subtitle in the corresponding video decoded image, the subtitle The character size of the subtitle image, the identification code and reliability of the subtitle image after OCR; and A video summary generation unit, configured to process at least one characteristic of the obtained subtitles according to the relationship between the content of the video signal and the characteristics of the subtitles, so as to generate a video summary suitable for the content of the video signal , Wherein, the relationship between the content of the video signal and the characteristics of the subtitles indicates that the subtitles with larger character size and located at a specific position in the video decoded image relate to more important content of the video signal, and The video summary generation unit is configured to: setting the size threshold H of the character size of the subtitle; and A subtitle whose character size is greater than H and located at the specific position of the video decoded image is determined, and a video summary of the video signal is composed from the video decoded image including the determined subtitle. 32. The apparatus according to claim 31 , wherein the subtitle characteristic obtaining unit is configured to obtain the character height of the subtitle as the character size of the subtitle through the following processing: processing the image of the subtitle to obtain a binarized image of the image of the subtitle; If the height H of the image of the subtitle is less than or equal to 2 times of the predefined character height MIN_CS, then the image height H is determined as the character height of the subtitle, otherwise enter the next step; For the image of the horizontal subtitle, horizontal projection is performed, and the stroke pixel value of the subtitle image is accumulated in the horizontal direction, and the accumulated value is stored in Hist(i), i=1,2,....H, or, for the image of the vertical subtitle, the Vertical projection, accumulating the stroke pixel values of the subtitle image in the vertical direction, the accumulated value is stored in Hist(i), i=1,2,....H, where i is an index value as a positive integer, representing the segmentation point; If Hist(i)<=T, it means that the subtitle is separable at the position of the split point i, record the position coordinate values i1, i2,...ij of all possible separable positions, and set (i2-i1) ,(i3-i2),...,(ij-i(j-1)) is the average value of the character height of the subtitle, and, when Hist(i)>T, it means that the subtitle is inseparable and enters the next step of processing , wherein j is a positive integer less than or equal to H, H is the number of split points, and T is a predetermined constant; Connected domain CCA analysis is performed on the image features of subtitles, and the average value of CC height of character components of subtitles is estimated to be H1; Perform an alignment operation on the character component CC of the subtitle, and obtain a new character component CC_New after alignment, and estimate the average height of the new character component CC_New to be H2; If the area of a certain character component in the new character component CC_new is greater than (α*H*W) or the number of the new character component CC_new is equal to 1, then the character height of the subtitle is determined as H1, otherwise the character height of the subtitle is determined is H2, where α is a predetermined constant, and W is the width of the subtitle image. 33. The device according to claim 31 or 32, further comprising a video summary output unit configured to output the video summary generated by the video summary generation unit. 34. An apparatus for generating a video summary of a video signal, comprising: A video decoding unit is used to decode video signals so as to obtain multi-frame video decoding images; A subtitle characteristic obtaining unit, configured to obtain at least one of the following characteristics of each subtitle in all the subtitles contained in the multi-frame video decoded image: the duration of the subtitle, the position of the subtitle in the corresponding video decoded image, the subtitle The character size of the subtitle image, the identification code and reliability of the subtitle image after OCR; and A video summary generation unit, configured to process at least one characteristic of the obtained subtitles according to the relationship between the content of the video signal and the characteristics of the subtitles, so as to generate a video summary suitable for the content of the video signal , Wherein, the relationship between the content of the video signal and the characteristics of the subtitle indicates that the subtitle located at a specific position in the video decoded image and whose subtitle image has a higher reliability after optical character recognition is more important in relation to the video signal the content of , and The video summary generation unit is configured to: Setting the threshold L of the reliability of the subtitle image after optical character recognition; and Determining the subtitles located at the specific position in the decoded video image and having a reliability greater than L after optical character recognition of the subtitle image, and composing a video abstract of the video signal from the decoded video image containing the determined subtitles. 35. The device according to claim 34, further comprising a video summary output unit configured to output the video summary generated by the video summary generation unit. 36. An apparatus for generating a video summary of a video signal, comprising: A video decoding unit is used to decode video signals so as to obtain multi-frame video decoding images; A subtitle characteristic obtaining unit, configured to obtain at least one of the following characteristics of each subtitle in all the subtitles contained in the multi-frame video decoded image: the duration of the subtitle, the position of the subtitle in the corresponding video decoded image, the subtitle The character size of the subtitle image, the identification code and reliability of the subtitle image after OCR; and A video summary generation unit, configured to process at least one characteristic of the obtained subtitles according to the relationship between the content of the video signal and the characteristics of the subtitles, so as to generate a video summary suitable for the content of the video signal , Wherein, the relationship between the content of the video signal and the characteristics of the subtitle indicates that the subtitle with a larger character size and a higher reliability of the subtitle image after optical character recognition involves more important content of the video signal, and The video summary generation unit is configured to: Setting the size threshold H of the character size of the subtitle, and setting the threshold L of the reliability of the subtitle image after optical character recognition; and Determine the subtitles whose character size is greater than H and whose reliability after optical character recognition is greater than L, and compose the video summary of the video signal from the video decoding images containing the determined subtitles. 37. The apparatus according to claim 36, wherein the subtitle characteristic obtaining unit is configured to obtain the character height of the subtitle as the character size of the subtitle through the following processing: processing the image of the subtitle to obtain a binarized image of the image of the subtitle; If the height H of the image of the subtitle is less than or equal to 2 times of the predefined character height MIN_CS, then the image height H is determined as the character height of the subtitle, otherwise enter the next step; For the image of the horizontal subtitle, horizontal projection is performed, and the stroke pixel value of the subtitle image is accumulated in the horizontal direction, and the accumulated value is stored in Hist(i), i=1,2,....H, or, for the image of the vertical subtitle, the Vertical projection, accumulating the stroke pixel values of the subtitle image in the vertical direction, the accumulated value is stored in Hist(i), i=1,2,....H, where i is an index value as a positive integer, representing the segmentation point; If Hist(i)<=T, it means that the subtitle is separable at the position of the split point i, record the position coordinate values i1, i2,...ij of all possible separable positions, and set (i2-i1) ,(i3-i2),...,(ij-i(j-1)) is the average value of the character height of the subtitle, and, when Hist(i)>T, it means that the subtitle is inseparable and enters the next step of processing , wherein j is a positive integer less than or equal to H, H is the number of split points, and T is a predetermined constant; Connected domain CCA analysis is performed on the image features of subtitles, and the average value of CC height of character components of subtitles is estimated to be H1; Perform an alignment operation on the character component CC of the subtitle, and obtain a new character component CC_New after alignment, and estimate the average height of the new character component CC_New to be H2; If the area of a certain character component in the new character component CC_new is greater than (α*H*W) or the number of the new character component CC_new is equal to 1, then the character height of the subtitle is determined as H1, otherwise the character height of the subtitle is determined is H2, where α is a predetermined constant, and W is the width of the subtitle image. 38. The device according to claim 36 or 37, further comprising a video summary output unit configured to output the video summary generated by the video summary generation unit.

Images