KR102685135B1 - Video editing automation system - Google Patents

Abstract

The present invention relates to an automated video editing system, and more specifically, to an automated video editing system that allows general users to edit video content or edit video more easily and conveniently. According to one aspect of the present invention, an automated video editing system is provided that can automatically search and edit video content on the web, and this system includes a text receiving unit that receives text input from a user, and an analysis unit that analyzes the input text. A text analysis unit, an image search unit that searches for images on the web based on keyword results analyzed in the text analysis unit, a sample image extraction unit that extracts metadata and sample images from the searched images, and an image analysis unit that analyzes the extracted sample images. It includes a correlation evaluation unit that analyzes the correlation with the user input text based on the analyzed image information, and a section video download unit that downloads the corresponding video for each section in response to the evaluation results of the relevance evaluation unit.

Description

Video editing automation system {VIDEO EDITING AUTOMATION SYSTEM}

The present invention relates to an automated video editing system, and more specifically, to an automated video editing system that allows general users to edit video content or edit video more easily and conveniently.

Recently, with the rapid development of IT and multimedia technologies, ordinary individuals, not professionals or companies, can directly take various types of photos and videos to create personal digital content, and edit the digital content created in this way to post it on various networks. By uploading to a server, it becomes possible to distribute, share, and sell to others.

However, despite these technological developments, most ordinary individuals share the original digital content they produce without editing, and are unable to give it commercial value as digital content. The majority of commercialized digital content is generally produced and serviced by broadcasters or content companies with specialized editing knowledge, and the reality is that individuals simply passively watch the videos provided from the perspective of viewers.

In particular, when ordinary users create digital video content, it takes a lot of effort and time to search for video clips that they want to use in digital video content from web pages scattered on the Internet, review them individually, and extract the desired video. Accordingly, there is a problem that it is not easy for ordinary users to produce high-quality digital video content.

Republic of Korea Patent Publication No. 10-2020-0044435 (April 29, 2020)

The present invention is an invention made based on the above-described problems, so that general users or individual users can more easily and conveniently create digital video content, so that it can be used on the Internet, etc. according to the user's desired topic or the storyline of the digital content that the user wants to create. The purpose is to provide an automated video editing system that can automatically provide users with appropriate videos among scattered videos.

According to one aspect of the present invention in order to solve the above-described problem, an automated video editing system is provided that can automatically search and edit video content on the web, and this system includes a text receiver that receives text input from the user. , a text analysis unit that analyzes the input text, an image search unit that searches for images on the web based on the keyword results analyzed in the text analysis unit, a sample image extraction unit that extracts metadata and sample images from the searched images, An image analysis unit that analyzes sample images, a relevance evaluation unit that analyzes the correlation with user input text based on the analyzed image information, and a section video that downloads the corresponding video for each section in response to the evaluation results of the relevance evaluation unit. Includes download section.

In the above-described aspect, the text receiving unit is configured to receive a sentence consisting of text input by the user, a text file received through file upload, a set of keywords, or a voice-recognized text, and the text analyzing unit analyzes the input text sentence It is configured to extract keywords for each sentence, and the text analysis unit includes a deep learning-based text mining model.

In addition, in any of the above-described aspects, the image analysis unit includes an image-based object detection unit and an image-based text generation unit, and the image-based object detection unit determines the type, size, number, and location of the object through deep learning analysis of the object included in the image. It includes a deep learning-based object detection module that identifies, and the image-based text generator includes a deep learning-based image caption module that outputs context information text through deep learning analysis of objects included in the image or situations containing the object. .

In addition, in any of the above-described aspects, the relevance evaluation unit uses the context information analyzed by the text analysis unit, the image detection result analyzed by the image-based object detection unit, and the context information text generated by the image-based text generation unit. , It functions to evaluate the degree of correlation between the text sentences entered by the user and the analysis results analyzed by the image analysis unit.

In addition, in any of the above-described aspects, the relevance evaluation unit

- First input information including information on the type, number, and size of the object detected by the image-based object detection unit;

- Second input information including context information text generated by the image-based text generator;

- Third input information including an image feature map obtained at an intermediate stage of deep learning models used in the image-based object detection unit and the image-based text generation unit;

- Fourth input information including keyword information extracted from the text mining model of the text analysis unit;

- Functions to evaluate the degree of correlation between the text sentence entered by the user and the analysis results analyzed by the image analysis unit using the fifth input information including context information extracted from the context analysis model of the text analysis unit. .

In addition, in any of the above-described aspects, the relevance evaluation unit generates a comparison vector A by combining the first to third input information; combining the fourth input information and the fifth input information to generate a comparison vector B; It functions to calculate a similarity score between comparison vector A and comparison vector B, where the similarity score may be a Word2Vec model that evaluates similarity between natural languages or a modified model thereof.

According to an embodiment of the present invention, when a user simply inputs a desired text sentence or set of search words into a text input box provided on a web page screen, a section image suitable for the text sentence or set of search words is automatically provided on the user's editing work template. Users can obtain relevant section images more easily and conveniently without having to search numerous images to obtain desired image information on the web, enabling faster and higher quality production of digital video content.

1 is a diagram showing an example of an automated video editing system according to an embodiment of the present invention;
Figure 2 is a diagram showing an example of a screen configuration provided to a user from a web server of an automated video editing system according to the present invention;
Figure 3 is a diagram showing another example of a screen configuration provided to a user from a web server of an automated video editing system according to the present invention;
Figure 4 is a diagram showing the internal structure of the automated video editing system according to the present invention by function; and
Figure 5 is a diagram showing the operational flow of the automated video editing system according to the present invention.

The advantages and features of the present invention and methods for achieving them will become clear by referring to the embodiments described in detail below along with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below and will be implemented in various different forms.

The examples herein are provided to make the disclosure of the present invention complete and to fully inform those skilled in the art of the scope of the invention. And the present invention is only defined by the scope of the claims. Accordingly, in some embodiments, well-known components, well-known operations and well-known techniques are not specifically described in order to avoid ambiguous interpretation of the present invention.

Like reference numerals refer to like elements throughout the specification. Also, the terms used (mentioned) in this specification are for describing embodiments and are not intended to limit the present invention. As used herein, singular forms also include plural forms, unless specifically stated otherwise in the context. Additionally, components and operations referred to as 'including (or, including)' do not exclude the presence or addition of one or more other components and operations.

Unless otherwise defined, all terms (including technical and scientific terms) used in this specification may be used with meanings that can be commonly understood by those skilled in the art to which the present invention pertains. Additionally, terms defined in commonly used dictionaries are not interpreted ideally or excessively unless they are defined.

Hereinafter, embodiments of the present invention will be described with reference to the attached drawings.

1 is a diagram showing an automated video editing system according to an embodiment of the present invention. As shown in FIG. 1, the video editing automation system 100 of the present invention is connected to one or more user terminals through a network, and includes a main menu selectable on a web page and selection of the main menu according to the request of the user terminal. A template determined in response is provided, and an editing tool for creating digital content using the template is provided to the user terminal.

It is a computing device that allows the user to access the video editing automation system through the network using the user terminal, search for videos using tools provided by the system, edit the searched videos, and create digital content. In particular, web browsers are used to create digital content. A portable terminal, PC, laptop, or workstation capable of running a web browser such as Internet Explorer, Firefox, and Chrome that can perform video editing services provided based on the browser can be used.

Accordingly, the user terminal may be equipped with a computing device, a storage device, an input device, an output device, etc., and an editing tool may be executed through a web browser to produce digital video content online.

Here, the network is a concept that includes not only short-range wireless communication networks such as WiFi and Zigbee, but also long-distance wireless communication networks such as CDMA, WCDMA, Wibro, 3G, and LTE, and may be the same or separate networks.

The video editing automation system 100 automatically searches for sources such as photos and videos on the Internet based on text input from each user terminal, and provides the searched photos and videos to the user screen interface so that users can more easily access digital content. It may be a computing device, for example, a web server, equipped with a high-performance microprocessor and large storage space capable of generating data.

Figure 2 is a diagram showing an example of a web page screen provided to a user when the user accesses the automated video editing system 100 through a web browser. As shown in Figure 2, a part of the user's web browser screen, a text input window for receiving text (or sentences) from the user is provided on the left side of the screen, and another part of the web browser screen, the drawing shown. A search result window 1520 is provided on the right side of the screen in which searched video clips corresponding to the text (or sentence) entered by the user are listed.

On this web page screen, the user enters keywords or sentences desired to be searched in the text input window 1501 and clicks the search button, or uploads a text file stored in the user terminal using the file upload function and then clicks the search button. When you perform a search by pressing , the automated video editing system 100 extracts keywords by analyzing sentences entered by the user, or searches for images from the Internet web using the keywords entered by the user, and selects highly relevant images among the searched images. Videos are provided to the user's screen in order.

FIG. 3 is a diagram showing another example of a web page screen provided to a user when the user accesses the automated video editing system 100 through a web browser. As shown in FIG. 3, the user's web browser screen is composed of a plurality of templates (Ta to Td), and the user can search for video clips according to the storyline of the video content to be produced.

For example, on the web page screen provided in this way, for each template, enter keywords or sentences you wish to search in the text input box and click the search button, or use the file upload button (not shown) to upload to the user terminal. When a search is performed by pressing the search button after uploading a saved text file, the automated video editing system 100 analyzes sentences entered by the user and extracts keywords or context information for each sentence, or extracts keywords entered by the user. Context information is extracted from the set, images are searched from the Internet web using keyword information or context information extracted for each sentence, and the searched images are provided to the user's screen in the order of highest relevance.

FIG. 4 is a block diagram showing the internal structure of the video editing automation system that provides the video editing automation service described above by function, and FIG. 5 is a block diagram schematically showing processing procedures in the video editing automation system. The internal components of the present invention and the functions performed by the internal components will be sequentially described with reference to FIGS. 4 and 5. As shown, the automated video editing system 100 according to the present invention includes a text receiver 110 that receives text input from a user, a text analysis unit 120 that analyzes the input text, and a text analysis unit 120. An image search unit 130 that searches for images from the web based on the analysis results (keywords and/or context information); a sample image extraction unit 140 that extracts metadata and sample images from the searched images; An image analysis unit 150 that analyzes the extracted sample image; A correlation evaluation unit 160 that analyzes the correlation with the user input text based on the analyzed image information; and a section video download unit 170 that downloads the corresponding video for each section in response to the evaluation result of the relevance evaluation unit 160.

As shown in FIGS. 2 and 3, the text receiving unit 110 is configured to receive a sentence consisting of text input in a screen configuration, a text file received through file upload, or a set of keywords. Although not shown, the present invention is not limited to this, and when a voice is input from a user terminal, the voice input by the user is converted into text using a voice recognition function, and the converted text may be configured to be received in the text receiver 110. there is.

A sentence consisting of text received from the text receiver 110 is transmitted to the text analysis unit 120, and the text analysis unit 120 is configured to analyze the input text and extract main keywords. To extract such keywords, the text analysis unit 120 may include an artificial learning-based text mining model.

The text analysis unit 120 performs text preprocessing to extract keywords from the received text sentences. In the case of text, it consists of natural forms of language that humans can perceive, such as Korean letters, English letters, and numbers. In order to provide keywords through text mining, the input text must be expressed in a form that the computer can understand, and the resulting technology is called Natural Language Processing (NLP).

Natural languages are distinct from artificial languages, such as computer program languages. Artificial language is a language created by humans to set rules so that computers can understand, so anyone can understand it if they learn the rules. However, in natural languages, although humans set rules, over a long period of time, customary rules, informal rules, etc. Because various changes are added, there are characteristics that cannot be understood through rules alone. The natural language processing process can be divided into four stages: morphological analysis, syntactic analysis, semantic analysis, and pragmatic analysis. Each stage is not a stage in which a clear theory is established through a specific theory, but a development stage in which theories are continuously presented and verified. there is.

- Handling of strings

- Automatic spacing processing

The most basic task in text analysis is tokenization, which identifies and extracts words from text. Korean considers 'word clauses' as the unit of token, and unlike languages such as Chinese or Japanese that do not have word boundary markers, it stipulates that words should be written with spaces between them. In Korean, incorrect spacing causes ambiguity or noise in text analysis, which actually hinders tokenization and reduces readability. Likewise, spacing in Korean is an important factor that affects machine readability as much as user readability of text. Spacing errors within a sentence cause a lot of grammatical and semantic ambiguity, and sometimes make morphological analysis impossible.

In the present invention, it is assumed that the user may input text without considering spacing, and preprocessing is performed by applying automatic spacing.

- WPM (Word Piece Model) processing

WPM is a method for building a voice search system that automatically generates vocabulary with minimal perplexity without prior knowledge of the language. Existing natural language processing involves four processes: morphological analysis, syntactic analysis, semantic analysis, and pragmatic analysis, but WPM encodes units into a pronunciation set based on the International Phonetic Alphabet (PA). Using statistical techniques, new units are created by combining them according to frequency of use. The advantage of WPM is that it is language-independent and uses statistical methods, so it can be applied to specific domains or languages whose meaning is not yet understood. In the present invention, the WPM algorithm is applied to separate the noun and the particle and remove the particle to extract only the noun.

In addition, when a noun and a noun appear consecutively, it is searched from the data stored in the database to determine whether it can be used as a compound noun, and when a matching result is found, it is treated as a compound noun.

After preprocessing is performed on the text, noise filtering is performed on the preprocessed text. Here, noise filtering refers to the process of excluding predefined stop words. Afterwards, the following processes may be further used to extract keyword information and/or context information from the text sentence.

° Morpheme (word) analysis: Analysis of morphemes or words, the smallest unit of meaning, is the most common method of analyzing text, including part of speech tagging and entity name recognition based on grammatical rules or probability. (named entity recognition), spelling correction, word identification (tokenization) techniques, etc.

°Key phrase extraction: Extract key phrases through chunking, dividing the text into lexically related words, or use it for entity name recognition, relationship extraction, etc.

°Vector space modeling: In the vector space model, statistical weights are obtained by matching documents and words to vectors and dimensions, respectively. Filter out unimportant words from the word frequency distribution and extract document similarity through TF-IDF, chi-square test, cosine similarity, etc.

°Co-occurrence word analysis: To determine the semantic relationship between words, calculate the frequency of co-occurrence of two words within a certain context and then extract meaningful word pairs using various statistical methods.

°Topic modeling: A statistical inference algorithm for discovering and categorizing hidden topic structures in large amounts of unstructured text, the LDA (Latent Dirichlet Allocation) model is mainly used.

°Semantic network analysis: Based on or beyond the limitations of keyword co-occurrence analysis techniques, define semantic or contextual connections between words, visualize the connections, or extract important concepts according to centrality in the network.

The set of keywords extracted for each sentence through the text analysis unit 120 is delivered to the video search unit 130, and the video search unit 130 searches for videos on the web using web crawling technology based on the extracted keywords. and create a list of searched videos. Additionally, when creating a list of videos, the video search unit 130 operates to store additional metadata if metadata for the video exists on the web page for each video. Metadata includes information such as the title, producer, genre, and production date of the video, but the present invention is not limited to the above-described metadata information. Meanwhile, metadata may further include header information of the video file.

After the list of images searched by the image search unit 130 is created, the sample image extraction unit 140 extracts a small number of sample images from random sections of each image and performs downloading. This is to not only reduce the burden of network traffic compared to downloading all searched videos, but also reduce the burden of storage space where videos can be downloaded.

Meanwhile, in the above description, the image search unit 130 was described as searching images on the web, but the present invention is not limited to this and performs a search for image files such as jpg, jepg, gif, bmp, png, or image files. It may also be configured to perform image searches in addition to searches.

The sample image extracted from the sample image extraction unit 140 is then provided to the image analysis unit 150. The image analysis unit 150 performs analysis on the sample image extracted through the image-based object detection unit 152 and the image-based text generation unit 154.

The image-based object detection unit 152 collectively refers to computer vision technology that identifies the type of object or object included in the image through deep learning analysis. In this context, the image-based object detection unit 152 includes object detection and object recognition technology. Object detection and object recognition are similar object identification technologies, but there are differences in how they are implemented. Object detection is the process of finding instances of objects in images. In the case of deep learning, object detection is a subset of object recognition that not only identifies objects in images but also determines their location. This allows you to identify multiple objects in one image and determine their respective locations. In the present invention, the image-based object detection unit preferably uses a deep learning-based or machine learning-based object detection module that can not only identify objects in an image but also determine their location and identify multiple objects in the image. Such deep learning analysis includes deep neural network methods such as CNN (Convolutional Neural Network) and RNN (Recurrent Neural Network), which include multiple hidden layers. Various improved methods are continuously being introduced, and rapid changes are made depending on the field of application. It provides improved performance and various analysis results.

The image-based text generator 154 outputs text information through deep learning analysis of objects included in the extracted image or situations containing objects. Similar to deep learning-based object detection modules, this deep learning analysis can use deep neural network methods such as CNN (Convolutional Neural Network) and RNN (Recurrent Neural Network) that include multiple hidden layers. Therefore, the image-based text generator 154 provides not only basic analysis results on the type, location, and size of the object in the image, but also recently provides the psychological state of the object, detailed descriptions of parts of the object, and recognition results for the situation. It is possible to provide a considerable amount of situational information through images in the form of text and analysis information (accuracy, weight, additional analysis information, etc.).

The relevance evaluation unit 160 includes context information analyzed by the previously described text analysis unit 120, image detection results analyzed by the image-based object detection unit 152, and context information generated by the image-based text generation unit 154. It functions to evaluate the degree of correlation between the text sentence input by the user and the analysis result analyzed by the image analysis unit 150 using text.

Specifically, the relevance evaluation unit 160 inputs the following:

- Input 1: Information on the type, number, and size of the object detected by the image-based object detection unit 152

- Input 2: Context information text generated by the image-based text generator 154

- Input 3: Image feature map obtained at the intermediate stage of the deep learning models used in the image-based object detection unit 152 and the image-based text generation unit 154.

- Input 4: Keyword information extracted from the text mining model of the text analysis unit 120

- Input 5: Context information extracted from the context analysis model of the text analysis unit 120

Analyze the correlation using .

The relevance evaluation unit 160 generates comparison vector A by combining input 1 to input 3, and generates comparison vector B by combining input 4 to input 5, and then calculates the similarity between comparison vector A and comparison vector B. Functions to calculate scores. In evaluating the similarity score, the Word2Vec model, which evaluates the similarity between natural languages, or a modified model thereof can be used.

The relevance evaluation unit 160 provides feedback to the sample image extraction unit 140 to attempt to extract more images for images containing images evaluated as having a high relevance, while the relevance evaluation unit 160 provides feedback to the sample image extraction unit 140 for images containing images with a low relevance. It is desirable to configure it to shorten the time required for image analysis and improve the quality of the final result by excluding images that are damaged. In this way, in the present invention, the advantage of being able to perform analysis directly from the image without tag information in the image can be obtained by analyzing the correlation between the image extracted from the image searched through the relevance evaluation unit and the information obtained from the user input text. there is.

In addition, after the relevance evaluation unit 160 gives a relevance evaluation score to each image, the section video download unit 170 downloads the section video for the image with a high degree of relevance, and the downloaded section video is stored in the database. Meanwhile, the analysis results and matching information about the image are stored together in the indexing database, thereby increasing the reusability of the image.

Once the section video is acquired or downloaded from the section video download unit 170, the downloaded section video is provided to the video editing template on the user's screen or is automatically added to the timeline of the selected template, and the user can access the desired video information on the web. You can acquire related section videos more easily and conveniently without having to search through numerous videos to acquire them, and section videos are automatically provided to the user according to the storyline (or editing template) of the digital content you want to create. Can help with production.

The device described above may be implemented with hardware components, software components, and/or a combination of hardware components and software components. For example, devices and components described in embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), etc. , may be implemented using one or more general-purpose or special-purpose computers, such as a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. A processing device may execute an operating system (OS) and one or more software applications that run on the operating system. Additionally, a processing device may access, store, manipulate, process, and generate data in response to the execution of software. For ease of understanding, a single processing device may be described as being used; however, those skilled in the art will understand that a processing device includes multiple processing elements and/or multiple types of processing elements. It can be seen that it can include . For example, a processing device may include a plurality of processors or one processor and one controller. Additionally, other processing configurations, such as parallel processors, are possible.

Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing unit to operate as desired, or may be processed independently or collectively. You can command the device. Software and/or data may be used on any type of machine, component, physical device, virtual equipment, computer storage medium or device to be interpreted by or to provide instructions or data to a processing device. It can be embodied in . Software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., singly or in combination. Program instructions recorded on the medium may be specially designed and configured for the embodiment or may be known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -Includes optical media (magneto-optical media) and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, etc. Examples of program instructions include machine language code, such as that produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc.

As described above, although the embodiments have been described with limited examples and drawings, various modifications and variations can be made by those skilled in the art from the above description. For example, the described techniques are performed in a different order than the described method, and/or components of the described system, structure, device, circuit, etc. are combined or combined in a different form than the described method, or other components are used. Alternatively, appropriate results may be achieved even if substituted or substituted by an equivalent.

Therefore, other implementations, other embodiments, and equivalents of the claims should also be construed as falling within the scope of the claims described below.

100: Video editing automation system 110: Text receiver
120: text analysis unit 130: video search unit
140: sample image extraction unit 150: image analysis unit
160: Relevance evaluation unit 170: Section video download unit
152: Image-based object detection unit 154: Image-based text generation unit

Claims (6)

A text receiver 110 that receives a sentence consisting of text input by the user, a text file received through file upload, a set of keywords, or a voice-recognized text;
A text analysis unit 120 that analyzes input text sentences and extracts keywords for each sentence,
The text analysis unit 120 includes a deep learning-based text mining model,
an image search unit 130 that searches for images on the web based on keyword results analyzed by the text analysis unit 120;
a sample image extraction unit 140 that extracts metadata and sample images from the searched images;
An image analysis unit 150 that analyzes the extracted sample image;
The image analysis unit 150 includes an image-based object detection unit 152 and an image-based text generation unit 154,
The image-based object detection unit 152 includes a deep learning-based object detection module that identifies the type, size, number, and location of the object included in the image through deep learning analysis,
The image-based text generator 154 includes a deep learning-based image caption module that outputs context information text through deep learning analysis of objects included in the image or situations containing objects,
A correlation evaluation unit 160 that analyzes the correlation with the user input text based on the analyzed image information,
The relevance evaluation unit 160
- First input information including information on the type, number, and size of the object detected by the image-based object detection unit 152;
- Second input information including context information text generated by the image-based text generator 154;
- Third input information including an image feature map obtained at an intermediate stage of deep learning models used in the image-based object detection unit 152 and the image-based text generation unit 154;
- Fourth input information including keyword information extracted from the text mining model of the text analysis unit 120;
- Between the text sentence entered by the user using the fifth input information including context information extracted from the context analysis model of the text analysis unit 120 and the analysis result analyzed by the image analysis unit 150 Functions to evaluate the degree of correlation,
An automated video editing system comprising a section video download unit 170 that downloads the corresponding video for each section in response to the evaluation result of the relevance evaluation unit 160. delete delete delete delete According to paragraph 1,
The relevance evaluation unit 160
Generate a comparison vector A by combining the first to third input information; combining the fourth input information and the fifth input information to generate a comparison vector B; Functions to calculate a similarity score between comparison vector A and comparison vector B,
An automated video editing system characterized in that the similarity score uses the Word2Vec model, which evaluates the similarity between natural languages, or a modified model thereof.