JP7669555B1 - Information processing device, information processing method, and program - Google Patents

Abstract

[Problem] To reduce the burden of modifying materials to fit content. [Solution] An information processing device 1 having a receiving unit 131 that receives materials for constituting video content and generation instructions for generating video content, an acquisition unit 132 that acquires metadata that is data describing the materials, a specification unit 133 that specifies modification content for applying the materials to the video content based on the generation instructions and the metadata of the materials, and a generation unit 134 that generates a prompt for inputting instructions to modify the materials to an image generation AI based on the specified modification content. [Selected Figure] Figure 2

Description

The present invention relates to an information processing device, an information processing method, and a program.

Devices that create content by combining materials are known (see, for example, Patent Document 1).

JP 2024-027453 A

Such materials may not match the desired video content, which creates limitations on content generation while also creating problems such as the heavy burden of manually correcting the materials.

The present invention was made in consideration of these points, and aims to reduce the burden of modifying materials to fit content.

The information processing device of the first aspect of the present invention includes a receiving unit that receives materials for constituting video content and generation instructions for generating the video content, an acquisition unit that acquires metadata that is data describing the materials, a specification unit that specifies corrections for applying the materials to the video content based on the generation instructions and the metadata of the materials, and a generation unit that generates a prompt for inputting instructions to modify the materials to an image generation AI based on the specified corrections.

The metadata includes the aspect ratio of the material, and the generation instruction includes a reference aspect ratio that is a standard aspect ratio for the video content. When the aspect ratio indicated by the metadata differs from the reference aspect ratio, the identification unit may identify a change in the aspect ratio of the material as the modification content, and the generation unit may generate a prompt to change the aspect ratio of the material to the aspect ratio identified by the identification unit.

The generation instruction includes a description describing the video content, and when the identification unit identifies a change in aspect ratio as the modification content, the generation unit may generate a prompt based on the generation instruction to instruct the user to supplement a portion that is missing in order to change the aspect ratio of the material based on the modification content, based on the description describing the video content indicated by the generation instruction.

When the identification unit identifies a change in aspect ratio as the modification content, the generation unit may generate a prompt based on the generation instruction to instruct the user to cut out a portion that would be redundant if the aspect ratio of the material were changed based on the modification content.

The material includes a photographic image, the metadata includes a description describing a subject of the photographic image, and the generation instructions include a description describing the video content, and the generation unit may generate a prompt to modify the photographic image based on the description describing the video content indicated by the generation instructions when the description describing the subject of the photographic image indicated by the metadata does not match the description describing the video content indicated by the generation instructions.

The system may further include a determination unit that determines whether or not modification is required to apply the material to the video content based on the metadata and the generation instructions, and the identification unit may identify the content of the modification when the determination unit determines that modification of the material is required.

The generation instructions may further include restrictions indicating matters that are restricted from being expressed in the video content, and the generation unit may generate a prompt that further includes the restrictions.

The information processing method of the second aspect of the present invention includes the steps of receiving materials for constituting video content and generation instructions for generating the video content, executed by a computer, acquiring metadata that is data describing the materials, identifying corrections for applying the materials to the video content based on the generation instructions and the metadata of the materials, and generating a prompt for inputting instructions to modify the materials to an image generation AI based on the identified corrections.

The computer is caused to execute the steps of: accepting materials for constructing video content and generation instructions for generating the video content; acquiring metadata, which is data describing the materials; identifying modifications for applying the materials to the video content based on the generation instructions and the metadata of the materials; and generating a prompt for inputting instructions to modify the materials to an image generation AI based on the identified modifications.

The present invention has the effect of reducing the burden of modifying materials to fit content.

FIG. 1 is a diagram for explaining an overview of an information processing device 1 according to an embodiment. 1 is a block diagram showing a configuration of an information processing device 1. FIG. FIG. 11 is a diagram showing an example of a reception screen displayed by a reception unit 131. FIG. 11 is a diagram illustrating an example of a process performed by a specification unit 133. 4 is a flowchart showing a process flow in the information processing device 1.

[Overview of Information Processing System S]
1 is a diagram for explaining an overview of an information processing device 1 according to an embodiment. The information processing system S is a system for providing video content. The information processing system S generates video content based on materials specified by a user. The information processing system S has an information processing device 1, an information terminal 2, and a generation device 3. The materials are contents that are elements that make up the video content. The materials are video data, still image data, text data, audio data, music data, etc.

The information processing device 1 is a device for generating a prompt for giving instructions to a machine learning model. The information processing device 1 determines whether or not the material needs to be modified based on the content specified by the user and the metadata of the material specified by the user. If modification is necessary, the information processing device 1 generates a prompt for instructing the generation device 3 to modify the material.

The information terminal 2 is a terminal used by a user of the information processing system S. The information terminal 2 is, for example, a smartphone, a tablet, or a personal computer. The information terminal 2 transmits instructions to the information processing device 1 regarding the content to be generated.

The generating device 3 generates video content, still images, etc. based on the received instructions. The generating device 3 has a generative model (hereinafter sometimes referred to as "image generation AI") that is a trained model trained to generate video or still images based on instructions written in natural language, and outputs video or still images generated by the generative model based on the input instructions.

The processing of the information processing device 1 will be described. The information processing device 1 acquires materials and a generation instruction from the information terminal 2 ((1) in FIG. 1). The generation instruction is an instruction to generate video content. As an example, the generation instruction includes a description that explains the video content. Specifically, the generation instruction includes information indicating the content expressed by the video content, such as "a man jogging", and subjects and their actions and scenes that appear in the video content. The generation instruction may include information specifying the playback time of the video content to be generated, and may include information indicating a standard aspect ratio for the video content (hereinafter sometimes referred to as a "standard aspect ratio").

The information processing device 1 acquires metadata ((2) in FIG. 1). The metadata is data describing the material. As an example, the metadata includes the aspect ratio of the material. The metadata may also include descriptions describing the material, such as subjects captured in the material or the theme expressed by the material.

The information processing device 1 identifies the modification content of the material based on the generation instruction and the metadata ((3) in FIG. 1). As an example, the information processing device 1 compares the aspect ratio of the video content included in the generation instruction with the aspect ratio included in the metadata of the material, and identifies changing the aspect ratio of the material as the modification content of the material when the aspect ratio of the video content included in the generation instruction differs from the aspect ratio included in the metadata of the material. For example, when the aspect ratio of the material is "4:3" and the standard aspect ratio of the video content is "16:9", the information processing device 1 identifies "changing the aspect ratio to 16:9" as the modification content.

The information processing device 1 generates a prompt for modifying the material based on the identified modification content ((4) in FIG. 1). The prompt is an instruction written in natural language and is an instruction to be given to the machine learning model. For example, if the identified modification content is a change in aspect ratio and the reference aspect ratio included in the generation instruction is "16:9", a prompt is generated stating "Please change the aspect ratio of the input content to 16:9".

The information processing device 1 transmits the generated prompt to the generation device 3 ((5) in FIG. 1). The information processing device 1 may display the generated prompt on the information terminal 2 to prompt the user for confirmation. When the user performs a confirmation operation on the information terminal 2, the information processing device 1 may transmit the generated prompt and the accepted material to the generation device 3 to modify the material. As an example, the generation device 3 modifies the material based on the prompt and displays the modified material on the information terminal 2.

By configuring the information processing system S in this way, it has the effect of reducing the burden of modifying materials to fit the content.

[Configuration of information processing device 1]
2 is a block diagram showing the configuration of the information processing device 1. The information processing device 1 has a communication unit 11, a storage unit 12, and a control unit 13. The control unit 13 has a reception unit 131, an acquisition unit 132, an identification unit 133, a generation unit 134, and a determination unit 135.

The communication unit 11 is a communication interface for transmitting and receiving data to and from other devices via a network. The storage unit 12 is a storage medium including a ROM (Read Only Memory), a RAM (Random Access Memory), an SSD (Solid State Drive), a hard disk drive, etc. The storage unit 12 pre-stores the programs to be executed by the control unit 13.

The control unit 13 is a processor such as a CPU (Central Processing Unit). The control unit 13 executes the programs stored in the storage unit 12, thereby functioning as a reception unit 131, an acquisition unit 132, an identification unit 133, a generation unit 134, and a determination unit 135.

The reception unit 131 receives materials for constructing video content and generation instructions for generating video content. The reception unit 131 displays the reception screen shown in FIG. 3 on the information terminal 2. The reception screen is a screen for receiving generation instructions. As an example, the reception screen includes an interface O1 for selecting materials, an interface O2 for specifying the aspect ratio of the generated video, and an interface O3 for specifying the content of the video. When the user presses the send button O4, the reception unit 131 acquires a generation instruction including the content specified by the user. Note that the reception screen may include an interface for selecting a template associated with video settings such as the aspect ratio and video playback time, instead of the interface for selecting the aspect ratio.

The acquisition unit 132 acquires metadata about the material. The acquisition unit 132 may acquire the metadata about the material from the information terminal 2. The acquisition unit 132 may acquire the metadata from a material management server (not shown) that manages the material. In this case, the acquisition unit 132 accepts a material ID of a specified material from the information terminal 2, and acquires metadata corresponding to the material ID accepted by the acceptance unit 131 from the material management server.
The determination unit 135 determines whether or not modification is necessary to apply the material to the video content based on the metadata and the generation instruction. The determination unit 135 compares the metadata with the generation instruction to determine whether or not modification is necessary. The determination unit 135 may determine whether or not modification is necessary by comparing the aspect ratios included in the metadata and the generation instruction, or may determine whether or not modification is necessary by comparing the contents of the material and the video content included in the metadata and the generation instruction, respectively. The identification unit 133 identifies the content of the modification when the determination unit 135 determines that modification of the material is necessary.

The identification unit 133 identifies the modification content for applying the material to the video content based on the generation instruction and the metadata of the material. As an example, when the aspect ratio indicated by the metadata differs from the reference aspect ratio, the identification unit 133 identifies a change in the aspect ratio of the material as the modification content.

The generation unit 134 generates a prompt for inputting an instruction to modify the material to the image generation AI based on the identified modification content. The generation unit 134 generates a prompt for changing the aspect ratio of the material to the aspect ratio identified by the identification unit 133. As an example, the storage unit 12 stores a template for generating a prompt. The generation unit 134 generates a prompt by fitting the identified modification content to the template stored in the storage unit 12. The storage unit 12 also stores a trained model (hereinafter referred to as a "prompt generation model") that has been trained to output a prompt written in natural language when the modification content is input, and the generation unit 134 inputs the modification content identified by the identification unit 133 into the prompt generation model and causes it to output a prompt corresponding to the modification content.

When the identification unit 133 identifies a change in aspect ratio as the modification content, the generation unit 134 generates a prompt based on the generation instruction to instruct the user to fill in the missing part in order to change the aspect ratio of the material based on the modification content based on the description explaining the video content indicated by the generation instruction. As an example, when generating a vertical video content based on a horizontal image material as shown in FIG. 4, a region R to which the material cannot be applied is generated in the video content. As an example, the identification unit 133 may identify as the region R an area that cannot be filled with the material when the material is placed in the center. In this case, the identification unit 133 identifies the specified region R to be filled based on the content of the video content included in the generation instruction as the modification content. In this case, the generation unit 134 generates a prompt that instructs the user to generate the region R specified by the identification unit 133 based on the content of the video included in the generation instruction. In this case, as an example, the generation unit 134 generates a prompt that describes "Please change the aspect ratio of the input content to 9:16. Please generate the part that is not filled by the input content based on 'a man running'."

In this way, it is desirable for the content to be supplemented in region R to be natural in relation to the material. Therefore, the prompt generation model may learn the user's evaluation of the generated prompt.

More specifically, the information processing device 1 acquires an evaluation indicating whether a user who has viewed material modified by the image generation AI based on a prompt generated by the prompt generation model feels the modified material is natural. The user's evaluation may be expressed as a binary value indicating whether it is felt to be natural or not, or may be expressed as information indicating the degree to which it is felt to be natural. Note that the user's evaluation is not limited to these examples. As an example, the generation model learns training data that associates the prompt generated by the prompt generation model with the evaluation. The prompt generation model may also learn training data that includes prompts that are rated by users at or above a predetermined threshold.

When the identification unit 133 identifies a change in aspect ratio as the modification content, the generation unit 134 generates a prompt based on the generation instruction to instruct the user to crop a portion that would become redundant if the aspect ratio of the material were changed based on the modification content. In this case, the identification unit 133 identifies an area of the material that would become redundant by changing the aspect ratio, and generates a prompt that instructs the user to trim the redundant area.

If the material is a video and the playback time of the material is different from the playback time of the video content associated with the template, a prompt may be generated that indicates how to handle the missing or surplus time. In this case, as an example, the template is associated with a processing method when the playback time of the material is shorter or longer than the playback time of the video content. As an example, when the playback time of the material is shorter than the playback time of the video content, it is associated with playing the material in a loop or generating a digest video in which the content indicated by the material is edited to fill the missing time. Also, when the playback time of the material is longer than the playback time of the video content, it may be associated with deleting a predetermined part of the material (e.g., the end).
When the playback time of the material differs from the playback time of the video content associated with the template, the identification unit 133 refers to a method of dealing with a shortage or surplus time associated with the template and identifies the correction content.

The generation unit 134 generates a prompt based on the correction content identified by the identification unit 133. As an example, if the playback time of the material is 9 minutes and the playback time of the video content is 10 minutes, and the method of handling the excess time is to generate a digest video, the generation unit 134 generates a prompt stating, "Please generate a digest that summarizes the input content into one minute, and connect the generated digest after the input content to generate a video." In addition, if the playback time of the material is 10 minutes and the playback time of the video content is 60 minutes, and the method of handling the excess time is to play in a loop, the generation unit 134 generates a prompt stating, "Please generate a video that repeats the input content six times."

The information processing device 1 may be configured to generate a prompt to correct content captured in a photographic image.

When the description describing the subject of the photographic image indicated by the metadata does not match the description describing the video content indicated by the generation instruction, the generation unit 134 generates a prompt to modify the photographic image based on the description describing the video content indicated by the generation instruction. As an example, when the metadata indicates that the content contained in the image, which is the material, is "5 dumplings" and the generation instruction indicates that the content of the video content is "6 dumplings," the generation unit 134 generates a prompt to modify the material to 6 dumplings.

The information processing device 1 may be configured to generate a prompt that further includes, as an instruction, restrictions on content generation.

The generation instruction further includes restrictions indicating matters that are to be restricted from being expressed in the video content. For example, restrictions are to ensure that brand logo images, characters that are the subject of copyright, and the like are not included in the generated content. For example, restrictions may be to ensure that the content does not include any content that suggests violence, crime, or the like. As an example, an interface for selecting restrictions to be specified for the generated content may be provided on the reception screen, and the reception unit 131 may receive the restrictions selected by the user on the reception screen as a generation instruction.

The generation unit 134 generates a prompt that further includes restrictions. As an example, the generation unit 134 may generate a prompt in which a sentence specifying the content is followed by a sentence indicating the restrictions.

[Processing flow in information processing device 1]
5 is a flowchart showing a flow of processing in the information processing device 1. The reception unit 131 receives a material and a generation instruction (S01). The acquisition unit 132 acquires metadata about the material received by the reception unit 131 (S02).

The identification unit 133 identifies the correction content of the material based on the generation instruction and the metadata of the material (S03). The generation unit 134 generates a prompt for inputting an instruction to correct the material to the image generation AI based on the identified correction content (S04). Then, the information processing device 1 ends the process.

[Effects of information processing device 1]
By configuring the information processing device 1 as described above, an effect is achieved in that the burden of correcting material so as to match it with content is reduced.

Furthermore, this invention will make it possible to contribute to Goal 9 of the United Nations' Sustainable Development Goals (SDGs), which is "Build resilient infrastructure, promote inclusive and sustainable industrialization, and promote innovation and infrastructure."

Although the present invention has been described above using embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments, and various modifications and changes are possible within the scope of the gist of the invention. For example, all or part of the device can be configured by distributing or integrating functionally or physically in any unit. In addition, new embodiments resulting from any combination of multiple embodiments are also included in the embodiments of the present invention. The effect of the new embodiment resulting from the combination also has the effect of the original embodiment.

Reference Signs List 1 Information processing device 2 Information terminal 3 Generation device 11 Communication unit 12 Storage unit 13 Control unit 131 Reception unit 132 Acquisition unit 133 Identification unit 134 Generation unit 135 Determination unit

Claims (8)

a receiving unit that receives a source of photographic images for constituting video content and a generation instruction for generating the video content , the generation instruction including an explanation of the content of the video content ;
an acquisition unit that acquires metadata, which is data describing the material , the metadata including a description of the subject of the photographic image ;
a specification unit that specifies a modification content for applying the material to the video content based on the generation instruction and metadata of the material;
A generation unit that generates a prompt for inputting an instruction to modify the material to an image generation AI based on the identified modification content;
having
When a description of a subject of the photographic image indicated by the metadata does not match a description of the video content indicated by the generation instruction, the generation unit generates a prompt for modifying the material, which is the photographic image, based on the description of the video content indicated by the generation instruction.
Information processing device. the metadata includes an aspect ratio of the material;
the generation instruction includes a reference aspect ratio that is a reference aspect ratio for the video content,
the identifying unit, when the aspect ratio indicated by the metadata is different from the reference aspect ratio, identifies a change in the aspect ratio of the material as the modification content;
the generating unit generates a prompt for changing the aspect ratio of the material to the aspect ratio specified by the specifying unit;
The information processing device according to claim 1 . The generation instruction includes a description explaining the video content,
When the specification unit specifies a change in aspect ratio as the modification content, the generation unit generates, based on the generation instruction, a prompt for instructing to supplement a portion that is insufficient for changing the aspect ratio of the material based on the modification content, based on a description explaining the video content indicated by the generation instruction.
The information processing device according to claim 2 . When the specification unit specifies a change in aspect ratio as the modification content, the generation unit generates, based on the generation instruction, a prompt for instructing to cut off a portion that would become redundant if the aspect ratio of the material is changed based on the modification content.
The information processing device according to claim 2 . a determination unit that determines whether or not the material needs to be modified to apply to the video content based on the metadata and the generation instruction,
The identification unit identifies the content of the correction when the determination unit determines that the material needs to be corrected.
The information processing device according to claim 1 . The generation instruction further includes a restriction indicating a restriction on an expression of the video content,
The generating unit generates a prompt further including the restriction.
The information processing device according to claim 1 . The computer executes
receiving a source of photographic images for constituting video content and a generation instruction for generating the video content, the generation instruction including a description of the content of the video content;
obtaining metadata, which is descriptive data about the material , the metadata including a description of the content of a subject of the photographic image ;
determining, based on the generation instructions and metadata of the material, modifications to be applied to the material to the video content;
generating a prompt for inputting instructions to the image generation AI to modify the material based on the identified modification content;
having
In the generating step, if a description of the subject of the photographic image indicated by the metadata does not match a description of the content of the video content indicated by the generation instruction, a prompt is generated for modifying the material, which is the photographic image, based on the description of the content of the video content indicated by the generation instruction.
Information processing methods. On the computer,
receiving a source of photographic images for constituting video content and a generation instruction for generating the video content, the generation instruction including a description of the content of the video content;
obtaining metadata, which is descriptive data about the material , the metadata including a description of the content of a subject of the photographic image ;
determining, based on the generation instructions and metadata of the material, modifications to be applied to the material to the video content;
generating a prompt for inputting instructions to the image generation AI to modify the material based on the identified modification content;
having
In the generating step, if a description of the subject of the photographic image indicated by the metadata does not match a description of the content of the video content indicated by the generation instruction, a prompt is generated for modifying the material, which is the photographic image, based on the description of the content of the video content indicated by the generation instruction.
A program that executes the following.

Images