KR102741006B1 - Method and apparatus for providing live home shopping in metaverse - Google Patents

Abstract

A method for providing live home shopping according to the present invention includes the steps of receiving a request for use of a predetermined object placed in a metaverse from a user, the step of determining the user's usage right for the predetermined object, the step of receiving live home shopping content from the user in real time if it is determined that the user has the usage right, and the step of displaying the live home shopping content by overlapping it on the predetermined object. According to the present invention, by granting a user in a real space the usage right for a predetermined object in a virtual space and displaying user content on an object in the virtual space in real time, a new revenue-generating environment and service quality can be improved.

Description

METHOD AND APPARATUS FOR PROVIDING LIVE HOME SHOPPING IN METAVERSE

The present invention relates to a method and device for providing live home shopping in a virtual environment such as a metaverse.

Recently, Virtual Reality, Augmented Reality, and Mixed Reality technologies that apply computer graphics technology are being developed.

Here, virtual reality technology refers to a technology that uses a computer to create a virtual space that does not exist in the real world and then makes that virtual space feel real, while augmented reality or mixed reality technology refers to a technology that adds computer-generated information on top of the real world and expresses it, that is, a technology that allows real-time interaction with users by combining the real world and the virtual world.

Among these, augmented reality and mixed reality technologies are being utilized in conjunction with technologies in various fields (e.g., broadcasting technology, medical technology, and game technology).

Representative examples of augmented reality technology being applied to broadcasting technology include cases where a weather map changes naturally in front of a weathercaster giving a weather forecast on TV, or where advertising images that do not exist in the stadium are inserted into the screen and broadcast as if they were actually in the stadium during a sports broadcast.

In particular, these augmented reality and mixed reality technologies are being implemented and provided as various application services with the advent of smartphones.

As a representative service that provides users with augmented reality or mixed reality, there is the Metaverse. The Metaverse is a compound word of 'Meta', meaning fiction or abstraction, and 'Universe', meaning the real world, and refers to a three-dimensional virtual world.

The metaverse is a more advanced concept than the existing virtual reality environment, and provides an augmented reality environment in which the virtual world, such as the web and the Internet, is absorbed into the real world.

However, this metaverse has poor revenue-generating environments for content creators in the real world to generate new revenue because the technology for real-time content playback service that can play user content created in the real world in the virtual space of the metaverse has not been developed, and there have been problems such as poor service quality due to unsatisfactory playback of various content within the metaverse.

Therefore, in the future, there is a need to develop a reality-linked content provision device within the metaverse that can improve the new revenue-generating environment and service quality by playing real-time user content from the real space in the virtual space within the metaverse.

Republic of Korea Patent Publication No. 10-2492985 (January 25, 2023)

The purpose of the present invention is to provide a device, method, and program for providing live home shopping in a metaverse, which can improve a new profit-generating environment and service quality by granting a user in a real space the right to use a certain object in a virtual space and displaying user content on an object in the virtual space in real time.

The problems to be solved by the present invention are not limited to the problems mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the description below.

A method for providing live home shopping according to the present invention for achieving the above purpose includes the steps of receiving a request for use of a predetermined object placed in a metaverse from a user, determining the user's usage authority for the predetermined object, receiving live home shopping content from the user in real time if it is determined that the user has usage authority, and displaying the live home shopping content by overlapping it on the predetermined object.

And, the above-mentioned predetermined object may be an object having a flat surface.

Additionally, the above-mentioned predetermined object may be any one of a screen, a building exterior wall, a building interior wall, a fence, a wall, and a billboard.

And, based on the request of the user, the step of providing an item for playing live home shopping content is further included; and based on whether the user owns the item for playing live home shopping content, the presence or absence of the usage right can be determined.

In addition, the method may further include: calculating the number of avatars adjacent to the predetermined object; and providing a reward to each avatar adjacent to the predetermined object based on the number of avatars.

And, the step of calculating the number of avatars can calculate the number of avatars located within a predetermined radius centered on the predetermined object.

In addition, the method further includes a step of determining whether the live home shopping content has actually been viewed; and the number of avatars may include only avatars that have actually viewed the live home shopping content.

And, the step of determining whether or not the actual viewing has occurred may include a step of extracting the line of sight of an avatar located within a predetermined radius centered on the predetermined object; and a step of counting only avatars that include all or part of the predetermined object within the line of sight.

Additionally, the step of providing the reward may provide a reward determined based on the number of avatars to each avatar adjacent to the predetermined object.

And, the method may further include a step of calculating the number of avatars adjacent to the predetermined object; and a step of providing a reward to the user who transmitted the live home shopping content based on the number of avatars.

In addition, the method may further include: receiving a summoning request of a second avatar from a first avatar adjacent to the predetermined object; transmitting the summoning request signal to the second avatar; and moving the second avatar to a position close to the first avatar when receiving an acceptance signal for the summoning request signal from the second avatar.

In addition, the method may further include a step of providing product information of a product included in the live home shopping content; and a step of changing the product information in real time based on the number of avatars calculated in real time.

In addition, the product information may include at least one of the product's price, discount rate, and gift information.

Meanwhile, a live home shopping providing device according to the present invention for achieving the above object includes a metaverse space generating unit for generating a predetermined object for playing back video content in a metaverse space; and a reality-linked content providing unit for overlapping and displaying live home shopping content on the predetermined object based on a user request, wherein the reality-linked content providing unit determines whether the user has a usage right for the predetermined object when receiving a request for use of the predetermined object from a user, and if it is determined that the user has a usage right for the predetermined object, receives live home shopping content from the user in real time and displays the received live home shopping content in real time.

And, the above-mentioned predetermined object may be any one of a screen, a building exterior wall, a building interior wall, a fence, a wall, and a billboard.

In addition, the reality-linked content provider can provide an item for playing live home shopping content based on the user's request, and determine whether or not the user has the usage rights based on whether or not the user owns the item for playing live home shopping content.

And, the method may further include a reward providing unit that calculates the number of avatars adjacent to the given object and provides a reward to each avatar adjacent to the given object based on the number of avatars.

In addition, the method further includes an actual viewing judgment unit that extracts the line of sight of avatars located within a predetermined radius centered on the predetermined object and determines whether actual viewing has occurred based on whether all or part of the predetermined object is included within the line of sight; and the number of avatars can be calculated based on whether actual viewing has occurred.

In addition, the reward providing unit can provide a reward to the user who transmitted the live home shopping content based on the number of avatars.

In addition, the reality-linked content providing unit provides product information of a product included in the live home shopping content, and changes the product information in real time based on the number of avatars calculated in real time, and the product information may include at least one of the price, discount rate, and gift information of the product.

According to the present invention, by granting a user in a real space the right to use a given object in a virtual space and displaying user content in real time on the object in the virtual space, a new profit-generating environment and service quality can be improved.

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

Figure 1 is a schematic diagram for explaining a device (or server) according to the present invention.
Figure 2 is a block diagram for explaining the server of Figure 1.
Figure 3 is a drawing for explaining the display process of reality-linked content in the metaverse according to the present invention.
Figure 4 is a drawing for explaining the purchase list of content playback items of Figure 3.
Figures 5 to 8 are drawings for explaining the process of providing rewards for viewing reality-linked content in the metaverse according to the present invention.
Figures 9 to 12 are flowcharts for explaining a method according to the present invention.
Figure 13 is a flow chart showing another embodiment of a method according to the present invention.
FIG. 14 is a drawing for explaining a method for determining whether or not actual viewing of reality-linked content has occurred in a method and device according to the present invention.
FIG. 15 is a diagram for explaining a learning method of an AI model for determining whether or not actual viewing has occurred in a method and device according to the present invention.
FIG. 16 is a schematic diagram showing a method for selecting an avatar as a target for acquiring gaze information or user status information in a method and device according to the present invention.
FIG. 17 is a schematic diagram illustrating a method for determining whether to view content based on avatar gaze information in a method and device according to the present invention.
Figure 18 is a schematic diagram for explaining a method for determining content viewing time in a method and device according to the present invention.
FIG. 19 is a drawing for explaining a method for determining whether to view content based on screen information of a user device in a method and device according to the present invention.
FIG. 20 is a drawing for explaining a method for determining whether to view content based on user gaze information extracted using electrooculogram (EOG) in a method and device according to the present invention.
FIG. 21 is a drawing for explaining an HMD structure for extracting user gaze information and a method for determining whether to view content based on user gaze information in a method and device according to the present invention.
Figure 22 is a block diagram showing the configuration of a device (or server) according to the present invention.

The advantages and features of the present invention, and the methods for achieving them, will become clearer with reference to the embodiments described in detail below together with the accompanying drawings. However, the present disclosure is not limited to the embodiments disclosed below, but may be implemented in various different forms, and these embodiments are provided only to make the present disclosure complete and to fully inform a person skilled in the art of the scope of the present disclosure, and the present disclosure is defined only by the scope of the claims.

The terminology used herein is for the purpose of describing embodiments only and is not intended to limit the present disclosure. In the present disclosure, the singular also includes the plural unless the context clearly dictates otherwise. The terms “comprises” and/or “comprising” as used herein do not exclude the presence or addition of one or more other components in addition to the mentioned components. Like reference numerals refer to like components throughout the specification, and “and/or” includes each and every combination of one or more of the mentioned components. Although “first”, “second”, etc. are used to describe various components, these components are not limited by these terms. These terms are only used to distinguish one component from another. Therefore, it should be understood that a first component mentioned below may also be a second component within the technical spirit of the present disclosure.

Unless otherwise defined, all terms (including technical and scientific terms) used in this specification may be used with the meaning commonly understood by those skilled in the art to which this disclosure belongs. In addition, terms defined in commonly used dictionaries shall not be ideally or excessively interpreted unless explicitly specifically defined.

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the attached drawings.

Before the explanation, the meaning of terms used in this specification is briefly explained. However, since the explanation of terms is intended to help the understanding of this specification, it should be noted that they are not used in a meaning that limits the technical idea of this disclosure unless explicitly stated to limit the disclosure.

FIG. 1 is a schematic diagram for explaining a device for providing reality-linked content within a metaverse according to the present disclosure, and FIG. 2 is a block diagram for explaining the server of FIG. 1.

As illustrated in FIG. 1, a device for providing reality-linked content within a metaverse according to the present disclosure may include a server (200) and a database (220) of a metaverse service platform that are connected to a user terminal (100) via a network.

Here, the user terminal (100) may include both a stationary device (standing device) such as a personal computer (PC), a network TV, a hybrid broadcast broadband TV (HBBTV), a smart TV, an Internet protocol TV (IPTV), and the like, and a mobile device (mobile device or handheld device) such as a smart phone, a tablet PC, a notebook, a personal digital assistant (PDA), and the like.

And, the network that connects the user terminal (100) and the server (200) for communication includes both wired and wireless networks, and collectively refers to a communication network that supports various communication standards or protocols for pairing or/and data transmission and reception between the user terminal (100) and the server (300).

These wired/wireless networks include all current or future communication networks supported by the standards, and can support one or more communication protocols for them.

These wired/wireless networks can be formed by networks for wired connections, such as USB (Universal Serial Bus), CVBS (Composite Video Banking Sync), Component, S-Video (analog), DVI (Digital Visual Interface), HDMI (High Definition Multimedia Interface), RGB, and D-SUB, and by communication standards or protocols therefor, and by networks for wireless connections, such as Bluetooth, RFID (Radio Frequency Identification), IrDA (infrared Data Association), UWB (Ultra Wideband), ZigBee, DLNA (Digital Living Network Alliance), WLAN (Wireless LAN) (Wi-Fi), Wibro (Wireless broadband), Wimax (World Interoperability for Microwave Access), HSDPA (High Speed Downlink Packet Access), LTE/LTE-A (Long Term Evolution/LTE-Advanced), and Wi-Fi Direct, and by communication standards or protocols therefor.

And, as illustrated in FIG. 2, the server (200) may include a communication unit (210) that is connected to a user terminal (100) in a real space, a database (220) that stores user terminal information and user information in a real space, and a control unit (230) that displays content on a predetermined object in a virtual space of the metaverse.

Here, the control unit (230) provides a content playback item in response to a request from a user terminal (100), grants a user terminal (100) that owns the content playback item a usage right for a predetermined object in the virtual space of the metaverse, and when a content image is received from a user terminal (100) that owns the content playback item, the content image can be displayed on a predetermined object corresponding to the usage right granted to the user terminal (100).

In addition, the control unit (230) may, prior to providing content playback items, select objects capable of displaying content images among objects located within the virtual space of the metaverse, calculate the number of content playback items to be created based on the total number of selected objects, and generate content playback items corresponding to the calculated number of creations.

For example, the control unit (230) can calculate the number of content playback items to be created to be less than or equal to the total number of pre-selected objects.

The reason is to grant usage rights to one or more objects to one content playback item, thereby allowing for smooth and uniform use of the objects.

In addition, objects capable of displaying content images may include, but are not limited to, building walls, screens, billboards, and flat-surfaced fixtures located within a virtual space, but are merely examples.

Next, when the control unit (230) receives a request to acquire a content playback item from a user terminal (100), it performs authentication based on the user terminal information and user information extracted from the database (220), and when authentication is completed, it can provide the content playback item to the user terminal (100).

Here, when performing authentication, the control unit (230) requests user terminal information and user information from the user terminal if the user terminal information and user information do not exist in the database (220), and when receiving user terminal information and user information from the user terminal (100), the control unit (230) can store the user terminal information and user information in the database (220) and perform authentication.

Additionally, the control unit (230) may assign an identification code to a content playback item provided to the user terminal (100) when authentication is completed.

For example, the identification code may be generated using at least one of numbers, letters, symbols, and combinations thereof, but this is only an example and is not limited thereto.

As another embodiment, when the control unit (230) receives a content playback item purchase request from a user terminal (100), it performs authentication based on user terminal information and user information extracted from a database (220), and when authentication is completed, it provides a list of content playback items with different purchase prices to the user terminal (100), and when the user terminal (100) selects one content playback item from the list of content playback items with different purchase prices and pays the corresponding purchase price, it may provide the user terminal (100) with the completed content playback item.

Here, the control unit (230) can set a different purchase price for the content playback item depending on the usage level of the object capable of displaying the content image.

For example, the usage level of an object may be set based on at least one of the location, number, area, and usage period of the object that can display the content image.

In other words, the usage level can be higher the more the object is located in an area of virtual space with a large number of avatars, the higher the number of objects, the higher the area of the object, and the longer the usage period of the object.

In this way, the control unit (230) can provide a list of content playback items in which the purchase price of the content playback item increases as the usage right level increases.

Here, the content playback item may have a usage right level that includes at least one of the location, number, area, and usage period of an object capable of displaying the content image.

Next, the control unit (230) may grant a usage right for a certain object within the virtual space of the metaverse based on the user level of the user terminal (100) that owns the content playback item when granting the usage right.

Here, the control unit (230) can set a user level based on the frequency of metaverse visits and metaverse usage, and grant different levels of usage rights according to the set user level.

At this time, the control unit (230) can increase the level of usage rights as the user level increases.

In some cases, the control unit (230) may set a user level based on at least one of the user's social awareness, preference, likability, and trustworthiness, and grant different levels of usage rights according to the set user level.

Here, the control unit (230) grants different levels of usage rights depending on the user level when granting usage rights, and the usage rights level can be set based on at least one of the location, number, area, and usage period of an object that can display content images.

In addition, when granting a usage right, the control unit (230) may grant a usage right for a certain object within the virtual space of the metaverse based on the usage right level of the content playback item purchased by the user terminal (100).

Here, the usage right level of the content playback item purchased by the user terminal (100) can be set based on at least one of the location, number, area, and usage period of the object capable of displaying the content image.

At this time, the usage level may be higher the more the location of the object is in an area of virtual space with a large number of avatars, the higher the number of objects, the higher the area of the object, and the longer the usage period of the object.

Next, the control unit (230) can receive the content image in real time from the user terminal (100) when displaying the content image and display it by overlapping the content image on the object.

In addition, the control unit (230) can calculate the number of avatars watching the content video when the content video is displayed on the object, and provide a reward to each avatar based on the calculated number of avatars.

For example, when calculating the number of avatars, the control unit (230) can track the gaze of each avatar positioned around an object on which a content image is displayed, and if all or part of the content image is included within the gaze of the avatar, recognize the avatar as an actual viewer watching the content image, and calculate the number of avatars watching the content image by counting the number of actual viewer avatars.

As another example, when calculating the number of avatars, the control unit (230) may track the gaze of each avatar positioned around an object on which a content image is displayed, recognize an avatar as an actual viewer watching the content image if all or part of the content image is included within the gaze of the avatar, determine a reward for the content image based on the number of actual viewer avatars, and transmit the determined reward to the user terminal (100) that provided the content image.

In addition, the control unit (230) may place a tip jar around the object to induce real-time donation of goods by an avatar watching a live busking-related content video when the live busking-related content video is displayed on the object. When another avatar watching the live busking-related content approaches the tip jar, a pop-up window is displayed on the user terminal corresponding to the other avatar asking whether to donate a tip, and when a tip is donated by the other avatar, the tip may be given to the avatar that provided the live busking-related content.

In addition, the control unit (230) may calculate the number of avatars watching the live home shopping related content video when the live home shopping related content video is displayed on the object, and provide a reward to at least one of the avatars and the user terminal that is the content video provider according to the calculated number of avatars.

Additionally, the control unit (230) may generate and provide an avatar invitation option for recruiting avatars to view the content video when displaying the content video on the object.

Meanwhile, the control unit (230) may be composed of one or more cores and may include a processor for data analysis and deep learning, such as a central processing unit (CPU), a general purpose graphics processing unit (GPGPU), and a tensor processing unit (TPU) of a computing device.

Additionally, the communication unit (210) may include a wireless Internet module, a short-range communication module, a location information module, etc.

A wireless Internet module is a module for wireless Internet access, and is configured to transmit and receive wireless signals in a communication network according to wireless Internet technologies.

Examples of wireless Internet technologies include WLAN (Wireless LAN), Wi-Fi (Wireless-Fidelity), Wi-Fi (Wireless Fidelity) Direct, DLNA (Digital Living Network Alliance), WiBro (Wireless Broadband), WiMAX (World Interoperability for Microwave Access), HSDPA (High Speed Downlink Packet Access), HSUPA (High Speed Uplink Packet Access), LTE (Long Term Evolution), and LTE-A (Long Term Evolution-Advanced), and the wireless Internet module transmits and receives data according to at least one wireless Internet technology, including Internet technologies not listed above.

From the perspective that wireless Internet access through WiBro, HSDPA, HSUPA, GSM, CDMA, WCDMA, LTE, LTE-A, etc. is achieved through a mobile communication network, a wireless Internet module that performs wireless Internet access through a mobile communication network can be understood as a type of the above mobile communication module.

The short-range communication module is for short-range communication and can support short-range communication using at least one of Bluetooth™, RFID (Radio Frequency Identification), Infrared Data Association (IrDA), UWB (Ultra Wideband), ZigBee, NFC (Near Field Communication), Wi-Fi (Wireless-Fidelity), Wi-Fi Direct, and Wireless USB (Wireless Universal Serial Bus) technologies.

The location information module is a module for obtaining the location (or current location) of the server, and representative examples include the GPS (Global Positioning System) module or the WiFi (Wireless Fidelity) module.

And, the database (220) may include at least one type of storage medium among a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (for example, an SD or XD memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, and an optical disk.

The server (200) of the present disclosure may operate in connection with a web storage that performs a storage function of a database (120) on the Internet. The description of the database described above is only an example and is not limited thereto.

Additionally, the server (200) of the present disclosure may further include an output unit and an input unit.

Here, the output unit can display a user interface (UI) for providing a virtual space of the metaverse. The output unit can output any form of information generated or determined by the control unit (230) and any form of information received by the communication unit (210).

For example, the output unit may include at least one of a liquid crystal display (LCD), a thin film transistor-liquid crystal display (TFT LCD), an organic light-emitting diode (OLED), a flexible display, and a three-dimensional display (3D display). Some of these display modules may be configured as transparent or light-transmitting so that the outside may be viewed therethrough. This may be referred to as a transparent display module, and a representative example of a transparent display module is TOLED (Transparent OLED).

Additionally, the input unit can receive user input. The input unit can have keys and/or buttons on a user interface for receiving user input, or physical keys and/or buttons. A computer program according to embodiments of the present disclosure can be executed according to user input through the input unit.

In addition, the input unit may receive a signal by detecting the user's button operation or touch input, or may receive the user's voice or motion through a camera or microphone and convert it into an input signal. For this purpose, speech recognition technology or motion recognition technology may be used.

In addition, the input unit may be implemented as an external input device connected to the server (200). For example, the input device may be at least one of a touch pad, a touch pen, a keyboard, or a mouse for receiving user input, but this is only an example and is not limited thereto.

For example, the input unit can recognize a user touch input. In some cases, the input unit may have the same configuration as the output unit. The input unit may be configured with a touch screen implemented to receive a user's selection input. The touch screen may use any one of a contact-type electrostatic capacitance method, an infrared light detection method, a surface-acoustic (SAW) method, a piezoelectric method, and a resistive film method. The detailed description of the above-described touch screen is only an example according to one embodiment of the present disclosure, and various touch screen panels may be employed in the platform server (100). The input unit configured with a touch screen may include a touch sensor. The touch sensor may be configured to convert a change in pressure applied to a specific portion of the input unit or electrostatic capacitance generated in a specific portion of the input unit into an electrical input signal. The touch sensor may be configured to detect not only the position and area of the touch, but also the pressure at the time of the touch. When there is a touch input to the touch sensor, a corresponding signal(s) is sent to the touch controller. The touch controller can process the signal(s) and then transmit corresponding data to the processor, thereby enabling the processor to recognize which area of the input section has been touched, etc.

The server (200) of the present disclosure may include other configurations for performing a server environment of the server. The server (200) may include any type of device. The server (200) may be a digital device having a processor, memory, and computing power, such as a laptop computer, a notebook computer, a desktop computer, a web pad, or a mobile phone.

The server (200) of the present disclosure may be a computing system that can generate a user interface according to embodiments of the present disclosure and provide information to a user terminal via a network. The server (200) may transmit a virtual space of a metaverse space to a user terminal. In this case, the user terminal may be any type of computing device that can access the server (200). The control unit (230) of the server (200) may transmit the user interface to the user terminal (100) via the communication unit (210). The server (200) of the present disclosure may be a cloud server. The server (200) may be a web server that processes a service. The types of the server (200) described above are merely examples and are not limited thereto.

In this way, the present disclosure can improve a new revenue-generating environment and service quality by granting a user in a real space the right to use a certain object in a virtual space and displaying user content on the object in the virtual space in real time.

FIG. 3 is a diagram for explaining the display process of reality-linked content within the metaverse according to the present disclosure.

As illustrated in FIG. 3, the present disclosure can display a content image (400) on a predetermined object (500) corresponding to a usage right granted to a user terminal when a content image (400) is received from a user terminal owning a content playback item.

Here, objects (500) capable of displaying content images (400) may include building walls (510), screens, billboards (520), and flat-surfaced fixtures located within the virtual space (300), but this is only an example and is not limited thereto.

In addition, the present disclosure can receive content images in real time from a user terminal and display the content images (400) by overlapping them on an object (500) in a virtual space (300).

Here, the present disclosure calculates the number of avatars (600) watching the content image (400) when the content image (400) is displayed on the object (500), and provides a reward to each avatar (600) based on the calculated number of avatars (600).

In addition, the present disclosure can set different purchase prices for content playback items depending on the usage rights level of the object (500).

For example, the usage level may be set based on at least one of the location, number, area, and usage period of an object (500) that can display the content image (400).

That is, the usage right level may be higher the more the location of the object (500) is in an area of the virtual space (300) with a large number of avatars, the higher the number of objects (500), the higher the area of the object (500), and the higher the usage right level may be the longer the usage period of the object (500).

Figure 4 is a drawing for explaining the purchase list of content playback items of Figure 3.

As illustrated in FIG. 4, the present disclosure performs authentication based on user terminal information and user information extracted from a database when receiving a content playback item purchase request from a user terminal, and upon completion of authentication, provides a list of content playback items with different purchase prices to the user terminal, and when the user terminal selects one content playback item from the list of content playback items with different purchase prices and pays the corresponding purchase price, provides the completed content playback item to the user terminal.

Here, you can set different purchase prices for content playback items depending on the object's usage level.

For example, the usage level may be set based on at least one of the location, number, area, and usage period of objects that can display content images.

In other words, the usage level can be higher the more the object is located in an area of virtual space with a large number of avatars, the higher the number of objects, the higher the area of the object, and the longer the usage period of the object.

As shown in Figure 4, the purchase price of content playback item 1 may be higher than that of other content playback item 2 because the object's usage right level is high.

For example, in the case of content playback item 1, if the location of the object is a billboard in an area of virtual space with a large number of avatars, the number of objects is 2, the area of the object is large, and the object's usage period is preset to 2 months, and in the case of content playback item 2, if the location of the object is a building wall in an area of virtual space with a small number of avatars, the number of objects is 2, the area of the object is small, and the object's usage period is preset to 1 month, the purchase price of content playback item 1 may be higher than that of content playback item 2 because the object's usage right level may be higher.

In this way, the present disclosure can provide a list of content playback items in which the purchase price of the content playback item increases as the usage right level increases.

In some cases, the present disclosure may grant a right to use a certain object within a virtual space of the metaverse based on the user level of a user terminal owning a content playback item.

Here, the present disclosure can set a user level based on the frequency of metaverse visits and metaverse usage, and grant different levels of usage rights according to the set user level.

At this time, the present disclosure can increase the level of usage rights of an object as the user level increases.

In another case, the present disclosure may set a user level based on at least one of the user's social awareness, preference, likability, and trustworthiness, and grant different levels of usage rights according to the set user level.

In addition, the present disclosure can grant a usage right for a certain object in a virtual space of the metaverse based on the usage right level of a content playback item purchased by a user terminal.

Here, the usage right level of the content playback item purchased by the user terminal can be set based on at least one of the location, number, area, and usage period of the object capable of displaying the content image.

At this time, the usage level may be higher the more the location of the object is in an area of virtual space with a large number of avatars, the higher the number of objects, the higher the area of the object, and the longer the usage period of the object.

FIGS. 5 to 8 are diagrams for explaining a process of providing rewards for viewing reality-linked content within the metaverse according to the present disclosure.

As illustrated in FIGS. 5 to 8, the present disclosure can receive a content image in real time from a user terminal and display the content image (400) by overlapping it on an object (500).

In addition, the present disclosure can calculate the number of avatars (600) watching the content image (400) when the content image (400) is displayed on the object (500), and provide a reward to each avatar (600) based on the calculated number of avatars (600).

Here, as shown in FIG. 5 and FIG. 6, the present disclosure can provide reward information (700) around an avatar (600).

Reward information (700) may include information such as avatar nickname, number of advertisements viewed, and reward points.

For example, as shown in FIGS. 5 and 6, when calculating the number of avatars, the present disclosure tracks the gaze of each avatar (600) positioned around an object (500) on which a content image (400) is displayed, recognizes an avatar as an actual viewer watching the content image (400) if all or part of the content image (400) is included within the gaze of the avatar (600), and calculates the number of avatars (600) viewing the content image (400) by counting the number of actual viewer avatars.

In addition, the present disclosure may determine a reward for a content video based on the number of actual viewing avatars, and transmit the determined reward to a user terminal that provided the content video (400).

In addition, as shown in FIG. 7, the present disclosure can place a tip jar (800) around an object to induce real-time donation of goods by an avatar (600) watching a live busking-related content video (400) when the live busking-related content video (400) is displayed on the object.

In addition, as shown in FIG. 8, the present disclosure calculates the number of avatars (600) watching the live home shopping related content video (400) when the live home shopping related content video (400) is displayed on an object, and provides a reward to at least one of the avatars and the user terminal that is the content video provider according to the calculated number of avatars (600).

Here, the present disclosure can generate and provide an avatar invitation option and a discount coupon (900) for recruiting an avatar (600) to watch the content image (400) when displaying the content image (400) on an object.

Figures 9 to 12 are flowcharts for explaining a method according to the present invention.

As illustrated in FIG. 9, the present disclosure can provide a content playback item corresponding to a request from a user terminal in a real space (S10).

Here, the present disclosure selects objects capable of displaying content images from among objects located within a virtual space of the metaverse before providing content playback items, calculates the number of content playback items to be generated based on the total number of selected objects, and generates content playback items corresponding to the calculated number of creations, but this is only one example and is not limited thereto.

For example, objects capable of displaying content images may include, but are not limited to, building walls, screens, billboards, and flat fixtures located within a virtual space.

In addition, the present disclosure can perform authentication based on user terminal information and user information extracted from a database when receiving a request to acquire a content playback item from a user terminal, and provide a content playback item to the user terminal when authentication is completed.

Here, the present disclosure may also assign an identification code to a content playback item provided to a user terminal when authentication is completed.

For example, the identification code may be generated using at least one of numbers, letters, symbols, and combinations thereof, but this is only an example and is not limited thereto.

In addition, the present disclosure performs authentication based on user terminal information and user information extracted from a database when receiving a content playback item purchase request from a user terminal, and when authentication is completed, provides a list of content playback items with different purchase prices to the user terminal, and when the user terminal selects one content playback item from the list of content playback items with different purchase prices and pays the corresponding purchase price, provides the user terminal with the completed content playback item.

Here, the present disclosure can set a different purchase price for a content playback item depending on the usage level of an object capable of displaying a content image.

For example, the usage level of an object may be set based on at least one of the location, number, area, and usage period of the object that can display the content image.

In addition, the present disclosure can grant a user terminal owning a content playback item a right to use a predetermined object within a virtual space of the metaverse (S20).

Here, the present disclosure can grant a right to use a certain object within a virtual space of the metaverse based on the user level of a user terminal owning a content playback item.

For example, the present disclosure can set user levels based on the frequency of metaverse visits and metaverse usage, and grant different levels of usage rights according to the set user levels.

As another example, the present disclosure may set a user level based on at least one of the user's social awareness, preference, likability, and trustworthiness, and grant different levels of usage rights according to the set user level.

As another example, the present disclosure may grant a usage right to a certain object within a virtual space of the metaverse based on the usage right level of a content playback item purchased by a user terminal.

Here, the usage right level of the content playback item purchased by the user terminal can be set based on at least one of the location, number, area, and usage period of the object capable of displaying the content image.

Next, the present disclosure can receive a content video from a user terminal owning a content playback item (S30).

Next, the present disclosure can display the content image on a predetermined object corresponding to the usage right granted to the user terminal when the content image is received (S40).

Here, the present disclosure can receive a content image in real time from a user terminal and display the content image by overlapping it on an object.

And, as illustrated in FIG. 10, the present disclosure can calculate the number of avatars watching the content image when the content image is displayed on the object (S50) and provide a reward to each avatar based on the calculated number of avatars (S60).

Here, the present disclosure, as illustrated in FIG. 11, when calculating the number of avatars, tracks the gaze of each avatar positioned around an object on which a content image is displayed (S51), confirms whether all or part of the content image is included within the gaze of the avatar (S52), and if all or part of the content image is included within the gaze of the avatar, recognizes the avatar as an actual viewing avatar viewing the content image and counts the number of actual viewing avatars (S53), and calculates the number of avatars viewing the content image based on the counted number of actual viewing avatars (S54).

In some cases, the present disclosure, as illustrated in FIG. 12, when calculating the number of avatars, tracks the gaze of each avatar positioned around an object on which a content image is displayed (S51), confirms whether all or part of the content image is included within the gaze of the avatar (S52), recognizes the avatar as an actual viewing avatar viewing the content image if all or part of the content image is included within the gaze of the avatar, counts the number of actual viewing avatars (S53), calculates the number of avatars viewing the content image based on the counted number of actual viewing avatars (S54), determines a reward for the content image based on the number of avatars (S55), and transmits the determined reward to a user terminal that provided the content image (S56).

In this way, the present disclosure can improve a new revenue-generating environment and service quality by granting a user in a real space the right to use a certain object in a virtual space and displaying user content on the object in the virtual space in real time.

Figure 13 is a flow chart showing another embodiment of a method according to the present invention. At this time, the reality-linked content may be, as described above, live home shopping, live busking, etc., but is not limited thereto. Here, the reality-linked content is explained assuming live home shopping content.

First, a request for use of a predetermined object placed in the metaverse is received from a user (S110). At this time, the predetermined object is an object having a flat surface, and may be any one of a screen, a building exterior wall, a building interior wall, a fence, a wall, and a billboard, but is not limited thereto. In other words, the predetermined object may be a structure in the metaverse capable of displaying reality-linked content in the form of a media façade.

Thereafter, the user's usage rights for the given object are determined (S120). The user's usage rights for the given object may correspond to the right to provide live home shopping content within the metaverse. In other words, a user who wants to earn a certain amount of revenue through live busking content or a user who wants to gain advertising effects through live home shopping content must pay a certain amount of compensation in order to provide reality-linked content to an avatar within the metaverse space.

To this end, although not shown in Fig. 13, a step (not shown) of providing an item for playing live home shopping content based on a user's request may be further included. That is, the user purchases an item for playing live home shopping content, and the server grants the user the authority to play live home shopping content within the metaverse. That is, whether or not the user has the right to use the item can be determined based on whether or not the user owns the item for playing live home shopping content.

Thereafter, if it is determined that the user has the right to use it, live home shopping content is received from the user in real time (S130), and the content is overlapped on a predetermined object such as a wall or screen and displayed in the form of a media facade (S140).

Meanwhile, the method according to the present invention can provide a certain reward to avatars who watch live home shopping or live busking. The reward may be a product discount coupon, product coupon, point, virtual currency, item, etc., but is not limited thereto. Accordingly, the method according to the present invention determines whether or not live home shopping or live busking is actually being watched.

FIG. 14 is a diagram for explaining a method for determining whether or not to actually view reality-linked content in a method and device according to the present invention. Whether or not to actually view content can be determined by comparing an avatar's field of view based on avatar gaze information or user status information with a content transmission area. The avatar's posture, motion, and position are rendered based on user manipulation. Accordingly, the avatar's gaze information can be obtained based on the directionality of the rendered avatar. In another embodiment, when a user manipulating the avatar uses an HMD and VR gear, etc., the avatar's gaze can be the same as the user's gaze viewing the screen through the HMD. Accordingly, the avatar's gaze information can also be obtained through the user's gaze information, the HMD's screen information, etc. In addition, in one embodiment, the determination of whether or not to actually view content can be automatically made based on a learned AI model.

FIG. 15 is a diagram for explaining a learning method of an AI model for determining whether or not actual viewing has occurred in a method and device according to the present invention. The deep neural network (DNN) illustrated in FIG. 15 may mean a neural network including a plurality of hidden layers in addition to an input layer and an output layer. By using a deep neural network, latent structures of data can be identified. The deep neural network may include, but is not limited to, a convolutional neural network (CNN), a recurrent neural network (RNN), a generative adversarial network (GAN), a restricted boltzmann machine (RBM), a deep belief network (DBN), a Q network, a U network, a Siamese network, etc.

A neural network can be trained by at least one of supervised learning, unsupervised learning, and semi-supervised learning. The training of a neural network is to minimize the error of the output. In the training of a neural network, training data is repeatedly input into the neural network, the output of the neural network and the target error for the training data are calculated, and the error of the neural network is backpropagated from the output layer of the neural network to the input layer in the direction of reducing the error, thereby updating the weight of each node of the neural network. In the case of supervised learning, training data in which the correct answer is labeled for each training data is used (i.e., labeled training data), and in the case of unsupervised learning, the correct answer may not be labeled for each training data. That is, for example, in the case of supervised learning for data classification, the training data may be data in which each training data category is labeled. Labeled training data is input to the neural network, and an error can be calculated by comparing the output (category) of the neural network with the label of the training data. As another example, in the case of unsupervised learning for data classification, an error can be calculated by comparing the input training data with the output of the neural network. The calculated error is backpropagated in the neural network in the reverse direction (i.e., from the output layer to the input layer), and the connection weights of each node of each layer of the neural network can be updated according to the backpropagation. The amount of change in the connection weights of each node to be updated can be determined according to the learning rate. The calculation of the neural network for the input data and the backpropagation of the error can constitute a learning cycle (epoch). The learning rate can be applied differently depending on the number of repetitions of the learning cycle of the neural network. For example, a high learning rate can be used in the early stage of learning of the neural network to allow the neural network to quickly acquire a certain level of performance, thereby increasing efficiency, and a low learning rate can be used in the later stage of learning to increase accuracy.

In a neural network, one or more nodes connected through a link can form a relationship between input nodes and output nodes. The concept of input nodes and output nodes is relative, and any node in an output node relationship for one node can be in an input node relationship with another node, and vice versa. As described above, the input node-to-output node relationship can be created based on links. One or more output nodes can be connected to one input node through links, and vice versa. In the input node and output node relationship connected through one link, the value of the output node can be determined based on the data input to the input node. Here, the node interconnecting the input node and the output node can have a weight. The weight can be variable and can be varied by the user or the algorithm so that the neural network can perform a desired function. For example, when one or more input nodes are interconnected to one output node through each link, the output node can determine the output node value based on the values input to the input nodes connected to the output node and the weight set for the link corresponding to each input node.

The characteristics of a neural network can be determined based on the number of nodes and links in the neural network, the relationship between the nodes and links, and the weight values assigned to each link. For example, if there are two neural networks with the same number of nodes and links and different weight values between the links, the two neural networks can be recognized as different from each other. Some of the nodes constituting the neural network can form a layer based on the distances from the initial input node. For example, a set of nodes whose distance is n from the initial input node can form an n layer. The distance from the initial input node can be defined by the minimum number of links that must be passed through to reach the node from the initial input node. However, this definition of a layer is arbitrary for explanation, and the order of a layer in the neural network can be defined in a different way than described above. For example, a layer of nodes can be defined by the distance from the final output node. An initial input node can mean one or more nodes into which data is directly input without passing through links in a relationship with other nodes in the neural network. Or, in a neural network, in a relationship between nodes based on a link, it may mean nodes that do not have other input nodes connected by a link. Similarly, a final output node may mean one or more nodes that do not have an output node in a relationship with other nodes among the nodes in the neural network. In addition, a hidden node may mean nodes that constitute a neural network other than the initial input node and the final output node. A neural network according to one embodiment of the present invention may be a neural network in which the number of nodes in an input layer may be greater than the number of nodes in a hidden layer close to the output layer, and the number of nodes decreases as it progresses from the input layer to the hidden layer. The neural network may include one or more hidden layers. A hidden node in a hidden layer may have as input the output of a previous layer and the output of a surrounding hidden node. The number of hidden nodes in each hidden layer may be the same or different. The number of nodes in the input layer may be determined based on the number of data fields of the input data, and may be the same as or different from the number of hidden nodes. The input data entered into the input layer can be operated on by the hidden nodes of the hidden layer and output by the fully connected layer (FCL), which is the output layer.

Avatar gaze information, user status information, and content transmission area can be preprocessed into data in a format that can be learned by a deep learning model through a data preprocessing process. The preprocessed learning data is provided to the input layer and an AI model is created through a learning stage.

Since the content transmission area is often set from the time the metaverse space is built, if only the gaze information of adjacent avatars, user status information, and/or avatar view acquired based on them are determined, it is possible to quickly determine whether an avatar is viewing reality-linked content or not through an AI model.

Meanwhile, in another embodiment, it is possible to determine whether or not actual viewing is taking place based on avatar gaze information and/or user status information. To do this, an avatar that is to obtain gaze information or user status information must first be selected. Avatar selection can be done by extracting an avatar adjacent to a given object (an object where live home shopping or live busking content is played).

FIG. 16 is a schematic diagram showing a method for selecting an avatar as a target for acquiring gaze information or user status information in a method and device according to the present invention. The reality-linked content transmission area may be designated when designing a metaverse space, but may also be set in various locations by a platform manager, advertiser, etc. First, the reality-linked content transmission area is determined. The reality-linked content transmission area can be determined based on the location coordinates and the content transmission area within the metaverse space. Thereafter, an area within a predetermined radius from the content transmission area can be set as an avatar selection area. For example, if the coordinates of the content transmission area are {x1∼x2, y1∼y2, z1∼z2}, since the avatar is active on the floor of the metaverse space (road, ground, etc.), an area of a circle with a predetermined radius (D) centered at {(x1+x2)/2, (y1+y2)/2} can be set as an avatar selection area (PA). Of course, in other embodiments, the avatar selection area may be a rectangular area, and in still other embodiments, if the straight-line distance between the content transmission area and the avatar is within a predetermined distance, the avatar may be selected to determine whether to view the content.

Hereinafter, a method for determining whether or not content has been viewed will be described. FIG. 17 is a schematic diagram illustrating a method for determining whether or not content has been viewed based on avatar gaze information in a method and device according to the present invention.

In the present invention, the gaze of an avatar is tracked, and if a content transmission area is included within the avatar field of view based on the gaze of the avatar by a predetermined ratio or more, it is determined that the user is actually viewing the advertising content. Then, the actual viewing time is counted to derive the actual viewing time.

Judgment of whether to view content based on avatar gaze information first starts with tracking the avatar's gaze. For example, when operating an avatar using an HMD, the avatar's gaze and posture are rendered based on the viewpoint of a camera installed in front of the HMD, and the avatar's gaze can be tracked through this. Specifically, assuming the 12 o'clock direction in Fig. 17 as a reference point (reference direction), based on the posture (direction) of the rendered avatar, (a) has a gaze of 0°, (b) has a gaze of -60°, and (c) has a gaze of 150°. When the avatar's gaze is tracked, the field of view can be set based on the avatar's gaze. The avatar's field of view (θ) can have a range of ±α° from the avatar's gaze. α is a constant, can be preset by the user, and can vary based on the display angle of the HMD, the distance between the HMD and the user's eyes, etc.

And, based on the location/area of the content transmission area (S) in the metaverse space where content is played, the avatar field of view (θ), and the distance between the content transmission area (S) and the avatar (A), the overlap area (V) and/or the overlap length (L) can be extracted. The overlap area (V) means the area where the avatar field of view and the content transmission area overlap, and the overlap ratio between the avatar field of view and the content transmission area can be determined based on the overlap area. In another embodiment, in order to minimize the amount of computation, only the overlap length (L) at one point of the overlap area (V) can be calculated to determine the overlap ratio between the avatar field of view and the content transmission area.

Figure 18 is a schematic diagram for explaining a method for determining content viewing time in a method and device according to the present invention.

The determination of whether content is viewed and the content viewing time may be continuously performed while the content is played, or may be performed intermittently at preset intervals. In Fig. 18, the determination of whether content is viewed and the content viewing time is performed every 0.5 hours (30 minutes), but is not limited thereto. In the present embodiment, it is set so that the content is viewed if the overlap ratio between the avatar's field of view and the content transmission area is 60% or higher. Accordingly, the section of 0 m to 1.5 hours where the overlap ratio is 100% and the section of 1.5 to 3 hours where the overlap ratio is 67% can be determined as the user actually viewing the content, and the section of 3 to 4 hours where the overlap ratio is only 18% can be determined as the user not actually viewing the content.

The content viewing time can be determined based on the time during which all or part of the content transmission area is included within the avatar's field of view. In the above explanation, the user's content viewing time can be extracted as 3 hours when determined by the overlap ratio. If the determination period is shortened, the viewing time can be extracted more precisely. However, in another embodiment, regardless of the preset overlap ratio, if all or part of the content transmission area is included within the avatar's field of view, it can be determined that the user has actually viewed the content. In this case, since all or part of the content transmission area (S) is included within the avatar's field of view for 4 hours during which the content is played, the content viewing time can be determined as 4 hours.

FIG. 19 is a drawing for explaining a method for determining whether to view content based on screen information of a user device in a method and device according to the present invention.

The user status information received from a user device (e.g., HMD, tablet PC, laptop, etc.) may be screen information. Although the screen information of the HMD is illustrated in FIG. 19, in another embodiment, it may be screen information displayed on a screen of a laptop, screen information displayed on a touch screen of a tablet PC, screen information displayed on a computer monitor, etc. In reality, the user sees the screen displayed on the above-mentioned device. That is, when media content is included on the screen, the user sees reality-linked content, and when the reality-linked content disappears from the screen, the user no longer sees the content. Therefore, it is possible to determine what the user is currently viewing based on the screen information displayed on the user device. Ultimately, the present invention further includes a step of receiving screen information displayed on the user device, and at least one of whether or not the content has been viewed and the viewing time can be determined based on the screen information.

Meanwhile, the method according to the present invention further includes a step of receiving user gaze information of a user tracked by a user device (e.g., HMD, tablet PC, laptop, etc.), and at least one of viewing status and viewing time can be determined based on the user gaze information.

FIG. 20 is a drawing for explaining a method for determining whether to view content based on user gaze information extracted using electrooculogram (EOG) in a method and device according to the present invention.

As illustrated in FIG. 20, the user device can extract user gaze information using the user's electrooculography. The electrooculography can be divided into horizontal electrooculography, which is an electrooculography according to horizontal (left-right) movement of the eyeball, and vertical electrooculography according to vertical (up-down) movement of the eyeball. The user device includes one or more sensors for measuring electrooculography, and can extract the user's gaze information based on the electrooculography measured by the sensor. At this time, the type, shape, size, number, and position of the sensor for sensing the electrooculography can be designed in various ways.

The HMD can measure an ocular electromyography signal by a sensor placed in an area in contact with the user's face, and generates a linear coefficient vector that separates the measured ocular electromyography signal into multiple signal components. At this time, in order to generate the linear coefficient vector, the user can be requested to perform a pre-movement for detecting horizontal/vertical eye movement. The ocular electromyography signal according to a specific pattern of pre-movement can include a signal component and a noise component according to the movement of the eye. The HMD can separate the ocular electromyography signal into a signal component for horizontal movement, a signal component for vertical movement, and a noise component. At this time, the mixed signal including the signal component for horizontal movement, the signal component for vertical movement, and the noise component can be separated by using an ICA (Independent Component Analysis) algorithm or a RICA (Reconstruction Independent component analysis) algorithm that separates each signal from a mixed signal, and the linear coefficient vector corresponding to the separated signal is output and then applied to the separated signal. And, using the linear coefficient vector, the horizontal eye conduction signal and the vertical eye conduction signal are estimated from the eye conduction signal for eye movement. That is, when the linear coefficient vector is obtained according to the pre-movement of the eye, the horizontal and vertical eye conduction signals can be estimated by applying the linear coefficient vector to the eye conduction signal. The horizontal and vertical eye conduction signals estimated in this way can also be used to render the eye movement of the avatar.

Based on the user gaze information extracted using EOG, it is possible to determine whether the user is actually viewing the reality-linked content displayed on the HMD. More specifically, using EOG, it is possible to determine even cases where the user is asleep and not actually viewing the content.

FIG. 21 is a drawing for explaining an HMD structure for extracting user gaze information and a method for determining whether content has been actually viewed based on the user gaze information in a method and device according to the present invention. Specifically, (a) of FIG. 21 is a drawing showing an HMD structure for extracting user gaze information, and (b) is a drawing for explaining a method for determining whether content has been viewed based on the user gaze information.

The user gaze information can also be extracted through the user's pupil movement acquired through an internal camera. The HMD includes a cushion, lens, display, housing that come into contact with the user's face, and an eye tracking sensor (e.g., camera) for tracking the user's pupil movement. Various other sensor modules (such as inertial sensors), head tracking modules, communication modules, variable focus modules, and controllers may be provided inside and outside the housing. The eye tracking sensor may include information about the orientation of the user's eyes, that is, information about the angle of the gaze. Through the user gaze information acquired from the eye tracking sensor and the image information displayed on the HMD, the content area that the user is currently viewing can be extracted, and based on this, whether or not the user is actually viewing the reality-linked content and the actual viewing time can be determined.

Meanwhile, the method according to the present invention may include a step of calculating the number of avatars adjacent to the predetermined object, and a step of providing a reward to each avatar adjacent to the predetermined object based on the number of avatars. That is, the greater the number of avatars viewing the reality-linked content, the greater the benefit can be provided to each avatar. In addition, the greater the number of avatars viewing the reality-linked content, the more rewards can be provided to the avatars providing the corresponding content. That is, the method according to the present invention may include a step of providing a reward determined based on the number of avatars to each avatar adjacent to the predetermined object.

At this time, the number of avatars can be calculated for avatars located within a predetermined radius centered on a predetermined object, as described above. More specifically, since it is unreasonable to provide a reward simply for the fact that the content is displayed adjacent to an area where reality-linked content is displayed, the actual viewing can be determined for avatars adjacent to the predetermined object, and the number of avatars can be calculated for only avatars that actually view the content. To this end, the method according to the present invention can include a step of extracting the line of sight of avatars located within a predetermined radius centered on a predetermined object, and a step of counting only avatars that include all or part of the predetermined object within the line of sight, and only the number of avatars that actually view the content can be calculated through these two steps.

Meanwhile, for content providers, the method according to the present invention may further include a step of calculating the number of avatars adjacent to the predetermined object, and a step of providing a reward to a user who has transmitted reality-linked content based on the number of avatars. In this case, the reward may be an item granting the right to provide reality-linked content, but is not limited thereto.

As described above, depending on the number of avatars viewing the reality-linked content, the viewer and the content provider can receive a predetermined reward. Therefore, in order to summon more avatars, the method according to the present invention may further include the steps of receiving a summoning request of a second avatar from a first avatar adjacent to a predetermined object, the step of transmitting a summoning request signal to the second avatar, and the step of moving the second avatar to a position close to the first avatar when an acceptance signal for the summoning request signal is received from the second avatar. That is, by allowing the avatar summoning function, the platform provider can make more users use the metaverse space, and the content provider and the viewer can receive more rewards, thereby promoting the activation of the metaverse platform.

Meanwhile, if the reality-linked content is live home shopping, the method according to the present invention may further include a step of providing product information of a product included in the live home shopping content, and a step of changing the product information in real time based on the number of avatars calculated in real time. At this time, the product information may include at least one of the price, discount rate, and gift information of the product. That is, the number of avatars watching the live home shopping is calculated in real time, and the corresponding benefits are visibly displayed, so that users make more active efforts to recruit avatars.

If we organize the method according to the present invention from the viewpoint of rewards for content providers and viewers, first, a predetermined object for playing video content in a metaverse space is created, and then, based on a user request, reality-linked content is displayed by overlapping it on the predetermined object, and then, in the manner described above, the number of avatars viewing the reality-linked content is calculated, and a reward is provided to each avatar viewing the reality-linked content based on the number of avatars.

At this time, with respect to the content provision authority, the method according to the present invention may further include the steps of receiving a request for use of a predetermined object from a user, determining the user's authority to use the predetermined object, and receiving real-time linked content from the user if it is determined that the user has the authority to use the predetermined object. The content received in real time from the user may be displayed in the form of a media facade by overlapping the predetermined object.

In order to provide a reward, the method according to the present invention determines whether an avatar has actually viewed the reality-linked content, and the number of avatars viewing the reality-linked content is calculated based on the actual viewing, as described above.

In a first embodiment of determining whether or not actual viewing is taking place, the avatar's gaze is tracked, and if the playback area of the reality-linked content is included within the avatar's field of view based on the avatar's gaze by a predetermined percentage or more, it can be determined that the avatar is actually viewing the reality-linked content.

In a second embodiment of determining whether or not actual viewing has occurred, screen information displayed on a user's HMD (Head-Mounted Display) may be received, and whether or not actual viewing has occurred may be determined based on the screen information.

In a third embodiment of determining whether or not actual viewing has occurred, the user's gaze information tracked by the user's HMD (Head-Mounted Display) is received, and whether or not actual viewing has occurred can be determined based on the user's gaze information.

In addition to the first to third embodiments above, it may be possible to determine whether an avatar has actually viewed the reality-linked content in various ways. Meanwhile, along with determining whether or not the content has actually been viewed, a step of calculating the actual viewing time of each avatar viewing the reality-linked content may be further included.

Meanwhile, when explaining the area setting method for extracting avatars, which are the target of determining whether or not to view content, the avatar extraction area can vary depending on the area of the reality-linked content transmission area. That is, the avatar extraction area (PA) for a transmission area (CA) with a small area can be a circular area with a short radius (D) from the center of the transmission area (CA), and the avatar extraction area (PA') for a transmission area (CA') with a large area can be a circular area with a long radius (D') from the center of the transmission area (CA'). This is because in order to consider that the content has actually been viewed, an avatar must exist at a distance where the content can be properly visually recognized while being transmitted in the metaverse space. It is unreasonable to consider that all of the content has been viewed just because several contents are included with a 100% overlap ratio within the avatar field of view of an avatar at a distance. Considering this, the avatar extraction area (PA, PA') must be adjacent to the transmission area (CA, CA'), and if the avatar exists within a circle with a predetermined radius from the transmission area, the avatar can be determined to be adjacent to the transmission area. In addition, the avatar becomes a target for determining whether to view the content and a target for providing rewards.

However, this is only an example, and the avatar extraction area may take the shape of a square, hexagon, etc. rather than a circle. In another embodiment, if the straight-line distance between the content transmission area and the avatar is within a predetermined distance, the avatar may be determined to be located within the avatar extraction area.

In addition, a reward provision method based on the number of avatars in the setting area for avatar extraction is described. If the avatar extraction area is not set, even if there are more avatars gathered in a place, the number of target avatars that can actually view reality-linked content may be smaller. Therefore, the present invention can determine rewards differently based on the number of avatars included in the avatar extraction area (adjacent to the content transmission area). This is because the greater the number of avatars included in the avatar extraction area (adjacent to the content transmission area), the more cost a provider of reality-linked content such as live home shopping/live busking can pay, and thus, it is desirable for the scale of rewards to be larger.

And, the number of avatars used to determine rewards can be limited to avatars that actually view the content. Accordingly, a given reward is determined based on the number of avatars located within a given radius from the center of the content transmission area, and the number of avatars can be calculated by counting only avatars whose content transmission area is included within the avatar field of view.

Figure 22 is a block diagram showing the configuration of a device (or server) according to the present invention.

The device (1000) according to the present invention can be implemented as various devices such as a server or a terminal. The device (1000) according to the present invention can be a digital device having a processor, memory, and computing power, such as a laptop computer, a notebook computer, a desktop computer, a web pad, or a mobile phone.

In addition, the device (1000) according to the present invention may be a web server providing a metaverse platform. In addition, the device (1000) according to the present invention may be a server providing a cloud computing service. More specifically, the device (1000) according to the present invention may be a server providing a cloud computing service that processes information with another computer connected to the Internet rather than the user's computer as a type of Internet-based computing. The cloud computing service may be a service that stores data on the Internet and allows the user to use the data or programs required by the user anytime and anywhere through an Internet connection without having to install them on his or her computer. Data stored on the Internet can be easily shared and transmitted with simple manipulation and clicks. In addition, the cloud computing service may be a service that not only simply stores data on a server on the Internet, but also performs desired tasks by utilizing the functions of application programs provided on the web without having to install a separate program, and allows multiple people to share documents and work simultaneously. In addition, the cloud computing service may be implemented in the form of at least one of IaaS (Infrastructure as a Service), PaaS (Platform as a Service), SaaS (Software as a Service), a virtual machine-based cloud server, and a container-based cloud server. That is, the device (1000) according to the present invention can be implemented in the form of at least one of the above-described cloud computing services. The specific description of the above-described cloud computing services is only an example, and may include any platform that constructs the cloud computing environment of the present invention.

A device (1000) according to the present invention may include a processor (1010), a metaverse space creation unit (1020), an avatar creation unit (1030), a reality-linked content provision unit (1040), a gaze tracking unit (1050), an actual viewing judgment unit (10060), a reward provision unit (1070), a storage unit (1080), and a communication unit (1090).

The device (1000) according to the present invention performs various embodiments of the method described above, and each component has a function of performing each step of the method. However, each component illustrated in FIG. 23 is merely functionally distinguished according to the performed operation, and the processor may perform the operation of the entire component or a part of the component. In addition, two or more components may be integrated into one component, and one component may be separated into two or more components. In other words, it should be understood that the block diagram illustrated in FIG. 22 is merely schematically illustrated to easily understand the operation of the device (1000) according to the present invention, and the name, structure, shape, number, etc. of each component do not limit the scope of the rights of the present invention. Meanwhile, each component of FIG. 22 may correspond to the component of FIG. 2, and the operation of each component described below may be performed or controlled by the control unit of FIG. 2, and may be performed by the communication unit and database of FIG. 2. Likewise, it is obvious to those skilled in the art that the operations of the control unit, communication unit, and database of Fig. 2 can be performed by each configuration described below.

The processor (1010) controls the overall operation of the metaverse space creation unit (1020), the avatar creation unit (1030), the reality-linked content provision unit (1040), the eye-tracking unit (1050), the actual viewing judgment unit (10060), the reward provision unit (1070), the storage unit (1080), and the communication unit (1090). The processor (1010) can read a computer program stored in the storage unit (1080) and perform a method according to the present invention. The processor (1010) may be composed of one or more cores, and may be implemented as a central processing unit (CPU), a general purpose graphics processing unit (GPGPU), a tensor processing unit (TPU), or the like of a computing device.

The metaverse space creation unit (1020) creates various objects within the metaverse space. The metaverse space creation unit (1020) can create an object for playing back reality-linked content. The reality-linked content can be video content, and in order to play back the content, it is preferable that the object has a flat surface. Accordingly, the object for playing back the content can include a building wall, a screen, a billboard, a fixture having a flat surface, etc. located within the virtual space, but this is only an example and is not limited thereto. In other words, it is sufficient if the video content can be displayed, and it may also be an object including a curved surface.

The user account registration unit (1020) registers user information. The user information may be the user's avatar information, item information, authority information, owned assets, shares, etc. Meanwhile, the user account registration unit (1020) may further include information on the creation history of media spaces, and the user's level may be determined based on this. That is, a user who has created multiple media spaces may have their level adjusted upward, and various rewards may be granted for activities within the metaverse space.

The avatar creation unit (1030) creates an avatar that is active in the metaverse space. The avatar is active in the metaverse space by user manipulation. The avatar creation unit (1030) presents the avatar appearance (face, eyes, nose, mouth, ears, skin color, hair, body shape, hands, feet, etc.), clothing, accessories, shoes, gloves, bags, etc. to the user, and can create the appearance, appearance, posture, etc. of the avatar by combining the options selected by the user. The avatar creation unit (1030) can assign an ID that can identify the user.

The reality-linked content provider (1040) overlaps reality-linked content on a predetermined object and displays it based on a user request. The reality-linked content provider (1040) performs location selection, transmission, modification, recording, etc. for content received from a content provider. The reality-linked content provider (1040) can receive various contents from users through the communication unit (1090), and the contents may be live busking, live home shopping, etc., but are not limited thereto. The reality-linked content provider (1040) can store the contents received from users and display them at a predetermined time. In addition, the reality-linked content provider (1040) can receive contents from users in real time and display the contents through real-time streaming. The selection of a reality-linked content transmission area may be performed arbitrarily by the reality-linked content provider (1040), or may be performed at the request of a platform manager, a user, a content provider, etc.

The reality-linked content provider (1040) determines whether the user has the right to use the object for content playback. That is, the reality-linked content provider (1040) receives a request for use of a given object from a user, determines whether the user has the right to use the given object, and if it is determined that the user has the right to use the object, it receives reality-linked content from the user in real time. Meanwhile, the reality-linked content provider (1040) provides product information of a product included in a live home shopping content, and changes the product information in real time based on the number of avatars calculated in real time. At this time, the product information may include at least one of the price, discount rate, and gift information of the product, but is not limited thereto.

If the reality-linked content is live busking content, the reality-linked content provider (1040) may place a tip jar around the object to induce real-time donations of goods by avatars watching the live busking content video when the live busking content video is displayed on the object.

The eye tracking unit (1050) tracks the eye of an avatar adjacent to the transmission area at the time when the reality-linked content is transmitted. The adjacent avatar may be an avatar located within a predetermined radius centered on the transmission area. Therefore, the eye tracking unit (1050) may set an avatar extraction area in order to extract an avatar adjacent to the transmission area. The eye tracking unit (1050) may track the eye of the avatar in various ways as mentioned above, but is not limited thereto.

The actual viewing judgment unit (1060) determines at least one of whether the user corresponding to the avatar has viewed the content and the viewing time. The actual viewing judgment unit (1060) tracks the gaze of the avatar and determines whether the actual viewing has occurred based on whether the content transmission area is included in the avatar field of view based on the gaze of the avatar. Specifically, the actual viewing judgment unit (1060) obtains the avatar field of view based on the gaze of the avatar and determines whether the avatar has actually viewed the reality-linked content based on whether the avatar field of view and the transmission area overlap. The actual viewing judgment unit (1060) can obtain the gaze and avatar field of view of the avatar based on the screen information of the user's HMD (Head-Mounted Display) received through the communication unit (1090). In addition, the actual viewing judgment unit (1060) can obtain the gaze and avatar field of view of the avatar based on the user's gaze information received through the communication unit (1090). The actual viewing judgment unit (1060) calculates the overlap ratio between the avatar's field of view and the content transmission area, and can determine that the avatar is actually viewing the content only when the overlap ratio is greater than or equal to a predetermined value. When two or more contents are included in the avatar's field of view, the actual viewing judgment unit (1060) can extract the overlap ratio between the avatar's field of view and the transmission area of each content, and determine the content with the highest overlap ratio as the content that the avatar is actually viewing. The actual viewing judgment unit (1060) can extract the actual viewing time during which the avatar's field of view and the content transmission area overlap.

The reward providing unit (1070) provides a predetermined reward to an avatar depending on whether the content is actually viewed. At this time, the predetermined reward may be selected differently depending on the actual viewing time obtained by the actual viewing determining unit (1060). The reward providing unit (1070) may calculate a reward level based on a ratio of the content's playback time and viewing time, and select a predetermined reward based on the reward level. The reward providing unit (1070) may select a predetermined reward based on the number of avatars adjacent to the content transmission area. At this time, the number of avatars may be calculated by counting only avatars that actually view the content, that is, avatars whose content transmission area is included within the avatar's field of view. The reward providing unit (1070) provides a reward selected based on the actual viewing time to each avatar that views the reality-linked content. Additionally, the reward provision unit (1070) provides a reward selected based on the number of avatars viewing the reality-linked content to the user who provided the reality-linked content.

The storage (1080) can store a computer program for performing a method according to the present invention. The storage (1080) can include various information about an administrator and a user, attribute information about an object included in a metaverse space, tokenization information about an object, etc. In addition, the storage (1080) can store various contents to be transmitted on the metaverse. The storage (1080) can store a LUT for the reward level described above. The storage (1080) can store basic reward information about contents, reward setting/change information according to the number of adjacent avatars, reward setting/change information according to the content viewing time, etc.

The communication unit (1090) is connected to external devices and external servers via wired or wireless means to transmit and receive various signals and data. Here, the external devices may be HMDs, smartphones, tablet PCs, laptops, etc., but are not limited thereto. The communication unit (1080) transmits and receives data with external devices such as HMDs, smartphones, tablet PCs, etc. via wired or wireless communication networks. The communication unit (1090) can receive screen information displayed on a user device (e.g., HMD), user gaze information tracked by a user device, etc.

In addition, although not shown in the drawing, it may further include an object minting unit (not shown) that tokenizes various spaces (landmarks, roads, virtual stores, etc.) and various objects within the metaverse created at the user's request into NFTs. The object minting unit (not shown) may create NFTs by linking with an external platform, or may create NFTs within the platform itself. Meanwhile, assets within the metaverse space may be created as NFTs and made the subject of transactions, and the subject of transactions (landmarks, etc.) within the metaspace may be sectioned and the subject of transactions may be traded through online and offline payments depending on the area of the subject of transaction.

In addition, although not shown in the drawing, it may further include a virtual space composition unit (not shown), an object modeling unit (not shown), an avatar management unit (not shown), etc., for visualizing the metaverse space. The virtual space design unit (not shown) has the function of pre-dividing, dividing, and arranging the virtual metaverse space to be the subject of the transaction, and setting the purpose of each space. The user can purchase the space divided by the virtual space design unit (not shown) and use it as a landmark, road, or virtual store. The transaction subject may be assigned various values depending on the size, purpose, etc. The object modeling unit (not shown) models an object according to user input, etc., assigns attribute information, and then arranges it in the virtual space. The object modeled by the object modeling unit (not shown) may be displayed on the user terminal. The avatar management unit (not shown) may create an avatar model that will be active in the metaverse space, and present it to the user for selection. The avatar management unit (not shown) may create various objects (clothes, bags, accessories, hair, poses, etc.) related to the avatar. Alternatively, you can input various objects (clothes, bags, accessories, hair, poses, etc.) created by users and sell them on consignment within the metaverse space.

The method according to one embodiment of the present disclosure described above can be implemented as a program (or application) to be executed in combination with a hardware server and stored in a medium.

The above-described program may include codes coded in a computer language, such as C, C++, JAVA, or machine language, that can be read by the processor (CPU) of the computer through the device interface of the computer, so that the computer reads the program and executes the methods implemented as a program. Such codes may include functional codes related to functions that define functions necessary for executing the methods, and may include control codes related to execution procedures necessary for the processor of the computer to execute the functions according to a predetermined procedure. In addition, such codes may further include memory reference-related codes regarding which location (address address) of the internal or external memory of the computer should be referenced for additional information or media necessary for the processor of the computer to execute the functions. In addition, if the processor of the computer needs to communicate with another computer or server located remotely in order to execute the functions, the code may further include communication-related code regarding how to communicate with another computer or server located remotely using the communication module of the computer, what information or media to send and receive during communication, etc.

The above storage medium means a medium that stores data semi-permanently and can be read by a device, rather than a medium that stores data for a short period of time, such as a register, cache, or memory. Specifically, examples of the storage medium include, but are not limited to, ROM, RAM, CD-ROM, magnetic tape, floppy disk, or optical data storage device. That is, the program can be stored in various storage media on various servers that the computer can access or in various storage media on the user's computer. In addition, the medium can be distributed to computer systems connected to a network, so that a computer-readable code can be stored in a distributed manner.

The steps of a method or algorithm described in connection with the embodiments of the present disclosure may be implemented directly in hardware, implemented in a software module executed by hardware, or implemented by a combination of these. The software module may reside in a Random Access Memory (RAM), a Read Only Memory (ROM), an Erasable Programmable ROM (EPROM), an Electrically Erasable Programmable ROM (EEPROM), a flash memory, a hard disk, a removable disk, a CD-ROM, or any other form of computer-readable recording medium well known in the art to which the present disclosure pertains.

Although the embodiments of the present disclosure have been described above with reference to the attached drawings, those skilled in the art will appreciate that the present disclosure may be implemented in other specific forms without changing the technical spirit or essential features thereof. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive.

Claims (20)

A method performed on a computing device having one or more processors and a memory storing one or more programs executed by the one or more processors,
A step of receiving a request for use of a specific object placed within the metaverse from a user;
A step of determining the user's usage rights for the above-mentioned predetermined object;
If it is determined that the above user has usage rights, a step of receiving live home shopping content from the above user in real time;
A step of displaying the above live home shopping content by overlapping it on the above predetermined object; and
It includes a step of determining the transmission area of the live home shopping content, calculating the number of avatars adjacent to the object based on the transmission area of the live home shopping content, and providing a reward based on the number of avatars.
The transmission area of the above live home shopping content is determined based on the location coordinates of the object within the metaverse space and the transmission area of the above live home shopping content.
The steps for providing the above rewards are:
A step of setting an avatar selection area for acquiring gaze information based on the above transmission area;
A step of acquiring gaze information of avatars within the above avatar selection area and determining whether the avatars actually watch the live home shopping content based on the acquired gaze information and the transmission area; and
A method for providing live home shopping, comprising a step of calculating the number of avatars that actually viewed the above live home shopping content. delete delete In the first paragraph,
Further comprising a step of providing an item for playing live home shopping content based on the request of the user;
A method for providing live home shopping, which determines whether or not there is the right to use the above based on whether or not the item for playing the above live home shopping content is owned. In the first paragraph,
The steps for determining whether or not the above actual viewing is:
A step of inputting the gaze information of avatars adjacent to the object, the user status information of the avatars, and the content transmission area into the pre-learned AI model; and
A method for providing live home shopping, comprising a step of obtaining a result on whether or not to view the live home shopping content from the AI model. In the first paragraph,
The step of calculating the number of avatars is a live home shopping providing method that calculates the number of avatars located within a predetermined radius centered on the predetermined object. In the first paragraph,
The steps for determining whether or not the above actual viewing is:
A method for providing live home shopping, wherein the live home shopping content is determined to have been actually viewed when the transmission area is included within the avatar's field of vision by a preset percentage or more. In the first paragraph,
The steps for determining whether or not the above actual viewing is:
A method for providing live home shopping, comprising: calculating an overlap ratio between the avatar field of view and the transmission area based on the location and area of the transmission area within the metaverse space, the avatar field of view based on the avatar's gaze, and the distance between the transmission area and the avatar. In the first paragraph,
The steps for providing the above rewards are:
A method for providing live home shopping, which provides a reward determined based on the number of avatars to each avatar adjacent to the predetermined object. In the first paragraph,
A method for providing live home shopping that provides rewards to users who transmit the live home shopping content based on the number of the above avatars. In the first paragraph,
A step of receiving a summoning request of a second avatar from a first avatar adjacent to the above-mentioned predetermined object;
A step of transmitting the above summoning request signal to the second avatar; and
A method for providing live home shopping, further comprising: a step of moving the second avatar to a position close to the first avatar when receiving an acceptance signal for the summoning request signal from the second avatar. In the first paragraph,
Step of providing product information of products included in the above live home shopping content; and
A method for providing live home shopping, further comprising: a step of changing the product information in real time based on the number of avatars generated in real time. In Article 12,
A method for providing live home shopping, wherein the above product information includes at least one of the product's price, discount rate, and gift information. A metaverse space creation unit that creates a predetermined object for playing video content in the metaverse space; and
A real-time linkage content provider that displays live home shopping content by overlapping it on a predetermined object based on a user request; and
A reward providing unit is included that determines the transmission area of the live home shopping content, calculates the number of avatars adjacent to the object based on the transmission area of the live home shopping content, and provides a reward based on the number of avatars.
The above reality-linked content provider department is,
When a request for use of the above-mentioned object is received from a user, the user's usage authority for the above-mentioned object is determined, and if it is determined that the user has usage authority for the above-mentioned object, the live home shopping content is received from the user in real time, and the received live home shopping content is displayed in real time.
The above reward provider,
A live home shopping providing device that determines the transmission area of the live home shopping content based on the position coordinates of the object in the metaverse space and the transmission area of the live home shopping content, sets an avatar selection area for acquiring line-of-sight information based on the transmission area, acquires line-of-sight information targeting avatars within the avatar selection area, determines whether the avatars actually viewed the live home shopping content based on the acquired line-of-sight information and the transmission area, and calculates the number of avatars that actually viewed the live home shopping content. delete In Article 14,
The above reality-linked content provider department is,
A live home shopping providing device that provides an item for playing live home shopping content based on the request of the user, and determines whether or not the user has the right to use the item based on whether or not the user owns the item for playing live home shopping content. delete In Article 14,
The above reward provider,
A live home shopping providing device that inputs the gaze information of avatars adjacent to the object, the user status information of the avatars, and the transmission area into a pre-learned AI model, and obtains a result on whether or not the live home shopping content has been viewed from the AI model. In Article 14,
The above reward provider,
A live home shopping providing device that determines that the live home shopping content has actually been viewed when the transmission area is included within the avatar's field of vision by a preset percentage or more based on the avatar's gaze. In Article 14,
The above reward provider,
A live home shopping providing device that calculates an overlap ratio between the avatar's field of view and the transmission area based on the location and area of the transmission area within the metaverse space, the avatar's field of view based on the avatar's gaze, and the distance between the transmission area and the avatar.

Images