KR102735183B1 - Contents providing apparatus based on virtual character and providing method thereof - Google Patents

Abstract

A content providing device based on a virtual character is disclosed. The content providing device based on a virtual character of the present invention includes a display unit, a LiDAR sensor for detecting an object within a certain range in front, a storage unit for storing information on a plurality of virtual characters, and a processor for recognizing three-dimensional position coordinates of at least one person detected through the LiDAR sensor and controlling display of one of a plurality of virtual characters stored in an area corresponding to each of the recognized three-dimensional position coordinates among a virtual background space displayed through the display unit.

Description

{CONTENTS PROVIDING APPARATUS BASED ON VIRTUAL CHARACTER AND PROVIDING METHOD THEREOF}

The present invention relates to a content providing device and method based on a virtual character, and more specifically, to a content providing device and method capable of creating a virtual character matching an actual person and providing the same in real time as content in the form of animation.

Recently, the popularity and market of the Metaverse are growing due to the development of computer graphics and AR/VR technology. In particular, as the concept of the Metaverse has been highlighted due to the spread of the untact culture following the COVID-19 incident, virtual reality contents that have adopted it are attracting attention.

Metaverse is a compound word of Meta, meaning virtual and transcendent, and Universe, meaning world, and refers to a market where reality and the virtual world can interact.

Recently, various metaverse platforms have emerged where you can decorate your own character (avatar) in the game and show off your individuality. According to Statista, a global market research firm, the market size of augmented reality (AR) and virtual reality (VR), which will be the basis of the metaverse platform, is expected to grow rapidly from approximately $37.7 billion in 2021 to approximately $296.9 billion in 2024.

In addition, the character industry has been steadily increasing in size in recent years as characters have expanded from simply telling stories to industrial products that directly generate revenue, and sales have also been growing explosively as various character products are developed and sold. This seems to be due to cultural changes in which characters are evolving from simply being entertainment consumed temporarily to a means of expressing oneself.

The present invention is intended to satisfy the need to experience a new and friendly virtual world by utilizing familiar and friendly characters in the metaverse in an environment where the metaverse and a high value-added character industry are growing.

The present invention is intended to satisfy the above-described needs, and an object of the present invention is to provide virtual reality content that can arouse interest by matching user-friendly characters to people, thereby increasing the added value of characters and arousing interest in viewers.

According to an embodiment of the present invention for achieving the above-described purpose, a virtual character-based content providing device includes a display unit, a LiDAR sensor for detecting an object within a certain range in front, a storage unit for storing information on a plurality of virtual characters, and a processor for recognizing three-dimensional position coordinates of at least one person detected through the LiDAR sensor and controlling display of one of the plurality of virtual characters stored in an area corresponding to each of the recognized three-dimensional position coordinates among a virtual background space displayed through the display unit.

At this time, the processor can recognize the physical characteristics and movements of the detected person, match and display a character corresponding to the recognized physical characteristics among the plurality of virtual characters, and control the displayed character to move according to the recognized movements.

In addition, when a plurality of people are detected within a predetermined range in front, the processor can recognize the distances between the plurality of detected people, determine at least two or more people as one group based on the recognized distances, and sequentially match and display characters with the highest correlation among the plurality of virtual characters for each person belonging to the determined one group based on information about the correlation between the virtual characters previously stored in the storage unit.

Meanwhile, a method for providing content based on a virtual character according to an embodiment of the present invention includes a step of detecting an object within a certain range in front through a lidar sensor, a step of recognizing three-dimensional position coordinates of at least one person detected through the lidar sensor, and a step of displaying one of a plurality of virtual characters previously stored in an area corresponding to each of the recognized three-dimensional position coordinates among a virtual background space displayed through a display unit.

At this time, the displaying step may match a character corresponding to the recognized physical characteristic among the plurality of virtual characters based on the physical characteristic and movement recognized from the detected person, and display the matched character to move according to the recognized movement.

In addition, the displaying step may include a step of recognizing the distance between the plurality of people detected within a predetermined range in front, a step of determining at least two or more people as one group based on the recognized distance, and a step of sequentially matching and displaying characters with the highest correlation among the plurality of virtual characters for people belonging to the determined one group based on information about the correlation between the virtual characters stored in advance.

According to various embodiments of the present invention as described above, a new entertainment experience can be provided by providing virtual reality content that can arouse interest by matching user-friendly characters to people.

In order to more fully understand the drawings cited in the detailed description of the present invention, a brief description of each drawing is provided.
Figure 1 is a block diagram briefly illustrating the configuration of a content providing device according to one embodiment of the present invention.
FIG. 2 is a drawing of a 3D image of data on surrounding objects detected by a lidar sensor according to an embodiment of the present invention.
FIG. 3 is a drawing for explaining an example of operation of a content providing device according to one embodiment of the present invention.
Figure 4 is a flowchart briefly explaining a content providing method according to one embodiment of the present invention.

First, the terms used in this specification and claims are general terms selected in consideration of functions in various embodiments of the present invention. However, these terms may vary depending on the intention of a person skilled in the art, legal or technical interpretation, and the emergence of new technologies. In addition, some terms may be terms arbitrarily selected by the applicant. These terms may be interpreted as defined in this specification, and if there is no specific definition of a term, it may be interpreted based on the overall content of this specification and common technical knowledge in the relevant technical field.

In addition, the same reference numbers or symbols described in each drawing attached to this specification represent parts or components that perform substantially the same function. For convenience of description and understanding, the same reference numbers or symbols are used in different embodiments to describe them. That is, even if components having the same reference numbers are all depicted in multiple drawings, the multiple drawings do not mean one embodiment.

In addition, terms including ordinal numbers such as "first", "second", etc. may be used in this specification and claims to distinguish between components. These ordinals are used to distinguish between identical or similar components, and the meaning of the terms should not be interpreted in a limited manner due to the use of these ordinals. For example, the order of use or arrangement of components combined with these ordinals should not be interpreted in a limited manner by their numbers. If necessary, each ordinal number may be used interchangeably.

In this specification, singular expressions include plural expressions unless the context clearly indicates otherwise. In this application, it should be understood that terms such as "comprise" or "comprises" are intended to specify the presence of a feature, number, step, operation, component, part or combination thereof described in the specification, but do not exclude in advance the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts or combinations thereof.

In addition, in the embodiment of the present invention, when it is said that a part is connected to another part, this includes not only a direct connection but also an indirect connection through another medium. In addition, when it is said that a part includes a certain component, it means that it may further include other components, rather than excluding other components, unless there is a specific description to the contrary.

Hereinafter, the present invention will be described in more detail with reference to the attached drawings.

FIG. 1 is a block diagram briefly illustrating the configuration of a content providing device according to one embodiment of the present invention.

The content providing device (100) of the present invention includes a display unit (110), a lidar sensor (120), a storage unit (130), and a processor (140).

The display unit (110) outputs image data. In particular, the display unit (110) can display various image contents based on virtual characters according to the control operation of the processor (140).

The display unit (110) may include a panel for displaying an image. The panel may be any one selected from a liquid crystal display panel, an electrophoretic display panel, an organic light emitting diode panel (OLED panel), a light emitting diode panel (LED panel), an inorganic EL panel (Electro Luminescent Display Panel), a field emission display panel (FED panel), a surface-conduction electron-emitter display panel (SED panel), a plasma display panel (PDP), and a cathode ray tube (CRT) display panel.

In addition, the display unit (110) may include a plurality of display modules that can be connected to each other to form a video wall in order to be implemented in the form of a large screen indoors or outdoors, such as on an outdoor stage, auditorium, or on the street.

A LiDAR (Light Detection And Ranging) sensor (120) is a configuration that can detect the distance, direction, shape, etc. to an object by emitting a high-power laser pulse and measuring the characteristics of the signal that returns when the laser hits a target. In this respect, the LiDAR sensor (120) is also called a time-of-flight (ToF) technology, a laser scanner, or a laser radar.

The lidar sensor (120) measures the distance, width, and height information to recognize the target in three dimensions, and fires the laser by splitting it into several branches. The higher the number of laser channels divided into 15, 32, and 64, the more detailed the recognition is. The pulse width can vary from several nanoseconds to several microseconds. Pulse power, round trip time, phase shift, and pulse width are general parameters used to extract information from light signals.

As illustrated in FIG. 2, the lidar sensor (120) can create a 3D map of the surrounding terrain and objects based on the detected data, and can distinguish and detect people from this.

There are various methods for detecting people using a lidar sensor (120), such as a method using the ratio of the human body and a method for detecting legs. The method using the ratio of the body is a method for determining whether a person is a person by calculating the ratio of the width and thickness of the human body. The method for detecting legs is a method for distinguishing people by calculating the distance correlation between two legs. These methods generally consist of two steps: clustering and classification. Clustering groups the distance data obtained as an output result of the lidar sensor (120) by considering the correlation, and in this process, human characteristics such as the width and thickness of the body can be obtained from the detected data. After this, a task of classifying the human characteristics can be performed.

Learning methods can be used for classification. Representative learning methods include Bayes classifier, AdaBoost, and SVM (Support Vector Machines).

In addition, the content providing device (100) of the present invention may further include a camera (not shown) according to an embodiment, and may detect a person by fusing a lidar sensor (120) and a camera. The location of a person can be obtained by using accurate distance information and angle information acquired by the lidar sensor (120), and feature information such as color, face, and outline can be acquired through the camera to detect a person with high accuracy.

The storage unit (130) serves to store programs and data necessary for the operation of the content providing device (100). In particular, the storage unit (130) can store information such as multiple images that constitute a virtual character or background space.

Character information stored in the storage unit (130) may be classified and stored by being divided into multiple categories. Specifically, character information may be classified and stored according to a preset criterion based on at least one of the world view, gender, size, body type, age, personality, and other physical characteristics of the character. Character information may include information on various characters that are not limited to human figures, and may be information expressed as 2D characters or 3D characters.

In addition, the character information stored in the storage unit (130) may further include data that quantifies the relationships between the stored characters. Specifically, if among the characters stored in the storage unit (130), there are characters that share the same world view and characters that do not, the relationships between the characters that share the same world view will be higher than the relationships with the remaining characters.

For example, it is assumed that the characters stored in the storage (130) include Dooly, Totchi, Mycol, Pikachu, Turtle, and Iron Man. At this time, Dooly, Totchi, and Mycol are all characters appearing in the same animation 'Dooly the Little Dinosaur', so they share the same worldview (if they appear in the same content work or series, they can generally be considered to share the same worldview). Pikachu and Turtle are characters appearing in 'Pokemon', so they also share the same worldview, but their worldviews are different from Dooly, Totchi, and Mycol. Therefore, the relationship between Dooly and Totchi can be considered to be higher than the relationship between Dooly and Pikachu.

If we were to represent the character relationships as numerical data by only considering whether they share the same worldview, the correlation between Dooly and Totchi would be '1', the correlation between Dooly and Pikachu, and the correlation between Dooly and Iron Man would be '0'. Depending on the embodiment, even within the same worldview, the correlation may be set differently depending on the frequency of appearance and the role (whether they are friends or enemies, etc.).

Meanwhile, considering the character characteristics such as the size and personality of the character, the correlation between the characters can be expressed as numerical data. Since Dooly, Tochi, Pikachu, and Turtle have the same characteristics of 'small' and 'cute', and Michael and Iron Man do not have these characteristics, the correlation between Dooly and Tochi is '1.5', the correlation between Dooly and Iron Man is still '0', but the correlation between Dooly and Pikachu can be '0.5'.

In this way, the character information stored in the storage unit (130) is classified based on various characteristics between characters, and may include numerical information indicating the relationship between the characteristics. In addition, by assigning different weights to each character characteristic, the degree of association can be reflected differently.

Meanwhile, the content providing device (100) may perform separate mutual communication with an external content server by including a communication unit (not shown) in addition to the storage unit (130), thereby receiving the above-described character information.

The network may include wireless communication networks such as WLAN (Wireless LAN), Wi-Fi, Wibro, WiMAX, and HSDPA (High Speed Downlink Packet Access), and depending on the system implementation method, may also include wired communication networks such as Ethernet, xDSL (ADSL, VDSL), HFC (Hybrid Fiber Coaxial Cable), FTTC (Fiber to The Curb), and FTTH (Fiber To The Home).

A content server (not shown) can provide character information including a character image constituting a character in response to a character modeling request from a content providing device (100).

The processor (140) is a component that controls the content providing device (100) as a whole. Specifically, the processor (140) recognizes the three-dimensional position coordinates of at least one person detected through the lidar sensor (120), and can control displaying one of a plurality of virtual characters stored in the storage unit (130) in an area corresponding to each recognized three-dimensional position coordinate among the virtual background space displayed through the display unit (110).

Specifically, an example of the operation of the content providing device will be described with reference to FIG. 3.

The processor (140) can generate 3D graphics representing a virtual background space by left/right reversing and graphically processing an image captured by a camera of an area in front detected by a lidar sensor (120) equipped in some area of the content providing device (100). That is, since the area actually detected by the lidar sensor (120) in front of the content providing device (100) is processed as a virtual background space, people feel as if the background space reflected on the display unit (110) reflects their actual location like a mirror, and thus the virtual reality sensation effect can be enhanced.

The processor (140) can generate content including a virtual character and control the display unit (110) so that the virtual character is overlapped and displayed in an area corresponding to the three-dimensional position coordinates of a person recognized within a virtual background space. The virtual character is an object within the content controlled by the movement of a person, and multiple virtual character images stored in the storage unit (130) can be used.

The processor (140) can recognize the physical characteristics and movements of the detected person, match and display a character corresponding to the physical characteristics of the recognized person among a plurality of virtual characters stored in the storage unit (130), and control the displayed character to move according to the recognized movements.

Specifically, the character displayed in the virtual background space can be selected or modified according to the characteristics of the recognized person. For example, if the recognized person is recognized as being tall or big, a character with related characteristics such as a strong character, a male character, a hero character, etc. can be selected depending on the degree of recognition. On the contrary, if the recognized person is recognized as being short or small, a cute and small character, an animal character, etc. can be selected. Of course, the opposite embodiment is also possible. In this case, by matching a small or small body type with a large hero character, it is possible to provide people with different kinds of enjoyment. In addition, if the recognized person is tall or fat, the length of the character can be increased accordingly or the character can be displayed as fat.

To this end, the storage unit (130) can store the characteristic information of the detected person, such as height and body shape, and the character characteristic information in a matched and classified manner.

In addition, the processor (140) can control the character to be expressed in real time with the same animation motion according to various motions of the person recognized by the lidar sensor (120). In this case, recognition data on various motions of the person are collected through the lidar sensor (120), and video recording equipment such as a camera is installed at the same time to collect video data of the person. The two data collected through the lidar sensor (120) and the video recording equipment are matched to build a data set for analysis, and the recognition data of the lidar sensor (120), which is derived to have a significant difference for each motion of the person, is set as an input variable of a deep neural network (DNN), so that the motion characteristics of the person can be learned and estimated.

Accordingly, the processor (140) can control the display unit (110) to change and display the movement of the matching character in real time to correspond to the movement of the estimated person.

Referring to FIG. 3, characters AA, BB, CC, and DD matching A, B, C, and D can be displayed in areas corresponding to the three-dimensional position coordinates of A, B, C, and D in the virtual background space, respectively.

At this time, according to one embodiment of the present invention, the processor (140) can recognize the distance between a plurality of people detected within a certain range in front, and determine at least two or more people as one group based on the recognized distance.

Specifically, the processor (140) may group two or more people into one group when they are separated by a preset distance, since they are often close friends, colleagues, family members, etc. Depending on the embodiment, the processor (140) may group two or more people into one group only when the time that they are separated by a preset distance is longer than a preset time. Here, the preset distance may be set to various values, such as 10 cm or less or 30 cm or less.

Thereafter, the processor (140) can control the sequential matching and display of characters with the highest correlation with respect to each person belonging to a determined group.

Referring to FIG. 3, the processor (140) may determine that C and D are a single group (31) separated by a preset distance. In this case, the processor (140) may select and assign characters (32) matching C and D in order of high relevance, such as Dooly the Little Dinosaur animation characters, Douner (CC) and Ddoochi (DD), which belong to the same world view.

In this embodiment, by giving related characters to people belonging to a group, it is possible to provide a story as well as generate interest.

In addition, the processor (140) can display a virtual character (X1, X2) generated by itself moving regardless of whether a person is detected in a blank area other than the three-dimensional position coordinates where a person is detected, thereby providing an effect in which characters (AA, BB, CC, and DD) matching people (A, B, C, and D) exist together with or interact with the virtual characters (X1, X2).

Meanwhile, further, the processor (140) can match characters with higher relevance by further considering the physical characteristic information of people belonging to a group determined as one. Specifically, if the group is composed of two people with a height difference of 10 cm or more based on the recognition information on the height and body shape of people belonging to the group, the group can be determined as a lover or married couple, and a matching lover character (Beauty and the Beast, Shrek and Fiona, etc.) can be matched. In addition, if the group is composed of three people with a height difference of 30 cm or more, the group can be determined as a married couple with a child, and a matching family character (The Simpsons, The Incredibles, etc.) can be matched.

That is, the processor (140) can determine the relationship between people in the group based on the physical characteristic information of people in the group and the relationship information according to the physical characteristic information stored in the storage unit (130), and match a character (lover, family member, etc.) that matches the determined relationship.

Meanwhile, according to an embodiment, the processor (140) may estimate the gender and age group based on the walking characteristics of the person detected by the lidar sensor (120). To this end, a dataset may be constructed based on pedestrian attribute data collected using the lidar sensor (120) and video recording equipment, and various feature information such as the pedestrian's average speed, stride, speed standard deviation, and direction angle standard deviation may be derived. The feature information derived in this way may be set as an input variable of the DNN to learn and estimate the pedestrian's characteristics. The processor (140) may thereby set more detailed character matching criteria based on the estimated gender and age group.

Figure 4 is a flowchart briefly explaining a content providing method according to one embodiment of the present invention.

First, an object within a certain range in front is detected using a lidar sensor (S410).

Thereafter, the 3D location coordinates of at least one person detected through the lidar sensor are recognized (S420).

Thereafter, one of the multiple virtual characters previously stored is displayed in an area corresponding to each recognized 3D position coordinate among the virtual background space displayed through the display unit (S430).

At this time, based on the physical characteristics and motions recognized from the detected person, a character corresponding to the recognized physical characteristics among a plurality of virtual characters can be matched, and the matched character can be displayed to move according to the recognized motion.

In addition, when multiple people are detected within a certain range in front, the distances between the multiple detected people are each recognized, and at least two or more people are determined as one group based on the recognized distances, and based on information about the relationships between previously stored virtual characters, characters with the highest relationship among the multiple virtual characters can be sequentially matched and displayed for people belonging to one determined group.

According to the various embodiments of the present invention as described above, it is possible to provide a new entertainment experience by providing virtual reality content that can stimulate interest by matching user-friendly characters to people. In addition, it is possible to increase the added value of characters and contribute to the development of related industries.

Meanwhile, the content providing method according to the various embodiments described above may be implemented as a program and stored in various recording media. That is, a computer program that can be processed by various processors and execute the content providing method described above may be used in a state of being stored in a recording medium.

For example, a non-transitory computer readable medium may be provided in which a program is stored that performs the steps of: i) detecting an object within a certain range in front using a lidar sensor, ii) recognizing three-dimensional position coordinates of at least one person detected using the lidar sensor, and iii) displaying one of a plurality of virtual characters stored in advance in an area corresponding to each of the recognized three-dimensional position coordinates among a virtual background space displayed through a display unit.

A non-transitory readable medium refers to a medium that semi-permanently stores data and can be read by a device, rather than a medium that stores data for a short period of time, such as a register, cache, or memory. Specifically, the various applications or programs described above can be stored and provided on a non-transitory readable medium, such as a CD, DVD, hard disk, Blu-ray disk, USB, memory card, or ROM.

Although the preferred embodiments of the present invention have been illustrated and described above, the present invention is not limited to the specific embodiments described above, and various modifications may be made by those skilled in the art without departing from the gist of the present invention as claimed in the claims. Furthermore, such modifications should not be individually understood from the technical idea or prospect of the present invention.

100: Content provider device 110: Display unit
120: Lidar sensor 130: Storage unit
140: Processor

Claims (6)

In a content providing device based on virtual characters,
Display section;
LiDAR sensor that detects objects within a certain range ahead;
A storage unit storing information about multiple virtual characters;
A processor that recognizes the three-dimensional position coordinates of at least one person detected through the lidar sensor and controls displaying one of the plurality of stored virtual characters in an area corresponding to each of the recognized three-dimensional position coordinates among the virtual background space displayed through the display unit;
The information about the plurality of characters stored in the above storage unit further includes data that quantifies the relationship between the plurality of characters stored,
The multiple characters stored in the above storage unit are stored in multiple categories according to the numerical data of the above relationship,
The above processor,
It is characterized by recognizing the physical characteristics and movements of the detected person, matching and displaying a character corresponding to the recognized physical characteristics among the plurality of virtual characters, and controlling the displayed character to move according to the recognized movements.
The above processor,
A content providing device characterized in that, when a plurality of people are detected within a predetermined range in front, the distance between the plurality of detected people is respectively recognized, and at least two or more people are determined as one group based on the recognized distance, and, based on the data in which the correlations between the virtual characters are digitized and stored in the storage unit, the content providing device sequentially matches and displays characters belonging to one of a plurality of classifications divided according to the digitized correlation data among the plurality of characters for people belonging to the determined one group.
delete delete In a method for providing content based on virtual characters,
A step of detecting an object within a certain range ahead using a lidar sensor;
A step of recognizing the three-dimensional position coordinates of at least one person detected through the above lidar sensor; and
A step of displaying one of a plurality of virtual characters stored in advance in an area corresponding to each of the recognized three-dimensional position coordinates among the virtual background space displayed through the display unit; including;
Information about multiple characters further includes data that quantifies the relationships between the multiple characters to be stored, and the multiple characters that have already been stored are classified into multiple categories and stored according to the quantified data of the relationships.
The steps indicated above are:
Based on the physical characteristics and movements recognized from the detected person, a character corresponding to the recognized physical characteristics among the plurality of virtual characters is matched, and the matched character is displayed to move according to the recognized movements.
The steps indicated above are:
When multiple people are detected within a predetermined range in front, a step of recognizing the distance between each of the multiple people detected;
A step of determining at least two or more people as one group based on the recognized distance; and
A content providing method characterized by comprising: a step of sequentially matching and displaying characters belonging to one of a plurality of classifications divided according to numerical data of the relationships among the plurality of virtual characters, for each person belonging to the determined group, based on information about the relationships between the virtual characters that have been previously stored; delete delete

Images