CN113728621A - Head-mounted information processing device - Google Patents
Head-mounted information processing device Download PDFInfo
- Publication number
- CN113728621A CN113728621A CN201980095249.6A CN201980095249A CN113728621A CN 113728621 A CN113728621 A CN 113728621A CN 201980095249 A CN201980095249 A CN 201980095249A CN 113728621 A CN113728621 A CN 113728621A
- Authority
- CN
- China
- Prior art keywords
- content
- key
- head
- registration
- information processing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/14—Digital output to display device ; Cooperation and interconnection of the display device with other functional units
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/18—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
- H04N7/183—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast for receiving images from a single remote source
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
- G06F3/012—Head tracking input arrangements
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
- G06F3/015—Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/14—Digital output to display device ; Cooperation and interconnection of the display device with other functional units
- G06F3/147—Digital output to display device ; Cooperation and interconnection of the display device with other functional units using display panels
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
- G09G5/00—Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators
- G09G5/36—Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators characterised by the display of a graphic pattern, e.g. using an all-points-addressable [APA] memory
- G09G5/39—Control of the bit-mapped memory
- G09G5/393—Arrangements for updating the contents of the bit-mapped memory
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
- G09G5/00—Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators
- G09G5/36—Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators characterised by the display of a graphic pattern, e.g. using an all-points-addressable [APA] memory
- G09G5/39—Control of the bit-mapped memory
- G09G5/395—Arrangements specially adapted for transferring the contents of the bit-mapped memory to the screen
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/66—Transforming electric information into light information
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/18—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
- G09G2354/00—Aspects of interface with display user
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Computer Hardware Design (AREA)
- Biomedical Technology (AREA)
- Health & Medical Sciences (AREA)
- Dermatology (AREA)
- General Health & Medical Sciences (AREA)
- Neurology (AREA)
- Neurosurgery (AREA)
- User Interface Of Digital Computer (AREA)
- Controls And Circuits For Display Device (AREA)
- Processing Or Creating Images (AREA)
- Position Input By Displaying (AREA)
- Transforming Electric Information Into Light Information (AREA)
Abstract
提供一种能够提高用户的便利性的头戴式信息处理装置。因此，密钥内容对信息登记部(211)在接收到来自用户的指示时或者装置停止时即事件发生时，将事件发生时的内容的状态与从事件发生时的摄像机(111)的图像得到的登记密钥对应起来，作为密钥内容对信息(216)保存于存储器(128)。密钥内容对恢复部(212)在装置启动时基于密钥内容对信息(216)判断装置启动时的摄像机(111)的图像与登记密钥是否匹配，在匹配的情况下，恢复与匹配的登记密钥相对应的内容的状态。
Provided is a head-mounted information processing device capable of improving user convenience. Therefore, the key content pair information registration unit (211) obtains the state of the content at the time of the event and the image of the camera (111) at the time of the event when an instruction from the user is received or when the device is stopped, that is, when an event occurs. The registered key of , and stored in the memory (128) as the key content pair information (216). The key content pair recovery unit (212) determines whether the image of the camera (111) at the time of device startup matches the registration key based on the key content pair information (216) at the time of device startup, and if it matches, recovers the matching key. The state of the content corresponding to the registration key.
Description
Technical Field
The present invention relates to a head-mounted information processing device that provides various information to a user visually or audibly while being worn on the head of the user.
Background
Documents of the prior art
Patent document
Patent document 1: japanese patent laid-open publication No. 2018-129732
Disclosure of Invention
Problems to be solved by the invention
The Head Mounted information processing apparatus is also called an HMD (Head Mounted Display), and can Display Augmented Reality (AR) information and Virtual Reality (VR) information on a Display unit while being worn on the Head of a user. In recent years, in a head-mounted information processing device, a real world and a virtual world are seamlessly integrated in real time, and a user can experience as if a virtual object exists on the spot. Such a head-mounted information processing apparatus is typified by content using AR or VR, and many contents are mounted thereon.
When a user selects various contents to be loaded on the head-mounted information processing device, the user needs to perform a selection operation using a remote controller or the like while viewing a menu or the like displayed on the display unit after wearing the head-mounted information processing device. As a result, it is necessary to search for the remote controller and perform the selection operation with time, and thus the user's convenience may be reduced. In particular, in the case where the hierarchy of the menu is deep, the contents of the object may need to be searched, and thus the convenience is further degraded. In addition, when such a selection operation is performed, the user may be less immersed in the content due to the behavior itself.
The present invention has been made in view of the above circumstances, and an object thereof is to provide a head-mounted information processing apparatus capable of improving user convenience.
The above object as well as other objects and novel features of the present invention will become apparent from the description of the present specification and the accompanying drawings.
Means for solving the problems
A brief description will be given of an outline of a typical embodiment of the invention disclosed in the present application, as follows.
A typical head-mounted information processing device is a device that provides various kinds of information to a user visually or audibly while being worn on the head of the user, and includes a display unit that displays a predetermined image, a camera that captures an image outside the device, a memory, a content execution unit that executes content, and a content control unit that controls the content execution unit. The content control unit includes a key content pair information registration unit and a key content pair recovery unit. The key content pair information registering unit stores, in the memory, the key content pair information in which the state of the content at the time of occurrence of an event when an instruction from a user is received or when the apparatus is stopped is associated with a registration key obtained from an image of the camera at the time of occurrence of the event. When the device is started, the key content pair recovery unit determines whether or not there is a match between the image of the camera and the registration key at the time of starting the device, based on the key content pair information, and if there is a match, recovers the state of the content corresponding to the matched registration key.
ADVANTAGEOUS EFFECTS OF INVENTION
In brief, by explaining the effects obtained by the representative embodiments of the invention disclosed in the present application, it is possible to improve the convenience of the user in the head-mounted information processing apparatus.
Drawings
Fig. 1 is a block diagram showing a schematic configuration example of a head-mounted information processing apparatus according to embodiment 1 of the present invention.
Fig. 2 is a block diagram showing a configuration example of a main part around the CPU in fig. 1.
Fig. 3 is a flowchart showing an example of the operation of the key content pair information registration unit in fig. 2.
Fig. 4 is a flowchart showing an example of the operation of the key content pair restoring unit in fig. 2.
Fig. 5 is a schematic diagram showing an example of the configuration of the key content pair information in fig. 2.
Fig. 6 (a), 6 (b), and 6 (c) are schematic diagrams showing an example of the usage status of the head-mounted information processing apparatus corresponding to fig. 5.
Fig. 7 (a), 7 (b), and 7 (c) are schematic views showing another example of the usage status of the head-mounted information processing apparatus corresponding to fig. 5.
Fig. 8 (a) and 8 (b) are schematic diagrams showing another configuration example of the key content pair information in fig. 2.
Fig. 9 (a) and 9 (b) are diagrams illustrating examples of use of the registration key by the user in fig. 2.
Fig. 10 (a) is a diagram illustrating another use example of the user pair registration key in fig. 2, and fig. 10 (b) is a schematic diagram showing a configuration example of key content pair information corresponding to fig. 10 (a).
Fig. 11 is a diagram for explaining an example of the operation of the key content pair restoration unit in fig. 2 in the head-mounted information processing device according to embodiment 2 of the present invention.
Fig. 12 (a), (b), (c), and (d) are diagrams illustrating an example of the operation of the key content pair restoration unit in fig. 2 in the head-mounted information processing device according to embodiment 3 of the present invention.
Fig. 13 (a), (b), (c), and (d) are diagrams illustrating another operation example of the key content pair restoration unit in fig. 2 in the head-mounted information processing device according to embodiment 3 of the present invention.
Fig. 14 is a block diagram showing a schematic configuration example around a head-mounted information processing apparatus according to embodiment 5 of the present invention.
Description of the reference numerals
100: a head mounted information processing device (HMD); 111: a camera; 112: a right eye sight line detector; 113: a left eye sight line detector; 114: a vibration generator; 115: a stimulus generator; 116: a peripheral sound microphone; 117: a sound producing microphone; 118: an earphone; 119: an operation input interface; 120: a display (display unit); 121: a depth sensor; 122: an acceleration sensor; 123: a gyroscope sensor; 124: a geomagnetic sensor; 125: a CPU; 126: carrying out a procedure; 127: information data; 128: a memory; 129: a biosensor; 132: a non-contact sensor; 140: a bus; 150: an antenna; 151: a communication interface; 201: various sensors; 205: a content control unit; 206: a content execution unit; 210: a candidate key extraction unit; 211: a key content pair information registration unit; 212: a key content pair recovery unit; 215: key setting information; 216: key content pair information; 601: a user; 602: a clock; 603: a calendar; 604: an action-rendering content; 605: making content by a document; 902: a server device; 903: a communication interface; 904: an antenna; 905: a CPU; 906: carrying out a procedure; 907: information data; 908: a memory; 910: a communication network; 920: a bus; 1001. 1002: an access point device.
Detailed Description
Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In all the drawings for describing the embodiments, the same members are denoted by the same reference numerals in principle, and redundant description thereof is omitted.
(embodiment mode 1)
Outline of head-mounted information processing device
Fig. 1 is a block diagram showing a schematic configuration example of a head-mounted information processing apparatus according to embodiment 1 of the present invention. The head-mounted information processing device 100 of fig. 1 provides various information to a user by visual or auditory sense while being worn on the head of the user. In the specification, the head-mounted information processing apparatus 100 is also referred to as an HMD 100. The HMD100 includes a camera 111, an external sound microphone 116, a sound-producing microphone 117, headphones 118, an operation input interface 119, a display (display unit) 120, a cpu (central Processing unit)125, a memory 128, an antenna 150, and a communication interface 151.
The HMD100 further includes a vibration generator 114, a stimulus generator 115, and various sensors. The various sensors include a right-eye line-of-sight detector 112, a left-eye line-of-sight detector 113, a noncontact sensor 132, a depth sensor 121, an acceleration sensor 122, a gyro sensor 123, a geomagnetic sensor 124, a biosensor 129, and the like. These components are connected to each other via a bus 140.
The display (display unit) 120 is, for example, a liquid crystal panel or the like, and displays a predetermined image, typically an AR or VR image or the like. The camera 111 is provided, for example, in front of both eyes of the user, and captures an image outside the device (for example, a landscape in front of the user). The captured image is displayed on the display 120 as an image in real space. The right-eye line-of-sight sensor 112 and the left-eye line-of-sight sensor 113 detect the lines of sight of the right eye and the left eye, respectively. A technique of detecting the line of sight of a user is generally known as eye tracking. For example, in eye tracking using corneal reflection, an infrared LED (light Emitting diode) is irradiated on a face and an infrared camera is used to take an image, and a position on a cornea of reflected light (corneal reflection) generated by irradiation of the infrared LED is used as a reference point, and a line of sight is detected based on a position of a pupil relative to a position of the corneal reflection.
The vibration generator 114 generates vibration in accordance with an instruction from the CPU 125. The vibration generator 114 converts notification information to the user generated by the CPU125 into vibration, for example, thereby notifying the user of a higher degree of recognition. The stimulus generator 115 generates a predetermined stimulus, for example, as represented by an electrical stimulus or the like. The peripheral sound microphone 116 is worn, for example, near the ear of the user, and collects peripheral sound generated outside the device and converts the sound into a sound signal. The sound emitting microphone 117 collects sound emitted from the user and converts it into a sound signal. The earphone 118 converts sound signals inside the device into sound (sound waves) and plays them to the user. The earphone 118 may be of the air conduction type or the bone conduction (bone conduction) type.
The operation input interface 119 is typically a remote controller or the like, and receives various operation input signals input by a user. The operation input interface 119 may also be capable of receiving various operation input signals via the communication interface 151. The proximity sensor 132 determines the distance by, for example, the time when the irradiated infrared ray is reflected by the object and returns, or determines the distance to the object by electromagnetic induction with the object or a change in capacitance with the object, and detects the object approaching the HMD 100. The depth sensor 121 measures the distance to the object in a planar manner. The depth sensor unit 121 is, for example, a sensor using reflection of infrared rays, laser light, or the like, but may be implemented in other ways, such as obtaining distance information from parallax of images captured by a plurality of cameras at different mounting positions.
The acceleration sensor 122 detects the acceleration of the HMD100, thereby detecting the motion, vibration, impact, and the like of the HMD 100. The gyro sensor 123 detects the longitudinal, lateral, and oblique postures of the HMD100 by detecting the angular velocity in the rotational direction. The geomagnetic sensor 124 detects the magnetic force of the earth, thereby detecting the direction in which the HMD100 is facing. Therefore, by using the gyro sensor 123 or the geomagnetic sensor 124, and further using the acceleration sensor 122 in combination as the case may be, the posture of the HMD100 (and the user wearing the HMD 100) can be detected. The biometric sensor 129 detects biometric information of a user represented by, for example, an iris or a retina, and thereby determines each user.
The memory 128 is, for example, a flash memory, an operating RAM, or the like, and stores the program 126 and various information data 127. The program 126 includes an OS (Operating System) and various content programs that operate on the OS. The information data 127 includes various parameters and the like used in the program 126. The CPU125 executes the program 126 stored in the memory 128, thereby providing a user with desired content and controlling the operation of the entire HMD 100. The CPU125 may include various dedicated processors for arithmetic Processing such as a GPU (Graphics Processing Unit).
The communication interface 151 performs short-range wireless communication via the antenna 150, for example. Examples of short-range wireless communication methods include an electronic tag, Bluetooth (registered trademark), IrDA (Infrared Data Association), Zigbee (registered trademark), HomeRF (registered trademark), and wireless LAN (ieee802.11a, ieee802.11b, ieee802.11g).
Details of the major part around the CPU
Fig. 2 is a block diagram showing a configuration example of a main part around the CPU in fig. 1. In fig. 2, the CPU125 includes a content control unit 205 and a content execution unit 206. The CPU125 executes the program 126 stored in the memory 128, and the content control unit 205 and the content execution unit 206 are installed on the CPU 125. The content execution unit 206 executes various contents in response to a request from a user or the like. The content control unit 205 controls the content execution unit 206. The content control unit 205 and the content execution unit 206 are not limited to program processing (software) performed by the CPU125, and may be installed in hardware such as an FPGA (Field Programmable Gate Array) or various accelerators, or a combination of hardware and software, depending on the case.
The content control unit 205 includes a candidate key extraction unit 210, a key content pair information registration unit 211, and a key content pair restoration unit 212. The candidate key extraction unit 210 extracts a candidate key based on at least one of the image of the camera 111, the detection results of the various sensors 201, and the reception data of the communication interface 151. At this time, which of the image of the camera 111, the detection result of the various sensors 201, and the reception data of the communication interface 151 is to be used as a candidate key is determined in advance by the key setting information 215 in the memory 128. The various sensors include a right-eye visual-line detector 112, a left-eye visual-line detector 113, a noncontact sensor 132, a depth sensor 121, an acceleration sensor 122, a gyro sensor 123, a geomagnetic sensor 124, a biosensor 129, and the like.
When an event occurs, the key content pair information registration unit 211 causes the candidate key extraction unit 210 to extract candidate keys, and specifies a key at the time of occurrence of the event from the candidate keys. Taking a camera image as an example, the candidate key corresponds to, for example, a single or a plurality of objects included in the camera image. The key content pair information registration section 211 determines 1 object (1 key of the single or multiple candidate keys) of the single or multiple objects as a key at the time of the event occurrence.
Then, the key content pair information registration unit 211 associates the state of the content execution unit 206 at the time of the event occurrence with the key at the time of the event occurrence, and stores the associated state as key content pair information 216 in the memory 128. In the specification, a key stored as the key content pair information 216 when the event specified by the key content pair information registration unit 211 occurs is referred to as a registration key. The event occurrence time is when the device stop signal (for example, a power off signal or the like) POFF is received, that is, when the device is stopped, or when a predetermined instruction is received from the user by the user input signal CMD.
On the other hand, the key content pair recovery unit 212 causes the candidate key extraction unit 210 to extract the candidate key when the PON receives an apparatus start signal (e.g., a power-on signal) that is the apparatus start-up. Then, the key content pair recovery unit 212 determines whether or not the extracted candidate key matches the registration key based on the key content pair information 216, and if there is a match, recovers the state of the content corresponding to the registration key.
Here, when the device of the HMD100 is stopped, typical examples include (a) when the user takes off the HMD100 from the head, (B) when the power supply is turned off in a state where the user has worn the HMD100 on the head, and (C) when the user takes off the HMD100 from the head, places the HMD100 somewhere and turns off the power supply. The power-off does not necessarily mean the power-off, and may mean the transition to the sleep mode. (A) For example, by using a non-contact sensor on the face side of the HMD 100. When the HMD100 is detached from the head, the HMD100 may automatically turn off the power supply or automatically shift to the sleep mode. (C) Unlike the case of (a), the subject is the HMD100 that does not automatically turn off the power supply (or does not shift to the sleep mode).
The user input signal CMD is issued, for example, in accordance with an interrupt button of a remote controller or the like or an interrupt instruction by a menu on the screen or the like, and functions as a bookmark, guide, restore point, for example. In addition, the state of the content indicates the category of the content, or indicates the execution state on the content in addition to the category of the content. The category of the content is information for distinguishing between action playback content such as a movie, game content such as role playing, experience content such as sightseeing, document production content, chat room content, and the like, for example. In this case, the key content pair information registration unit 211 may store, for example, a file path to the content as the state of the content in the key content pair information 216. The key content pair recovery unit 212 may start the content based on the file path or the like.
On the other hand, the execution state on the content indicates, for example, a reproduction position on the action reproduction content, game content, a progress position on the experience content, a document file name on the document production content, an environment of a participant in the chat room content, and the like. In this case, the key content pair information registration unit 211 may appropriately store, as the state of the content in the key content pair information 216, for example, a file path to the content, a file path to various setting files, a file path to a document file, or a parameter value (register value, counter value) of the CPU. The key content pair restoring unit 212 may restore the execution state on the content in addition to the startup content, based on the stored various information.
Details of the Key Contents Pair information registration section and Key Contents Pair recovery section
Fig. 3 is a flowchart showing an example of the operation of the key content pair information registration unit in fig. 2. Fig. 4 is a flowchart showing an example of the operation of the key content pair restoring unit in fig. 2. Here, a case where the key (candidate key and registration key) is the image of the camera 111 will be described as an example. In fig. 3, the key content pair information registration unit 211 executes the process of step S102 upon receiving an event signal (the apparatus stop signal POFF or the user input signal CMD) (step S101). In step S102, the key content pair information registration unit 211 acquires the state (e) of the content at the time of occurrence of the event from the content execution unit 206.
Next, the key content pair information registration section 211 causes the candidate key extraction section 210 to extract a candidate key based on the image of the camera 111 at the time of the occurrence of the event, and determines the registration key (E) therefrom (step S103). Next, the key content pair information registration unit 211 associates the state (E) of the content acquired in step S102 with the registration key (E) specified in step S103, stores the state (E) in the memory 128 as the key content pair information 216 (step S104), and ends the processing.
On the other hand, in fig. 4, the key content pair recovery unit 212 executes the processing of step S202 upon receiving the device activation signal (step S201). In step S202, the key content pair recovery unit 212 determines whether or not the key content pair information 216 is stored in the memory 128. When the key content pair information 216 is not stored, the key content pair restoring unit 212 displays a predetermined initial screen (for example, a menu screen) after the device is started on the display 120 (step S211), and ends the process.
On the other hand, when the key content pair information 216 is stored in step S202, the key content pair recovery unit 212 causes the candidate key extraction unit 210 to extract the candidate key (E') based on the image of the camera 111 at the time of device activation (step S203). Next, the key content pair restoring section 212 sequentially reads the registration key (E) within the key content pair information 216 from the memory 128 (step S204), and determines whether there is a match of the registration key (E) with the candidate key (E') extracted in step S203 (in other words, the camera image) (step S205).
If there is no match in step S205, the key content pair restoration unit 212 displays a predetermined initial screen in step S211, and ends the process. On the other hand, in the case where there is a match in step S205, the key content pair recovery section 212 acquires the state of the content corresponding to the matched registration key (E) based on the key content pair information 216 (step S206). Then, the key content pair restoring unit 212 starts the corresponding content based on the device stop time setting and the initial setting (steps S207 and S208) (step S209), or restores the execution state of the content based on the start time setting and the initial setting (step S210), and ends the processing.
In this example, it is possible for the user to arbitrarily select whether to resume the category of content described in the description of fig. 2 or resume the execution state on the content when the apparatus is stopped (step S207), or to fixedly determine in the initial setting for each content (step S208). Information on the stop of these devices or information on the initial setting is stored in the memory 128. For example, in the case of video reproduction content, the user can arbitrarily select whether to reproduce the video from the initial reproduction or the interrupted position when the apparatus is stopped, and can also be fixedly determined in the initial setting. The determination method is not particularly limited to user setting or initial setting when the apparatus is stopped, and various methods can be used.
Fig. 5 is a schematic diagram showing an example of the configuration of the key content pair information in fig. 2. Fig. 6 (a), 6 (b), and 6 (c) are schematic diagrams showing an example of the usage status of the head-mounted information processing device corresponding to fig. 5, and fig. 7 (a), 7 (b), and 7 (c) are schematic diagrams showing another example of the usage status of the head-mounted information processing device corresponding to fig. 5.
In fig. 6 (a), the user 601 wearing the HMD100 enjoys the movie "mountain climbing" using the video playback content 604 of the HMD100 as shown in fig. 6 (c), and during this playback time "1: 10: 15 "toward the clock 602 while indicating to the HMD100 that the device is stopped (e.g., power is off). At this time, as shown in fig. 6 (b), the camera 111 captures a clock 602. On the other hand, in fig. 7 (a), the user 601 wearing the HMD100 performs document creation using the document creation content 605 of the HMD100, and instructs the HMD100 to stop the apparatus while heading toward the calendar 603 after storing the document file. At this time, as shown in fig. 7 (b), the camera 111 photographs the calendar 603.
As a result, the key content pair information 216a in fig. 5 indicates the reproduction time "1: 10: 15 "is associated with the clock 602 as the registration key and stored. In addition, the key content pair information 216a in fig. 5 stores a document file on the document creation content 605 as a state of content (execution state on content) in association with the calendar 603 as a registration key.
Thus, when the user 601 wears the HMD100 later and instructs the apparatus to start (e.g., turns on the power) while heading in the direction of the clock 602, the HMD100 changes the display mode from the reproduction time "1: 10: 15 "start reproducing the movie" mountain climbing "automatically. As a result, the user 601 can resume movie viewing immediately before without performing a special selection operation. In addition, when the user 601 wears the HMD100 and instructs the apparatus to start while heading in the direction of the calendar 603, the HMD100 automatically starts a predetermined document file on the document production content 605. As a result, the user 601 can create a document following the previous resume without performing a special selection operation.
Note that, here, for simplicity of explanation, the clock 602 and the calendar 603 are specified as the registration key, but in practice, for example, a sofa provided in a living room, a bookshelf provided in a study room, or the like may be specified as the registration key. As a specific example of use, a case is assumed where a user enjoys a movie in the middle of enjoying the movie in a living room using the HMD100, and then document creation is performed in the middle of producing a document in a study using the HMD 100. In this case, when the user starts the HMD100 while watching a sofa in the living room, the subsequent content of the movie is automatically reproduced, and when the HMD100 is started while watching a bookshelf in the study, an environment in which the subsequent document creation is possible is automatically created.
Here, as the registration key specified by the key content pair information registration unit 211, there may be mentioned, for example, a method of directly using the entire image of the camera 111, a method of detecting 1 object included in the center of the image of the camera 111, and the like. The latter method is used in the examples of fig. 5, 6, and 7. In the latter method, the key content pair information registration unit 211 causes the candidate key extraction unit 210 to extract a single object or a plurality of objects as candidate keys from the entire camera image when the apparatus is stopped, for example. Then, the key content pair information registration unit 211 determines 1 object included in the center portion of the camera image from the extracted single or plurality of objects (candidate keys) as a registration key and stores it as the key content pair information 216.
At this time, the candidate key extraction unit 210 extracts information (for example, the shape, the type, the feature points, and the like) of a single object or a plurality of objects included in the entire camera image by performing image processing typified by edge detection or corner detection, image processing using Artificial Intelligence (AI), or the like. The information of 1 object in the extracted information (shape, type, feature point, etc.) of the single or multiple objects becomes a registration key. In the latter method, the key content pair recovery unit 212 causes the candidate key extraction unit 210 to extract a single or a plurality of candidate keys (information of the target in this example) from the entire camera image at the time of device startup, for example. Then, for example, when the extracted single or plural candidate keys (in other words, camera images) include the registration key, the key content is determined as a match to the restoring unit 212, and when the extracted single or plural candidate keys do not include the registration key, the key content is determined as a non-match.
On the other hand, in the former method, the key content pair restoration unit 212 compares the entire camera image stored as the registration key with the entire camera image captured at the time of device startup (in other words, 1 candidate key), for example, and determines whether there is a match between the candidate key and the registration key based on whether or not the matching rate is equal to or greater than a reference value. In this case, although the processing can be facilitated, the accuracy of the determination result of the presence or absence of matching may be lowered particularly by the influence of the scale of the camera image. Therefore, from the viewpoint of the accuracy of the determination result of the presence or absence of matching, it is desirable to use the latter method.
Various modifications of the Key
Fig. 8 (a) and 8 (b) are schematic diagrams showing another configuration example of the key content pair information in fig. 2. Fig. 8 (a) shows an example in which detection results of various sensors 201 of fig. 2 are used as a registration key instead of an image of the camera 111. Fig. 8 (b) shows an example of using a combination of the image of the camera 111 and the detection results of the various sensors 201 as the registration key.
In the key content pair information 216b shown in fig. 8 (a), the HMD100 that can be powered on/off with the key removed from the head is taken as an example, and the registration key is set to the orientation of the HMD100 (for example, left behind, right side down, etc.) based on the key setting information 215 of fig. 2. For example, if the user turns on the power supply while the HMD100 is turned back on, the HMD100 automatically reproduces the movie "mountain climbing". The orientation of the HMD100 is detected by, for example, the gyro sensor 123, the geomagnetic sensor 124, and the like, and a sensor output value and the like as a detection result thereof become keys (a candidate key and a registration key).
In the key content pair information 216c shown in fig. 8 (b), the registration key is set as a combination of the pose (e.g., left-out) of the HMD100 and the camera image (e.g., clock). For example, when the user turns the HMD100 upside down and turns on the power supply by pointing the direction of the camera 111 toward the clock in this state, the HMD100 automatically reproduces the movie "mountain climbing". On the other hand, when the user turns the HMD100 right side down, and in this state, turns the camera 111 toward the clock and turns the power on, the HMD100 automatically starts up the document file. In this manner, by using a combination of the camera image and the sensor detection result, the number of registration keys that can be used can be increased, and the reliability (accuracy) of the determination of whether or not there is a match between the keys can be improved.
Fig. 9 (a) and 9 (b) are diagrams illustrating examples of use of the registration key by the user in fig. 2. In particular, when the user 601 uses VR content on the HMD100, the user may not be able to visually recognize where the HMD100 is removed because the entire real field of view is blocked. Therefore, for example, by setting the direction of the wall toward which the user 601 faces when the apparatus is stopped as the registration key, the effective range of the registration key can be expanded.
In the examples of fig. 9 (a) and 9 (b), any of the 4 walls of the room is determined as the registration key. Thus, the user 601 can specify the registration key by removing the HMD100 in a state of being directed to any of the 4 walls without paying attention to a delicate direction, and storing the direction of the wall. As a result, the user 601 can easily process the registration key. Further, when the 4-sided wall is subdivided into the upper half, the lower half, and the like, the number of keys can be increased while maintaining the ease of processing.
In the case of using such a scheme, the registration key is determined based on, for example, a sensor output value that is a detection result of the geomagnetic sensor 124 or the like. Here, in this case, for example, the registration key has a certain valid range such that the northeast to northwest range (i.e., the predetermined sensor output value range) is the wall V and the northwest to southwest range is the wall W. When the sensor output value (candidate key) is obtained via the candidate key extraction unit 210 at the time of device startup, it is determined whether or not the sensor output value (candidate key) matches the sensor output value range (registration key). When the sensor output value range (registration key) contains the sensor output value (candidate key), the sensor output value range (registration key) matches.
Fig. 10 (a) is a diagram illustrating another use example of the user pair registration key in fig. 2, and fig. 10 (b) is a schematic diagram showing a configuration example of key content pair information corresponding to fig. 10 (a). In the examples of fig. 10 (a) and 10 (b), the received data of the communication interface 151 of fig. 2 is used instead of the camera image as the registration key. Specifically, the reception data is, for example, identification information of a beacon. As described above, the communication interface 151 performs short-range wireless communication.
In general, an apparatus that performs short-range wireless communication periodically outputs a beacon including identification information of the apparatus itself. For example, the access point device of the wireless LAN periodically outputs an SSID (Service Set Identifier) as identification information of the beacon. Thus, the HMD100 can determine the registration key using the identification information of the beacon by receiving the identification information of the beacon via the communication interface 151.
In fig. 10 (a), the user 601 executes content a using the HMD100 within the reception area of SSID # 1 from the access point device 1001, and instructs the HMD100 to stop the device in the middle thereof. After that, the user 601 moves the location, executes the content B using the HMD100 in the reception area of the SSID # 2 from the access point device 1002, and instructs the HMD100 to stop the device in the middle of the execution.
As a result, the key content pair information 216d in fig. 10 (B) includes the correspondence between the content a and the SSID # 1 as the registration key and the correspondence between the content B and the SSID # 2 as the registration key. After that, when the user 601 activates the HMD100 in the reception area of SSID # 1, the key content pair recovery unit 212 determines whether or not the candidate key extracted by the candidate key extraction unit 210 (here, the reception data "SSID # 1" of the communication interface 151) matches the registration key stored in (b) of fig. 10. In this case, since the registration key "SSID # 1" matches, the key content pair restoring unit 212 automatically activates the corresponding content a. Similarly, when the user 601 activates the HMD100 in the reception area of SSID # 2, the key content pair recovery unit 212 automatically activates content B.
As described above, when different beacon transmitters (access point devices in the example of fig. 10 (a)) are provided for each location, the user 601 can restore different content states depending on the location where the HMD100 is started. Examples of the units of the site include room units, building units, and area units. In addition to the case where the beacon transmitter is installed in each location, the present invention can be applied to a case where the user carries the beacon transmitter. In this case, when the user holding the beacon transmitter stops the HMD in a state of a certain content, if the same or a different user holding the same beacon transmitter starts the HMD later, the state of the content at the time of the stop is restored.
Note that, although the received data of the communication interface 151 is used as the registration key, the image of the camera 111 and the detection results of the various sensors 201 may be combined with each other in some cases. For example, in the case of combining images of the cameras 111, even if rooms of the same interior are decorated, the registration key can be distinguished by the identification information of the beacon only if the rooms themselves are different, and in addition, the registration key can be distinguished by the image of the camera 111 for each room.
As another example of using the registration key, the following method may be mentioned: the user stops the HMD indicating device in a state where a package of content to be reproduced is captured by the camera, and starts the HMD indicating device in a state where the package can be captured by the camera next time. In addition, a method of using the positions of the lines of sight detected by the right-eye line-of-sight detector 112 and the left-eye line-of-sight detector 113 as a registration key is exemplified. For example, when the user instructs the apparatus to stop in the state of viewing the upper right of the screen while playing back a predetermined content, the user can automatically play back the predetermined content when the apparatus is instructed to start in the state of viewing the upper right of the screen next time. In addition, even when the HMD100 includes a GPS receiver, the GPS information can be determined as the registration key.
Handling of HMD usage by multiple users
When the HMD100 is used by a plurality of users, for example, a combination of the image of the camera 111 and the detection results of the various sensors 201 is used as a registration key, whereby the registration key can be independently determined for each user. Specifically, the various sensors 201 may be, for example, a biosensor 129 that detects biometric information of a user, such as an iris or a retina. In some cases, the user may be distinguished from the pattern of blood flow, pulse, or the like acquired by the biosensor 129, or from the habit of wearing the device that can be detected by the acceleration sensor 122 or the gyro sensor 123.
Main effects of embodiment 1
As described above, by using the head mounted information processing device (HMD) according to embodiment 1, it is possible to improve the convenience of the user representatively. Specifically, the user can use the HMD in the state following the previous stop without performing a selection operation with a remote controller or the like for a long time. In this case, the HMD may continue the state at the time of the previous stop while discriminating the content using the registration key, as in the case of movie viewing or continuing movie viewing after document creation, instead of continuing the state at the time of the previous stop directly, for example, as in the case of continuing the document creation after movie viewing or document creation. Further, the user can continue the state at the time of the previous stop without performing an unnecessary operation such as a selection operation, and thus the user can continue to feel immersed in the content.
(embodiment mode 2)
Handling of case where desired content is not activated
Fig. 11 is a diagram for explaining an example of the operation of the key content pair restoration unit in fig. 2 in the head-mounted information processing device according to embodiment 2 of the present invention. In embodiment 1 described above, it is assumed that a desired content is activated when there is a match between the registration keys, but depending on the accuracy of determination as to whether or not there is a match between the keys, an error in determination of the registration key by the user, and the like, in some cases, an erroneous content is activated. Therefore, the key content pair restoring unit 212 determines whether or not a cancel instruction is received from the user within a predetermined period after restoring the state of the content, and when the cancel instruction is received, stops the current content and displays a predetermined initial screen after the device is started on the display (display unit) 120.
In the example of fig. 11, the HMD100 starts the content 2 in a state where the user 601 expects the start of the content 1 at the time of device start. Therefore, the user 601 issues a cancel instruction within a predetermined period thereafter, for example, by panning or the like. Upon receiving the cancel instruction, the key content pair restoration unit 212 displays a predetermined initial screen (e.g., a menu screen) after the device is started on the display (display unit) 120. Note that the motion of the head is detected by the gyro sensor 123.
The method of canceling the instruction is not limited to this, and various methods can be used including, for example, a cancel button of a remote controller. Note that, although the key content pair restoring unit 212 displays a predetermined initial screen here, in some cases, a method may be used in which the content registered in the key content pair information 216 is sequentially reproduced, and whether or not a cancel instruction is issued by the user is determined every reproduction.
Main effects of embodiment 2
As described above, by using the head mounted information processing device (HMD) according to embodiment 2, it is possible to cope with the case where the wrong content is activated in addition to the various effects described in embodiment 1, and it is possible to improve the convenience of the user.
(embodiment mode 3)
Handling of case where a plurality of registration keys are matched
Fig. 12 (a), 12 (b), 12 (c), and 12 (d) are diagrams for explaining an example of the operation of the key content pair restoration unit in fig. 2 in the head-mounted information processing device according to embodiment 3 of the present invention. Here, assume a case where the key content pair information 216a as shown in fig. 5 is stored in the memory 128. In fig. 12 (a) and 12 (b), the user 601 wearing the HMD100 instructs the device to start in a state where both the clock 602 and the calendar 603 are captured by the camera 111.
In this case, the key content pair recovery section 212 extracts two objects of the clock 602 and the calendar 603 as candidate keys using the candidate key extraction section 210. These two candidate keys are saved to the memory 128 as registration keys, as shown by the key content pair information 216a of fig. 5. As described above, when the image of the camera 111 at the time of device startup includes a plurality of registration keys, the key content pair recovery unit 212 displays a content selection screen on the display (display unit) 120, as shown in fig. 12 (c) or 12 (d). The selection screen is a screen for allowing the user to select any one of a plurality of contents corresponding to a plurality of registration keys, and is, for example, a screen on which thumbnails of the respective contents are displayed.
Thus, the user 601 can reliably start the desired content even when a positional shift, a scale shift, or the like occurs to some extent between the camera image when the apparatus is stopped and the camera image when the apparatus is started. The selection method by the user on the selection screen includes, for example, a method of directing the head or the line of sight to a predetermined time, a method of pressing a determination button such as a remote controller while turning the head or the line of sight, and the like, in addition to a method of performing an operation by the remote controller or the like. The direction of the head or the line of sight is detected by the gyro sensor 123, the geomagnetic sensor 124, or each line of sight detector (112, 113).
In addition, with the method of fig. 12, it is also possible to deal with a case where the user 601 does not clearly memorize the registration key. Specifically, for example, when the user 601 does not perform a selection operation on the selection screen, the key content pair restoration unit 212 sequentially determines whether or not each camera image matches the registration key in conjunction with the surrounding movement (further, the change in the camera image) of the user 601. Then, the key content pair restoration unit 212 sequentially displays a selection screen as shown in (d) of fig. 12 for each camera image. In addition, according to circumstances, the content may be automatically started at the time of device startup, instead of using the method of fig. 12, for example, a method of always displaying a selection screen as shown in (d) of fig. 12 and allowing the user to select the selection screen may be used.
Fig. 13 (a), 13 (b), 13 (c), and 13 (d) are diagrams for explaining another operation example of the key content pair restoration unit in fig. 2 in the head-mounted information processing device according to embodiment 3 of the present invention. In fig. 13 (c) and 13 (d), the thumbnails of the respective contents on the selection screen are weighted, unlike the cases of fig. 12 (c) and 12 (d).
Specifically, the key content pair restoration unit 212 largely weights the registration key closer to the center in the image of the camera 111 at the time of device activation, and displays the weighted selection screen. Examples of the weighting method include a method of displaying the corresponding thumbnail image larger as the weighting of the registration key is larger as shown in fig. 13 (c), and a method of displaying the corresponding thumbnail image in front as the weighting of the registration key is larger as shown in fig. 13 (d).
Main effects of embodiment 3
As described above, by using the head mounted information processing device (HMD) according to embodiment 3, it is possible to cope with a case where a plurality of registration keys are matched, in addition to the various effects described in embodiment 1, and it is possible to improve the convenience of the user. Further, since the user can specify a target close to a certain extent as the registration key, the restriction in specifying the registration key can be alleviated, and the convenience of the user can be improved.
(embodiment mode 4)
Assistance in the memory of registration keys
In particular, when the HMD100 for VR of a non-transmissive type is used, it may be difficult for the user to remember which direction (in other words, the imaging direction of the camera 111) the device is facing when it is stopped. Therefore, the key content pair information registration unit 211 in fig. 2 may display the image of the camera 111 on the display 120 for a certain period of time when the apparatus is stopped. Thereby, the user can clearly memorize the registration key.
Specifically, the key content pair information registration unit 211 displays, for example, an image of the content execution unit 206 and an image of the camera 111 alternately a plurality of times on the display 120, and displays the images of both continuously in a short time. This makes it possible for the user to have a strong impression of the relationship between the two images, and to remember the registration key more clearly. In addition, at this time, the user can explicitly select an object determined as the registration key by moving the head after displaying the image of the camera 111 in the display 120.
Main effects of embodiment 4
As described above, by using the head mounted information processing device (HMD) according to embodiment 4, it is possible to assist the user in memorizing the registration key in addition to the various effects described in embodiment 1, and thus it is possible to improve the convenience of the user.
(embodiment 5)
Outline of head-mounted information processing device
Fig. 14 is a block diagram showing a schematic configuration example around a head-mounted information processing apparatus according to embodiment 5 of the present invention. Fig. 14 shows a server apparatus 902 connected to a head mounted information processing apparatus (HMD)100 via a communication network 910, in addition to the HMD100 shown in fig. 1. The server 902 includes a communication interface 903, a CPU905, and a memory 908 connected to each other via a bus 920. The communication interface 903 communicates with the communication interface 151 of the HMD100 via the antenna 904 and the communication network 910. The memory 908 stores therein the program 906 and the information data 907.
Here, the key content pair information registration unit 211 in the HMD100 shown in fig. 2 transmits the generated key content pair information 216 to the server apparatus 902 outside the apparatus via the communication interface 151 when an event occurs, and stores the key content pair information in the memory 908 of the server apparatus 902. On the other hand, the key content pair recovery unit 212 acquires the key content pair information 216 from the server 902 via the communication interface 151 at the time of device startup, and stores the key content pair information in the memory 128 of the device itself.
Main effects of embodiment 5
As described above, by using the head mounted information processing device (HMD) according to embodiment 5, it is possible to further improve the convenience of the user in addition to the various effects described in embodiment 1. Specifically, since the key content pair information 216 is stored in the server device 902, the user can use the registration key even when using another HMD100 of the same model, for example.
The storage destination of the program corresponding to various contents of the HMD100 is not limited to the memory 128 of the present apparatus, and may be the memory 908 of the server apparatus 902. In this case, the HMD100 appropriately acquires the desired program 906 from the server apparatus 902 by streaming or downloading. This can reduce the memory capacity of the HMD100, for example.
The present invention is not limited to the above embodiments, and includes various modifications. For example, the above embodiments are described in detail to explain the present invention easily and understandably, and are not necessarily limited to having all the configurations described. In addition, a part of the structure of one embodiment may be replaced with the structure of another embodiment, and the structure of another embodiment may be added to the structure of one embodiment. In addition, a part of the structures of the respective embodiments can be added, deleted, and replaced with other structures.
Further, the above-described respective structures, functions, processing units, processing means, and the like may be partially or entirely realized by hardware by designing them in an integrated circuit, for example. The respective structures, functions, and the like described above may be realized by software by interpreting and executing a program for realizing the respective functions by a processor. Information such as programs, tables, and files for realizing the respective functions can be stored in a recording device such as a memory, a hard disk, an SSD (Solid State Drive), or a recording medium such as an IC card, an SD card, or a DVD.
The control lines and the information lines are illustrated as parts deemed necessary in the description, and not all the control lines and the information lines are necessarily illustrated in the product. In practice, it is also contemplated that substantially all structures may be interconnected.
Claims (19)
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| PCT/JP2019/015501 WO2020208717A1 (en) | 2019-04-09 | 2019-04-09 | Head-mounted information processing device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113728621A true CN113728621A (en) | 2021-11-30 |
Family
ID=72751889
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201980095249.6A Pending CN113728621A (en) | 2019-04-09 | 2019-04-09 | Head-mounted information processing device |
Country Status (4)
| Country | Link |
|---|---|
| US (3) | US12019937B2 (en) |
| JP (3) | JP7286760B2 (en) |
| CN (1) | CN113728621A (en) |
| WO (1) | WO2020208717A1 (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114999030A (en) * | 2022-05-25 | 2022-09-02 | 杭州萤石软件有限公司 | Unlocking method, system, electronic device and storage medium |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113728621A (en) * | 2019-04-09 | 2021-11-30 | 麦克赛尔株式会社 | Head-mounted information processing device |
Family Cites Families (33)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2000023065A (en) | 1998-06-29 | 2000-01-21 | Dainippon Printing Co Ltd | Goggle type display with gaze input |
| JP2005065214A (en) | 2003-07-30 | 2005-03-10 | Sanyo Electric Co Ltd | Content reproducing apparatus |
| US8433173B2 (en) * | 2005-09-29 | 2013-04-30 | Nikon Corporation | Portable media player system |
| JP2007259343A (en) | 2006-03-24 | 2007-10-04 | Toshiba Corp | REPRODUCTION CONTROL DEVICE AND REPRODUCTION CONTROL METHOD |
| US7599789B2 (en) * | 2006-05-24 | 2009-10-06 | Raytheon Company | Beacon-augmented pose estimation |
| JP5026294B2 (en) | 2008-01-29 | 2012-09-12 | 京セラ株式会社 | Content playback device |
| JP5428174B2 (en) | 2008-03-24 | 2014-02-26 | 株式会社ニコン | Head mounted display device |
| JP5277801B2 (en) | 2008-09-04 | 2013-08-28 | カシオ計算機株式会社 | Image reproducing apparatus and program |
| US20120212406A1 (en) * | 2010-02-28 | 2012-08-23 | Osterhout Group, Inc. | Ar glasses with event and sensor triggered ar eyepiece command and control facility of the ar eyepiece |
| US20110239142A1 (en) * | 2010-03-25 | 2011-09-29 | Nokia Corporation | Method and apparatus for providing content over multiple displays |
| CN103329075B (en) | 2011-01-06 | 2017-12-26 | TiVo解决方案有限公司 | For the method and apparatus based on gesture control |
| AU2015255652B2 (en) * | 2014-05-09 | 2018-03-29 | Google Llc | Systems and methods for using eye signals with secure mobile communications |
| KR102247817B1 (en) * | 2014-09-11 | 2021-05-04 | 삼성전자 주식회사 | Method and apparatus for providing lockscreen |
| JP5767386B1 (en) * | 2014-12-15 | 2015-08-19 | 株式会社コロプラ | Head mounted display system, method for displaying on head mounted display, and program |
| JP2016115329A (en) | 2015-05-22 | 2016-06-23 | 株式会社コロプラ | Head-mounted display system, method of displaying on head-mounted display, and program |
| KR20170051013A (en) | 2015-11-02 | 2017-05-11 | 엘지전자 주식회사 | Tethering type head mounted display and method for controlling the same |
| KR20170059163A (en) * | 2015-11-20 | 2017-05-30 | 삼성전자주식회사 | Electronic apparatus and operating method thereof |
| KR20170060485A (en) * | 2015-11-24 | 2017-06-01 | 삼성전자주식회사 | Electronic device and method for displaying content according to display mode |
| US10665020B2 (en) * | 2016-02-15 | 2020-05-26 | Meta View, Inc. | Apparatuses, methods and systems for tethering 3-D virtual elements to digital content |
| US11487353B2 (en) * | 2016-11-14 | 2022-11-01 | Logitech Europe S.A. | Systems and methods for configuring a hub-centric virtual/augmented reality environment |
| US10754416B2 (en) * | 2016-11-14 | 2020-08-25 | Logitech Europe S.A. | Systems and methods for a peripheral-centric augmented/virtual reality environment |
| JP2018097160A (en) * | 2016-12-14 | 2018-06-21 | セイコーエプソン株式会社 | Display system, display device, and control method for display device |
| JP2018129732A (en) | 2017-02-09 | 2018-08-16 | 三菱自動車工業株式会社 | Video display device for vehicle |
| KR102330829B1 (en) * | 2017-03-27 | 2021-11-24 | 삼성전자주식회사 | Method and apparatus for providing augmented reality function in electornic device |
| KR102555443B1 (en) * | 2017-05-01 | 2023-07-12 | 매직 립, 인코포레이티드 | Matching content to a spatial 3d environment |
| JP6271794B1 (en) | 2017-05-18 | 2018-01-31 | 株式会社プランネット・アソシエイツ | Application layer multicast delivery method |
| KR102374404B1 (en) * | 2017-07-25 | 2022-03-15 | 삼성전자주식회사 | Device and method for providing content |
| US11314399B2 (en) * | 2017-10-21 | 2022-04-26 | Eyecam, Inc. | Adaptive graphic user interfacing system |
| JP2019105885A (en) * | 2017-12-08 | 2019-06-27 | セイコーエプソン株式会社 | Head-mounted display device, information processing device, information processing system and method for controlling head-mounted display device |
| WO2019143117A1 (en) * | 2018-01-18 | 2019-07-25 | Samsung Electronics Co., Ltd. | Method and apparatus for adjusting augmented reality content |
| US11010025B2 (en) * | 2018-08-17 | 2021-05-18 | Goodrich Corporation | Multiple portable control panels in aircraft cargo handling system |
| CN113728621A (en) * | 2019-04-09 | 2021-11-30 | 麦克赛尔株式会社 | Head-mounted information processing device |
| US11748679B2 (en) * | 2019-05-10 | 2023-09-05 | Accenture Global Solutions Limited | Extended reality based immersive project workspace creation |
-
2019
- 2019-04-09 CN CN201980095249.6A patent/CN113728621A/en active Pending
- 2019-04-09 WO PCT/JP2019/015501 patent/WO2020208717A1/en not_active Ceased
- 2019-04-09 US US17/601,936 patent/US12019937B2/en active Active
- 2019-04-09 JP JP2021513070A patent/JP7286760B2/en active Active
-
2023
- 2023-05-24 JP JP2023085365A patent/JP7425245B2/en active Active
-
2024
- 2024-01-18 JP JP2024006036A patent/JP7607155B2/en active Active
- 2024-05-16 US US18/666,061 patent/US12353779B2/en active Active
-
2025
- 2025-06-11 US US19/234,357 patent/US20250306835A1/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114999030A (en) * | 2022-05-25 | 2022-09-02 | 杭州萤石软件有限公司 | Unlocking method, system, electronic device and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| US20220137908A1 (en) | 2022-05-05 |
| JPWO2020208717A1 (en) | 2020-10-15 |
| US20240303018A1 (en) | 2024-09-12 |
| JP2023111925A (en) | 2023-08-10 |
| JP2024050643A (en) | 2024-04-10 |
| JP7607155B2 (en) | 2024-12-26 |
| JP7286760B2 (en) | 2023-06-05 |
| US12353779B2 (en) | 2025-07-08 |
| JP7425245B2 (en) | 2024-01-30 |
| WO2020208717A1 (en) | 2020-10-15 |
| US20250306835A1 (en) | 2025-10-02 |
| US12019937B2 (en) | 2024-06-25 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| EP4342196B1 (en) | Beacons for localization and content delivery to wearable devices | |
| US11520399B2 (en) | Interactive augmented reality experiences using positional tracking | |
| CN112911182B (en) | Game interaction method, device, terminal and storage medium | |
| JP7607155B2 (en) | Information processing device | |
| US12299933B2 (en) | Collaborative augmented reality eyewear with ego motion alignment | |
| US20150378433A1 (en) | Detecting a primary user of a device | |
| CN110544280A (en) | AR system and method | |
| US20210303258A1 (en) | Information processing device, information processing method, and recording medium | |
| WO2019205868A1 (en) | Method, device and apparatus for repositioning in camera orientation tracking process, and storage medium | |
| CN112245912B (en) | Sound prompting method, device, equipment and storage medium in virtual scene | |
| CN112711335A (en) | Virtual environment picture display method, device, equipment and storage medium | |
| KR20230070308A (en) | Location identification of controllable devices using wearable devices | |
| CN111344776B (en) | Information processing device, information processing method, and program | |
| CN110875042B (en) | Designated crowd monitoring method and device and storage medium | |
| JP7616216B2 (en) | Information processing device, information processing method, and program | |
| WO2021176947A1 (en) | Information processing apparatus and information processing method | |
| WO2023233488A1 (en) | Information processing device and information processing method | |
| US20250068228A1 (en) | Reducing power consumption of extended reality devices | |
| HK40037812B (en) | Sound prompt method and apparatus in virtual scene, device and storage medium | |
| HK40037812A (en) | Sound prompt method and apparatus in virtual scene, device and storage medium | |
| HK40046421B (en) | Game interaction method and device, terminal and storage medium | |
| HK40046421A (en) | Game interaction method and device, terminal and storage medium | |
| HK40042025B (en) | Method and device for displaying a virtual environment screen, apparatus and storage medium | |
| KR20160027813A (en) | Glass type terminal |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |