CN109983784A - Information processing unit, methods and procedures - Google Patents

Abstract

A kind of information processing unit is provided with Activity recognition unit, voice data selecting unit and acoustic information generation unit, wherein the Activity recognition unit is configured to identify the behavior pattern of user based on sensor information.The voice data selecting unit is configured to select voice data corresponding with the behavior pattern for the user that Activity recognition unit is identified.The acoustic information generation unit generates multichannel audio information, which positions the acoustic image of sound source in the real space around user based on the voice data selected by voice data selecting unit.The information processing unit allows to obtain the augmented reality for the variation for following the matter in user behavior.

Description

Information processing apparatus, method and program

technical field

The present technology relates to improvements in the quality of augmented reality technology.

Background technique

In the wearable computing technology field, a technology is known that estimates the amount of spatial displacement of the user by a sensor device included in the wearable computer when the user wearing the wearable computer moves (for example, see Patent Document 1).

Patent Document 1 also discloses a technique regarding the generation of multi-channel audio information. The technique described in Patent Document 1 is to synthesize audio so that a sound can be perceived as if the sound is emitted from a spatial position, and in particular to synthesize audio so that the sound can be perceived as emitted even when the user changes the position or direction The spatial position of the sound also does not change.

Patent Document 2 discloses a technique of displaying a virtual object related to the information of the past action of the other person by using the information of the past real action of the other person when the person performs an action. The application example disclosed in Patent Document 2 shows an example in which running images of other people running the same route are displayed on the glasses-type display device during running.

Citation List

Patent Literature

Patent Document 1: Japanese Patent Application Laid-Open No. 2013-005021

Patent Document 2: Japanese Patent Application Laid-Open No. 2013-167941

SUMMARY OF THE INVENTION

technical problem

In the technical field of providing augmented reality to users through the use of wearable computers, it is desirable to provide a more realistic presentation. However, the above examples do not focus on the content of user actions to provide augmented reality. For example, even when the content of the motion is changed from a "walking" motion to a "running" motion or from a "light fatigue" motion to a "heavy fatigue" motion, an output following a qualitative change in the motion is not performed.

In view of the above circumstances, an object of the present technology is to provide an information processing apparatus that makes it possible to realize augmented reality that follows qualitative changes in user actions.

solution to the problem

To achieve the above object, an information processing apparatus according to an aspect of the present technology includes: an action recognition unit; an audio data selection unit; and an audio information generation unit.

The motion recognition unit is configured to recognize a motion pattern of the user based on the sensor information.

The audio data selection unit is configured to select audio data corresponding to the motion pattern of the user recognized by the motion recognition unit.

An audio information generation unit that generates multi-channel audio information for localizing a sound image of a sound source in a real space around a user based on the audio data selected by the audio data selection unit.

According to the information processing apparatus according to an aspect of the present technology, it is possible to provide the user with augmented reality that follows qualitative changes in the user's actions.

The audio data selection unit is configured to select audio data as audio emitted from a virtual object to be placed in the real space.

In this case, the audio information generating unit may be configured to perform sound image localization by generating multi-channel audio information, and the virtual object is placed at the position of the sound source by the sound image localization.

The audio data selection unit may be configured to, when audio data to be selected is changed as a recognition result of the action recognition unit, select audio data corresponding to an audio data switching pattern from the audio data before the change to the audio data after the change and the changed audio data.

The audio data selection unit may be configured to, when audio data to be selected is changed as a recognition result of the motion recognition unit, select the audio data associated with the virtual object in the presence of a plurality of pieces of audio data corresponding to the motion pattern of the user. information to match the audio data.

The information processing apparatus further includes a displacement calculation unit that outputs a user displacement including a relative change in the user's position based on the sensor information.

The audio information generation unit may be configured to modulate the audio data selected by the audio data selection unit based on the user displacement output by the displacement calculation unit, thereby generating the multi-channel audio information.

The audio information generation unit may be configured to modulate the audio data selected by the audio data selection unit such that a sound source whose sound image is located by the multi-channel audio information is placed at a position following the user displacement output by the displacement calculation unit, thereby generating Multichannel audio information.

The audio information generating unit may be configured to generate the multi-channel audio information such that the sound source positioned by the sound image by the multi-channel audio information follows a position in space with a time delay from the position of the user identified by the user displacement start.

The audio information generating unit may be configured to generate the multi-channel audio information based on the user's displacement output by the displacement calculating unit and the map information including the position coordinates of the building obtained from the outside so that the virtual object is not placed in the map information Included in the range of the location coordinates of the building.

The audio information generating unit may be configured to generate multi-channel audio information including the collision sound in a case where the range of the position coordinates of the building included in the map information overlaps with the position where the virtual object is placed.

The information processing apparatus further includes a state analysis unit configured to analyze a state of the user that can be changed according to one of the sensor information and the motion pattern of the user recognized by the motion recognition unit.

The audio data selection unit may be configured to select audio data corresponding to the action pattern of the user and audio data corresponding to the state of the user analyzed by the state analysis unit.

The audio information generating unit may be configured to synthesize the audio data corresponding to the action pattern of the user selected by the audio data selecting unit and the audio data corresponding to the state of the user, thereby generating multi-channel audio information based on the synthesized audio data.

The state analysis unit may be configured to assign the fatigue level per unit time according to one of the sensor information and the action pattern of the user recognized by the action recognition unit, and to accumulate the assigned fatigue level per unit time, thereby calculating the fatigue level as User's status.

The audio data selection unit may be configured to select audio data different from audio data corresponding to the motion pattern of the user recognized by the motion recognition unit if the motion pattern of the user recognized by the motion recognition unit continues to exceed a predetermined threshold.

An information processing method according to another aspect of the present technology includes: an action recognition step; an audio data selection step; and an audio information generation step.

In the motion recognition step, the motion pattern of the user is recognized based on sensor information.

In the audio data selection step, audio data corresponding to the motion pattern of the user recognized by the motion recognition step is selected.

In the audio information generating step, based on the audio data selected by the audio data selecting step, multi-channel audio information for locating the sound image of the sound source in the real space around the user is generated.

A program according to still another aspect of the present technology causes a computer to perform the steps of: a motion recognition step; an audio data selection step; and an audio information generation step.

In the motion recognition step, the motion pattern of the user is recognized based on sensor information.

In the audio data selection step, audio data corresponding to the motion pattern of the user recognized by the motion recognition step is selected.

In the audio information generating step, based on the audio data selected by the audio data selecting step, multi-channel audio information for locating the sound image of the sound source in the real space around the user is generated.

The beneficial effects of the present invention

As described above, according to the present technology, augmented reality that follows qualitative changes in user actions can be realized.

It should be noted that the above-described effects are not necessarily limiting. In addition to or in place of the above-described effects, any of the effects described in this specification or other effects that can be understood from this specification may be employed.

Description of drawings

FIG. 1 is a diagram (part 1) showing an example of augmented reality provided to a user as an output result of the information processing apparatus according to the embodiment of the present technology.

FIG. 2 is a diagram (part 2) showing an example of augmented reality provided to a user as an output result of the information processing apparatus according to the embodiment of the present technology.

FIG. 3 is a diagram showing an example of the external configuration of the above-described information processing apparatus.

FIG. 4 is a block diagram showing an example of the internal configuration of the above-described information processing apparatus.

FIG. 5 is a flowchart showing the flow of processing performed by the above-described information processing apparatus.

FIG. 6 is a diagram describing information processing by an audio data selection unit of the above-described information processing apparatus.

FIG. 7 is a diagram describing information processing by a sound image position calculation unit of the above-described information processing apparatus.

FIG. 8 is a block diagram showing an example of a configuration of another embodiment of the present technology.

FIG. 9 is a block diagram showing an example of a configuration of another embodiment of the present technology.

Detailed ways

Hereinafter, preferred embodiments of the present technology will be described in detail with reference to the accompanying drawings. It should be noted that components having substantially the same functional configuration are denoted by the same reference numerals, and overlapping descriptions will be omitted in this specification and the drawings.

Note that the description will be given in the following order.

1. Outline of Information Processing Device According to Embodiment of Present Technology

2. Configuration

2-1. External configuration

2-2. Internal configuration

3. Operation

4 Conclusion

5. Other implementations

5-1. Another Embodiment 1

5-2. Another Embodiment 2

5-3. Another Embodiment 3

5-4. Additional Embodiment 4

5-5. Additional Embodiment 5

5-6. Additional Embodiment 6

5-7. Additional Embodiment 7

6. Appendix

<1. Outline of Information Processing Device According to Embodiment of the Present Technology>

1 and 2 are each a diagram showing an example of augmented reality provided to a user as an output result of the information processing apparatus 1 according to this embodiment. The information processing apparatus 1 outputs multi-channel audio information in which sound images are positioned so that the sound can be heard from a specific direction around the user. Sound image localization is performed by, for example, adjusting the volume of sound entering each of the left and right ears.

Part (a) of FIG. 1 shows a state in which a virtual dog, which is an example of a virtual object, is walking 50 cm in front of the user. The dog's footsteps and breathing sounds are multi-channel, and the volume or effect of the sound entering the left and right ears is adjusted to provide the user with augmented reality as shown. Here, changing the balance between the left and right volume in the multi-channel audio information produces a feeling as if the virtual object is walking at a position 100 cm behind the user's left, as shown in part (b) of FIG. 1 .

Such sound image localization technology enables users to feel the presence of virtual objects to a certain extent. Meanwhile, it is unnatural if the sound emitted from the virtual object does not change when the user's action changes qualitatively or the user's state changes. For example, in the case where the state of the user is changed from the walking state (part (a) of FIG. 2 ) to the running state (part (b) of FIG. 2 ), if a virtual object simulated as a dog is made to sound the same as the breathing when walking The breathing sound follows the user, then it is unnatural. If the virtual object doesn't seem to tire at all after running with the user for a long time, then this is unnatural.

In view of the above, in this embodiment described below, in order to provide augmented reality with higher quality, the augmented reality follows qualitative changes in user actions.

Here, the qualitative change includes a change in the type of the user's action ("running", "walking", etc.). In existing wearable computing, the system has been able to comprehend that the user's action is "moving" through methods such as absolute position measurement. However, in the case where the action is changed from the "walking" action type to the "running" action type, the follow-up to the qualitative change is insufficient. For this reason, there has been a possibility to provide augmented reality that makes users feel uncomfortable.

As an example, in the case where a virtual object is regarded as a character that exists virtually, the augmented reality to be provided to the user needs to be changed according to the action type of the character. For example, there is a possibility that if the footsteps of the character when running are different from the footsteps of the character when walking (although both are "footsteps"), the user will feel uncomfortable.

In this embodiment, the user is provided with augmented reality that follows qualitative changes in the user's actions by recognizing the user's motion pattern based on sensor information input from the sensor 101, selecting audio data corresponding to the recognized motion pattern, The selected audio data is then shifted.

Note that in the following description, a virtual dog is used as an example of a virtual object. Furthermore, an application that enables the user to walk with the virtual dog by wearing the information processing apparatus 1 will be described as an example as the entire application.

The outline of the information processing apparatus 1 according to this embodiment has been described so far. Next, the configuration of the information processing apparatus 1 will be described with reference to FIGS. 3 and 4 .

<2-1. External configuration>

FIG. 3 is a diagram showing an example of the external configuration of the information processing apparatus according to this embodiment. As shown in FIG. 3 , the information processing apparatus 1 is, for example, a neck-mounted wearable computer. As shown in FIG. 3 , the neck-mounted information processing apparatus 1 has a horseshoe shape as a whole, and the user wears it by hanging from the back side of the neck.

Furthermore, as shown in FIG. 3 , the information processing apparatus 1 includes an audio output unit 109 and various sensors 101 . The audio output unit 109 reproduces audio data. In particular, the speaker 15 according to this embodiment reproduces the audio signal of the virtual object on which the sound image localization process has been performed, which makes the user perceive the virtual object as if the virtual object actually exists in the real space.

<2-1. Internal configuration>

FIG. 4 is a diagram showing an example of the internal configuration of the information processing apparatus according to this embodiment. As shown in FIG. 4 , the information processing apparatus 1 includes a central processing unit (hereinafter, referred to as a CPU) 100 as hardware, a sensor 101 , a storage unit 107 , and an audio output unit 109 . Through information processing by a software program, the CPU 100 is configured to have the respective functional blocks shown in FIG. 4 .

The sensor 101 is shown as an abstraction layer for various sensor device groups of the information processing device 1 . Specific examples of the sensor device include an acceleration sensor that detects acceleration in three directions of longitudinal, horizontal, and vertical directions, a gyroscope sensor that detects axial velocity in three directions, an atmospheric pressure sensor that measures atmospheric pressure, and a sensor that detects Geomagnetic orientation sensor. A mechanism for receiving a signal in a GPS (Global Positioning System), a mobile communication system, or a wireless local area network and detecting position information of the information processing apparatus 1 (hereinafter, referred to as "absolute position information") can be regarded as constituting the sensor 101 A sensor device group. In addition, specific examples of the sensor 101 include a sensor device that detects a user's pulse and body temperature and a rise in body temperature, and a microphone for inputting sound. Note that information input from the sensor 101 to the CPU 100 will be referred to as sensor information.

The storage unit 107 includes a non-volatile storage device such as an electrically erasable programmable read only memory (EEPROM). The storage unit 107 stores various types of audio data.

The audio output unit 109 is a device having a function of outputting multi-channel audio information for locating a sound image of a sound source in a real space around the user as sound waves, the multi-channel audio information Generated by the audio information generating unit 108 . A specific example of the audio output unit 109 may include a speaker in the form shown in FIG. 3 .

The CPU 100 is an arithmetic processing device of the information processing device 1 . The CPU 100 is not necessarily the main arithmetic processing device of the information processing device 1 . The CPU 100 may include auxiliary devices for the information processing apparatus 1 . The CPU 100 executes a software program stored in the storage unit 107 or a software program downloaded from the outside. Therefore, the CPU 100 is configured to include an action recognition unit 102, a fatigue degree calculation unit 103, an audio data selection unit 104, a displacement calculation unit 105, a sound image position calculation unit 106, and an audio information generation unit 108 having the following functions.

The action recognition unit 102 recognizes the type of user action as an example of a user action pattern.

The fatigue degree calculation unit 103 is an example of a state analysis unit that analyzes the user's state and calculates the user's fatigue degree.

The audio data selection unit 104 selects appropriate audio data based on the type of user action recognized by the action recognition unit 102 and the fatigue level calculated by the fatigue level calculation unit 103 .

The displacement calculation unit 105 calculates the spatial displacement of the user that occurs between time T0 and later time Tn based on the information input from the sensor 101 .

The sound image position calculation unit 106 calculates the position where the virtual object to be superimposed in the real space should be located in the real space based on the user displacement calculated by the displacement calculation unit 105 .

The audio information generation unit 108 generates multi-channel audio information for locating the sound image of the sound source in the real space around the user by modulating the audio data selected by the audio data selection unit 104 . At the time of modulation, the sound image position set by the sound image position calculation unit 106 is used as a parameter.

<3. Operation>

FIG. 5 is a flowchart showing the processing performed by the information processing apparatus according to this embodiment. In the description of the process shown in FIG. 5 described below, unless otherwise indicated, the CPU 100 is the subject of the operation. First, the CPU 100 acquires sensor information and inputs it to each unit (S101).

Next, the action recognition unit 102 recognizes the user's action type (an example of the user's action pattern) based on the information input from the sensor 101 ( S102 ). Examples of information processing to identify action types include methods using a learned determination device obtained by subjecting a determination device to machine learning, wherein the determination device uses sensor information as input and the action type as output. Alternatively, a method of determining stationary/walking/running based on changes in acceleration included in sensor information may be used.

Next, the fatigue degree calculation unit 103 calculates the user's fatigue degree ( S103 ). User fatigue may be an accumulated parameter. In this case, the parameter regarding the fatigue degree is calculated by, for example, multiplying the fatigue degree per unit time determined based on the type of motion recognized by the motion recognition unit 102 by the duration of the action. Fatigue may be a parameter that gradually decreases over time.

For example, as the degree of fatigue per unit time, the following values can be used: a value assigned to each motion type by using the result of recognizing the motion using the motion recognition unit 102, for example, -α is stationary, +β is walking, and +γ for running (α, β and γ are positive, β<γ). Note that in the case where the user's action type changes as a result of action recognition, the fatigue level per unit time may be updated accordingly. The degree of fatigue can be calculated by integrating the degree of fatigue per unit time allocated in this way.

The fatigue degree calculation unit 103 may calculate the fatigue degree by a simpler method, instead of calculating the fatigue degree using the recognition result of the action recognition unit 102 . For example, a value directly calculated from the accumulation of the number of steps of the user captured by the sensor 101 or the displacement detected by the acceleration sensor or the gyro sensor may be output as the degree of fatigue. Alternatively, the activity amount based on the output of the sensor device that detects the user's pulse and body temperature and the rise in body temperature (an example of sensor information) may be output as the fatigue level.

Next, the audio data selection unit 104 selects audio data according to the motion type of the user recognized by the motion recognition unit 102 (S104). The storage unit 107 stores in advance a plurality of audio data patterns corresponding to assumed action types. Multiple audio data patterns may correspond to one action type. In this case, the audio data selection unit 104 randomly selects one of the plurality of audio data patterns. It is to be noted that, depending on the type of action, it is not necessarily necessary to select audio data, for example in cases where outputting audio may be uncomfortable for the user.

For example, in a case where the motion recognition unit 102 has recognized that the user motion is "walking", the audio data selection unit 104 randomly selects a plurality of sounds associated with "walking" from the motion sounds of virtual objects stored in the storage unit 107 in advance. one of the pieces of audio data. In the case where the virtual object is the virtual dog described above, if the user is walking, audio is selected that makes the user feel as if the virtual dog is walking at the same pace as his/her pace.

Similarly, the motion recognition unit 102 has recognized that the user's motion is "running", and the audio data selection unit 104 randomly selects a plurality of audio pieces associated with "running" from the motion sounds of virtual objects pre-stored in the storage unit 107 one of the data. Furthermore, in the case where the motion recognition unit 102 has recognized that the user's motion is "still", the audio data selection unit 104 randomly selects a motion sound associated with "still" from the motion sounds of the virtual objects pre-stored in the storage unit 107 One of multiple pieces of audio data. In the case of "still", it can be configured not to select audio data.

For example, audio data of footsteps or motion sounds while walking is associated with "walking". Footsteps during running or audio data of motion sounds representing breathing thicker than breathing during walking are associated with "running".

The audio data selection unit 104 also selects audio data according to the fatigue degree calculated by the fatigue degree calculation unit 103 (S104). The audio data selection unit 104 classifies the degree of fatigue into "large degree of fatigue" and "small degree of fatigue" by a predetermined threshold. In a case where it is determined that the degree of fatigue is “large degree of fatigue”, the audio data selection unit 104 randomly selects pieces of audio data associated with “a degree of high degree of fatigue” from motion sounds of virtual objects stored in advance in the storage unit 108 . one. Meanwhile, in a case where it is determined that the degree of fatigue is "small degree of fatigue", the audio data selection unit 104 randomly selects a plurality of pieces of audio associated with "little degree of fatigue" from motion sounds of virtual objects pre-stored in the storage unit 108 one of the data.

For example, the sound of shortness of breath can be associated with "great fatigue." Furthermore, fatigue can be divided into three or more levels, such as a large level, a medium level, and a small level, and understood.

FIG. 6 is an explanatory diagram for understanding information processing by the audio data selection unit 104 . The two tables in part (a) of FIG. 6 are tables showing the association between audio data of motion sounds of virtual objects and the like and the action type of the user, and showing the association between the audio data and the degree of fatigue properties, which are stored in the storage unit 107 in advance.

The audio data selection unit 104 selects audio data corresponding to the user action type identified by the action identification unit 102, selects audio data according to the fatigue level calculated by the fatigue level calculation unit 103, and outputs the selected two pieces of audio data to the subsequent stage . In part (a) of Fig. 6, the file mb002.mp3 corresponding to "walking" and the file tc001.mp3 corresponding to "large fatigue" are selected. The selected pieces of audio data are synthesized by the audio information generating unit 108 .

As another example of the audio data selection by the audio data selection unit 104, a table shown in part (b) of FIG. 6 in which the types of actions recognized by the action recognition unit 102 and calculated by the fatigue level may be prepared in advance The fatigue levels calculated by the unit 103 may be combined, and a process of selecting the synthesized audio data may be performed. In this case, for example, as audio data corresponding to "walking" and "great fatigue", "sound of breathing while walking" may be placed.

As another example of audio data selection by the audio data selection unit 104, there is an example in which the audio data selection unit 104 dynamically generates an audio data pattern, and the dynamically generated audio data is selected. In this case, the user's motion recognized by the motion recognition unit 102 is a parameter with continuous values, and the fatigue level calculated by the fatigue level calculation unit 103 is also a parameter with continuous values. As described above, the audio data selection unit 104 may dynamically generate audio data based on a parameter group including parameters of the degree of motion speed and the degree of fatigue from "walking" to "running".

In a case where audio data has been selected, the audio data selection unit 104 stores the paired time and selection result in the storage unit 107 (S105). The paired time and selection result are used by the sound image position calculation unit 106 or the like.

Next, the displacement calculation unit 105 calculates the displacement from an arbitrary time point (S106). The displacement calculated here represents the spatial displacement of the information processing apparatus 1 . Since the premise is that the user wears the information processing apparatus 1, hereinafter, the displacement calculated by the displacement calculation unit 105 will be referred to as "user displacement". User displacement includes relative changes in the user's spatial position, as well as orientation, horizontal position, vertical position, and displacement thereof.

For example, the displacement of the orientation can be calculated by integrating the output of the gyro sensor among the pieces of sensor information input from the sensor 101 . In addition, there is a method of acquiring the absolute orientation from the output of the geomagnetic sensor. To compensate for the accuracy of the geomagnetic sensor, the output of the gyro sensor can be integrated. Through these methods, the displacement calculation unit 105 calculates the orientation in the user's displacement (the user's orientation, the front direction).

The displacement calculation unit 105 also calculates the displacement of the horizontal position as one of the user displacements. The displacement of the horizontal position can be calculated by receiving absolute position measurement of radio waves of GPS satellites, or by performing wireless communication with a plurality of base stations to determine the absolute position. As another method, the displacement calculation unit 105 calculates the displacement of the relative position from a point in time based on the travel distance and the travel direction (the above-described displacement of the orientation). Here, there is a method of obtaining the travel distance by integrating the output value of the acceleration sensor.

Further, the travel distance can be obtained by detecting the walking step from the change in acceleration, and multiplying the step length corresponding to the walking step by the number of steps. In this case, as the step size, the average step size is fixedly used, or the step size is set by, for example, calculating the average step size of the user from the relationship between the horizontal movement distance and the number of steps.

The displacement calculation unit 105 also calculates the displacement of the height as one of the user displacements. The displacement in the height direction (vertical direction) can be calculated by the method of using the measurement value of the atmospheric pressure sensor or the method of calculating the displacement of the height corresponding to the case in which it is recognized that the action recognition unit 102 The user action type identified as "standing" and the user action type identified as "sit" are alternately repeated. Note that "standing"/"sitting" can be identified based on the change pattern of the measurement value of the acceleration sensor.

The user displacement calculated by the displacement calculation unit 105 is used by the sound image position calculation unit 106 . The sound image position calculation unit 106 calculates the relative position of the virtual object viewed from the user (S107). This position is a perceptual sound source (position in real space) that the user feels due to the final synthesized audio in the case of outputting audio from the audio output unit 109 .

As the information processing of the sound image position calculation unit 106, an appropriate one can be selected according to the character this application desires to give to the virtual object. In this case, depending on what character the virtual object is or what the virtual object is, several modes are set in the calculation method performed by the sound image position calculation unit 106 . In the following, two typical modes will be described with reference to FIG. 7 . FIG. 7 is a schematic diagram describing the information processing of the sound image position calculation unit 106, and shows the displacement of the user and the displacement of the virtual object in each mode.

Part (a) of FIG. 7 shows a sound image position calculation mode in which the movement of the virtual object tracks the same position as the position displaced by the user with a certain time delay. In this figure, the vertical axis is a one-dimensional representation of six-dimensional information including three-axis position and three-axis orientation, as an example of displacement. The horizontal axis represents time t.

The sound image position calculation mode that can track the user's displacement shown in part (a) of FIG. 7 can be realized by, for example, the following formula. However, X(t) represents the user displacement, X'(t) represents the displacement of the virtual object, and K represents the time delay before the virtual object starts to move. The larger the value of K, the larger the time (delay) before the virtual object starts to move.

X'(t)=X(t-K)

In the case of presenting a presence that moves with the user's movement, the tracking of the user's displacement by the virtual object shown in part (a) of FIG. 7 is effective. For example, in the case where it is desired to provide the user with augmented reality in which the virtual object is a person, robot, car, animal, etc., that is, in the case where the sound image localization position tracks the same position as the position displaced by the user with a certain delay, it is possible to This mode of panning position calculation is used.

Part (b) of FIG. 7 shows a sound image position calculation mode in which the virtual object moves with a certain time delay relative to the user displacement to go directly to the position where the user exists. This sound image position calculation mode can be realized by, for example, the following formula. However, a represents the moving speed of the virtual object. The closer the value of a, the longer it takes to catch up with the user. That is, move slowly.

X'(t)=aX(t-K)+(1-a)X'(t-1)

Tracking the user's displacement by the virtual object shown in part (b) of Fig. 7 is effective, for example, in the case of presenting a presence that follows the user through a wall. For example, it is suitable for representing ghost characters as virtual objects.

The sound image position calculation unit 106 uses the displacement X'(t) calculated by the above-described information processing or the point at time t that can be calculated therefrom, as the position of the virtual object at time t. Note that this point can be used as a base point, and a point obtained by adding a predetermined position change to this base point can be used as the position of the virtual object. For example, in the case where the virtual object is a dog character, a point obtained by moving the base point calculated by the calculation to a lower position closer to the ground is output. Alternatively, in the case where the virtual object is a ghost character, in order to generate a floating feeling, calculation is performed so as to add upward and downward positional changes at regular intervals. According to this configuration, more realistic character movement can be reproduced.

In addition, the audio-visual position calculation unit 106 calculates the relative position of the character viewed from the user in consideration of the orientation displacement of the user with respect to the position of the character (X'(t)-X(t)) with the user's position as the starting point Location. As the sound image localization method, the method described in Patent Document 1 can be used.

The audio information generating unit 108 performs information processing for spatially arranging the audio information ( S108 ). The information processing locates the sound image at a relative position to the user, eg, a distance or a direction from the user centered on the user, and the method described in Patent Document 1 can be used, for example.

The audio information generation unit 108 uses the audio data selected by the audio data selection unit 104 as audio information to be used for output. However, in the case where there is a delay in the sound image position calculation, the selected audio data when referring to the user position is used. That is, in the case where the sound image position is calculated by the calculation formula X'(t)=X(t-K), the audio data selected at the time (t-K) is used at the time t. The audio information generation unit 108 extracts and uses the audio data stored in the storage unit 107 in association with the time information through the audio data selection unit 104 .

Further, the audio information generation unit 108 specifies the position calculated by the sound image position calculation unit 106 based on the user displacement output by the displacement calculation unit 105 as the perceptual sound source position (the sound image is localized) that the user feels due to the output audio information. s position). The audio information generating unit 108 modulates the audio data so that it is heard from a designated position. In this embodiment, it is generated as 2-channel audio information. However, according to the specific implementation of the audio output unit 109, it may be generated as 5.1 channel audio information.

Further, the audio information generating unit 108 can adjust the reproduction speed of the modulated audio data according to the moving speed of the user calculated based on the user's displacement calculated by the displacement calculating unit 105 . For example, even in the case where the audio data selected by the audio data selection unit 104 is also audio data corresponding to "walking", reproduction is performed at different reproduction speeds depending on the difference in moving speed.

Next, the audio output unit 109 physically outputs the audio data generated by the audio information generation unit 108 as sound waves.

<4. Conclusion>

According to the above-described embodiment, by recognizing the action pattern of the user and switching audio data to be reproduced based thereon, it is possible to generate an expression that follows a change in the action pattern of the user. In addition, by changing the sound image localization position based on changes in three-dimensional position or user orientation, it is possible to generate a sound representation that follows the result of the user's actions or the position the user occupies in space. In addition, since the sound image localization position or its base point is moved with a predetermined delay with respect to the user's action, it is possible to generate a sound expression that follows the position change of the user. As described above, according to the above-described embodiments, in the case where a virtual character (eg, a dog) is set as a virtual object, a voice expression as if the virtual character actually exists in the vicinity of the user is realized.

<5. Other Embodiments>

Note that the present technology can also take the following configurations.

Although the neck-mounted speaker hung around the neck is shown as an appearance configuration example in FIG. 3 , the technology disclosed in the above-described embodiment can also be applied to other embodiments, for example, a head-mounted display including a glasses-type display. In this case, it is also advantageous to present an image at the position of the virtual object output by the information processing of the displacement calculation unit 105 and the sound image position calculation unit 106 . Synergy can be achieved by adding visual stimuli to auditory stimuli according to the present technology, and augmented reality with higher quality can be provided to the user.

<5-1. Another embodiment 1>

A configuration according to another embodiment 1 of the present technology will be described with reference to FIG. 8 . In this embodiment, it is assumed that there are multiple users of the information processing apparatus 1 . In this embodiment, there are two kinds of sensor information to be input to the CPU 100 . One or more sensors that output sensor information to be used in information processing in which the displacement calculation unit 105 calculates the user's displacement is set as the sensor 101, and one or more sensors that output sensor information to be used in motion recognition and fatigue calculation A plurality of sensors are set as other human sensors 110 .

The displacement calculation unit 105 calculates the user displacement of the user who perceives the audio output by the audio output unit 109 . The action recognition unit 102 and the fatigue level calculation unit 103 recognize the action pattern and the fatigue level of the other person who is not the user based on the sensor information of the other person. Other information processing is similar to that in the above-described embodiment.

According to this embodiment, the motion of the further user is recognized, the audio data according to the motion pattern is selected, and the position in the real space where the sound image is located follows the spatial displacement of the user listening to the audio. According to this embodiment, the user may be provided with an augmented reality in which an avatar of another person follows the user. The sound image position calculation unit 106 may set the sound image localization position in the air to create a feeling as if the virtual avatar of another person is floating.

This embodiment can be applied to, but is not limited to, running applications where the user can compete against another person running at a distance. Alternatively, this embodiment may be applied to an application where a user experiences another person's experience. For example, by applying this embodiment to a head mounted display in which a user experiences another person's field of view, an augmented reality can be provided to the user in which he/she tracks the movement of a distant athlete.

<5-2. Another embodiment 2>

For execution, the information processing described in the above-described embodiments does not depend on the hardware and software configurations shown in the above-described embodiments. The present technology may be implemented in a form in which some or all of the functional blocks shown in FIG. 4 are executed on separate hardware. As shown in FIG. 9 , this embodiment is an embodiment in which the information processing apparatus 1 is configured as a server client system in which the server 2 including the CPU 100 and the storage unit 107 and the wearable device 3 are connected via a network 4 communicate with each other.

In this embodiment, the neck-mounted speaker shown in FIG. 3 can be used as the wearable device 3 . In addition, a smartphone can be used as an example of the wearable device 3 . The server 2 is placed in the cloud, and information processing according to the present technology is performed on the side of the server 2 . The present technology may also be implemented in such a form.

<5-3. Another embodiment 3>

In the above-mentioned embodiment, the action recognition unit 102 recognizes the action type of the user in S102 of FIG. 5 . In this embodiment, when a change in the action type of the user is recognized here, the audio data selection unit 104 selects the audio data corresponding to the audio data switching pattern from the audio data before the change to the audio data after the change and the change After the audio data both.

The timing of switching the audio data is the timing of the start or the end of the action. For example, when the character to be provided to the virtual object is "a bell-tethered character" and the action type is changed from "running" to "still", the audio data selection unit 104 selects a jingle sound that sounds like a jingle. The audio data selection unit 104 selects audio data corresponding to "still" and audio data on jingle sound.

Such a configuration in which audio data is selected according to a change in motion of a character corresponding to a virtual object can provide a user with a more entertaining augmented reality or a more realistic augmented reality.

In addition to the jingle sound, when the action type changes from "running" to "still", the audio data selection unit 104 may select a dialogue indicating that the character is surprised. In this case, the following effect can be produced: the character of the virtual object is surprised when the running user suddenly stops. Characters can be brought to life and users can be provided with more entertaining augmented reality.

<5-4. Another embodiment 4>

In the above-mentioned embodiment, the action recognition unit 102 recognizes the action type of the user in S102 of FIG. 5 . In this embodiment, when a change in the user action type is recognized here, the CPU 100 performs predetermined condition determination related to the changed action type. In this condition determination, it is determined whether the changed action type matches the information associated with the virtual object.

In the above-described embodiment or this embodiment, the user is provided with augmented reality (AR) as if the virtual object follows the user. In this regard, the audio emitted from the virtual object is also changed, if possible, according to the type of action of the user. The information processing apparatus 1 gives a personality, characteristic, belonging, etc. to a virtual object as a character that does not exist. In cases where this information associated with the virtual object does not match the changed action type, augmented reality is degraded.

In this regard, in this embodiment, whether the changed action type matches the information associated with the virtual object. The audio data selection unit 104 may select predetermined audio data when performing condition determination on matching.

For example, in the case where the action type is changed from "walking" to "cycling", the conditional determination of whether or not the belonging of the character of the virtual object includes "bicycle" is performed. In this example, the conditional determination of whether the belonging includes a "bicycle" corresponds to a determination of whether the changed action type matches the information associated with the virtual object.

In the case where the information associated with the virtual object (the character's possession) does not include a bicycle as a result of the determination in this example, that is, in the case where the character does not have "bicycle", the audio data selection unit 104 does not select the corresponding bicycle audio data. Instead, you can choose the voice where the character whispers "I want to ride a bike, too."

Such a configuration in which the speaking timing of the character according to the virtual object is controlled, can provide the user with a more entertaining augmented reality or a realistic augmented reality.

<5-5. Another embodiment 5>

In the above-described embodiment, the user's fatigue or the degree of fatigue is calculated as an example of the user's state, and the fatigue or the degree of fatigue is used to select audio data. However, as another example of the user's state, the user's emotions (eg, happy, angry, sad, and happy emotions) may be obtained through the sensor 101, and audio data may be selected based on the emotions. The sensor 101 is not particularly limited. As long as the user's emotion can be obtained from the blood pressure or body temperature through the biosensing device, the audio data may be selected based on the emotion.

Furthermore, instead of, or in addition to, the user's state or mood, environmental information around the user may be obtained, and audio information may be selected based on the environmental information. For example, in the case where rainfall is detected as environmental information, the audio data selection unit 104 selects the sound of walking on puddles according to this. According to this configuration, more entertaining augmented reality or realistic augmented reality can be provided to the user.

<5-6. Another embodiment 6>

The sound image position calculation unit 106 may determine the position where the virtual object is placed based on one or more of the following combinations.

· Information obtained from sensor 101

・Information obtained from outside (map data, etc.)

· Information about personalities or possessions given to virtual objects

For example, in the case where the information processing apparatus 1 can obtain detailed map data, and the user's absolute position based on the user's displacement calculated by the displacement calculating unit 105 is near a wall of a building, the audio-visual position calculating unit 106 places a virtual object such that the The virtual object is not positioned across the wall towards the user. For example, in the case where the virtual object is a character such as a dog, since it is unnatural if the character enters the other side of the wall during walking, if detailed map data can be obtained, the sound image position calculation unit 106 places a virtual The object causes the virtual object to turn to the user's side.

The map data that can be obtained from the outside includes not only the approximate latitude and longitude of the building, but also the positional coordinates of the wall dividing the boundary between the building and the road or the like. When the information processing apparatus 1 can use such map data, the building can be regarded as a range surrounded by the position coordinates of the wall. In this regard, the sound image position calculation unit 106 sets the sound image localization position of the virtual object determined based on the user displacement output by the displacement calculation unit 105 in the range excluding the coordinate position of the building. Specifically, for example, a virtual object is placed on one side of a road. Alternatively, the sound image position calculation unit 106 places the virtual object along a direction in which the space is open (eg, a direction in which a road exists).

Also, in this example, in the case where a virtual object collides with an object such as a wall of a building on the map data, audio data such as a collision sound can be reproduced. For example, in the case where the position of the virtual object placed by the sound-image position calculating unit 106 overlaps the coordinates of the position range of the building, the audio information generating unit 108 reproduces audio data such as collision sound.

Note that the character given to the virtual object is a character (eg, a ghost) that can pass through a wall, and the sound image position calculation unit 106 can place the character on the other side of the wall. Additionally, a specific jingle sound can be played when the ghost walks through the wall.

According to the configuration of this embodiment, more entertaining augmented reality or realistic augmented reality can be provided to the user.

<5-7. Another embodiment 7>

In this embodiment, in addition to the configuration disclosed in the above-described embodiment, the audio information generating unit 108 has a configuration of generating different audio information according to the action state of the character of the virtual object. For example, when the motion recognition unit 102 recognizes that the user's motion pattern is running for a long time exceeding a predetermined threshold, the audio data selection unit 104 may select different audio data, and the audio information finally generated by the audio information generation unit 108 may different. In this case, the audio information generation unit 108 may switch the audio data to be selected by the audio data selection unit 104 from normal audio data while running to, for example, audio data indicating that the character is tired, which includes breathing, for example Short voice or voice "tired".

According to such a configuration, it is possible to generate an effect that the character of the virtual object also has an action state (tiredness, boredom, etc.), and more entertaining augmented reality or realistic augmented reality can be provided to the user.

<6. Appendix>

Parts of the technical idea disclosed in this specification can be described as the following (1) to (11).

(1)

An information processing device, comprising:

a motion recognition unit configured to recognize a user's motion pattern based on sensor information;

an audio data selection unit configured to select audio data corresponding to the motion pattern identified by the motion recognition unit; and

An audio information generation unit that generates multi-channel audio information for localizing a sound image of a sound source in a real space around the user based on the audio data selected by the audio data selection unit.

(2)

The information processing apparatus according to (1) above, wherein

the audio data selection unit is configured to select the audio data as audio emitted from a virtual object to be placed in the real space, and

The audio information generating unit is configured to perform sound image localization by generating the multi-channel audio information, by which the virtual object is placed at the position of the sound source.

(3)

The information processing apparatus according to (1) or (2) above, wherein

The audio data selection unit is configured to select an audio data switching mode corresponding to the audio data before the change to the audio data after the change when the audio data to be selected is changed as a result of the recognition by the action recognition unit the audio data and the changed audio data.

(4)

The information processing apparatus according to any one of (1) to (3) above, wherein

The audio data selection unit is configured to, when audio data to be selected is changed as a recognition result of the motion recognition unit, select the audio data corresponding to the user's motion pattern in the presence of a plurality of pieces of audio data. Audio data matching the information associated with the virtual object.

(5)

The information processing device according to any one of (1) to (4) above, further comprising:

A displacement calculation unit that outputs a user displacement including a relative change in the position of the user based on the sensor information.

(6)

The information processing apparatus according to (5) above, wherein:

The audio information generation unit is configured to: modulate the audio data selected by the audio data selection unit based on the user displacement output by the displacement calculation unit, thereby generating the multi-channel audio information.

(7)

The information processing apparatus according to the above item (6), wherein,

The audio information generation unit is configured to modulate the audio data selected by the audio data selection unit such that a sound source whose sound image is localized by the multi-channel audio information is placed in accordance with the output of the displacement calculation unit. at the position displaced by the user, thereby generating the multi-channel audio information.

(8)

The information processing apparatus according to (7) above, wherein:

The audio information generating unit is configured to generate the multi-channel audio information such that a sound source whose sound image is located by the multi-channel audio information follows a position in space with a time delay, the position is The position of the user identified by the user displacement begins.

(9)

The information processing apparatus according to any one of (5) to (8) above, wherein

The audio information generation unit generates the multi-channel audio information based on the user displacement output by the displacement calculation unit and map information including position coordinates of buildings obtained from the outside so that the virtual object does not is placed in the range of the location coordinates of the building included in the map information.

(10)

The information processing apparatus according to (9) above, wherein:

The audio information generating unit generates the multi-channel audio information including the collision sound in a case where the range of the position coordinates of the building included in the map information overlaps with the position where the virtual object is placed.

(11)

The information processing device according to any one of (1) to (10) above, further comprising:

A state analysis unit configured to analyze the state of the user, the state being changeable according to one of the sensor information and the motion pattern of the user recognized by the motion recognition unit.

(12)

The information processing apparatus according to (11) above, wherein:

The audio data selection unit is configured to select audio data corresponding to the action pattern of the user and audio data corresponding to the state of the user analyzed by the state analysis unit.

(13)

The information processing apparatus according to (12) above, wherein:

The audio information generating unit is configured to synthesize the audio data corresponding to the action pattern of the user selected by the audio data selecting unit and the audio data corresponding to the state of the user, thereby based on the synthesized audio data The multi-channel audio information is generated.

(14)

The information processing apparatus according to any one of (11) to (13) above, wherein,

The state analysis unit is configured to assign fatigue per unit time according to one of the sensor information and the action pattern of the user identified by the action identification unit, and to accumulate the assigned fatigue per unit time degree, so as to calculate the fatigue degree as the state of the user.

(15)

The information processing apparatus according to any one of (1) to (14) above, wherein,

The audio data selection unit selects audio data different from the audio data corresponding to the motion pattern of the user recognized by the motion recognition unit if the motion pattern of the user recognized by the motion recognition unit continues to exceed a predetermined threshold audio data.

(16)

An information processing method, comprising:

Action recognition step: based on sensor information to identify the user's action pattern;

Audio data selection step: selecting audio data corresponding to the motion pattern of the user identified by the motion recognition step; and

Audio information generating step: generating multi-channel audio information for localizing a sound image of a sound source in a real space around the user based on the audio data selected by the audio data selecting step.

(17)

A program that causes a computer to perform the following steps:

Action recognition step: based on sensor information to identify the user's action pattern;

Audio data selection step: selecting audio data corresponding to the motion pattern of the user identified by the motion recognition step; and

Audio information generating step: generating multi-channel audio information for localizing a sound image of a sound source in a real space around the user based on the audio data selected by the audio data selecting step.

List of reference signs

1 Information processing device

100 CPUs

101 Sensors

102 Action Recognition Unit

103 Fatigue calculation unit (state analysis unit)

104 Audio data selection unit

105 Displacement calculation unit

106 Sound image position calculation unit

107 storage unit

108 Audio information generation unit

109 Audio output unit

110 Other people sensor

Claims (17)

1. a kind of information processing unit, comprising:

Action recognition unit is configured to identify the action mode of user based on sensor information；

Audio data selecting unit is configured to select the movement mould of the user identified with the action recognition unit The corresponding audio data of formula；And

Audio-frequency information generation unit is generated based on the audio data selected by the audio data selecting unit in institute State the multichannel audio information positioned in the real space around user to the acoustic image of sound source.

2. information processing unit according to claim 1, wherein

The audio data selecting unit is configured to select the audio data as from being placed in the real space Virtual objects issue audio, and

The audio-frequency information generation unit is configured to execute Sound image localization, the void by generating the multichannel audio information Quasi- object is placed at the position of the sound source by the Sound image localization.

3. information processing unit according to claim 1, wherein

The audio data selecting unit is configured to: in the recognition result as the action recognition unit, the sound to be selected When frequency evidence is changed, selection and the audio data switch mode pair from the audio data after the audio data to variation before variation Audio data after the audio data answered and variation.

4. information processing unit according to claim 1, wherein

The audio data selecting unit is configured to: in the recognition result as the action recognition unit, the sound to be selected When frequency evidence is changed, selection and and institute in the case where there is a plurality of audio data corresponding with the action mode of the user State the audio data that the associated information of virtual objects matches.

5. information processing unit according to claim 1, further includes:

It is displaced computing unit, the user of the opposite variation of the position including the user is exported based on the sensor information Displacement.

6. information processing unit according to claim 5, wherein

The audio-frequency information generation unit is configured to: being displaced, is adjusted based on the user by the displacement computing unit output The audio data selected by the audio data selecting unit is made, to generate the multichannel audio information.

7. information processing unit according to claim 6, wherein

The audio-frequency information generation unit is configured to modulate the audio data selected by the audio data selecting unit, so that Acoustic image is placed on by the sound source that the multichannel audio information positions to be followed described in the displacement computing unit output At the position of user's displacement, to generate the multichannel audio information.

8. information processing unit according to claim 7, wherein

The audio-frequency information generation unit is configured to generate the multichannel audio information, so that acoustic image passes through the multichannel Audio-frequency information and the sound source that positions follow the position in space with a time delay, and the position is displaced mark from by the user The position of the user starts.

9. information processing unit according to claim 5, wherein

The audio-frequency information generation unit is obtained based on the user displacement by the displacement computing unit output and from outside The cartographic informations of the position coordinates including building generate the multichannel audio information so that the virtual objects are not It is placed in the cartographic information in the range of the position coordinates for the building for including.

10. information processing unit according to claim 9, wherein

The range of the position coordinates for the building that the audio-frequency information generation unit includes in the cartographic information and institute It states in the case that the position that virtual objects are placed overlaps and generates the multichannel audio information including impact sound.

11. information processing unit according to claim 1, further includes:

State analysis unit, is configured to analyze the state of the user, and the state can be according to the sensor information And one in the action mode of the user that is identified of the action recognition unit and change.

12. information processing unit according to claim 11, wherein

The audio data selecting unit be configured to select corresponding with the action mode of user audio data and with The corresponding audio data of state for the user that the state analysis unit is analyzed.

13. information processing unit according to claim 12, wherein

The audio-frequency information generation unit is configured to the audio data selecting unit is selected dynamic with the user The corresponding audio data of operation mode and the audio data for the state for corresponding to the user synthesize, thus the sound based on synthesis Frequency is according to the generation multichannel audio information.

14. information processing unit according to claim 11, wherein

The state analysis unit is configured to the institute identified according to the sensor information and the action recognition unit One in the action mode of user is stated to distribute fatigue strength per unit time, and accumulate distributed per unit time Fatigue strength, to calculate state of the fatigue strength as the user.

15. information processing unit according to claim 1, wherein

The audio data selecting unit is continued above in the action mode for the user that the action recognition unit is identified The audio with the action mode for corresponding to the user that the action recognition unit is identified is selected in the case where predetermined threshold The different audio data of data.

16. a kind of information processing method, comprising:

Action recognition step: the action mode based on sensor information identification user；

Audio data selects step: selection sound corresponding with the action mode of the user that the action recognition step is identified Frequency evidence；And

Audio-frequency information generation step: it based on the audio data by audio data selection step selection, generates for described The multichannel audio information that the acoustic image of sound source is positioned in real space around user.

17. a kind of program makes computer execute following steps:

Action recognition step: the action mode based on sensor information identification user；

Audio data selects step: selection sound corresponding with the action mode of the user that the action recognition step is identified Frequency evidence；And

Audio-frequency information generation step: it based on the audio data by audio data selection step selection, generates for described The multichannel audio information that the acoustic image of sound source is positioned in real space around user.