CN116965060A - Audio system using optical microphone - Google Patents

Abstract

An audio system includes an optical microphone and an audio controller. The optical microphone includes a light source and a detector. In some embodiments, the light source illuminates the user's skin. Alternatively, the optical microphone further includes a membrane, and the light source illuminates a portion of the membrane. Sound from a local area causes vibrations in the skin (or vibrations in the membrane). The detector can be in an interference configuration or a non-interference configuration with the light source. The audio controller uses the signal output from the detector to monitor vibrations of the skin (or membrane) and uses the monitored vibrations to make sound measurements.

Description

Audio systems using optical microphones

Cross-references to related applications

This application claims the benefit of U.S. Provisional Application No. 63/126,669, filed on December 17, 2020, which is incorporated by reference in its entirety.

Technical field

The present disclosure relates generally to audio systems, and more particularly, to audio systems using optical microphones.

Background technique

In noisy environments (e.g., noisy restaurants), conventional audio systems may have difficulty selectively capturing sounds from target sound sources (e.g., speakers, the user's own speech, etc.). Whether the user is speaking affects the selective capture of sounds. But in noisy environments, audio systems often cannot distinguish between user speech and ambient noise. Conventional audio systems attempt to alleviate this problem using voice activity detectors that rely on the temporal and spectral characteristics of the wearer's speech (eg, detected via conventional microphones) that is audible among interfering sounds. However, in low signal-to-noise ratio (SNR) environments (i.e., noisy environments), this method often fails because the wearer's voice is completely masked by the noise.

Contents of the invention

Accordingly, the present invention discloses audio systems, methods, computer readable media and computer programs in accordance with the appended claims. Audio systems use optical microphones. The audio system includes an audio controller and may also include a microphone array. In some embodiments, the audio system may be part of a headset, necklace, watch, hearable device, etc. Optical microphones include light sources and detectors. The light source is configured to emit light. The light includes a reference beam and a sensing beam, and the light source is configured to illuminate the user's skin with the sensing beam. Sound from a local area (eg, the user's voice, other people, etc.) causes vibrations in the user's skin. The detector is in an interference configuration with the light source (eg, self-mixing, low coherence interferometry (LCI), etc.) such that the detector is configured to detect mixed signals. This mixed signal corresponds to the reference beam mixed with the portion of the sensing beam that is reflected by the skin. The audio controller is configured to use the signal to make measurements of sound in the local area.

In an embodiment of the audio system according to the present invention, the audio system includes: an optical microphone including a light source and a detector, the light source being configured to emit light including a reference beam and a sensing beam, the light source being configured To illuminate the user's skin with a sensing beam in which sound from a localized area induces vibrations in the skin, the detector is in an interference configuration with the light source such that the detector is configured to detect a mixed signal corresponding to a reference beam mixed with the portion of the sensing beam reflected by the skin; and an audio controller configured to use the mixed signal to measure the sound.

In embodiments of the audio system according to the present invention, the interference configuration may be such that the light source and detector form at least one of the following: a self-mixing interferometer, a Michelson interferometer, a low-coherence interference system, a laser Doppler vibrometer ( laser doppler vibrometer (LDV) or some other type of interferometric system.

In an embodiment of the audio system according to the present invention, the light source may be adjacent to the detector, and the optical microphone may further comprise a lens coupled to the light source, the lens may be configured to split light emitted from the light source into a reference beam. and a sensing beam; and directing the sensing beam to the skin and reflecting the reference beam to the detector.

In an embodiment of the audio system according to the present invention, the system may further include: a second optical microphone including a second light source and a second detector, the second light source being configured to emit light, the light Comprising a second reference beam and a second sensing beam, the light source is configured to illuminate the user's skin with the sensing beam, and the second detector is in an interference configuration with the second light source such that the second detector is configured to detecting a second mixed signal corresponding to a second reference beam mixed with a portion of the second sensing beam reflected by the skin; a block including a first side and a second side, the first side coupled to the optical microphone, and the second side coupled to the second optical microphone; and a lens coupled to the light source, the block, and the second light source, the lens configured to split light emitted from the light source into a reference beam and sensing the light beam, dividing the light emitted from the second light source into a second reference beam and a second sensing beam, directing the sensing beam and the second sensing beam to the skin, reflecting the reference beam to the detector, and The two reference beams are reflected to the second detector.

In embodiments of the audio system according to the invention, the detector and the light source may be separated from each other by a threshold distance.

In embodiments of the audio system according to the present invention, the optical microphone may be located on a headset including a nose pad, and the optical microphone may be integrated into the nose pad, and the light source may be configured to illuminate the user with the sensing beam nose skin.

In an embodiment of the audio system according to the present invention, the optical microphone may be located on a head-mounted viewer including a spectacle frame, and the optical microphone may be integrated into the spectacle frame, and the light source may be configured to illuminate the user with the sensing beam skin on the face. In addition, the audio system may further include a second optical microphone, which may be integrated into the spectacle frame at a different location than the first optical microphone, which may be configured to use the second sensing beam. A portion of the skin on the user's face that is different from the optical microphone is illuminated.

In an embodiment of the audio system according to the present invention, the optical microphone may be located on the headset, and the audio system may further include: a microphone array located on the headset, the microphone array being configured to Sound from the local area is detected; wherein the audio controller is further configured to use the detected sound to calibrate the optical microphone.

In an embodiment of the audio system according to the present invention, the optical microphone may be located on the headset, and the audio system may further include: a microphone array located on the headset, the microphone array being configured to Sound from a local area is detected; wherein the audio controller may also be configured to enhance the measured sound based in part on the detected sound.

In embodiments of the audio system according to the present invention, the audio controller may be further configured to determine an expression of the user's face based in part on the measured sound.

In an embodiment of the audio system according to the present invention, the optical microphone may be located on the headset, and the audio controller may be further configured to: identify noise in the measured sound; generate a sound filter to suppress the The identified noise, and applying a sound filter to modify the audio signal corresponding to the audio content, wherein the audio system may further include: a transducer array integrated into the headset, the transducer The array is configured to present the modified audio signal to the user as modified audio content, the modified audio content including the audio content and a suppression component that suppresses noise.

In an embodiment of the audio system according to the present invention, wherein the optical microphone may be located on the head-mounted viewer, and the audio system may further include: a microphone array located on the head-mounted viewer, the microphone array may Configured to detect sounds from a local area, where the sounds from the local area may include the voice of a user of the audio system; wherein the audio controller may also be configured to: use the measured sounds to identify among the detected sounds the user's voice; and updating the sound filter based on the recognized user's voice, wherein the updated sound filter can be used to modify the audio content, and the modified audio content can be presented by at least one audio system. In addition, the updated sound filter can also enhance the user's voice, and the audio controller can also be configured to: modify the audio content with the updated filter, wherein the modified audio content enhances the user's voice ; and providing the modified audio content to a second audio system, wherein the second audio system presents the modified audio content.

In an embodiment of the audio system according to the present invention, wherein the optical microphone may be located on the head-mounted viewer, and the audio system may further include: a microphone array located on the head-mounted viewer, the microphone array may Configured to detect sounds from a local area, where the sounds from the local area may include the voice of a user of the audio system; wherein the audio controller may also be configured to: use the measured sounds to identify among the detected sounds the user's voice; and updating the sound filter based on the recognized user's voice, wherein the updated sound filter can be used to modify the audio content, and the modified audio content can be presented by at least one audio system. In addition, the updated sound filter can enhance the user's voice, and the audio controller can also be configured to: modify the audio content with the updated filter, wherein the modified audio content can enhance the user's voice; Determining that the modified audio content includes a command; and performing an action based on the command.

In some embodiments, a method for using an optical microphone as part of an audio system is described. Light is emitted from the optical microphone's light source. The emitted light includes a reference beam and a sensing beam. Sound from a local area (eg, the user's voice, other people, etc.) causes vibrations in the user's skin. The user's skin (eg, part of the face) is illuminated with a sensing beam. A detector in an interference configuration with the light source detects the mixed signal. This mixed signal corresponds to the reference beam mixed with the portion of the sensing beam that is reflected by the skin. Use this mixed signal to make measurements on the sound.

In an embodiment of the method according to the present invention, the method includes: emitting light from a light source of an optical microphone, the light including a reference beam and a sensing beam; irradiating the user's skin with the sensing beam, wherein the sound from the local area is inducing vibrations in the skin; detecting a mixed signal corresponding to a reference beam mixed with a portion of the sensing beam reflected by the skin by a detector in an interference configuration with the light source; and measuring the sound using the mixed signal.

In an embodiment of the method according to the present invention, the vibration of the skin may be caused in part by the user's voice, and the method may further include: detecting sound from a local area through a microphone array; using the measured sound to detect identifying the user's voice in the voice; and updating the voice filter based on the recognized voice of the user, wherein the updated voice filter can be used to modify the audio content, and the modified audio can be presented by at least one audio system content.

In an embodiment of the method according to the invention, wherein the interference configuration may be such that the light source and the detector form at least one of the following: a self-mixing interferometer, a Michelson interferometer, a low-coherence interference system, a laser Doppler vibrometer or some other type of interference system.

In an embodiment of the method according to the present invention, the measured sound may include the user's voice, and the high-frequency component of the voice may be attenuated relative to the low-frequency component of the voice, and the method may further include: performing a reconstruction; and updating the measured sound of the speech with the reconstructed high-frequency components.

In some embodiments, a non-transitory computer-readable medium configured to store program code instructions is described. The instructions, when executed by the processor of the audio system, cause the audio system to perform steps of the above methods and/or other methods described herein.

In an embodiment of a computer-readable medium in accordance with the present invention, the non-transitory computer-readable medium is configured to store program code instructions that, when executed by a processor of an audio system, cause the audio The system performs steps including: emitting light from a light source of an optical microphone, the light including a reference beam and a sensing beam; irradiating the user's skin with the sensing beam, wherein the sound from the local area induces vibrations in the skin; and A detector with the light source in an interference configuration detects a mixed signal corresponding to a reference beam mixed with a portion of the sensing beam reflected by the skin; and the sound is measured using the mixed signal.

In some embodiments, a computer program is described. The computer program includes instructions that, when executed by a processor of the audio system, cause the audio system to perform the steps of the methods described above and/or other methods described herein.

In some embodiments, the detector and light source are not in an interference configuration, and the optical microphone measures localized sound based on intensity modulation of light reflected and/or scattered from the skin (non-interference).

Note that in some embodiments, the optical microphone also includes a membrane, and the light source illuminates the membrane and/or a portion of the diaphragm with a sensing beam (or, more generally, light from the light source) instead of illuminating the skin. In these embodiments, sound from a localized area causes the membrane to vibrate. Optical microphones therefore measure sound in a local area by monitoring the vibrations of the membrane caused by the sound coming from the local area. In this embodiment, the light source and detector may be in an interference configuration or a non-interference configuration.

Description of the drawings

1A is a perspective view of a head-mounted viewer implemented as an eyewear device and including at least one optical microphone in accordance with one or more embodiments.

FIG. 1B is a perspective view of a head-mounted display (HMD) implemented as a head-mounted display (HMD) including at least one optical microphone in accordance with one or more embodiments.

Figure 2 is a block diagram of an audio system in accordance with one or more embodiments.

Figure 3 is an example nose pad including an optical microphone having a light source and detector located at different locations within the nose pad, in accordance with one or more embodiments.

Figure 4 is an example optical microphone configured as a self-mixing interferometer in accordance with one or more embodiments.

Figure 5A is an optical microphone configured as a self-mixing interferometer with multiple components of the optical microphone in a series configuration in accordance with one or more embodiments.

Figure 5B is an optical microphone configured as a self-mixing interferometer with multiple components of the optical microphone in a parallel configuration in accordance with one or more embodiments.

Figure 5C is an example of a pair of optical microphones including two self-mixing interferometers according to one or more environments.

Figure 6 is an example of an optical microphone configured as a laser Doppler vibrometer in accordance with one or more embodiments.

Figure 7 is an example of an optical microphone configured to use optical coherence tomography (OCT) in accordance with one or more embodiments.

8 is a flowchart illustrating a process for using an optical contact transducer in an interference configuration in accordance with one or more embodiments.

Figure 9 is a system including a head-mounted viewer in accordance with one or more embodiments.

The drawings depict various embodiments for purposes of illustration only. Those skilled in the art will readily recognize from the following discussion that alternative embodiments of the structures and methods illustrated herein may be employed without departing from the principles described herein.

Detailed ways

An audio system including one or more optical microphones and an audio controller. In some embodiments, the audio system is part of a headset, and one or more optical microphones are positioned to detect sounds in a local area of the headset (e.g., the user's voice, other people's voices, Vibrations of the headset user's skin caused by noise sources, etc.) are monitored. One or more optical microphones may be positioned at one or more locations on the headset (eg, nose pads, eyeglass frames, etc.). The audio system can use the detected vibrations to measure sound in a local area and perform various actions based on the measured sound (e.g., active noise cancellation, user voice enhancement, voice activity detection (e.g., active noise cancellation), user voice enhancement, etc. voice activity detection, VAD), etc.). Note that although the above is with one or more optical microphones on the headset. In other embodiments, one or more optical microphones and/or audio systems may be located on other devices (eg, necklaces, smart watches, etc.).

Optical microphones monitor the vibrations of the user's skin. The optical microphone includes a light source and a detector. The light source (eg, vertical cavity surface emitting laser (VCSEL)) is configured to emit light. The emitted light has a band of light such that the skin primarily reflects the light (as opposed to, for example, primarily absorbing it, between 250 and 1800 nanometers (nm)). In some embodiments, the emitted light is a continuous wave and includes a reference beam and a sensing beam. Sound from localized areas of the audio system causes vibrations in the user's skin. The light source is configured to illuminate the user's skin (eg, a portion of the face) with the sensing beam.

The detector is in an interference configuration with the light source such that the detector is configured to detect the mixed signal. In an interference configuration, the optical microphone becomes an interferometer-based system, where constructive or destructive interference between the light from the laser and the reflected light provides a signal that changes with varying distances. For example, a wavelength of 850nm can provide accuracy down to the 10nm range or below. The interference configuration enables the light source and detector to form an interference system (eg, self-mixing interferometer, Michelson interferometer, low coherence interferometry (LCI), laser Doppler vibrometer (LDV), etc.). The mixed signal corresponds to the reference beam mixed with the portion of the sensing beam reflected by the skin.

The audio controller processes the information from the detector. The audio controller is configured to use the mixed signal to measure sound. The audio controller analyzes the mixed signal to measure some or all of the sound that causes vibrations in the user's skin. The sound may include: for example, the user's voice, and/or other sound sources in the local area (eg, other people, noise sources (eg, fans), etc.).

In some embodiments, the audio system may also include a microphone array. The microphone array is configured to detect sounds from a localized area. The sound from the local area may include, for example, the voice of a user of the audio system, sounds from other sound sources in the local area, or some combination thereof.

The audio system may perform various actions based in part on sound measured by one or more optical microphones, sound detected by a microphone array, or some combination thereof. Various actions may include, for example: enhancing the user's speech, using one or more optical microphones for voice activity detection (VAD), performing active noise reduction, capturing information for identifying the user's microexpressions, monitoring the headset's Positioning etc. Note that in some embodiments, some or all of the respective optical microphones may be coupled to a vibration attenuating structure that is coupled to the headset. The vibration attenuating structure reduces vibrations transmitted to the optical microphone from the headset (or, more generally, the device to which the optical microphone is coupled).

Note that in some embodiments, the optics include a membrane, and sound from the local area is monitored by monitoring vibrations of the membrane rather than monitoring vibrations of the user's skin. Sound from a local area causes the membrane to vibrate. In this embodiment, the light source is configured to illuminate a portion of the film, and the film scatters and/or reflects a portion of the light. The light source and detector can be in an interference configuration or a non-interference configuration. The detector detects scattered light and/or reflected light. The audio controller uses the signal from the detector to measure the sound coming from a local area.

Regular VADs don't work well in low SNR environments (e.g., noisy, crowded restaurants). These systems use microphones to detect sounds coming from a localized area and then attempt to separate the user's speech from the low SNR environment. However, in low SNR environments, this method often fails because the wearer's voice is completely masked by other sounds (for example, other people talking in a crowded restaurant). In contrast, the audio system described here uses one or more contact optical microphones to monitor vibrations on the user's skin and uses these vibrations to measure sound. The noise in the detected signal is much lower than conventional signals, and it allows reliable identification of when a user is speaking. Furthermore, since only relative distance changes are observed in interferometric systems, for embodiments in which the detector and light source are in an interference configuration, in the case of distance changes (e.g., moving glasses), no calibration of the absolute distance is required or required. alignment. Furthermore, conventional VADs (eg, bone conduction microphones with diaphragms) have a resonant frequency, and using conventional VADs near or above this resonant frequency can be difficult and inaccurate. In contrast, optical microphones have no moving or vibrating elements, so the above limitations in conventional VADs are not a problem for optical contact microphones.

Embodiments of the present invention may include artificial reality systems or may be implemented in conjunction with artificial reality systems. Artificial reality is a form of reality that has been adjusted in some way before being presented to the user. Artificial reality can include, for example, virtual reality (VR), augmented reality (AR), and mixed reality (MR). , hybrid reality or some combination and/or derivative thereof. Artificial reality content may include fully generated content or generated content combined with captured (eg, real-world) content. Artificial reality content can include video, audio, haptic feedback, or some combination thereof, any of which can be presented in a single channel or in multiple channels (such as stereoscopic video that produces a three-dimensional effect to the viewer). Furthermore, in some embodiments, artificial reality may also be associated with applications, products, accessories, services, or some combination thereof, which are used to operate in artificial reality. Create content in and/or otherwise use it in artificial realities. Artificial reality systems that provide artificial reality content can be implemented on a variety of platforms, including wearable devices (e.g., head-mounted viewers) connected to a host computer system, stand-alone wearable devices (e.g., head-mounted viewers) (server), mobile device or computing system, or any other hardware platform capable of delivering artificial reality content to one or more viewers.

FIG. 1A is a perspective view of a headset 100 implemented as an eyewear device in accordance with one or more embodiments. In some embodiments, the eyewear device is a near eye display (NED). Generally, the headset 100 may be worn on the user's face such that content (eg, media content) is presented using display components and/or audio systems. However, the headset 100 may also be used in such a manner that media content is presented to the user in a different manner. Examples of media content presented by headset 100 include one or more images, video, audio, or some combination thereof. The headset 100 includes eyeglass frames and may include, among other components, a display assembly including one or more display elements 120 , a depth camera assembly (DCA), an audio system, and a positioning sensor 190 . Although FIG. 1A shows components of the headset 100 in an example position on the headset 100 , these components may be located at other locations on the headset 100 , in conjunction with the headset 100 . on the peripheral device to which the processor 100 is paired, or on some combination thereof. Similarly, there may be more or fewer components on headset 100 than shown in FIG. 1A .

Eyeglass frame 110 holds other components of headset 100 . Eyeglass frame 110 includes a front portion that holds one or more display elements 120, and end pieces (eg, temples) that attach to the user's head. The front of the eyeglass frame 110 spans the top of the user's nose. The length of the end piece may be adjustable (eg, adjustable temple length) to suit different users. The end piece may also include a portion that curves behind the user's ear (eg, foot cuff, earpiece).

One or more display elements 120 provide light to a user wearing headset 100 . As shown, the headset includes a display element 120 for each eye of the user. In some embodiments, display element 120 generates image light that is provided to an eyebox of headset 100 . The eye zone is the position in space occupied by the user's eyes when wearing the headset 100 . For example, display element 120 may be a waveguide display. A waveguide display includes a light source (eg, a two-dimensional source, one or more line sources, one or more point sources, etc.) and one or more waveguides. Light from the light source is incoupled into one or more waveguides that output the light in such a manner that there is pupil replication in the eye zone of the headset 100 . In-coupling and/or out-coupling of light from one or more waveguides may be accomplished using one or more diffraction gratings. In some embodiments, a waveguide display includes a scanning element (eg, waveguide, mirror, etc.) that scans light from a light source as the light is incoupled into one or more waveguides. Note that in some embodiments, one or both of the display elements 120 are opaque and do not transmit light from a localized area around the headset 100 . The local area is the area around the headset 100 . For example, the local area may be a room inside which the user wearing headset 100 is, or the user wearing headset 100 may be outside and the local area is an external area. In this context, the headset 100 generates VR content. Alternatively, in some embodiments, one or both of the display elements 120 are at least partially transparent such that light from the localized area can be combined with light from one or more display elements to produce AR content and/or MR content.

In some embodiments, display element 120 does not generate image light, but rather the lens transmits light from a localized area to an eye-friendly area. For example, one or both of the display elements 120 may be uncorrected (non-prescription) lenses or prescription lenses (eg, single vision lenses, bifocal and trifocal lenses, or progressive lenses) to help correct the user's vision. defects in. In some embodiments, display element 120 may be polarized and/or tinted to protect the user's eyes from sun damage.

In some embodiments, display element 120 may include additional optics blocks (not shown). The optics block may include one or more optical elements (eg, lenses, Fresnel lenses, etc.) that direct light from display element 120 to an eye zone. The optics block may, for example, correct aberrations in some or all of the image content, amplify some or all of the image, or some combination thereof.

DCA determines depth information for a portion of a local area around the headset 100 . The DCA includes one or more imaging devices 130 and a DCA controller (not shown in Figure 1A), and may also include an illuminator 140. In some embodiments, illuminator 140 illuminates a portion of the local area with light. The light may be, for example, infrared (IR) structured light (eg, point pattern structured light, strip structured light, etc.), IR flash for time-of-flight (ToF), etc. In some embodiments, the one or more imaging devices 130 capture images of portions of the local area that include light from the illuminator 140 . As shown, Figure 1A shows a single illuminator 140 and two imaging devices 130. In an alternative embodiment, there is no illuminator 140 and at least two imaging devices 130 are present.

The DCA controller uses the captured image and one or more depth determination techniques to calculate depth information for a portion of the local area. Depth determination techniques may be, for example, direct time-of-flight (ToF) depth sensing, indirect ToF depth sensing, structured light, passive stereo analysis, active stereo analysis (using texture added to the scene by light from the illuminator 140), scene determination some other technology or some combination of them.

The audio system provides the audio content. The audio system includes a transducer array, a sensor array, one or more optical microphones 145, and an audio controller 150. However, in other embodiments, the audio system may include different components and/or additional components. Similarly, in some cases, functionality described with reference to multiple components of an audio system may be distributed among the multiple components in a manner different from that described herein. For example, some or all functions of the audio controller may be performed by the remote server.

The transducer array presents sound to the user. The transducer array includes a plurality of transducers. The transducer may be a speaker 160 or a tissue transducer 170 (eg, a bone conduction transducer or a cartilage conduction transducer). Although speaker 160 is shown external to eyeglass frame 110 , speaker 160 may be enclosed within eyeglass frame 110 . In some embodiments, instead of a separate speaker for each ear, the headset 100 includes a speaker array that includes multiple speakers integrated into the eyeglass frame 110 to improve the presentation of the audio content. Directionality. Tissue transducer 170 couples to the user's head and directly vibrates the user's tissue (eg, bone or cartilage) to generate sound. The number and/or location of transducers may differ from that shown in Figure 1A.

The sensor array detects sounds within a local area of the headset 100 . The sensor array includes a plurality of acoustic sensors 180 . Acoustic sensor 180 captures sound emitted from one or more sound sources in a local area (eg, a room). Each acoustic sensor is configured to detect sound and convert the detected sound into an electronic format (analog or digital). Acoustic sensor 180 may be an acoustic wave sensor, a microphone, a sound transducer, or a similar sensor suitable for detecting sound.

In some embodiments, one or more acoustic sensors 180 may be placed in the ear canal of each ear (eg, acting as binaural microphones). In some embodiments, acoustic sensor 180 may be placed on an outer surface of headset 100 , placed on an inner surface of headset 100 , separate from headset 100 (e.g., part of some other device) or some combination thereof. The number and/or location of acoustic sensors 180 may differ from that shown in Figure 1A. For example, the number of acoustic detection locations can be increased to increase the amount of audio information collected as well as the sensitivity and/or accuracy of the information. The acoustic detection locations may be oriented so that the microphones can detect sounds in a wide range of directions around a user wearing headset 100 .

In some embodiments, one or more optical microphones 145 detect tissue-based vibrations of the skin caused by sound in a localized area. These sounds may include, for example, the user's voice and sounds from other sound sources in the local area. For example, when a user speaks, part of the speech is actually transmitted through the user's tissue via tissue conduction. Part of this speech appears as slight tissue-based vibrations on the skin of the user's head. One or more optical microphones 145 detect these tissue-based vibrations. Similarly, sound sources external to the user (e.g., fans, other speakers, etc.) produce sound, which may also manifest as vibrations on the user's skin. Optical microphone 145 includes at least one light source and at least one detector, and may optionally include one or more optical elements. The optical microphone 145 can be configured in a variety of ways. For example, the light source and detector may be configured in a series configuration or a parallel configuration (eg, as described below with respect to Figures 5A and 5B). And in some cases, the optical microphone may be a pair of optical microphones including at least two self-mixing interferometers (eg, as described below with respect to Figure 5C). And in some cases, the light source and detector of the optical microphone may be located at different locations on the headset 100 (eg, different positioning within the nose pad, different positioning on the eyeglass frame 110, etc.). The detector and light source can be on different or the same die, depending for example on whether they are dual or common interferometers, in which case the threshold distance between them is determined by the interferometer arm length .

The light source is configured to emit light. The light source may be, for example, a vertical cavity surface emitting laser (VCSEL), an edge emitting laser, a tunable laser, some other coherent light source, or some combination thereof. The band of light emitted is such that the skin mainly reflects the light (as opposed to, for example, mainly absorbing it). One or more optical microphones 145 may emit light at, for example, 850 nm, 940 nm, 1300 nm, 1050 nm, etc. The emitted light is a continuous wave and, in some embodiments, includes a reference beam and a sensing beam. In some embodiments, the light source is configured to illuminate (eg, with a sensing beam) the user's skin (eg, one or more of the same or different portions of the face).

The detector monitors light in the band of light emitted by the light source. The detector may be, for example, one or more photodetectors. In some embodiments, the detector and light source are configured to form an interferometric system (eg, self-mixing interferometer, Michelson interferometer, LCI (eg, optical coherence tomography), LDV, etc.). The detector is therefore configured to detect a mixed signal corresponding to the reference beam mixed with a portion of the sensing beam that is reflected by the skin (eg via Fresnel reflection and/or scattering reflection). In other embodiments, the light source and detector are in a non-interference configuration. In this configuration, the detector measures the intensity modulation of reflected and/or scattered light from the skin.

In an alternative embodiment, the optical microphone 145 includes a membrane, and the vibrations of the membrane are monitored rather than the vibrations of the user's skin. In this embodiment, the light source is configured to illuminate a portion of the film, and the film scatters and/or reflects a portion of the light. In some embodiments, at least the portion of the film illuminated by the light source is highly reflective in the band of light emitted by the light source. The light source and detector can be in an interference configuration or a non-interference configuration. The detector detects scattered and/or reflected light from the membrane, and the output signal (eg, mixed signal, modulation intensity) can be used to monitor sound in a local area.

In the example shown in FIG. 1A , optical microphone 145 is located in an area of spectacle frame 110 that would come into contact with a portion of the nose of a user wearing headset 100 . For example, optical microphone 145 may be integrated into one or both nose pads of a pair of glasses. In other embodiments, one or more of the plurality of optical microphones 145 may alternatively or additionally be located at other locations on the headset 100 and/or may be present on the headset 100 One or more additional optical microphones 145. For example, some or all of the one or more optical microphones 145 may be positioned on the inward side of the eyeglass frame 110 at side fire positions 147A, 147B, 147C, 147D and/or at the center beam position. Some or all of the 148 locations. Various embodiments of optical microphone 145 are described below with respect to Figures 2, 3, 4, and 5A-5C.

The audio controller 150 processes the detected mixed signal(s) from the detector(s) of the optical microphone(s) 145 to measure sound from the local area. Audio controller 150 may analyze the mixed signal from some or all of the various optical microphones 145 to measure some or all of the sound that causes vibrations in the user's skin. The sound may include: for example, the user's voice, and/or other sound sources in the local area (eg, other people, noise sources (eg, fans), etc.).

Audio controller 150 may include a processor and computer-readable storage media. Audio controller 150 may be configured to generate a direction of arrival (DOA) estimate, generate an acoustic transfer function (eg, array transfer function (ATF)) and/or a head-related transfer function (ATF). , HRTF)), track the location of the sound source, form a beam in the direction of the sound source, classify the sound source, generate a sound filter for the transducer array, instruct the transducer array to perform active noise reduction, identify the user voice, recognize and execute commands based on the recognized user's voice, capture information that can be used to identify the user's micro-expressions, or some combination thereof. Additional details regarding how audio controller 150 may use detected tissue vibrations are detailed with respect to the figures discussed below.

Positioning sensor 190 generates one or more measurement signals in response to movement of headset 100 . Positioning sensor 190 may be located on a portion of eyeglass frame 110 of headset 100 . The positioning sensor 190 may include an inertial measurement unit (IMU). Examples of positioning sensors 190 include: one or more accelerometers, one or more gyroscopes, one or more magnetometers, another suitable type of sensor that detects motion, a type of sensor used for error correction of the IMU , or some combination of them. Positioning sensor 190 may be located external to the IMU, internal to the IMU, or some combination thereof.

In some embodiments, the headset 100 may provide simultaneous localization and mapping (SLAM) for positioning the headset 100 and updating the model of the local area. For example, the headset 100 may include a passive camera assembly (PCA) that generates color image data. PCA may include one or more RGB cameras that detect images of some or all of the local area. In some embodiments, some or all of the DCA's imaging devices 130 may also serve as PCAs. The image detected by PCA and the depth information determined by DCA can be used to determine parameters of the local area, generate a model of the local area, update a model of the local area, or some combination thereof. Additionally, positioning sensor 190 tracks the positioning (eg, position and attitude) of headset 100 within the room. Additional details regarding various components of headset 100 are discussed below in conjunction with FIG. 4 .

FIG. 1B is a perspective view of a head-mounted viewer 105 implemented as an HMD, in accordance with one or more embodiments. In embodiments describing AR systems and/or MR systems, portions of the front side of the HMD are at least partially transparent in the visible band (~380 nm to 750 nm), and portions of the HMD located on the front side of the HMD are in contact with the user's The portions between the eyes are at least partially transparent (eg, a partially transparent electronic display). The HMD includes a front rigid body 115 and a belt 175 . The headset 105 includes many of the same components described above with reference to Figure 1A, but these components are modified to incorporate the HMD form factor. For example, the HMD includes a display component, a DCA, an audio system including one or more optical microphones 145, and a positioning sensor 190. FIG. 1B shows an illuminator 140, a plurality of speakers 160, a plurality of imaging devices 130, a plurality of acoustic sensors 180, and a positioning sensor 190. These speakers 160 may be located in various locations: for example, coupled to the strap 175 (as shown), coupled to the front rigid body 115 , or may be configured to be inserted into the user's ear canal.

Figure 2 is a block diagram of an audio system 200 in accordance with one or more embodiments. The audio system in FIG. 1A or FIG. 1B may be an embodiment of audio system 200. Audio system 200 generates one or more acoustic transfer functions for the user. Audio system 200 may then use the one or more acoustic transfer functions to generate audio content for the user. In the embodiment of FIG. 2 , audio system 200 includes transducer array 210 , sensor array 220 , optical microphone assembly 222 , and audio controller 230 . Some embodiments of audio system 200 have different components than those described herein. Similarly, in some cases, functionality may be distributed among multiple components in a manner different from that described herein.

Transducer array 210 is configured to present audio content. Transducer array 210 includes a plurality of transducers. A transducer is a device that provides audio content. The transducer may be, for example, a speaker (eg, speaker 160), a tissue transducer (eg, tissue transducer 170), some other device that provides audio content, or some combination thereof. The tissue transducer may be configured to function as a bone conduction transducer or a cartilage conduction transducer. Transducer array 210 may present audio content via air conduction (e.g., via one or more speakers), via bone conduction (via one or more bone conduction transducers), via cartilage conduction audio systems (e.g., via one or more speakers). via one or more cartilage conduction transducers), or some combination thereof. In some embodiments, transducer array 210 may include one or more transducers to cover different portions of the frequency range. For example, piezoelectric transducers may be used to cover a first part of the frequency range, and moving coil transducers may be used to cover a second part of the frequency range.

Bone conduction transducers generate sound pressure waves by causing bone/tissue in the user's head to vibrate. The bone conduction transducer may be coupled to a portion of the headset and may be configured behind an auricle connected to a portion of the user's skull. The bone conduction transducer receives vibration instructions from the audio controller 230 and vibrates a portion of the user's skull based on the received instructions. Vibrations from the bone conduction transducer generate sound pressure waves experienced by the tissue that travel around the eardrum toward the user's cochlea.

Cartilage conduction transducers generate sound pressure waves by vibrating one or more portions of the auricle cartilage of the user's ear. The cartilage conduction transducer may be coupled to a portion of the headset and may be configured to couple to one or more portions of the auricular cartilage of the ear. For example, a cartilage conduction transducer may be coupled to the back of the pinna of the user's ear. The cartilage conduction transducer may be located anywhere along the auricular cartilage surrounding the external ear (eg, the pinna, the tragus, some other portion of the auricular cartilage, or some combination thereof). Vibrating one or more parts of the auricle cartilage can generate: air-borne sound pressure waves outside the ear canal; sound pressure waves generated by tissue that causes parts of the ear canal to vibrate, thereby generating Sound pressure waves propagated through the air in the tunnel; or some combination thereof. The resulting airborne sound pressure wave travels down the ear canal toward the eardrum.

Transducer array 210 generates audio content based on instructions from audio controller 230. In some embodiments, the audio content is spatialized. Spatialized audio content is audio content that appears to originate from a specific direction and/or target area (eg, objects in a local area, and/or virtual objects). For example, spatialized audio content may make the sound appear to a user of audio system 200 to be originating from a virtual singer across the room. Transducer array 210 may be coupled to a wearable device (eg, headset 100 or headset 105). In alternative embodiments, transducer array 210 may be a plurality of speakers separate from the wearable device (eg, coupled to an external console).

The sensor array 220 detects sound in a local area around the sensor array 220 . The detected sounds may, for example, come from a user of audio system 200 (eg, the user's voice), and/or may be sounds from other sound sources in the local area (eg, other people). Sensor array 220 may include a plurality of acoustic sensors, each of which detects changes in air pressure of sound waves and converts the detected sound into an electronic format (analog or digital). The plurality of acoustic sensors may be positioned on the headset (eg, headset 100 and/or headset 105 ), on the user (eg, in the user's ear canal), be positioned on the neckband, or some combination thereof. An acoustic sensor may be, for example, a microphone, a vibration sensor, an accelerometer, or any combination thereof. In some embodiments, sensor array 220 is configured to monitor audio content generated by transducer array 210 using at least some of the plurality of acoustic sensors. Increasing the number of sensors can improve the accuracy of information (eg, directivity) describing the sound field generated by the transducer array 210, and/or the sound coming from a local area.

In some embodiments, optical microphone assembly 222 is configured to detect tissue-based (ie, temporary) vibrations of the skin caused by sound in a localized area. The optical microphone assembly 222 includes one or more of the plurality of optical microphones 145 . As described above with respect to FIG. 1A , optical microphone 145 includes at least one light source and at least one detector, and may optionally include one or more optical elements (eg, lenses). Optical microphone 145 may be configured in a variety of ways (eg, as described below with respect to Figures 3-5C). The signal (eg, mixed signal, modulation intensity) from the output of each optical microphone 145 can be used to monitor sound in a local area.

Note that in some embodiments, the one or more optical microphones 145 each include a membrane, and the vibrations of the membrane are monitored rather than the vibrations of the user's skin. In this embodiment, the light source is configured to illuminate a portion of the film, and the film scatters and/or reflects a portion of the light. In some embodiments, at least the portion of the film illuminated by the light source is highly reflective. The light source and detector can be in an interference configuration or a non-interference configuration. The detector detects scattered and/or reflected light from the membrane, and the output signal (eg, mixed signal, modulation intensity) can be used to monitor sound in a local area.

Audio controller 230 controls the operation of audio system 200. In the embodiment of FIG. 2 , audio controller 230 includes data memory 235 , DOA estimation module 240 , transfer function module 250 , tracking module 260 , beamforming module 270 , processing module 275 , and sound filter module 280 . In some embodiments, audio controller 230 may be located inside the headset. Some embodiments of audio controller 230 have different components than those described herein. Similarly, various functions may be distributed among multiple components in a manner different from that described herein. For example, some functions of the controller can be performed outside the headset. The user can opt-in to allow audio controller 230 to send data detected by the headset to systems external to the headset, and the user can select privacy settings that control access to any such data.

Data memory 235 stores data used by audio system 200. Data in data store 235 may include sounds recorded in a local area of audio system 200, audio content, a head-related transfer function (HRTF), a transfer function of one or more sensors, one or more of a plurality of acoustic sensors. Array transfer function (ATF) of the acoustic sensor, sound source location, virtual model of the local area, direction of arrival estimation, acoustic filter, tissue vibration detected by one or more optical microphones 145, sound detected by the sensor array 220, A model that maps light amplitude to the distance from the detector of the optical microphone 145, and other relevant data used by the audio system 200, or any combination thereof.

The user may opt-in to allow data storage 235 to record data detected by sensor array 220 and/or one or more optical microphones 145 . In some embodiments, audio system 200 may employ always-on recording in which audio system 200 records all sounds detected by sensor array 220 and/or optical microphone assembly 222 . The user may opt-in or opt-out to allow or prevent audio system 200 from recording, storing, or sending recorded data to other entities.

DOA estimation module 240 is configured to locate sound sources in a local area based in part on information from sensor array 220 . Localization is the process of determining where a sound source is located relative to the user of the audio system 200 . DOA estimation module 240 performs DOA analysis to locate one or more sound sources within a local area. DOA analysis may include analyzing the intensity, spectrum, and/or arrival time of each sound at sensor array 220 to determine from which direction the sound originates. In some cases, DOA analysis may include any suitable algorithm for analyzing the surrounding acoustic environment in which audio system 200 is located.

For example, a DOA analysis may be designed to receive an input signal from sensor array 220 and apply a digital signal processing algorithm to the input signal to estimate the direction of arrival. These algorithms may include, for example, delayed sum algorithms in which an input signal is sampled and the final weighted and delayed versions of the sampled signals are averaged together to determine the DOA. The least mean square (LMS) algorithm can also be implemented to create adaptive filters. This adaptive filter can then be used to identify differences in signal strength or arrival times, for example. These differences can then be used to estimate DOA. In another embodiment, the DOA may be determined by converting the input signal into the frequency domain and selecting specific bins within the time-frequency (TF) domain to be processed. Each selected TF interval can be processed to determine whether the interval includes a portion of the audio spectrum with a direct path audio signal. Those intervals that have a portion of the direct path signal can then be analyzed to identify the angle at which the direct path audio signal is received by the sensor array 220 . The determined angle can then be used to identify the DOA of the received input signal. Other algorithms not listed above may also be used alone or in combination with the above algorithms to determine DOA.

In some embodiments, DOA estimation module 240 may also determine the DOA relative to the absolute positioning of audio system 200 within the local area. The positioning of sensor array 220 may be received from an external system (eg, some other component of the headset, an artificial reality console, a mapping server, a positioning sensor (eg, positioning sensor 190), etc.). The external system may create a virtual model of the local area in which the local area and positioning of the audio system 200 is mapped. The received positioning information may include the location and/or orientation of some or all of audio system 200 (eg, sensor array 220). DOA estimation module 240 may update the estimated DOA based on the received positioning information.

Transfer function module 250 is configured to generate one or more acoustic transfer functions. Typically, a transfer function is a mathematical function that gives a corresponding output value for each possible input value. Based on the parameters of the detected sound, the transfer function module 250 generates one or more acoustic transfer functions associated with the audio system. The acoustic transfer function may be an array transfer function (ATF), a head-related transfer function (HRTF), other types of acoustic transfer functions, or some combination thereof. ATF characterizes how a microphone receives sound from a point in space.

The ATF includes a plurality of transfer functions that characterize relationships between sound sources and corresponding sounds received by a plurality of acoustic sensors in sensor array 220 . Therefore, for one sound source, there is a corresponding transfer function for each of the plurality of acoustic sensors in sensor array 220 . And this set of transfer functions is collectively called ATF. Therefore, for each sound source, there is a corresponding ATF. Note that a sound source may be, for example, someone or something generating sound in a local area, a user, or one or more transducers of transducer array 210. Because human anatomy (eg, ear shape, shoulders, etc.) affects sound as it travels to the human ear, the ATF for a particular sound source location relative to sensor array 220 may vary from user to user. Therefore, the ATF of sensor array 220 is personalized for each user of audio system 200.

In some embodiments, transfer function module 250 determines one or more HRTFs for a user of audio system 200 . HRTF characterizes how the ear receives sound from a point in space. Because a person's anatomy (e.g., ear shape, shoulders, etc.) affects sound as it travels to the person's ears, the HRTF relative to a person's specific source location is unique to each of the person's ears (and is Unique). In some embodiments, transfer function module 250 may use a calibration process to determine the HRTF for the user. In some embodiments, transfer function module 250 can provide information about the user to the remote system. The user can adjust the privacy settings to allow or prevent the transfer function module 250 from providing information about the user to any remote system. The remote system determines a set of HRTFs customized for the user using, for example, machine learning, and provides the customized set of HRTFs to the audio system 200 .

Tracking module 260 is configured to track the location of one or more sound sources. Tracking module 260 may compare multiple current DOA estimates and compare them to a stored history of multiple previous DOA estimates. In some embodiments, audio system 200 may recalculate the DOA estimate on a periodic schedule (eg, once every second or every millisecond). The tracking module may compare the current DOA estimate to the previous DOA estimate, and in response to changes in the DOA estimate of the sound source, the tracking module 260 may determine that the sound source has moved. In some embodiments, tracking module 260 may detect changes in position based on visual information received from a head-mounted viewer or some other external source. Tracking module 260 can track the movement of one or more sound sources over time. Tracking module 260 may store a value for the number of sound sources and the location of each sound source at each point in time. In response to a change in the value of the number of sound sources or the location of the sound source, tracking module 260 may determine that the sound source has moved. Tracking module 260 may calculate an estimate of the positioning variance. Positioning variance can be used as a confidence level for each determined movement change.

Beamforming module 270 is configured to process one or more ATFs to selectively emphasize sounds from sources within a certain area while attenuating sounds from other areas, thereby functioning as an adaptive beamformer. . When analyzing sounds detected by sensor array 220, and in some cases, optical microphone 145, beamforming module 270 may combine information from different acoustic sensors to emphasize associated sounds from specific areas of the local area, Also attenuates sounds coming from outside the area. Beamforming module 270 may isolate audio signals associated with sound from a particular sound source from other sound sources in a local area based on, for example, different DOA estimates from DOA estimation module 240 and tracking module 260 . The beamforming module 270 can thus selectively analyze discrete sound sources in local areas. In some embodiments, beamforming module 270 can enhance signals from sound sources. For example, beamforming module 270 may apply an acoustic filter that eliminates signals above certain frequencies, below certain frequencies, or between certain frequencies. Signal enhancement is used to enhance the sounds associated with a given identified sound source relative to other sounds detected by sensor array 220 .

The processing module 275 uses the output signal from the optical microphone assembly 222 to measure the sound in the local area. The output signal may be, for example, a signal corresponding to the amplitude of the portion of light reflected from the skin. For example, the audio controller can input detected light into a model that maps light amplitude to distance from the detector (i.e., skin localization). This is discussed further below with respect to Figure 3.

In the case of one or more optical microphones 145 in an interference configuration, the output signal may be, for example, a mixed signal. The detected mixed signal includes dynamic high-frequency components and modulation components. This dynamic high-frequency component changes (frequency shift) as the distance between the illuminated part of the skin and the detector changes. Note that the amplitude of the vibration measured may be, for example, 50 nm (eg, user whispering) to 1.5 micron (eg, user shouting, some other loud noise in the local area). Therefore, the vibrations induced in the skin by sounds in local areas lead to changes in dynamic high-frequency components. The processing module measures sounds by inferring corresponding sounds that cause vibrations of the user's skin based on dynamic high-frequency components. Furthermore, vibrations of the user's skin caused by sound have very different frequencies from vibrations caused by, for example, the user's movement (eg, walking, running) (called motion noise). The processing module 275 may isolate and/or filter out portions of the dynamic high-frequency components corresponding to motion noise.

Note that the amplitude of speech-driven tissue vibrations decreases at higher audio frequencies. This is likely due to the low-pass nature of bone-conducted speech as speech-driven vibrations propagate through bone and soft tissue. The low-pass filter characteristics of this structure can affect the effectiveness of optical microphones in sensing speech content information at higher frequencies. Therefore, in some cases, high frequency components of speech (eg, a user's speech) may be attenuated. The high frequency component may be, for example, audio content with a frequency exceeding 2 kHz.

Processing module 275 may enhance and/or reconstruct the sensed speech content (eg, for higher frequencies). The processing module 275 may reconstruct the high-frequency content of the sensed speech using, for example, a matrix decomposition-based bandwidth extension method. Clean speech recordings obtained using one or more acoustic microphones are used to learn the user's broadband spectral basis. These broadband bases include both the low frequency content and the high frequency content of speech. Their low-frequency content is then used for narrowband speech (obtained with optical microphones) to learn how the wideband basis should be combined to obtain wideband speech.

In another embodiment, the processing module 275 may use a neural network-based audio super-resolution method to reconstruct the high-frequency content. If the network is trained in the spectral domain, audio super-resolution enables the extrapolation (repair) of high-frequency content based on low-frequency content. If the network is trained in the time domain, the narrowband waveforms are interpolated in the time domain to obtain wideband speech with high frequency content. Alternatively, it is also possible to jointly train two networks, one in the time domain and the other in the frequency domain. The results of both networks can be combined with a fusion layer, or the two networks can be cascaded. Simultaneous recordings from one or more optical and acoustic microphones can be used for training of these neural networks. The learned network can then be used to reconstruct the high-frequency content of speech acquired with one or more optical microphones.

Note that similar approaches have been successfully used in the literature to reconstruct wideband speech from narrowband telephone signals. In telephony applications, the low-frequency content of narrowband speech and wideband speech is the same. However, the low-frequency content of speech captured with an optical microphone may differ from the low-frequency content of speech captured with an acoustic microphone. To account for this difference, one can include in the decomposition-based approach a weighting matrix that maps the basis obtained from the optical microphone to the acoustic microphone. This matrix can be learned during training. When training an audio super-resolution network, one or more convolutional layers can be inserted as one or more input layers. With appropriate training, one or more additional layers can help learn the mapping of the low-frequency content of optical microphones to acoustic microphones.

The processing module 275 may update the measured sound of the speech with the reconstructed high-frequency component. The reconstructed high-frequency components alleviate the attenuation of the high-frequency components.

Processing module 275 may use signals (eg, mixed signals) from the output of optical microphone assembly 222 to identify the user's speech among the detected sounds. These output signals are vibrations that occur on the user's skin, for example when the user speaks and/or sounds from a local area cause the skin to vibrate. In some embodiments, one or more optical microphones 145 may function as a VAD. Accordingly, the processing module 275 may input the output signal from one or more optical microphone assemblies 222, as well as the sound from the local area, into a model that uses the input to predict the user's behavior in the detected sound from the local area. Voice recognition.

In some embodiments, processing module 275 may determine that the recognized user's voice includes a command. And then, audio system 200 and/or headset 100 can perform actions based on the command. This action may control certain operations of the audio system 200 and/or the headset 100 . The action may, for example, specify a sound source, decrease/increase volume, some other action that controls the operation of audio system 200 and/or headset 100, or some combination thereof.

The processing module 275 uses the detected tissue vibrations (ie, output signals) from the optical microphone assembly 222 and the detected sound from the sensor array 220 in a variety of ways. In some embodiments, processing module 275 uses detected sound from sensor array 220 to calibrate one or more optical microphones 145 .

In some embodiments, processing module 275 identifies one or more sounds (eg, background noise) for suppression among the measured sounds from optical microphone 145 and/or sensor array 220 . The processing module 275 may then provide this information to the sound filter module 280 as part of the active noise reduction process. Sound filters can be applied to modify the audio signal corresponding to the audio content. And then, the transducer array 210 may present the modified audio signal to the user as modified audio content, the modified audio content including the audio content and the suppression component that suppresses the noise.

In some embodiments, processing module 275 may use output signals from optical microphone assembly 222 to monitor sliding of the headset on the user's head. For example, if the headset is moved to a new resting position on the user's head, it will introduce a distance offset. Processing module 275 may identify offsets in the output signal to identify and/or monitor the positioning of the headset on the user. For example, sound filter module 280 may use the new positioning information to generate a more accurate sound filter.

The sound filter module 280 determines the sound filter for the transducer array 210 . In some embodiments, the sound filter causes the audio content to be spatialized so that the audio content appears to originate from the target area. In some embodiments, a sound filter may cause positive or negative amplification of sound as a function of frequency. The sound filter module 280 may use HRTFs and/or acoustic parameters to generate sound filters. Acoustic parameters describe the acoustic properties of a local area. Acoustic parameters may include, for example, reverberation time, reverberation level, room impulse response, etc. In some embodiments, sound filter module 280 calculates one or more of these acoustic parameters. In some embodiments, the acoustic filter module 280 requests the acoustic parameters from the map building server (eg, as described below with respect to Figure 9).

The sound filter module 280 may update one or more sound filters based on the user's voice identified in the detected sounds. One or more updated sound filters may be applied to the audio content to generate modified audio content. For example, the sound filter module 280 may update the sound filter such that when the sound filter is applied to audio content, the modified audio content will enhance the recognized user's speech. In some embodiments, sound filter module 280 may update the sound filter to suppress one or more sounds detected by one or more optical microphones 145 and/or sensor array 220 (ie, perform active noise reduction). In some embodiments, sound filter module 280 provides sound filters and/or modified audio content to transducer array 210, and/or one or more other audio systems in the local area. The sound filter module 280 may provide one or more updated sound filters, and/or modified audio content, to one or more other audio systems via, for example, a local wireless network (eg, WIFI, Bluetooth, etc.). In this way, the user's speech can be presented to the user of another audio system in real time - this is possible in noisy environments where other users would have difficulty hearing the user's speech (e.g., in a crowd at a football match or in a This is particularly useful in other low SNR environments).

3 is an example nose pad 310 including an optical microphone 320 with a light source 330 and a detector 340 located at different locations within the nose pad 310, in accordance with one or more embodiments. Nose pad 310 is an example nose pad for a headset (eg, headset 100 ). Optical microphone 320 is an embodiment of optical microphone 145 in which light source 330 and detector 340 are located at different locations within nose pad 310 and are separated from each other by a threshold distance. Detector 340 and light source 330 are on different or the same die, depending on whether they are dual or common interferometers, in which case the threshold distance is determined by the interferometer arm length. As shown in Figure 3, the light source 330 illuminates a portion of the nose with light. And detector 340 detects the portion of light scattered and reflected by the nose.

Vibrations in the skin may affect how much of the emitted light is reflected and/or scattered from the user's skin. In some embodiments, the audio controller processes the detected light to measure sound from the local area based on the modulation intensity of the detected light. For example, the audio controller may input the detected light into a model that maps light amplitude to distance from detector 340 (ie, skin location). For example, at a first time, the detected light may have a relatively low amplitude signal, and at a second time, the detected light may have an increased amplitude signal. Therefore, the skin will be farther apart during the first time than during the second time. In this way, the audio controller can use the amplitude of the detected signal to monitor the vibrations of the skin.

In some embodiments, light emitted from light source 330 is divided into a reference beam and a sensing beam. A reference beam is provided to detector 340. In these embodiments, detector 340 and light source 330 are in an interference configuration such that detector 340 is configured to detect the mixed signal. For example, there may be an optical waveguide (eg, an optical fiber) that provides a reference beam to detector 340. The reference beam will be mixed with the portion of the sensing beam that is reflected and scattered from the nose to produce a mixed beam that is detected by the detector as a mixed signal. The audio controller processes the detected mixed signal to measure the sound coming from the local area.

Figure 4 is an example optical microphone 400 configured as a self-mixing interferometer in accordance with one or more embodiments. Optical microphone 400 includes a light source (not shown), a detector (not shown), and optical elements. Optical microphone 400 is an embodiment of optical microphone 145 in which the light source and detector are part of the same device and are configured to function as configured as a self-mixing interferometer. Optical microphone 400 may be, for example, embedded in a nose pad of a headset, coupled to an eyeglass frame of a headset (eg, positioned in a lateral radiating position), or the like. Various embodiments of optical microphone 400 are described in detail below with respect to Figures 5A, 5B and 5C.

Optical microphone 400 is configured as a self-mixing interferometer. A system based on self-mixing interferometry is a system in which reflected light is fed back into a laser cavity that modulates the power of the laser, and the laser cavity acts as a lock-in amplifier, thereby increasing the microphone SNR. In a self-mixing interferometer system, a detector (eg, a photodiode) can be placed on a laterally displaced or vertically displaced laser die to measure laser intensity. In some embodiments, the light source divides the emitted light into a reference beam and a sensing beam, and provides the reference beam to the detector. Additionally or alternatively, a portion of the light emitted from the light source may be reflected back to the detector by the optical element, and this portion of the reflected light is the reference beam. The detector is in an interference configuration with the light source (to function as a self-mixing interferometer), and the detector is configured to detect the mixed signal. The mixed signal corresponds to a mixed beam formed by mixing the reference beam with the portion of the sensing beam that is reflected and scattered from the nose. The audio controller processes the detected mixed signal to measure the sound coming from the local area.

Figure 5A is an optical microphone 500 configured as a self-mixing interferometer with multiple components in a series configuration in accordance with one or more embodiments. The optical microphone 500 includes a light source 510, an optical element 520 and a detector 530. In some embodiments, light source 510 and detector 530 are coupled to the same die. Optical microphone 500 is an embodiment of optical microphone 400 .

As shown, light source 510 (eg, VCSEL) emits light. The emitted light is incident on optical element 520 (eg, a lens), and a portion of the emitted light is transmitted by optical element 520 as a sensing beam, and a portion of the emitted light is reflected to detector 530 as a reference beam. Note that in some embodiments (not shown), the light source 510 may emit light as a sensing beam to the optical element and as a reference beam to the detector 530 . A portion of the sensing beam is scattered and/or reflected from the user's skin and then passes through optical element 520 and light source 510 to mix with the reference beam at detector 530 to form a mixed beam. The detector 530 detects this mixed light beam as a mixed signal. The audio controller processes the detected mixed signal to measure the sound coming from the local area.

Figure 5B is an optical microphone 540 configured as a self-mixing interferometer with multiple components in a parallel configuration in accordance with one or more embodiments. Optical microphone 540 includes a light source 545, an optical element 520, and a detector 550. In some embodiments, light source 510 and detector 530 are coupled to the same die. Light source 545 and detector 550 are essentially the same as light source 510 and detector 530 , except that light source 545 and detector 550 are arranged in a parallel configuration and detector 550 is now also coupled to optical element 520 . Optical microphone 500 is an embodiment of optical microphone 400 .

As shown, light source 545 (eg, VCSEL) emits light. The emitted light is incident on optical element 520 (eg, a lens), and a portion of the emitted light is transmitted by optical element 520 as a sensing beam, and a portion of the emitted light is reflected to detector 550 as a reference beam. A portion of the sensing beam is scattered and/or reflected from the user's skin and then passes through optical element 520 to mix with the reference beam at detector 550 to form a mixed beam. The detector 550 detects this mixed light beam as a mixed signal. The audio controller processes the detected mixed signal to measure the sound coming from the local area.

Figure 5C is an example of a pair of optical microphones 560 including two self-mixing interferometers according to one or more environments. Optical microphone 560 includes light source 510a, light source 510b, optical element 570, detector 530a, detector 530b, and block 580. In some embodiments, light source 510a, light source 510b, detector 530a, detector 530b, and block 580 are coupled to the same die. The light source 510a and the light source 510b are substantially the same as the light source 510, and the detector 530a and the detector 530b are substantially the same as the detector 530. Optical element 570 is substantially the same as optical element 520 except that optical element 570 is coupled to multiple light sources. Optical microphone 560 is an embodiment of optical microphone 400 .

Paired optical microphone 560 includes one or more optical elements, multiple light sources, and multiple corresponding detectors. And each light source and its corresponding detector are in a parallel configuration or a series configuration to form a corresponding optical microphone. For example, as shown, light source 510a is in a series configuration with corresponding detector 530a, and light source 510b is in a series configuration with corresponding detector 530b. In some embodiments, light source 510a and light source 510b emit at the same wavelength. In alternative embodiments, light source 510a and light source 510b emit at different wavelengths. For example, light source 510a may emit light at 780 nm, and light source 510b may emit light at 850 nm. Accordingly, light source 510a, detector 520a, and optical element 570 form a first optical microphone configured as a self-mixing interferometer with its multiple components in a series configuration (e.g., as shown in Figure 5A shown); and the light source 510b, the detector 520b and the optical element 570 form a second optical microphone configured as a self-mixing interferometer, with multiple components of the second optical microphone in a series configuration. Note that although two optical microphones are shown in the illustrated example, in other embodiments there may be multiple additional optical microphones formed that are also coupled to optical element 570 .

Crosstalk between the two optical microphones is mitigated via block 580 . Block 580 is made of a material that is non-transmissive (eg, absorptive or reflective) to light emitted by light source 510a, 510b. In some embodiments, block 580 may be part of a semiconductor die to form a single chip with the emitter. In some embodiments, block 580 is the metal used to bond two separate chips together. Note that the different positioning of the two optical elements relative to optical element 570 allows for a different emission angle for each of the plurality of optical microphones. For example, an optical microphone formed using light source 510a and detector 530a emits sensing beam 585, and an optical microphone formed using light source 510a and detector 530a emits sensing beam 590. And the sensing beam 585 and the sensing beam 590 are emitted from the optical element 570 at different angles.

Paired optical microphones 560 can monitor two different locations simultaneously. In contrast, optical microphones 500, 540 monitor a single location. Note that optical microphone 540 may be more easily implemented using two-component packaging techniques rather than a monolithic process.

Figure 6 is an example of an optical microphone 600 configured as an LDV in accordance with one or more embodiments. Optical microphone 600 includes a light source 610 (eg, an edge-emitting laser), a detector 620 (eg, a photodiode), a waveguide structure 630, and an optical antenna 640, all located on a substrate 650 as part of a photonic integrated circuit. In some embodiments, optical microphone 600 may include additional components. Optical microphone 600 is an embodiment of optical microphone 145 using an LDV configuration. Optical microphone 600 may be, for example, embedded in a nose pad of a headset, coupled to an eyeglass frame of a headset (eg, positioned in a lateral radiating position), or the like.

Waveguide structure 630 is an optical waveguide that guides light to the various components of optical microphone 600 . The waveguide structure 630 may couple together via various portions, for example, a light source 610, a beam splitter 692, one or more optical antennas 640, an optical combiner 695, a detector 620, one or more optical amplifiers, or some combination thereof. . The plurality of sections includes a transmit section 660, a reference section 670, a transmit sensing section 680, a receive sensing section 685, and a mixing section 690. The waveguide structure also includes a beam splitter 692, an optical combiner 694, and may also include a laser amplifier. Splitter 692 splits a portion (eg, 50%) of the light from emission portion 660 into emission sensing portion 680 and the remaining portion of the light from emission portion 660 into reference portion 670 . In some embodiments, some other portion (eg, 80%) of the optical power is split into emission sensing portion 680 instead of reference portion 670. Similarly, an optical combiner combines light from receive sensing portion 685 with light from reference portion 670 into mixing portion 690 . Optical microphone 600 may include one or more optical amplifiers that amplify light. For example, the optical amplifier may be positioned to amplify light prior to output by optical antenna 640 and/or to amplify light coupled in-coupled by optical antenna 640.

Optical antenna 640 out-couples and in-couples light into optical microphone 600 . Optical antenna 640 may be, for example, a grating coupler. Note that the illustrated embodiment includes a common input/output path for light via optical antenna 640. In other embodiments (not shown), there may be an optical antenna for outcoupling the sensing beam, and a separate optical antenna for receiving the portion of the sensing beam that is reflected and/or scattered from the user's skin.

Light source 610 emits light, which couples into emission portion 660 of waveguide structure 630 . The beam splitter 692 divides the emitted light into a reference beam and a sensing beam, provides the reference beam to the reference portion 670 and provides the sensing beam to the emission sensing portion 680 . Transmit sensing portion 680 provides the sensing beam to optical antenna 640, which outcouples the light into a localized area (eg, to illuminate the user's skin). Note that in some embodiments, an optical amplifier may be used to amplify the light before it is emitted by optical antenna 640. A portion of the sensing beam is reflected and/or scattered by the user's skin and is incoupled into waveguide structure 630 via optical antenna 640 . The receiving sensing portion 685 provides this light to the optical combiner 695 . Note that in some embodiments, an optical amplifier may be used to amplify the light before it is passed to optical combiner 695. Optical combiner 695 combines the received portion of the sensing beam with the reference beam to generate a mixed beam that is incoupled to mixing portion 690 . Detector 620 receives the mixed beam and detects the corresponding mixed signal. The audio controller processes the detected mixed signal to measure the sound coming from the local area.

Substrate 650 may be formed from any standard chip substrate material, such as semiconductor materials, silicon, silicon on insulator, gallium arsenide, aluminum gallium arsenide, silicon on sapphire, etc. Substrate 650 may also be formed from any transparent material in the visible spectral band (400 nm to 700 nm), such as glass, plastic, polymer, polymethylmethacrylate (PMMA), silicon dioxide, and any form of crystal (e.g., Lithium niobate, tellurium dioxide, etc.). The surface of substrate 650 may be bonded to a headset (eg, headset 100). The various components of optical microphone 600 may be bonded to substrate 650 by any standard bonding technique, and/or formed on the substrate by any standard etching or epitaxial growth technique.

Figure 7 is an example of an optical microphone 700 configured to use optical coherence tomography (OCT) in accordance with one or more embodiments. OCT is a form of LCI. Optical microphone 700 includes a light source 710, a detector 720, a waveguide structure 730, and an optical antenna 740, all of which are part of a photonic integrated circuit. In some embodiments, optical microphone 700 may include additional components. Optical microphone 700 is an embodiment of optical microphone 145 in an OCT configuration. Optical microphone 700 may be, for example, embedded in a nose pad of a headset, coupled to an eyeglass frame of a headset (eg, positioned in a lateral radiating position), or the like. In some embodiments, some or all of the components of optical microphone 700 may be bonded to and/or formed on a substrate (eg, substrate 650).

Waveguide structure 730 is an optical waveguide that guides light to the various components of optical microphone 700 . For example, the waveguide structure 730 may combine a light source 710 (eg, a tunable laser source), one or more beam splitters, one or more combiners, one or more optical antennas 740, a detector 720, a or Multiple optical amplifiers or some combination thereof are coupled together. The plurality of sections includes a transmit section 760, a reference section 770, a transmit/receive section 780, a sensing section 785, and a mixing section 790.

The waveguide structure also includes a beam splitter 792, a beam splitter 794, a beam splitter 796, and may also include a laser amplifier. Optical splitter 794 splits a portion of the light from emission portion 760 that is incoupled into the K clock. Optical splitter 794 splits a portion (eg, 50%) of the light from transmit portion 760 into transmit/receive portion 780 and a remaining portion (eg, the remaining 50%) of the light from transmit portion 760 into the reference portion 770 in. In some embodiments, some other portion of the optical power is split into transmit/receive portion 780 instead of reference portion 770. Note that beam splitter 792 also splits a portion (eg, 50%) of light traveling in the opposite direction from transmit/receive portion 780 (ie, light coupled within optical antenna 740) into sensing portion 785. Optical splitter 796 combines a first portion (eg, 50%) of light from sensing portion 785 and a first portion (eg, 50%) of light from reference portion 770 into a first waveguide of mixing portion 790, and The remainder of the light in both the measurement portion 785 and the reference portion 770 is combined into the second channel of the mixing portion 790 . Optical microphone 700 may include one or more optical amplifiers that amplify light. For example, the optical amplifier may be positioned to amplify light prior to output by optical antenna 740 and/or to amplify light coupled in-coupled by optical antenna 740.

Light source 710 (eg, a tunable laser source) emits light, which couples into emission portion 760 of waveguide structure 730 . Optical splitter 794 splits a portion of the light from emission portion 760, which portion is then incoupled into the K clock. The K clock synchronizes the light source 710 so that the output wavelength is linearly scanned, resulting in equal wavenumber intervals at the analog-to-digital converter (not shown - but its function may be performed by the audio controller) that detects The mixed signal detected by the processor 720 is processed to convert the mixed signal into a digital signal. The remaining light is transmitted by emission portion 760 to beam splitter 792. The beam splitter 792 divides the emitted light into a reference beam and a sensing beam, provides the reference beam to the reference part 770 and provides the sensing beam to the transmitting/receiving part 780 . The transmit/receive portion 780 directs the sensing beam to an optical antenna 740, which outcouples the light into a localized area (eg, to illuminate the user's skin). Note that in some embodiments, an optical amplifier may be used to amplify the light before it is emitted by optical antenna 740.

A portion of the sensing beam is reflected and/or scattered by the user's skin and is coupled into the transmit-receive portion 780 via the optical antenna 740 . Transmit/receive section 780 guides the incoupled light to beam splitter 792 . Beam splitter 792 splits a portion of this light into sensing portion 785 . Sensing portion 785 directs light to beam splitter 796 . Note that in some embodiments, an optical amplifier may be used to amplify the light before it is passed to beam splitter 796. Optical splitter 796 combines a first portion (eg, 50%) of light from sensing portion 785 and a first portion (eg, 50%) of light from reference portion 770 into a first waveguide of mixing portion 790, and The remainder of the light in both the measurement portion 785 and the reference portion 770 is combined into the second channel of the mixing portion 790 . Detector 720 receives the mixed beam and detects the corresponding mixed signal via a pair of balanced photodetectors.

The audio controller processes the detected mixed signal to measure the sound coming from the local area. In the OCT configuration, the detected interference pattern (expressed as a mixed signal) is a function of wavelength/wavenumber and provides an axial profile of the skin along the beam axis, where the fringe frequency corresponds to the depth of the skin, and the amplitude of this fringe frequency Corresponds to the reflectance of the skin. Note that there are various forms of OCT, and in other embodiments, optical microphone 700 may be configured to operate in one of these other forms (eg, phase-sensitive OCT).

8 is a flowchart illustrating a process for using an optical contact transducer in an interference configuration in accordance with one or more embodiments. The process illustrated in Figure 8 may be performed by components of an audio system (eg, audio system 200). In other embodiments, other entities may perform some or all of the various steps in Figure 8. Embodiments may include different steps and/or additional steps, or perform the multiple steps in a different order.

The audio system emits 810 light from the light source of the optical microphone, which light includes a reference beam and a sensing beam. This optical microphone is an embodiment of optical microphone 145 and may be constructed as described above with reference to Figures 3, 4, 5A, 5B and 5C. Optical microphones can be integrated into headsets. The emitted light is a continuous wave, and its driving current can be modulated (e.g., 10kHz). Optical microphones are positioned to monitor vibrations of the user's skin caused by sounds in localized areas of the audio system (eg, the user's voice, other people, noise sources, etc.).

The audio system illuminates 820 the user's skin (eg, one or more different or the same portions of the face) with the sensing beam. For example, the sensing beam may be refracted through the optical elements of the optical microphone to illuminate the user's skin.

A portion of the sensing beam is scattered and/or reflected from the user's skin to mix with the reference beam at the detector to form a mixed beam.

The audio system detects 830 the mixed signal (detected mixed light beam) via a detector in an interference configuration with the light source. The interference configuration enables the light source and detector to form an interference system (eg, self-mixing interferometer, Michelson interferometer, OCT, LDV, etc.).

Audio systems use mixed signals to measure sound in 840 local areas. The detected mixed signal includes dynamic high-frequency components and modulation components. This dynamic high-frequency component changes (frequency shift) as the distance between the illuminated part of the skin and the detector changes. Note that the amplitude of the vibration measurement may be, for example, 50 nm (eg, user whispers) to 1.5 microns (eg, user shouts). Therefore, the vibrations induced in the skin by sounds in local areas lead to changes in dynamic high-frequency components. The audio system measures 640 sounds by inferring corresponding sounds based on dynamic high frequency components that cause vibrations in the user's skin. In addition, since the vibration of the user's skin caused by the sound has a very different frequency from the vibration caused by the user's movement (e.g., walking, running), the audio system can convert the dynamic high-frequency component corresponding to the vibration from the local area. Partial isolation and/or filtering out of sound.

In some embodiments, the measured sound includes the user's speech, and the high frequency components of the speech are attenuated relative to the low frequencies of the speech. The audio system can reconstruct the high-frequency components of the speech through, for example, matrix decomposition-based bandwidth expansion, neural network-based audio super-resolution, etc. The audio system can then update the measured sound of the speech with the reconstructed high-frequency components.

Audio systems can perform various actions based in part on sound measured by optical microphones. And in some embodiments, where the audio system also includes a microphone array, the audio system may perform a variety of actions based in part on sound measured by the optical microphone, sound detected by the microphone array, or some combination thereof. Various actions may include, for example, enhancing the user's voice, using one or more optical microphones for voice activity detection (VAD), performing active noise reduction, etc.

Figure 9 is a system 900 including a headset 905 in accordance with one or more embodiments. In some embodiments, headset 905 may be headset 100 of Figure IA, or headset 105 of Figure IB. System 900 may operate in an artificial reality environment (eg, a virtual reality environment, an augmented reality environment, a mixed reality environment, or some combination thereof). System 900 shown in FIG. 9 includes a headset 905, an input/output (I/O) interface 910 coupled to a console 915, a network 920, and a map construction server 925. Although FIG. 9 shows an example system 900 that includes a headset 905 and an I/O interface 910, in other embodiments, any number of these components may be included in the system 900. For example, there may be multiple headsets, each having an associated I/O interface 910 , where each headset and I/O interface 910 communicate with a console 915 . In alternative configurations, system 900 may include different and/or additional components. Additionally, in some embodiments, functionality described in connection with one or more components shown in FIG. 9 may be distributed among the components in a manner different from that described in connection with FIG. 9 . For example, some or all of the functionality of console 915 may be provided by headset 905 .

Headset 905 includes display assembly 930 , optics block 935 , one or more positioning sensors 990 , and DCA 945 . Some embodiments of the headset 905 have different components than those described in conjunction with FIG. 9 . Additionally, in other embodiments, the functionality provided by the various components described in connection with FIG. 9 may be distributed differently among the components of the headset 905 , or in a separate component remote from the headset 905 . component was detected.

Display component 930 displays content to the user based on data received from console 915. Display component 930 displays content using one or more display elements (eg, display element 120). The display element may be, for example, an electronic display. In various embodiments, display component 930 includes a single display element or multiple display elements (eg, one display for each eye of the user). Examples of electronic displays include: liquid crystal display (LCD), organic light emitting diode (OLED) display, active-matrix organic light-emitting diode display (AMOLED), A waveguide display, some other display, or some combination thereof. Note that in some embodiments, display element 120 may also include some or all of the functionality of optics block 935 .

Optics block 935 may amplify image light received from the electronic display, correct optical errors associated with the image light, and present the corrected image light to one or both eye zones of the headset 905 . In various embodiments, optics block 935 includes one or more optical elements. Example optical elements included in optics block 935 include: apertures, Fresnel lenses, convex lenses, concave lenses, filters, reflective surfaces, or any other suitable optical element that affects image light. Additionally, optics block 935 may include a combination of different optical elements. In some embodiments, one or more optical elements in optics block 935 may have one or more coatings, such as a partially reflective coating or an anti-reflective coating.

The amplification and focusing of the image light by the optics block 935 allows the electronic display to be physically smaller, lighter weight and consume less power than larger displays. Additionally, magnification can increase the field of view of content presented on an electronic display. For example, the field of view of the displayed content is such that the displayed content is presented using substantially all of the user's field of view (e.g., approximately 110 degrees diagonal), and in some cases, the displayed content uses all of the user's field of view. The field of view is presented. Additionally, in some embodiments, the amount of amplification can be adjusted by adding or removing optical elements.

In some embodiments, optics block 935 may be designed to correct for one or more types of optical errors. Examples of optical errors include barrel or pincushion distortion, longitudinal chromatic aberration, or lateral chromatic aberration. Other types of optical errors may also include spherical aberration; chromatic aberration; or errors due to lens field curvature, astigmatism; or any other type of optical error. In some embodiments, the content provided to the electronic display for display is pre-distorted, and the optics block 935 corrects the distortion when it receives image light from the electronic display that is generated based on the content.

Position sensor 940 is an electronic device that generates data indicative of the position of headset 905 . Positioning sensor 940 generates one or more measurement signals in response to movement of headset 905 . Position sensor 190 is an embodiment of position sensor 940 . Examples of positioning sensors 940 include: one or more IMUs, one or more accelerometers, one or more gyroscopes, one or more magnetometers, another suitable type of sensor that detects motion, or some combination thereof . Positioning sensors 940 may include a plurality of accelerometers for measuring translational motion (forward/backward, up/down, left/right) and for measuring rotational motion (e.g., pitch, yaw, roll) of multiple gyroscopes. In some embodiments, the IMU rapidly samples the measurement signal and calculates an estimated position of the headset 905 based on the sampled data. For example, the IMU integrates measurement signals received from the accelerometer over time to estimate a velocity vector, and integrates the velocity vector over time to determine the estimated position of a reference point on the headset 905 . A reference point is a point that can be used to describe the positioning of headset 905 . Although the reference point may generally be defined as a point in space, in practice, the reference point is defined as a point within the headset 905 .

DCA 945 generates depth information for a portion of a local area. DCA includes one or more imaging devices and a DCA controller. The DCA 945 can also include an illuminator. The operation and structure of DCA 945 are described above with respect to Figure 1A.

Audio system 950 provides audio content to the user of headset 905 . Audio system 950 is substantially the same as audio system 200 described above. Audio system 950 may include one or more acoustic sensors (e.g., as part of a sensor array), one or more transducers (e.g., as part of a transducer array), one or more optical microphones, and audio controls device. As described above with respect to, for example, FIGS. 1-6, the output signal from one or more optical microphones facilitates the audio system 950 to perform well in low-pitched SNR environments. In some embodiments, output signals from one or more optical microphones may be used, for example, to calibrate a sensor array; for active noise reduction, VAD, etc. Audio system 950 can provide spatialized audio content to users. In some embodiments, audio system 950 may request acoustic parameters from map building server 925 over network 920 . Acoustic parameters describe one or more acoustic properties of a local area (e.g., room impulse response, reverberation time, reverberation level, etc.). Audio system 950 may provide information describing at least a portion of the local area and/or position information of headset 905 from positioning sensor 940 , for example. Audio system 950 may generate one or more sound filters using one or more of the plurality of acoustic parameters received from map building server 925 and use the sound filters to provide audio content to the user.

I/O interface 910 is a device that allows a user to send action requests to console 915 and receive responses from console 915 . An action request is a request to perform a specific action. For example, an action request may be an instruction to start or end detection of image or video data, or an instruction to perform a specific action within an application. I/O interface 910 may include one or more input devices. Example input devices include a keyboard, mouse, game controller, or any other suitable device for receiving and communicating action requests to console 915 . Action requests received by I/O interface 910 are passed to console 915, which performs actions corresponding to the action requests. In some embodiments, I/O interface 910 includes an IMU that detects calibration data indicating an estimated position of I/O interface 910 relative to an initial position of I/O interface 910 . In some embodiments, I/O interface 910 can provide tactile feedback to the user based on instructions received from console 915 . For example, tactile feedback is provided when an action request is received, or the console 915 transmits instructions to the I/O interface 910 when the console 915 performs an action, causing the I/O interface 910 to generate tactile feedback.

Console 915 provides content to headset 905 for processing based on information received from one or more of: DCA 945, headset 905, and I/O interface 910. In the example shown in FIG. 9 , console 915 includes application repository 955 , tracking module 960 and engine 965 . Some embodiments of the console 915 have different modules or components than those described in conjunction with FIG. 9 . Similarly, functionality described further below may be distributed among the various components of console 915 in a manner different from that described in connection with FIG. 9 . In some embodiments, the functionality discussed herein with respect to console 915 may be implemented in headset 905 or a remote system.

Application repository 955 stores one or more applications for execution by console 915. An application is a set of instructions that, when executed by a processor, generate content for presentation to a user. Content generated by the application may be responsive to input received from the user via movement of the headset 905 or I/O interface 910 . Examples of applications include: gaming applications, conferencing applications, video playback applications, or other suitable applications.

Tracking module 960 tracks the movement of headset 905 or the movement of I/O interface 910 using information from DCA 945, one or more positioning sensors 940, or some combination thereof. For example, tracking module 960 determines the location of a reference point of headset 905 in the construction of a map of the local area based on information from headset 905 . Tracking module 960 may also determine the location of an object or virtual object. Additionally, in some embodiments, tracking module 960 may use portions of the data from positioning sensor 940 indicating the positioning of headset 905 and representations of local areas from DCA 945 to predict the positioning of headset 905 future location. Tracking module 960 provides an estimated or predicted future position of headset 905 or I/O interface 910 to engine 965 .

Engine 965 executes the application and receives positioning information, acceleration information, velocity information, predicted future positioning, or some combination thereof of headset 905 from tracking module 960 . Based on the received information, engine 965 determines content to be provided to headset 905 for presentation to the user. For example, if information received indicates that the user has looked to the left, engine 965 generates content for headset 905 that reflects the user's presence in a virtual local area or a local area enhanced with additional content. of movement. Additionally, engine 965 performs actions within the application executing on console 915 in response to action requests received from I/O interface 910 and provides feedback to the user that the action was performed. The feedback provided may be visual feedback or auditory feedback via the headset 905 , or tactile feedback via the I/O interface 910 .

Network 920 couples headset 905 and/or console 915 to map building server 925 . Network 920 may include any combination of local area networks and/or wide area networks using wireless communication systems and/or wired communication systems. For example, network 920 may include the Internet and mobile phone networks. In one embodiment, network 920 uses standard communications technologies and/or protocols. Therefore, network 920 may include links using technologies such as Ethernet, 802.11, worldwide interoperability for microwave access (WiMAX), 2G/3G/9G mobile communication protocols, digital subscriber lines (DSL), etc. subscriber line (DSL), asynchronous transfer mode (ATM), unlimited bandwidth (InfiniBand), PCI Express Advanced Switching (PCIExpress Advanced Switching), etc. Similarly, networking protocols used on network 920 may include multiprotocol label switching (MPLS), transmission control protocol/Internet protocol (TCP/IP), User Datagram Protocol (UserDatagram) Protocol (UDP), hypertext transport protocol (HTTP), simple mail transfer protocol (SMTP), file transfer protocol (FTP), etc. Data exchanged over network 920 may be represented using technologies and/or formats including binary forms of image data (eg, Portable Network Graphics (PNG)), hypertext markup language (HTML), extensible markup language (XML), etc. In addition, conventional encryption technologies (such as secure sockets layer (SSL), transport layer security (TLS), virtual private network (VPN), Internet Protocol security, IPsec), etc.) to encrypt all or some links.

The map building server 925 may include a database storing virtual models describing a plurality of spaces, where a location in the virtual model corresponds to the current configuration of the local area of the headset 905 . The map construction server 925 receives information describing at least a portion of the local area and/or location information of the local area from the headset 905 via the network 920 . The user can adjust privacy settings to allow or prevent the headset 905 from sending information to the map building server 925 . The map construction server 925 determines the location in the virtual model associated with the local area of the headset 905 based on the received information and/or location information. The map building server 925 determines (eg, retrieves) one or more acoustic parameters associated with the local area based in part on the determined location in the virtual model and any acoustic parameters associated with the determined location. The map building server 925 may send the location of the local area and the values of any acoustic parameters associated with the local area to the headset 905 .

One or more components of system 900 may include a privacy module that stores one or more privacy settings for user data elements. The user data element describes the user or headset 905. For example, the user data elements may describe the user's physical characteristics, actions performed by the user, the user's location of the headset 905 , the location of the headset 905 , the user's HRTF, etc. Privacy settings (or "access settings") for user data elements may be stored in any suitable manner, such as, for example, stored in association with the user data element, stored in an index on the authorization server, stored in another suitable manner or any suitable combination of them.

Privacy settings for a User Data Element specify how the User Data Element (or certain information associated with the User Data Element) may be accessed, stored, or otherwise used (e.g., viewed, shared, modified, copied, performed, exposed, or identified). In some embodiments, a user data element's privacy setting may specify a "blacklist" of entities that may not have access to certain information associated with the user data element. Privacy settings associated with user data elements can specify any suitable granularity for granting access or denying access. For example, some entities may have permission to ascertain the existence of a specific user data element, some entities may have permission to view the content of a specific user data element, and some entities may have permission to modify a specific user data element. Privacy settings may allow users to allow other entities to access or store user data elements for a limited period of time.

Privacy settings may allow users to specify one or more geographic locations from which user data elements may be accessed. Access or denial of access to a user data element may depend on the geographic location of the entity attempting to access the user data element. For example, a user can allow access to user data elements and specify that the user data elements are accessible to an entity only when the user is in a specific location. If the user leaves that particular location, the user data element may no longer be accessible to that entity. As another example, a user may specify that user data elements are only accessible to entities within a threshold distance from the user (eg, another user of the headset within the same local area as the user). If the user subsequently changes location, entities with access to that user's data element may lose access while a new set of entities gain access when they come within a threshold distance of the user.

System 900 may include one or more authorization/privacy servers for enforcing privacy settings. A request from an entity for a specific user data element may identify the entity associated with the request and may only be requested if the authorization server determines that the entity is authorized to access the user data element based on the privacy settings associated with the user data element. Send this user data element to this entity. If the requesting entity is not authorized to access the user data element, the authorization server may prevent the requested user data element from being retrieved or may prevent the requested user data element from being sent to the entity. Although this disclosure describes implementing privacy settings in a particular manner, this disclosure contemplates implementing privacy settings in any suitable manner.

Additional configuration information

The foregoing description of embodiments is presented for the purpose of illustration, and is not intended to be exhaustive or to limit patent rights to the precise forms disclosed. Those skilled in the relevant art will appreciate that many modifications and variations are possible with respect to the above disclosure.

Some portions of this description describe various embodiments in terms of algorithms and symbolic representations of operations on information. These algorithmic descriptions and representations are commonly used by those skilled in the data processing arts to effectively convey the substance of their work to others skilled in the art. Although these operations are described functionally, computationally, or logically, these operations should be understood to be implemented by computer programs or equivalent circuits, microcode, or the like. Furthermore, it has proven convenient, without loss of generality, to sometimes refer to arrangements of these operations as modules. The described operations and their associated modules may be implemented in software, firmware, hardware, or any combination thereof.

Any steps, operations or processes described herein may be performed or implemented using one or more hardware or software modules, alone or in combination with other devices. In one embodiment, the software modules are implemented as a computer program product comprising a computer-readable medium embodying computer program code, the computer program code being executable by a computer processor for performing any or all of the described A step, operation, or process.

Embodiments may also relate to an apparatus for performing the operations herein. Such apparatus may be specially constructed for the required purposes, and/or such apparatus may comprise a general-purpose computing device selectively activated or reconfigured by a computer program stored in the computer. Such computer program may be stored on a non-transitory tangible computer-readable storage medium or any type of medium suitable for storing electronic instructions, which may be coupled to a computer system bus. Furthermore, any computing system mentioned in this specification may include a single processor, or may be an architecture designed with multiple processors for increased computing power.

Embodiments may also relate to a product resulting from the computational processes described herein. Such a product may include information resulting from a computing process, where the information is stored on a non-transitory tangible computer-readable storage medium and may include any embodiment of a computer program product or other data combination described herein.

Finally, the language used in this specification has been selected primarily for readability and instructional purposes, and the language may not have been selected to describe or qualify patent rights. Accordingly, it is intended that the scope of patent rights be limited not by this detailed description, but by any claims published on an application based hereof. Accordingly, the disclosure of the embodiments is intended to be illustrative and not limiting of the scope of patent rights, which is set forth in the appended claims.

Claims (20)

1. An audio system, comprising: Optical microphone, the optical microphone includes: a light source, the light source is configured to emit light, the light includes a reference beam and a sensing beam, the light source is configured to illuminate the skin of the user with the sensing beam, wherein the sound from the local area is heard on the skin causing vibration in a detector in an interference configuration with the light source such that the detector is configured to detect a mixed signal corresponding to mixing with a portion of the sensing beam reflected by the skin the reference beam; and An audio controller configured to measure the sound using the mixed signal. 2. The audio system of claim 1, wherein the interference configuration is such that the light source and the detector form at least one of: a self-mixing interferometer, a Michelson interferometer, a low-coherence interference system, a laser Doppler vibrometer or some other type of interferometric system. 3. The audio system of claim 1, wherein the light source is adjacent the detector, the optical microphone further comprising: a lens coupled to the light source, the lens configured to: splitting the light emitted from the light source into the reference beam and the sensing beam; and The sensing beam is directed to the skin and the reference beam is reflected to the detector. 4. The audio system of claim 1, further comprising: Second optical microphone, including: a second light source configured to emit light including a second reference beam and a second sensing beam, the light source configured to illuminate the skin of the user with the sensing beam, a second detector, the second detector and the second light source being in the interference configuration such that the second detector is configured to detect a second mixed signal, the second mixed signal corresponding to the second reference beam mixed with the portion of the second sensing beam reflected by the skin; a block including a first side coupled to the optical microphone and a second side coupled to the second optical microphone; and a lens coupled to the light source, the block, and the second light source, the lens being configured to: splitting the light emitted from the light source into the reference beam and the sensing beam, and splitting the light emitted from the second light source into the second reference beam and the second sensing beam, directing the sensing beam and the second sensing beam to the skin, reflect the reference beam to the detector, and The second reference beam is reflected to the second detector. 5. The audio system of claim 1, wherein the detector and the light source are separated from each other by a threshold distance. 6. The audio system of claim 1, wherein the optical microphone is located on a headset including a nose pad, the optical microphone is integrated into the nose pad, and the light source is configured to The skin of the user's nose is illuminated with the sensing beam. 7. The audio system of claim 1, wherein the optical microphone is located on a headset including an eyeglass frame, and the optical microphone is integrated into the eyeglass frame, and the light source is configured to The skin on the user's face is irradiated with the sensing beam, and optionally, the audio system further includes a second optical microphone integrated into the glasses frame in connection with the The first optical microphone is in a different position, and the second optical microphone is configured to illuminate a portion of the skin on the face of the user with a second sensing beam that is different from the optical microphone. 8. The audio system of claim 1, wherein the optical microphone is located on a headset, the audio system further comprising: a microphone array located on the headset, the microphone array configured to detect the sound from the local area; Wherein, the audio controller is further configured to use the detected sound to calibrate the optical microphone. 9. The audio system of claim 1, wherein the optical microphone is located on a headset, the audio system further comprising: a microphone array located on the headset, the microphone array configured to detect the sound from the local area; wherein the audio controller is further configured to enhance the measured sound based in part on the detected sound. 10. The audio system of claim 1, wherein the audio controller is further configured to determine an expression of the user's face based in part on the measured sound. 11. The audio system of claim 1, wherein the optical microphone is located on a headset, and the audio controller is further configured to: identify noise in measured sounds; Generate sound filters to suppress the identified noise, and applying said sound filter to modify an audio signal corresponding to audio content, Wherein, the audio system also includes: a transducer array integrated into the headset, the transducer array configured to present a modified audio signal to the user as modified audio content, The modified audio content includes the audio content and a suppression component that suppresses the noise. 12. The audio system of claim 1, wherein the optical microphone is located on a headset, the audio system further comprising: a microphone array located on the headset, the microphone array configured to detect the sound from the local area, the sound from the local area including the audio The voice of the user of the system; Wherein, the audio controller is also configured as: using the measured sounds to identify the user's voice among the detected sounds; and updating a sound filter based on the recognized speech of the user, Wherein, the audio content is modified using the updated sound filter, and the modified audio content is presented by at least one audio system. 13. The audio system of claim 12, wherein the updated sound filter enhances the user's voice, and the audio controller is further configured to: Modify the audio content with the updated filter, wherein the modified audio content enhances the user's voice; and The modified audio content is provided to a second audio system, wherein the second audio system presents the modified audio content. 14. The audio system of claim 12, wherein the updated sound filter enhances the user's voice, and the audio controller is further configured to: Modify the audio content with the updated filter, wherein the modified audio content enhances the user's voice; determining that the modified audio content includes a command; and Perform actions based on the command. 15. A method comprising: Emit light from a light source of the optical microphone, the light including a reference beam and a sensing beam; irradiating the user's skin with the sensing beam, wherein sound from the localized area induces vibrations in the skin; detecting a mixed signal by a detector in an interference configuration with the light source, the mixed signal corresponding to the reference beam mixed with the portion of the sensing beam reflected by the skin; and The sound is measured using the mixed signal. 16. The method of claim 15, wherein the vibration of the skin is caused in part by the user's voice, the method further comprising: detecting the sound from the local area through a microphone array; identifying the voice of the user among the detected sounds using the measured sounds; and updating a sound filter based on the recognized speech of the user, Wherein, the audio content is modified using the updated sound filter, and the modified audio content is presented by at least one audio system. 17. The method of claim 15, wherein the interference configuration is such that the light source and the detector form at least one of: a self-mixing interferometer, a Michelson interferometer, a low-coherence interference system, a laser multi- Puller vibrometer or some other type of interferometric system. 18. The method of claim 15, wherein the measured sound includes the user's voice, and the high frequency components of the voice are attenuated relative to the low frequencies of the voice, the method further comprising: reconstructing the high-frequency components of the speech; and The measured sound of the speech is updated with the reconstructed high frequency components. 19. A non-transitory computer-readable medium configured to store program code instructions that, when executed by a processor of an audio system, cause the audio system to perform The method according to any one of claims 15 to 18 or a plurality of steps comprising the following steps: Emit light from a light source of the optical microphone, the light including a reference beam and a sensing beam; irradiating the user's skin with the sensing beam, wherein sound from the localized area induces vibrations in the skin; detecting a mixed signal by a detector in an interference configuration with the light source, the mixed signal corresponding to the reference beam mixed with the portion of the sensing beam reflected by the skin; and The sound is measured using the mixed signal. 20. A computer program comprising instructions which, when executed by a processor of an audio system, cause the audio system to perform a method according to any one of claims 15 to 18 Or multiple steps including: Emit light from a light source of the optical microphone, the light including a reference beam and a sensing beam; irradiating the user's skin with the sensing beam, wherein sound from the localized area induces vibrations in the skin; detecting a mixed signal by a detector in an interference configuration with the light source, the mixed signal corresponding to the reference beam mixed with the portion of the sensing beam reflected by the skin; and The sound is measured using the mixed signal.