CN115211144A - Hearing aid system and method - Google Patents

Abstract

A hearing aid and associated systems and methods. In one implementation, a hearing aid system may include a wearable camera configured to capture images from an environment of a user, a microphone configured to capture sound from the environment of the user, and a processor. The processor may be programmed to receive an image captured by the camera; receiving an audio signal representing sound captured by a microphone; operating in a first mode to cause a first selective adjustment of a first audio signal; determining, based on an analysis of at least one of the image or the audio signal, to switch to a second mode to cause a second selective adjustment to the first audio signal; and causing the first audio signal, selectively adjusted in the second mode, to be transmitted to an auditory interface device configured to provide sound to an ear of a user.

Description

Hearing aid systems and methods

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims US Provisional Patent Application No. 62/956,744, filed January 3, 2020; US Provisional Patent Application No. 62/970,726, filed February 6, 2020; Priority of Provisional Patent Application No. 63/050,890. All of the foregoing applications are incorporated herein by reference in their entirety.

technical field

The present disclosure generally relates to apparatus and methods for capturing and processing images and audio from a user's environment and using information derived from the captured images and audio.

Background technique

Advances in technology today enable wearable devices to automatically capture images and audio, and store information associated with the captured images and audio. Certain devices have been used to digitally record aspects of an individual's life and individual experiences, a practice commonly referred to as "life logging." Some people record their lives so they can retrieve moments from past activities such as social events, travel, etc. Life logging may also have significant benefits in other fields (eg, business, fitness and healthcare, and social research). Life-logging devices, while useful for tracking daily activities, can also be improved through feedback and other advanced features based on analysis of captured image and audio data to enhance an individual's ability to interact in their environment.

Although users can capture images and audio with their smartphones, and some smartphone applications can process the captured information, given their size and design, smartphones may not be the best platform for use as life-logging devices. Life recording devices should be small and light so they can be worn easily. In addition, as image capture devices, including wearables, improve, additional functionality can be provided to help users navigate in and around their environment, identify people and objects they encounter, and provide users with information about their surroundings and activities feedback of. Accordingly, there is a need for apparatus and methods for automatically capturing and processing images and audio to provide useful information to a user of the apparatus, as well as systems and methods for processing and utilizing the information collected by the apparatus.

SUMMARY OF THE INVENTION

Embodiments in accordance with the present disclosure provide apparatus and methods for automatically capturing and processing images and audio from a user's environment, and systems and methods for processing information related to images and audio captured from a user's environment.

In one embodiment, a hearing aid system for selectively adjusting sound is provided. The hearing aid system includes a wearable camera configured to capture a plurality of images from the user's environment and at least one microphone configured to capture sound from the user's environment. Additionally, the hearing aid system includes at least one processor programmed to receive a plurality of images captured by the camera, receive a plurality of audio signals from the user's environment representing sounds captured by the at least one microphone, and operate in the first mode to cause a first selective adjustment of a first audio signal of the plurality of audio signals. The processor is further programmed to, based on the analysis of at least one of the plurality of images or the plurality of audio signals, determine to switch to the second mode to cause a second selective adjustment of the first audio signal, the second selective adjustment Different in at least one respect relative to the first selective adjustment. The processor is also programmed to transmit the selectively conditioned first audio signal in the second mode to an auditory interface device configured to provide sound to the ear of the user.

In one embodiment, a hearing aid system for selectively adjusting sound is provided. The hearing aid system includes a wearable camera configured to capture a plurality of images from the user's environment and at least one microphone configured to capture sound from the user's environment. The hearing aid system also includes at least one processor programmed to receive a plurality of images captured by the camera, receive a plurality of audio signals from the user's environment representing sounds captured by the at least one microphone, and operate in a plurality of modes, wherein the plurality of modes includes a first mode and a second mode, wherein operating in the first mode causes a first selective adjustment of at least one audio signal of the plurality of audio signals, and wherein operating in the second mode causes a A second selective adjustment of the at least one audio signal, the second selective adjustment being different from the first selective adjustment in at least one respect. Further, the processor is programmed to select the first mode or the second mode based on analysis of at least one of the plurality of images or the plurality of audio signals, and to transmit the at least one audio signal selectively adjusted based on the selected mode To the auditory interface device, the auditory interface device is configured to provide sound to the user's ear.

In one embodiment, a hearing aid system for selectively adjusting sound is provided. The hearing aid system includes a wearable camera configured to capture a plurality of images from a user's environment, at least one microphone configured to capture a plurality of audio signals from the user's environment, and at least one processor. The processor is programmed to receive a plurality of images captured by the camera, receive a plurality of audio signals from the user's environment representing sounds captured by the at least one microphone, identify at least one of the plurality of images or a plurality of audio signals from the plurality of audio signals at least one identified individual of at least one representation, and a conditioning profile associated with the at least one identified individual is retrieved from memory. Further, the processor is programmed to cause selective adjustment of a first audio signal of the plurality of audio signals associated with the at least one identified individual, the selective adjustment determined based on the adjustment profile and to cause The conditioned first audio signal is transmitted to an auditory interface device configured to provide sound to a user's ear.

In one embodiment, a hearing aid system for selectively adjusting sound is provided. The hearing aid system includes a wearable camera configured to capture a plurality of images from a user's environment, at least one microphone configured to capture a plurality of audio signals from the user's environment, and at least one processor. The processor is programmed to receive a plurality of images captured by the camera, receive a plurality of audio signals from the user's environment representing sounds captured by the at least one microphone, identify at least one of the plurality of images or a plurality of audio signals from the plurality of audio signals at least one represented group of individuals, and retrieving from memory a conditioning profile associated with the group of individuals. Further, the processor is programmed to cause selective adjustment of a first audio signal of the plurality of audio signals associated with the individual group, the selective adjustment determined based on the adjustment profile and to cause The conditioned first audio signal is transmitted to an auditory interface device configured to provide sound to a user's ear.

In one embodiment, a hearing aid system for selectively amplifying sound may include a wearable camera configured to capture a plurality of images from a user's environment, at least one microphone configured to capture sound from the user's environment, and at least one processor. The processor may be programmed to receive a plurality of images captured by the wearable camera; receive at least one audio signal from the user's environment representing sound captured by the at least one microphone; analysis of the signal to identify at least one action of the user; based on the identified action, causing selective adjustment of the at least one audio signal received by the at least one microphone; and causing the at least one adjusted audio signal to be transmitted to the auditory interface device, the The auditory interface device is configured to provide sound to the user's ear.

In one embodiment, a method for selectively amplifying sound may include: capturing a plurality of images from a user's environment via a wearable camera; capturing sound from the user's environment via at least one microphone; receiving captures by the wearable camera receiving at least one audio signal representing sound captured by at least one microphone from the user's environment; identifying at least one action of the user based on an analysis of at least one of the plurality of images or the at least one audio signal; identifying and causing the at least one adjusted audio signal to be transmitted to the auditory interface device configured to provide sound to the user's ear.

In one embodiment, a hearing aid system for selectively amplifying sound may include a wearable camera configured to capture a plurality of images from a user's environment, at least one microphone configured to capture sound from the user's environment, and at least one processor. The processor may be programmed to: receive a plurality of images captured by the camera; identify a representation of a first individual in at least one of the plurality of images; receive an audio signal representing sound captured by the at least one microphone; analysis of the audio signal, identifying a first audio signal associated with a first speech associated with the first individual, and a second audio signal associated with a second speech of the second individual; determining the first audio signal based on the at least one processor The audio signal is associated with a priority higher than the priority of the second audio signal, causing selective conditioning of the first audio signal; and causing the selectively conditioned first audio signal to be transmitted to the auditory interface device, the auditory interface device is configured to provide sound to a user's ear.

In one embodiment, a method for selectively amplifying sound may include: capturing a plurality of images from a user's environment via a wearable camera; capturing sound from the user's environment via at least one microphone; receiving captures by the wearable camera identifying a representation of a first individual in at least one of the plurality of images; receiving an audio signal representing sound captured by the at least one microphone; identifying an association with the first individual based on analysis of the audio signal a first audio signal associated with an associated first voice, and a second audio signal associated with a second voice of a second individual; determining the priority of the first audio signal with a higher priority than the second audio signal based on the at least one processor cause a selective adjustment of the first audio signal; and cause the selectively adjusted first audio signal to be transmitted to an auditory interface device configured to provide sound to a user's ear.

In one embodiment, a hearing aid system can selectively amplify sound. The hearing aid system may include a wearable camera device including: at least one camera configured to capture a plurality of images from the user's environment; at least one microphone configured to capture sound from the user's environment; and at least one first processor programmed to selectively condition audio signals received from at least one microphone representing sound captured by the at least one microphone; and a hearing aid device comprising: at least one speaker that is configured to provide sound to a user's ear; and at least one second processor programmed to: cause one or more instructions to be transmitted to the wearable camera device; receive the adjusted audio signal from the wearable camera device; and based on the The conditioned audio signal provides sound to the ear of the user using at least one speaker.

In one embodiment, a method for amplifying sound in a hearing aid system may include: capturing a plurality of images from a wearable user's environment using at least one camera of a wearable camera device; using at least one of the wearable camera devices a microphone to capture sound from a user's environment; to selectively condition an audio signal received from the at least one microphone using the at least one first processor, the audio signal representing the sound captured by the at least one microphone; and by using at least one second of the hearing aid device a processor for providing sound to a user's ear using at least one speaker of the hearing aid device, the at least one second processor being programmed to: cause one or more instructions to be transmitted to the wearable camera device; an audio signal; and using the at least one speaker to provide sound to a user's ear based on the conditioned audio signal.

In one embodiment, a hearing aid system for selectively amplifying sound includes: a wearable camera configured to capture a plurality of images from a user's environment, the wearable camera having image capture parameters; at least one microphone, configured by configured to capture sound from a user's environment; and at least one processor programmed to: receive a plurality of images captured by the camera; receive an audio signal representing the sound captured by the at least one microphone; identify at least one image of the plurality of images a representation of at least one individual in; detecting a speech rate associated with the at least one individual based on at least one of a plurality of images or audio signals; and causing adjustments to image capture parameters of the wearable camera based on the detected speech rate .

In one embodiment, a method for amplifying sound includes: receiving a plurality of images captured by a camera; receiving an audio signal representing sound captured by at least one microphone; identifying at least one of the at least one image of the plurality of images a representation of an individual; detecting a speech rate associated with the at least one individual based on at least one of a plurality of image or audio signals; and causing adjustments to image capture parameters of the wearable camera based on the detected speech rate.

In one embodiment, a hearing aid system may include at least one microphone configured to capture sound from a user's environment; and at least one processor. The at least one processor may be programmed to: receive an audio signal representing sound captured by the at least one microphone; receive an indication of a time delay associated with processing the audio signal; and store a plurality of audios representing portions of the audio signal in a buffer samples; and processing a first audio sample of the plurality of audio samples to generate a processed first audio sample. Processing the first audio sample may include analyzing a second audio sample of the plurality of audio samples, the second audio sample being represented in the audio signal after the first audio sample and having a length defined by a time delay, wherein the processed first audio sample The audio quality of one audio sample depends on the length of the second audio sample.

In one embodiment, a method for selectively amplifying an audio signal is disclosed. The method may include: receiving an audio signal representing sound received by at least one microphone from a user's environment; receiving an indication of a time delay associated with processing the audio signal; and storing in a buffer a plurality of audio samples representing portions of the audio signal ; and processing a first audio sample of the plurality of audio samples to generate a processed first audio sample. Processing the first audio sample may include analyzing a second audio sample represented in the audio signal after the first audio sample and having a length defined by a time delay, wherein the audio quality of the processed first audio sample Depends on the length of the second audio sample.

In one embodiment, a hearing aid system may include at least one microphone configured to capture sound from a user's environment; and at least one processor. The at least one processor may be programmed to: receive an audio signal representing sound captured by the at least one microphone; process the audio signal using the first value of the at least one aspect to generate a first processed audio signal; and use the first value of the at least one aspect The audio signal is binary processed to generate a second processed audio signal, the second value being different from the first value. The at least one processor may also be programmed to compare the first processed audio signal to the second processed audio signal to compare the audio quality of the first processed audio signal and the second processed audio signal based on at least one aspect selecting the first processed audio signal or the second processed audio signal; and sending the selected processed audio signal to the user's auditory interface device.

In one embodiment, a method for selectively amplifying an audio signal is disclosed. The method may include: receiving an audio signal representing sound received by at least one microphone from a user's environment; processing the audio signal using the first value of the at least one aspect to generate a first processed audio signal; and using the second value of the at least one aspect The value processes the audio signal to generate a second processed audio signal, the second value being different from the first value. The method may also include comparing the first processed audio signal to the second processed audio signal to determine based on at least one aspect and a compromise between audio quality of the first processed audio signal and the second processed audio signal selecting the first processed audio signal or the second processed audio signal; and sending the selected processed audio signal to the user's auditory interface device.

In one embodiment, a hearing aid system for selectively replacing audio signals may include a wearable camera configured to capture a plurality of images from a user's environment, at least one microphone configured to capture sounds from the user's environment ; and at least one processor. The at least one processor may be programmed to: receive a plurality of images captured by the camera; receive a plurality of audio signals representing sounds captured by the at least one microphone; and, based on an analysis of the plurality of images or the plurality of audio signals, select from a plurality of Audio signals associated with sound-emitting objects in the user's environment are identified from among the audio signals. The at least one processor may also be configured to: predict, based on the plurality of audio signals, the sound to be received at the user's ear from the user's environment; generate a cancellation audio signal configured to cancel at least the predicted sound at the user's ear; based on generating a selectively conditioned audio signal from the identified audio signal; and sending the canceling audio signal and the selectively conditioned audio signal to a hearing aid interface device configured to provide sound to a user's ear.

In one embodiment, a method for selectively replacing an audio signal is disclosed. The method may include receiving a plurality of images captured from the user's environment by the wearable camera; receiving a plurality of audio signals representing sounds captured from the user's environment by the at least one microphone; and based on the analysis of the plurality of images or the plurality of audio signals The analysis identifies, from the plurality of audio signals, audio signals associated with sound-emitting objects in the user's environment. The method may further include: predicting, based on the plurality of audio signals, sounds to be received at the user's ears from the user's environment; generating a cancellation audio signal configured to cancel at least the predicted sounds at the user's ears; based on the identified audio signals to generate a selectively conditioned audio signal; and sending the cancel audio signal and the selectively conditioned audio signal to a hearing aid interface device configured to provide sound to a user's ear.

In one embodiment, a hearing aid system for selectively amplifying sound includes a wearable camera configured to capture a plurality of images from a user's environment, at least one microphone configured to capture sound from the user's environment, and at least one a processor. The at least one processor is configured to receive a plurality of images captured by the camera; receive an audio signal representing sound captured by the at least one microphone; and determine, based on an analysis of at least one of the plurality of images or audio signals, the sound originating object's a location; generating a stereo representation based on the location of the sound-emitting object, the stereo representation including a first audio signal and a second audio signal, the first audio signal being different from the second audio signal in at least one respect to simulate the location of the object relative to the user; and The stereo representation is transmitted to a hearing aid interface device configured to provide sound based on the first audio signal to a first ear of the user and sound based on the second audio signal to a second ear of the user.

In one embodiment, a method for selectively amplifying sound includes: receiving a plurality of images from a user's environment, the images being captured by a wearable camera; receiving audio signals from the user's environment representing sound, the sound being captured by a wearable camera at least one microphone capture; determining the location of the sound emitting object based on analysis of at least one of the plurality of images or audio signals; generating a stereo representation based on the location of the sound emitting object, the stereo representation including the first audio signal and the second audio signal , the first audio signal differs from the second audio signal in at least one aspect to simulate the position of the object relative to the user; and causing the stereo representation to be transmitted to a hearing aid interface device configured to provide sound based on the first audio signal to the first ear of the user, and the sound based on the second audio signal is provided to the second ear of the user.

In one embodiment, a hearing aid system for selectively amplifying sound includes at least one microphone configured to capture sound from a user's environment; and at least one processor. The at least one processor is configured to receive a plurality of audio signals representing sounds captured by the at least one microphone; identify a first audio signal of the plurality of audio signals, the first audio signal being associated with an individual; and process the first audio signal to selectively adjust at least one speech characteristic of the individual; and causing the processed first audio signal to be transmitted to an auditory interface device configured to provide sound to a user's ear.

In one embodiment, a method for selectively amplifying sound includes: receiving a plurality of audio signals representing sound captured by at least one microphone configured to capture sound from a user's environment; identifying multiple audio signals a first audio signal of the audio signals associated with the individual; processing the first audio signal to selectively adjust at least one speech characteristic of the individual; and transmitting the processed first audio signal to the auditory interface A device configured to provide sound to a user's ear.

In one embodiment, a hearing aid system is provided. The hearing aid system can selectively adjust the sound. The hearing aid system may include a wearable camera configured to capture a plurality of images from the user's environment, the wearable camera having an image capture rate; at least one microphone configured to capture sound from the user's environment; and at least one processing. The processor may be programmed to receive a plurality of images captured by the camera; receive a plurality of audio signals representing sounds captured by the at least one microphone; obtain a voiceprint associated with an individual in the user's environment, the individual based on a plurality of at least one of an image or a plurality of audio signals is identified; based on the analysis of the plurality of images, detecting at least one lip movement associated with the individual's mouth; based on at least one of the voiceprint or the detected lip movement, identifying a first audio signal of the plurality of audio signals that is associated with the individual's speech; processing the first audio signal; and transmitting the selectively conditioned first audio signal to an auditory interface device configured to communicate to a user ear provides sound.

In one embodiment, a computer-implemented method for selectively adjusting sound in a hearing aid system is provided. The method may include receiving a plurality of images from the user's environment, the plurality of images being captured by the wearable camera. The method may also include receiving a plurality of audio signals representing sound captured by the at least one microphone. The method may also include obtaining a voiceprint associated with an individual in the user's environment, the individual being identified based on at least one of the plurality of images or the plurality of audio signals. The method may also include detecting at least one lip movement associated with the individual's mouth based on the analysis of the plurality of images. The method may also include identifying a first audio signal of the plurality of audio signals associated with the individual's speech based on at least one of the voiceprint or the detected lip movement. The method may also include processing the first audio signal. The method may also include transmitting the selectively conditioned first audio signal to an auditory interface device configured to provide sound to a user's ear.

Consistent with other disclosed embodiments, a non-transitory computer-readable storage medium may store program instructions that are executed by at least one processor and perform any of the methods described herein.

The foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the claims.

Description of drawings

The accompanying drawings, which are incorporated in and constitute a part of this disclosure, illustrate various disclosed embodiments. In the attached image:

1A is a schematic diagram of an example of a user wearing a wearable device according to disclosed embodiments.

1B is a schematic diagram of an example of a user wearing a wearable device in accordance with disclosed embodiments.

1C is a schematic diagram of an example of a user wearing a wearable device according to disclosed embodiments.

1D is a schematic diagram of an example of a user wearing a wearable device, according to disclosed embodiments.

2 is a schematic diagram of an example system consistent with the disclosed embodiments.

3A is a schematic diagram of an example of the wearable device shown in FIG. 1A .

3B is an exploded view of the example of the wearable device shown in FIG. 3A.

4A-4K are schematic diagrams of the example of the wearable device shown in FIG. 1B from various perspectives.

5A is a block diagram showing an example of components of the wearable device according to the first embodiment.

5B is a block diagram illustrating an example of components of the wearable device according to the second embodiment.

5C is a block diagram illustrating an example of components of the wearable device according to the third embodiment.

6 illustrates an exemplary embodiment of a memory containing software modules consistent with the present disclosure.

7 is a schematic diagram of an embodiment of a wearable device including an orientable image capture unit.

8 is a schematic diagram of an embodiment of a wearable device secureable to clothing consistent with the present disclosure.

9 is a schematic diagram of a user wearing a wearable device consistent with embodiments of the present disclosure.

10 is a schematic diagram of an embodiment of a wearable device that can be secured to clothing in accordance with the present disclosure.

11 is a schematic diagram of an embodiment of a wearable device secureable to clothing consistent with the present disclosure.

12 is a schematic diagram of an embodiment of a wearable device secureable to clothing consistent with the present disclosure.

13 is a schematic diagram of an embodiment of a wearable device secureable to clothing consistent with the present disclosure.

14 is a schematic diagram of an embodiment of a wearable device secureable to clothing consistent with the present disclosure.

15 is a schematic diagram of an embodiment of a wearable device power supply unit including a power supply.

16 is a schematic diagram of an exemplary embodiment of a wearable device including a protection circuit.

17A is a schematic diagram of an example of a user wearing an apparatus for a camera-based hearing aid device according to disclosed embodiments.

17B is a schematic diagram of an embodiment of a device that is secureable to a garment in accordance with the present disclosure.

18 is a schematic diagram illustrating an exemplary environment for using a camera-based hearing aid consistent with the present disclosure.

19 is a flow diagram illustrating an exemplary process for selectively amplifying sound emanating from a detected line of sight of a user, consistent with the disclosed embodiments.

20A is a schematic diagram illustrating an exemplary environment for using a hearing aid with speech and/or image recognition consistent with the present disclosure.

20B illustrates an exemplary embodiment of an apparatus including a face and voice recognition component consistent with the present disclosure.

21 is a flowchart illustrating an exemplary process for selectively amplifying an audio signal associated with a recognized individual's speech, consistent with the disclosed embodiments.

22 is a flowchart illustrating an exemplary process for selectively transmitting an audio signal associated with a recognized voice of a user, consistent with the disclosed embodiments.

23A is a schematic diagram illustrating an exemplary individual identifiable in a user's environment consistent with the present disclosure.

23B is a schematic diagram illustrating an exemplary individual identifiable in a user's environment consistent with the present disclosure.

23C illustrates an exemplary lip tracking system consistent with the disclosed embodiments.

24 is a schematic diagram illustrating an exemplary environment for using a lip tracking hearing aid consistent with the present disclosure.

25 is a flowchart illustrating an exemplary process for selectively amplifying an audio signal based on tracked lip movement, consistent with the disclosed embodiments.

26 is a schematic diagram of an example of a user wearing an apparatus for a camera-based hearing aid device according to disclosed embodiments.

27A and 27B are flowcharts illustrating exemplary processes for selectively conditioning audio signals, consistent with the disclosed embodiments.

28A and 28B are schematic diagrams illustrating exemplary processes for selectively conditioning audio signals consistent with disclosed embodiments.

29 is a schematic diagram of an example of a user wearing an apparatus for a camera-based hearing aid device according to disclosed embodiments.

30A and 30B are flowcharts illustrating exemplary processes for selectively conditioning audio signals, consistent with the disclosed embodiments.

31 is another flow diagram illustrating an exemplary process for selectively conditioning an audio signal, consistent with the disclosed embodiments.

32 is a schematic diagram illustrating an exemplary environment for using a hearing aid with speech and/or image recognition in accordance with the disclosed embodiments.

33 is an exemplary depiction of a user with a hearing aid system in accordance with the disclosed embodiments.

34 is a flowchart illustrating an exemplary process for selectively amplifying sound in accordance with disclosed embodiments.

35 is a schematic diagram illustrating an exemplary environment including a hearing aid with speech and/or image recognition in accordance with the disclosed embodiments.

36 is an illustration of an exemplary computing device for use with a hearing aid with speech and/or image recognition in accordance with disclosed embodiments.

37 is a flowchart illustrating an exemplary process for selectively amplifying sound in accordance with disclosed embodiments.

38 is a schematic diagram of an example of a hearing aid system including a wearable camera device, a hearing aid device, and a mobile device consistent with the disclosed embodiments.

39 is a schematic diagram of an example of a hearing aid system attached to a user consistent with the disclosed embodiments.

40 is a flowchart illustrating an exemplary process for a hearing aid and paired camera system consistent with the disclosed embodiments.

41 is a schematic diagram of an example of a hearing aid system attached to a user consistent with the disclosed embodiments.

42 is a schematic diagram of an example of a hearing aid system that captures images and audio from a user's environment, consistent with the disclosed embodiments.

43 is a flowchart illustrating an exemplary process for adjusting capture parameters of a wearable camera consistent with disclosed embodiments.

Figure 44 illustrates an example audio signal that may be processed in accordance with the disclosed embodiments.

45A illustrates an example user interface through which a user may define aspects of processing audio signals, consistent with the disclosed embodiments.

45B illustrates an example process for parallel processing of audio signals consistent with disclosed embodiments.

46A is a flowchart illustrating an example process for selectively amplifying an audio signal, consistent with the disclosed embodiments.

46B is a flowchart illustrating an example process for selectively amplifying an audio signal, consistent with the disclosed embodiments.

47 is a block diagram illustrating an example process for active sound replacement consistent with the disclosed embodiments.

48A, 48B, and 48C illustrate example wearable devices for active sound replacement consistent with disclosed embodiments.

49 is a flowchart illustrating an example process for selectively replacing audio signals consistent with disclosed embodiments.

50A is a schematic diagram illustrating an exemplary environment for using a hearing aid with sound localization consistent with the disclosed embodiments.

50B is a schematic diagram of an exemplary image captured by an imaging capture device consistent with disclosed embodiments.

51 is a schematic diagram of an audio signal acquired and played back by a hearing aid system consistent with disclosed embodiments.

52 is a flowchart illustrating an exemplary process for generating a stereo representation consistent with the disclosed embodiments.

53 is a schematic diagram illustrating an exemplary environment for using a hearing aid with sound localization consistent with the present disclosure.

54A is a schematic diagram of an audio signal acquired by a hearing aid system consistent with the present disclosure.

54B is a schematic diagram of an audio signal played back by a hearing aid system consistent with the present disclosure.

55A is a flowchart illustrating an exemplary process for selectively conditioning an audio signal, consistent with the disclosed embodiments.

55B is a flowchart illustrating an exemplary process for determining speech characteristics based on visual recognition of an individual, consistent with disclosed embodiments.

56 is a schematic diagram of an exemplary hearing aid system for selectively modulating sound in accordance with disclosed embodiments.

57 is a schematic diagram illustrating an exemplary environment for a user of a hearing system consistent with the disclosed embodiments.

58 is a schematic diagram illustrating a flow diagram of an exemplary method for selectively adjusting sound in a hearing aid system consistent with disclosed embodiments.

Detailed ways

The following detailed description refers to the accompanying drawings. Wherever possible, the same reference numbers will be used in the drawings and the following description to refer to the same or like parts. Although several illustrative embodiments are described herein, modifications, adaptations, and other implementations are possible. For example, substitutions, additions, or modifications may be made to components shown in the figures, and the illustrative methods described herein may be modified by substituting, reordering, removing, or adding steps to the disclosed methods. Therefore, the following detailed description is not limited to the disclosed embodiments and examples. Rather, the appropriate scope is defined by the appended claims.

FIG. 1A shows a user 100 wearing a device 110 physically connected (or integrated) to glasses 130 in accordance with the disclosed embodiments. The glasses 130 may be prescription glasses, magnifying glasses, non- prescription glasses, safety glasses, sunglasses, and the like. Additionally, in some embodiments, eyeglasses 130 may include portions of a frame and earpiece, nose piece, etc., and one or no lenses. Accordingly, in some embodiments, glasses 130 may be primarily used to support apparatus 110, and/or augmented reality display devices or other optical display devices. In some embodiments, the apparatus 110 may include an image sensor (not shown in FIG. 1A ) for capturing real-time image data of the user's 100 field of view. The term "image data" includes any form of data retrieved from optical signals in the near infrared, infrared, visible and ultraviolet spectrums. Image data may include video clips and/or photographs.

In some embodiments, apparatus 110 may communicate wirelessly or via wired communication with computing device 120 . In some embodiments, computing device 120 may include, for example, a smartphone, tablet, or dedicated processing unit, which may be portable (eg, may be carried in the pocket of user 100). Although shown as an external device in FIG. 1A , in some embodiments, computing device 120 may be provided as part of wearable device 110 or glasses 130 , whether integrated or mounted thereon. In some embodiments, computing device 120 may be included in an augmented reality display device or an optical head-mounted display that is integrally provided or mounted on glasses 130 . In other embodiments, computing device 120 may be provided as part of another wearable or portable device of user 100 including a wristband, multifunction watch, buttons, clips, and the like. In other embodiments, computing device 120 may be provided as part of another system, such as an in-vehicle automotive computing or navigation system. Those skilled in the art will appreciate that different types of computing devices and arrangements of devices may implement the functionality of the disclosed embodiments. Thus, in other implementations, computing device 120 may include a personal computer (PC), a laptop computer, an Internet server, and the like.

Figure IB shows a user 100 wearing a device 110 physically connected to a necklace 140 in accordance with the disclosed embodiments. This configuration of device 110 may be suitable for users who do not wear glasses some or all of the time. In this embodiment, the user 100 can easily wear the device 110 and take it off.

FIG. 1C shows a user 100 wearing a device 110 physically connected to a belt 150 in accordance with the disclosed embodiments. This configuration of device 110 may be designed as a belt buckle. Alternatively, device 110 may include clips for attachment to various items of clothing, such as belt 150, or vests, pockets, collars, hats or hats, or other parts of clothing items.

Figure ID shows a user 100 wearing a device 110 physically connected to a wristband 160 in accordance with the disclosed embodiments. Although according to this embodiment, the aiming direction of device 110 may not match the field of view of user 100 , device 110 may include identifying the hand based on the tracked eye movements of user 100 indicating that user 100 is looking in the direction of wristband 160 Abilities related to triggers. Wristband 160 may also include an accelerometer, gyroscope, or other sensor for determining the movement or orientation of the user's 100 hand to identify hand-related triggers.

FIG. 2 is a schematic diagram of an exemplary system 200 that includes a wearable device 110 worn by a user 100 , and optionally a computing device 120 and/or a server capable of communicating with the device 110 via a network 240 , consistent with the disclosed embodiments 250. In some embodiments, device 110 may capture and analyze image data, identify hand-related triggers present in the image data, and perform actions and/or provide feedback to user 100 based at least in part on the identification of the hand-related triggers. In some embodiments, optional computing device 120 and/or server 250 may provide additional functionality to enhance user 100 interaction with his or her environment, as described in more detail below.

According to the disclosed embodiments, the apparatus 110 may include an image sensor system 220 for capturing real-time image data of the user's 100 field of view. In some embodiments, device 110 may also include functions for controlling and performing disclosed functions of device 110 (such as controlling the capture of image data, analyzing image data, and performing actions and/or based on hand-related triggers identified in the image data). or output feedback) of the processing unit 210. According to the disclosed embodiments, the hand-related trigger may include a gesture performed by the user 100 involving a portion of the user's 100 hand. Furthermore, consistent with some embodiments, hand-related triggers may include wrist-related triggers. Additionally, in some embodiments, the apparatus 110 may include a feedback output unit 230 for generating information output to the user 100 .

As discussed above, apparatus 110 may include image sensor 220 for capturing image data. The term "image sensor" refers to a device capable of detecting and converting light signals in the near-infrared, infrared, visible and ultraviolet spectrums into electrical signals. The electrical signals may be used to form images or video streams (ie, image data) based on the detected signals. The term "image data" includes any form of data retrieved from optical signals in the near infrared, infrared, visible and ultraviolet spectrums. Examples of image sensors may include semiconductor charge coupled devices (CCDs), active pixel sensors in complementary metal oxide semiconductors (CMOS), or N-type metal oxide semiconductors (NMOS, active MOS). In some cases, image sensor 220 may be part of a camera included in device 110 .

According to the disclosed embodiments, the apparatus 110 may also include a processor 210 for controlling the image sensor 220 to capture image data and for analyzing the image data. As discussed in further detail below with respect to FIG. 5A, the processor 210 may include a “processing device for performing logical operations on one or more inputs of image data and other data in accordance with stored or accessible software instructions that provide the desired functionality. ". In some embodiments, the processor 210 may also control the feedback output unit 230 to provide feedback to the user 100 including information based on the analyzed image data and stored software instructions. As the term is used herein, a "processing device" may access memory in which executable instructions are stored, or, in some embodiments, the "processing device" may itself include executable instructions (eg, memory stored in the processing device) middle).

In some embodiments, the information or feedback information provided to the user 100 may include time information. The time information may include any information related to the current time of day, and, as described further below, may be presented in any sensory perception manner. In some embodiments, the time information may include the current time of day (eg, 2:30 PM or 14:30 PM) in a preconfigured format. The time information may include the time in the user's current time zone (eg, based on the determined location of the user 100), as well as an indication of the time zone and/or time of day in another desired location. In some embodiments, the time information may include hours or minutes relative to one or more predetermined times of day. For example, in some embodiments, the time information may include an indication that three hours and fifteen minutes remain until a particular hour (eg, by 6:00 PM) or some other predetermined time. The time information may also include the duration elapsed since the start of a particular activity, such as the start of a meeting or the start of a jog or any other activity. In some embodiments, the activity may be determined based on the analyzed image data. In other embodiments, the time information may also include additional information related to the current time and one or more other routines, time periods, or scheduled events. For example, as discussed in further detail below, the time information may include an indication of the number of minutes remaining until the next scheduled event, which may be determined from a calendar function or other information retrieved from computing device 120 or server 250 .

Feedback output unit 230 may include one or more feedback systems for providing information output to user 100 . In the disclosed embodiments, audible or visual feedback may be provided via any type of connected audible or visual system or both. Information feedback in accordance with the disclosed embodiments may include audible feedback to the user 100 (eg, using Bluetooth ^™ or other wired or wireless connected speakers, or bone conduction headphones). Feedback output unit 230 of some embodiments may additionally or alternatively generate visual output of information to user 100 , eg, as projected onto the lenses of eyeglasses 130 or provided via a separate heads-up display in communication with device 110 . A portion of an augmented reality display, such as display 260 provided as part of computing device 120, which may include a vehicle head-up display, augmented reality device, virtual reality device, smartphone, PC, tablet, and the like.

等)以及近场电容耦合和其他短距离无线技术，或经由有线连接，通过网络240连接到计算设备120。在其中计算设备120是智能手机的实施例中，计算设备120可以具有安装在其中的专用应用程序。例如，用户100可以在显示器260上查看源自装置110或由装置110触发的数据(例如，图像、视频剪辑、提取的信息、反馈信息等)。另外，用户100可以选择数据的一部分以存储在服务器250中。The term "computing device" refers to a device that includes a processing unit and has computing capabilities. Some examples of computing devices 120 include PCs, laptops, tablets, or other computing systems such as an in-vehicle computing system of an automobile, for example, each computing system configured to communicate directly with device 110 or server 250 via network 240 . Another example of computing device 120 includes a smartphone with display 260 . In some embodiments, computing device 120 may be a computing system specifically configured for apparatus 110 , and may be provided integrally with or connected to apparatus 110 . Device 110 may also communicate via any known wireless standard (eg, WiFi, Bluetooth

etc.) and near-field capacitive coupling and other short-range wireless techniques, or via a wired connection, are connected to computing device 120 through network 240. In embodiments in which computing device 120 is a smartphone, computing device 120 may have dedicated applications installed therein. For example, user 100 may view data (eg, images, video clips, extracted information, feedback information, etc.) originating from or triggered by device 110 on display 260 . Additionally, the user 100 may select a portion of the data to store in the server 250 .

The network 240 may be a shared, public or private network, may include a wide area or a local area, and may be implemented by any suitable combination of wired and/or wireless communication networks. Network 240 may also include an intranet or the Internet. In some embodiments, network 240 may include a short-range or near-field wireless communication system for enabling communication between apparatus 110 and computing device 120 provided in close proximity to each other (eg, on or near a user's person). Device 110 may autonomously establish a connection to network 240, eg, using a wireless module (eg, Wi-Fi, cellular). In some embodiments, the device 110 may use a wireless module when connected to an external power source to extend battery life. Furthermore, the communication between the device 110 and the server 250 may be accomplished through any suitable communication channel, such as a telephone network, extranet, intranet, Internet, satellite communication, offline communication, wireless communication, repeater communication, local area network ( LAN), Wide Area Network (WAN), and Virtual Private Network (VPN).

As shown in FIG. 2 , the device 110 may transmit data to or receive data from the server 250 via the network 240 . In the disclosed embodiments, data received from server 250 and/or computing device 120 may include many different types of information based on the analyzed image data, including identities related to commercial products or people, identified landmarks, and Information about any other information stored in or accessed by server 250 . In some embodiments, data may be received and transmitted via computing device 120 . Server 250 and/or computing device 120 may retrieve information from various data sources (eg, user-specific databases or users' social network accounts or other accounts, the Internet, and other managed or accessible databases), and in accordance with the disclosed implementations For example, information related to the analyzed image data and the identified triggers is provided to the device 110 . In some embodiments, calendar-related information retrieved from different data sources may be analyzed to provide specific time information or time-based context for providing specific information based on the analyzed image data.

An example of a wearable device 110 integrated with eyewear 130 is shown in more detail in FIG. 3A (as discussed in connection with FIG. 1A ) in accordance with some embodiments. In some embodiments, the device 110 may be associated with a structure (not shown in FIG. 3A ) that enables the device 110 to be easily detached and reattached to the glasses 130 . In some embodiments, when the device 110 is attached to the glasses 130, the image sensor 220 acquires the set aiming direction without directional calibration. The set aiming direction of the image sensor 220 may substantially match the field of view of the user 100 . For example, a camera associated with image sensor 220 may be mounted slightly downward (eg, 5-15 degrees from the horizon) within device 110 at a predetermined angle. Therefore, the set aiming direction of the image sensor 220 may substantially match the field of view of the user 100 .

Figure 3B is an exploded view of components of the embodiment discussed with respect to Figure 3A. Attaching the device 110 to the glasses 130 can be done in the following manner. First, the bracket 310 may be mounted on the glasses 130 using the screws 320 on the side of the bracket 310 . The device 110 can then be clamped on the stand 310 so that it is aligned with the user's 100 field of view. The term "mount" includes any device or structure capable of detaching and reattaching a device, including a camera, to a pair of glasses or another object (eg, a helmet). The bracket 310 may be made of plastic (eg, polycarbonate), metal (eg, aluminum), or a combination of plastic and metal (eg, carbon fiber graphite). The bracket 310 may be mounted on any type of eyewear (eg, glasses, sunglasses, 3D glasses, safety glasses, etc.) using screws, bolts, snaps, or any fastening device used in the art.

In some embodiments, the bracket 310 may include a quick release mechanism for disengaging and re-engaging the device 110 . For example, bracket 310 and device 110 may include magnetic elements. As an alternative example, bracket 310 may include a male pin member, while device 110 may include a female jack plate. In other embodiments, the bracket 310 may be an integral part of a pair of eyeglasses, or sold separately and installed by an optometrist. For example, the bracket 310 may be configured to be mounted on the temple of the eyeglass 130 near the front of the frame but before the hinge. Alternatively, the bracket 310 may be configured to be mounted on the bridge of the eyeglasses 130 .

In some embodiments, device 110 may be provided as part of a spectacle frame 130 with or without lenses. Additionally, in some embodiments, the apparatus 110 may be configured to provide an augmented reality display (if provided) projected onto the lenses of the eyeglasses 130, or alternatively, such as in accordance with the disclosed embodiments, may include time for projection information display. Device 110 may include an additional display, or, alternatively, may communicate with a separately provided display system that may or may not be attached to glasses 130 .

In some embodiments, device 110 may be implemented in forms other than wearable glasses, eg, as described above with respect to FIGS. 1B-1D . FIG. 4A is a schematic diagram of an example of an additional embodiment of the device 110 from a front view of the device 110 . Device 110 includes image sensor 220, clips (not shown), function buttons (not shown), and suspension loop 410 for attaching device 110 to necklace 140 such as shown in Figure IB. When the device 110 is suspended from the necklace 140 , the aiming direction of the image sensor 220 may not exactly match the user's 100 field of view, but the aiming direction is still relative to the user's 100 field of view.

FIG. 4B is a schematic diagram of an example of a second embodiment of the device 110 from the side of the device 110 . In addition to the suspension ring 410, the device 110 may also include a clip 420 as shown in FIG. 4B. As shown in FIG. 1C , user 100 may use clip 420 to attach device 110 to shirt or waistband 150 . The clip 420 may provide an easy mechanism for detaching and re-engaging the device 110 from different garments. In other embodiments, the device 110 may include a female jack plate for connecting with the male pins of a car mount or universal mount.

In some embodiments, device 110 includes function buttons 430 for enabling user 100 to provide input to device 110 . Function buttons 430 may accept different types of tactile inputs (eg, tap, tap, double tap, long press, swipe from right to left, swipe from left to right). In some embodiments, each type of input may be associated with a different action. For example, a tap can be associated with the ability to take a picture, while a right-to-left swipe can be associated with the ability to record a video.

As shown in Figure 4C, the device 110 may be attached to clothing (eg, shirts, belts, pants, etc.) at the edges of the clothing of the user 100 using clips 431 . For example, the body of the device 100 may reside adjacent to the inner surface of the garment, with the clips 431 engaging the outer surface of the garment. In such an embodiment, as shown in Figure 4C, the image sensor 220 (eg, a camera for visible light) may protrude beyond the edge of the garment. Alternatively, the clip 431 may engage the inner surface of the garment with the body of the device 110 proximate the exterior of the garment. In various embodiments, a garment may be positioned between the clip 431 and the body of the device 110 .

An example embodiment of apparatus 110 is shown in Figure 4D. Device 110 includes clips 431 that may include points (eg, 432A and 432B) proximate front surface 434 of body 435 of device 110 . In an exemplary embodiment, the distance between the points 432A, 432B and the front surface 434 may be less than a typical thickness of the fabric of the user's 100 garment. For example, the distance between the points 432A, 432B and the surface 434 may be less than the thickness of the T-shirt, eg, less than 1 mm, less than 2 mm, less than 3 mm, etc., or in some cases the points 432A, 432B of the clip 431 may touch Surface 434. In various embodiments, clip 431 may include points 433 that do not contact surface 434, allowing garments to be inserted between clip 431 and surface 434.

Figure 4D schematically illustrates the different views of the device 110 defined as a front view (F view), rear view (R view), top view (T view), side view (S view) and bottom view (B view) . These views will be referred to when describing the device 110 in the subsequent figures. FIG. 4D shows an example embodiment in which the clip 431 is located on the same side of the device 110 (eg, the front side of the device 110 ) as the sensor 220 . Alternatively, the clip 431 may be located on the opposite side of the device 110 (eg, the rear side of the device 110 ) from the sensor 220 . In various embodiments, as shown in FIG. 4D , device 110 may include function buttons 430 .

Various views of device 110 are shown in Figures 4E-4K. For example, FIG. 4E shows a view of device 110 with electrical connections 441 . Electrical connection 441 may be, for example, a USB port, which may be used to transfer data to/from device 110 and to provide power to device 110 . In an example embodiment, connection 441 may be used to charge battery 442, shown schematically in Figure 4E. FIG. 4F shows a view F of device 110 including sensor 220 and one or more microphones 443 . In some embodiments, device 110 may include outwardly facing several microphones 443 , where microphones 443 are configured to pick up ambient sounds and sounds from various speakers in communication with user 100 . FIG. 4G shows an R view of device 110 . In some embodiments, as shown in FIG. 4G , the microphone 444 may be located on the rear side of the device 110 . Microphone 444 may be used to detect audio signals from user 100 . It should be noted that device 110 may have microphones placed on any side of device 110 (eg, front, rear, left, right, top, or bottom). In various embodiments, some microphones may be on a first side (eg, microphone 443 may be on the front of device 110), while other microphones may be on a second side (eg, microphone 444 may be on the rear side of device 110).

Figures 4H and 4I illustrate different sides of device 110 (ie, S-views of device 110) consistent with the disclosed embodiments. For example, FIG. 4H shows the location of sensor 220 and an example shape of clip 431 . FIG. 4J shows a T view of the device 110 including the function button 430 , and FIG. 4K shows a B view of the device 110 with electrical connections 441 .

The example embodiments discussed above with respect to Figures 3A, 3B, 4A and 4B are not limiting. In some embodiments, apparatus 110 may be implemented in any suitable configuration to perform the disclosed methods. For example, referring back to FIG. 2 , the disclosed embodiments may implement apparatus 110 according to any configuration including image sensor 220 and processor unit 210 for performing image analysis and for communicating with feedback unit 230 .

FIG. 5A is a block diagram illustrating components of apparatus 110 according to an example embodiment. As shown in FIG. 5A , and as similarly discussed above, device 110 includes image sensor 220 , memory 550 , processor 210 , feedback output unit 230 , wireless transceiver 530 , and power bank 520 . In other embodiments, apparatus 110 may also include buttons, other sensors such as microphones, and inertial measurement devices such as accelerometers, gyroscopes, magnetometers, temperature sensors, color sensors, light sensors, and the like. The apparatus 110 may also include a data port 570 and a power connection 510 with a suitable interface for connection with an external power source or external device (not shown).

The processor 210 shown in FIG. 5A may comprise any suitable processing device. The term "processing device" includes any physical device that has circuitry that performs logical operations on inputs. For example, a processing device may include one or more integrated circuits, microchips, microcontrollers, microprocessors, central processing units (CPUs), graphics processing units (GPUs), digital signal processors (DSPs), field programmable gates All or part of an array (FPGA), or other circuitry suitable for executing instructions or performing logical operations. Instructions executed by the processing device may, for example, be preloaded into memory integrated with or embedded in the processing device, or may be stored in a separate memory (eg, memory 550). Memory 550 may include random access memory (RAM), read only memory (ROM), hard disk, optical disk, magnetic media, flash memory, other persistent, fixed or volatile memory, or any other mechanism capable of storing instructions.

Although in the embodiment shown in FIG. 5A, apparatus 110 includes one processing device (eg, processor 210), apparatus 110 may include more than one processing device. Each processing device may have a similar structure, or the processing devices may have different structures that are electrically connected or disconnected from each other. For example, the processing device may be a separate circuit or integrated in a single circuit. When more than one processing device is used, the processing devices may be configured to operate independently or cooperatively. The processing devices may be coupled electrically, magnetically, optically, acoustically, mechanically, or by other means that allow them to interact.

In some embodiments, processor 210 may process multiple images captured from the environment of user 100 to determine different parameters related to capturing subsequent images. For example, processor 210 may determine values for at least one of the following: image resolution, compression ratio, cropping parameters, frame rate, focus, exposure time, aperture size, and light sensitivity based on information derived from captured image data. The determined value can be used to capture at least one subsequent image. Additionally, the processor 210 may detect images that include at least one hand-related trigger in the user's environment, and via the feedback output unit 230 perform actions and/or provide information output to the user.

In another embodiment, the processor 210 may change the aiming direction of the image sensor 220 . For example, when the device 110 is attached with the clip 420, the aiming direction of the image sensor 220 may not be aligned with the user's 100 field of view. The processor 210 may identify certain conditions from the analyzed image data and adjust the aiming direction of the image sensor 220 to capture the relevant image data. For example, in one embodiment, the processor 210 may detect an interaction with another subject and sense that the subject is not fully in the field of view because the image sensor 220 is tilted downward. In response to this, the processor 210 may adjust the aiming direction of the image sensor 220 to capture image data of the individual. Other scenarios are also contemplated in which processor 210 may identify the need to adjust the aiming direction of image sensor 220.

In some embodiments, processor 210 may communicate data to feedback output unit 230 , which may include any device configured to provide information to user 100 . Feedback output unit 230 may be provided as part of device 110 (as shown), or may be provided external to and communicatively coupled to device 110 . Feedback output unit 230 may be configured to output visual or non-visual feedback based on signals received from processor 210, such as when processor 210 identifies a hand-related trigger in the analyzed image data.

The term "feedback" refers to any output or information provided in response to at least one image in the processing environment. In some embodiments, as similarly described above, the feedback may include an audible or visible indication of time information, detected text or numbers, monetary value, branded products, the identity of a person, the identity of a landmark, or including at an intersection. other environmental circumstances or conditions, such as street names or the color of traffic lights, and other information associated with each of these. For example, in some embodiments, the feedback may include additional information about the amount of money still needed to complete the transaction, information about the person identified, historical information or the time and entry price of the landmark detected, and the like. In some embodiments, the feedback may include audible tones, haptic responses, and/or information previously recorded by the user 100 . Feedback output unit 230 may include appropriate components for outputting acoustic and haptic feedback. For example, the feedback output unit 230 may include audio headphones, hearing aid type devices, speakers, bone conduction headphones, interfaces that provide haptic cues, vibrotactile stimulators, and the like. In some embodiments, processor 210 may communicate signals with external feedback output unit 230 via wireless transceiver 530, a wired connection, or some other communication interface. In some embodiments, feedback output unit 230 may also include any suitable display device for visually displaying information to user 100 .

As shown in FIG. 5A , device 110 includes memory 550 . Memory 550 may include one or more sets of instructions accessible to processor 210 for performing the disclosed methods, including instructions for identifying hand-related triggers in image data. In some embodiments, memory 550 may store image data (eg, images, videos) captured from the environment of user 100 . In addition, memory 550 may store information specific to user 100, such as image representations of known individuals, favorite products, individual items, and calendar or appointment information, among others. In some embodiments, the processor 210 may determine, for example, which type of image data to store based on available storage space in the memory 550 . In another embodiment, the processor 210 may extract information from image data stored in the memory 550 .

As further shown in FIG. 5A , the device 110 includes a power bank 520 . The term "power bank" includes any device capable of providing power that can be easily carried by hand (eg, power bank 520 may weigh less than a pound). The mobility of the power source enables the user 100 to use the device 110 in a variety of situations. In some embodiments, the power bank 520 may include one or more batteries (eg, nickel-cadmium batteries, nickel-metal hydride batteries, and lithium-ion batteries) or any other type of power source. In other embodiments, the power bank 520 may be rechargeable and contained within the housing of the receiving device 110 . In other embodiments, the power bank 520 may include one or more energy harvesting devices (eg, portable solar units, human body vibration units, etc.) for converting ambient energy into electrical energy.

蓝牙智能、802.15.4或ZigBee)。在一些实施例中，无线收发器530可以将数据(例如，原始图像数据、经处理的图像数据、提取的信息)从装置110发送到计算设备120和/或服务器250。无线收发器530还可以从计算设备120和/或服务器250接收数据。在其他实施例中，无线收发器530可以将数据和指令发送到外部反馈输出单元230。Power bank 520 may power one or more wireless transceivers (eg, wireless transceiver 530 in Figure 5A). The term "wireless transceiver" refers to any device configured to exchange transmissions over an air interface using radio, infrared, magnetic, or electric fields. Wireless transceiver 530 may transmit and/or receive data using any known standard (eg, WiFi, Bluetooth

Bluetooth Smart, 802.15.4 or ZigBee). In some embodiments, wireless transceiver 530 may transmit data (eg, raw image data, processed image data, extracted information) from apparatus 110 to computing device 120 and/or server 250 . Wireless transceiver 530 may also receive data from computing device 120 and/or server 250 . In other embodiments, the wireless transceiver 530 may send data and instructions to the external feedback output unit 230 .

FIG. 5B is a block diagram illustrating components of apparatus 110 according to another example embodiment. In some embodiments, the device 110 includes a first image sensor 220a, a second image sensor 220b, a memory 550, a first processor 210a, a second processor 210b, a feedback output unit 230, a wireless transceiver 530, a power bank 520, and Power connector 510. In the arrangement shown in Figure 5B, each image sensor may provide images of different image resolutions, or face images in different directions. Alternatively, each image sensor may be associated with a different camera (eg, wide-angle camera, narrow-angle camera, IR camera, etc.). In some embodiments, apparatus 110 may select which image sensor to use based on various factors. For example, processor 210a may determine to capture subsequent images at a certain resolution based on available storage space in memory 550 .

The apparatus 110 may operate in a first processing mode and a second processing mode such that the first processing mode may consume less power than the second processing mode. For example, in the first processing mode, the device 110 may capture images and process the captured images to make real-time decisions, eg, based on recognizing hand-related triggers. In the second processing mode, apparatus 110 may extract information from images stored in memory 550 and delete images from memory 550 . In some embodiments, the power bank 520 can provide more than fifteen hours of processing in the first processing mode and approximately three hours of processing in the second processing mode. Thus, different processing modes may allow power bank 520 to generate enough power to power device 110 for different time periods (eg, over two hours, over four hours, over ten hours, etc.).

In some embodiments, the device 110 may use the first processor 210a in the first processing mode when powered by the mobile power supply 520 and may use the first processor 210a in the second processing mode when powered by the external power supply 580 connectable via the power connector 510 The second processor 210b is used in processing mode. In other embodiments, the apparatus 110 may determine which processors or which processing modes to use based on predefined conditions. The device 110 may operate in the second processing mode even when the device 110 is not powered by the external power source 580 . For example, if the available storage space in memory 550 for storing new image data is below a predefined threshold, device 110 may determine that device 110 should operate in the second processing mode when device 110 is not powered by external power source 580 .

)与服务器250或计算设备120通信。在一些实施例中，当可穿戴装置由包括在可穿戴装置中的移动电源供电时，装置110可以使用第一无线收发器，而当可穿戴装置由外部电源供电时，装置110可以使用第二无线收发器。Although one wireless transceiver is depicted in Figure 5B, apparatus 110 may include more than one wireless transceiver (eg, two wireless transceivers). In devices with more than one wireless transceiver, each wireless transceiver may transmit and/or receive data using a different standard. In some embodiments, the first wireless transceiver may communicate with server 250 or computing device 120 using a cellular standard (eg, LTE or GSM) and the second wireless transceiver may use a short-range standard (eg, WiFi or Bluetooth)

) communicates with server 250 or computing device 120 . In some embodiments, device 110 may use a first wireless transceiver when the wearable device is powered by a power bank included in the wearable device, and may use a second wireless transceiver when the wearable device is powered by an external power source wireless transceiver.

5C is a block diagram illustrating components of apparatus 110 including computing device 120, according to another example embodiment. In this embodiment, the device 110 includes an image sensor 220 , a memory 550 a , a first processor 210 , a feedback output unit 230 , a wireless transceiver 530 a , a mobile power source 520 and a power connector 510 . As further shown in FIG. 5C , computing device 120 includes processor 540 , feedback output unit 545 , memory 550b , wireless transceiver 530b , and display 260 . An example of computing device 120 is a smartphone or tablet with dedicated applications installed therein. In other embodiments, computing device 120 may include any configuration such as an in-vehicle automotive computing system, a PC, a laptop computer, and any other system consistent with the disclosed embodiments. In this example, user 100 may view feedback output in response to identification of hand-related triggers on display 260 . Additionally, user 100 may view other data (eg, images, video clips, object information, schedule information, extracted information, etc.) on display 260 . Additionally, user 100 may communicate with server 250 via computing device 120 .

In some embodiments, processor 210 and processor 540 are configured to extract information from captured image data. The term "extracting information" includes any process of identifying information associated with objects, individuals, locations, events, etc. in captured image data by any means known to those of ordinary skill in the art. In some embodiments, apparatus 110 may use the extracted information to send feedback or other real-time indications to feedback output unit 230 or to computing device 120 . In some embodiments, processor 210 may identify an individual standing in front of user 100 in the image data and send to computing device 120 the name of the individual and the last time user 100 encountered the individual. In another embodiment, the processor 210 may identify one or more visible triggers in the image data, including hand-related triggers, and determine whether the trigger is associated with a person other than the user of the wearable device to selectively determine Whether to perform the action associated with this trigger. One such action may be to provide feedback to user 100 via feedback output unit 230 provided as part of (or in communication with) apparatus 110 or via feedback unit 545 provided as part of computing device 120 . For example, the feedback output unit 545 may communicate with the display 260 to cause the display 260 to visibly output information. In some embodiments, the processor 210 may identify a hand-related trigger in the image data and send an indication of the trigger to the computing device 120 . The processor 540 may then process the received trigger information and provide output via the feedback output unit 545 or the display 260 based on the hand-related trigger. In other embodiments, processor 540 may determine hand-related triggers based on image data received from device 110 and provide appropriate feedback similar to those described above. In some embodiments, processor 540 may provide instructions or other information (such as contextual information) to device 110 based on the identified hand-related triggers.

In some embodiments, processor 210 may identify other contextual information in the analyzed image, such as an individual standing in front of user 100, and send information related to the analyzed information to computing device 120, such as the name of the individual and user 100 The last time the individual was encountered. In various embodiments, the processor 540 may extract statistical information from the captured image data and forward the statistical information to the server 250 . For example, certain information may be determined by processor 540 regarding the types of items the user purchases or how often the user visits a particular business. Based on this information, server 250 may send coupons and discounts associated with the user's preferences to computing device 120 .

When apparatus 110 is connected or wirelessly connected to computing device 120, apparatus 110 may transmit at least a portion of the image data stored in memory 550a for storage in memory 550b. In some embodiments, the processor 540 may delete the portion of the image data after the computing device 120 confirms that the portion of the image data was successfully transmitted. The term "deleted" means that the image is marked as "deleted" and other image data may be stored in its place, but does not necessarily mean that the image data is physically removed from memory.

Many variations and/or modifications of the disclosed embodiments are possible, as will be understood by those skilled in the art having the benefit of this disclosure. Not all components are necessary for the operation of device 110 . Any of the components may be located in any suitable apparatus and the components may be rearranged into various configurations while providing the functionality of the disclosed embodiments. For example, in some embodiments, apparatus 110 may include a camera, a processor, and a wireless transceiver for transmitting data to another device. Accordingly, the foregoing configurations are examples and apparatus 110 may capture, store, and/or process images regardless of the configurations discussed above.

Furthermore, the preceding and following descriptions relate to storing and/or processing images or image data. In embodiments disclosed herein, stored and/or processed images or image data may include representations of one or more images captured by image sensor 220 . As the term is used herein, a "representation" of an image (or image data) may include an entire image or a portion of an image. The representation of the image (or image data) may have the same resolution as the image (or image data) or a lower resolution, and/or the representation of the image (or image data) may be altered in some way (eg, compressed , have a lower resolution, have one or more colors changed, etc.).

For example, device 110 may capture an image and store a representation of the image compressed as a JPG file. As another example, device 110 may capture a color image, but store a black and white representation of the color image. As yet another example, device 110 may capture an image and store different representations of the image (eg, a portion of the image). For example, device 110 may store a portion of an image that includes the face of a person appearing in the image, but does not substantially include the environment surrounding the person. Similarly, device 110 may, for example, store a portion of an image that includes the product that appears in the image, but does not substantially include the environment surrounding the product. As yet another example, the apparatus 110 may store the representation of the image at a reduced resolution (ie, at a lower resolution than the captured image). Storing the representation of the image may allow device 110 to conserve storage space in memory 550 . Additionally, processing the representation of the image may allow device 110 to increase processing efficiency and/or help maintain battery life.

In addition to the above, in some embodiments, any of apparatus 110 or computing device 120 may further process the captured image data via processor 210 or processor 540 to provide for identifying objects in the captured image data and /or additional functions of gestures and/or other information. In some embodiments, actions may be taken based on recognized objects, gestures, or other information. In some embodiments, processor 210 or processor 540 may identify one or more visible triggers in the image data, including hand-related triggers, and determine whether the trigger is associated with a person other than the user, to determine whether to perform a Trigger the associated action.

Some embodiments of the present disclosure may include devices that may be secured to a user's clothing. Such a device may comprise two parts connectable by a connector. A capture unit may be designed to be worn on the outside of a user's clothing and may include an image sensor for capturing images of the user's environment. The capture unit may be connected or connectable to a power supply unit, which may be configured to house a power supply and processing equipment. The capture unit may be a small device including a camera or other device for capturing images. The capture unit may be designed to be unobtrusive and unobtrusive, and may be configured to communicate with a power supply unit hidden by the user's clothing. The power supply unit may include larger aspects of the system, such as transceiver antennas, at least one battery, processing devices, and the like. In some embodiments, the communication between the capture unit and the power supply unit may be provided through a data cable included in the connector, while in other embodiments, the communication between the capture unit and the power supply unit may be achieved wirelessly. Some embodiments may allow changing the orientation of the image sensor of the capture unit, eg, to better capture the image of interest.

6 illustrates an exemplary embodiment of a memory containing software modules consistent with the present disclosure. The memory 550 includes an orientation recognition module 601 , an orientation adjustment module 602 and a motion tracking module 603 . Modules 601, 602, 603 may contain software instructions for execution by at least one processing device (eg, processor 210) included in the wearable device. Orientation identification module 601 , orientation adjustment module 602 , and motion tracking module 603 may cooperate to provide orientation adjustment for a capture unit incorporated into wireless device 110 .

FIG. 7 shows an exemplary capture unit 710 including an orientation adjustment unit 705 . The orientation adjustment unit 705 may be configured to allow adjustment of the image sensor 220 . As shown in FIG. 7 , the orientation adjustment unit 705 may include an eyeball type adjustment mechanism. In alternative embodiments, the orientation adjustment unit 705 may include a gimbal, an adjustable rod, a pivotable mount, and any other suitable unit for adjusting the orientation of the image sensor 220 .

The image sensor 220 may be configured to move with the head of the user 100 in a manner such that the aiming direction of the image sensor 220 substantially coincides with the field of view of the user 100 . For example, as described above, a camera associated with the image sensor 220 may be mounted within the capture unit 710 at a predetermined angle in a slightly upward or downward facing position, depending on the intended position of the capture unit 710 . Therefore, the set aiming direction of the image sensor 220 may match the field of view of the user 100 . In some embodiments, processor 210 may use image data provided from image sensor 220 to change the orientation of image sensor 220 . For example, the processor 210 may recognize that the user is reading a book and determine that the aiming direction of the image sensor 220 is offset from the text. That is, the processor 210 may determine that the image sensor 220 is tilted in the wrong direction because the words at the beginning of each line of text are not completely within the field of view. In response to this, the processor 210 may adjust the aiming direction of the image sensor 220 .

The orientation recognition module 601 may be configured to recognize the orientation of the image sensor 220 of the capture unit 710 . For example, the orientation of image sensor 220 may be determined by analyzing images captured by image sensor 220 of capture unit 710 , by tilt or attitude sensing devices within capture unit 710 , and by measuring orientation adjustment unit 705 relative to the remainder of capture unit 710 to identify the relative orientation.

The orientation adjustment module 602 may be configured to adjust the orientation of the image sensor 220 of the capture unit 710 . As discussed above, the image sensor 220 may be mounted on the orientation adjustment unit 705 configured for movement. The orientation adjustment unit 705 may be configured to rotate and/or laterally move in response to commands from the orientation adjustment module 602 . In some embodiments, the orientation adjustment unit 705 may adjust the orientation of the image sensor 220 via a motor, an electromagnet, a permanent magnet, and/or any suitable combination thereof.

In some embodiments, monitoring module 603 may be provided for continuous monitoring. Such continuous monitoring may include tracking movement of at least a portion of an object included in one or more images captured by the image sensor. For example, in one embodiment, device 110 may track an object as long as the object remains substantially within the field of view of image sensor 220 . In additional embodiments, the monitoring module 603 may engage the orientation adjustment module 602 to instruct the orientation adjustment unit 705 to continuously orient the image sensor 220 toward the object of interest. For example, in one embodiment, monitoring module 603 may cause image sensor 220 to be oriented to ensure that a particular designated object, such as a particular person's face, remains within the field of view of image sensor 220 even as the designated object moves around. In another embodiment, the monitoring module 603 may continuously monitor the region of interest included in one or more images captured by the image sensor. For example, a user may be occupied with a particular task, such as typing on a laptop computer, while the image sensor 220 remains oriented in a particular direction and continuously monitors a portion of each image from a series of images to detect triggers or other events. For example, the image sensor 210 may be directed towards a piece of laboratory equipment, and the monitoring module 603 may be configured to monitor a status light on the laboratory equipment for status changes while the user's attention is engaged.

In some embodiments consistent with the present disclosure, capture unit 710 may include multiple image sensors 220 . The plurality of image sensors 220 may each be configured to capture different image data. For example, when multiple image sensors 220 are provided, the image sensors 220 may capture images with different resolutions, may capture wider or narrower fields of view, and may have different magnification levels. Image sensor 220 may be provided with different lenses to allow for these different configurations. In some embodiments, the plurality of image sensors 220 may include image sensors 220 having different orientations. Thus, each of the plurality of image sensors 220 may be pointed in different directions to capture different images. In some embodiments, the fields of view of image sensors 220 may overlap. The plurality of image sensors 220 may be configured for orientation adjustment, eg, by pairing with the image adjustment unit 705 . In some embodiments, the monitoring module 603 or another module associated with the memory 550 may be configured to individually adjust the orientation of the plurality of image sensors 220 and turn each of the plurality of image sensors 220 on or off as desired. In some embodiments, monitoring an object or person captured by the image sensor 220 may include tracking the movement of the object in the field of view of the plurality of image sensors 220 .

Embodiments consistent with the present disclosure may include a connector configured to connect a capture unit and a power supply unit of a wearable device. A capture unit consistent with the present disclosure may include at least one image sensor configured to capture an image of a user's environment. A power supply unit consistent with the present disclosure may be configured to house a power source and/or at least one processing device. Connectors consistent with the present disclosure may be configured to connect the capture unit and the power supply unit, and may be configured to secure the device to the garment such that the capture unit is above the outer surface of the garment and the power supply unit is below the inner surface of the garment. Exemplary embodiments of capture units, connectors, and power supply units consistent with the present disclosure are discussed in further detail with respect to FIGS. 8-14 .

18 is a schematic diagram of an embodiment of a wearable device 110 that may be secured to clothing in accordance with the present disclosure. As shown in FIG. 8 , the capturing unit 710 and the power supply unit 720 may be connected through the connector 730 such that the capturing unit 710 is located on one side of the clothing 750 and the power supply unit 720 is located on the opposite side of the clothing 750 . In some embodiments, the capture unit 710 may be positioned above the outer surface of the garment 750 and the power supply unit 720 may be positioned below the inner surface of the garment 750 . The power supply unit 720 may be configured to be placed against the user's skin.

The capture unit 710 may include the image sensor 220 and an orientation adjustment unit 705 (as shown in FIG. 7 ). The power supply unit 720 may include the mobile power supply 520 and the processor 210 . Power supply unit 720 may also include any combination of previously discussed elements that may be part of wearable device 110 , including but not limited to wireless transceiver 530 , feedback output unit 230 , memory 550 , and data port 570 .

The connector 730 may include a clip 715 or other mechanical connection designed to clip or attach the capture unit 710 and the power supply unit 720 to the garment 750 as shown in FIG. 8 . As shown, clips 715 may be attached to each of capture unit 710 and power supply unit 720 at their perimeter, and may wrap around the edges of garment 750 to secure capture unit 710 and power supply unit 720 in place. Connector 730 may also include power cable 760 and data cable 770 . The power supply cable 760 may be capable of delivering power from the power bank 520 to the image sensor 220 of the capture unit 710 . The power cable 760 may also be configured to provide power to any other element of the capture unit 710 (eg, toward the adjustment unit 705). Data cable 770 may be capable of transmitting captured image data from image sensor 220 in capture unit 710 to processor 800 in power supply unit 720 . The data cable 770 is also capable of conveying additional data between the capture unit 710 and the processor 800 , eg, for control instructions towards the adjustment unit 705 .

9 is a schematic diagram of a user 100 wearing a wearable device 110 consistent with embodiments of the present disclosure. As shown in FIG. 9 , the capturing unit 710 is located on the outer surface of the clothing 750 of the user 100 . The capture unit 710 is connected to the power supply unit 720 (not seen in this illustration) via a connector 730 that wraps around the edge of the garment 750 .

In some embodiments, the connector 730 may comprise a flexible printed circuit board (PCB). FIG. 10 shows an exemplary embodiment in which the connector 730 includes a flexible printed circuit board 765 . The flexible printed circuit board 765 may include data and power connections between the capture unit 710 and the power supply unit 720 . Thus, in some embodiments, the flexible printed circuit board 765 may be used in place of the power cable 760 and the data cable 770 . In alternate embodiments, a flexible printed circuit board 765 may be included in addition to at least one of the power cable 760 and the data cable 770 . In various embodiments discussed herein, the flexible printed circuit board 765 may replace the power cable 760 and the data cable 770 or include the flexible printed circuit board 765 in addition to the power cable 760 and the data cable 770 .

11 is a schematic diagram of another embodiment of a wearable device secureable to clothing consistent with the present disclosure. As shown in FIG. 11 , the connector 730 may be centrally located relative to the capture unit 710 and the power supply unit 720 . The central location of the connector 730 may facilitate securing the device 110 to the garment 750 through a hole in the garment 750, such as a button hole in an existing garment 750 or a special hole in the garment 750 designed to receive the wearable device 110.

Figure 12 is a schematic diagram of yet another embodiment of a wearable device 110 that can be secured to clothing. As shown in FIG. 12 , the connector 730 may include a first magnet 731 and a second magnet 732 . The first magnet 731 and the second magnet 732 may fix the capturing unit 710 to the power supply unit 720 , wherein the laundry is located between the first magnet 731 and the second magnet 732 . In the embodiment including the first magnet 731 and the second magnet 732, a power supply cable 760 and a data cable 770 may also be included. In these embodiments, power cable 760 and data cable 770 may be of any length and may provide flexible power and data connections between capture unit 710 and power supply unit 720 . Embodiments including the first magnet 731 and the second magnet 732 may also include flexible PCB 765 connections in addition to or in place of the power cable 760 and/or the data cable 770 . In some embodiments, the first magnet 731 or the second magnet 732 may be replaced by an object comprising a metallic material.

13 is a schematic diagram of yet another embodiment of a wearable device 110 that can be secured to clothing. FIG. 13 shows an embodiment in which power and data may be wirelessly transferred between capture unit 710 and power supply unit 720 . As shown in FIG. 13 , the first magnet 731 and the second magnet 732 may be provided as connectors 730 to fix the capturing unit 710 and the power supply unit 720 to the clothing 750 . Power and/or data may be captured via any suitable wireless technology (eg, magnetic coupling and/or capacitive coupling, near field communication technology, radio frequency transmission, and any other wireless technology suitable for transferring data and/or power over short distances). unit 710 and power supply unit 720.

FIG. 14 illustrates yet another embodiment of a wearable device 110 that may be secured to a user's clothing 750 . As shown in FIG. 14, the connector 730 may include features designed for contact mounting. For example, the capture unit 710 may include a ring 733 with a hollow center that is slightly larger in diameter than the disk-like protrusions 734 on the power supply unit 720 . When pressed together with the fabric of the garment 750 , the disc-shaped protrusions 734 may be tightly fitted within the ring 733 to secure the capturing unit 710 to the power supply unit 720 . FIG. 14 shows an embodiment that does not include any cables or other physical connections between the capture unit 710 and the power supply unit 720 . In this embodiment, the capture unit 710 and the power supply unit 720 may wirelessly transmit power and data. In alternative embodiments, capture unit 710 and power supply unit 720 may transmit power and data via at least one of cable 760 , data cable 770 , and flexible printed circuit board 765 .

FIG. 15 illustrates another aspect of a power supply unit 720 consistent with embodiments described herein. The power supply unit 720 may be configured to be positioned directly against the user's skin. To facilitate such placement, the power supply unit 720 may also include at least one surface coated with a biocompatible material 740 . Biocompatible material 740 may include materials that do not negatively react with the user's skin when held against the skin for extended periods of time. These materials may include, for example, silicone, PTFE, polyimide tape, polyimide, titanium, nitinol, platinum, and the like. As also shown in FIG. 15 , the power supply unit 720 may be sized such that the interior volume of the power supply unit is substantially filled by the power bank 520 . That is, in some embodiments, the internal volume of the power supply unit 720 may be such that the volume does not accommodate any additional components other than the power bank 520 . In some embodiments, the power bank 520 can take advantage of its proximity to the user's skin. For example, the power bank 520 may use the Peltier effect to generate power and/or charge the power source.

In other embodiments, the garment-secured device may also include a protection circuit associated with the power supply 520 housed in the power supply unit 720 . FIG. 16 shows an exemplary embodiment including a protection circuit 775 . As shown in FIG. 16 , the protection circuit 775 may be located remotely relative to the power supply unit 720 . In alternative embodiments, the protection circuit 775 may also be located in the capture unit 710 , on the flexible printed circuit board 765 , or in the power supply unit 720 .

Protection circuitry 775 may be configured to protect image sensor 220 and/or other elements of capture unit 710 from potentially dangerous currents and/or voltages generated by power bank 520 . The protection circuit 775 may include passive components (such as capacitors, resistors, diodes, inductors, etc.) to provide protection to the elements of the capture unit 710 . In some embodiments, the protection circuit 775 may also include active components, such as transistors, to provide protection to elements of the capture unit 710 . For example, in some embodiments, the protection circuit 775 may include one or more resistors that function as fuses. Each fuse may include melting (thus braking) when the current flowing through the fuse exceeds a predetermined limit (eg, 500 mA, 900 mA, 1 amp, 1.1 amp, 2 amp, 2.1 amp, 3 amp, etc.) The connection between the circuit of the image capture unit 710 and the circuit of the power supply unit 720) is a wire or metal strip. Any or all of the previously described embodiments may include protection circuitry 775 .

等)，或经由近场电容耦合、其他短程无线技术，或经由有线连接，在一个或多个网络上向计算设备(例如，智能手机、平板电脑、手表、计算机等)发送数据。类似地，可穿戴装置可以经由任何已知的无线标准(例如，蜂窝、WiFi、蓝牙

等)，或经由近场电容耦合、其他短程无线技术，或经由有线连接，在一个或多个网络上从计算设备接收数据。发送到可穿戴装置和/或由无线装置接收的数据可以包括图像、图像的部分、与出现在经分析的图像中的信息有关的或与经分析的音频相关联的标识符，或表示图像和/或音频数据的任何其他数据。例如，可以分析图像，并且可以将与在图像中发生的活动相关的标识符发送到计算设备(例如，“配对设备”)。在本文描述的实施例中，可穿戴装置可以本地(在可穿戴装置上)和/或远程(经由计算设备)处理图像和/或音频。此外，在本文描述的实施例中，可穿戴装置可以将与图像和/或音频的分析有关的数据发送到计算设备以进行进一步的分析、显示，和/或向另一设备(例如，配对设备)发送。此外，配对设备可以执行一个或多个应用程序(apps)以处理、显示和/或分析从可穿戴装置接收的数据(例如，标识符、文本、图像、音频等)。In some embodiments, the wearable device can communicate via any known wireless standard (eg, cellular, WiFi, Bluetooth

etc.), or via near-field capacitive coupling, other short-range wireless techniques, or via wired connections, to send data to computing devices (eg, smartphones, tablets, watches, computers, etc.) over one or more networks. Similarly, wearable devices can communicate via any known wireless standard (eg, cellular, WiFi, Bluetooth

etc.), or via near-field capacitive coupling, other short-range wireless techniques, or via wired connections, receive data from computing devices over one or more networks. Data sent to the wearable device and/or received by the wireless device may include images, portions of images, identifiers related to information present in the analyzed images or associated with the analyzed audio, or representations of images and / or any other data for audio data. For example, the image can be analyzed and an identifier associated with the activity occurring in the image can be sent to the computing device (eg, a "paired device"). In the embodiments described herein, the wearable device may process images and/or audio locally (on the wearable device) and/or remotely (via a computing device). Additionally, in the embodiments described herein, the wearable device may send data related to the analysis of the image and/or audio to the computing device for further analysis, display, and/or to another device (eg, a paired device) )send. Additionally, the paired device may execute one or more applications to process, display and/or analyze data (eg, identifiers, text, images, audio, etc.) received from the wearable device.

Some of the disclosed embodiments may relate to systems, apparatus, methods, and software products for determining at least one keyword. For example, the at least one keyword may be determined based on data collected by the device 110 . At least one search query can be determined based on at least one keyword. At least one search query can be sent to the search engine.

In some embodiments, at least one keyword may be determined based on at least one or more images captured by image sensor 220 . In some cases, at least one keyword may be selected from a pool of keywords stored in memory. In some cases, optical character recognition (OCR) may be performed on at least one image captured by image sensor 220, and at least one keyword may be determined based on the OCR results. In some cases, at least one image captured by image sensor 220 may be analyzed to identify: people, objects, locations, scenes, and the like. Furthermore, at least one keyword may be determined based on the identified person, object, location, scene, and the like. For example, the at least one keyword may include: a person's name, an object's name, a place name, a date, a sports team name, a movie title, a book title, and the like.

In some embodiments, the at least one keyword may be determined based on the user's behavior. The user's behavior may be determined based on analysis of one or more images captured by image sensor 220 . In some embodiments, the at least one keyword may be determined based on the activity of the user and/or others. The one or more images captured by image sensor 220 may be analyzed to identify the activities of the user and/or other persons present in the one or more images captured by image sensor 220 . In some embodiments, the at least one keyword may be determined based on at least one or more audio segments captured by device 110 . In some embodiments, the at least one keyword may be determined based on at least GPS information associated with the user. In some embodiments, the at least one keyword may be determined based on at least the current time and/or date.

In some embodiments, at least one search query may be determined based on the at least one keyword. In some cases, at least one search query may include at least one keyword. In some cases, at least one search query may include at least one keyword and additional keywords provided by the user. In some cases, at least one search query may include at least one keyword and one or more images, such as images captured by image sensor 220 . In some cases, at least one search query may include at least one keyword and one or more audio segments, such as audio segments captured by device 110 .

In some embodiments, at least one search query may be sent to a search engine. In some embodiments, search results provided by a search engine in response to at least one search query may be provided to a user. In some embodiments, at least one search query may be used to access the database.

For example, in one embodiment, the keywords may include the name of a food type (such as quinoa), or the brand name of the food; and the search will output information related to desired consumption, facts about nutritional profiles, etc. . In another example, in one embodiment, the keywords may include the name of the restaurant, and the search will output information related to the restaurant, such as menus, opening times, reviews, and the like. The name of the restaurant can be obtained using OCR on the signage image, using GPS information, etc. In another example, in one embodiment, a keyword may include a person's name, and the search will provide information from that person's social network profile. The person's name may be obtained using OCR on a name tag attached to the person's shirt, using a facial recognition algorithm, or the like. In another example, in one embodiment, the keyword may include the title of the book, and the search will output information related to the book, such as reviews, sales statistics, information about the author of the book, and the like. In another example, in one embodiment, the keywords may include the title of the movie, and the search will output information related to the movie, such as reviews, box office statistics, information about the movie's cast, show times, and the like. In another example, in one embodiment, the keyword may include the name of the sports team, and the search will output information related to the sports team, such as statistics, recent results, future schedules, information about the players of the sports team, etc. Wait. For example, the name of a sports team can be obtained using an audio recognition algorithm.

Camera-based directional hearing aids

As previously described, the disclosed embodiments may include providing feedback (such as acoustic and haptic feedback) to one or more auxiliary devices in response to processing at least one image in the environment. In some embodiments, the assistive device may be a headset or other device for providing auditory feedback to the user, such as a hearing aid. Conventional hearing aids often use microphones to amplify sounds in the user's environment. However, these conventional systems often fail to distinguish sounds that may be of particular importance to the wearer of the device, or may do so on a limited basis. Using the systems and methods of the disclosed embodiments, as described in detail below, provide various improvements over conventional hearing aids.

In one embodiment, a camera-based directional hearing aid may be provided for selectively amplifying sound based on the user's gaze direction. The hearing aid may communicate with an image capture device, such as apparatus 110, to determine the user's gaze direction. This gaze direction may be used to isolate and/or selectively amplify sound received from that direction (eg, sounds from individuals in the user's gaze direction, etc.). Sound received from directions other than the user's line of sight may be suppressed, attenuated, filtered, and the like.

17A is a schematic diagram of an example of a user 100 wearing an apparatus 110 for a camera-based auditory interface device 1710 in accordance with the disclosed embodiments. As shown, user 100 may wear device 110 that is physically connected to user 100's shirt or other clothing. Consistent with the disclosed embodiments, the device 110 may be positioned in other locations, as previously described. For example, device 110 may be physically attached to necklaces, belts, glasses, wristbands, buttons, and the like. Apparatus 110 may be configured to communicate with an auditory interface device, such as auditory interface device 1710 . Such communication may be via a wired connection, or may be performed wirelessly (eg, using Bluetooth ^™ , NFC or wireless forms of communication). In some embodiments, one or more additional devices such as computing device 120 may also be included. Accordingly, one or more of the processes or functions described herein with respect to apparatus 110 or processor 210 may be performed by computing device 120 and/or processor 540 .

Auditory interface device 1710 may be any device configured to provide auditory feedback to user 100 . The auditory interface device 1710 may correspond to the feedback output unit 230 as described above, and thus any description of the feedback output unit 230 may also apply to the auditory interface device 1710 . In some embodiments, auditory interface device 1710 may be separate from feedback output unit 230 and may be configured to receive signals from feedback output unit 230 . As shown in Figure 17A, auditory interface device 1710 may be placed in one or both ears of user 100, similar to conventional auditory interface devices. The auditory interface device 1710 may be of various styles, including in-canal, fully in-canal, in-ear, behind-the-ear, supra-ear, in-canal receiver, open mount, or various other styles. Auditory interface device 1710 may include one or more speakers for providing auditory feedback to user 100, a microphone for detecting sounds in the environment of user 100, internal electronics, a processor, memory, and the like. In some embodiments, auditory interface device 1710 may include, in addition to or in place of a microphone, one or more communication units, particularly one or more receivers, for receiving and transmitting signals from device 110 to user 100 .

The auditory interface device 1710 may have various other configurations or placements. In some embodiments, the auditory interface device 1710 may include a bone conduction headset 1711, as shown in FIG. 17A. Bone conduction earphones 1711 may be surgically implanted and may provide audible feedback to user 100 through bone conduction of sound vibrations to the inner ear. The auditory interface device 1710 may also include one or more earphones (eg, wireless earphones, over-ear earphones, etc.) or portable speakers carried or worn by the user 100 . In some embodiments, the auditory interface device 1710 may be integrated into other devices, such as the user's Bluetooth ^™ headset, glasses, helmets (eg, motorcycle helmets, bicycle helmets, etc.), hats, and the like.

The apparatus 110 may be configured to determine the user's gaze direction 1750 of the user 100 . In some embodiments, the user's gaze direction 1750 may be tracked by monitoring the orientation of the user's 100 jaw, or another body part or face part, relative to the optical axis of the camera sensor 1751 . Device 110 may be configured to capture one or more images of the user's surroundings, eg, using image sensor 220 . The captured image may include a representation of the user's 100 jaw, which may be used to determine the user's gaze direction 1750 . Processor 210 (and/or processors 210a and 210b) may be configured to use various image detection or processing algorithms (eg, using convolutional neural networks (CNN), scale-invariant feature transforms (SIFT), histograms of oriented gradients) (HOG feature or other techniques) to analyze the captured image and detect the jaw or another part of the user 100 . Based on the detected representation of the jaw of the user 100, the gaze direction 1750 can be determined. The gaze direction 1750 may be determined in part by comparing the detected representation of the jaw of the user 100 to the optical axis of the camera sensor 1751 . For example, the optical axis 1751 may be known or fixed in each image, and the processor 210 may determine the gaze direction 1750 by comparing a representative angle of the chin of the user 100 to the direction of the optical axis 1751 . While the process is described using a representation of the user's 100 jaw, various other features may be detected to determine the user's gaze direction 1750, including the user's face, nose, eyes, hands, and the like.

In other embodiments, the user's line of sight direction 1750 may be more closely aligned with the optical axis 1751 . For example, as described above, device 110 may be affixed to a pair of glasses of user 100, as shown in FIG. 1A. In this embodiment, the user's line of sight direction 1750 may be the same as or close to the direction of the optical axis 1751 . Therefore, the user's gaze direction 1750 may be determined or roughly estimated based on the field of view of the image sensor 220 .

17B is a schematic diagram of an embodiment of a device that is secureable to a garment in accordance with the present disclosure. As shown in FIG. 17A, device 110 may be secured to a piece of clothing, such as user 110's shirt. As mentioned above, the device 110 may be secured to other clothing items, such as the user's 100 belt or pants. Device 110 may have one or more cameras 1730 , which may correspond to image sensor 220 . The camera 1730 may be configured to capture images of the user's 100 surroundings. In some embodiments, camera 1730 may be configured to detect a representation of the user's jaw in the same image that captures the user's surroundings, which image may be used for other functions described in this disclosure. In other embodiments, the camera 1730 may be a secondary or separate camera dedicated to determining the user's gaze direction 1750.

The device 110 may also include one or more microphones 1720 for capturing sound from the user's 100 environment. Microphone 1720 may also be configured to determine the directionality of sound in the environment of user 100 . For example, microphone 1720 may include one or more directional microphones, which may be more sensitive to picking up sounds in certain directions. For example, the microphone 1720 may comprise a unidirectional microphone designed to pick up sound from a single direction or a small range of directions. Microphone 1720 may also include a cardioid microphone, which may be sensitive to sound from the front and sides. Microphone 1720 may also include a microphone array, which may include additional microphones, such as microphone 1721 on the front of device 110 , or microphone 1722 placed on the side of device 110 . In some embodiments, the microphone 1720 may be a multi-port microphone for capturing multiple audio signals. The microphones shown in Figure 17B are by way of example only, and any suitable number, configuration, or location of microphones may be used. The processor 210 may be configured to differentiate sounds within the environment of the user 100 and determine the approximate directionality of each sound. For example, using the microphone array 1720 , the processor 210 may compare the relative timing or amplitude of individual sounds between the microphones 1720 to determine the directionality relative to the device 100 .

As a preliminary step before other audio analysis operations, any audio classification technique can be used to classify sounds captured from the user's environment. For example, sounds may be classified into segments containing music, tones, laughter, screams, and the like. Indications of individual segments can be recorded in a database and can prove very useful for life-logging applications. As one example, the recorded information may enable the system to retrieve and/or determine the user's mood when encountering another entity. Additionally, such processing is relatively fast and efficient, and does not require significant computing resources, and does not require significant bandwidth to send the information to the destination. Also, once some parts of the audio are classified as non-speech, more computing resources are available to process other segments.

Based on the determined user's gaze direction 1750, the processor 210 may selectively adjust or amplify sound from the area associated with the user's gaze direction 1750. 18 is a schematic diagram illustrating an exemplary environment for using a camera-based hearing aid consistent with the present disclosure. Microphone 1720 may detect one or more sounds 1820 , 1821 and 1822 within the environment of user 100 . Based on the user's gaze direction 1750 determined by the processor 210, a region 1830 associated with the user's gaze direction 1750 can be determined. As shown in FIG. 18, the area 1830 may be defined by a cone or range of directions based on the user's gaze direction 1750. As shown in FIG. 18, the angular range can be defined by the angle θ. The angle θ may be any suitable angle (eg, 10 degrees, 20 degrees, 45 degrees) used to define the range in which the sound within the environment of the user 100 is adjusted.

The processor 210 may be configured to selectively adjust the sound in the environment of the user 100 based on the zone 1830 . The conditioned audio signal can be sent to the auditory interface device 1710 and thus can provide the user 100 with auditory feedback corresponding to the user's gaze direction. For example, processor 210 may determine that sound 1820 (which may correspond to the speech of individual 1810 , or, for example, to noise) is within region 1830 . The processor 210 may then perform various conditioning techniques on the audio signal received from the microphone 1720 . Adjusting may include amplifying the audio signal determined to correspond to sound 1820 relative to other audio signals. Amplification may be accomplished digitally, for example, by processing the audio signal associated with 1820 relative to other signals. Amplification may also be accomplished by changing one or more parameters of the microphone 1720 to focus audio sounds emanating from the area 1830 associated with the user's gaze direction 1750 (eg, a region of interest). For example, microphone 1720 may be a directional microphone and processor 210 may perform operations to focus microphone 1720 on sound 1820 or other sounds within area 1830. Various other techniques for amplifying sound 1820 may be used, such as the use of beamforming microphone arrays, acoustic telescope techniques, and the like.

Conditioning may also include attenuating or suppressing one or more audio signals received from directions outside of zone 1830 . For example, processor 1820 may attenuate sounds 1821 and 1822. Similar to the amplification of sound 1820, attenuation of sound may occur by processing the audio signal, or by changing one or more parameters associated with one or more microphones 1720 to direct focus away from sound emanating from outside area 1830.

In some embodiments, adjusting may also include changing the pitch of the audio signal corresponding to sound 1820 to make sound 1820 more perceptible to user 100 . For example, user 100 may have less sensitivity to tones within a certain range, and adjustment of the audio signal may adjust the pitch of sound 1820 to make it more perceptible to user 100 . For example, user 100 may experience hearing loss in frequencies above 10 kHz. Therefore, the processor 210 can remap higher frequencies (eg, at 15khz) to 10khz. In some embodiments, the processor 210 may be configured to vary the speech rate associated with one or more audio signals. Accordingly, processor 210 may be configured to detect speech within one or more audio signals received by microphone 1720, eg, using a voice activity detection (VAD) algorithm or technique. If it is determined that the sound 1820 corresponds to, for example, speech or speech from the individual 1810, the processor 220 may be configured to change the playback rate of the sound 1820. For example, the speaking rate of the individual 1810 may be reduced to make the detected speech more perceptible to the user 100 . Various other processing may be performed, such as modifying the pitch of the sound 1820, to maintain the same pitch as the original audio signal, or to reduce noise within the audio signal. If speech recognition has been performed on the audio signal associated with sound 1820, adjusting may also include modifying the audio signal based on the detected speech. For example, processor 210 may introduce pauses or increase the duration of pauses between words and/or sentences, which may make speech easier to understand.

The conditioned audio signal may then be sent to auditory interface device 1710 and an audio signal generated for user 100 . Thus, in the conditioned audio signal, sound 1820 may be more easily heard by user 100, louder and/or more distinguishable than sounds 1821 and 1822, which may represent background noise within the environment.

FIG. 19 is a flowchart illustrating an exemplary process 1900 for selectively amplifying sound emanating from a detected line of sight of a user, consistent with the disclosed embodiments. Process 1900 may be performed by one or more processors associated with apparatus 110, such as processor 210. In some embodiments, some or all of process 1900 may be performed on a processor external to device 110 . In other words, the processor performing process 1900 may be included in a common housing with microphone 1720 and camera 1730, or may be included in a second housing. For example, one or more portions of process 1900 may be performed by a processor in auditory interface device 1710 or an auxiliary device such as computing device 120 .

In step 1910, the process 1900 may include receiving a plurality of images captured by the camera from the user's environment. The camera may be a wearable camera such as camera 1730 of device 110 . In step 1912, the process 1900 can include receiving an audio signal representing sound received by the at least one microphone. The microphone can be configured to capture sound from the user's environment. For example, the microphone may be microphone 1720, as described above. Thus, the microphones may include directional microphones, microphone arrays, multi-port microphones, or various other types of microphones. In some embodiments, the microphone and wearable camera may be included in a common housing, such as that of device 110 . One or more processors that perform process 1900 may also be included in this enclosure, or may be included in a second enclosure. In such an embodiment, the processor may be configured to receive image and/or audio signals from the common housing via a wireless link (eg, Bluetooth ^™ , NFC, etc.). Thus, the common housing (eg, apparatus 110) and the second housing (eg, computing device 120) may also include a transmitter or various other communication components.

In step 1914, the process 1900 can include determining the user's gaze direction based on the analysis of at least one of the plurality of images. As mentioned above, various techniques can be used to determine the user's gaze direction. In some embodiments, the gaze direction may be determined based at least in part on a detected representation of the user's jaw in one or more images. As mentioned above, the images can be processed to determine the direction in which the jaw is pointing relative to the optical axis of the wearable camera.

In step 1916, the process 1900 can include selectively adjusting at least one audio signal received by the at least one microphone from a region associated with the user's line-of-sight direction. As described above, the area may be determined based on the user's gaze direction determined in step 1914 . The range may be associated with an angular width (eg, 10 degrees, 20 degrees, 45 degrees, etc.) with respect to the line-of-sight direction. As mentioned above, various forms of conditioning can be performed on the audio signal. In some embodiments, adjusting may include changing the pitch or playback speed of the audio signal. For example, adjusting may include changing the rate of speech associated with the audio signal. In some embodiments, adjusting may include amplifying the audio signal relative to other audio signals received from outside the area associated with the user's line of sight. Amplification may be performed by various means, such as operating a directional microphone configured to focus on audio sounds emanating from the area, or changing one or more parameters associated with the microphone to focus the microphone on audio emanating from the area sound. Amplifying may include attenuating or suppressing one or more audio signals received by the microphone from directions outside the area associated with the direction of the user's 110 line of sight.

In step 1918, the process 1900 can include transmitting the at least one conditioned audio signal to an auditory interface device configured to provide sound to a user's ear. For example, the conditioned audio signal may be sent to auditory interface device 1710, which may provide user 100 with sounds corresponding to the audio signal. The processor performing process 1900 may also be configured to cause one or more audio signals representing background noise, which may be attenuated relative to the at least one conditioned audio signal, to be transmitted to the auditory interface device. For example, processor 220 may be configured to transmit audio signals corresponding to sounds 1820 , 1821 and 1822 . However, based on the determination that sound 1820 is within region 1830, the signal associated with sound 1820 may be modified (eg, amplified) differently from sound 1821 and sound 1822. In some embodiments, auditory interface device 1710 may include a speaker associated with the earpiece. For example, an auditory interface device may be inserted at least partially into a user's ear for providing audio to the user. The auditory interface device may also be external to the ear, such as a behind-the-ear hearing device, one or more earphones, small portable speakers, and the like. In some embodiments, the auditory interface device may include a bone conduction microphone configured to provide audio signals to the user through vibrations of the user's skull. Such devices may be placed in external contact with the user's skin, or may be surgically implanted and attached to the user's bone.

Hearing aids that use speech and/or image recognition

Consistent with the disclosed embodiments, the hearing aid may selectively amplify the audio signal associated with the recognized speech of the individual. The hearing aid system may store the recognized speech features and/or facial features of the person to aid in recognition and selective amplification. For example, when an individual enters the field of view of device 110, the individual may be identified as an individual who has been introduced to the device, or who may have interacted with user 100 in the past (eg, friends, colleagues, relatives, previous acquaintances) Wait). Accordingly, the audio signal associated with the recognized individual's speech may be isolated and/or selectively amplified relative to other sounds in the user's environment. Audio signals associated with sound received from directions other than the individual's direction may be suppressed, attenuated, filtered, and the like.

The user 100 may wear a hearing aid device similar to the camera-based hearing aid device described above. For example, the hearing aid device may be an auditory interface device 1720 as shown in Figure 17A. Auditory interface device 1710 may be any device configured to provide auditory feedback to user 100 . The auditory interface device 1710 may be placed in one or both ears of the user 100, similar to conventional auditory interface devices. As mentioned above, the auditory interface device 1710 may be of various styles, including in-canal, completely in-canal, in-ear, behind-the-ear, supra-aural, in-canal receiver, open mount, or various other styles. Auditory interface device 1710 may include one or more speakers for providing auditory feedback to user 100, a communication unit for receiving signals from another system (such as device 110), a microphone for detecting sounds in the environment of user 100 , internal electronics, processors, memory, etc. The auditory interface device 1710 may correspond to the feedback output unit 230 or may be separate from the feedback output unit 230 and may be configured to receive signals from the feedback output unit 230 .

In some embodiments, the auditory interface device 1710 may include a bone conduction headset 1711, as shown in FIG. 17A. Bone conduction earphones 1711 may be surgically implanted and may provide audible feedback to user 100 through bone conduction of sound vibrations to the inner ear. The auditory interface device 1710 may also include one or more earphones (eg, wireless earphones, over-ear earphones, etc.) or portable speakers carried or worn by the user 100 . In some embodiments, the auditory interface device 1710 may be integrated into other devices, such as the user's Bluetooth ^™ headset, glasses, helmets (eg, motorcycle helmets, bicycle helmets, etc.), hats, and the like.

Auditory interface device 1710 may be configured to communicate with a camera device such as apparatus 110 . Such communication may be via a wired connection, or may be performed wirelessly (eg, using Bluetooth ^™ , NFC or wireless forms of communication). As described above, device 110 may be worn by user 100 in various configurations, including physically attached to a shirt, necklace, belt, eyeglasses, wristband, button, or other item associated with user 100 . In some embodiments, one or more additional devices such as computing device 120 may also be included. Accordingly, one or more of the processes or functions described herein with respect to apparatus 110 or processor 210 may be performed by computing device 120 and/or processor 540 .

As mentioned above, the apparatus 110 may include at least one microphone and at least one image capture device. Device 110 may include microphone 1720 as described with respect to FIG. 17B . Microphone 1720 may be configured to determine the directionality of sound in the environment of user 100 . For example, microphone 1720 may include one or more directional microphones, microphone arrays, multi-port microphones, and the like. The microphones shown in Figure 17B are by way of example only, and any suitable number, configuration, or location of microphones may be used. The processor 210 may be configured to differentiate sounds within the environment of the user 100 and determine the approximate directionality of each sound. For example, using the microphone array 1720 , the processor 210 may compare the relative timing or amplitude of individual sounds between the microphones 1720 to determine the directionality relative to the device 100 . Device 110 may include one or more cameras, such as camera 1730 , which may correspond to image sensor 220 . The camera 1730 may be configured to capture images of the user's 100 surroundings.

Apparatus 110 may be configured to identify individuals in the environment of user 100 . 20A is a schematic diagram illustrating an exemplary environment for using a hearing aid with speech and/or image recognition consistent with the present disclosure. Apparatus 110 may be configured to recognize faces 2011 or voices 2012 associated with individuals 2010 within the environment of user 100 . For example, device 110 may be configured to capture one or more images of user 100's surroundings using camera 1730 . The captured image may include a representation of an identified individual 2010 , which may be a friend, colleague, relative, or previous acquaintance of the user 100 . Processor 210 (and/or processors 210a and 210b ) may be configured to analyze captured images and detect identified users using various facial recognition techniques, as represented by element 2011 . Accordingly, device 110, or memory 550 in particular, may include one or more facial or voice recognition components.

FIG. 20B illustrates an exemplary embodiment of an apparatus 110 including a face and voice recognition component consistent with the present disclosure. Apparatus 110 is shown in simplified form in Figure 20B, and apparatus 110 may contain additional elements or may have alternative configurations, eg, as shown in Figures 5A-5C. Memory 550 (or 550a or 550b) may include facial recognition component 2040 and speech recognition component 2041. These components may be in place of or in addition to the orientation recognition module 601 , orientation adjustment module 602 and motion tracking module 603 as shown in FIG. 6 . Components 2040 and 2041 may contain software instructions for execution by at least one processing device (eg, processor 210) included in the wearable device. Components 2040 and 2041 are shown within memory 550 by way of example only, and may be located elsewhere within the system. For example, components 2040 and 2041 may reside in auditory interface device 1710, in computing device 120, on a remote server, or in another associated device.

Facial recognition component 2040 can be configured to recognize one or more faces within the environment of user 100 . For example, facial recognition component 2040 can recognize facial features on individual 2010's face 2011, such as eyes, nose, cheekbones, chin, or other features. The facial recognition component 2040 can then analyze the relative size and location of these features to identify the user. The facial recognition component 2040 can utilize one or more algorithms to analyze the detected features, such as principal component analysis (eg, using eigenfaces), linear discriminant analysis, elastic bundle map matching (eg, using Fisher faces), local binary Pattern Histogram (LBPH), Scale Invariant Feature Transform (SIFT), Accelerated Robust Feature (SURF), etc. Other facial recognition techniques such as three-dimensional recognition, skin texture analysis, and/or thermal imaging may also be used to identify individuals. Features other than facial features may also be used for identification, such as height, body size, or other distinguishing features of the individual 2010 .

Facial recognition component 2040 can access a database or data associated with user 100 to determine whether detected facial features correspond to an identified individual. For example, processor 210 may access database 2050 that contains information about individuals known to user 100 and data representing associated facial or other identifying features. Such data may include one or more images of an individual, or data representing a user's face that may be used for identification by facial recognition. Database 2050 may be any device capable of storing information about one or more individuals, and may include hard drives, solid state drives, network storage platforms, remote servers, and the like. Database 2050 may be located within device 110 (eg, within memory 550) or external to device 110, as shown in Figure 20B. In some embodiments, the database 2050 may be associated with a social networking platform, such as Facebook ^™ , LinkedIn ^™ , Instagram ^™ , and the like. The facial recognition component 2040 can also access a contact list of the user 100, such as a contact list on the user's phone, a web-based contact list (eg, through Outlook ^™ , Skype ^™ , Google ^™ , Salesforce ^™ , etc.) or with an auditory interface A private contact list associated with device 1710. In some embodiments, database 2050 may be compiled by device 110 from previous facial recognition analysis. For example, processor 210 may be configured to store data associated with one or more faces identified in images captured by device 110 in database 2050 . Each time a face is detected in an image, the detected facial features or other data may be compared to previously identified faces in database 2050. Facial recognition component 2040 can determine whether the individual is an identified individual of user 100, whether the individual has been previously identified by the system in instances exceeding a certain threshold, whether the individual has been explicitly introduced to device 110, and the like.

In some embodiments, user 100 may access database 2050, such as through a web interface, an application on a mobile device, or through apparatus 110 or an associated device. For example, user 100 may be able to select which contacts are recognized by device 110, and/or manually delete or add certain contacts. In some embodiments, a user or administrator may be able to train facial recognition component 2040. For example, user 100 may have the option of confirming or rejecting identifications made by facial recognition component 2040, which may improve the accuracy of the system. This training may occur in real-time as the individual 2010 is being identified, or at some point in the future.

Other data or information can also inform the facial recognition process. In some embodiments, the processor 210 may recognize the speech of the individual 2010 using various techniques, as described in further detail below. The recognized speech patterns and detected facial features may be used alone or in combination to determine that the individual 2010 is recognized by the device 110 . The processor 210 can also determine the user's gaze direction 1750 as described above, which can be used to verify the identity of the individual 2010. For example, if user 100 is looking in the direction of individual 2010 (especially for an extended period of time), this may indicate that individual 2010 is recognized by user 100, which may be used to increase confidence in facial recognition component 2040 or other identification means.

The processor 210 may also be configured to determine whether the individual 2010 is recognized by the user 100 based on one or more detected audio characteristics of sounds associated with the individual 2010's speech. Returning to FIG. 20A , processor 210 may determine that sound 2020 corresponds to speech 2012 of user 2010 . The processor 210 may analyze the audio signal representing the sound 2020 captured by the microphone 1720 to determine whether the individual 2010 is recognized by the user 100 . This can be performed using speech recognition component 2041 (FIG. 20B), and can include one or more speech recognition algorithms, such as hidden Markov models, dynamic time warping, neural networks, or other techniques. The speech recognition component and/or processor 210 may access a database 2050, which may also include the voiceprints of one or more individuals. Speech recognition component 2041 can analyze the audio signal representing sound 2020 to determine whether speech 2012 matches the voiceprint of an individual in database 2050. Thus, database 2050 may contain voiceprint data associated with multiple individuals, similar to the stored facial recognition data described above. After a match is determined, individual 2010 may be determined as an identified individual of user 100 . This process can be used alone or in conjunction with the facial recognition technology described above. For example, the individual 2010 can be identified using the facial recognition component 2040 and the individual 2010 can be authenticated using the speech recognition component 2041, and vice versa.

In some embodiments, device 110 may detect the speech of an individual who is not within device 110's field of view. For example, speech can be heard on a speakerphone, from the back seat, or the like. In such an embodiment, in the absence of a speaker in the field of view, the identification of the individual may be based solely on the individual's speech. The processor 110 may analyze the individual's speech as described above, eg, by determining whether the detected sound matches the individual's voiceprint in the database 2050.

After determining that individual 2010 is an identified individual of user 100, processor 210 may selectively adjust audio associated with the identified individual. The adjusted audio signal may be sent to auditory interface device 1710, and thus adjusted audio based on the identified individual may be provided to user 100. For example, adjusting may include amplifying an audio signal determined to correspond to sound 2020 (which may correspond to speech 2012 of individual 2010 ) relative to other audio signals. In some embodiments, the amplification may be accomplished digitally, for example, by processing the audio signal associated with the sound 2020 relative to other signals. Additionally or alternatively, amplification may be achieved by changing one or more parameters of the microphone 1720 to focus on audio sounds associated with the individual 2010 . For example, the microphone 1720 may be a directional microphone, and the processor 210 may perform operations to focus the microphone 1720 on the sound 2020. Various other techniques for amplifying sound 2020 may be used, such as the use of beamforming microphone arrays, acoustic telescope techniques, and the like.

In some embodiments, the selective adjustment may include attenuating or suppressing one or more audio signals received from directions not associated with the individual 2010 . For example, processor 210 may attenuate sound 2021 and/or 2022. Similar to the amplification of sound 2020, attenuation of sound may occur by processing the audio signal or by changing one or more parameters associated with microphone 1720 to direct focus away from sounds not associated with individual 2010.

The selective adjustment may also include determining whether the individual 2010 is speaking. For example, the processor 210 may be configured to analyze an image or video containing a representation of the individual 2010 to determine when the individual 2010 is speaking, eg, based on the detected movement of the individual's lips identified. This can also be determined by analyzing the audio signal received by the microphone 1720 , for example by detecting the speech 2012 of the individual 2010 . In some embodiments, selective adjustment may occur (initiated and/or terminated) dynamically based on whether the identified individual is speaking or not.

In some embodiments, adjusting may also include changing the pitch of one or more audio signals corresponding to sound 2020 to make the sound more perceptible to user 100 . For example, the user 100 may have less sensitivity to tones within a certain range, and the adjustment of the audio signal may adjust the pitch of the sound 2020. In some embodiments, the processor 210 may be configured to vary the speech rate associated with one or more audio signals. For example, sound 2020 may be determined to correspond to speech 2012 of individual 2010 . The processor 210 may be configured to alter the speech rate of the individual 2010 to make the detected speech more perceptible to the user 100 . Various other processing may be performed, such as modifying the pitch of the sound 2020, to maintain the same pitch as the original audio signal, or to reduce noise within the audio signal.

In some embodiments, the processor 210 may determine the area 2030 associated with the individual 2010 . Area 2030 may be associated with the orientation of individual 2010 relative to device 110 or user 100 . The orientation of the individual 2010 may be determined using the camera 1730 and/or the microphone 1720 using the methods described above. As shown in FIG. 20A, a region 2030 may be defined by a cone or range of directions based on the determined direction of the individual 2010. As shown in FIG. 20A, the angular range can be defined by the angle θ. The angle θ may be any suitable angle (eg, 10 degrees, 20 degrees, 45 degrees) used to define the range in which the sound within the environment of the user 100 is adjusted. The area 2030 may be calculated dynamically as the location of the individual 2010 relative to the device 110 changes. For example, as the user 100 turns, or if the individual 1020 moves within the environment, the processor 210 may be configured to track the individual 2010 within the environment and update the area 2030 dynamically. Zone 2030 may be used for selective adjustment, such as by amplifying sounds associated with zone 2030 and/or attenuating sounds determined to be emanating from outside zone 2030 .

The conditioned audio signal may then be sent to auditory interface device 1710 and an audio signal generated for user 100 . Thus, in the conditioned audio signal, sound 2020 (particularly speech 2012) may be louder and/or more distinguishable than sounds 2021 and 2022, which may represent background noise within the environment.

In some embodiments, the processor 210 may perform further analysis based on the captured images or video to determine how to selectively adjust the audio signal associated with the identified individual. In some embodiments, the processor 210 may analyze the captured images to selectively adjust audio associated with the individual relative to others. For example, the processor 210 may determine the orientation of the identified individual relative to the user based on the image, and may determine how to selectively adjust the audio signal associated with the individual based on the orientation. If an identified individual is standing in front of the user, the audio associated with that user may be amplified (or otherwise selectively adjusted) relative to the audio associated with the individual standing at the user's side. Similarly, the processor 210 may selectively adjust the audio signal associated with the individual based on the proximity to the user. The processor 210 can determine the distance from the user to each individual based on the captured images, and can selectively adjust audio signals associated with the individuals based on the distance. For example, individuals closer to the user may have higher priority than individuals further away from the user. In some embodiments, the angle between the user's viewing direction and the individual may also be considered. For example, individuals positioned at a smaller angle relative to the user's line of sight may be prioritized over individuals positioned at a larger angle to the user's line of sight.

In some embodiments, the selective adjustment of the audio signal associated with the identified individual may be based on the identity of the individual in the user's environment. For example, where multiple individuals are detected in the image, the processor 210 may identify the individuals using one or more facial recognition techniques as described above. Audio signals associated with individuals known to user 100 may be selectively amplified or otherwise adjusted to have priority over unknown individuals. For example, the processor 210 may be configured to attenuate or mute audio signals associated with bystanders in the user's environment (such as noisy office colleagues, etc.). In some embodiments, the processor 210 may also determine a hierarchy of individuals and give priority based on the relative status of the individuals. The hierarchy may be based on the location of the individual relative to the user within the family or organization (eg, company, sports team, club, etc.). For example, a user's boss may be ranked higher than a co-worker or a member of the maintenance team and thus may have priority in the selective moderation process. In some embodiments, the hierarchy may be determined based on a list or database. Individuals identified by the system can be sorted individually or grouped into layers of priority. The database can be maintained specifically for this purpose or accessed externally. For example, a database can be associated with a user's social network (eg, Facebook ^™ , LinkedIn ^™ , etc.), and individuals can be prioritized based on their groupings or relationship to the user. For example, individuals identified as "close friends" or family members may take precedence over the user's acquaintances.

Selective adjustments may be based on the determined behavior of one or more individuals determined from the captured images. In some embodiments, the processor 210 may be configured to determine the gaze direction of the individual in the image. Thus, selective conditioning can be based on the behavior of other individuals with respect to the identified individual. For example, the processor 210 may selectively adjust audio associated with the first individual being viewed by one or more other users. If the individual's attention is diverted to the second individual, the processor 210 may then switch to selectively adjust the audio associated with the second user. In some embodiments, the processor 210 may be configured to selectively adjust the audio based on whether the identified individual is speaking to the user or to another individual. For example, when the identified individual is speaking to the user, the selective adjustment may include amplifying the audio signal associated with the identified individual relative to other audio signals received from directions outside the area associated with the identified individual . When the identified individual is speaking to another individual, the selective adjustment may include attenuating the audio signal relative to other audio signals received from directions outside the area associated with the identified individual.

In some embodiments, the processor 210 may access one or more voiceprints of the individual, which may facilitate selective adjustment of the individual's 2010 speech 2012 relative to other sounds or speeches. Having a speaker's voiceprint, especially a high-quality voiceprint, can provide fast and efficient speaker separation. For example, when the user speaks alone, preferably in a quiet environment, high quality voiceprints can be collected. By having the voiceprints of one or more speakers, an ongoing speech signal can be separated in near real-time (eg, with minimal delay) using a sliding time window. The delay may be, for example, 10 milliseconds, 20 milliseconds, 30 milliseconds, 50 milliseconds, 100 milliseconds, and the like. Different time windows can be chosen depending on the quality of the voiceprint, the quality of the captured audio, the differences in characteristics between the speaker and other speakers, the processing resources available, the quality of separation required, etc. In some embodiments, voiceprints may be extracted from segments of a conversation in which an individual speaks alone, and then used to separate the individual's speech later in the conversation, whether or not the individual's voice is recognized.

Separating speech can be performed as follows: Spectral features (also known as spectral properties, spectral envelopes, or spectrograms) can be extracted from the clean audio of a single speaker and fed into a pretrained first neural network that The neural network generates or updates the signature of the speaker's speech based on the extracted features. The audio may be, for example, one second of clean speech. The output signature may be a vector representing the speaker's speech such that the distance between the vector and another vector extracted from the same speaker's speech is typically less than the distance between the vector and the vector extracted from the other speaker's voice the distance. Speaker models can be pre-generated from captured audio. Alternatively or additionally, the model may be generated after an audio segment in which only the speaker is speaking, followed by another audio segment in which the speaker and another speaker (or background noise) are heard and need to be separated. a fragment.

Then, to separate the speaker's speech from other speakers or background noise in the noise audio, a second pretrained neural network can receive the noise audio and the speaker's signature, and output the speaker's speech extracted from the noise audio The audio of speech (which can also be represented as an attribute) that is separated from other speech or background noise. It will be appreciated that the same or additional neural networks can be used to separate the speech of multiple speakers. For example, if there are two possible speakers, two neural networks can be activated, each with the same noise output and a model for one of the two speakers. Alternatively, the neural network may receive the speech signatures of two or more speakers and output each speaker's speech separately. Thus, the system can generate two or more distinct audio outputs, each audio output including the speech of a corresponding speaker. In some embodiments, the input speech may only be cleaned from background noise if separation is not possible.

21 is a flowchart illustrating an exemplary process 2100 for selectively amplifying an audio signal associated with a recognized individual's speech, consistent with the disclosed embodiments. Process 2100 may be performed by one or more processors associated with apparatus 110, such as processor 210. In some embodiments, some or all of process 2100 may be performed on a processor external to device 110 . In other words, the processor performing process 2100 may be included in the same common housing as microphone 1720 and camera 1730, or may be included in a second housing. For example, one or more portions of process 2100 may be performed by a processor in auditory interface device 1710 or an auxiliary device such as computing device 120 .

In step 2110, the process 2100 may include receiving a plurality of images captured by the camera from the user's environment. The images may be captured by a wearable camera such as camera 1730 of device 110 . In step 2112, the process 2100 can include identifying a representation of the identified individual in at least one of the plurality of images. As described above, individual 2010 can be identified by processor 210 using facial recognition component 2040. For example, individual 2010 may be a friend, colleague, relative, or former acquaintance of the user. Processor 210 may determine whether an individual represented in at least one of the plurality of images is an identified individual based on one or more detected facial features associated with the individual. As described above, the processor 210 may also determine whether to identify an individual based on one or more detected audio characteristics of sounds determined to be associated with the individual's speech.

In step 2114, the process 2100 can include receiving an audio signal representing the sound captured by the microphone. For example, device 110 may receive audio signals representing sounds 2020 , 2021 , and 2022 captured by microphone 1720 . Thus, as described above, the microphones may include directional microphones, microphone arrays, multi-port microphones, or various other types of microphones. In some embodiments, the microphone and wearable camera may be included in a common housing, such as that of device 110 . One or more processors that perform process 2100 may also be included in the enclosure (eg, processor 210), or may be included in a second enclosure. Where a second housing is used, the processor may be configured to receive image and/or audio signals from the common housing via a wireless link (eg, Bluetooth ^™ , NFC, etc.). Thus, the common housing (eg, apparatus 110) and the second housing (eg, computing device 120) may also include transmitters, receivers, and/or various other communication components.

In step 2116, process 2100 can include selectively adjusting at least one audio signal received by at least one microphone from a region associated with at least one identified individual. As described above, the area may be determined based on the orientation of the identified individual determined from one or more of the plurality of image or audio signals. The range may be associated with an angular width (eg, 10 degrees, 20 degrees, 45 degrees, etc.) with respect to the direction of the identified individual.

As mentioned above, various forms of conditioning can be performed on the audio signal. In some embodiments, adjusting may include changing the pitch or playback speed of the audio signal. For example, adjusting may include changing the rate of speech associated with the audio signal. In some embodiments, conditioning may include amplifying the audio signal relative to other audio signals received from outside the region associated with the identified individual. Amplification may be performed by various means, such as operating a directional microphone configured to focus on audio sounds emanating from the area, or changing one or more parameters associated with the microphone to focus the microphone on audio emanating from the area sound. Amplifying may include attenuating or suppressing one or more audio signals received by the microphone from directions outside the area. In some embodiments, step 2116 may also include determining that the identified individual is speaking based on the analysis of the plurality of images, and triggering selective adjustments based on the determination that the identified individual is speaking. For example, it may be determined that the identified individual is speaking based on the detected movement of the identified individual's lips. In some embodiments, selective adjustments may be based on further analysis of the captured images as described above, eg, based on the orientation or proximity of the identified individual, the identity of the identified individual, the behavior of other individuals, and the like.

In step 2118, the process 2100 may include transmitting the at least one conditioned audio signal to an auditory interface device configured to provide sound to a user's ear. For example, the conditioned audio signal may be sent to auditory interface device 1710, which may provide user 100 with sounds corresponding to the audio signal. The processor performing process 2100 may also be configured to cause one or more audio signals representing background noise, which may be attenuated relative to the at least one conditioned audio signal, to be transmitted to the auditory interface device. For example, processor 210 may be configured to transmit audio signals corresponding to sounds 2020 , 2021 and 2022 . However, based on the determination that sound 2020 is within region 2030, the signal associated with sound 2020 may be amplified relative to sounds 2021 and 2022. In some embodiments, auditory interface device 1710 may include a speaker associated with the earpiece. For example, auditory interface device 1710 may be inserted at least partially into a user's ear for providing audio to the user. The auditory interface device may also be external to the ear, such as a behind-the-ear hearing device, one or more earphones, small portable speakers, and the like. In some embodiments, the auditory interface device may include a bone conduction microphone configured to provide audio signals to the user through vibrations of the user's skull. Such devices may be placed in external contact with the user's skin, or may be surgically implanted and attached to the user's bone.

In addition to recognizing the speech of an individual speaking to the user 100, the systems and methods described above may also be used to recognize the speech of the user 100. For example, the speech recognition unit 2041 may be configured to analyze audio signals representing sounds collected from the user's environment to recognize the speech of the user 100 . Similar to the selective adjustment of the recognized individual's speech, the user's 100 speech may be selectively adjusted. For example, the sound may be collected by the microphone 1720 or by the microphone of another device such as a mobile phone (or a device linked to a mobile phone). For example, by amplifying the voice of the user 100 and/or attenuating or eliminating all sounds other than the user's voice, an audio signal corresponding to the voice of the user 100 may be selectively sent to the remote device. Accordingly, the voiceprints of one or more users of device 110 may be collected and/or stored to facilitate detection and/or isolation of the user's voice as described in further detail above.

FIG. 22 is a flow diagram illustrating an exemplary process 2200 for selectively transmitting audio signals associated with recognized speech of a user, consistent with the disclosed embodiments. Process 2200 may be performed by one or more processors associated with apparatus 110, such as processor 210.

In step 2210, the process 2200 can include receiving an audio signal representing sound captured by the microphone. For example, device 110 may receive audio signals representing sounds 2020 , 2021 , and 2022 captured by microphone 1720 . Thus, as described above, the microphones may include directional microphones, microphone arrays, multi-port microphones, or various other types of microphones. In step 2212, process 2200 may include identifying one or more speech audio signals representing the recognized speech of the user based on the analysis of the received audio signals. For example, the user's voice may be identified based on a voiceprint associated with the user, which may be stored in memory 550, database 2050, or other suitable location. The processor 210 can recognize the user's speech, eg, using the speech recognition component 2041 . The processor 210 may separate the ongoing speech signal associated with the user in near real-time (eg, with minimal delay) using a sliding time window. The speech can be separated by extracting the spectral features of the audio signal according to the above method.

In step 2214, the process 2200 may include transmitting one or more speech audio signals representing the recognized speech of the user to the remote device. The remotely located device may be any device configured to receive audio signals remotely via wired or wireless communication. In some embodiments, the remotely located device may be another device of the user, such as a mobile phone, audio interface device, or another form of computing device. In some embodiments, the speech audio signal may be processed and/or further transmitted by a remotely located device. In step 2216, process 2200 can include preventing transmission to the remotely located device of at least one background noise audio signal that is different from one or more speech audio signals representing the user's recognized speech. For example, processor 210 may attenuate and/or cancel audio signals associated with sounds 2020, 2021 or 2023, which may represent background noise. The user's speech can be separated from other noise using the audio processing techniques described above.

In an exemplary illustration, the speech audio signal may be captured by a headset or other device worn by the user. The user's speech can be recognized and isolated from background noise in the user's environment. The headset may transmit the conditioned audio signal of the user's speech to the user's mobile phone. For example, the user may be on a phone call and the conditioned audio signal may be sent by the mobile phone to the recipient of the call. The user's voice may also be recorded by a remotely located device. For example, the audio signal may be stored on a remote server or other computing device. In some embodiments, the remotely located device may process the received audio signal, eg, to convert the recognized speech of the user into text.

lip tracking hearing aids

Consistent with the disclosed embodiments, the hearing aid system may selectively amplify the audio signal based on the tracked lip movement. The hearing aid system analyzes the captured images of the user's environment to detect the individual's lips and track the movement of the individual's lips. The tracked lip movement can be used as a cue to selectively amplify the audio received by the hearing aid system. For example, speech signals determined to be synchronized with or consistent with tracked lip movement may be selectively amplified or otherwise conditioned. Audio signals not associated with detected lip movements may be suppressed, attenuated, filtered, etc.

The user 100 may wear a hearing aid device that conforms to the camera-based hearing aid device described above. For example, the hearing aid device may be an auditory interface device 1710 as shown in Figure 17A. Auditory interface device 1710 may be any device configured to provide auditory feedback to user 100 . The auditory interface device 1710 may be placed in one or both ears of the user 100, similar to conventional auditory interface devices. As mentioned above, the auditory interface device 1710 may be of various styles, including in-canal, completely in-canal, in-ear, behind-the-ear, supra-aural, in-canal receiver, open mount, or various other styles. Auditory interface device 1710 may include one or more speakers for providing auditory feedback to user 100, a microphone for detecting sounds in the environment of user 100, internal electronics, a processor, memory, and the like. In some embodiments, auditory interface device 1710 may include one or more communication units in addition to or instead of a microphone, and be one or more receivers for receiving signals from device 110 and transmitting signals to user 100 . The auditory interface device 1710 may correspond to the feedback output unit 230 or may be separate from the feedback output unit 230 and may be configured to receive signals from the feedback output unit 230 .

In some embodiments, the auditory interface device 1710 may include a bone conduction headset 1711, as shown in FIG. 17A. Bone conduction earphones 1711 may be surgically implanted and may provide audible feedback to user 100 through bone conduction of sound vibrations to the inner ear. The auditory interface device 1710 may also include one or more earphones (eg, wireless earphones, over-ear earphones, etc.) or portable speakers carried or worn by the user 100 . In some embodiments, the auditory interface device 1710 may be integrated into other devices, such as the user's Bluetooth ^™ headset, glasses, helmets (eg, motorcycle helmets, bicycle helmets, etc.), hats, and the like.

Auditory interface device 1710 may be configured to communicate with a camera device such as apparatus 110 . Such communication may be via a wired connection, or may be performed wirelessly (eg, using Bluetooth ^™ , NFC or wireless forms of communication). As described above, device 110 may be worn by user 100 in various configurations, including physically attached to a shirt, necklace, belt, eyeglasses, wristband, button, or other item associated with user 100 . In some embodiments, one or more additional devices such as computing device 120 may also be included. Accordingly, one or more of the processes or functions described herein with respect to apparatus 110 or processor 210 may be performed by computing device 120 and/or processor 540 .

As mentioned above, the apparatus 110 may include at least one microphone and at least one image capture device. Device 110 may include microphone 1720 as described with respect to FIG. 17B . Microphone 1720 may be configured to determine the directionality of sound in the environment of user 100 . For example, microphone 1720 may include one or more directional microphones, microphone arrays, multi-port microphones, and the like. The processor 210 may be configured to differentiate sounds within the environment of the user 100 and determine the approximate directionality of each sound. For example, using the microphone array 1720 , the processor 210 may compare the relative timing or amplitude of individual sounds between the microphones 1720 to determine the directionality relative to the device 100 . Device 110 may include one or more cameras, such as camera 1730 , which may correspond to image sensor 220 . The camera 1730 may be configured to capture images of the user's 100 surroundings. Apparatus 110 may also use one or more microphones of auditory interface device 1710 , and thus, references to microphone 1720 used herein may also refer to a microphone on auditory interface device 1710 .

Processor 210 (and/or processors 210a and 210b ) may be configured to detect mouths and/or lips associated with individuals within the environment of user 100 . 23A and 23B illustrate an exemplary individual 2310 that may be captured by camera 1730 in a user environment consistent with the present disclosure. As shown in FIG. 23 , an individual 2310 may physically exist in the environment of the user 100 . Processor 210 may be configured to analyze images captured by camera 1730 to detect representations of individuals 2310 in the images. Processor 210 may use a facial recognition component, such as facial recognition component 2040 described above, to detect and identify individuals in the environment of user 100 . The processor 210 may be configured to detect one or more facial features of the user 2310, including the lips 2311 of the individual 2310. Accordingly, the processor 210 may use one or more facial recognition and/or feature recognition techniques, as described further below.

In some embodiments, processor 210 may detect a visual representation of individual 2310 from the environment of user 100, such as a video of user 2310. As shown in FIG. 23B, user 2310 may be detected on the display of display device 2301. Display device 2301 may be any device capable of displaying a visual representation of an individual. For example, the display device may be a personal computer, laptop computer, mobile phone, tablet computer, television, movie screen, handheld gaming device, videoconferencing device (eg, Facebook Portal ^™ , etc.), baby monitor, and the like. The visual representation of the individual 2310 may be a real-time video feed of the individual 2310, such as a video call, conference call, surveillance video, or the like. In other embodiments, the visual representation of the individual 2310 may be a pre-recorded video or image, such as a video message, a television show, or a movie. The processor 210 may detect one or more facial features, including the mouth 2311 of the individual 2310, based on the visual representation of the individual 2310.

23C illustrates an exemplary lip tracking system consistent with the disclosed embodiments. The processor 210 may be configured to detect one or more facial features of the individual 2310, which may include, but are not limited to, the individual's mouth 2311. Accordingly, the processor 210 may identify the user's facial features using one or more image processing techniques, such as convolutional neural network (CNN), scale-invariant feature transform (SIFT), histogram of oriented gradient (HOG) features, or other technology. In some embodiments, the processor 210 may be configured to detect one or more points 2320 associated with the mouth 2311 of the individual 2310. Points 2320 may represent one or more characteristic points of the individual's mouth, such as one or more points along the individual's lips or the corners of the individual's mouth. The points shown in Figure 23C are for illustration purposes only, and it should be understood that any point for tracking an individual's lips may be determined or identified via one or more image processing techniques. Points 2320 may be detected at various other locations, including points associated with an individual's teeth, tongue, cheeks, chin, eyes, and the like. The processor 210 may determine one or more contours (eg, represented by lines or polygons) of the mouth 2311 based on the points 2320 or based on the captured image. The contour may represent the entire mouth 2311, or may include multiple contours, including, for example, a contour representing the upper lip and a contour representing the lower lip. Each lip may also be represented by a plurality of contours, such as the contour of the upper edge of each lip and the contour of the lower edge of each lip. The processor 210 may also identify the lips of the individual 2310 using various other techniques or characteristics, such as color, edge, shape, or motion detection algorithms. Identified lips can be tracked over multiple frames or images. The processor 210 may use one or more video tracking algorithms, such as mean-shift tracking, contour tracking (eg, compression algorithms), or various other techniques. Accordingly, the processor 210 may be configured to track the movement of the lips of the individual 2310 in real time.

If desired, the tracked lip movement of the individual 2310 can be used to isolate and selectively adjust one or more sounds in the user's 100 environment. 24 is a schematic diagram illustrating an exemplary environment 2400 for using a lip tracking hearing aid consistent with the present disclosure. Device 110 worn by user 100 may be configured to identify one or more individuals within environment 2400 . For example, device 110 may be configured to capture one or more images of surrounding environment 2400 using camera 1730 . The captured images may include representations of individuals 2310 and 2410 , which may exist in environment 2400 . Processor 210 may be configured to detect the mouths of individuals 2310 and 2410 and track their respective lip movements using the methods described above. In some embodiments, processor 210 may also be configured to identify individuals 2310 and 2410 by detecting facial features of individuals 2310 and 2410 and comparing them to a database, for example, as previously discussed.

In addition to detecting images, apparatus 110 may be configured to detect one or more sounds in the environment of user 100 . For example, microphone 1720 may detect one or more sounds 2421 , 2422 and 2423 within environment 2400 . In some embodiments, the sounds may represent the speech of various individuals. For example, as shown in FIG. 24, sound 2421 may represent the speech of individual 2310, and sound 2422 may represent the speech of individual 2410. Sound 2423 may represent additional speech and/or background noise within environment 2400. Processor 210 may be configured to analyze sounds 2421, 2422, and 2423 to separate and identify audio signals associated with speech. For example, the processor 210 may use one or more speech or voice activity detection (VAD) algorithms and/or the above-described speech separation techniques. When multiple sounds are detected in the environment, the processor 210 may isolate the audio signal associated with each speech. In some embodiments, the processor 210 may perform further analysis on the audio signal associated with the detected voice activity to identify the individual's voice. For example, the processor 210 may use one or more speech recognition algorithms (eg, hidden Markov models, dynamic time warping, neural networks, or other techniques) to recognize an individual's speech. Processor 210 may also be configured to recognize words spoken by individual 2310 using various speech-to-text algorithms. In some embodiments, instead of using microphone 1710, apparatus 110 may receive audio signals from another device through a communication component, such as wireless transceiver 530. For example, if the user 100 is on a video call, the apparatus 110 may receive an audio signal representing the voice of the user 2310 from the display device 2301 or another auxiliary device.

Processor 210 may determine which individuals in environment 2400 are speaking based on lip movements and detected sounds. For example, processor 2310 may track lip movements associated with mouth 2311 to determine that individual 2310 is speaking. A comparative analysis can be performed between the detected lip movement and the received audio signal. In some embodiments, processor 210 may determine that individual 2310 is speaking based on a determination that mouth 2311 is moving while sound 2421 is detected. For example, when the lips of individual 2310 cease to move, this may correspond to periods of silence or reduced volume in the audio signal associated with sound 2421. In some embodiments, the processor 210 may be configured to determine whether a particular movement of the mouth 2311 corresponds to a received audio signal. For example, the processor 210 may analyze the received audio signal to identify particular phonemes, phoneme combinations, or words in the received audio signal. The processor 210 may identify whether a particular lip movement of the mouth 2311 corresponds to a recognized word or phoneme. Various machine learning or deep learning techniques can be implemented to correlate expected lip movements with detected audio. For example, a training dataset of known sounds and corresponding lip movements can be fed to a machine learning algorithm to develop a model for correlating detected sounds with expected lip movements. Other data associated with device 110 may also be used in conjunction with detected lip movements to determine and/or verify whether individual 2310 is speaking, such as user 100 or individual 2310 gaze direction, detected identity of user 2310, identified Voiceprint of user 2310, etc.

Based on the detected lip movement, the processor 210 may cause selective adjustments to the audio associated with the individual 2310. Adjusting may include amplifying the audio signal determined to correspond to sound 2421 (which may correspond to the speech of individual 2310 ) relative to other audio signals. In some embodiments, amplification may be accomplished digitally, for example, by processing the audio signal associated with sound 2421 relative to other signals. Additionally or alternatively, amplification may be achieved by changing one or more parameters of the microphone 1720 to focus on audio sounds associated with the individual 2310. For example, the microphone 1720 may be a directional microphone, and the processor 210 may perform an operation to focus the microphone 1720 on the sound 2421. Various other techniques for amplifying sound 2421 may be used, such as the use of beamforming microphone arrays, acoustic telescope techniques, and the like. The adjusted audio signal may be sent to auditory interface device 1710, and thus may provide user 100 with audio adjusted based on the individual speaking.

In some embodiments, the selective adjustment may include attenuating or suppressing one or more audio signals, such as sounds 2422 and 2423, not associated with the individual 2310. Similar to the amplification of sound 2421, attenuation of sound may occur by processing the audio signal or by changing one or more parameters associated with microphone 1720 to direct focus away from sounds not associated with individual 2310.

In some embodiments, adjusting may also include changing the pitch of one or more audio signals corresponding to sound 2421 to make the sound more perceptible to user 100 . For example, the user 100 may have less sensitivity to tones within a certain range, and the adjustment of the audio signal may adjust the pitch of the sound 2421. For example, user 100 may experience hearing loss in frequencies above 10 kHz, and processor 210 may remap higher frequencies (eg, at 15 kHz) to 10 kHz. In some embodiments, the processor 210 may be configured to vary the speech rate associated with one or more audio signals. The processor 210 may be configured to alter the speech rate of the individual 2310 to make the detected speech more perceptible to the user 100 . If speech recognition has been performed on the audio signal associated with the sound 2421, the conditioning may also include modifying the audio signal based on the detected speech. For example, processor 210 may introduce pauses or increase the duration of pauses between words and/or sentences, which may make speech easier to understand. Various other processing may be performed, such as modifying the pitch of the sound 2421, to maintain the same pitch as the original audio signal, or to reduce noise within the audio signal.

The conditioned audio signal may then be sent to auditory interface device 1710 , which then generates an audio signal for user 100 . Thus, in the conditioned audio signal, sound 2421 (may be louder and/or easier to distinguish than sounds 2422 and 2423).

The processor 210 may be configured to selectively condition the plurality of audio signals based on which individuals associated with the audio signals are currently speaking. For example, individual 2310 and individual 2410 may engage in a conversation within environment 2400, and processor 210 may be configured to transition from the modulation of the audio signal associated with sound 2421 to the modulation of the audio signal associated with sound 2421 based on the respective lip movements of individual 2310 and individual 2410 2422 Adjustment of the associated audio signal. For example, lip movement of individual 2310 may indicate that individual 2310 has stopped speaking, or lip movement associated with individual 2410 may indicate that individual 2410 has begun to speak. Accordingly, the processor 210 may switch between selectively adjusting the audio signal associated with sound 2421 to the audio signal associated with sound 2422. In some embodiments, processor 210 may be configured to process and/or condition both audio signals simultaneously, but selectively send the conditioned audio to auditory interface device 1710 based only on which individual is speaking. Where speech recognition is implemented, the processor 210 may determine and/or anticipate transitions between speakers based on the context of the speech. For example, processor 210 may analyze the audio signal associated with sound 2421 to determine that individual 2310 has reached the end of a sentence or has asked a question, which may indicate that individual 2310 has finished or is about to finish speaking.

In some embodiments, the processor 210 may be configured to select between multiple active speakers to selectively condition the audio signal. For example, individuals 2310 and 2410 may both be speaking at the same time, or their speech may overlap during the conversation. The processor 210 may selectively adjust audio associated with a speaking individual relative to others. This may include giving priority to a speaker who has started but has not completed a word or sentence or who has not fully finished speaking when another speaker starts speaking. As mentioned above, this determination can also be driven by the context of the speech.

Various other factors can also be considered when selecting active speakers. For example, the user's gaze direction may be determined, and individuals in the user's gaze direction may be given higher priority among active speakers. Priority can also be assigned based on the speaker's gaze direction. For example, if individual 2310 is looking at user 100 and individual 2410 is looking elsewhere, the audio signal associated with individual 2310 may be selectively adjusted. In some embodiments, priorities may be assigned based on the relative behavior of other individuals in the environment 2400 . For example, if both individual 2310 and individual 2410 are talking, and more other individuals are looking at individual 2410 than at individual 2310, the audio signal associated with individual 2410 may be prioritized over the audio signal associated with individual 2310 Selectively adjust. In embodiments where the identity of an individual is determined, as discussed in greater detail above, priorities may be assigned based on the relative status of the speakers. User 100 may also provide input on which speakers are to be prioritized through predefined settings or by actively selecting which speakers to focus on.

The processor 210 may also assign priorities based on how the representation of the individual 2310 is detected. Although individuals 2310 and 2410 are shown physically present in environment 2400, as shown in Figure 23B, one or more individuals may be detected as visual representations of the individuals (eg, on a display device). Processor 210 may prioritize speakers based on whether they are physically present in environment 2400 . For example, the processor 210 may prioritize physically present speakers over speakers on the display. Alternatively, for example, if the user 100 is on a video conference, or if the user 100 is watching a movie, the processor 210 may prioritize the video over the speakers in the room. User 100 may also indicate a prioritized speaker or speaker type (eg, presence or absence) using a user interface associated with device 110 .

25 is a flowchart illustrating an exemplary process 2500 for selectively amplifying an audio signal based on tracked lip movement, consistent with the disclosed embodiments. Process 2500 may be performed by one or more processors associated with apparatus 110, such as processor 210. The processor may be included in the same common housing as the microphone 1720 and camera 1730 that may also be used in process 2500 . In some embodiments, some or all of process 2500 may be performed on a processor external to device 110, which may be included in the second housing. For example, one or more portions of process 2500 may be performed by a processor in auditory interface device 1710 or an auxiliary device such as computing device 120 or display device 2301 . In such an embodiment, the processor may be configured to receive the captured image via a wireless link between the transmitter in the common housing and the receiver in the second housing.

In step 2510, the process 2500 can include receiving a plurality of images captured by the wearable camera from the user's environment. The images may be captured by a wearable camera such as camera 1730 of device 110 . In step 2520, process 2500 can include identifying a representation of at least one individual in at least one of the plurality of images. Various image detection algorithms can be used to identify individuals, such as Haar cascade, Histogram of Oriented Gradients (HOG), Deep Convolutional Neural Networks (CNN), Scale Invariant Feature Transform (SIFT), and others. In some embodiments, processor 210 may be configured to detect a visual representation of an individual from a display device, eg, as shown in Figure 23B.

In step 2530, the process 2500 can include identifying at least one lip movement or lip position associated with the individual's mouth based on the analysis of the plurality of images. The processor 210 may be configured to identify one or more points associated with the individual's mouth. In some embodiments, the processor 210 may develop a profile associated with the individual's mouth, which may define a boundary associated with the individual's mouth or lips. Lips identified in an image can be tracked over multiple frames or images to identify lip movement. Accordingly, the processor 210 may use various video tracking algorithms as described above.

In step 2540, the process 2500 may include receiving an audio signal representing sound captured by the microphone from the user's environment. For example, device 110 may receive audio signals representing sounds 2421 , 2422 , and 2423 captured by microphone 1720 . In step 2550, process 2500 may include identifying a first audio signal associated with the first speech and a second audio signal associated with a second speech different from the first speech based on the analysis of the sound captured by the microphone. For example, processor 210 may identify audio signals associated with sounds 2421 and 2422 representing the speech of individuals 2310 and 2410, respectively. The processor 210 may use any currently known or future developed technique or algorithm to analyze the sound received from the microphone 1720 to separate the first and second speech. Step 2550 may also include identifying additional sounds (such as sounds 2423), which may include additional sounds or background noise in the user's environment. In some embodiments, the processor 210 may perform further analysis on the first and second audio signals, eg, by using the available voiceprints of the individuals 2310 and 2410 to determine their identities. Alternatively or additionally, the processor 210 may use speech recognition tools or algorithms to recognize the individual's speech.

In step 2560, process 2500 may include selectively adjusting the first audio signal based on determining that the first audio signal is associated with the identified lip movement associated with the individual's mouth. The processor 210 may compare the identified lip movement to the first and second audio signals identified in step 2550. For example, the processor 210 may compare the timing of the detected lip movement to the timing of speech patterns in the audio signal. In embodiments where speech is detected, the processor 210 may also compare certain lip movements to phonemes or other features detected in the audio signal, as described above. Accordingly, the processor 210 may determine that the first audio signal is associated with the detected lip movement, and thus with the speaking individual.

As mentioned above, various forms of selective adjustment can be performed. In some embodiments, adjusting may include changing the pitch or playback speed of the audio signal. For example, the adjustment may include remapping the audio frequency or changing the speech rate associated with the audio signal. In some embodiments, adjusting may include amplifying the first audio signal relative to other audio signals. Amplification may be performed by various means, such as operation of a directional microphone, changing one or more parameters associated with the microphone, or digitally processing the audio signal. Adjusting may include attenuating or suppressing one or more audio signals not associated with the detected lip movement. The attenuated audio signal may include audio signals associated with other sounds detected in the user's environment, including other speech such as the second audio signal. For example, the processor 210 may selectively attenuate the second audio signal based on determining that the second audio signal is not associated with the identified lip movement associated with the individual's mouth. In some embodiments, the processor may be configured to extract from the audio signal associated with the first individual when the identified lip movement of the first individual indicates that the first individual has completed the sentence or has completed speaking The conditioning translates to conditioning of the audio signal associated with the second individual.

In step 2570, process 2500 may include causing transmission of the selectively conditioned first audio signal to an auditory interface device configured to provide sound to the user's ear. For example, the conditioned audio signal may be sent to auditory interface device 1710, which may provide user 100 with a sound corresponding to the first audio signal. Additional sounds such as a second audio signal may also be sent. For example, processor 210 may be configured to transmit audio signals corresponding to sounds 2421 , 2422 and 2423 . However, the first audio signal that may be associated with the detected lip movement of the individual 2310 may be amplified relative to the sounds 2422 and 2423 as described above. In some embodiments, auditory interface device 1710 may include a speaker associated with the earpiece. For example, an auditory interface device may be inserted at least partially into a user's ear for providing audio to the user. The auditory interface device may also be external to the ear, such as a behind-the-ear hearing device, one or more earphones, small portable speakers, and the like. In some embodiments, the auditory interface device may include a bone conduction microphone configured to provide audio signals to the user through vibrations of the user's skull. Such devices may be placed in external contact with the user's skin, or may be surgically implanted and attached to the user's bone.

Multimodal Hearing Aid

According to embodiments of the present disclosure, a hearing aid system may include a wearable camera configured to capture multiple images from the user's environment. In various embodiments, the hearing aid system may include at least one microphone configured to capture sound from the user's environment. In some embodiments, the hearing aid system may include more than one microphone. In an example embodiment, a hearing aid system may include a first microphone for capturing audio signals in a first wavelength range and a second microphone for capturing audio signals in a second wavelength range. In example embodiments, the hearing aid system may include multiple cameras and/or multiple microphones. For example, FIG. 26 shows a user 2601 who may wear device 110, which may include cameras 2617a and 2617b and microphone 2613. As described herein, device 110 may be attached to user 2601 in various locations. For example, device 110 may be physically attached to shirts, necklaces, belts, glasses, wristbands, buttons, and the like.

Apparatus 110 may be configured to communicate with an auditory interface device, such as auditory interface device 2615 shown in FIG. 26 . In one example embodiment, the auditory interface device 2615, the apparatus 110, and various cameras and microphones form a hearing aid system. In some embodiments, device 110 may receive video and audio data from other cameras and microphones, respectively. Camera 2617A may point in a first direction (eg, forward), and camera 2617B may point forward or sideways. It should be understood that the particular orientation of cameras 2617A and 2617B is illustrative only and any other suitable orientation of these cameras may be used. While auditory interface device 2615 is shown attached to one of user 2601's ears, in some embodiments auditory interface device 2615 may have a left portion (shown) configured to attach to the left ear and configured to attach to the left ear. Connected to the right part of the right ear (not shown).

It should be understood that cameras 2617A-2617B may be any suitable cameras with any suitable optics. For example, camera 2617A may have a first resolution, and camera 2617B may have a second (eg, higher) resolution. The camera may be configured to capture image data via an image sensor capable of detecting any suitable optical signal in any suitable wavelength spectrum (eg, the near-infrared, infrared, visible, and ultraviolet spectrums). In some cases, cameras 2617B-2617B can be configured to capture images, and in other cases, cameras 2617A-2617B can be configured to capture video data. Cameras 2617A-2617B may include optical lenses (eg, fisheye ultra-wide angle lenses for creating wide panoramic or hemispherical images or video). In some cases, a zoom lens such as a periscope lens may be used to zoom to different objects in the user's 2601 environment. In an example embodiment, cameras 2617A-2617B may be configured to zoom in the direction of the audio signal detected by microphone 2613. In some cases, the cameras 2617A-2617B may have a frame system to eliminate vibration and/or maintain certain orientations of the cameras. As mentioned above, the cameras 2617A-2617B can work in the infrared spectrum, especially in dark environments. Such cameras may include infrared flashlights, and may be configured to detect the skin temperature of surrounding people (eg, skin temperature may be used to detect nearby speakers in dark environments).

Device 110 may include at least one processor 2641 programmed to receive a plurality of images captured by a wearable camera (eg, by camera 2617A). Processor 2641 may be configured to analyze the environment of user 2601 using a computer-based model by analyzing images collected by a wearable camera (eg, camera 2617B). In an example embodiment, camera 2617B can detect the presence of children 2602 that can produce audio signals, and can detect other objects in the environment of user 2601 that can produce audio signals (eg, camera 2617B can detect cats 2603 that can produce audio signals, Presence of computer 2619 that can generate audio signals via conference software 2618, etc.). In various embodiments, processor 2641 may execute a suitable software application configured to analyze and identify objects, people or animals within image data (or video data) as discussed herein. In addition to the processor 2641 being part of the apparatus 110, the auditory interface device 2615 may also include a processor configured to modify various audio signals and provide the modified audio signals to the ear of the user 2601. In some cases, a processor associated with auditory interface device 2615 may perform some (or all) of the functions performed by processor 2641.

In an example embodiment, the processor 2641 of the device 110 may analyze one or more captured images. For example, processor 2641 may be configured to receive various images or characteristics of a person captured by cameras 2617A-2617B. Additionally, the apparatus 110 may be configured to communicate with a server that may store images of various objects and/or people. In an example embodiment, the apparatus 110 may upload or download images from the server. Furthermore, the device 110 may perform a search for images (or videos) stored at the server. Similar to the embodiments discussed above, the processor 2641 may use computer-based models to analyze and identify objects. In an example embodiment, the computer-based model may include a trained neural network, such as a convolutional neural network (CNN). In some cases, facial features can be analyzed by suitable computer-based models. For example, a computer-based model, such as a CNN, can be used to analyze images and correlate facial features or relationships between facial features of persons identified in captured images with persons found in images stored in the server's database Facial features or the relationship between the two are compared. In some embodiments, a video of the dynamic movement of a person's face may be compared to video data records of various persons obtained from a database in order to determine that the person captured in the video is an identified individual.

As described herein, auditory interface device 2615 may be any device configured to provide auditory feedback to user 2601. The auditory interface device 2615 may be placed in one or both ears of the user 2601, similar to conventional auditory interface devices. The auditory interface device 2615 may be of various styles, including in-canal, fully in-canal, in-ear, behind-the-ear, supra-ear, in-canal receiver, open mount, or various other styles. Auditory interface device 2615 may include one or more speakers for providing auditory feedback to user 2601, a microphone for detecting sounds in the environment of user 2601, internal electronics, a processor, memory, and the like. The auditory interface device 2615 may include an auditory interface (eg, buttons) for manually adjusting audio signal parameters (eg, loudness, pitch, etc.) of the audio signal sent by the auditory interface device. In some cases, device 2615 may include a processor for performing audio signal manipulation, a power source (eg, a rechargeable battery), an optional wireless communication device (which may include an antenna), and a set of microphones.

In various embodiments, the processor 2641 is configured to receive a plurality of audio signals representing sounds captured by the at least one microphone from the environment of the user 2601. Such a signal may be a combination of sounds produced by various entities in the user's 2601 environment. For example, the sounds may include children 2602 talking while cats 2603 meowing, or/and a group of people attempting to communicate with user 2601 via computer 2619 (eg, through conferencing software 2618). In various embodiments, the audio signals may overlap, resulting in noise in the sound. Such audio environments with multiple sounds may be referred to as noisy environments, and such environments may differ significantly from relatively noise-free (also referred to herein as calm) environments. A noisy environment as shown in Figure 26 is an example of such an environment. Other examples of noisy environments may include parties with multiple individuals speaking, television programs with multiple individuals speaking on the background (the background may include street sounds, music, etc.), plays, conferences, dinners, lectures, computer-based conferences, Conversations in bars or restaurants, conversations in the background (eg, next to a busy road or construction site), public transport (eg, buses, trains, boats, or planes), etc. Examples of calm environments may include a private office, a library, two persons conversing in a quiet place, and so on.

In various embodiments, the processor 2641 of the device 110 may execute software instructions configured to analyze visual and audio data of the user's 2601 environment and determine whether the user is in a noisy environment or a calm environment. While it will be appreciated that the analysis is performed by a processor 2641 executing software instructions, the software instructions may be any suitable instructions and may include machine learning algorithms, for the sake of brevity, the processor may execute program instructions to analyze various sounds and cause Various actions that affect the sound characteristics of the audio signal received by the user 2601 via the auditory interface device 2615.

Depending on the type of environment (eg, noisy or calm environment, or various other distinctions), the processor 2641 may be configured to operate in different modes when receiving different types of audio signals from the environment. In an example embodiment, when the processor 2641 determines that the environment is calm, the processor may operate in a first mode, which may include a ) specific selective modulation (referred to herein as the first selective modulation). In some cases, when the environment is sufficiently calm, the processor 2641 may be configured not to provide conditioning of the first audio signal and to send the first audio signal directly to the user 2601 via the auditory interface device 2615. Alternatively, at least some of the first selective adjustments may be performed. For example, noise (eg, background noise from a fan) can be suppressed and the sound of human speech can be amplified. In some cases, the hearing system 2650 may be configured to determine whether a person's speech is partially inaudible or unclear (eg, the processor 2641 may be configured to perform speech recognition), and when it is determined that the speech is partially inaudible or unclear (eg, , the person does not speak some words loud enough or clearly), the person's speech can be modified so that the speech intelligibility is improved. In an example embodiment, human speech may be transcribed, and the transcribed text may be read to user 2601 via natural reading speech (this process may be referred to as speech rendering) to clarify the speech of the person interacting with user 2601. In an example embodiment, speech rendering may be used to remove a speaker's accent or reapply a different accent to the speech. In an example embodiment, the original audio signal of the human speech may be combined (eg, morphed) with the rendered speech in order to preserve some of the speaker's natural characteristics while maintaining clear speech. Speech rendering may include changing the pitch of the person speaking (eg, such rendering may be beneficial if the user 2601 has difficulty recognizing a particular frequency), the cadence of the person's speech, the loudness of the person's speech, or any other aspect of the person's speech characteristics (eg, filters can be applied to human speech to change the human voice from male to female).

It is important to note that even in a calm environment, there can be several sources of sound (eg, two bodies quietly communicating with a third person, the sound of quiet music playing in the background, etc.). In an example embodiment, the apparatus 110 may include an interface for adjusting the parameters of the first selective adjustment to be applied. In an example embodiment, such an interface may include a smartphone-based interface using, for example, an application with graphical user elements, buttons that may be part of the hearing system 2650, or a voice interface (eg, the user 2601 may be configured to Control device 110) via voice commands. Among other things, example commands requested by user 2601 to control at least some parameters of the first selective adjustment may include a command from a speaker (or other source) speaking to user 2601 (or emitting any type of audio signal to user 2601). quantity. As an example, when user 2601 is participating in a speech (the speech is a relatively calm environment), he or she may wish to hear only the speaker and not background noise or other people's speech. For this case, the user 2601 may instruct the hearing system 2650 (ie, the processor 2641 via an appropriate software application) to reduce the amplitude of the ambient audio signal (ie, the signal unrelated to the speaker's speech), and in some cases Amplifies the speaker's voice. User 2601 may instruct hearing system 2650 to improve the intelligibility of the speaker's speech when the speaker is far away from the user. In some cases, instructions to hearing system 2650 may also include changing the speaker's accent as described above, translating the speaker's speech into a different language selected by user 2601, or applying any other form of modification. As another example, when user 2601 is at a social event and talking to several entities, he or she may wish to hear each of the several entities as they speak. In some cases, user 2601 may or may not wish to hear herself, or she may prefer to hear herself at a lower amplitude. In this case, the user 2601 may instruct the device 110 to reduce the amplitude of his or her voice. This change in the user's perceived speech may be another example of a first selective adjustment. Additionally, as described above, when capturing other audio signals (eg, speech by other individuals), the user 2601 may wish to hear lower amplitude background noise (or not), but when one or more microphones of the hearing aid 2650 User 2601 may wish to hear higher amplitude background noise when no other audio signal is detected.

Where the user is in a relatively noisy environment, different modes of operation of the hearing aid system may be performed (herein, such different modes of operation are referred to as second modes of operation). In an example embodiment, the processor 2641 of the apparatus 110 may operate in a second mode, which may include a particular selective adjustment mode (herein referred to as a second selection) of at least one audio signal of the plurality of audio signals sexual regulation). In some cases, the processor 2641 may determine to switch to the second mode to cause a second selective adjustment of the first audio signal based on analysis of at least one of the plurality of images or the plurality of audio signals, the second selection The sexual conditioning differs from the first selective conditioning in at least one respect. For example, when the environment is noisy, the processor 2641 may be configured to provide stronger audio signal conditioning than the first selective conditioning. In an example embodiment, the selectively adjusted strength may be defined as the difference in the audio signal when the adjusted audio signal is compared to the unadjusted audio signal under a suitable metric. A suitable metric may determine the pitch difference between the conditioned audio signal and the conditioned audio signal, or the difference in amplitude, tempo, time stretch, or any other suitable parameter that can be used to characterize the audio signal. In various embodiments, the second selective adjustment may differ in at least one respect relative to the first selective adjustment.

An example of a second selective adjustment may include reducing the amplitude of the audio signal from the child 2602 when using the computer 2619 to communicate via a web conference. Similarly, the second selective adjustment may include reducing ambient noise (eg, noise from cat 2603 or any other indoor noise) when communicating via a web conference.

In various embodiments, the determination of whether to use the first selective adjustment or the second selective adjustment may be made automatically or manually (ie, by using a suitable interface such as a graphical interface, buttons, or voice commands via commands from the user 2601 ) ). In an example embodiment, the processor 2641 may perform the automatic determination by determining whether the environment of the user 2601 is calm or noisy. For example, the processor 2641 may switch to a first selective adjustment when it determines that the environment is calm, and may switch to a second selective adjustment when it determines that the environment is noisy. In some cases, during switching operations, the first selective adjustment is maintained in part, while the second selective adjustment is superimposed on the first selective adjustment. For example, if User 2601 is communicating via a web conference without ambient noise interference, the first selective adjustment may include reducing the amplitude of User 2601's speech as described above. However, when the child 2602 enters the room creating a noisy environment, the processor 2641 can switch to the second selective adjustment and reduce the perceived amplitude of the child's voice while maintaining the first selective adjustment. In an example embodiment, image data associated with the arrival of the child 2602 (or other image data such as the arrival of the cat 2603) may trigger the processor 2641 to determine that the user 2601 is about to be immersed in a noisy environment, and as a result, the processor 2641 It may be determined that a second selective adjustment needs to be used when processing audio signals in the environment of user 2601 .

In some cases, user 2601 may determine a set of parameters (eg, via an appropriate interface for device 110) such that when these parameters are observed, processor 2641 may determine that user 2601 may be in a noisy environment. For example, the list of possible parameters includes audio and image/video parameters. Audio parameters may include maximum amplitude of the audio signal during a given time interval, maximum change in audio frequency, maximum amplitude of the audio signal averaged over a given time interval, distribution of maximum amplitudes as a function of audio frequency, and the like. The image/video parameters may include the speed at which the object moves in the user's 2601 environment, the rate of change of the image gradient of the image captured in the user's 2601 environment, the image recognition software by the processor 2641 (or the processor 2641 via the The transmission of the image data captured in the environment is the presence of objects, people or animals in the environment of the user 2601 identified by the image recognition software of the device with which it communicates. In some cases, a temporal correlation between motion of objects in user 2601's environment and audio signals detected in user 2601's environment can be used to determine whether user 2601's environment is noisy or calm. For example, if the motion of objects around user 2601 may not be correlated with the sound produced, and the amplitude of the sound produced is low, processor 2641 may conclude that user 2601's environment is relatively calm. Alternatively, processor 2641 may conclude that user 2601's environment is chaotic if processor 2641 determines that the sound may have occurred after objects in user 2641's environment moved quickly (possibly with some time delay). In some cases, the determination of whether the environment is calm or noisy may be based on actions performed around the user 2601 (eg, the processor 2641 may detect, by analyzing the image data, that the phone is being picked up by the person next to the user 2601, a band is about to play, User 2601 is entering a busy street, etc.).

In some cases, apparatus 110 may also be configured to exchange data with various other devices to determine whether user 2601 is placed in a noisy or calm environment. For example, the apparatus 110 may be configured to interact with a smartphone (or similar electronic device capable of exchanging data with the system 2650) having various sensors that the apparatus 110 would otherwise have unnecessary access to. In an example embodiment, the smartphone may use the GPS location to determine if the user 2601 is in a noisy environment (eg, if the GPS indicates that the user 2601 is in a bar, the smartphone may conclude (via suitable software associated with the device 110) that the user 2601 is in a bar in a noisy environment). Additionally, the smartphone can determine if the user 2601 is in a noisy environment if the smartphone records loud vibrations combined with some other factor (eg, excessive noise, which in turn may follow/precede/consistent with the vibration). In some embodiments, events in the user's calendar, such as lectures, meetings, concerts, etc., may provide an indication that the user is in a calm or noisy environment. In some cases, identifying a particular individual (eg, child 2602 ) in image data collected by device 110 or a smartphone interacting with device 110 may indicate to processor 2641 that user 2601 is in a noisy environment. Similarly, identifying a particular audio signal (eg, the voice of the child 2602) can indicate to the processor 2641 that the user 2601 is in a noisy environment.

In some embodiments, the operating mode of the device 110 may automatically change according to the environment of the user 2601, wherein the environment need not be classified as noisy or calm. For example, certain circumstances (eg, events associated with user 2601) may cause processor 2641 (or an associated device, such as a smartphone) to determine that apparatus 110 should operate in a certain mode. For example, when user 2601 is entering a particular location (eg, a lecture room), processor 2641 may determine to switch to a particular mode of operation (eg, a first selective adjustment) in which audio signals from only one speaker are sent to the auditory Interface device 2615. As another example, when the user 2601 leaves a location (eg, a lecture room), the processor 2641 (and/or associated device) may determine to switch to another mode of operation (eg, a second selective adjustment) at which Mode when one of those individuals is speaking (when several individuals are speaking at the same time) audio signals from multiple individuals may be sent to the auditory interface device 2615, and the processor 2641 may be based on various possible factors discussed herein (eg, The proximity of the individual to the user 2601, whether the individual is in front of the user 2601, whether the user 2601 is looking at the individual, etc.) determines which audio signal is amplified and which audio signal is attenuated.

Whether the user 2601 is in a noisy or calm environment is determined by the processor 2641 (or by an associated device such as a smartphone in communication with the processor 2641), and automatically between the first selective adjustment and the second selective adjustment Toggle, which can be one possible way to select audio signal conditioning. Alternatively, as discussed above, the user 2601 may be provided with a suitable interface for manually determining whether to apply the first selection adjustment or the second selection adjustment. For example, user 2601 may select a desired mode of operation by operating a user interface, such as displayed on a device (such as a smartphone, laptop, etc.) coupled to apparatus 110, for example.

As discussed, the processor 2641 (or the user 2601 via a suitable interface) may be configured to distinguish between a noisy environment or a calm environment. However, such an environment classification is only one possible example, and any other suitable classification may be used, which may result in a differentiation of modes of operation of the device 110 .

FIG. 27A schematically illustrates a process 2701 for determining the operating mode of the device 110 . For example, at step 2711 of process 2701, processor 2641 (or a related device such as a smartphone) may be configured to determine the type of environment of a user (eg, user 2601). At step 2713, the processor 2641 may evaluate whether the environment type corresponds to one of a number of possible types (eg, the environment may be evaluated as a Type A, B, or C environment), and based on the type of environment, may choose to have Corresponding operating modes corresponding to the selective adjustments (eg, operating modes A-C with corresponding selective adjustments 2715A-C may be selected).

FIG. 27B illustrates an example process 2702 for sending selectively conditioned audio signals to auditory interface device 2615, consistent with the disclosed embodiments. At step 2721 of process 2702, processor 2641 of device 110 is configured to receive images/videos of the environment of a user (eg, user 2601). Audio signals from the user's 2601 environment may also be received by the processor 2641 at step 2723. In various embodiments, the audio signal from the environment may include all audio sounds detected in user 2601's environment, such as individual speech and ambient noise in user 2601's environment. At step 2725 of process 2702, processor 2641 may be configured to analyze the audio and image data collected in steps 2711 and 2713. At step 2725, the processor 2641 may be configured to operate in the first mode as described above to cause the first selective conditioning of the first audio signal. At step 2727, the processor 2641 may be configured to determine, based on the image analysis or the audio analysis, to switch to the second mode to cause a second selective adjustment to the first audio signal, the second selective adjustment being different in at least one respect The first selective adjustment. For example, as described above, when the environment of the user 2601 is noisy, the processor 2641 may be configured to provide stronger conditioning of the first audio signal (eg, increasing the amplitude of the first audio signal).

At step 2729, the processor 2641 may be configured to send the conditioned signal to the auditory interface device 2615. Note that in some embodiments of process 2702, only the conditioned signal is sent to device 2615. In other embodiments, both the conditioned signal and other unconditioned signals may be sent to the device 2615.

Consistent with the disclosed embodiments, the mode of operation (eg, the first mode of operation or the second mode of operation) to apply may be determined by identifying whether a speaking individual is present in the environment of the user 2601 . For example, if the individual is detected, the mode of operation may be switched from a first mode of operation to a second mode of operation for which a second selective adjustment may be used and may include reducing ambient noise and amplifying noise from the speaking individual audio signal. In an exemplary embodiment, background (ie, ambient) noise may constitute the first audio signal, and the second selective adjustment may reduce such first audio signal. In some cases, the second selective adjustment may include changing the pitch of one of the audio signals (eg, the first audio signal), changing the amplitude of the audio signal, or time-stretching the audio signal.

As described above, the second selective adjustment may be applied in conjunction with the first selective adjustment. For example, the second selective adjustment can be applied to the second audio signal and the first selective adjustment can be applied to the first audio signal. For example, if the first audio signal is the individual's speech and the second audio signal is background noise, the first selective adjustment may include amplifying the volume of the speech, and the second selective adjustment may include reducing the amplitude (or modifying) the background noise the pitch of the background noise so that it is not easily confused with the first audio signal). In some cases, the second selective adjustment may include attenuating at least one second audio signal of the plurality of audio signals relative to the first audio signal. Additionally or alternatively, one of the first or second selective adjustment may include attenuating the pitch of at least one second audio signal of the plurality of audio signals relative to the first audio signal. In an example embodiment, device 110 may use the first selective adjustment when individuals in user 2601's environment are not speaking. However, when the individual begins to speak, the system 2650 can automatically switch to the second selective adjustment.

In some cases, the first audio signal may be associated with the speech of the first individual and the second audio signal may be associated with the speech of the second individual. Additionally, in some embodiments, the first audio signal may be associated with a first group of individuals and the second audio signal may be associated with a second group of individuals.

As previously mentioned, the user 2601 may select a particular selective adjustment by providing instructions to the device 110 . In an example embodiment, the instructions may include selecting a number of individuals that may generate a subset of audio signals from multiple audio signals in the environment of the user 2601, and requesting the use of a particular type of selective adjustment (eg, a second selective adjustment) to selectively adjust a subset of the audio signal. In some cases, a particular type of selective adjustment (eg, a second selective adjustment) may be performed by first determining the speech audio signal of each speaker from the plurality of audio signals, and for a given point in time, determining that the audio signal has less than a threshold value The difference signal is a pair of speech audio signals to adjust at least some of the audio signals in the environment of the user 2601 at each point in time. As previously mentioned, the signal difference can be measured using a suitable metric. For example, the difference can be measured by measuring the difference in pitch, amplitude, rhythm, time stretch, or any other suitable parameter that can be used to characterize an audio signal. The second selective adjustment may then include amplifying the signal difference above a threshold by changing the base height, amplitude or duration of one of the speech audio signals from the pair of speech audio signals. An example of such a process is schematically illustrated by Figures 28A and 28B using corresponding processes 2801 and 2802. During process 2801, composite audio signal 2811 (which may contain overlapping dialogue) may be decomposed into individual audio signals 2821-2825 using any suitable method as described further herein. In some cases, these separate audio signals (eg, signals 2821 and 2823) may overlap and may be similar for some time interval indicated by time domains 2815 and 2817 (eg, the signals may be similar enough that they may be User 2601 confused). For this case, the selective adjustment may also include further distinguishing the signals 2821 and 2823 at least for the time domains 2815 and 2817 . As shown in process 2802 in Figure 28B, to differentiate between signals 2821 and 2823, at least one of signals 2821 and 2823 may be altered via computer-based application 2830 to amplify the difference measured using a suitable metric (eg, Amplify the difference so that it is higher than the threshold difference). For example, as shown in FIG. 28B , signal 2817 may be altered (eg, time stretched) to yield signal 2837 , while signal 2815 may be time compressed and its amplitude increased to yield signal 2835 . In an example embodiment, the difference between signal 2835 and signal 2837 is above a threshold, making it easy for user 2601 to distinguish these modified signals.

In some embodiments, particular modes of operation may be selected to optimize resource usage of device 110 . For example, if there is only one person in the vicinity of user 2601, device 110 may be configured to switch to a single-speaker mode of operation (eg, a single-speaker mode of operation may not require determining the active speaker in the speaker's group, so, The power consumption of the device 110 is reduced, which might otherwise be associated with determining the amplitude ratio between different speakers). Various other modes of operation can be used to reduce power consumption and extend battery life of the device 110 (eg, modes of operation that do not rely heavily on wireless communication with peripherals such as smartphones, laptops, etc., to collect only per minute Several image-featured operating modes, operating modes that do not require any audio analysis, etc.).

The processor 2641 of the hearing aid system (eg, the hearing aid system may include apparatus 110, auditory interface device 2615, cameras 2617A-2617B, and microphone 2613) may be based on analysis of multiple images in the environment of user 2601 or based on the environment of user 2601 Analysis of the plurality of audio signals in to determine switching to the second mode (as described above). For example, if an individual in the environment of the user 2601 is speaking, the processor 2641 may switch to the second mode.

In an example embodiment, the processor 2641 may operate in the first mode to cause the first selective adjustment. The first selective adjustment (as described above) includes amplification of the first audio signal. Additionally, processor 2641 may determine, based on analysis of at least one of the plurality of images or the plurality of audio signals, to switch to the second mode to cause a second selective adjustment of the first audio signal, the second selective adjustment relative to Different from the first selective adjustment in at least one respect. In an example embodiment, the second selective adjustment may include attenuating at least one second audio signal of the plurality of audio signals relative to the first audio signal.

In an example embodiment, the first audio signal may be associated with the speech of the first individual and the second audio signal may be associated with the speech of the second individual.

In an example embodiment, the processor 2641 may be configured to determine to switch to the second mode based on a context associated with at least one of the plurality of images or the plurality of audio signals.

In an example embodiment, the processor 2641 may operate under active mode control. In this mode, the processor 2641 can automatically switch between the first mode and the second mode, for example. In active mode, the processor 2641 may control, among other things, several speakers whose audio is sent to the user. For example, if a user is attending a lecture, he or she may only want to hear the speaker and not background noise, or other people's speech, and so on. However, if the user is at a social event and talking to many people, the user may want to hear each of them as they speak. The user may or may not want to hear himself or herself, or may want to hear himself or herself at a lower amplitude. When capturing other audio, the user may want to hear or not hear background noise, but when no other audio is being captured, the user may want to hear background noise of some amplitude, and so on. In some embodiments, the operating mode of the processor 2641 may vary according to user selection. A user may select a desired mode of operation by operating a user interface, such as displayed on a device (such as device 110, smartphone, laptop, etc.) coupled to the auditory interface device. In other embodiments, the mode of operation may change automatically depending on the user and the context (eg, environment) of the hearing aid system. For example, certain events may cause the processor 2641 to assume a certain mode of operation (eg, a first mode or a second mode). For example, when entering a venue identified as a lecture room, the processor 2641 may switch to a mode in which only one speaker is sent, and when leaving the room, the processor 2641 may switch to sending multiple speakers based on active speakers mode of speech. In an example embodiment, the background indicates that the user is entering the room. Alternatively, the background instructs the user to leave the room.

In an example embodiment, the processor 2641 may be configured to select the first mode or the second mode based on a context associated with at least one of the plurality of images or the plurality of audio signals. The context indicates that the user is attending a speech or that the user is attending a social event. The context may indicate that the user is entering a lecture room, wherein the at least one audio signal includes a speaker's speech, and wherein the first selective adjustment includes amplifying a first audio signal of the plurality of audio signals and attenuating a first audio signal of the plurality of audio signals at least another signal (such as background noise). The context may instruct the user to leave the lecture room, wherein the at least one audio signal includes the speech of an active speaker outside the lecture room, and wherein the first selective adjustment includes amplification of the speech. In some cases, the context may indicate that the user is only within a predetermined distance of a person, and wherein the at least one audio signal includes the person's speech, and wherein the first selective adjustment includes amplification of the speech. The predetermined distance may be any suitable distance (eg, a distance in the range of 0.1 meters to 5 meters). In some cases, the predetermined distance may be greater than five meters (eg, 10 meters).

In various embodiments, as described above, the first selective adjustment includes changing the amplitude of the at least one audio signal. Additionally or alternatively, the first selective adjustment includes changing the pitch of the at least one audio signal. In some cases, as previously described, the first selective adjustment includes time-stretching the at least one audio signal.

In some embodiments, the processor 2641 may be configured to select the first mode or the second mode based on a selection received from a user. In an example embodiment, the selection is received from an electronic device such as a smartphone, laptop, smart watch, bracelet, or any other suitable wearable electronic device.

Custom filter for acquaintances

According to embodiments of the present disclosure, a hearing aid system for selectively adjusting sound may include a wearable camera (or multiple wearable cameras) configured to capture multiple images from a user's environment as described herein. In various embodiments, the hearing aid system may include one or more microphones configured to capture sound from the user's environment as described herein. As described below with respect to FIG. 26 , the hearing aid system may include apparatus 110 and processor 2641 .

In an example embodiment, the processor 2641 may be configured to receive a plurality of images captured by a camera (eg, camera 2617A or 2617B as shown in FIG. 26). Additionally, the processor 2641 may be configured to receive a plurality of audio signals representing sounds captured by the at least one microphone from the user's environment as described herein. The processor 2641 may be configured to identify an individual identified by at least one of the plurality of images or at least one represented by at least one of the plurality of audio signals. In an example embodiment, the processor 2641 of the device 110 may analyze one or more captured images. For example, processor 2641 may be configured to receive various images or characteristics of a person captured by camera 2617A or camera 2617B. Additionally, the apparatus 110 may be configured to communicate with a server that may store images of various objects and/or people. In an example embodiment, the apparatus 110 may upload or download images from the server. Furthermore, the device 110 may perform a search for images (or videos) stored at the server.

Similar to the embodiments discussed above, the processor 2641 may analyze and identify objects using computer-based models as described herein. In some cases, facial features can be analyzed by suitable computer-based models. For example, a computer-based model can be used to analyze images and compare facial features or relationships between facial features of persons identified in captured images with facial features of persons found in images stored in the server's database or Compare the relationship between the two. In some embodiments, a video of the dynamic movement of a person's face may be compared to video data records of various persons obtained from a database in order to determine that the person captured in the video is an identified individual.

29 shows user 100 with device 110, which may include camera 2617A and microphone 2613. In an example embodiment, user 100 may face individual 2911 and individual 2912 that produce respective audio signals 2921 and 2922 detected by microphone 2613 . Images of individual 2911 and individual 2912 can be detected by camera 2617A. Additionally, the microphone 2613 can detect audio signals 2923 (eg, voices from objects or people not visible to the user 100). Using image data to identify individuals (eg, identify individual 2911 ) may be one possible approach. Alternatively, the individual 2911 may be identified based on speech detected in the audio signal 2921. In an example embodiment, as discussed above, the voiceprint of individual 2911 may be detected. For example, processor 2641 may determine that sound 2921 corresponds to the speech of individual 2911. This may be performed using speech recognition software such as hidden Markov models, dynamic time warping, neural networks, or other techniques (eg, such speech recognition software may be executed by processor 2641). In some cases, processor 2641 may be configured to upload audio signals (eg, 2921 and 2922) to a server, and the server may be configured to further use the voiceprint of individual 2911 by isolating audio signal 2921 from audio signal 2922 to determine that signals 2921 belong to individuals 2911 to process these signals. For speech recognition, the server may access a database (eg, database 2050 as shown in FIG. 20B ), which may also include the voiceprints of one or more individuals. After determining that the audio signal 2921 matches the individual 2911 based on, for example, the voiceprint of the individual 2911, the hearing aid system may identify the individual 2911 as an identified individual.

This recognition process can be used alone or in combination with the image recognition techniques described above (eg, facial recognition techniques). For example, facial recognition technology can be used to identify individual 2911 and speech recognition can be used to authenticate individual 2010, and vice versa. In some cases, video of an individual speaking can be used for identification of the individual. For example, synchronization of facial features (eg, lip movement of individual 2911) with audio signal 2921 may be used to identify individual 2911 as a speaker. The voiceprints extracted for that speaker can then be used to separate the voice associated with the individual 2911 from other voices represented in the audio signal 2921. In some embodiments, additional processing such as identifying words, or other forms of processing described throughout this disclosure, may be performed on the audio uttered by the individual 2911. In various embodiments, the processor 2641 may identify one or more individuals. For example, processor 2641 may be configured to identify individual 2911 and individual 2912.

In some embodiments, device 110 may detect speech of individuals not within the field of view of device 110 as shown in FIG. 29 as audio signal 2923. For example, speech can be heard on a speakerphone, from the back seat of a vehicle, or the like. In such an embodiment, in the absence of a speaker in the field of view, the identification of the individual may be based solely on the individual's speech.

A processor of the hearing aid system (eg, processor 2641) may be configured to retrieve from memory an adjustment profile associated with the at least one identified individual. The conditioning profile may be any suitable set of instructions executable by the processor 2641 for selectively conditioning the audio signal. Selective conditioning may include amplification or attenuation of the audio signal, removing noise from the audio signal (eg, suppressing some frequencies identified in the audio signal), and the like. In some cases, some portions of the audio signal may be amplified (eg, the audio signal corresponding to individual 2911's speech may be amplified), while other portions of the audio signal (eg, background music or individual 2912's speech) may be suppressed . In some cases, the processor 2641 may be configured to analyze the speech of the individual 2911 and recognize words within the speech. If a word cannot be clearly identified in a portion of the audio signal 2921 (eg, the processor 2641 determines that the probability of correctly identifying a word in the portion is low), the processor 2641 may be configured to selectively adjust the portion ( For example, amplify that part of the audio signal). In some cases, processor 2641 may be configured to selectively condition portions of audio signal 2921 for which words cannot be recognized, and then repeatedly attempt to zoom in to recognize words in that portion. Such iterations may be performed multiple times to optimize the selective conditioning of the audio signal 2921.

The adjustment profile may allow for the audio signal to be selectively adjusted such that, for example, the speech of the individual 2911 is modified to improve the intelligibility of the speech. In an example embodiment, speech rendering may be used to remove a speaker's accent or reapply a different accent to the speech. In an example embodiment, both the original audio signal of the human speech may be combined (eg, morphed) with the rendered speech in order to preserve some of the speaker's natural characteristics while maintaining clear speech. The speech rendering may include changing the pitch of the person speaking (eg, such rendering may be beneficial if the user 100 has difficulty recognizing a particular frequency), the cadence of the human speech, the loudness of the human speech, or any other aspect of the human speech characteristics (eg, filters can be applied to human speech to change the human voice from male to female). In addition, selective adjustments may be used for any suitable modification of background sounds unrelated to the individual's 2911 speech.

In various embodiments, the conditioning profile may include information for selectively conditioning the audio signal. In some cases, the instructions corresponding to the adjustment profile may include any suitable logical elements. For example, an IF logic clause may be used in the adjustment profile when the selective adjustment is subject to a particular characteristic (eg, pitch or loudness) observed for a portion of the audio signal (eg, within audio signal 2921). After performing the selective adjustment, the processor 2641 may be configured to cause transmission of the adjusted first audio signal to an auditory interface device configured to provide sound to the ear of a user (eg, user 100). In an example embodiment, the conditioning profile may include predefined filters for selectively conditioning the audio signal (eg, the audio signal 2921 ) based on at least one of a frequency rate or an amplitude of the audio signal 2921 .

In an example embodiment, the memory may be located within the same housing as the wearable camera (ie, the memory may be local memory). The memory may be any suitable memory (eg, solid state memory, hard drive, etc.). In some cases, at least a portion of the adjustment configuration file may be stored in local storage, while another portion may be stored in remote storage (eg, in a remote database). In some cases, adjustment profiles may be selected and retrieved from a remote database based on the identified individual.

In an example embodiment, the hearing aid system's processor 2641 is also programmed to determine at least one modification to the selective adjustment associated with the identified individual (eg, individual 2911), and to update the adjustment profile based on the modification. For example, determining at least one modification may include determining that amplification of the audio signal 2921 needs to be performed based on how well the user 100 can hear the individual 2911 (or how well the user 100 can discern the words of the individual 2911). In an example embodiment, the user 100 may provide feedback to the processor 2641 using any suitable means (eg, via an audio signal or via a suitable interface for a hearing aid system as discussed herein). The interface of the hearing aid system may include buttons on the apparatus 110, or may be, for example, an application displayed on a device coupled to the hearing aid system (such as a smartphone, laptop, etc.). The user 100 may provide specific instructions to the hearing aid system (eg, the user 100 may request to increase the amplitude of the audio signal 2921 from the individual 2911) or may provide more complex instructions (eg, increase the intelligibility of the individual's 2911 speech, increase the intelligibility from the camera 2617A) the clarity of the received audio signal at the point pointed to, or suppress the background sound). Such complex instructions can be interpreted by the hearing aid system and modified accordingly. In some cases, an instruction may be selected from a list of possible modifications. Alternatively, when instructions are provided via voice commands from user 100, such instructions may be analyzed by a language processing application of device 110, and modifications corresponding to the instructions may be made. In one example embodiment, the language processing application of device 110 may be any suitable software application capable of transcribing human speech and determining instructions for modifying audio signals from the resulting transcribed text.

Additionally, the modification determined by the processor 2641 may be based on various environmental factors (presence of noise, background music, other speakers), and may be done automatically without receiving an indication from the user 100 . In each case, the adjustment profile may be updated as appropriate based on the determined modifications. In some cases, the hearing aid system may need to receive approval from the user before updating the adjustment profile. Once the modifications are determined, the processor 2641 may be configured to apply the modifications.

Modifications may include adjustments (ie, instructions) by the user, as previously described. Adjustments can be made via a suitable interface of the hearing aid system. As described, at least one modification may be determined based on an indication of hearing difficulty from the user. The indication of difficulty may be determined via one of an audio signal from the user or an input from the user received via an interface of the hearing aid system. In an example embodiment, the at least one modification includes one of amplification of the audio signal (eg, signal 2921 ) or modification of the frequency spectrum of the audio signal 2921 . In some cases, the selective adjustment includes attenuating at least another audio signal (eg, audio signal 2922 or signal 2923 ) not associated with the identified individual 2921 .

In an example embodiment, the processor 2641 of the hearing aid system may be programmed to identify another identified individual represented by at least one of the plurality of images or by at least one of the plurality of audio signals. For example, processor 2641 may be configured to identify individual 2912. In some cases, the processor 2641 may be configured to identify the individual 2912 when the user 100 points the camera 2617A at the individual 2912. In some cases, when device 110 includes two cameras (eg, cameras 2617A and 2617B as shown in FIG. 26 ), camera 2617A may be configured to identify individual 2911 and camera 2617B may be configured to identify individual 2912. Once the individual 2912 is identified (identification of individuals using image or audio data is discussed above), the processor 2641 may retrieve another conditioning profile associated with another identified individual from memory. For example, if device 110 includes local memory, processor 2641 may retrieve the conditioning profile associated with individual 2912 from the local memory. In some cases, the conditioning profile associated with individual 2912 (eg, CP2912) may be the same as the conditioning profile associated with individual 2911 (eg, CP2911). Alternatively, CP2912 may be different from CP2911. The processor 2641 can selectively condition the audio signal 2921 using the CP2911 and can selectively condition the audio signal 2922 using the CP2922. In some cases, CP2911 and CP2922 can be applied to a combined audio signal comprising signals 2921 and 2922 when signals 2921 and 2922 cannot be clearly separated.

In an example embodiment, as shown in Figure 26, an audio signal 2921 conditioned using CP2911 may be sent to an auditory interface device (eg, auditory interface device 2615), and an audio signal 2922 conditioned using CP2912 may also be sent to an auditory interface Device 2615. In an exemplary embodiment, processor 2641 may separate audio signals 2921 and 2922 in time. For example, when audio signal 2921 and signal 2922 are emitted at about the same time, processor 2641 may separate them by a sufficient amount of time for user 100 to better discern each signal. In some cases, when auditory interface device 2615 sends audio signals to both ears of user 100, the first selectively conditioned audio signal (eg, selectively conditioned signal 2921 ) may be sent to the left ear (or right ear), and a second selectively conditioned audio signal (eg, selectively conditioned signal 2922) may be sent to the right ear (or left ear).

In some cases, the hearing aid system may be configured to identify groups of individuals (rather than a single identified individual) and to selectively condition sounds from such groups. For example, camera 2617A can capture an image of a group of people, and processor 2641 can be configured to identify the group. In one example embodiment, there may be an adjustment profile for the group (as opposed to an adjustment profile for a single individual). For example, if the processor 2641 identifies that an audio signal is received from a group of school children in a classroom, the adjustment profile may include instructions to reduce the amplitude of the audio signal emitted by each member of the group.

30A illustrates an example process 3001 for selectively conditioning and sending audio signals to a user's ear. At step 3011 of process 3001, processor 2641 may receive images captured by a camera (eg, camera 2617A) associated with the hearing aid system. At step 3013, the processor 2641 may receive audio signals from a microphone (eg, microphone 2613) associated with the hearing aid system. At step 3015, processor 2641 may use any suitable image recognition technique to identify the individual (eg, individual 2911) represented by the image captured using camera 2617A, as described above. In some embodiments, individuals may be identified based on their voiceprints. For example, voiceprints may be obtained during periods of time during which only the individual is speaking, or in other parts of the audio signal where the individual's voiceprint may be obtained. The voiceprint can then be compared to multiple voiceprints pre-stored in a database to identify the individual. At step 3017, the processor 2641 may retrieve the adjustment profile from memory (eg, local memory associated with the hearing aid system or remote memory associated with a suitable cloud computing resource) as described above. At step 3019, the processor 2641 may selectively adjust the audio signal received from the environment of the user (eg, user 100 as described in FIG. 29) based on the retrieved adjustment profile. An audio signal (eg, audio signal 2921 ) may be received from the identified individual 2911 . In some cases, user 100 may point camera 2617A at recognized individual 2911 and microphone 2613. Process 3001 may end at step 3021, where processor 2641 may send, via auditory interface device 2615 (eg, shown in FIG. 26), an audio signal that is selectively adjusted using instructions obtained from the adjustment profile to The ear of the user 100 .

30B illustrates an example process 3002 for selectively conditioning and sending audio signals to a user's ear. Process 3002 is a variation of process 3001. For example, process 3002 may include steps 3011 and 3013 as discussed above in connection with Figure 30A. At step 3025, process 3002 may differ from process 3001. At step 3025, the processor 2641 may not use the image data to identify the individual (as described in step 3015), but instead use the audio data received from the individual to identify the individual. For example, audio data may be identified via an individual's voiceprint. As mentioned above, for example, voiceprints can be obtained during periods of time when only the individual is speaking. This voiceprint can then be compared to voiceprints pre-stored in a database to identify the individual. In some embodiments, it should be understood that steps 3015 and 3025 may be performed concurrently by processor 2641 as described above. Process 3002 may also include steps 3017, 3019, and 3021 as discussed above in connection with Figure 30A.

Figure 31 shows an example process 3101 for selectively conditioning and sending audio signals to a user's ear. Process 3101 is a variation of process 3001 or process 3002. For example, process 3101 may include steps 3011 and 3013 as discussed above in connection with Figure 30A. At steps 3115 and 3117, process 3101 may be different from process 3001 or 3002. At step 3115, the processor 2641 may identify the group of individuals represented by the image captured by the camera 2617A or the audio captured by the microphone 2613 as described above. At step 3117, the processor 2641A may retrieve the conditioning profile corresponding to the identified group from memory as described above. Process 3101 may also include steps 3019 and 3021 as discussed above in connection with Figure 30A.

Selective adjustments based on user prompts

Hearing aid systems are designed to improve and enhance the user's interaction with the environment. Users may rely on hearing aid systems to navigate their surroundings and daily activities. However, different users may require different levels of assistance depending on the circumstances. Typical hearing aid systems may not be able to adequately correct or adjust the audio signal based on the needs of the user. Accordingly, there is a need for an apparatus and method for automatically adjusting a user's audio signal based on prompts from the user.

The disclosed embodiments include hearing aid systems that may be configured to correct or adjust audio signals based on cues from a hearing aid user. For example, the prompt may be physical (eg, the user tilts or turns the ear to a sound source or speech, the user raises his or her hand to the ear, etc.) or verbal (eg, the user may state "What?" or "Repeat" ” etc.), cues can be collected by sensors and/or microphones on the wearable camera device. Based on the detected cues, the system can identify the user's hearing difficulties (eg, understand the individual speaking, etc.) and automatically correct or adjust at least one audio signal. For example, the system can selectively amplify sound from a sound source.

32 is a schematic diagram illustrating an exemplary environment for using a hearing aid with speech and/or image recognition consistent with the disclosed embodiments. A wearable camera (eg, the wearable camera of device 110 ) may be configured to capture multiple images from the environment of user 100 . Alternatively or additionally, the at least one microphone may be configured to capture sound from the environment of the user 100 .

For example, the processor 210 may receive at least one audio signal 3203 , 3205 or 3207 representing one or more sounds captured by the at least one microphone 1750 from the environment of the user 100 . In some embodiments, processor 210 may identify at least one action of user 100 based on analysis of at least one audio signal (eg, audio signal 3203). In some embodiments, the at least one action may include speech by the user 100 . For example, identifying the at least one action may include detecting words spoken by the user 100 based on analysis of the at least one audio signal 3203 . In some embodiments, the words may indicate that the user 100 did not hear clearly. For example, user 100 may say "What?" or "Can you repeat what you said?" or "I don't understand." In some embodiments, processor 210 may associate greater hearing difficulties with user 100 based on the type and/or frequency of detected words. For example, "repetition" may be associated with greater hearing difficulties than "I don't understand", and/or repeated words (eg, user 100 repeating the statement "what?") may be associated with greater hearing than unrepeated words difficulty associated.

In some embodiments, the microphone 1720 may be configured to determine the directionality of sound in the environment of the user 100 . For example, microphone 1720 may include one or more directional microphones, which may be more sensitive to picking up sounds in certain directions. The processor 210 may be configured to differentiate sounds within the environment of the user 100 and determine the approximate directionality of each sound. For example, using the microphone array 1720 , the processor 210 may compare the relative timing or amplitude of sounds between the microphones 1720 to determine the directionality relative to the device 100 .

In some embodiments, the sounds captured from the environment of the user 100 may be classified into segments containing speech, music, tones, laughter, screams, and the like. The indication of each segment may be recorded in database 2050.

In some embodiments, the recorded information may enable processor 210 to identify at least one action of user 100 based on analysis of at least one audio signal 3203 . As previously discussed, the at least one action may include speech by the user 100 . For example, identifying the at least one action may include detecting words spoken by the user 100 based on analysis of the at least one audio signal 3203 . The words may indicate that the user 100 is not hearing the sound well (eg, from another entity 3210, the sound associated with the audio signal 3205, the sound associated with the audio signal 3207, etc.).

Processor 210 may be configured to selectively adjust at least one audio signal (eg, from individual 3210, audio signal 3205, audio signal 3207, etc.) in the environment of user 100 received by the at least one microphone based on the identified action . The at least one conditioned audio signal can be sent to auditory interface device 1710, which is configured to provide sound to the ear of user 100, and thus can provide user 100 with a source corresponding to the at least one audio signal (eg, auditory feedback of individual 3210 associated with audio signal 3205, individual 3210 associated with audio signal 3207, etc.). The processor 210 may perform various conditioning techniques on the audio signal received from the microphone 1720 . Adjusting may include amplifying audio signals determined to correspond to sounds (eg, individual 3210 associated with audio signal 3205, individual 3210 associated with audio signal 3207, etc.) relative to other audio signals. Amplification may be accomplished digitally, for example, by processing the audio signal associated with the individual 3210 relative to other audio signals 3205 or 3207. Amplification may also be accomplished by changing one or more parameters of the microphone 1720 to focus on audio sounds emanating from the individual 3210 associated with the user 100 (eg, a region of interest). For example, microphone 1720 may be a directional microphone, and processor 210 may perform operations to focus microphone 1720 on individual 3210 or other sounds in the environment of user 100 . Various other techniques for amplifying sound may be used, such as the use of beamforming microphone arrays, acoustic telescope techniques, and the like. In some embodiments, the selective adjustment of the at least one audio signal may be implemented based on the determined hearing difficulty of the user 100 . For example, the amplification of the audio signal may increase as the hearing difficulty associated with the user 100 increases.

Conditioning may also include attenuating or suppressing one or more audio signals received from directions other than the region of interest (eg, individual 3210). For example, processor 210 may attenuate audio signals 3205 and 3207. Similar to the amplification of sound from the individual 3210, attenuation of the sound can occur by processing the audio signal, or by changing one or more parameters associated with the one or more microphones 1720 to direct focus away from the area containing the individual 3210. sound from outside. In some embodiments, the attenuation of audio signals received from outside the region of interest may be increased if the user 100 has determined through analysis of past interactions to have a predetermined level of hearing loss.

In some embodiments, adjusting may also include changing the pitch of the audio signal corresponding to the sound from the region of interest to make it easier for the user 100 to perceive the sound. For example, the user 100 may have less sensitivity to tones within a certain range, and the adjustment of the audio signal may adjust the pitch of the sound from the area of interest to make it more perceptible to the user 100 . For example, user 100 may experience hearing loss in frequencies above 10 kHz. Thus, the processor 210 may remap higher frequencies (eg, at 15khz) to frequencies below 10khz. In some embodiments, the processor 210 may be configured to vary the speech rate associated with one or more audio signals. Accordingly, processor 210 may be configured to detect speech within one or more audio signals received by microphone 1720, eg, using a voice activity detection (VAD) algorithm or technique. For example, if it is determined that the sound corresponds to speech or speech from the individual 3210, the processor 210 may be configured to change the playback rate of the sound from the individual 3210. For example, the speaking rate of the individual 3210 may be reduced to make the detected speech more perceptible to the user 100 . Various other processing may be performed, such as modifying the pitch of the sound from the individual 3210, to maintain the same pitch as the original audio signal, or to reduce noise within the audio signal. If speech recognition has been performed on the audio signal associated with the sound from the individual 3210, the conditioning may also include modifying the audio signal based on the detected speech. In some embodiments, adjusting may include modifying the detected rate of speech. For example, the rate of speech can be modified by increasing the duration of words included in the audio signal and reducing the duration of pauses between words (or vice versa), which can make speech easier to understand.

The conditioned audio signal may then be sent to auditory interface device 1710 and an audio signal generated for user 100 . Thus, in the conditioned audio signal, the sound from the individual 3210 may be more easily heard by the user 100, louder and/or more distinguishable than the sound from the audio signal 3205 or 3207, the sound from the audio signal 3205 or 3207 Can represent background noise within the environment.

In some embodiments, at least one microphone 1750 can capture one or more audio signals received during a moving time window of predetermined length, and processor 210 can be programmed to cause selective adjustment and Transmission of part of an audio signal. For example, processor 210 may store a portion of at least one audio signal (eg, from individual 3210) in database 2050, where the portion was received prior to at least one action by user 100 (eg, emitting audio signal 3203). The portion of the at least one audio signal may be sent to the auditory interface device 1710 configured to provide sound to the ear of the user 100, and thus may provide the user 100 with a source corresponding to the portion of the at least one audio signal (eg, another user ) auditory feedback. The portion of the at least one audio signal may be sent to the hearing device 1710 based on at least one action of the user 100 . For example, after user 100 indicates that he or she has difficulty hearing one or more sounds, the portion may be sent to hearing device 1710, thereby replicating at least one sound that user 100 has difficulty hearing. Periods missed due to playback of previous sound periods may then be provided to user 100 at an increased rate, such as by reducing periods of silence between words, or in any other suitable manner. In some embodiments, automatic playback of the sound may be performed based on gestures performed by the user rather than verbal instructions from the user. For example, if the user verbally states that he or she has difficulty understanding, another entity may be prompted to repeat the words.

33 is an exemplary depiction of a user 100 having a hearing aid system consistent with the disclosed embodiments. In some embodiments, the wearable camera may be a component of the device 110 (eg, a camera-based directional hearing aid device) for selectively selectively based on the motion 3201 of the user 100 (eg, hand motion, tilt motion, gaze direction, etc.) Change the amplification of the sound. User 100 may also wear auditory interface device 1710, for example. In some embodiments, device 110 may capture at least one image from the environment of user 100 . In some embodiments, the processor 210 may receive at least one image captured by the device 110 . The processor 210 may identify the at least one action by detecting the motion 3201 of the user 100 based on the analysis of the at least one image. In some embodiments, motion 3201 may include hand motion of user 100 or tilt motion of user 100 . For example, the user 100 may wrap his hands around the ears, indicating that the user 100 is having difficulty hearing the individual 3210. In some embodiments, the user 100 may tilt towards the individual 3210, indicating that the user 100 is having difficulty hearing the individual 3210.

In some embodiments, this can be done by monitoring the orientation of the user's 100 body part (eg, hand, arm, etc.) or face part (eg, nose, eyes, ears, hand near the ear, etc.) relative to the optical axis of the camera sensor Movement 3201 of user 100 is tracked. For example, the wearable camera of device 110 may be configured to capture one or more images of user 100's surroundings, eg, using image sensor 220 . For example, the captured images may include representations of body parts or facial parts of the user 100 , which may be used to determine the hand movements of the user 100 . Processor 210 (and/or processors 210a and 210b) may be configured to use various image detection or processing algorithms (eg, using convolutional neural networks (CNN), scale-invariant feature transforms (SIFT), histograms of oriented gradients) (HOG) feature or other techniques) to analyze the captured image and detect movement 3201 of the body part or face part of the user 100 . Based on the detected representation of the body part or face part of the user 100, the motion 3201 of the user 100 can be determined.

Motion 3201 may be determined in part by comparing a detected representation of the body part or face part of user 100 to the optical axis of camera sensor 1751 . For example, the optical axis 1751 may be known or fixed in each image, and the processor 210 may determine the motion 1750 by comparing a representative angle of the body part or face part of the user 100 to the direction of the optical axis 1751 . For example, the determined motion may include the user 100 wrapping his hand around the ear, indicating that the user 100 is having difficulty hearing the individual 3210. In some embodiments, the determined motion may include the user 100 leaning toward the individual 3210, indicating that they are having difficulty hearing the individual 3210. For example, a tilting motion of the user 100 towards a sound emitting object may be identified by a decrease in the distance between the user 100 and the object. In some embodiments, tilt motion may be detected based on comparing the reduction to a predetermined threshold or range (eg, 5-30 centimeters). This distance can be assessed by analyzing one or more images, based on a rangefinder embedded within the device 110, or various other methods.

In some embodiments, processor 210 may cause selective adjustment of at least one audio signal received by at least one microphone 1720 (eg, from individual 3210 ) based on the identified action and cause the at least one adjusted audio signal to be directed toward Transmission of auditory interface device 1710 configured to provide sound to user 100 ears. In some embodiments, user 110 may tilt in a particular direction (eg, tilt toward an individual in the environment of user 110), and causing selective adjustment of the at least one audio signal may include amplifying at least one received from the direction of tilt motion audio signal.

The at least one conditioned audio signal can be sent to auditory interface device 1710, which is configured to provide sound to the ear of user 100, and thus can provide user 100 with a source corresponding to the at least one audio signal (eg, auditory feedback from individual 3210). The processor 210 may perform various conditioning techniques on the audio signal received from the microphone 1720 . Adjusting may include amplifying the audio signal determined to correspond to the sound from the individual 3210 relative to other audio signals (eg, audio signal 3205 or 3207). Amplification may be accomplished digitally, for example, by digitally processing the audio signal associated with the individual 3210 relative to other signals. Amplification may also be accomplished by changing one or more parameters of the microphone 1720 to focus on audio sounds emanating from the individual 3210 associated with the user 100 (eg, a region of interest). For example, microphone 1720 may be a directional microphone and processor 210 may perform operations to focus microphone 1720 on sound 1820 or other sounds within the area of individual 3210. Various other techniques for amplifying sound from the individual 3210 may be used, such as the use of beamforming microphone arrays, acoustic telescope techniques, and the like.

Conditioning may also include attenuating or suppressing one or more audio signals received from directions other than the region of interest (eg, individual 3210). For example, processor 210 may attenuate audio signals 3205 and 3207. Similar to the amplification of sound from the individual 3210, attenuation of the sound can occur by processing the audio signal, or by changing one or more parameters associated with the one or more microphones 1720 to direct focus away from the area containing the individual 3210. sound from outside.

In some embodiments, adjusting may also include changing the pitch of the audio signal corresponding to the sound from the region of interest to make it easier for the user 100 to perceive the sound. For example, the user 100 may have less sensitivity to tones within a certain range, and the adjustment of the audio signal may adjust the pitch of the sound from the area of interest to make it more perceptible to the user 100 . For example, user 100 may experience hearing loss in frequencies above 10 kHz. Thus, the processor 210 may remap higher frequencies (eg, at 15khz) to frequencies below 10khz. In some embodiments, the processor 210 may be configured to vary the speech rate associated with one or more audio signals. Accordingly, processor 210 may be configured to detect speech within one or more audio signals received by microphone 1720, eg, using a voice activity detection (VAD) algorithm or technique. For example, if it is determined that the sound corresponds to speech or speech from the individual 3210, the processor 210 may be configured to change the playback rate of the sound from the individual 3210. For example, the speaking rate of the individual 3210 may be reduced to make the detected speech more perceptible to the user 100 . For example, as discussed above, the speech rate may be modified by lengthening or reducing the duration of spoken words and/or the period of silence between spoken words. Various other processing may be performed, such as modifying the pitch of the sound from the individual 3210, to maintain the same pitch as the original audio signal, or to reduce noise within the audio signal. If speech recognition has been performed on the audio signal associated with the sound from the individual 3210, the conditioning may also include modifying the audio signal based on the detected speech. For example, processor 210 may introduce pauses or increase the duration of pauses between words and/or sentences, which may make speech easier to understand.

The conditioned audio signal may then be sent to auditory interface device 1710 and an audio signal generated for user 100 . Thus, in the conditioned audio signal, the sound from the individual 3210 may be more easily heard by the user 100, louder and/or more distinguishable than the sound from the audio signal 3205 or 3207, the sound from the audio signal 3205 or 3207 Can represent background noise within the environment.

In some embodiments, at least one microphone 1750 can capture one or more audio signals received during a moving time window of predetermined length, and processor 210 can be programmed to cause selective adjustment and Transmission of part of an audio signal. For example, processor 210 may store a portion of at least one audio signal (eg, from individual 3210 ) in database 2050 , where the portion was received prior to at least one action (eg, movement 3203 ) of user 100 . The portion of the at least one audio signal may be sent to the auditory interface device 1710 configured to provide sound to the ear of the user 100, and thus may provide the user 100 with a source corresponding to the portion of the at least one audio signal (eg, another user ) auditory feedback. The portion of the at least one audio signal may be sent to the hearing device 1710 based on at least one action of the user 100 . For example, after the user 100 status is "repeat", the portion may be sent to the hearing device 1710, thereby duplicating at least one sound that is difficult for the user 100 to hear.

34 is a flowchart illustrating an exemplary process 3400 for selectively amplifying sound, consistent with the disclosed embodiments.

In step 3401 , a wearable camera (eg, the wearable camera of device 110 ) may capture multiple images from the environment of user 100 . In some embodiments, the wearable camera may be a component of the device 110 (eg, a camera-based directional hearing aid device) for selectively selectively based on the motion 3201 of the user 100 (eg, hand motion, tilt motion, gaze direction, etc.) Change the amplification of the sound. In some embodiments, the wearable camera can capture at least one image from the user's 100 environment.

In step 3403, at least one microphone may capture sound from the user's 100 environment. In some embodiments, the microphone 1720 may be configured to determine the directionality of sound in the environment of the user 100 . For example, microphone 1720 may include one or more directional microphones, which may be more sensitive to picking up sounds in certain directions. The processor 210 may be configured to differentiate sounds within the environment of the user 100 and determine the approximate directionality of each sound. For example, using the microphone array 1720 , the processor 210 may compare the relative timing or amplitude of individual sounds between the microphones 1720 to determine the directionality relative to the device 100 .

In step 3405, the processor 210 may receive a plurality of images captured by the wearable camera. For example, the wearable camera may be configured to capture one or more images of the user's 100 surroundings, eg, using the image sensor 220 .

In step 3407, the processor 210 may receive at least one audio signal representing sound captured by the at least one microphone from the user's environment. For example, the processor 210 may receive at least one audio signal 3203 , 3205 or 3207 representing sound captured by the at least one microphone 1750 from the environment of the user 100 .

In step 3409, the processor 210 may identify at least one action of the user based on the analysis of at least one of the plurality of images or at least one audio signal. In some embodiments, processor 210 may identify at least one action of user 100 based on analysis of at least one audio signal (eg, audio signal 3203). In some embodiments, the at least one action may include speech by the user 100 . For example, identifying the at least one action may include detecting words spoken by the user 100 based on analysis of the at least one audio signal 3203 . In some embodiments, the words may indicate that the user 100 did not hear clearly. For example, user 100 may say "what?", "Can you repeat what you said?", "I don't understand", or similar phrases. In some embodiments, processor 210 may associate greater hearing difficulties with user 100 based on the type and/or frequency of detected words. For example, "repetition" may be associated with greater hearing difficulties than "I don't understand", and/or repeated words (eg, user 100 repeating the statement "what?") may be associated with greater hearing than unrepeated words difficulty associated.

In some embodiments, the sounds captured from the environment of the user 10 may be classified using any audio classification technique. The processor 210 may be configured to analyze the sound to separate and identify different sources of the audio signal. For example, the processor 210 may use one or more speech or voice activity detection (VAD) algorithms, speech separation techniques, and/or sound classification techniques. When multiple sounds are detected in the environment of the user 100, the processor 210 may isolate audio signals associated with different sound sources. In some embodiments, the processor 210 may perform further analysis on the audio signal associated with the detected voice activity to identify the individual's voice. For example, processor 210 may use one or more speech recognition algorithms (eg, hidden Markov models, dynamic time warping, neural networks, or other techniques) to recognize an individual's speech and/or spoken words. For example, sounds may be classified into segments containing speech, music, tones, laughter, screams, and the like. The indication of each segment may be recorded in database 2050.

In some embodiments, the recorded information may enable processor 210 to identify at least one action of user 100 based on analysis of at least one audio signal 3203 . In some embodiments, the at least one action may include speech by the user 100 . For example, identifying the at least one action may include detecting words spoken by the user 100 based on analysis of the at least one audio signal 3203 . In some embodiments, the words may indicate that the user 100 is not hearing the sound well (eg, from another entity 3210, the sound associated with the audio signal 3205, the sound associated with the audio signal 3207, etc.).

In some embodiments, the processor 210 may receive at least one image captured by the wearable camera, and may identify at least one action of the user 100 based on the analysis of the at least one image. The processor 210 may identify the at least one action by detecting the motion 3201 of the user 100 based on the analysis of the at least one image. In some embodiments, motion 3201 may include hand motion of user 100 or tilt motion of user 100 . For example, the user 100 may wrap his hands around the ears, indicating that they are having difficulty hearing the individual 3210. In some embodiments, the user 100 may tilt towards the individual 3210, indicating that they are having difficulty hearing the individual 3210.

In some embodiments, this can be done by monitoring the orientation of the user's 100 body part (eg, hand, arm, etc.) or face part (eg, nose, eyes, ears, hand near the ear, etc.) relative to the optical axis of the camera sensor Movement 3201 of user 100 is tracked. For example, the captured image may include a representation of a body part or a face part of the user 100 , which may be used to determine hand movement or tilting movement of the user 100 . Processor 210 (and/or processors 210a and 210b) may be configured to use various image detection or processing algorithms (eg, using convolutional neural networks (CNN), scale-invariant feature transforms (SIFT), histograms of oriented gradients) (HOG) feature or other techniques) to analyze the captured image and detect movement 3201 of the body part or face part of the user 100 . Based on the detected representation of the body part or face part of the user 100, the motion 3201 of the user 100 can be determined.

Motion 3201 may be determined in part by comparing a detected representation of the body part or face part of user 100 to the optical axis of camera sensor 1751 . For example, the optical axis 1751 may be known or fixed in each image, and the processor 210 may determine the motion 1750 by comparing a representative angle of the body part or face part of the user 100 to the direction of the optical axis 1751 . For example, the determined motion may include the user 100 wrapping his hands around the ears, indicating that they are having difficulty hearing the individual 3210. In some embodiments, the determined motion may include the user 100 leaning toward the individual 3210, indicating that they are having difficulty hearing the individual 3210.

In step 3411, the processor 210 may cause selective adjustment of the at least one audio signal received by the at least one microphone based on the identified action as described in more detail above.

In step 3413, the processor 210 may cause the at least one conditioned audio signal to be transmitted to an auditory interface device configured to provide sound to the user's ear. For example, at least one conditioned audio signal may be sent to auditory interface device 1710, which is configured to provide sound to the ear of user 100, and thus may provide user 100 with a source corresponding to the at least one audio signal ( For example, auditory feedback of individual 3210 associated with audio signal 3205, individual 3210 associated with audio signal 3207, etc.). For example, in the adjusted audio signal, the sound from the individual 3210 may be more easily heard by the user 100, louder and/or more distinguishable than the sound from the audio signal 3205 or 3207, the sound from the audio signal 3205 or 3207 Can represent background noise within the environment.

Intuitive control of active speakers

Hearing aid systems are designed to improve and enhance the user's interaction with the environment. Users may rely on hearing aid systems to navigate their surroundings and daily activities. However, different users may require different levels of assistance depending on the environment. In some cases, a user may hear sound from sources in his or her environment preferentially over one or more additional sources in his or her environment. For example, a user may preferentially hear voices from family members in his environment over strangers or background noise in his environment. Typical hearing aid systems may not be able to adequately correct or adjust the audio signal based on the needs of the user. Accordingly, there is a need for an apparatus and method for automatically adjusting a user's audio signal based on cues from the user's environment.

The disclosed embodiments include hearing aid systems that may be configured to correct or adjust audio signals based on cues from the hearing aid user's environment. The cues may include the distance between the user and one or more individuals, the direction of the individual relative to the user's gaze direction, the active speaker's gesture, the individual's gaze direction toward the active speaker, or other visual cues. For example, sounds from individuals closer to the user may have higher priority than sounds from individuals further away from the user. In some embodiments, the user may manually define or assign priorities to different sound sources via the device. For example, a user may assign a higher priority to his identified individuals (eg, family members, friends, etc.) than other sound sources (eg, sounds from the device, strangers, background noise, etc.). In some embodiments, the hearing aid system can identify the individual and use the priority assigned to the individual accordingly.

In some embodiments, the hearing aid system may correct or adjust the audio signal by selectively conditioning (eg, amplifying, attenuating, muting, etc.) the audio signal based on the priority of the sound source. For example, the hearing aid system may preferably amplify audio signals from sound sources with higher priority. In some embodiments, the hearing aid system may preferably attenuate or mute audio signals from sound sources with lower priority.

35 is a schematic diagram illustrating an exemplary environment including a hearing aid with speech and/or image recognition consistent with the disclosed embodiments. In some embodiments, a wearable camera (eg, the wearable camera of device 110 ) may be configured to capture multiple images from the environment of user 100 . In some embodiments, at least one microphone may be configured to capture sound from the environment of the user 100 .

For example, the processor 210 may receive at least one audio signal 3511 , 3513 or 3515 representing sound captured by the at least one microphone 1750 from the environment of the user 100 . In some embodiments, processor 210 may identify a first audio signal associated with a first speech associated with first individual 3501 based on analysis of at least one audio signal (eg, audio signal 3511, 3513, or 3515) 3511 and a second audio signal 3513 associated with the second speech associated with the second individual 3503.

The hearing aid system may store speech samples, images, speech features and/or facial features of the recognized person to aid in recognition and selective amplification. For example, when an individual (either the first individual 3501 or the second individual 3503) enters the field of view of the device 110, the individual may be identified as an individual who has been introduced to the user 110, or who may have interacted with the user 100 in the past (eg, friends, colleagues, relatives, previous acquaintances, etc.). Accordingly, an audio signal (eg, audio signal 3511 or audio signal 3513 ) associated with a recognized individual's speech may be isolated and/or selectively amplified relative to other sounds in the user's environment. Audio signals (eg, audio signal 3515) associated with sound received from directions other than the individual's direction may be suppressed, attenuated, filtered, and the like.

The user 100 may want to amplify the audio signal based on the priority of the sounds that the user 100 wishes to receive. For example, the processor 210 may determine a hierarchy of individuals and assign priorities based on the relative status of the individuals. The hierarchy may be based on the location of the individual relative to the user 100 within the family or organization (eg, company, sports team, club, etc.). For example, user 100 may be in a work environment and may need to hear his boss before his colleagues. Therefore, the boss of the user 100 may be ranked higher than colleagues or people from different departments, and thus may have priority in the selective adjustment process. In some embodiments, user 100 may be in an environment with "close friends," family, and acquaintances. For example, individuals identified as close friends or family members may take precedence over acquaintances of user 100 because user 100 may wish to hear from close friends or family members rather than acquaintances.

In some embodiments, the wearable camera may be a component of the apparatus 110 (eg, a camera-based directional hearing aid device) for an individual (eg, the first individual 3501 or the second individual 3503 ) in the user 100-based environment identification to selectively change the amplification of the sound. In some embodiments, the wearable camera may capture at least one image from the environment of the user 100 using the image sensor 220 . In some embodiments, processor 210 may receive at least one image captured by the wearable camera, and may identify at least one individual in the environment of user 100 based on analysis of the at least one image. Processor 210 (and/or processors 210a and 210b) may be configured to use various image detection or processing algorithms (eg, using convolutional neural networks (CNN), scale-invariant feature transforms (SIFT), histograms of oriented gradients) (HOG) features or other techniques) to analyze the captured images and detect features of the body part or face part of the at least one individual. The at least one individual may be identified based on the detected representation of the body part or face part of the at least one individual. In some embodiments, processor 210 may be configured to identify at least one individual using facial and/or voice recognition components, as described for apparatus 110 in FIG. 20A or FIG. 20B.

For example, facial recognition component 2040 can be configured to recognize one or more faces within the environment of user 100 . Facial recognition component 2040 can identify facial features on an individual's face, such as eyes, nose, cheekbones, chin, or other features. The facial recognition component 2040 can analyze the relative size and location of these features to identify the individual. In some embodiments, the facial recognition component 2040 can utilize one or more algorithms to analyze the detected features, such as principal component analysis (eg, using eigenfaces), linear discriminant analysis, elastic bundle map matching (eg, using Fisher face), Local Binary Pattern Histogram (LBPH), Scale Invariant Feature Transform (SIFT), Accelerated Robust Feature (SURF), etc. Individuals may be identified using additional facial recognition techniques such as three-dimensional recognition, skin texture analysis, and/or thermal imaging. Features other than the individual's facial features may also be used for identification, such as height, body size, or other distinguishing characteristics of the individual.

Facial recognition component 2040 can access a database or data associated with user 100 to determine whether detected facial features correspond to an identified individual. For example, processor 210 may access database 2050 that contains information about individuals known to user 100 and data representing associated facial or other identifying features. Such data may include one or more images of an individual, or data representing a user's face that may be used for identification by facial recognition. The facial recognition component 2040 can also access a contact list of the user 100, such as a contact list on the user's phone, a web-based contact list (eg, through Outlook ^™ , Skype ^™ , Google ^™ , Salesforce ^™ , etc.) or with an auditory interface A private contact list associated with device 1710. In some embodiments, database 2050 may be compiled by device 110 from previous facial recognition analysis. For example, processor 210 may be configured to store data associated with one or more faces identified in images captured by device 110 in database 2050 . Each time a face is detected in an image, the detected facial features or other data may be compared to previously identified faces in database 2050. Facial recognition component 2040 can determine whether the individual is an identified individual of user 100, whether the individual has been previously identified by the system in instances exceeding a certain threshold, whether the individual has been explicitly introduced to device 110, and the like.

Apparatus 110 may be configured to identify an individual (eg, first individual 3501 or second individual 3503 ) in the environment of user 100 based on the plurality of received images captured by the wearable camera. For example, apparatus 110 may be configured to identify face 3521 associated with first individual 3501 or face 3523 associated with second individual 3503 within the environment of user 100 . For example, device 110 may be configured to capture one or more images of user 100's surroundings using camera 1730 . The captured image may include a representation of an identified individual (eg, first individual 3501 or second individual 3503 ), which may be a friend, colleague, relative, or previous acquaintance of user 100 . Processor 210 (and/or processors 210a and 210b) may be configured to analyze captured images and detect identified individuals using various facial recognition techniques. Accordingly, apparatus 110, or memory 550 in particular, may include one or more facial recognition components (eg, software programs, modules, libraries, etc.).

In some embodiments, the processor 210 may be configured to cause a response to the audio signal (eg, audio signal 3511 or 3513) based on the processor 210 determining that the first audio signal is associated with a higher priority than the second audio signal. ) selective adjustment. Various methods can be used to determine the hierarchy of sounds. For example, the hierarchy of sounds can be determined by comparative analysis of two or more sounds. In some embodiments, sound sources may include people, objects (eg, televisions, cars, etc.), environments (eg, flowing water, wind, etc.), and the like. For example, the processor 210 may use comparative analysis to determine that sound from a person has priority over sound from an object or environment.

In some embodiments, the selective adjustment of the audio signal associated with the identified individual may be based on the identity of the individual in the environment of the user 100 . For example, where multiple individuals are detected in the image, the processor 210 may identify the individuals using one or more facial recognition techniques as described above. Audio signals associated with individuals known to user 100 may be selectively amplified or otherwise adjusted to have priority over unknown individuals. For example, processor 210 may be configured to attenuate or mute audio signals associated with bystanders in the environment of user 100 (such as noisy office colleagues, etc.). In some embodiments, the processor 210 may also determine a hierarchy of individuals and assign priorities based on the relative status of the individuals. The hierarchy may be based on the location of the individual relative to the user 100 within the family or organization (eg, company, sports team, club, etc.). For example, the boss of user 100 may be ranked higher than colleagues or people from different departments and thus may have priority in the selective moderation process. In some embodiments, the hierarchy may be determined based on a list or database. Individuals identified by the system can be sorted individually or grouped into layers of priority. The database can be maintained specifically for this purpose or can be accessed externally. For example, a database can be associated with a user's social network (eg, Facebook ^™ , LinkedIn ^™ , etc.), and individuals can be prioritized based on their groupings or relationship to the user. For example, individuals identified as "close friends" or family members may take precedence over acquaintances of user 100 .

In some embodiments, the processor 210 may selectively adjust the audio signal associated with the individual based on the individual's proximity to the user 100 . The processor 210 may determine the distance from the user 100 to each individual based on captured images, rangefinders, or other methods, and may selectively adjust audio signals associated with the individuals based on the distance. For example, an individual who is physically closer to the user 100 may be given higher priority and his or her voice may be amplified to a greater extent than an individual who is farther away from the user 100 . In some embodiments, the processor 210 may determine the orientation of the individual relative to the user's gaze direction. Individuals at closer angles relative to the line-of-sight direction may be given higher priority.

In some embodiments, the processor 210 may determine the priority level by identifying at least one action based on an analysis of at least one of the plurality of images. For example, processor 210 may determine which individuals in the environment of user 100 are speaking based on lip movements and detected sounds. For example, processor 210 may track lip movements associated with individual 3501 or 3503 to determine that individual 3501 or 3503 is speaking. A comparative analysis can be performed between the detected lip movement and the received audio signal. For example, the processor 210 may determine that the individual 3501 is speaking based on a determination that the individual 3501's mouth is moving while the sound associated with the audio signal 3511 is detected. In some embodiments, when the lips of the individual 3501 cease to move, this may correspond to a period of silence or reduced volume in the sound associated with the audio signal 3511.

In some embodiments, data associated with device 110 may also be used in conjunction with detected lip movements to determine and/or verify whether an individual is speaking, such as the gaze direction of user 100 or individual 3501 or 3503, detected The identity of the individual 3501 or 3503, the voiceprint of the recognized individual 3501 or 3503, etc.

In some embodiments, the processor 210 may be configured to selectively condition the plurality of audio signals based on which individuals associated with the audio signals are currently speaking. That is, in some embodiments, the processor 210 may prioritize individuals who are speaking over individuals who are not speaking. For example, individual 3501 and individual 3503 may engage in a conversation within the environment of user 100, and processor 210 may be configured to transition from amplifying audio signal 3511 associated with individual 3501 to amplifying and Audio signal 3513 associated with individual 3503. For example, lip movement of individual 3501 may indicate that individual 3501 has stopped speaking, or lip movement associated with individual 3503 may indicate that individual 3503 has begun to speak. Therefore, the processor 210 can switch between the amplified audio signal 3511 to the audio signal 3513 . In some embodiments, processor 210 may be configured to process and/or condition both audio signals simultaneously, but selectively send the conditioned audio to auditory interface device 1710 based only on which individual is speaking. Where speech recognition is implemented, the processor 210 may determine and/or anticipate transitions between speakers based on the context of the speech. For example, processor 210 may analyze audio signal 3511 to determine that individual 3501 has reached the end of a sentence or has asked a question, which may indicate that individual 3501 has finished or is about to finish speaking.

In some embodiments, the processor 210 may be configured to select between multiple active speakers to selectively condition the audio signal. For example, individuals 3501 and 3503 may both be speaking at the same time, or their speech may overlap during the conversation. The processor 210 may amplify audio associated with one speaking individual in preference to other individuals. This may include giving priority to a speaker who has started but has not completed a word or sentence or who has not fully finished speaking when another speaker starts speaking. As mentioned above, this determination can also be driven by the context of the speech.

In some embodiments, the gaze direction of the user 100 or individual 3501 or 3503 may be determined, and the individual pointed by the gaze direction may be given higher priority among active speakers. For example, if individual 3503 is looking at individual 3501, the audio signal 3511 associated with individual 3501 may be selectively adjusted (eg, amplified). In some embodiments, priorities may be assigned based on the relative behavior of other individuals in the user's 100 environment. For example, if both individual 3501 and individual 3503 are speaking, and an additional individual is looking at individual 3501 instead of individual 3503, the audio signal 3511 associated with individual 3501 may be amplified in preference to the audio signal 3511 associated with individual 3503 . In embodiments where the identity of an individual is determined, as previously discussed, priorities may be assigned based on the relative status of the speakers.

In some embodiments, processor 210 may be configured to select at least one audio signal (eg, audio signal 3511 or audio signal 3513 ) in the environment of user 100 received by at least one microphone based on the determined priority Sexual regulation. The at least one conditioned audio signal may be sent to auditory interface device 1710 , and auditory interface device 1710 may be configured to provide sound to user 100 ears. Accordingly, auditory interface device 1710 may provide user 100 with auditory feedback corresponding to the source of at least one audio signal (eg, associated with audio signal 3511, associated with audio signal 3513, etc.). The processor 210 may perform various conditioning techniques on the audio signal received from the microphone 1720 . Adjusting may include amplifying audio signals determined to have higher priority than other audio signals. Amplification can be accomplished digitally, for example, by processing audio signals that are associated with higher priority relative to other signals. Amplification can also be achieved by changing one or more parameters of the microphone 1720 to focus on audio sounds emanating from higher priority sources. For example, microphone 1720 may be a directional microphone, and processor 210 may perform operations to focus microphone 1720 on individual 3501 or 3503 or other sounds in the environment of user 100 . Various other techniques for amplifying sound may be used, such as the use of beamforming microphone arrays, acoustic telescope techniques, and the like.

Conditioning may also include attenuating or suppressing one or more audio signals received from lower priority sources. For example, processor 210 may determine that individual 3501 has a higher priority than individual 3503, and attenuate audio signals 3513 and 3515. Similar to sound amplification, sound attenuation can occur by processing the audio signal, or by changing one or more parameters associated with one or more microphones 1720 to direct focus away from sound emanating from lower priority sources.

In some embodiments, the adjustment may also include changing the pitch of the audio signal corresponding to the sound from the higher priority source to make the sound more easily perceived by the user 100 . For example, the user 100 may have less sensitivity to tones within a certain range, and conditioning of the audio signal may adjust the pitch of the sound from a higher priority source to make it more perceptible to the user 100 . For example, user 100 may experience hearing loss in frequencies above 10 kHz. Thus, the processor 210 may remap higher frequencies (eg, at 15khz) to frequencies below 10khz. In some embodiments, the processor 210 may be configured to vary the speech rate associated with one or more audio signals. Accordingly, processor 210 may be configured to detect speech within one or more audio signals received by microphone 1720, eg, using a voice activity detection (VAD) algorithm or technique. For example, if it is determined that the sound corresponds to speech or speech from the individual 3501, the processor 210 may be configured to change the playback rate of the sound from the individual 3501. For example, the speaking rate of the individual 3501 may be reduced to make the detected speech more perceptible to the user 100 . Various other processing may be performed, such as modifying the pitch of the sound from the individual 3501, to maintain the same pitch as the original audio signal, or to reduce noise within the audio signal. If speech recognition has been performed on the audio signal associated with the sound from the individual 3501, the conditioning may also include modifying the audio signal based on the detected speech. For example, processor 210 may introduce pauses or increase or decrease the duration of pauses between words and/or sentences, which may make speech easier to understand.

The conditioned audio signal may then be sent to auditory interface device 1710 and an audio signal generated for user 100 . Thus, in the conditioned audio signal, sounds from higher priority sources may be more easily heard by the user 100, louder and/or more distinguishable than sounds from lower priority sources, from lower priority sources The sound of the source may represent background noise within the environment.

36 illustrates an exemplary computing device 120 for use with a hearing aid with speech and/or image recognition consistent with the disclosed embodiments. In some embodiments, the user 100 may provide input for prioritizing speakers through predefined settings or by actively selecting which speaker to focus on. In some embodiments, computing device 120 may be paired with apparatus 110 . For example, computing device 120 (eg, a mobile device) may display at least one audio signal priority interface 3601 (eg, a graphical user interface) associated with an individual (eg, individual 3501). In some embodiments, user 100 may interact with at least interface 3601 to submit at least one priority setting. For example, user 100 may enter a priority setting via interface 3601 on computing device 120 indicating that individual 3501 has a higher priority than other individuals.

In some embodiments, user 100 may enter priority settings for one or more sound sources by interacting with various interfaces that indicate volume adjustment of sound sources (eg, individuals, objects, environments, etc.) via computing device 120 . For example, the user 100 may enter the priority setting of the individual 3501 by interacting with the interface indicating the volume increase of the individual 3501 (eg, selecting an icon, moving a slider icon, etc.). In some embodiments, user 100 may enter priority settings for individual 3501 by interacting with an interface that reduces or mute the volume of one or more sound sources (eg, other individuals, objects, environments, etc.).

In some embodiments, the user 100 may assign a hierarchy to the sound sources by ordinal sorting the priority of the sound sources via the interface of the computing device 120 . In some embodiments, the user 100 may "favor" (eg, assign an asterisk via the interface of the computing device 120) some sound sources that he prioritizes over others.

In some embodiments, audio signal priority interface 3601 may display images captured by device 110 from the environment of user 100 . In some embodiments, computing device 120 may include a plurality of audio signal priority interfaces, wherein each audio signal priority interface includes an image captured by apparatus 110 from the environment of user 100 . As previously described, the processor 210 may identify at least one action based on analysis of at least one of the plurality of captured images. For example, the processor 210 may determine a hierarchy of sound sources based on the user 100 pointing at one or more sound sources (eg, individuals). In some embodiments, the user 100 may direct at least one sound source of each sound source in order from highest priority to lowest priority. Based on the actions of the user 100, the audio signal priority interface of the computing device 120 may display a hierarchy of sounds associated with each sound source. The processor 210 may selectively adjust the audio signal associated with each sound source based on a hierarchy of sound sources. For example, in the environment of user 100, higher priority audio signals may be isolated and/or selectively amplified relative to lower priority audio signals. In some embodiments, low priority audio signals may be suppressed, attenuated, filtered, unchanged, and the like.

In some embodiments, user 100 may enter priority settings for one or more sound sources by entering conditions via computing device 120 . For example, user 100 may input conditions such that sound sources uttering keywords (eg, individuals saying "be careful," "emergency," etc.) are prioritized over other sound sources (eg, amplified). In some embodiments, user 100 may enter priority settings for sound sources by entering other conditions or criteria. For example, user 100 may prioritize sound sources based on time (eg, time of day, time of week, time of year, etc.) so that the sound is modulated within a specified time span. For example, user 100 may enter priority settings such that sound is amplified Monday through Friday from 9:00 am to 5:00 pm during business hours.

37 is a flowchart illustrating an exemplary process 3700 for selectively amplifying sound, consistent with the disclosed embodiments. For example, according to process 3700, the hearing aid system may be configured to correct or adjust the audio signal by selectively adjusting (eg, amplifying, attenuating, muting, etc.) the audio signal based on the priority of the sound source.

In step 3701 , a wearable camera (eg, the wearable camera of device 110 ) may capture multiple images from the environment of user 100 . In some embodiments, the wearable camera may be a component of the apparatus 110 (eg, a camera-based directional hearing aid device) for an individual (eg, the first individual 3501 or the second individual 3503 ) in the user 100-based environment identification to selectively change the amplification of the sound. In some embodiments, the wearable camera may capture at least one image from the environment of the user 100 using the image sensor 220 . In some embodiments, the processor 210 may receive at least one image captured by the wearable camera.

In step 3703, the processor 210 may identify at least one individual in the environment of the user 100 based on the analysis of the at least one image. As noted above, processor 210 (and/or processors 210a and 210b) may be configured to use various image detection or processing algorithms (eg, using convolutional neural networks (CNN), scale-invariant feature transform (SIFT), Histogram of Oriented Gradients (HOG) feature or other techniques) to analyze the captured image and detect features of the body part or face part of the at least one individual. Based on the detected representation of the body part or face part of the at least one individual, at least one identity of the individual can be determined. In some embodiments, processor 210 may be configured to identify at least one individual using facial and/or voice recognition components, as described for apparatus 110 in FIG. 20A or FIG. 20B.

Apparatus 110 may be configured to identify an individual (eg, first individual 3501 or second individual 3503 ) in the environment of user 100 based on the plurality of received images captured by the wearable camera. The apparatus 110 may be configured to identify the face 3521 associated with the first individual 3501 or the face 3523 associated with the second individual 3503 within the environment of the user 100 . For example, device 110 may be configured to capture one or more images of user 100's surroundings using camera 1730 . The captured image may include a representation of an identified individual (eg, first individual 3501 or second individual 3503 ), which may be a friend, colleague, relative, or other previous acquaintance of user 100 . Processor 210 (and/or processors 210a and 210b) may be configured to analyze captured images and detect identified individuals using various facial recognition techniques. Accordingly, device 110, or memory 550 in particular, may include one or more facial recognition components.

For example, facial recognition component 2040 can access a database or data associated with user 100 to determine whether detected facial features correspond to an identified individual. For example, processor 210 may access database 2050 (eg, remotely, via a network, etc.) that contains information about individuals known to user 100 and data representing associated facial or other identifying features. Such data may include one or more images of an individual, or data representing a user's face that may be used for identification by facial recognition. In some embodiments, database 2050 may be compiled by device 110 through previous facial recognition. For example, processor 210 may be configured to store data associated with one or more faces identified in images captured by device 110 in database 2050 . Each time a face is detected in an image, the detected facial features or other data may be compared to previously identified faces in database 2050. Facial recognition component 2040 can determine whether the individual is an identified individual of user 100, whether the individual has been previously identified by the system in instances exceeding a certain threshold, whether the individual has been explicitly introduced to device 110, and the like.

In some embodiments, audio signal priority interface 3601 may display images captured by device 110 from the environment of user 100 . For example, computing device 120 may include a plurality of audio signal priority interfaces, where each audio signal priority interface includes an image (eg, of an individual or other sound source) captured by apparatus 110 from the environment of user 100 .

In step 3705, at least one microphone may capture sound from the user's 100 environment. The processor 210 may receive at least one audio signal 3511 , 3513 or 3515 representing sound captured by the at least one microphone 1750 from the environment of the user 100 .

In step 3707, the processor 210 may identify a first audio signal 3511 associated with a first speech associated with the first individual 3501 based on the analysis of at least one audio signal (eg, audio signal 3511, 3513 or 3515) A second audio signal 3513 associated with a second speech associated with the second individual 3503. Various methods can be used to determine the hierarchy of sounds. For example, the hierarchy of sounds can be determined by comparative analysis of two or more sounds. In some embodiments, sound sources may include people, objects (eg, televisions, cars, etc.), environments (eg, flowing water, wind, etc.), and the like. For example, the processor 210 may use comparative analysis to determine that sound from a person has priority over sound from an object or environment. As described above, the hierarchy of sounds may be determined based on user input, default settings, and the like.

In step 3709, the processor 210 may be configured to cause the first audio signal and the second audio signal (eg, , selective conditioning of audio signals 3511 and 3513).

In some embodiments, the selective adjustment of the audio signal associated with the identified individual may be based on the identity of the individual in the environment of the user 100 . For example, where multiple individuals are detected in the image, the processor 210 may identify the individuals using one or more facial recognition techniques as described above. Audio signals associated with individuals known to user 100 may be selectively amplified or otherwise adjusted to have priority over unknown individuals. For example, processor 210 may be configured to attenuate or mute audio signals associated with bystanders in the environment of user 100 (such as noisy office colleagues, etc.). In some embodiments, the processor 210 may also determine a hierarchy of individuals and give priority based on the relative status of the individuals. In some embodiments, the hierarchy may be determined based on a list or database. Individuals identified by the system can be sorted individually or grouped into layers of priority. For example, individuals identified as "close friends" or family members may take precedence over acquaintances of user 100 .

In some embodiments, processor 210 may selectively adjust audio signals associated with one or more individuals based on the individual's proximity to user 100 . The processor 210 may determine the distance from the user 100 to each individual based on the captured images, and may selectively adjust audio signals associated with the individuals based on the distance. For example, individuals closer to user 100 may be given higher priority than individuals further away from user 100 . Similarly, individuals at angles closer to the user's line of sight may be given higher priority than individuals at greater angles from the user's line of sight.

In some embodiments, the processor 210 may determine the priority level by identifying at least one action based on an analysis of at least one of the plurality of images. For example, processor 210 may determine which individuals in the environment of user 100 are speaking based on lip movements and detected sounds. For example, processor 210 may track lip movements associated with individual 3501 or 3503 to determine that individual 3501 or 3503 is speaking. A comparative analysis can be performed between the detected lip movement and the received audio signal. For example, the processor 210 may determine that the individual 3501 is speaking based on a determination that the individual 3501's mouth is moving while the sound associated with the audio signal 3511 is detected. In some embodiments, when the lips of the individual 3501 cease to move, this may correspond to a period of silence or reduced volume in the sound associated with the audio signal 3511.

In some embodiments, data associated with device 110 may also be used in conjunction with detected lip movements to determine and/or verify whether an individual is speaking, such as the gaze direction of user 100 or individual 3501 or 3503, detected The identity of the individual 3501 or 3503, the voiceprint of the recognized individual 3501 or 3503, etc.

In some embodiments, the gaze direction of the user 100 or individual 3501 or 3503 may be determined, and the individual pointed by the gaze direction may be given higher priority among active speakers. For example, if individual 3503 is looking at individual 3501, the audio signal 3511 associated with individual 3501 may be selectively adjusted. In some embodiments, priorities may be assigned based on the relative behavior of other individuals in the user's 100 environment. For example, if both individual 3501 and individual 3503 are speaking, and more other individuals are looking at individual 3501 than individual 3503, the audio signal 3511 associated with individual 3501 may take precedence over the audio signal associated with individual 3503 is selectively regulated. In embodiments where the identity of an individual is determined, as previously discussed, priorities may be assigned based on the relative status of the speakers.

In some embodiments, the processor 210 may identify at least one action based on analysis of at least one of the plurality of captured images. For example, the processor 210 may determine a hierarchy of sound sources based on the user 100 pointing at one or more sound sources (eg, individuals). In some embodiments, the user 100 may direct at least one sound source of each sound source in order from highest priority to lowest priority. Based on the actions of the user 100, the audio signal priority interface of the computing device 120 may display a hierarchy of sounds associated with each sound source. The processor 210 may selectively adjust the audio signal associated with each sound source based on a hierarchy of sound sources. For example, in the environment of user 100, higher priority audio signals may be isolated and/or selectively amplified relative to lower priority audio signals. In some embodiments, low priority audio signals may be suppressed, attenuated, filtered, unchanged, and the like.

In step 3711 , the processor 210 may be configured to transmit the selectively conditioned first audio signal to the auditory interface device 1710 configured to provide sound to the ear of the user 100 . Thus, in the conditioned audio signal, sounds from higher priority sources may be more easily heard by the user 100, louder and/or more distinguishable than sounds from lower priority sources, from lower priority sources The sound of the source may represent background noise within the environment.

Hearing aids and paired camera systems

According to embodiments of the present disclosure, the hearing aid system may selectively amplify sound. Hearing aid systems may include wearable camera devices and hearing aid devices. A wearable camera device may refer to a device with image, sound, audio and/or video capture capabilities that may be attached to a user or to a user's clothing or accessories. A hearing aid device may refer to a device that outputs audio or sound to a user's ear. The output of the hearing aid device may be generated based on input received from or by the wearable camera device. A hearing aid system may include several devices that are paired together to provide improved functionality. For example, a hearing aid system may include a hearing aid device for providing sound to a user's ear, where the sound may be acoustically captured by the hearing aid or received electronically from another source such as a wearable camera device. The hearing aid device can be paired with the wearable camera device (and vice versa), and the wearable camera device can capture images and/or audio. The wearable camera device may capture images and/or audio according to instructions received from the hearing aid device. In response, the hearing aid device can receive audio and other information from the wearable camera device. The system may also include a mobile device paired with the wearable camera device and the hearing aid device. For example, a GUI displayed on a display of a mobile device may enable a user to provide input to control how sound is processed and received at the hearing aid device. The hearing aid system may also include pairing of other devices such as rangefinders.

As discussed above, in some embodiments, the hearing aid system may include a mobile device. The mobile device may be a mobile phone, or other examples of devices such as PDAs, tablet computers, wearable electronic devices, and other types of portable electronic devices. The mobile device may be paired with at least one or both of a hearing aid device or a wearable camera device. Pairing can refer to enabling communication between two or more devices, and a mobile device can be wirelessly paired with a hearing aid device or a wearable camera device. Examples of wireless pairing include Wi-Fi, Bluetooth, NFC, and other similar wireless communication technologies.

In some embodiments, the hearing aid system may include a rangefinder. A rangefinder may refer to a device capable of determining the range (or distance) between an object and itself. In some embodiments, the rangefinder may be wirelessly paired to a wearable camera device, hearing aid device, and/or mobile device, one or more of these paired devices receiving range measurements generated by the rangefinder. In some embodiments, the rangefinder may be incorporated into the wearable camera device.

In some embodiments, the wearable camera device may include at least one camera configured to capture multiple images from the user's environment. The camera may include one or more image sensors (such as image sensor 220 discussed above) for capturing one or more images (and/or video) from the environment of the user of the wearable camera device. In some embodiments, the wearable camera device includes at least one microphone configured to capture sound from the user's environment. The at least one microphone may refer to a component or device capable of receiving sound waves and generating audio signals based on the received sound waves. In some embodiments, the hearing aid device may include at least one speaker configured to provide sound to a user's ear. At least one speaker may refer to a component or device capable of generating sound, typically based on an audio signal.

For example, Figure 38 shows a hearing aid system 3800 including a wearable camera device and a hearing aid device. The hearing aid system 3800 includes a hearing aid device 3802 and a wearable camera device 3804 . In some embodiments, the hearing aid device 3802 and the wearable camera device 3804 are paired to communicate with the mobile device 3806. In some embodiments, the hearing aid device 3802 can be paired with the wearable camera device 3804 and data can be communicated between the paired devices. In some embodiments, the wearable camera device 3804 may correspond to the apparatus 110 shown in FIGS. 4A-4K. In some alternative embodiments, the wearable camera device 3804 may correspond to the apparatus 110 shown in Figures 3A-3B. In some embodiments, hearing aid device 3802 may correspond to auditory interface device 1710 .

As shown in Figure 38, the wearable camera device 3804 includes a camera 3804A. The camera 3804A can be placed on the wearable camera device 3804 so as to face in the direction of the object or person to be imaged. Camera 3804A may include an image sensor (eg, image sensor 220) for capturing real-time image data from a user's field of view. Camera 3804A may be a device capable of detecting and converting light signals into electrical signals in the near infrared, infrared, visible, ultraviolet spectrum, or any combination thereof. The electrical signals may be used to form images or video streams (ie, image data) based on the detected signals. The term "image data" includes any form of data retrieved from optical signals in the near infrared, infrared, visible, ultraviolet spectrum, or any combination thereof. Examples of image sensors may include semiconductor charge coupled devices (CCDs), active pixel sensors in complementary metal oxide semiconductors (CMOS), or N-type metal oxide semiconductors (NMOS, active MOS). In some embodiments, camera 3804A may have ranging features. For example, camera 3804A may determine range measurements 3807 for one or more objects in image 3805.

As an example, FIG. 39 shows an example of a user using a hearing aid system 3800. User 100 may wear wearable camera device 3804 and hearing aid device 3082 in accordance with the disclosed embodiments. As shown, user 100 may wear wearable camera device 3804 that is physically connected to user 100's shirt or other clothing. Consistent with the disclosed embodiments, the wearable camera device 3804 may be placed in other locations such as attached to necklaces, belts, glasses, wristbands, buttons, hats, and the like. Wearable camera device 3804 can be wirelessly paired with hearing aid device 3802 and/or with mobile device 3806.

As shown in Figure 39, a hearing aid device 3802 may be placed in one or both ears of the user 100, similar to a conventional auditory interface device. The hearing aid device 3802 can be of various styles, including in-canal, fully in-canal, in-ear, behind-the-ear, supra-ear, in-canal receiver, open mount, or various other styles. Hearing aid device 3802 may include one or more speakers for providing auditory feedback to user 100 . In some embodiments, hearing aid device 3802 may be auditory interface device 1710 as shown in Figure 17A.

In some embodiments, as shown in FIG. 17A , the hearing aid device 3802 may include a bone conduction earphone 1711. Bone conduction earphones 1711 may be surgically implanted and may provide audible feedback to user 100 through bone conduction of sound vibrations to the inner ear.

The user's environment generally refers to the environment of the user who is using the wearable camera device 3804 . The user's environment may include objects and people, some of which may generate sound waves 3803 that are received by one or more microphones (not shown) located in the wearable camera device 3804. Depending on the orientation and field of view of the camera 3804A, the camera 3804A may capture an image 3805 representing objects and people in the user's environment as seen by the camera 3804A along the optical axis 3903. In some embodiments, sound waves 3803 may be generated by objects or people captured in image 3805 . In some embodiments, image 3805 may include a representation of the chin of user 100 , which may be used to determine user's gaze direction 3901 , which may coincide with the user's 100 field of view.

Camera device 3804 may include at least one processor (may be referred to as at least one first processor in this disclosure). The term "processor" includes any physical device having circuitry that performs logical operations on inputs. For example, a processing device may include one or more integrated circuits, microchips, microcontrollers, microprocessors, central processing units (CPUs), graphics processing units (GPUs), digital signal processors (DSPs), field programmable gates All or part of an array (FPGA), or other circuitry suitable for executing instructions or performing logical operations. In some embodiments, the processor 210 shown in FIGS. 5A-5C may be an example of at least one first processor.

In some embodiments, the at least one first processor is programmed to selectively condition an audio signal received from at least one microphone representing sound captured by the at least one microphone. Conditioning may refer to editing, changing, or otherwise manipulating an audio signal. In some embodiments, the processor 210 may modulate the sound waves 3083 based on instructions received from the hearing aid device 3802.

In some embodiments, the hearing aid device may include at least one second processor. In some embodiments, the at least one second processor is programmed to cause the transmission of one or more instructions to the wearable camera device 3804. One or more instructions may be transmitted wirelessly through the channel created by the pairing between the hearing aid device 3802 and the wearable camera device 3804.

In some embodiments, a mobile device may include a user interface for providing output to a user. A user interface refers to a system or device capable of interacting with a user, such as providing output information to a display or receiving input from a user. In some embodiments, the user interface may be provided by the mobile device 3806. For example, mobile device 3806 may include a display for displaying user interface 3806A to allow user 100 to interact with hearing aid system 3800. In some embodiments, user interface 3806A may include an interface for receiving visual, audio, haptic, or any other suitable signals or input from user 100 . For example, user interface 3806A can include a display that can be part of mobile device 3800, such as with a display that can be accessed by user gestures (eg, touch gestures on a touch screen) or by a suitable physical or virtual (ie, on-screen) device (eg, keyboard, mouse, etc.) to manipulate the touch screen of GUI elements. In some embodiments, user interface 3806A may be an audio interface capable of receiving user 100 audio input (eg, user 100 speech input) for adjusting one or more parameters of system 3800 . For example, user 100 may provide input via user interface 3806A via audio commands. An audio interface may be provided by the mobile device 3806, such as a microphone associated with the mobile device 3806.

In some embodiments, user interface 3806A may also present information related to hearing aid system 3800 , such as information related to the operation of hearing aid device 3802 and/or wearable camera device 3804 . Such information can be used to inform the user 100 of the status of the hearing aid system 3800 so that the user 100 can control parameters of the hearing aid device 3802 and/or the wearable camera device 3804.

In some embodiments, the information related to the operation of the hearing aid device 3802 may include status information related to the pairing between the mobile device 3806 , the wearable camera 3804 , and/or the hearing aid device 3802 . In some embodiments, user 100 may initiate or terminate pairing between mobile device 3806, wearable camera 3804, and/or hearing aid device 3802 through user interface 3806A. In some embodiments, information related to the operation of the hearing aid system 3800 may include operational status information of the hearing aid device 3802 (such as battery level and/or volume level), and the user 100 may control parameters of the hearing aid device 3802 (such as adjusting the hearing aid device) the volume level of one or more speakers of the 3802). In some embodiments, the information related to the operation of the wearable camera device 3804 may include the operational status of the wearable camera device 3804 (such as battery levels), information related to images that have been captured by the wearable camera device 3804 and/or Audio adjustment operation.

In some embodiments, images captured by wearable camera device 3804, such as image 3805, may be displayed to user 100 in real-time via user interface 3806A. Image 3805 may be presented in a manner that enables user 100 to manipulate image 3805 in various ways. For example, user 100 may zoom in or out to maximize or minimize image 3805, crop image 3805, edit image 3805, and/or save image 3805 in memory, as well as other image manipulation techniques known in the art. In some embodiments, range measurements 3807 determined by camera 3804A may be presented to user 100 via user interface 3806A. Range measurements may be associated with objects or people represented in image 3805.

In some embodiments, the status of one or more audio adjustment operations may be presented to user 100 via user interface 3806A. For example, the user 100 may be presented with any one or more ongoing audio adjustment settings. In some embodiments, user interface 3806A may display options to user 100 to cancel an audio adjustment operation in progress, and/or to select a different audio adjustment operation by modifying one or more audio adjustment settings.

User interface 3806A may also be capable of receiving input from user 100 . For example, based on the output displayed on the display, the user 100 may wish to change one or more operations of the hearing aid system 3800. Input from user 100 may be processed into instructions for hearing aid system 3800 . In some embodiments, the at least one second processor may be configured to determine one or more instructions based on input from a user. In some embodiments, the mobile device 3806 can include a user interface for receiving input from a user. User input received via user interface 3806A may be wirelessly transmitted from mobile device 3806 to hearing aid device 3802 where a second processor (eg, processor 210 ) may convert the user input into a device for controlling hearing aid system 3800. or instructions for multiple operations.

In some embodiments, the wearable camera device 3804 may be configured to capture multiple images and sounds based on one or more instructions. User 100 can manually change how wearable camera device 3804 captures images through user interface 3806A. For example, user 100 may wish to focus on a person or object, so user input may result in instructions for reducing the field of view of camera 3804A and/or zooming in on one or more captured images. In another example, the image 3805 displayed by the user interface 3806A may be out of focus or blurred, and the user 100 may wish to change the focus of the camera 3804A to improve image quality, thereby instructing the camera 3804A to refocus. In yet another example, the user 100 may wish to change the lighting conditions of the image 3805 to compensate for low/high light conditions, and thus the user input may result in instructions to change the lighting conditions of the camera 3804A accordingly.

In some embodiments, the at least one first processor may be programmed to selectively condition the audio signal based on one or more instructions. For example, at least one first processor (eg, processor 210) of the wearable camera device 3804 can edit, alter, or otherwise process the sound waves 3803 captured from the user's environment. The conditioned audio signal may be provided to the hearing aid device 3802 so that a sound 3801 may be generated for the user 100 to hear. Sound 3801 may be generated based on the conditioned audio signal output by wearable camera device 3804 .

In some embodiments, selectively adjusting the audio signal may include modifying the amplitude, pitch, pitch, bass, and/or other audio effects of the sound waves 3803 . For example, user 100 may be presented with a menu-like interface (eg, an audio mix slider) on user interface 3806A, and user 100 may select one or more audio effects as desired. In some embodiments, user 100 may alter the pitch of one or more audio signals corresponding to sound waves 3803 to make the sound more perceptible to user 100 . For example, the user 100 may have less sensitivity to tones within a certain range, and the conditioning of the audio signal may include adjusting the pitch of the sound 3803. For example, user 100 may experience hearing loss in frequencies above 10 kHz, and a first processor (eg, processor 210 ) may remap higher frequencies (eg, at 15 kHz) to frequencies below 10 kHz. In some embodiments, the first processor (processor 210) may be configured to change the speech rate associated with the one or more audio signals. For example, the first processor (eg, processor 210 ) may be configured to change the language of individuals in the adjusted audio signal, for example by making each word longer in duration and reducing the silence period between consecutive words accordingly speed so that the detected speech is more perceptible to the user 100.

In some embodiments, selectively conditioning the audio signal may include classifying the sounds into different categories of sounds. For example, a first processor (eg, processor 210) may classify sound waves 3803 into segments containing music, tones, laughter, speech, screams, background noise, and the like. Indications of individual segments can be recorded in a database and can prove very useful for life-logging applications. As one example, the recorded information may enable the hearing aid system 3800 to retrieve and/or determine the user's mood when encountering another entity. Additionally, such processing may occur relatively quickly and efficiently, and may not use significant computing resources. Therefore, sending information to a destination (eg, another device, an external server, etc.) may not require significant bandwidth. Also, once some parts of the audio are classified as non-speech, more computing resources are available to process other segments. In some embodiments, user 100 may provide input via user interface 3806A to apply the different audio effects discussed above to different segments of sound wave 3803.

In some embodiments, selectively adjusting the audio signal may include attenuation of the audio signal. For example, a first processor (eg, processor 210 ) may apply one or more filters (eg, digital filters) to sound waves 3803 based on input from user 100 . In some cases, the filter can selectively attenuate audio signals (such as sound waves 3803). In some cases, the sound waves 3803 may include ambient noise (eg, various background sounds such as music, sounds/noise from people not engaged in the conversation with the user 100, etc.). The user 100 can select various filtering options so that ambient noise can be removed or attenuated from the conditioned audio signal. For example, user 100 may wish to attenuate sound waves 3803 in an environment with high background noise levels.

In some embodiments, the selective adjustment may include amplification of the audio signal. For example, a first processor (eg, processor 210) may select one or more portions of sound wave 3803 to amplify. In some embodiments, selected portions of sound waves 3803 may correspond to audio related to a conversation of user 100 with another entity, or audio from an audio source of interest to user 100 (such as TV, radio, speakers, etc.). For example, user 100 may provide input to user interface 3806A to amplify selected portions of sound waves 3803.

In some embodiments, the selective adjustment may include separating the speaker's voice from the background voice. Separation may be performed using any suitable method, such as using multiple microphones (such as those included on the wearable camera device 3804). In some cases, the at least one microphone may be a directional microphone or a microphone array. For example, one microphone can capture background noise, while another microphone can capture an audio signal that includes the background noise as well as a particular person's speech. Speech can then be obtained by subtracting background noise from the combined audio. In some embodiments, the first processor (eg, processor 210 ) can analyze the image 3805 to determine the source of the sound wave 3803 . For example, the image 3805 can help identify the object or person that generated the sound wave 3803. In some embodiments, user 100 may select from user interface 3806A an object or person to filter audio from.

In some embodiments, user 100 may provide input to selectively engage or disable various audio processing features. For example, the user 100 may provide input to selectively adjust the sound waves 3803 based on the lip reading function discussed above. For example, a person's lip movement can be captured in image 3805 and a first processor (eg, processor 210 ) can selectively amplify or attenuate sound waves 3803 based on image 3805 . In other examples, the user 100 may provide input to selectively adjust the sound waves 3803 based on the speech recognition discussed above. The first processor (eg, processor 210 ) can perform speech recognition of the velocity content sound waves 3803 and can selectively amplify or attenuate the sound waves 3803 based on whether words are recognized from the sound waves 3803 .

In some embodiments, user 100 may select a particular audio signal source for selective amplification or attenuation. For example, a first processor (eg, processor 210 ) may identify different components of sound wave 3803 and their respective sources based on analysis of image 3805 . In some examples, user 100 may interact with image 3805 displayed on user interface 3806A and select different portions of image 3805. Based on the user selection, the first processor (eg, processor 210 ) may selectively amplify portions of the sound wave 3803 emanating from the selected portion of the image 3805 and selectively attenuate one or more non-selected portions of the image 3805 other parts of the sound wave 3803 emitted by certain parts. In some embodiments, the first processor may amplify sound obtained from an area within the user's 100 field of view or a portion of the user's 100 field of view.

In some embodiments, the at least one second processor may receive the conditioned audio signal from the wearable camera device. The conditioned audio signal may be sent wirelessly from the wearable camera device 3804 to the hearing aid device 3802 via a pairing connection.

In some embodiments, the at least one second processor may provide sound to the user's ear using the at least one speaker based on the conditioned audio signal. At least one speaker of the hearing aid device 3802 may generate sound 3801 to the ear of the user 100 .

40 illustrates a flowchart of an exemplary process 4000 for a hearing aid and paired camera system consistent with the disclosed embodiments. In some embodiments, the hearing aid and paired camera system may be the system 3800 shown in FIGS. 38 and 39 that includes a hearing aid device 3802 , a wearable camera device 3804 and/or a mobile device 3806 .

In step 4002 , the hearing aid device 3802 and the wearable camera device 3804 may be paired to communicate with each other, and/or with the mobile device 3806 . Examples of wireless pairing include Wi-Fi, Bluetooth, NFC, and other similar wireless communication technologies. In some embodiments, pairing may be initiated by user 100 using mobile device 3806. For example, when the hearing aid device 3802 and/or the wearable camera device 3804 are within communication range with the mobile device and/or each other, a notification may be presented to the user 100 on the mobile device 3806 allowing the user 100 to select pairing of the devices. In some embodiments, the pairing between the hearing aid device 3802 , the wearable camera device 3804 , and/or the mobile device 3806 may be terminated by the user 100 on the mobile device 3806 . In other embodiments, pairing may be initiated automatically (eg, when the hearing aid device 3802 and/or the wearable camera device 3804 are within communication range with the mobile device 3806 and/or each other).

In step 4004, mobile device 3806 may generate or display user interface 3806A. User interface 3806A may be configured to receive visual, audio, haptic, or any other suitable signals from user 100 . For example, user interface 3806A may be displayed on a display that may be part of mobile device 3806, such as including GUI elements that may be manipulated by user gestures or by suitable physical or virtual (ie, on-screen) devices (eg, keyboard, mouse, etc.) touch screen. In some embodiments, user interface 3806A may be an audio interface capable of receiving user 100 audio input (eg, user 100 speech input) for adjusting one or more parameters of system 3800 . The audio interface may be provided by the mobile device 3806, which may include a microphone.

In some embodiments, user interface 3806A may also present information related to hearing aid system 3800 , such as information related to the operation of hearing aid device 3802 and/or wearable camera device 3804 . Such information can be used to inform the user 100 of the status of the hearing aid system 3800 so that the user 100 can control parameters of the hearing aid device 3802 and/or the wearable camera device 3804. For example, user 100 may initiate or terminate a pairing between mobile device 3806, wearable camera 3804, and/or hearing aid device 3802 through user interface 3806A.

In some embodiments, images captured by wearable camera device 3804, such as image 3805, may be displayed to user 100 in real-time via user interface 3806A. Image 3805 may be presented in a manner that can be manipulated by user 100 . For example, user 100 may zoom in/out to maximize or minimize image 3805, crop image 3805, edit image 3805, and/or save image 3805 in memory, as well as other image manipulations known in the art. In some embodiments, the range measurements determined by camera 3804A may be presented to user 100 via user interface 3806A. Range measurements may be associated with image 3805.

In step 4006, mobile device 3806 may receive input from user 100 via user interface 3806A. In some embodiments, the status of the audio adjustment operation may be presented to the user 100 via the user interface 3806A. For example, the user 100 may be presented with any ongoing audio adjustment settings. In some embodiments, the user interface 3806A may display options to the user to cancel an audio adjustment operation in progress, and/or to select a different audio adjustment operation by modifying one or more audio adjustment settings.

In step 4008, the mobile device 3806 may transmit the user input received via the user interface 3806A to the hearing aid device 3802 or the wearable camera device 3804. At least one second processor (eg, processor 210 ) of hearing aid device 3802 or wearable camera device 3804 may be programmed to determine instructions for system 3800 based on user input. The instructions may then be executed by components of the hearing aid device 3802 and/or by the wearable camera device 3804.

In step 4010, the wearable camera device 3804 may capture audio (such as sound waves 3803) from the user's environment. The wearable camera device 3804 can use at least one microphone to generate audio signals based on the received sound waves 3803.

In step 4012, wearable camera device 3804 may capture a plurality of images (such as image 3805). As shown in FIGS. 38 and 39 , the wearable camera device 3804 can use the camera 3804A to capture real-time image data of the user's 100 field of view. Camera 3804A can capture images of objects and people, some of which can generate sound waves 3803 that are received by one or more microphones located in wearable camera device 3804.

In step 4014, the wearable camera device 3804 may determine a range measurement 3807. For example, the wearable camera device 3804 may use a rangefinder to determine the range (or distance) between an object or person and the wearable camera device 3804 . In some embodiments, the angle relative to the user's gaze direction may be determined.

In step 4016, the wearable camera device 3804 may modulate the sound waves 3803. Adjusting may include operations performed by at least one first processor (eg, processor 210) to modify the pitch, pitch, bass, and/or other audio effects of sound waves 3803; classifying sounds into different sound categories; and/or cause amplification of the sound wave 3803. The adjustments may be based on instructions sent from the hearing aid device 3802 and/or the mobile device 3806 , which in turn are generated based on input from the user 100 .

For example, user 100 may be presented with a menu-like interface (eg, an audio mix slider) on user interface 3806A, and user 100 may select one or more audio effects as desired. In some embodiments, changing the pitch of one or more audio signals corresponding to sound wave 3803 may make the sound more perceptible to user 100 . For example, the user 100 may have less sensitivity to tones within a certain range, and the adjustment of the audio signal may adjust the pitch of the sound 3803. For example, user 100 may experience hearing loss in frequencies above 10 kHz, and a first processor (eg, processor 210 ) may remap higher frequencies (eg, at 15 kHz) to frequencies below 10 kHz. In some embodiments, the first processor (processor 210) may be configured to change the speech rate associated with the one or more audio signals. The first processor (eg, processor 210 ) may be configured to change the speech rate of the individuals in the adjusted audio signal to make the detected speech more perceptible to the user 100 .

For example, a first processor (eg, processor 210) may classify sound waves 3803 into segments containing music, tones, laughter, speech, screams, background noise, and the like. Once some parts of the audio are classified as non-speech, more computing resources are available to process other segments. In some embodiments, user 100 may provide input via user interface 3806A to apply the different audio effects discussed above to different segments of sound wave 3803.

For example, a first processor (eg, processor 210 ) may apply one or more filters (eg, digital filters) to sound waves 3803 based on input from user 100 . In some cases, the filter can selectively attenuate audio signals (such as sound waves 3803). In some cases, the sound waves 3803 may include ambient noise (eg, various background sounds such as music, sounds/noise from people not engaged in the conversation with the user 100, etc.). The user 100 can select various filtering options so that ambient noise can be removed or attenuated from the conditioned audio. For example, user 100 may wish to attenuate sound waves 3803 in the environment associated with background noise.

For example, a first processor (eg, processor 210) may select one or more portions of sound wave 3803 to amplify. In some embodiments, selected portions of sound waves 3803 may correspond to audio related to a conversation of user 100 with another entity, or audio from an audio source of interest to user 100 (such as TV, radio, speakers, etc.). For example, user 100 may provide input to user interface 3806A to amplify selected portions of sound waves 3803. The separation of the speaker's speech from the background sound may be performed using any suitable method, such as using multiple microphones (such as those included on the wearable camera device 3804). In some cases, the at least one microphone may be a directional microphone or a microphone array. For example, one microphone can capture background noise, while another microphone can capture an audio signal that includes the background noise as well as a particular person's speech. Speech can then be obtained by subtracting background noise from the combined audio. In some embodiments, the first processor (eg, processor 210 ) may utilize image 3805 to assist user 100 in determining the source of sound wave 3803 . For example, the image 3805 can be analyzed to identify the object or person generating the sound wave 3803, and this can then be displayed to the user 100 via the user interface 3806A.

For example, user 100 may provide input for selective adjustment of sound waves 3803 based on the lip reading function discussed above. Human lip movements can be captured in image 3805 and a first processor (eg, processor 210 ) can selectively amplify or attenuate sound waves 3803 based on image 3805 . In some other examples, the user 100 may provide input to selectively adjust the sound waves 3803 based on the speech recognition discussed above. The first processor (eg, processor 210 ) can perform speech recognition of the content sound waves 3803 and can selectively amplify or attenuate the sound waves 3803 based on whether words are recognized from the sound waves 3803 .

For example, a first processor (eg, processor 210 ) can identify the different components of sound wave 3803 and their respective sources as they are captured in image 3805 . User 100 may interact with image 3805 displayed on user interface 3806A and select different portions of image 3805. Based on the user selection, the first processor (eg, processor 210 ) may selectively amplify portions of the sound wave 3803 emanating from selected portions of the image 3805 and selectively attenuate portions of the sound waves 3803 emanating from non-selected portions of the image 3805 Other parts of Sonic 3803.

In step 4018 , the wearable camera device 3804 may provide the conditioned audio signal to the hearing aid device 3802 . For example, the conditioned audio signal may be sent wirelessly from the wearable camera device 3804 to the hearing aid device 3802 via a pairing connection. Transmission of the conditioned audio signal may include transmission over one or more networks and using one or more transmission protocols.

In step 4020 , one or more speakers of hearing aid device 3802 may generate sound 3801 to one or more ears of user 100 .

For example, in some embodiments, the at least one second processor may receive the conditioned audio signal from the wearable camera device, and may provide sound to the user's ear using the at least one speaker based on the conditioned audio signal. At least one speaker of the hearing aid device 3802 may generate sound 3801 to one or both ears of the user 100 .

Adaptive capture rate

In some embodiments, the hearing aid system may have an adaptive capture rate. For example, parameters associated with a microphone and/or a camera associated with the hearing aid system may be adjusted based on a particular situation or context. For example, a hearing aid system may analyze images captured by a camera and/or sounds captured by a microphone to determine that certain parameters should be changed. Depending on the situation or context of the user of the hearing aid system, different parameters can be optimized. For example, the hearing aid system may increase the capture rate (eg, frames per second) of the camera when the user is interacting with a fast-speaking individual. This may allow the hearing aid system to more efficiently analyze the captured image of the speaker (eg, for detecting lip movements, etc.). In situations where the individual is speaking slowly, or when there are no active speakers, the system can reduce the capture rate of the camera. This may be beneficial, for example, to reduce the power consumption of the camera device, reduce the amount of memory used, and so on. Other sensors or information can also be used to control parameters of the camera or microphone, such as location information, detected light levels, time of day, etc.

The disclosed hearing aid system can selectively amplify sound. In one embodiment, a hearing aid system may include a wearable camera, a hearing aid, and at least one microphone. Wearable cameras can refer to devices with image, sound, audio and/or video capture capabilities that can be attached to the user or to the user's clothing or accessories. A hearing aid may refer to a device that outputs audio or sound to a user's ear. The output of the hearing aid may be generated based on input received from or by the wearable camera and/or at least one microphone. A hearing aid system may include several devices that are paired together to provide improved functionality. For example, a hearing aid system may include a hearing aid for providing sound to a user's ear, wherein the sound may be captured acoustically by the hearing aid or received electronically from another source such as a wearable camera or at least one microphone.

In some embodiments, the wearable camera may be configured to capture multiple images from the user's environment, and the wearable camera may have image capture parameters. A camera may refer to a component or device capable of receiving light from a person, object, and/or environment, and forming an image or video based on the received light. The plurality of images may be video clips containing a plurality of still images (referred to as frames). In some embodiments, the hearing aid system may include at least one microphone configured to capture sound from the user's environment. A microphone may be a component or device capable of receiving sound waves and generating audio signals based on the received sound waves.

As an example, FIG. 41 shows a hearing aid system including a wearable camera 4104 and a hearing aid 4102. In some embodiments, the hearing aid device 4102 and the wearable camera device 4104 are paired to communicate with a mobile device (not shown) having a graphical user interface (GUI). In some embodiments, the hearing aid 4102 can be paired with the wearable camera 4104, and data can be communicated between the paired devices. In some embodiments, the wearable camera 4104 may correspond to the device 110 shown in Figures 4A-4K. In some alternative embodiments, the wearable camera device 4104 may correspond to the apparatus 110 shown in Figures 3A and 3B. Hearing aid 4102 may correspond to auditory interface device 1710 .

Wearable camera 4104 may include an image sensor (eg, image sensor 220) for capturing real-time image data substantially corresponding to a field of view from a user. For example, wearable camera 4104 may refer to a device capable of detecting and converting light signals in the near-infrared, infrared, visible, and ultraviolet spectrums into electrical signals. The electrical signals may be used to form images or video streams (ie, image data) based on the detected signals. The term "image data" includes any form of data retrieved from optical signals in the near infrared, infrared, visible and ultraviolet spectrums. Examples of image sensors may include semiconductor charge coupled devices (CCDs), active pixel sensors in complementary metal oxide semiconductors (CMOS), or N-type metal oxide semiconductors (NMOS, active MOS).

According to the example shown in FIG. 41 , the user 100 may wear a wearable camera 4104 and a hearing aid 4102 . As shown, user 100 may wear wearable camera 4104 that is physically attached to user 100's shirt or other clothing. Consistent with the disclosed embodiments, the wearable camera 4104 may be placed in other locations such as attached to necklaces, belts, glasses, wristbands, buttons, and the like. As shown in Figure 41, a hearing aid 4102 may be placed in one or both ears of the user 100, similar to a conventional auditory interface device. The hearing aid 4102 may be of various styles, including in-canal, fully in-canal, in-ear, behind-the-ear, supra-ear, in-canal receiver, open mount, or various other styles. Hearing aid 4102 may include one or more speakers for providing auditory feedback to user 100 .

The user's environment generally refers to the surrounding environment of the user who is using the wearable camera device 4104 . The user's environment may include objects and people, some of which may generate sound waves that are received by one or more microphones (not shown) located in the wearable camera device 4104 . Depending on the orientation and field of view of the wearable camera 4104, the wearable camera 4104 may capture images that include representations of objects and people in the user's environment as seen by the wearable camera 4104 along the optical axis 4103.

The wearable camera can include at least one processor. The term "processor" includes any physical device having circuitry that performs logical operations on inputs. For example, a processing device may include one or more integrated circuits, microchips, microcontrollers, microprocessors, central processing units (CPUs), graphics processing units (GPUs), digital signal processors (DSPs), field programmable gates All or part of an array (FPGA), or other circuitry suitable for executing instructions or performing logical operations. In some embodiments, the processor 210 shown in FIGS. 5A-5C may be an example of at least one processor.

In some embodiments, the at least one processor may receive a plurality of images captured by the wearable camera. In some embodiments, the at least one processor may receive audio signals representing sounds captured by the at least one microphone. The audio signal may represent sounds emanating from the user's environment.

By way of example, Figure 42 shows a hearing aid system that captures images and audio from a user's environment. The area, region and/or space around the user 100 may constitute the environment of the user 100 . In some embodiments, the wearable camera 4104 may have a field of view defined by the cone 4203 shown in FIG. 42 along the optical axis 4103. The width of the cone 4203 may be a property of the wearable camera 4104 as defined by its components or settings, such as a lens or aperture, zoom, etc. Within the cone 4203, there may be a person or object (such as person 4200) within the view of the wearable camera 4104. In some embodiments, the wearable camera 4104 can capture the image 4214. Image 4214 may include representations of people or objects within the view of pyramid 4203 . In some embodiments, image 4214 is an image of a person, such as person 4200, and person 4200's face can be imaged such that person 4200's lips 4214a and lip movement can be seen in image 4214.

In some embodiments, sound waves can be emitted from within cone 4203 , such as sound waves 4202 generated by person 4200 . In some embodiments, other sound waves may be emitted from the exterior of cone 4203, such as sound waves 4204a and/or sound waves 4204b. Thus, the audio signal captured by the at least one microphone may be from a person or object captured in image 4214. In some embodiments, sound waves not generated by persons or objects from within cone 4203 may be considered background sounds. In some embodiments, sound waves may be characterized by various physical properties, such as amplitude, frequency, pitch, pitch, and the like.

The processor 210 may separate the speech of the person 4200 from background sounds by performing any suitable method, such as by using multiple microphones, such as those included on the wearable camera 4104. In some cases, the at least one microphone may be a directional microphone or a microphone array. For example, one microphone may capture background noise (eg, sound waves 4204a and/or sound waves 4204b), while another microphone may capture audio that includes background noise (eg, sound waves 4204a and/or sound waves 4204b) and a particular person's speech (sound waves 4204) Signal. Speech can then be obtained by subtracting background noise from the combined audio. In some embodiments, a processor (eg, processor 210 ) can analyze image 4214 to determine the source of sound wave 4202 . For example, sound waves 4202 may contain speech 4212 spoken by person 4200. Speech 4212 may match the movement of lips 4214a seen in image 4214.

In some embodiments, the hearing aid system may change or adjust the image capture parameters of the wearable camera 4104. Image capture parameters may refer to operational settings, parameters, conditions, and/or other factors that characterize one or more operations of the wearable camera 4104 . For example, adjusting image capture may increase the performance characteristics of the wearable camera 4104, while reducing the performance characteristics of the wearable camera 4104 may reduce the power consumption of the wearable camera 4104, or may reduce the amount of memory used, or the like.

In some embodiments, the image capture parameter may be the frame rate of the camera. A frame may refer to a single image of multiple images, such as video. Therefore, frame rate can refer to the number of images per unit of time. In some embodiments, the frame rate of the wearable camera 4104 may refer to the number of images captured per unit of time when the wearable camera captures a video clip. For example, if the wearable camera 4104 captures video at 100 frames per second (fps), then when the wearable camera 4104 captures video, 100 still images are captured per second. The camera's frame rate can affect the quality of the captured video clips. For example, a camera using a higher frame rate can capture more images during a given time frame, and a higher number of images can increase video quality, e.g. provide more motion, than a camera using a slower frame rate detail. For example, in situations or contexts where an object of interest, such as person 4200, exhibits rapid or rapid motion, it may be more desirable for wearable camera 4104 to operate at a higher frame rate. Alternatively, higher frame rates may not be optimal in certain situations or contexts. For example, when an object of interest (such as person 4200) does not exhibit fast or rapid motion, a video with a slower frame rate (or a lower number of still images) may not significantly impair video quality. Conversely, having the camera operate at a slower frame rate can consume less power and the captured images can require less memory to store.

In additional or alternative embodiments, the image capture parameters may include the resolution of the captured image, the compression rate of the captured image, and/or parameters for optimizing the compression quality of the captured audio signal.

In some embodiments, image capture parameters may be adjusted based on the detected speech rate. For example, speech rate can refer to the pace or speed of speech or vocalization. In some cases, the rate of speech may be related to the movement of the lips of an individual such as person 4200 . For example, a high speech rate might imply rapid lip movement, while a low speech rate might imply slower lip movement. In some embodiments, the hearing aid system may include features such as a lip tracking algorithm, and the plurality of captured images may be used by the lip tracking algorithm.

Additionally or alternatively, image capture parameters may be adjusted based on detected light levels. For example, higher frame rate settings may require more light than lower frame rate settings. The processor 210 may determine an optimal frame rate for the wearable camera 4104 under certain environmental conditions (eg, certain lighting conditions) based on one or more light sensor readings (eg, from the wearable camera 4104).

Additionally or alternatively, image capture parameters may be adjusted based on location information. For example, user 100's environment may be determined based on user 100 location information (eg, GPS information, location information determined from analyzing captured images and/or audio, location information provided by user 100, etc.). The processor 210 may determine factors relevant to adjusting the frame rate of the wearable camera 4104 based on the location information, such as whether the user 100 is in a busy location, the number and type of objects used for imaging at the location, the number of electrical outlets used for recharging. Availability, or other such factors that may determine optimal performance of the wearable camera 4104 and/or optimal battery life management. In some embodiments, the processor 210 may receive location information based on GPS coordinates of the hearing aid system or input from the user 100 . In some embodiments, the location information may be a user setting selectable by the user 100 . For example, the wearable camera 4104 may be configured to operate at a predetermined frame rate based on location-based user settings, such as in urban environments, rural environments, crowded locations, sparse locations, low lighting environments, and the like.

Additionally or alternatively, image capture parameters may be adjusted based on user settings. For example, the user 100 may increase or decrease the frame rate of the wearable camera 4104 as needed based on the circumstances of the particular situation. In some embodiments, the user 100 may select predefined settings for the wearable camera 4104 (eg, via a verbal command or via the paired device's user interface) and cause adjustments to image capture parameters. For example, the wearable camera 4104 may be configured to allow the user 100 to select among several settings, each programmed with a frame rate of the wearable camera 4104 . For example, user 100 may select a power saving setting to save power, causing wearable camera 4104 to operate at a slower frame rate. For example, user 100 may select a high performance setting to maximize video quality, which may result in wearable camera 4104 operating at a higher frame rate. In some embodiments, the wearable camera 4104 can adjust the frame rate based on the identity of the speaker. For example, based on previous interactions, person 4200 may be known to have a specific speech rate, and wearable camera 4104 may automatically select a frame rate for wearable camera 4104 whenever person 4200 comes into view of wearable camera 4104. Alternatively, user 100 may select certain frame rate settings based on persons previously detected by wearable camera 4104 .

43 shows a flowchart of an exemplary process for adjusting capture parameters of a wearable camera consistent with disclosed embodiments.

In step 4302, at least one processor (eg, processor 210) may receive a plurality of images captured by the camera. The multiple images may be still images or video clips containing multiple still images. The images may include objects or people in the field of view of the wearable camera 4104 . For example, wearable camera 4104 may capture image 4214 showing the face of person 4200 located within cone 4203. In some embodiments, image 4214 may include an image of person 4200's lips 4214a or movement thereof.

In step 4304, at least one processor (eg, processor 210) may receive an audio signal representing sound captured by at least one microphone. For example, wearable camera 4104 may capture sound waves emanating from within cone 4203 , such as sound waves 4202 generated by person 4200 . Wearable camera 4104 may also capture other sound waves emanating from outside of cone 4203, such as sound waves 4204a and/or sound waves 4204b.

In step 4306, at least one processor (eg, processor 210) may identify a representation of at least one of the at least one of the plurality of images. In some embodiments, processor 210 may be configured to execute one or more image classification algorithms to identify whether a person is present in image 4214. In some embodiments, processor 210 may execute a facial recognition program or algorithm to recognize one or more faces within image 4214. For example, processor 210 may identify facial features on the face of person 4200 captured in image 4214, such as eyes, nose, cheekbones, chin, lips, or other features. The processor 210 may analyze the detected features using one or more algorithms, such as principal component analysis (eg, using eigenfaces), linear discriminant analysis, elastic bundle map matching (eg, using Fisher faces), local binary patterns Histogram (LBPH), Scale Invariant Feature Transform (SIFT), Accelerated Robust Feature (SURF), etc. Other facial recognition techniques such as three-dimensional recognition, skin texture analysis, and/or thermal imaging may also be used to identify individuals. Features other than facial features can also be used for identification, such as height, body size, or other distinguishing features of the person 4200 .

In step 4308, at least one processor (eg, processor 210) may detect a speech rate associated with the at least one individual. At least one processor (eg, processor 210 ) can detect a speech rate associated with the at least one based on the captured plurality of images or audio signals, or both.

In some embodiments, at least one processor (eg, processor 210) may identify at least one lip movement associated with the mouth of at least one individual based on analysis of the plurality of images. Processor 210 may identify at least one lip movement or lip position associated with the individual's mouth based on analysis of the plurality of images (eg, image 4214). The processor 210 may be configured to identify one or more points associated with the individual's mouth. In some embodiments, the processor 210 may develop a profile associated with the individual's mouth, which may define a boundary associated with the individual's mouth or lips. Lips identified in an image can be tracked over multiple frames or images to identify lip movement. Accordingly, the processor 210 may use various video tracking algorithms as described above. For example, processor 210 may identify lips 4214a of person 4200 within image 4214, and may analyze its motion. In some cases, processor 210 may identify correlations between particular facial expressions, such as lip movements, and particular sounds or sound fluctuations. For example, a facial expression associated with a particular lip movement may be associated with sounds or words that may have been spoken during the conversation captured in the first audio signal. In some embodiments, the analysis of the plurality of images is performed by a computer-based model, such as a trained neural network. For example, a trained neural network may be trained to receive image and/or video data associated with an individual's facial expressions, and to predict sounds associated with the received image and/or video data. As another example, a trained neural network may be trained to receive image and/or video data related to an individual's facial expression and voice, and output whether the facial expression corresponds to the voice. In some embodiments, other factors may be identified in one or more images captured by the wearable camera, such as the individual's gestures, the individual's location, the orientation of the individual's face, and the like. By associating lip movements with predicted sounds or words, processor 210 can determine the number of words or sounds associated with lip movements captured in image 4214 over a period of time. At least one processor (eg, processor 210 ) can determine the speech rate based on the lip movement.

In some embodiments, at least one processor (eg, processor 210) can analyze the received audio. For example, the processor 210 may identify speech 4212 associated with the sound wave 4202, the speech 4212 comprising the speech of the person 4200. In some embodiments, processor 210 may analyze sound received from the microphone of wearable camera 4104 to separate sound waves 4202 from sound waves 4204a and 4204b using any currently known or future developed techniques or algorithms. For example, processor 210 may receive audio signals representing sounds emanating from objects in the environment of user 100 and analyze the received audio signals to obtain an isolated audio stream associated with a sound emitting object.

Based on the audio analysis, at least one processor (eg, processor 210) can identify a plurality of words spoken by the individual. Processor 210 may be configured to identify words spoken by individual 4200. For example, the processor 210 may analyze the sound waves 4202 to identify particular phonemes, phoneme combinations, or words in the sound waves 4202. In some embodiments, processor 210 may use various speech-to-text algorithms to recognize spoken words. In some embodiments, recognizing the plurality of words may include using a speech recognition algorithm. Speech recognition may refer to the ability of a machine, processor or program to receive and interpret sound, speech, dictation or the like. For example, speech recognition algorithms may be executed by the processor 210 to recognize or interpret words or commands received in the voice. Examples of speech recognition algorithms can be implemented through AI, deep learning algorithms, neural embedding models, or other methods known in the art. At least one processor (eg, processor 210) may determine the speech rate based on the plurality of words.

In step 4310, at least one processor (eg, processor 210) may adjust one or more image capture parameters of the wearable camera based on the detected speech rate. In one example, the processor 210 may determine a high speech rate when the processor 210 detects that the amount of lip movement of the person 4200 is greater than a threshold. The processor 210 may increase the frame rate of the wearable camera 4104 when the lip movement of the person 4200 is high. This may allow the hearing aid system to maintain the accuracy of the lip tracking algorithm by ensuring that a sufficient number of lip movement images are captured. In another example, when processor 210 detects a speech rate that is less than a threshold, processor 210 may determine that a low amount of lip movement of person 4200 has been detected. The processor 210 may reduce the frame rate of the wearable camera 4104 when the lip movement of the person 4200 is low. This can reduce power usage and/or memory storage usage without compromising the accuracy of the lip tracking algorithm.

At least one microphone may have one or more parameters for capturing audio signals (particularly speech), and may also be adjusted according to the rate of speech or other parameters or settings. In some embodiments, the at least one processor may also be programmed to cause adjustments to one or more audio capture parameters of the at least one microphone based on the detected rate of speech. Examples of audio capture parameters may include pitch, pitch, amplitude, sensitivity level, frequency, and/or sampling rate. For example, the at least one processor may apply a filter to the audio signal arriving at the at least one microphone to selectively capture the audio signal having a particular tone, pitch or frequency. As another example, the at least one processor may amplify or attenuate the audio signal arriving at the at least one microphone to change the amplitude of the captured audio signal, and/or change the sensitivity of the at least one microphone.

In some embodiments, at least one processor (eg, processor 210 ) may adjust the sampling rate of at least one microphone based on the rate of speech detected in step 4308 . The sampling rate may refer to the time frequency at which the device samples the signal. For example, an audio signal recorded (ie, sampled) at a higher sampling rate will include more audio signal than one recorded at a lower sampling rate. For example, when the processor 210 determines that the detected speech rate is above a threshold, the processor 210 may increase the sampling rate of at least one microphone to record more sound waves 4202 per unit of time. This may be desirable because when speech rates are high, more words can be spoken per unit time, and lower sampling rates may affect features such as word/speech recognition.

Alternatively, for example, when the processor determines that the detected speech rate is below a threshold, the processor may reduce the sampling rate of at least one microphone to record fewer sound waves 4202 per unit of time. When speech rates are low, the lower sampling rates can still adequately capture enough detail to preserve the integrity of features such as word and/or speech recognition.

Handling audio signals with variable audio quality

As described above, audio signals captured from within the user's environment may be processed prior to presentation of the audio to the user. The processing may include various adjustments or enhancements to improve the user's experience. For example, as described above, speech from the individual the user is looking at may be amplified, while other audio such as background noise, speech from other speakers, etc. may be muted or attenuated. Therefore, the user can more easily hear and understand the speech from the individual with whom the user is conversing.

The quality and/or effectiveness of this processing may depend on various aspects or variables of how the audio signal is captured and/or processed. These aspects can lead to various tradeoffs between the quality of the processed audio signal and other factors such as audio latency, battery life, and the like. For example, many audio processing techniques use buffers of collected audio to perform processing. Specifically, the system can analyze samples captured before and after the audio sample currently being processed to gather additional information that may improve sample processing. For example, the system could use a sliding window of 0-30 seconds of accumulated audio. The sliding window may include samples preceding the processed samples, but may also include "lookahead" samples that follow the processed samples. Longer time windows provide a larger amount of audio sample data, resulting in higher quality output.

However, longer look-ahead means longer latency. Due to the longer look-ahead period, in order to process a given sample, the system has to wait longer until future samples are collected, which inevitably creates a delay in processing samples. For example, if the desired look-ahead period is one second, the audio output by processing the current sample will be delayed by at least one second. In some cases, this delay may be unpleasant or distracting to the user. For example, when speaking to another body, the speech from that individual may not match his or her lip movement. Also, delays can create uncomfortable pauses in conversations. Therefore, there is a trade-off between the time delay of processing the audio and the quality of the audio processing performed. Similar tradeoffs may also be associated with other aspects of audio signal capture and processing. For example, audio quality may depend on other aspects, such as whether a camera is used to help identify active speakers, the number of microphones used, etc. These variables can create similar tradeoffs for audio quality versus processing latency, audio quality versus battery consumption, or other tradeoffs.

The disclosed systems and methods can determine how to balance these tradeoffs in various ways. In some embodiments, the user may manually adjust one or more settings to balance these tradeoffs. Different users may have different preferences and thus different settings regarding audio quality may be applied. For example, some users may be more tolerant of lower audio processing quality and may therefore prefer shorter delay times. Other users may rely more on audio processing to hear and therefore may be more tolerant of delays. Furthermore, the same user may have different preferences in different situations. For example, in face-to-face meetings with low background noise, a shorter time delay may be preferred. On the other hand, users may prefer higher audio processing quality when in a noisy multi-speaker environment.

In other embodiments, the system may automatically adjust one or more settings to achieve the best compromise in audio quality. For example, the user can enter feedback on audio quality or time delay, and the system can adjust one or more settings. In some embodiments, the system may process audio signals in parallel according to multiple schemes and determine which scheme provides the best compromise. For example, the system may perform a first process with a short delay and a second process with a longer delay, and the resulting processed audio signals may be compared to determine which approach provides the best results. Thus, the disclosed embodiments may provide better efficiency, convenience and functionality than prior art hearing aid devices.

FIG. 44 shows an example audio signal 4410 that may be processed in accordance with the disclosed embodiments. As described above, audio signal 4410 may be captured by one or more microphones of wearable device 110, such as microphone 443 or 444. In some embodiments, audio signal 4410 may be received from multiple microphones, such as a microphone array. Audio signal 4410 may include a representation of sounds from the environment of the user of wearable device 110 . The audio signal may thus include speech from one or more individuals, background noise, music, and/or other sounds that the wearable device 110 processes before presenting it to the user. For example, the system may selectively adjust audio samples 4410 to attenuate background noise, amplify sound from a particular source (eg, an object or person the user is looking at), adjust the pitch of the audio signal, adjust the playback rate of the audio signal, Remove noise or artifacts from the signal, perform audio compression, or perform other enhancements to improve the user's audio quality.

As mentioned above, the quality of the processed audio signal depends on various factors, including the time delay allowed for additional audio samples to be collected and analyzed after the audio sample currently being analyzed. For example, as shown in FIG. 44, the system may process audio samples 4412 included within audio signal 4410. The processing may include any of the various enhancements or adjustments throughout this disclosure. The disclosed system may also analyze previous and/or subsequent audio samples to improve processing of the current sample. As shown in Figure 44, this may include one or more "look-ahead" samples 4414. Increasing the amount of information included in the look-ahead samples 4414 may result in higher quality audio processing, as the system may be better able to determine the progress of the audio signal. For example, given a larger range of sample data, the system may be able to reduce background or other noise from audio signal 4410 more effectively. In order to process audio samples 4412, the system may introduce a time delay (t) to allow time for look-ahead samples 4414 to be processed. Therefore, a larger time delay (t) may allow for increased audio quality to the processed audio signal. The time delay experienced by the user may be greater than the time delay (t) shown in FIG. 44 . For example, the actual time delay experienced by the user may include additional delays for processing the audio samples 4412, sending them to the hearing aid device, or other steps that may increase the delay. Although Figure 44 shows a single look-ahead sample 4414, it should be understood that this may include more than one sample, and thus the look-ahead sample 4414 may be divided into multiple samples.

The appropriate or desired balance between time delay and audio quality can be determined in various ways. In some embodiments, the time delay may be defined based on input from the user. For example, a user may provide input indicating a preference for higher audio quality or a preference for shorter processing delays. As used herein, audio quality can refer to any measure of how effectively a system processes an audio signal. In some embodiments, audio quality may refer to the degree to which a particular form of adjustment or enhancement is applied. For example, if selective conditioning of the audio signal is performed to attenuate background noise relative to individual speech, audio quality may refer to the degree to which background noise is attenuated, the degree to which the system distinguishes between speech and background noise, the clarity of speech, the ability of the system to The degree to which the speech of a particular individual is recognized or the degree to which the speech is amplified. Audio quality can also refer to more general properties of the resulting audio signal, such as the sampling rate, how much noise is in the signal, and so on.

FIG. 45A illustrates an example user interface 4510 through which a user may define aspects of processing audio signals, consistent with the disclosed embodiments. User interface 4510 may be a graphical user interface displayed on computing device 4500, as shown in Figure 45A. Computing device 4500 may be any device associated with wearable device 110 and/or auditory interface device 1710 through which a user may provide input. For example, computing device 4500 may include mobile phones, tablets, laptops, desktop computers, televisions, wearable devices (eg, smart watches, smart jewelry, etc.), home IoT devices, and the like. In some embodiments, computing device 4500 may correspond to computing device 120 described above. In some embodiments, user interface 4510 may be presented on wearable device 110 .

User interface 4510 may include one or more controls through which a user may provide input. For example, user interface 4510 may include one or more slider controls 4512 and 4514, as shown in FIG. 44 . Using the slider control 4512, the user can indicate a preference regarding the tradeoff between audio processing delay and audio quality. Dragging the slider control 4512 to the left reduces the time delay, thereby limiting the amount of look-ahead audio available for processing. Alternatively, the user can drag the slider control 4512 to the right, which can increase the time delay to produce a higher quality processed audio signal. User 4512 may access user interface 4510 when he or she feels that the balance between time delay and audio quality is not ideal. For example, in the case of a user talking one-on-one with an individual with minimal background noise, audio quality may not be as important, and thus, the user may prefer to minimize time delays. However, in other environments, such as crowded restaurants, the user may have hearing difficulties and therefore may be more tolerant of delays to improve the quality of the audio processing.

In some embodiments, slider controller 4512 may control other aspects of how audio is processed in addition to audio sample time delay. For example, factors such as the number of microphones used to capture audio, whether image processing is performed to enhance the processing of the audio, or any other variable that affects processing time, can also be adjusted or controlled by slider controller 4512. In some embodiments, separate controls may also be provided for each of these variables. For example, user interface 4510 may include one or more checkboxes, radio buttons, switches, or similar controls that allow the user to implement enhanced processing through the use of a camera or the like. In some embodiments, user interface 4510 may allow the user to control other aspects of audio processing. For example, slider controller 4514 may allow a user to define a preference between battery life (eg, wearable device 110, computing device 120, or auditory interface device 1710) and audio processing quality. For example, using a camera to enhance the processing of audio signals (eg, performing lip tracking techniques, determining active speakers, etc.) may drain the wearable device 110 battery faster, so the user can use the slider control 4514 to manage or define This compromise. In some embodiments, this may be presented as a binary option, such as an option to enter battery saver mode. Furthermore, although slider controls 4512 and 4514 are shown in FIG. 45A as an example, various other forms of controls may be used. For example, a user can type a value in a text field, which can correspond to a time delay (eg, in seconds or milliseconds), a range or scale that defines the time delay (eg, 0-5, 0-100, etc.) value, percentage, or any other value that can indicate a time delay preference. Controls can also include checkboxes, radio buttons, drop-down lists, buttons, drop-down buttons, toggle switches, icons, and more.

In some embodiments, time delays or other aspects of how to process audio may be specified based on feedback from the user. Rather than directly controlling one or more variables through the user interface 4510, the user can provide feedback on the processed audio, which the system can use to adjust one or more aspects that affect audio quality. For example, the system may provide a prompt to the user asking "How is the sound quality?" and provide the user with one or more response options (eg, select the number of stars or otherwise select a numerical rating, select "thumbs up" or "big" Thumbs Down" option, select options such as "Unclear Speech" or "Audio Delay"). Based on the response, the system can be configured to adjust one or more aspects of audio processing. Feedback from users can be obtained in various other ways. For example, the system can detect the user's actions, which can indicate feedback from the user. In some embodiments, the feedback may be explicit. For example, the user can make a thumb up or down gesture that can be recognized by the system. In some embodiments, feedback may also be implicit. For example, the system can detect, based on images or captured audio, if the user is leaning toward the speaker, placing his or her hands around their ears, asking the speaker to repeat themselves, or other actions that may indicate the user is having difficulty hearing, which The system may be prompted to adjust one or more aspects of audio processing.

According to some embodiments of the present disclosure, the system may be configured to automatically and dynamically adjust various aspects of audio processing. For example, the system may analyze the captured or processed audio signal to determine the appropriate number or duration of look-ahead samples. In some embodiments, this may include processing the captured audio signals in parallel according to different setups or setup schemes. The system can then compare the resulting processed audio signals to determine the best tradeoff or other tradeoff between time delay and audio quality. Thus, the system can automatically adjust one or more aspects of audio processing without user input.

45B illustrates an example process for parallel processing of audio signals consistent with disclosed embodiments. The system may receive audio signal 4540, as shown in Figure 45B. Similar to audio signal 4410, audio signal 4540 may be captured by one or more microphones of wearable device 110, such as microphone 443 or 444. In some embodiments, the audio signal 4540 may be received from multiple microphones, such as a microphone array. Audio signal 4540 may include a representation of sound from the user's environment of wearable device 110, which may include speech from one or more individuals, background noise, music, and/or wearable device 110 prior to presenting it to the user Other sounds processed.

To determine the value of the time delay period or other aspects of audio processing, the system may execute multiple parallel streams of audio processing and compare the results. As shown in FIG. 45B, the system may process the audio stream 4540 according to the first scheme 4552 and the second scheme 4562. As used herein, a scheme may be a set of one or more defined parameters, aspects or variables that affect how an audio signal is processed. For example, scheme 4552 may include a first value for the look-ahead time delay period, and scheme 4562 may include a different value for the look-ahead time delay. For example, scheme 4552 may be associated with longer delays, while scheme 4562 may be associated with shorter delays. Schemes 4552 and 4562 may define other aspects of how the audio signal is processed, such as the number of microphones used to capture the audio signal, whether a camera is used to process the audio signal, or any other variables or settings that may affect audio quality. Other aspects may involve internal parameters or variables used to process the audio signal through various schemes. The system may generate a first processed audio signal 4556 and a second processed audio signal 4566 as a result of the parallel processing.

The system may then compare processed audio signals 4556 and 4566 to determine which scheme provides a better compromise between one or more of the aspects defined in schemes 4552 and 4562 and the audio quality of processed audio signals 4556 and 4566 . Accordingly, the system may be configured to evaluate the resulting audio quality of the processed audio signals 4556 and 4566, which may be performed in various ways. As one example, when processing the audio signal 4540, certain frequencies may be removed to "clean" the audio signal. For example, this may include removing certain frequencies associated with background noise, other speakers, or other audio sources. This processing may result in a reduction in the amount of accumulated energy of the signal. Thus, after parallel processing, the energy level 4554 of the processed audio signal 4556 and the energy level 4564 of the processed audio signal 4566 can be compared. If the difference in energy levels is small, this may indicate that both schemes are equally effective at attenuating unwanted noise in the signal. Therefore, the additional time delay introduced by scheme 4552 may not provide a significant advantage over the shorter time delay introduced by scheme 4562. On the other hand, if the difference between energy levels 4554 and 4564 is relatively large, this may indicate that the additional time delay introduced by scheme 4552 significantly improves the quality of the processed audio signal, so scheme 4552 may be selected. Thus, comparing processed audio signals 4556 and 4566 may include comparing the difference between energy levels 4554 and 4564 to a threshold energy level difference.

Various other comparisons between processed audio signals 4556 and 4566 may be used. For example, the system may analyze the sampling rate of the audio signal, the amount of noise in the signal, or other characteristics of the processed audio signals 4556 and 4566. In some embodiments, audio quality may be assessed based on input from a user. For example, the system may present the processed audio signals 4556 and 4566 to the user, and the user may provide input as to how much improvement one provides over the other or whether they are relatively close in audio quality.

Based on a comparison of processed audio signals 4556 and 4566, the system may select a scheme that provides a better compromise between time delay (or otherwise) and audio quality. The system may also present the selected processed audio signal to the user, eg, through the auditory interface device 1710 . In some embodiments, this parallel processing may be performed periodically to dynamically adjust time delays or other aspects of audio processing. For example, parallel processing may be performed as checks per second, 2 seconds, 10 seconds, 60 seconds, 5 minutes, 10 minutes, hours, or other suitable time period. In some embodiments, the user may manually initiate parallel processing, eg, by selecting a calibration button on the user interface, a physical button on the wearable device 110, or the like. In some embodiments, parallel processing may be initiated based on other cues detected by wearable device 110 , computing device 210 , or auditory interface device 1710 . In some embodiments, the camera of the wearable device 110 can detect when the user enters different environments, such as moving from a quiet vehicle to a noisy restaurant. Therefore, when the user is in a restaurant, the look-ahead time delay may become more important because there may be more background noise that must be attenuated. As another example, a camera may detect whether an individual in the user's environment is speaking to the user, which may indicate that an additional look-ahead time delay is required. Parallel processing may be initiated based on other sensor data, such as changes in the user's GPS location, changes in light detected by the sensors, changes in noise levels, or various other sensor data. In some embodiments, the system may be configured to vary the predetermined time interval based on this information. For example, in environments where conditioning the audio signal may be more important or longer look-ahead samples may be required, the system may perform parallel processing more frequently.

Although two parallel processing schemes are shown in Figure 45B, in some embodiments parallel processing may be performed according to more than two schemes. Thus, the system can compare the difference in energy levels (or other audio metrics) of two or more processed audio signals. In some embodiments, the system may not have to select a value defined in one of the schemes. For example, the system may interpolate or extrapolate the energy levels output across multiple processed audio signals, and may determine a look-ahead time delay or other value that represents the best compromise in audio quality.

Where the system determines that the look-ahead time delay or other aspects should be changed, the system may implement the change in various ways. In some embodiments, changes may be implemented immediately or shortly after parallel processing. In some embodiments, changes may not have to be implemented immediately. For example, when extending the look-ahead delay, it can be done by extending fixed segments of the audio signal (such as periods of silence, periods of relatively consistent noise (eg, water flow, etc.), or other periods where changes in audio processing settings may be less pronounced) to extend the delay. Therefore, the user may not notice the change. In some embodiments, if a period of stillness is not detected within a predetermined period of time, the transition may be performed on another portion of the audio signal. In some embodiments, the delay may be gradually varied to reduce the impact on the user. For example, if the delay is to be extended from 30ms to 100ms, it can be executed in 7 loops, where the delay is extended by 10ms on each loop. If a still segment of the audio signal (eg, a quiet period) is detected after one or more cycles, the remainder of the delay may be extended during the still segment.

In some embodiments, although one processed audio signal is selected over the other, multiple processed audio signals may be presented to the user. For example, when a user speaks, he or she may want to hear himself speak. In particular, the user may want to hear his or her voice against other people. In such an embodiment, the user may wish to hear himself with minimal delay. Thus, a processed audio signal based on a scheme with minimal time delay can be presented to the user. For example, the audio signal may be transmitted within 300 milliseconds of receiving the audio signal. In some embodiments, this may be faster (eg, at 200 milliseconds, 100 milliseconds, 80 milliseconds, 40 milliseconds, etc.). When the wearable device 110 detects that the user is speaking, it can present a processed audio signal with minimal delay. In some embodiments, this processed audio signal with minimal delay may be presented (eg, as a mixed or combined audio signal) with a selected processed audio signal, which may be presented in a longer delayed rendering. In other embodiments, the system can switch back and forth between the preferred scheme and the scheme with the least delay.

46A is a flowchart illustrating an example process 4600A for selectively amplifying an audio signal, consistent with the disclosed embodiments. Process 4600A may be performed by at least one processing device of the wearable device, such as processor 210 as described above. In some embodiments, some or all of process 4600A may be performed by a different device, such as computing device 120 . It should be understood that throughout this disclosure, the term "processor" is used as a shorthand for "at least one processor." In other words, a processor may include one or more structures that perform logical operations, whether collocated, connected, or distributed. In some embodiments, a non-transitory computer-readable medium may contain instructions that, when executed by a processor, cause the processor to perform process 4600A. Furthermore, process 4600A is not necessarily limited to the steps shown in Figure 46A, and any steps or processes of various embodiments described throughout this disclosure may also be included in process 4600A, including those described above with respect to Figures 44 and 45A step or process. Although process 4600A is described in the context of time delays, it should be understood that process 4600A may be applied to other aspects of processing audio signals, including the number of microphones used to capture audio signals, whether cameras are used to process audio signals, or may affect any other aspect of audio quality.

In step 4610, process 4600A can include receiving an audio signal representing sound received by at least one microphone from the user's environment. For example, microphone 443 or 444 (or microphone 1720 ) can capture sounds from the user's environment and can send them to processor 210 . This may include audio signal 4410 as described above.

In step 4612, the processor 4600A may include receiving an indication of a time delay associated with processing the audio signal. As described in more detail above, the time delay may be determined in various ways. In some embodiments, the time delay may be defined by the user through the user interface. The user interface may be included on a device, such as computing device 120, that interfaces with the hearing aid system. For example, the device may be at least one of a mobile phone, desktop, laptop, or tablet. In some embodiments, the set points defining the time delay may be stored on a remote storage device. For example, the setpoints may be stored on the computing device 120, the hearing aid device 1710, a remote server (eg, a cloud storage platform, a web-based server, etc.), or the like. Accordingly, receiving an indication of a time delay may include accessing a remote storage device. In some embodiments, the time delay may be determined by the hearing aid system based on feedback from the user regarding previously processed audio signals. For example, the user may provide a rating of audio quality, may provide feedback on the delay associated with the audio signal, or may indicate other feedback that indicates a preferred time delay or audio quality setting.

In step 4614, the processor 4600A may include storing in a buffer a plurality of audio samples representing the portion of the audio signal. For example, the plurality of audio samples may include audio samples 4412 and 4414 as described above with respect to FIG. 44 . A buffer may be any storage location that stores audio samples at least temporarily for analysis and/or processing. In some embodiments, the buffer may correspond to memory 550, as described above.

In step 4616, process 4600A may include processing a first audio sample of the plurality of audio samples to generate a processed first audio sample. For example, the first audio sample may correspond to audio sample 4412. Processing the first audio sample may include analyzing a second audio sample, such as the look-ahead audio sample 4414 as described above. Thus, as shown in Figure 44, the second audio sample may be represented in the audio signal following the first audio sample, and may have a length defined by the time delay. Although step 4616 describes a single second audio sample, it should be understood that processing the first audio sample may include analyzing multiple look-ahead audio samples. Thus, in some embodiments, the second audio sample may comprise a plurality of audio samples. In some embodiments, the audio quality of the processed first audio sample may depend on the length of the second audio sample. For example, larger look-ahead samples 4414 may result in more data available for processing the audio signal 4412, which may result in a better ability to condition or enhance the audio signal. Therefore, longer second audio samples can be associated with higher audio quality.

46B is a flowchart illustrating an example process 4600B for selectively amplifying an audio signal, consistent with the disclosed embodiments. Process 4600B may be performed by at least one processing device of the wearable device, such as processor 210 as described above. In some embodiments, some or all of process 4600B may be performed by another device, such as computing device 120 . In some embodiments, a non-transitory computer-readable medium may contain instructions that, when executed by a processor, cause the processor to perform process 4600B. Furthermore, process 4600B is not necessarily limited to the steps shown in Figure 46B, and any steps or processes of various embodiments described throughout this disclosure may also be included in process 4600B, including those described above with respect to Figures 44-46A step or process.

In step 4640, process 4600B can include receiving an audio signal representing sound captured by at least one microphone from the user's environment. For example, microphone 443 or 444 (or microphone 1720 ) can capture sounds from the user's environment and can send them to processor 210 . This may include audio signal 4540 as described above.

In step 4642, process 4600B can include processing the audio signal using the first value of at least one aspect to generate a first processed audio signal. For example, audio signal 4540 may be processed to generate processed audio signal 4556 as described above with respect to FIG. 45B. At least one aspect may include any form of variables, settings, options or other parameters associated with the processed audio signal that may affect the audio quality of the processed audio signal. In some embodiments, at least one aspect may include a time delay for processing the audio signal. As mentioned above, the time delay may define the length of the look-ahead samples used to process the audio signal samples. In some embodiments, at least one aspect may include the number of microphones used to capture the audio signal. In some embodiments, at least one aspect may include whether to process images in addition to audio signals to improve processing of audio signals. Accordingly, process 4600B can also include receiving, from the wearable camera, at least one image captured from the user's environment, and at least one aspect can include whether to process the at least one image.

In step 4644, the process 4600B can include processing the audio signal using the second value of the at least one aspect to generate a second processed audio signal. For example, audio signal 4540 may be processed to generate processed audio signal 4566. Accordingly, at least a portion of step 4644 may be performed in parallel with step 4642. It should be understood, however, that step 4644 may be performed before or after step 4642 in some embodiments. In some embodiments, the second value may be different from the first value. For example, if at least one aspect includes a time delay, the first value may be a shorter time delay than the second value, and vice versa. In some embodiments, the first processed audio signal or the second processed audio signal may be processed to minimize time delay. In some embodiments, the first value and the second value may be defined according to the first and second schemes. For example, the audio signal may be processed according to schemes 4552 and 4562, as described above.

In step 4646, process 4600B may include comparing the first processed audio signal to the second processed audio signal to select either the first processed audio signal or the second processed audio signal. The processed audio signal may be selected based on a compromise between at least one aspect of audio quality of the first processed audio signal and the second processed audio signal. For example, where at least one aspect includes a time delay, the selected processed audio signal may be one that provides the lowest time delay without compromising audio quality. The comparison can be performed in various ways. For example, as described in greater detail above, comparing the first processed audio signal and the second processed audio signal may include determining an energy level difference between the first processed audio signal and the second processed audio signal. In some embodiments, selecting the first processed audio signal or the second processed audio signal includes determining that the difference in energy levels is below a predetermined threshold. For example, if the difference in energy levels is relatively low, this may indicate that a minimum gain in audio quality is achieved by the difference between the first value and the second value. In some embodiments, multiple aspects may be considered and weighted along with delay and audio quality. For example, the processing power or other resources required for processing can be considered. Therefore, the value that provides the greatest benefit (eg, shorter time delay and lower battery consumption) can be selected. In some embodiments, the energy level of the selected processed audio signal is lower than the energy level of the unselected processed audio signal. For example, if the energy level difference is below a certain threshold, step 4646 may include selecting the processed audio signal with the lowest energy level, which may indicate better audio quality.

In step 4648, process 4600B may include sending the selected processed audio signal to the user's auditory interface device. For example, wearable device 110 may transmit the selected processed audio signal to auditory interface device 1710 via wireless transceiver 530 . Thus, the selected processed audio signal may be presented audibly in the ear of the user. In some embodiments, transmitting the selected processed audio signal may include converting the value of at least one aspect to a different value (eg, to a first or second value). As mentioned above, this change may occur gradually over several cycles during periods of stillness of the detected audio signal, or in another manner to reduce user perceptibility.

As described above, process 4600B can be performed as a calibration process such that at least one aspect can be updated automatically and dynamically without requiring user input. As such, some or all of process 4600B may be performed periodically. For example, process 4600B may also include comparing the first processed audio signal and the second processed audio signal at predetermined time intervals. As noted above, some or all of process 4600B may be performed based on other prompts. For example, process 4600B may be based on detecting changes in the user's environment (eg, via camera images, GPS sensor data, light sensor data, or other sensor data), based on changes in audio signals, based on input from the user (eg, pressing a calibration button, etc.) ) or various other indicators.

Wearable device for active sound replacement

As described above, audio signals captured from within the user's environment may be processed prior to presentation of the audio to the user. The processing may include various adjustments or enhancements to improve the user's experience. For example, as described above, speech from the individual the user is looking at may be amplified, while other audio such as background noise, speech from other speakers, etc. may be muted or attenuated. Therefore, the user can more easily hear and understand the speech from the individual with whom the user is conversing. Due to the processing time required to condition the audio, the user may initially hear the speaker's voice from his or her surroundings, and then the delayed, adjusted voice of the speaker may be heard through the hearing aid device. Consequently, the user may experience unwanted "echoes" due to the processing time used to condition the audio, which may be undesirable to the user.

Accordingly, the hearing aid system may be configured to cancel sound, at least in part, in real time and provide the user with sound adjusted to reduce or eliminate echo. For example, a hearing aid can cancel the speaker's voice in real-time and then provide the user with an adjusted version of the speaker's voice through the hearing aid device. To cancel noise in real time, the system can determine the time difference between the sound that reaches the microphone and the sound that reaches the user's ear. This can be accomplished using the time difference between the transit time of an individual sound traveling in air at the speed of sound and the transmission time of a noise cancelling signal traveling at the speed of electricity (eg, at the speed of light). Determining the difference enables cancellation of the user's noise/voice in real time.

47 is a block diagram illustrating an example process 4700 for active sound replacement consistent with the disclosed embodiments. Process 4700 may be used to process audio signal 4710, as shown in FIG. 47 . As described above, audio signal 4710 may be captured by one or more microphones of wearable device 110, such as microphone 443 or 444. In some embodiments, the audio signal 4710 may be received from multiple microphones, such as a microphone array. Audio signal 4710 may include a representation of sounds from the environment of the user of wearable device 110 . For example, the audio signal may thus include speech from one or more individuals, background noise, music, and/or other sounds that the wearable device 110 processes before presenting it to the user.

Process 4700 may include performing audio processing 4720 on audio signal 4710. Audio processing 4720 may include any form of conditioning or enhancement of the audio signal, including any form of selective conditioning throughout this disclosure. For example, as disclosed in this application, the system may selectively condition the audio signal 4710 to attenuate background noise, amplify sound from a particular source (eg, an object or person the user is looking at), adjust the pitch of the audio signal , adjust the playback rate of the audio signal, remove noise or artifacts from the signal, perform audio compression, or perform other enhancements to improve the user's audio quality. As shown in FIG. 47 , audio processing 4720 can be used to generate a selectively conditioned audio signal 4722 that can be sent to auditory interface device 4740 .

In addition to audio processing 4720, process 4700 may also perform noise cancellation 4730. Noise cancellation 4730 may be any form of processing of an audio signal that is configured to cancel or attenuate one or more sounds from the audio signal. As shown in FIG. 47, noise cancellation 4730 may be used to generate at least one cancellation audio signal 4732. Noise cancelling audio signal 4732 may be any audio signal that is configured to cancel or cancel another audio signal. For example, noise cancellation 4730 may include an active noise control (ANC) process. Thus, the cancel audio signal 4732 may include a "negative" audio signal that is out of phase with another audio signal having undesired sounds, such as sounds predicted to reach the user's ears. The cancel audio signal 4732 may be configured such that when the sound pressure of the sound wave of the undesired sound is high, the sound pressure of the sound wave of the cancel audio signal 4732 is low. Thus, when the cancellation audio signal is combined with the predicted audio signal, the waves of the two audio signals can be cancelled or "cancelled". Like the selectively conditioned audio signal 4722, the cancel audio signal 4732 may be sent to the auditory interface device 4740.

Cancel audio signal 4732 may be configured to cancel at least a portion of audio signal 4710. In some embodiments, canceling audio signal 4732 may be configured to cancel a particular portion of audio signal 4710, such as an individual's speech, at a desired phase. Therefore, when the specific part reaches the user's ear, the specific part can be canceled. In some embodiments, this portion may also be included in audio processing 4720. For example, if the audio processing 4720 is configured to selectively condition the individual's speech from the audio signal 4710, the cancellation audio signal 4732 may cancel or cancel the individual's voice from the sound reaching the user's ear. Accordingly, canceling the audio signal 4732 can eliminate or reduce the echo that the user may experience when the selectively conditioned audio signal 4722 is sent to the auditory interface device 4740 . In some embodiments, cancel audio signal 4732 may be configured to cancel all sounds from the user's environment. Thus, the user can hear the selectively conditioned audio signal 4722 without hearing any sound or sounds from the user's environment. For example, audio processing 4720 may selectively adjust multiple sounds within the user's environment to adjust the volume or other properties of the sounds relative to each other, and the user may hear the selectively adjusted sounds without echo effects.

To effectively cancel sound from the user's environment, a cancel audio signal can be sent so that it appears in the user's ear at the same time as the sound being canceled. Accordingly, the system may be configured to determine a time interval that defines the time between when the audio signal 4710 is received by the device 110 (or, in some embodiments, when the cancel audio signal 4732 is generated) and when the cancel signal is presented to the user. time delay. Accordingly, the system can predict the sound to be received at the user's ear based on the audio signal 4710, and can determine a time delay such that the canceling audio signal 4732 cancels the predicted sound. In some embodiments, the time delay may be a preset or predefined value associated with the wearable device 110 . For example, the system may assume the time required for the predicted sound to be heard by the user after being captured as an audio signal 4710. The predetermined time delay can be adjusted based on input from the user. For example, the user may be able to fine-tune the delay through the user interface, or may be able to provide feedback that the echo is being experienced.

In some embodiments, the time delay may be determined based on the distance the sound will travel between the location where it will be captured as audio signal 4710 and the user's ear. 48A, 48B, and 48C illustrate example configurations of wearable device 110 and auditory interface device 4820 for active sound replacement consistent with the disclosed embodiments. In some embodiments, the wearable device 110 may be worn as glasses, as shown in Figure 48A. As described above, the wearable device 110 may be worn at other locations on the user's body. For example, wearable device 110 may be worn on a user's body on a belt, shirt, wrist, or various other locations. Wearable device 110 may include microphone 4812, which may be configured to capture audio signals (such as audio signal 4710 described above). Microphone 4812 may be any device configured to capture sound from the user's environment. For example, the microphone 4812 may correspond to the microphones 443, 444 or 1720 described above. As described above, the wearable device 110 may transmit one or more signals to the hearing aid device 4740. For example, wearable device 110 may transmit selectively conditioned audio signal 4722 and cancel audio signal 4732 to auditory interface device 4740 using wireless transceiver 530 . Auditory interface device 4740 may correspond to auditory interface device 1710 as described above. Accordingly, any of the details or embodiments described above with respect to auditory interface device 1710 may be applied to auditory interface device 4740. For example, while auditory interface device 4740 is shown as an in-ear device, auditory interface device 4740 may include another form of auditory interface device, such as a bone conduction headset, an out-of-ear device, and the like.

The disclosed methods and systems may include determining a distance d that represents the difference between the distance the sound wave 4810 will travel before being captured at the microphone 4812 and the distance the sound wave will travel before reaching the ear of the user (or auditory interface device 4740). value. The time delay discussed above can be determined based on how long it will take for the sound wave 4810 to travel the distance d (assuming it travels at the speed of sound). Thus, the distance d can be determined perpendicular to the propagation of the sound wave 4810. In some embodiments, other factors may be considered to determine the time delay, such as the time required to send the cancellation audio signal 4732, the time the cancellation audio signal 4732 is processed and played at the auditory interface device 4740, or that may affect presentation of the cancellation audio to the user Various other factors (or combinations thereof) of the timing of the signal. In some embodiments, the disclosed system may require accumulated audio background for processing canceled audio signals. For example, the system may introduce a delay of 40-80 milliseconds (or other suitable delay) to allow the audio signal 4710 to be processed. In some embodiments, this accumulated audio background delay may be variable or adjustable as described above with respect to Figures 44-46B.

As mentioned above, the wearable device 110 may be placed in various other locations on the user. The distance d may depend, at least in part, on the placement of the wearable device 110 . Figure 48B shows the wearable device 110 clipped to the user's collar. As shown in FIG. 48B , the distance d may vary depending on where the wearable device 110 is placed. Distance d may similarly vary based on the type of wearable device 110 or other characteristics. For example, wearable device 110, when implemented as a pair of glasses, may have a different predetermined time delay than a device that may be attached to a user's clothing. In some embodiments, the distance d (and time delay) may depend on where the user places the device. For example, the same device may be clipped or otherwise secured in different locations on the user. Therefore, the device can receive data indicating the placement location. In some embodiments, the data may be user input indicating a placement location. For example, a user may provide input through a user interface of computing device 120 (or other computing device). This may include selecting from a list of locations, clicking on a user image or other interface indicating an approximate placement location. User input may be received through other devices, such as physical switches or buttons on the wearable device 110 . In some embodiments, the wearable device 110 may infer the placement location based on the perceived camera position based on images captured by the image sensor 220 of the wearable device 110, the perceived directionality of the user's voice or other sounds, and the like. In some embodiments, the time delay may depend on the size of the device. For example, when implemented as a pair of glasses, a device with longer temples may assume a longer time delay than a device with shorter temples. The time delay may also be configured based on the placement location of the auditory interface device 4740 or other properties (eg, in-ear and bone conduction, processing speed, etc.).

The distance d and associated time delay may also depend on the location of the sounding object relative to the user. Figure 48C shows sound waves 4810 received from a higher position relative to Figure 48B, where the placement of the wearable device 110 and the auditory interface device 4740 remains the same. Because the angle at which the sound wave 4810 reaches the microphone 4812 and the user's ear is different, the distance d is also different (shortened in this example). Accordingly, the wearable device 110 may be configured to be responsible for generating the sound wave 4810 to cancel the location of the sound emitting object when the time delay is determined. The wearable device 110 may determine or estimate the location of the sound-emitting object in various ways. In some embodiments, the location may be assumed based on the type of sound-emitting object. For example, if wearable device 110 determines that sound wave 4810 is associated with an individual, wearable device 110 may assume that the sound emitting object is at the average height of the individual's mouth. As another example, if the sound waves 4810 are associated with animals such as dogs or cats, the wearable device may assume that the sound waves 4810 are emanating from near the ground. Various other types of sound-emitting objects may be identified and associated with predetermined heights.

In some embodiments, the location of the sound emitting object may be determined based on sensor data received by the wearable device 110 . For example, wearable device 110 may analyze one or more images captured by image sensor 220 to determine the location of the sound-emitting object. This may include various object or feature detection or other image processing techniques, as described throughout this disclosure. In some embodiments, the location of the sounding object may be determined based on input from the microphone 4812. For example, microphone 4812 may include a multidirectional array of microphones or other types of microphones configured to determine the direction in which to capture sound. Thus, the time delay (and direction d) may depend on various inputs to the wearable device 110 . As described above, the user may be able to use a graphical user interface (eg, on computing device 120, on wearable device 110, etc.), through physical controls (eg, buttons, dials, switches, etc.), or various other input devices to tune or adjust the determined delay.

49 is a flowchart illustrating an example process 4900 for selectively replacing an audio signal, consistent with the disclosed embodiments. Process 4900 may be performed by at least one processing device of the wearable device, such as processor 210 as described above. In some embodiments, some or all of process 4900 may be performed by a different device, such as computing device 120 or auditory interface device 4740. It should be understood that throughout this disclosure, the term "processor" is used as a shorthand for "at least one processor." In other words, a processor may include one or more structures that perform logical operations, whether collocated, connected, or distributed. In some embodiments, a non-transitory computer-readable medium may contain instructions that, when executed by a processor, cause the processor to perform process 4900. Furthermore, process 4900 is not necessarily limited to the steps shown in FIG. 49, and any steps or processes of various embodiments described throughout this disclosure may also be included in process 4900, including above with respect to FIGS. 47, 48A, 48B and those steps or processes described in Figure 48C.

In step 4910, process 4900 can include receiving a plurality of images captured by the wearable camera from the user's environment. For example, step 4910 may include receiving an image captured by image sensor 220 . The captured images may include representations of individuals or other sound-emitting objects within the user's environment.

In step 4912, the process 4900 can include receiving an audio signal representing sound captured by the at least one microphone from the user's environment. For example, microphone 443 or 444 (or microphone 1720 ) can capture sounds from the user's environment and can send them to processor 210 . This may include the audio signal 4710 as described above.

In step 4914, process 4900 may include identifying audio signals from a plurality of audio signals associated with one or more sound-emitting objects in the user's environment based on analysis of the plurality of image or audio signals. In some embodiments, the sound emitting object may be an individual. For example, step 4914 may include processing the plurality of images to identify the individual who is speaking to the user. This may be based on the individual's lip movement, the user's gaze direction, voiceprint, or various other methods described throughout this disclosure for identifying audio signals associated with the individual.

In step 4916, the process 4900 can include predicting the sound to be received at the user's ear from the user's environment based on the plurality of audio signals. The predicted sound may correspond to the sound that the user will hear when the sound waves associated with the identified audio signal reach the user's ear. For example, referring to FIG. 48A, the predicted sound may be the sound that the user is expected to hear once the sound wave 4810 reaches the user's ear after being recorded by the microphone 4812.

In step 4918, the process 4900 may include generating a cancellation audio signal configured to cancel at least the predicted sound at the user's ear. For example, as described above, this may include generating a cancellation audio signal 4732 through noise cancellation 4730. In some embodiments, the noise cancelling audio signal may be configured to cancel at least one sound from the user's environment other than the sound from the individual speaking to the user. For example, the noise-cancelling audio signal may be configured to cancel background noise, other speakers, or other sounds from the user's environment emitting the object's voice.

In step 4920, process 4900 may include generating a selectively conditioned audio signal based on the identified audio signal. For example, step 4920 may include generating a selectively conditioned audio signal 4722 by audio processing 4720 as described above. Thus, generating the selectively adjusted audio signal may include any form of adjustment or enhancement of the identified audio signal. For example, the selective adjustment may include amplifying the identified audio signal relative to additional audio signals of the plurality of audio signals. Various other forms of selective modulation are described throughout this disclosure.

In step 4922, the process 4900 may include sending the cancel audio signal and the selectively adjusted audio signal to a hearing aid interface device configured to provide sound to the user's ear. For example, as shown in FIG. 47 , selectively conditioned audio signal 4722 and cancel audio signal 4732 may be sent to auditory interface device 4740 . In some embodiments, the cancel audio signal and the selectively conditioned audio signal may be sent together, although they may be time-shifted relative to each other. For example, the cancelling audio signal and the selectively conditioned audio signal may be combined or mixed such that when presented to the user's ear along with the predicted sound, only the conditioned audio signal is heard. In some embodiments, the cancel audio signal and the selectively adjusted audio signal may be sent separately. For example, the cancel audio signal may be sent before the selectively adjusted audio signal is sent. Thus, the predicted sound can be canceled at the user's ear so that the selectively conditioned audio signal does not introduce echoes to the user.

As described above, the presentation of the canceled audio signal may be timed to coincide with the predicted sound arriving at the user's ear. Accordingly, in some embodiments, process 4900 may also include determining a time delay between when the plurality of audio signals are received and when the predicted sound will be received at the user's ear. In these embodiments, the cancel audio signal may be sent at a time based on a time delay. In some embodiments, the time delay may be determined based at least in part on the speed of sound propagating in air. For example, as described above with respect to Figures 48A, 48B, and 48C, the time delay may correspond to the time required for the acoustic wave 4810 to travel the distance d. In some embodiments, the time delay may be determined based at least in part on the position of the sound emitting object relative to the at least one microphone and hearing aid interface device. For example, as described above with respect to FIG. 48C, the location of the sound-emitting object may be determined based on a plurality of images. The location of the sound-emitting object may be determined based on other inputs, such as directionality determined based on the microphone, or other data. In some embodiments, the time delay may be determined based at least in part on input from a user. For example, input is received through a user interface of an external device, such as computing device 120 . The user interface may include controls similar to those shown in Figure 45A and described above. In some embodiments, user input may be provided through physical controls such as buttons, dials, switches, and the like.

Analog directionality of sound

Consistent with the disclosed embodiments, the hearing aid system may selectively amplify sound based on the location of the object emitting the sound. Existing hearing aid systems may not be able to reproduce the timing and volume of sounds with sufficient fidelity and precision to allow users to identify the source of the sounds. In some cases, the user may have learning difficulties, brain damage, a deformed ear or ear canal, or the effects of damage that impairs the user's ability to locate objects based on sound location. For example, a user may have unequal hearing loss in his or her ears, leading to the false belief that the object is closer to the user's less hearing-impaired ear than to the user's more hearing-impaired ear. Additionally, in some cases the user may wish to enhance sound localization capabilities by combining delay perception and sound intensity perception.

Thus, the hearing aid system can analyze the captured images and sounds of the user's environment to determine the location of the sound source. The hearing aid system can then send sounds to the user so that the sounds reach the user's ears at different times and volumes, creating a stereo effect. By doing so, the hearing aid system may provide the user with alternative and/or enhanced sound localization capabilities.

The user 100 may wear a hearing aid device that conforms to the camera-based hearing aid device described above. For example, the hearing aid device may be an auditory interface device 1710 as shown in Figure 17A. Auditory interface device 1710 may be any device configured to provide auditory feedback to user 100 . An auditory interface device 1710 may be placed in each ear of the user 100, similar to conventional auditory interface devices. As mentioned above, the auditory interface device 1710 may be of various styles, including in-canal, completely in-canal, in-ear, behind-the-ear, supra-aural, in-canal receiver, open mount, or various other styles. Auditory interface device 1710 may include one or more speakers for providing auditory feedback to user 100, a microphone for detecting sounds in the environment of user 100, internal electronics, a processor, memory, and the like. In some embodiments, auditory interface device 1710 may include one or more communication units in addition to or instead of a microphone, and be one or more receivers for receiving signals from device 110 and transmitting signals to user 100 . The auditory interface device 1710 may correspond to the feedback output unit 230 or may be separate from the feedback output unit 230 and may be configured to receive signals from the feedback output unit 230 .

In some embodiments, the auditory interface device 1710 may include a bone conduction headset 1711, as shown in FIG. 17A. Bone conduction earphones 1711 may be surgically implanted and may provide audible feedback to user 100 through bone conduction of sound vibrations to the inner ear. The auditory interface device 1710 may also include one or more earphones (eg, wireless earphones, over-ear earphones, etc.) or portable speakers carried or worn by the user 100 . In some embodiments, the auditory interface device 1710 may be integrated into other devices, such as the user's Bluetooth ^™ headset, glasses, helmets (eg, motorcycle helmets, bicycle helmets, etc.), hats, and the like. In some embodiments, two auditory interface devices 1710 may be provided, one for each ear. The two auditory interface devices 1710 may be wired or may be wirelessly connected. Additionally, the first auditory interface device may receive instructions or audio from the second auditory interface device. Additionally, the two auditory interface devices 1710 may receive audio from another source, such as the device 110 or a paired device.

Auditory interface device 1710 may be configured to communicate with a camera device such as apparatus 110 . Such communication may be via a wired connection, or may be performed wirelessly (eg, using Bluetooth ^™ , NFC or wireless forms of communication). As described above, device 110 may be worn by user 100 in various configurations, including physically attached to a shirt, necklace, belt, eyeglasses, wristband, button, or other item associated with user 100 . In some embodiments, one or more additional devices such as computing device 120 may also be included. Accordingly, one or more of the processes or functions described herein with respect to apparatus 110 or processor 210 may be performed by computing device 120 and/or processor 540 .

As mentioned above, the apparatus 110 may include at least one microphone and at least one image capture device. Device 110 may include microphone 1720 as described with respect to FIG. 17B . Microphone 1720 may be configured to determine the directionality of sound in the environment of user 100 . For example, microphone 1720 may include one or more directional microphones, microphone arrays, multi-port microphones, and the like. The processor 210 may be configured to differentiate sounds within the environment of the user 100 and determine the approximate directionality of each sound. For example, using the microphone array 1720 , the processor 210 may compare the relative timing or amplitude of individual sounds between the microphones 1720 to determine the directionality relative to the device 100 . Device 110 may include one or more cameras such as camera 1730 , which may correspond to one or more image sensors such as image sensor 220 . The camera 1730 may be configured to capture images of the user's 100 surroundings. Apparatus 110 may also use one or more microphones of auditory interface device 1710 , and thus, references to microphone 1720 used herein may also refer to a microphone on auditory interface device 1710 .

Processor 210 (and/or processors 210a and 210b ) may be configured to detect sound-emitting objects (such as individuals) within the environment of user 100 . 50A is a schematic diagram illustrating an exemplary environment. As shown in FIG. 50A , user 100 wearing device 110 may be physically present in the environment and individual 5002 produces sound 5004 . Thus, in the scene presented in Figure 50A, the individual 5002 is the sound-emitting object. Although FIG. 50A shows an individual as the sound-emitting object, the sound-emitting object may be any object in the environment that produces sound waves that can be heard by the user 100 or detected by the device 100 . For example, the sound emitting object may be a machine, an animal, or a naturally occurring sound source such as wind. In some cases, the sound emitting object may move while producing the sound. Alternatively, the sound emitting object may produce sound without observable motion (such as a speaker of a radio).

The location of the sound emitting object may be determined by detecting the difference in arrival times of the sound at the plurality of listening devices and/or by detecting the difference in volume. For humans, for example, sound localization is performed by determining the time difference between the sound reaching the left and right ears of the person. Sound localization is also achieved by determining the difference in sound volume at the left and right ears. For example, in Figure 50A, user 100 may determine that individual 5002 is standing in front and to the right of user 100 by noting that sound 5004 reaches user 100's right ear earlier and louder than user 100's left ear. However, if user 100 is hearing impaired, user 100 may have to rely on vision to determine the location of individual 5002. If the user 100 is visually impaired, or looks in a different direction, the user 100 may not be able to determine the location of the sound and thus the source of the sound.

To compensate for this, certain embodiments of the present disclosure may provide a stereo signal based on the determined location of the sound-emitting object. In some embodiments, the device 110 may use the image captured by the camera 1730 to determine the location of the sound emitting object. For example, FIG. 50B is a schematic diagram of an exemplary image captured by an imaging capture device consistent with the present disclosure. Processor 210 may be configured to analyze images captured by camera 1730 to detect sound-emitting objects, such as individual 5002, in field of view 5006 of camera 1730, and to determine the direction from user 100 to the sound-emitting objects. For example, the processor 210 may determine the location of the individual 5002 within the field of view 5006. As shown, the field of view 5006 may be divided into sections representing angles. For example, the field of view 5006 may have a centerline 5008 aligned with the user's gaze direction 1750. Field of view 5006 may be divided into portions representing the area delineated by lines 5008 and 5010 0-5 degrees off center; the portion delimiting the area 5-10 degrees from center delimited by lines 5010 and 5012; and The portion of the area delimited by a line that is 10-15 degrees off center. The location of the individual 5002 or other sound-emitting object can be determined by referring to these portions of the field of view 5006 . For example, as shown in Figure 50B, the mouth of individual 5002 is located within the area delineated by lines 5010 and 5012. Thus, in order to determine the direction towards the sound emitting object, the processor 210 may use motion detection and/or sound localization techniques as described further below. Although FIG. 50B shows lines 5008-5014, the processor 210 may use alternative metrics to determine the orientation of the sound-emitting object relative to the user 100, such as the x and/or y coordinates of the field of view 5006, or the smaller of the field of view 5006 and/or larger portions. Additionally, processor 210 may use sound time of arrival information in addition to or in addition to images captured by camera 1730, as will be described in more detail below.

Based on the detected location of the sound emitting object, the processor 210 may cause selective adjustments to the audio to communicate the location of the sound emitting object to the user 100 . Adjusting may include amplifying the audio signal determined to correspond to sound 5004 (which may correspond to the speech of individual 5002 ) relative to other audio signals. In some embodiments, amplification may be accomplished digitally, for example, by processing the audio signal associated with sound 5004 relative to other signals. Additionally or alternatively, amplification may be achieved by changing one or more parameters of the microphone 1720 to focus on the audio sounds associated with the individual 5002. For example, microphone 1720 may be a directional microphone and processor 210 may perform operations to focus microphone 1720 on sound 5004. Various other techniques for amplifying sound 5004 may be used, such as the use of beamforming microphone arrays, acoustic telescope techniques, and the like. The adjusted audio signal may be sent to the two auditory interface devices 1710 and thus may provide adjusted audio to the user 100 based on the location of the sound source. The selective adjustment may also include introducing an amplitude difference or delay between the sound being delivered to the first ear and the sound being delivered to the second ear. Further details of selective adjustment will be provided below.

The degree of modulation of the sound, such as the length of the amplification difference or delay, may be based on the determined direction to the sound-emitting object relative to the user 100 . For example, referring to Figure 50B, if the individual 5002 is located near the centerline 5008 of the field of view 5006, the processor 210 may provide a small degree of accommodation or no accommodation. Alternatively, if the individual 5002 is located near the right edge of the field of view 5006, the processor 210 may provide a greater degree of accommodation. Additionally, if the individual 5002 is outside the field of view 5006, the process 210 may determine the location of the individual 5002 based on the arrival times of sound at multiple microphones and introduce a greater degree of accommodation.

Sound conditioning can be further understood by referring to Figure 51, which shows a schematic diagram of an audio signal acquired and played back by a hearing aid system consistent with the present invention. In accordance with the present disclosure, device 110 may receive audio signals acquired by wearable microphone 1720 that reflect sounds generated by sound-emitting objects such as individual 5002 . Received signal 5102 is an audio signal representing sound captured by at least one microphone. The received signal 5102 may be, for example, the sound 5004 of Figure 50A. Once the received signal 5102 reaches the process 210, the processor 210 can determine the location of the sounding object corresponding to the received signal 5102. The processor 210 may determine position by, for example, identifying moving objects in the field of view of the camera as shown in Figure 50B, by calculating the time difference of arrival of received signals at multiple microphones, by manipulating directional microphones, and other methods.

After processor 210 analyzes received signal 5102, processor 210 may cause transmission of a stereo representation. For example, as shown in FIG. 51, the processor 210 may generate the first signal 5104 and the second signal 5106. For example, first signal 5104 may be sent to auditory interface device 1710 associated with the right ear of user 100 and second signal 5106 may be sent to auditory interface device 1710 associated with user 100's left ear.

To create a stereo representation, the processor 210 may delay the transmission of the second signal 5106 for a period of time after the transmission of the first signal 5014 . For example, processor 210 may begin transmission of first signal 5104 100 milliseconds after the sound arrives at device 110, as indicated by first transmission start line 5108. Although 100 milliseconds is used as an example, other suitable time periods may be used. For example, the time period may be in the range of 50-400 milliseconds, or any other suitable time period. Processor 210 may then send second signal 5106 800 milliseconds after the sound reaches device 110, as indicated by second transmission start line 5110. Thus, the second signal 5106 may be sent approximately 100 milliseconds after the first signal 5104, as shown by delay 5112 in FIG. 51 . Therefore, the user 100 will hear the first signal 5104 before the second signal 5106. If the first signal 5104 is associated with eg the user's right ear, the user 100 will perceive the sound source to be towards his right. Although the first transmission is shown to begin 100 milliseconds after the arrival of the received signal 5102, in some embodiments, the processor 210 may enable the first signal in other time frames (such as less than 100 milliseconds after the arrival of the received signal 5102). 5104 is transmitted to auditory interface device 1710. Similarly, delay 5112 can be of other durations and can vary depending on the location of the sounding object.

Figure 51 also shows that the received signal can be amplified and reduced in each signal to provide an additional indication of the location of the sounding object. For example, the maximum strength of the received signal 5102 is approximately 30,000 units. However, the first signal 5104 has a maximum strength close to 40,000, indicating that the processor 210 has amplified the first signal 5104. On the other hand, the second signal 5106 has a maximum strength of approximately 20,000, indicating that the processor 210 has attenuated the second signal 5106. Thus, assuming that the first signal 5104 is sent to the auditory interface device 1710 in the user's right ear, and the second signal 5106 is sent to the auditory interface device 1710 in the user's left ear, the user will hear in his right ear better than in The louder sound in his or her left ear enhances the user's ability to determine the source of the sound.

In some embodiments, the selective adjustment may include attenuating or suppressing one or more audio signals (such as background noise) not associated with the determined sound-emitting object. Adjusting may also include changing the pitch of the received signal 5102 to make the sound more perceptible to the user 100 . For example, the user 100 may have less sensitivity to tones within a certain range, and the adjustment of the audio signal may adjust the pitch of the received signal 5102. For example, user 100 may experience hearing loss in frequencies above 10 kHz, and processor 210 may remap higher frequencies (eg, at 15 kHz) to frequencies below 10 kHz. In some embodiments, the processor 210 may be configured to vary the speech rate associated with one or more audio signals.

52 is a flowchart illustrating an exemplary process for generating a stereo representation consistent with the disclosed embodiments. Process 5200 may be performed by one or more processors associated with apparatus 110, such as processor 210. The processor may be included in the same common housing as the microphone 1720 and camera 1730 that may also be used in process 5200. For example, apparatus 110 may include at least one microphone configured to capture sound from the user's environment. Apparatus 110 may include a wearable camera configured to capture multiple images from the user's environment. In some embodiments, some or all of process 5200 may be performed on a processor external to device 110, which may be included in the second housing. For example, one or more portions of process 5200 may be performed by a processor in auditory interface device 1710 or an auxiliary device such as computing device 120 or display device 2301 . In such an embodiment, the processor may be configured to receive the captured image via a wireless link between the transmitter in the common housing and the receiver in the second housing.

In step 5202, process 5200 may include receiving a plurality of images captured by the camera. For example, device 110 may capture an image and store a representation of the image compressed as a JPG file. As another example, device 110 may capture a color image, but store a black and white representation of the color image. As yet another example, device 110 may capture an image and store different representations of the image (eg, a portion of the image). For example, device 110 may store a portion of an image that includes the face of a person appearing in the image, but does not substantially include the environment surrounding the person. Similarly, device 110 may, for example, store a portion of an image that includes an object that appears in the image, but does not substantially include the environment surrounding the object. As yet another example, the apparatus 110 may store the representation of the image at a reduced resolution (ie, at a lower resolution than the captured image). Storing the representation of the image may allow device 110 to conserve storage space in memory 550 . Additionally, processing the representation of the image may allow device 110 to increase processing efficiency and/or help maintain battery life.

Step 5202 may also include determining differences in the series of images. For example, the previous image may be subtracted (such as pixel-by-pixel subtraction) to indicate the portion of the image that moved between the time of the first image and the time of the second image. In some embodiments, the differential image may be stored as a representation of the image.

In step 5204, process 5200 can include receiving an audio signal representing sound captured by at least one microphone. In some embodiments, process 5200 may receive and combine multiple audio signals from multiple microphones. For example, the device 110 may include a microphone designed to collect sounds with low frequencies, and a microphone designed to collect sounds with high frequencies. Step 5204 may then combine the sounds into a single audio signal representing both low and high frequencies. Step 5204 may also include determining a delay between arrival times of sounds in each of the plurality of microphones.

In step 5206, the process 5200 may include determining the location of the sound emitting object based on the analysis of the plurality of images and/or based on the analysis of the received sound. The position may be the angle of the sound-emitting object relative to the user, as shown in Figure 50B. Additionally or alternatively, the location may be the distance from the user to the object emitting the sound. The processor 210 may use the reference object in the captured image or the focal length of the lens of the camera 1730 to determine the distance. In some embodiments, the apparatus 110 may include a plurality of cameras that provide stereoscopic images, which may enable the processor 210 to determine the distance to the object. In some embodiments, process 5200 may include determining the Doppler effect of the sound to identify the location of the sound emitting object, eg, by calculating frequency changes of the sound waves. Additionally, process 5200 can match sounds to sound profiles to help identify types of sound-emitting objects in the plurality of images. For example, the processor 210 may identify a number of possible sound-emitting objects (such as cars and lawn mowers) based on analysis of the multiple images. The processor 210 may also compare the received sound to the sound profiles of the car and the lawn mower and determine that the lawn mower is the sound emitting object. Additionally, the processor 210 may use data from inertial measurements or from a rangefinder of the device 110 to determine the distance. For example, processor 210 may calculate, based on inertial measurements, that device 110 moves two feet to the left while objects in the field of view of camera 1730 move 15 degrees to the right of the center of the field of view. The processor 210 may use measurements from, for example, a rangefinder provided on the device 110 to identify the distance to the closest object. For example, a rangefinder can measure the distance in front of the user or to the side of the user, and this distance can be used to determine location. Based on these measurements, the processor 210 can determine the distance to the sound-emitting object.

If the processor 210 detects multiple possible sound-emitting objects, additional steps may be required to determine which of the multiple possible objects is associated with the received sound. For example, step 5206 may include determining the object closest to user 100 . Thus, step 5206 may include determining the distance of the first object to the user, and determining the distance of the second object to the user. Once the processor 210 determines the distance from the user to the first object and the second object, the processor 210 may select one of the first and second objects as the sound emitting object based on the determined distance. For example, the processor 210 may select the closest object as the sound emitting object.

In step 5206, the process 5200 may also include identifying a representation of the sound-emitting object in at least one of the plurality of images. In one embodiment, step 5206 may include identifying at least one lip movement or lip position associated with the individual's mouth based on analysis of the plurality of images. The processor 210 may be configured to identify one or more points associated with the individual's mouth. In some embodiments, the processor 210 may develop a profile associated with the individual's mouth, which may define a boundary associated with the individual's mouth or lips. Lips identified in an image can be tracked over multiple frames or images to identify lip movement. The processor 210 may also use one or more video tracking algorithms, such as mean-shift tracking, contour tracking (eg, compression algorithms), or various other techniques.

In some cases, the sound-emitting object may be associated with other movements. For example, the sound can be a moving car, a person splashing in a pool, or a hammer hitting a nail. Thus, in some embodiments, the processor 210 may identify a representation of a sound-emitting object by identifying at least one moving object in the user's environment. Step 5206 may include background subtraction between a series of captured images to identify motion, which may aid motion detection if the user 100 is stationary. Step 5206 may use other subtraction techniques to determine whether the observed motion is temporal, and to determine whether the observed motion is associated with the received audio signal based on the motion period and the sound period. In step 5208, process 5200 can include generating a stereo representation based on the location of the sound-emitting object, the stereo representation including a first audio signal and a second audio signal, the first audio signal being different from the second audio signal in at least one respect to simulate the object The location relative to the user. For example, as shown in the first signal 5104 and the second signal 5106 of FIG. 51 , the first audio signal and the second audio signal may be different.

For example, in some embodiments, the sound associated with the sound-emitting object may be attenuated in the first audio signal relative to the second audio signal. The processor 210 may increase the amplitude of the sound by a factor of less than 1 to generate a first audio signal, which may have a smaller intensity than the original sound. Alternatively or additionally, the processor 210 may multiply the sound by a factor greater than 1 to generate a second audio signal having an intensity greater than the original sound and greater than the first audio signal. In some embodiments, the amplification and attenuation may be performed digitally by the processor 210 . Amplification and attenuation may also be performed by amplifier or attenuator circuits housed in device 110 .

Additionally, the degree of attenuation or amplification may be determined based on the location of the sound emitting object (eg, the location determined at step 5206 of process 5200). The processor 210 may perform the conversion of position to attenuation and/or amplification. For example, the processor 210 may calculate the attenuation factor as a function of the direction toward the sound-emitting object relative to the centerline of the field of view of the camera 1730 . The processor 210 may determine the attenuation factor by multiplying the angle towards the sound emitting object by a factor (eg, 0.5). For example, referring to FIG. 50B , the sound emitting object (ie, the individual 5002 ) is located between 5 and 10 degrees from the center of the field of view 5006 . The processor 210 may divide the sound from the individual 5002 to determine an attenuation factor of (10 degrees x 0.5) = 5, making the first signal five times quieter than the second signal. As an additional example, if the individual 5002 were located on the centerline 5008 of Figure 50B, the attenuation factor would be 0 (0 degrees x 0.5 = 0) such that the first signal has equal volume as the second signal. In this way, the sound of the sounding object from far from the centerline of the field of view of the camera 1730 results in a more attenuated first signal than the sound of the sounding object from near the centerline of the field of view of the camera 1730 . This example is for illustrative purposes only and does not necessarily limit this embodiment.

In embodiments where the distance from the user to the object emitting the sound is determined, the attenuation factor may be based on distance rather than direction, or in addition to direction. For example, if an object is farther away from user 100 , the attenuation factor may be less than an object that is close to user 100 but in the same direction relative to user 100 . This can simulate stereo sound and enable more accurate sound localization by the user 100, as the difference in arrival time and volume in each ear decreases as the distance from the ear increases. Additional methods of determining attenuation factors are envisaged, such as alternative conversion methods. Furthermore, the processor 210 may calculate the amplification factor based on the position of the sound emitting object.

The first audio signal may also or alternatively differ from the second audio signal by being delayed. That is, the sound associated with the sound-emitting object may be delayed in the first audio signal relative to the second audio signal, as previously shown by delay 5112 in FIG. 51 . The processor 210 may calculate the delay based on the position of the sound emitting object, similar to the calculation of the attenuation or amplification factor described above. For example, in some embodiments, the sound associated with the sound-emitting object is delayed in the first audio signal by a delay duration based on the distance of the sound-emitting object from the user. Furthermore, the sound associated with the sound-emitting object is delayed in the first audio signal by a delay duration based on the direction of the sound-emitting object relative to the user. As mentioned above, in some embodiments, the delay may be based on both the direction and distance of the sounding object. For example, if the sound-emitting object is close to the user and to the user's right side, the sound will reach the user's right ear earlier than the user's left ear. The difference in arrival times can be large compared to the total travel time of the sound. For example, if the sound originating object is 6 inches from the user's right ear, and the user's right ear is 6 inches from the user's left ear, the sound takes twice as long to reach the user's left ear as it takes to reach the user's right ear. However, if the sound emitting object is 100 feet away from the user, the time to reach the user's ear is small compared to the total time the sound travels. Therefore, the processor 210 can determine the attenuation factor based on both the direction and the distance of the sound emitting object, so as to better realize the sound localization ability of the user.

In some scenarios, the sound-emitting object may be outside the field of view of the camera 1730, or may not produce motion detectable by the camera 1730, such as a radio. The processor 210 may then be unable to determine the direction and/or distance to the sound based on the plurality of images received. Thus, in order to enable sound localization even when the sound emitting object is outside the field of view of the camera 1730, the device 110 may include multiple microphones. For example, the at least one microphone may include a first microphone, and the hearing aid system may also include a second microphone. The processor 210 may be configured to determine the difference between the time of arrival of the sound associated with the sound emitting object at the first microphone and the time of arrival of the sound associated with the sound emitting object at the second microphone. For example, the processor 210 may analyze audio signals received from multiple microphones to determine whether the audio signals match, such as determining the timing of peak intensities or comparing other waveform characteristics. Additionally, the processor 210 may apply a timing window based on the speed of sound and the distance between the microphones, such that when a sound is measured at the first microphone, its similarity is only analyzed when the sound reaches the second microphone within the timing window. If the received audio signals are from the same source, the processor 210 may determine the difference in arrival times. The duration of the delay may then be based on the difference between the sound arrival time associated with the sound emitting object at the first microphone and the sound arrival time associated with the sound emitting object at the second microphone. In some embodiments, the duration of the delay may be decreased or increased relative to the difference in arrival time, which may enable the user to enhance sound localization capabilities.

In some embodiments, generating the first audio signal and the second audio signal in step 5208 may further include selectively adjusting at least one audio signal associated with the sound-emitting object. That is, adjusting may include changing the pitch or playback speed of the audio signal. For example, the adjustment may include remapping the audio frequency or changing the speech rate associated with the audio signal. In some embodiments, the adjustment may include other methods of amplifying the first audio signal relative to other audio signals, such as operation of a directional microphone, changing one or more parameters associated with the microphone, or digitally processing the audio signal. Conditioning may include attenuating or suppressing one or more audio signals associated with the person. The attenuated audio signal may include audio signals associated with other sounds detected in the user's environment, including other speech such as the second audio signal. For example, the processor 210 may selectively attenuate the second audio signal based on determining that the second audio signal is not associated with a person.

In step 5210, process 5200 may include transmitting the stereo representation to a hearing aid interface device configured to provide sound based on the first audio signal to a first ear of the user and sound based on the second audio signal To the user's second ear. For example, a stereo representation may be sent to the auditory interface device 1710 connected to the user's right ear, and to the auditory interface device 1710 connected to the user's left ear, thereby providing the user 100 with sound corresponding to the received audio signal. In some embodiments, auditory interface device 1710 may include a speaker associated with the earpiece. For example, an auditory interface device may be inserted at least partially into a user's ear for providing audio to the user. The auditory interface device may also be external to the ear, such as a behind-the-ear hearing device, one or more earphones, small portable speakers, and the like. In some embodiments, the auditory interface device may include a bone conduction microphone configured to provide audio signals to the user through vibrations of the user's skull. Such devices may be placed in external contact with the user's skin, or may be surgically implanted and attached to the user's bone.

On-the-spot changes in vocal characteristics

Consistent with the disclosed embodiments, hearing aid systems may selectively adjust sounds based on speech characteristics to enable users to better understand individuals with speech impairments, accents, or other speech characteristics that may hinder user understanding. While existing hearing aid systems can amplify sound to overcome hearing loss, these systems may not remove barriers to understanding. For example, users of hearing aid systems may have cognitive impairments in addition to hearing loss, and the simple amplification methods provided by traditional hearing aid systems do not address these cognitive impairments. Furthermore, even when the user does not exhibit cognitive impairment, the user may encounter someone with an accent or speech impairment, and loudspeaker does not solve these problems, or worse. Thus, the hearing aid system of the present invention can selectively adjust the audio associated with speech to reduce comprehension barriers.

The user 100 may wear a hearing aid device that conforms to the camera-based hearing aid device described above. For example, the hearing aid device may be an auditory interface device 1710 as shown in Figure 17A. Auditory interface device 1710 may be any device configured to provide auditory feedback to user 100 . An auditory interface device 1710 may be placed in each ear of the user 100, similar to conventional auditory interface devices. As mentioned above, the auditory interface device 1710 may be of various styles, including in-canal, completely in-canal, in-ear, behind-the-ear, supra-aural, in-canal receiver, open mount, or various other styles. Auditory interface device 1710 may include one or more speakers for providing auditory feedback to user 100, a microphone for detecting sounds in the environment of user 100, internal electronics, a processor, memory, and the like. In some embodiments, auditory interface device 1710 may include one or more communication units in addition to or instead of a microphone, and be one or more receivers for receiving signals from device 110 and transmitting signals to user 100 . The auditory interface device 1710 may correspond to the feedback output unit 230 or may be separate from the feedback output unit 230 and may be configured to receive signals from the feedback output unit 230 .

In some embodiments, the auditory interface device 1710 may include a bone conduction headset 1711, as shown in FIG. 17A. Bone conduction earphones 1711 may be surgically implanted and may provide audible feedback to user 100 through bone conduction of sound vibrations to the inner ear. The auditory interface device 1710 may also include one or more earphones (eg, wireless earphones, over-ear earphones, etc.) or portable speakers carried or worn by the user 100 . In some embodiments, the auditory interface device 1710 may be integrated into other devices, such as the user's Bluetooth ^™ headset, glasses, helmets (eg, motorcycle helmets, bicycle helmets, etc.), hats, and the like.

Auditory interface device 1710 may be configured to communicate with a camera device such as apparatus 110 . Such communication may be via a wired connection, or may be performed wirelessly (eg, using Bluetooth ^™ , NFC or wireless forms of communication). As described above, device 110 may be worn by user 100 in various configurations, including physically attached to a shirt, necklace, belt, eyeglasses, wristband, button, or other item associated with user 100 . In some embodiments, one or more additional devices such as computing device 120 may also be included. Accordingly, one or more of the processes or functions described herein with respect to apparatus 110 or processor 210 may be performed by computing device 120 and/or processor 540 . Apparatus 110 may also use one or more microphones of auditory interface device 1710 , and thus, references to microphone 1720 used herein may also refer to a microphone on auditory interface device 1710 .

Processor 210 (and/or processors 210a and 210b ) may be configured to detect individuals within the environment of user 100 . 53 is a schematic diagram illustrating an exemplary environment for using a hearing aid that provides field modification of sound characteristics consistent with the present disclosure. As shown in FIG. 53 , user 100 wearing device 110 may be physically present in the environment and individual 5302 produces sound 5304 . Although Figure 53 shows the individual 5302 as speaking, the microphone of the device 110 may also capture other sounds from the user's 100 environment, such as machines, animals, or naturally occurring sound sources such as wind. Accordingly, the processor 210 may be configured to identify and isolate the sound corresponding to speech from among the plurality of sounds captured by the microphone of the device 110 . For example, the sound produced by a machine usually has a period of time corresponding to the movement of the machine, such as the sound of each cycle of a running motor, or the sound of each stroke of a hand drill. In these cases, the processor 210 may filter the temporal sounds to isolate speech. As another example, the processor 210 may filter sounds that are louder or quieter than a typical speech volume range, sounds that have harmonics other than those of typical speech, or pitches that are outside the typical speech pitch. The processor 210 may employ signal analysis methods such as Fourier transform to isolate the speech signal.

The isolation of speech from other sounds may allow a user wearing a hearing aid system according to the present disclosure to better understand conversations with others. For example, the processor 210 may selectively adjust the sound by increasing the volume of the sound associated with speech and reducing or even eliminating the sound associated with sources other than speech. Therefore, hearing aid systems can help users focus on the conversation and avoid distractions. However, in some cases, selectively increasing the volume of the sound may not be sufficient for the user to understand the speaker. For example, as discussed above, users may have cognitive impairments that inhibit understanding typical speech, such as taking longer than usual to distinguish words. Users may also have physical barriers to understanding speech, such as hearing loss in certain frequencies corresponding to the human sound range. The user may also have cultural differences with the speaker, making the speaker difficult to understand (such as accent).

To address these issues, certain embodiments of the present disclosure may provide additional selective adjustment of speech to further improve user understanding. For example, Figure 54A is a schematic diagram of an audio signal acquired by a hearing aid system consistent with the present disclosure, and Figure 54B is a schematic diagram of an audio signal played back by a hearing aid system consistent with the present disclosure. 54A and 54B may be spectrograms derived from a microphone of device 110. FIG. A comparison of the acquired audio signal of FIG. 54A with the played back audio signal of FIG. 54B illustrates an embodiment of selective adjustment as in an embodiment of the present disclosure.

The processor 210 may separate words from longer sound samples corresponding to speech by removing non-speech sounds as described above, and separating the sounds associated with speech into segments based on periods of silence indicating word breaks. For example, Figure 54A conceptually represents words extracted from speech. In this example, the word may be a word that is difficult for a user to understand when spoken by a speaker, or a word that is easily confused with similarly pronounced words due to, for example, the speaker's accent. For example, the graph shown in the words of Figure 54A may correspond to a speaker saying the word "hearing" which may sound likely in certain accents or to some people with certain speech impairments Like "earring" or even "erin". That is, some speakers may drop the leading "h" or the final "g" in the word "hearing," or may change the vowel pronunciation.

The processor 210 may also divide the identified words into phonemes. For example, as shown in Figure 54A, the processor 210 may separate the word "hearing" into four phonemes: the "h" phoneme in region 5402, the "ea" phoneme in region 5404, the "r" phoneme in region 5406, and the region The "ng" phoneme in 5408. The processor 210 may identify phonemes by matching a spectrogram derived from the audio with a spectrogram library stored in association with a string describing the sound. For example, the processor 210 may store the library in the memory 550 of the device 110, or the processor 210 may access a database of the storage library. For example, a spectrogram for region 5402 may be stored in a library and linked to the letter "h". Additionally, the library can store indications that phonemes should be emphasized to enhance user understanding. For example, as mentioned above, some accents light the "h" sound, or drop the "h" sound entirely, while the library can store an indication that the "h" sound should be emphasized when it is detected. Additionally, the library can store conditional emphasis instructions, such as "h" should be emphasized at the beginning of a word, not a rule in the middle. The library can also store emphasis rules, such as introducing a delay before a phoneme, increasing the volume, decreasing the volume, increasing the duration, or decreasing the duration.

Thus, Figure 54B shows the rules that the reproduced sound and the "h" sound generated by the processor 210 based on the received sound of Figure 54A should be elongated and amplified at the beginning of the word to enhance the user's understanding of the word. Thus, the "h" phoneme in region 5402 of Fig. 54B has a greater intensity than the sound in region 5402 of Fig. 54A, amplified from the initial intensity (if around 20,000 units) to over 30,000 arbitrary intensity units. Additionally, the "h" phoneme in Figure 54B begins earlier than the beginning of region 5402, showing that the phoneme has an increased duration to further enhance user understanding. In some embodiments, the processor 210 may also shorten other phonemes in the word so that the total word duration is constant, thereby preventing speech and speech that may be confusing to a user, such as for a user reading lips to aid listening comprehension. The delay between hearing. In some embodiments, one or more phonemes may have an increased duration, while other phonemes may be unchanged. Therefore, the duration of the entire word can be increased. In some embodiments, the duration of one or more spaces between consecutive words may be reduced to avoid cumulative delays due to increased phoneme durations.

In some embodiments, the selective adjustment may include changing the pitch of the speaker's speech to make the sound more perceptible to the user 100 . For example, the user 100 may have less sensitivity to tones within a certain range, and the adjustment of the audio signal may adjust the pitch of the received signal 5102. For example, user 100 may experience hearing loss in frequencies above 10 kHz, and processor 210 may remap higher frequencies (eg, at 15 kHz) to frequencies below 10 kHz. In some embodiments, the processor 210 may be configured to vary the speech rate associated with one or more audio signals.

55A is a flowchart illustrating an exemplary process for selectively conditioning an audio signal, consistent with the disclosed embodiments. Process 5500A may be performed by one or more processors associated with apparatus 110, such as processor 210. The processor may be included in the same common housing as the microphone 1720 and camera 1730, which may also be used during process 5500A. For example, apparatus 110 may include at least one microphone configured to capture sound from the user's environment. In some embodiments, some or all of process 5500A may be performed on a processor external to device 110, which may be included in the second housing. For example, one or more portions of process 5500A may be performed by a processor in auditory interface device 1710 or an auxiliary device such as computing device 120 or display device 2301 . In such an embodiment, the processor may be configured to receive the captured images and sounds via a wireless link between the transmitter in the common housing and the receiver in the second housing.

In step 5502, process 5500A can include receiving a plurality of audio signals representing sounds captured by at least one microphone. In some embodiments, process 5500A may receive and combine multiple audio signals from multiple microphones. For example, device 110 may include a first microphone designed to collect sounds having low frequencies, and a second microphone designed to collect sounds having high frequencies. Step 5502 may then combine the sounds into a single audio signal representing both low and high frequencies.

In step 5504, process 5500A may include identifying a first audio signal of the plurality of audio signals, the first audio signal being associated with the individual. For example, as described above, process 5500A may remove audio signals from the plurality of audio signals having frequencies, periods, pitches, and volumes that are outside the range of normal human conversation. In some cases, the user may be in the vicinity of multiple speakers. In these cases, process 5500A may select the loudest audio signal, which may correspond to the closest speaker, which may be the speaker the user wishes to hear clearly. Alternatively, process 5500A may receive an indication from the user to focus on signals from different speakers. For example, process 5500A may cause auditory interface device 1710 to play audio from a first individual, receive a button press or other indication from the user, and then cause audio from a second individual to be played by auditory interface device 1710.

In step 5506, process 5500A can include processing the first audio signal to selectively adjust at least one speech characteristic of the individual. In process 5500A, a speech feature may be any characteristic of an individual speech that may inhibit a user's comprehension. Examples of speech features may include, but are not limited to, accent, speech disturbances (such as slurred speech, stuttering, speech jerks, abnormal pressure, abnormal tongue movement, or missing teeth), disturbing sounds such as whistling, or sound quality (such as high pitch or dysphonia) , i.e. hoarse voice).

For example, in some embodiments, the at least one speech characteristic may include the individual's accent, and the selective adjustment may include changing the accent. A memory such as memory 550 or another database accessed by device 110 may store characteristics of multiple accents. For example, memory 550 may store selective adjustment rules for British accents for causing phonemic substitution of the user's local accent. For example, some speakers of British English may replace the "t" phoneme with a glottal pause. As a result, memory 550 may include selective adjustment rules to replace the detected glottal pause in the first audio signal with a "t" tone, thereby making it easier for British English speakers to understand for American English speakers.

As another example, in some embodiments, the at least one speech feature may include slurred speech by the individual, and the selective adjustment includes removing the slurred speech. In this case, the processor 210 may replace the "th" sound with the "s" sound in the first audio signal. However, since slurred speakers may appropriately use the "th" sound in some words, the processor 210 may include a natural language processing algorithm to determine whether the words identified in the first audio signal should use "th" instead of the "s" sound, and thus avoid replacing "th" with the "s" sound. Alternatively, the processor 210 may access the dictionary and determine whether the observed word is a real word, and replace the "th" sound with the "s" sound. For example, the first audio signal may include a representation of the speaker saying "thit". The processor 210 may determine that "thit" is not a word in the dictionary. The processor 210 may then determine that "sit" is a word in the dictionary and selectively adjust the individual's lisp characteristics.

As yet another example, the at least one phonetic feature may include the pronunciation of a word, and the selective adjustment includes changing the pronunciation of the word. For example, an individual (such as a child) might mispronounce the word "cookie" as "tootie". Furthermore, the individual may have no other identifiable vocal characteristics that interfere with the user's comprehension. The processor 210 may then selectively adjust the phonetic characteristics of the mispronounced word "cookie" by playing a recording of the individual correctly saying the word "cookie" each time the individual mispronounced it. In other words, the processor 210 may replace entire words with correctly pronounced words, rather than replacing individual phonemes. Additionally, the speech features may include multiple pronunciations of words, and the processor 210 may access memory containing replacement audio files for each of the multiple words. Alternatively, the processor 210 may generate a correct pronunciation when a mispronounced word is detected. The processor 210 may also adjust the generated pronunciation to match the individual's speech characteristics, such as pitch, pitch, and quality.

In some embodiments, the selective adjustment may also include altering the individual's speech to simulate the speech of a second individual. For example, the processor 210 may create a transcription of the first audio signal and provide the transcription to a speech synthesis algorithm with speech selected to match the user's preference. Alternatively, the processor 210 may filter components or add other components to the first audio signal to enhance or reduce certain characteristics and to make the first audio signal more closely match characteristics of another speech. For example, the processor 210 may change the individual's speech by changing the pitch of the individual's speech. Thus, if the individual is male, the processor 210 may process the first audio signal to increase its pitch to more clearly match the female's speech, or may add the overtone properties of the particular person's speech. Selective conditioning may also include translation, such as providing machine translation transcribing into the user's language, and then reading the translated text aloud using a speech synthesis algorithm. Selective adjustments may also include playing the individual's speech at a slower or faster rate, eg, by playing one or more words faster or slower. In some embodiments, this may also include reducing or increasing the duration of silence periods between words to account for changes in the duration of the spoken words.

An individual's speech characteristics can be identified during processing. Thus, in some embodiments, step 5506 may include analyzing the first audio signal to determine at least one speech feature by, for example, frequency analysis or phoneme strength and duration analysis. Step 5506 may also include determining the user's preference for selective adjustments, such as instructions for selectively adjusting British accents or emphasizing the letter "h" in speech. For example, user preferences may be stored in memory 550 . Memory 550 may also store processing algorithms and constants, such as an algorithm that emphasizes the letter "h" by amplifying the identified portion of the audio signal corresponding to the letter "h" to 1.5 times the original audio and extending the duration by 10%.

Alternatively or additionally, step 5506 may include identifying the voice signature in the first audio signal, and determining the identity of the individual based on the voice signature. For example, speech feature extraction can be performed by extracting spectral features (also known as spectral properties, spectral envelopes, or spectrograms) from clean audio of a single speaker. The audio signal may include short samples (eg, one second long, two seconds long, etc.) of a single speaker's speech isolated from any other sounds, such as background noise or other sounds. This clean audio can be input into a computer-based model, such as a pre-trained neural network, which can output a signature of the speaker's speech based on the extracted features. In some embodiments, the speech signature may be speech impairment, pronunciation errors, speech rate, or accent. For example, British accents may share common spectral characteristics that can be identified as speech signatures. Furthermore, individuals may mispronounce words (such as commonly used words) in unique ways, and a spectrogram of mispronounced words may be part of an individual's speech signature. Likewise, an individual's speed disorder may cause certain phonemes to be absent from his speech, or alternatively, a certain phoneme to appear at an unusual rate. The presence or absence of such phonemes can also form a speech signature.

The output signature can be a vector of numbers. For example, for each audio sample submitted to a computer-based model (eg, a trained neural network), the computer-based model may output a set of numbers that form a vector. Audio data captured by one or more microphones of the hearing aid system may be processed using any suitable computer-based model to return an output signature. In an example embodiment, the computer-based model may detect and output various statistical properties of the captured audio, such as the average loudness or average pitch of the audio, the spectral frequency of the audio, the loudness or pitch variation of the audio, the rhythm of the audio mode, etc. These parameters can be used to form an output signature comprising a set of numbers forming a vector.

Once the speech signature is established, step 5506 may include executing one or more speech recognition algorithms, such as hidden Markov models, dynamic time warping, neural networks, or other techniques, by accessing a database that includes the voiceprints of one or more individuals . Accordingly, the processor 210 may determine the identity of the individual based on the voice signature. Additionally, after determining the identity, the processor 210 may also access memory to determine at least one voice characteristic stored in the memory in association with the individual.

To illustrate further, user 100 may have a low-pitched friend who cannot pronounce the letter "1". The processor 210 may establish a voice signature for the friend based on the friend's pitch and the overtones produced by the husky quality of the friend's voice. Additionally, the voice signature may notice that the letter "l" is not recognized in the audio signal from the friend. The voice signature may be stored in memory 220 . Additionally, the user may specify that the friend's inability to speak words containing "l" inhibits the user's understanding of the friend. The processor 210 may store a rule that when the friend's voice signature matches the first audio signal (indicating that the user is conversing with the friend), some segments of the first audio signal should be replaced by audio signals appropriately representing the "1" sound. For example, the processor 210 may also use natural language processing methods to determine whether the original sound is correct and avoid inserting the "1" sound. The processor 210 may also selectively adjust the speech to a higher pitch and remove the overtones responsible for the husky quality of the friend's speech.

In some cases, speech features may be general to a language, rather than limited to specific speakers. For example, some words in English, known as near-homonyms, sound similar except for a small difference (such as a stronger emphasis on individual letters). 'refuse' and 'refuge', 'hiss' and 'his', 'advice' and 'advise' can all be considered nearly homophones Character. In certain embodiments, the hearing aid system of the present disclosure may enhance a user's ability to distinguish near homophones. For example, as previously described, the processor 210 may identify words in the first audio signal. The processor 210 can then access the database to determine the near homophones of the word. For example, the database may be stored in memory 220 or accessed through a mobile device such as computing device 120 . The database may store a pre-populated list of near homophones of the language understood by the user. If no near homophones exist in the database, the processor 210 may move to analyzing the next word in the first audio signal. Alternatively, if the word exists, the processor 210 may compare the word in the received audio to the audio file or distinguishing characteristics to determine the difference between the word and the near homophone. Once the difference is identified, the processor 210 may increase at least one of the volume or the duration of the phoneme corresponding to the difference. For example, an entity near the user might say the word "his", which is commonly pronounced "hiz" in American English. The processor 210 may determine that the individual spoke the word "his" and that the word "hiss" (pronounced with the elongated, soft "s" sound in American English) is a near homophone. The processor 210 may then determine the difference between "his" and "hiss" as the pronunciation of the ending "s" and increase the volume of the segment of the first audio signal corresponding to the ending "s". In this way, a user wearing the hearing aid system of the present disclosure can understand the individual more clearly.

After processor 210 has selectively adjusted the speech features, if adjustment is required, processor 210 proceeds to step 5508 of process 5500A. In step 5508, process 5500A may cause the processed first audio signal to be transmitted to an auditory interface device configured to provide sound to the user's ear. For example, the first audio signal may be sent to the auditory interface device 1710 connected to the user's ear, or to two auditory interface devices connected to both ears of the user, thereby providing the user 100 with an audio signal corresponding to the received audio signal the sound of. In some embodiments, auditory interface device 1710 may include a speaker associated with the earpiece. For example, an auditory interface device may be inserted at least partially into a user's ear for providing audio to the user. The auditory interface device 1710 may also be external to the ear, such as a behind-the-ear hearing device, one or more earphones, a small portable speaker, and the like. In some embodiments, the auditory interface device may include a bone conduction microphone configured to provide audio signals to the user through vibrations of the user's skull. Such devices may be placed in external contact with the user's skin, or may be surgically implanted and attached to the user's bone.

In addition to the speech signature matching process, hearing aid systems according to the present disclosure may also rely on visual recognition techniques to identify individuals speaking. 55B is a flowchart illustrating an exemplary process for determining speech characteristics based on visual recognition of an individual, consistent with disclosed embodiments. The visual recognition method of process 5500B shown in Figure 55B may be used in place of or in conjunction with step 5504 of process 5500A.

For example, apparatus 110 may include a wearable camera configured to capture multiple images from the user's environment, and processor 210 may perform the steps of process 5500B. Accordingly, in step 5512, process 5500B may include receiving a plurality of images captured by the camera. For example, device 110 may capture an image and store a representation of the image compressed as a JPG file. As another example, device 110 may capture a color image, but store a black and white representation of the color image. As yet another example, device 110 may capture an image and store different representations of the image (eg, a portion of the image). For example, device 110 may store a portion of an image that includes the face of a person appearing in the image, but does not substantially include the environment surrounding the person. As yet another example, the apparatus 110 may store the representation of the image at a reduced resolution (ie, at a lower resolution than the captured image). Storing the representation of the image may allow device 110 to conserve storage space in memory 550 . Additionally, processing the representation of the image may allow device 110 to increase processing efficiency and/or help maintain battery life.

In step 5514, process 5500B can include identifying a representation of the individual in at least one of the plurality of images. Various image detection algorithms can be used to identify individuals, such as Haar cascade, Histogram of Oriented Gradients (HOG), Deep Convolutional Neural Networks (CNN), Scale Invariant Feature Transform (SIFT), and others. In some embodiments, the processor 210 may be configured to detect a visual representation of the individual, eg, from a display device.

In some embodiments, step 5512 may include identifying at least one lip movement or lip position associated with the individual's mouth based on analysis of the plurality of images to assist in identifying the individual's representation in the plurality of images, as above as described with reference to FIG. 23 . For example, many individuals may be within the field of view of the camera 1760, but one individual may be speaking. Thus, in order to determine how to selectively adjust the first audio signal, the processor 210 may identify the speaking individual from among the plurality of individuals. The processor 210 may also identify the face of the individual 2310 using various other techniques or characteristics, such as color, edge, shape, or motion detection algorithms.

In step 5516, process 5500B may include determining the identity of the individual based on the representation. Step 5516 may include performing a facial analysis of the individual image. Thus, the processor 210 may identify facial features on the individual's face, such as eyes, nose, cheekbones, chin, or other features. The processor 210 may analyze the detected features using one or more algorithms, such as principal component analysis (eg, using eigenfaces), linear discriminant analysis, elastic bundle map matching (eg, using Fisher faces), local binary patterns Histogram (LBPH), Scale Invariant Feature Transform (SIFT), Accelerated Robust Feature (SURF), etc.

In step 5518, process 5500B can include accessing memory to determine at least one speech characteristic stored in memory in association with the identity. For example, processor 210 may identify individuals in the image in step 5514. The processor 210 may further analyze the representation of the individual in the image at step 5516 to determine characteristics of the person's face. The processor 210 may then compare the determined characteristics to a database having sets of characteristics associated with the identity of the individual and obtain the identity of the individual (such as a name). At step 5518, the processor 210 may access the same or an alternate database to obtain selective adjustment rules applicable to the speech characteristics of the identified individual. Additionally, as described above, process 5500B can be combined with a voice signature recognition process to provide greater visibility into the identity of the individual, for example, where the individual's face is obscured by glasses or facial hair, or the individual's speech is obscured by ambient noise certainty.

Selective conditioning based on voice signature and lip reading

Humans have distinct and distinct voices. While some people have excellent phonetic memory and can easily recognize their first elementary school teacher, others may have difficulty recognizing their closest friends from just their phonetics, especially when an environment has several voices. Therefore, there is a need to identify active speakers or determine which speech of multiple speeches to focus on. For example, when user 100 is at the park with his child, he may wish to amplify his child's speech relative to the speech of other nearby children.

The disclosed hearing aid systems may be configured to use speech signatures in conjunction with lip reading to selectively condition or otherwise process an individual's speech. The hearing aid system may receive audio signals from the user's environment representing sounds captured by the microphone. Sound can be used to determine an individual's voice signature that can be stored. The voice signature can be used to identify individuals within the user's environment. For example, a hearing aid system can determine whether an individual is known to the user by comparing the detected speech signature to the stored speech signature. The hearing aid system can also detect individual lip movements based on images received from the camera, which can also be used to identify active speakers. While speech signature detection and lip reading can be performed separately to identify an individual or active speaker, each of these alone may lead to some degree of uncertainty. When used in combination (ie, speech signature and lip reading), the hearing aid system can identify speech to be selectively conditioned, transcribed, or otherwise processed in a more efficient and/or effective manner.

56 is a schematic diagram of an exemplary hearing aid system 5600 for selectively adjusting sound, consistent with disclosed embodiments. Hearing aid system 5600 is shown in simplified form in Figure 56, and hearing aid system 5600 may include additional elements or may have alternative configurations, eg, as shown in Figures 5A-5C. As shown, the hearing aid system 5600 includes a wearable camera 5601 , a microphone 5602 , a processor 5603 , a transceiver 5604 and a memory 5605 .

Wearable camera 5601 may be configured to capture multiple images from the user's 100 environment. For example, wearable camera 5601 may be camera 1730, as described above. The wearable camera 5601 may have an image capture rate that may be configured by the user or based on predetermined settings. In some embodiments, wearable camera 5601 may include one or more cameras, each camera may correspond to image sensor 220 .

Microphone 5602 may be configured to capture sound from the environment of user 100 . For example, microphone 5601 may be microphone 1720, as described above. Microphone 5602 may include one or more microphones. Microphones 5602 may include directional microphones, microphone arrays, multi-port microphones, or various other types of microphones. In some embodiments, microphone 5602 and wearable camera 5601 may be included in a common housing, such as that of device 110.

蓝牙智能、802.15.4或ZigBee)。在一些实施例中，收发器5604可以将数据(例如，原始图像数据、经处理的图像和/或音频数据、提取的信息)从助听器系统5600发送到听觉接口设备和/或服务器250。收发器5604还可以从听觉接口设备和/或服务器250接收数据。在一些实施例中，收发器5604可以将数据和指令发送到外部反馈输出单元230。The transceiver 5604 may be configured to transmit audio signals to an auditory interface device (eg, 1710 ) that is configured to provide sound to the ear of the user 100 . Transceiver 5604 may include one or more wireless transceivers. The one or more wireless transceivers may be any device configured to exchange transmissions over the air interface using radio, infrared, magnetic or electric fields. One or more wireless transceivers may transmit and/or receive data using any known standard (eg, WiFi, Bluetooth

Bluetooth Smart, 802.15.4 or ZigBee). In some embodiments, transceiver 5604 may transmit data (eg, raw image data, processed image and/or audio data, extracted information) from hearing aid system 5600 to auditory interface device and/or server 250. The transceiver 5604 may also receive data from the auditory interface device and/or the server 250. In some embodiments, transceiver 5604 may send data and instructions to external feedback output unit 230 .

The memory 5605 may include an individual information database 5606 and a voiceprint database 5607. Voiceprint database 5607 may include one or more voiceprints of one or more individuals. Individual information database 5606 may include information associating one or more voiceprints stored in voiceprint database 5607 with one or more individuals. The information associating one or more voiceprints with one or more individuals may include a mapping table. Individual information database 5606 may also include information indicating whether one or more individuals are known to user 100 . For example, the mapping table may also include information indicating the relationship of the individual to the user 100 . Optionally, the memory 5605 may also include other components, such as shown in FIG. 20B . Optionally, the memory 5605 may further include an orientation identification module 601 , an orientation adjustment module 602 and a monitoring module 603 as shown in FIG. 6 . Individual information database 5606 and voiceprint database 5607 are shown within memory 5605 by way of example only, and may be located elsewhere. For example, the database may reside in auditory interface device 1710, on a remote server, or in another associated device. Individual information database 5606 and voiceprint database 5607 may be implemented within the same database, or may be implemented as two or more separate databases.

Processor 5603 may include one or more processing units. Processor 5603 may be programmed to receive a plurality of images captured by wearable camera 5601. Processor 5603 may also be programmed to receive a plurality of audio signals representing sounds captured by microphone 5602. In one embodiment, the processor 5603 may be included in the same housing as the microphone 5602 and the wearable camera 5601. In another embodiment, the microphone 5602 and the wearable camera 5601 may be included in the first housing and the processor 5603 may be included in the second housing. In such an embodiment, the processor 5603 may be configured to receive a plurality of image and/or audio signals from the first housing via a wireless link (eg, Bluetooth ^™ , NFC, etc.). Accordingly, the first and second housings may also include transmitters or various other communication components. The processor 5603 may be programmed to analyze the received plurality of audio signals to identify the speech of an individual in the environment of the user 100 using the individual's voiceprint created ad hoc or stored in the memory 5605. The processor 5603 may also be programmed to detect at least one lip movement associated with the individual's mouth based on analysis of the plurality of images. The processor 5603 may also be programmed to identify a first audio signal of the plurality of audio signals associated with the individual's speech based on at least one of a voiceprint or detected lip movement. The processor 5603 may also be programmed to cause selective conditioning or other processing of the first audio signal. The processor 5603 may also be programmed to cause the transceiver 5604 to transmit the selectively conditioned first audio signal to an auditory interface device configured to provide sound to the ear of the user 100 .

In some embodiments, the processor 5603 may be programmed to acquire the individual's voiceprint. For example, a voiceprint may be identified based on an individual's speech collected earlier in the conversation (eg, when the individual is speaking alone without other background noise). As used herein, "earlier" may refer to a previous segment within the same event, or to a previous experience during which a voiceprint was created and stored. The first audio signal can then be identified based on a combination of the voiceprint and the detected lip movement. In some embodiments, upon causing the selective adjustment of the first audio signal, the processor 5603 may be programmed to amplify the first audio signal relative to the at least one second audio signal of the plurality of audio signals or from the first audio signal Remove background noise from audio signals. In some embodiments, upon causing the selective adjustment of the first audio signal, the processor 5603 may be programmed to attenuate at least one second audio signal of the plurality of audio signals relative to the first audio signal, or to filter out a plurality of audio signals at least one second audio signal of the audio signals. In some embodiments, the processor 5603 may be programmed to alter the rate of the recognized speech or to introduce one or more pauses between words or sentences of the recognized speech when causing the selective adjustment of the first audio signal.

In some embodiments, identifying the first audio signal may include determining that the individual is known to the user 100 . Determining that the individual is known to the user 100 may include retrieving information from an individual information database 5606 stored in memory 5605 . Individual information database 5606 may associate voiceprints with individuals and/or images of individuals.

57 is a schematic diagram illustrating an exemplary environment for a user 100 of a hearing aid system 5600 consistent with the disclosed embodiments. Hearing aid system 5600 worn by user 100 may be configured to capture multiple sounds 5704, 5705 and 5706 and identify one or more individuals within the user's environment. For example, among multiple voices 5704, 5705, and 5706, user 100 may wish to focus on voice 5704 from individual 5701. In some embodiments, individual 5701 may be a friend, colleague, relative, or former acquaintance of user 100 . In some embodiments, individual 5701 may not be known to user 100 . As shown in FIG. 57 , the hearing aid system 5600 may be configured to detect one or more movements of the lips 5703 or to recognize speech 5707 associated with an individual 5701 within the environment of the user 100 .

Hearing aid system 5600 may be configured to capture sounds 5704, 5705, and 5706 using microphone 5602. Sound 5704 is associated with speech 5707 of individual 5701 , and sounds 5705 and 5706 may be associated with additional sounds or background noise in the environment of user 100 . In some embodiments, the plurality of sounds may include speech or non-speech sounds of one or more bodies and/or one or more objects in the vicinity of the user 100, ambient sounds (eg, music, tones, or ambient noise), and the like. The processor 5603 may be configured to analyze the audio signal captured by the microphone 5602 to separate the sound 5704 associated with the speech 5707 of the individual 5701. For example, processor 5603 may use a pre-acquired voiceprint of individual 5701, who may be determined to be speaking. If the user 100 knows the individual 5701, the previously stored voiceprint can be retrieved and used. For example, processor 5603 can access voiceprint database 5607, which can include one or more voiceprints corresponding to one or more individuals. Processor 5603 may compare the voiceprint representing voice 5704 with the voiceprint stored in voiceprint database 5607 to determine if a better voiceprint for individual 5701 exists in the database.

Hearing aid system 5600 may be configured to capture one or more facial images 5702 of individual 5701 using wearable camera 5601 . The processor 5603 may be configured to analyze the captured facial image 5702 of the individual 5701. For example, as described above with respect to Figures 23A-23C, the processor 5603 may be configured to use one or more image processing techniques such as convolutional neural networks (CNN), scale-invariant feature transforms (SIFT), oriented gradients Histogram (HOG) features or other techniques) to detect one or more facial features of the individual 5701, which may include, but are not limited to, the mouth 5703 of the individual 5701. The processor 5603 may also be configured to detect one or more points associated with the mouth 5703 of the individual 5701 and track the movement of the lips of the individual 5701 in real time. Based on the detected lip movement, the processor 5603 can identify the audio signal associated with the voice 5704 of the individual 5701 from the plurality of audio signals. For example, the processor 5603 may compare the timing of the detected lip movement to the timing of speech patterns in the received audio signal to determine the audio signal corresponding to the lip movement. Thus, the speech of the individual 5701 can be identified as being separated from other signals and/or processed using lip-reading in conjunction with a pre-acquired voiceprint. This combined analysis can provide better results than using each technique individually.

In some embodiments, the processor 5603 may be programmed to perform selective adjustment of the voice of the user 100 prior to transmitting the audio signal associated with the voice 5704 of the individual 5701. In some embodiments, the selective adjustment of the user's 100 voice may include amplifying or removing background noise from the user's audio signal relative to at least one second audio signal from the user's environment. In some embodiments, the selective adjustment of the user's 100 voice may include attenuating or filtering at least one second audio signal from the user's environment relative to the user's audio signal. In some embodiments, selective adjustment of the user's 100 voice may include changing the user's speech rate or introducing one or more pauses between words or sentences.

58 is a flowchart illustrating an exemplary method 5800 for selectively conditioning or otherwise processing sound in a hearing aid system, consistent with disclosed embodiments. The processor 5603 can perform the process 5800 to selectively adjust the sound from the user's 100 surroundings after the system 5600 captures the audio signal of the individual 5701's speech and/or the image of the individual 5701.

The method 5800 can include the step 5801 of receiving a plurality of images from a user's environment. Multiple images can be captured by the wearable camera. For example, at step 5801, the processor 5603 may receive a plurality of images captured by the wearable camera 5601. In some embodiments, the plurality of images may include a facial image 5702 of the individual 5701 .

The method 5800 can include the step 5802 of detecting at least one lip movement associated with the individual's mouth based on analysis of the plurality of images. For example, at step 5802, process 5603 may detect at least one lip movement or lip position associated with mouth 5703 of individual 5701 based on analysis of the plurality of images. The processor 5603 can identify one or more points associated with the mouth 5703 of the individual 5701. In some embodiments, the processor 5603 can develop a profile associated with the individual's 5701 mouth 5703, which can define a boundary associated with the individual's mouth or lips. Lips identified in an image can be tracked over multiple frames or images to identify lip movement. Thus, the processor 5603 can use various video tracking algorithms as described above.

The method 5800 may also include the step 5803 of receiving a plurality of audio signals representing sounds captured by the at least one microphone. For example, at step 5802, the microphone 5602 can capture a plurality of sounds 5704, 5705 and 5706 and the processor 5603 can receive a plurality of audio signals representing the plurality of sounds 5704, 5705 and 5706. Sound 5704 is associated with the speech of individual 5701 , and sounds 5705 and 5706 may be additional sounds or background noise in the environment of user 100 . In some embodiments, sounds 5705 and 5706 may include speech or non-speech sounds of one or more entities other than individual 5701, ambient sounds (eg, music, tones, or ambient noise), and the like.

The method 5800 can include the step 5804 of obtaining a voiceprint associated with an individual within the user's environment. In some embodiments, the individual may be identified based on at least one of the plurality of images or the plurality of audio signals. For example, step 5804 may include the use of facial recognition, voice recognition, or other means for identifying individuals. Voiceprints can be obtained in various ways. In some embodiments, obtaining the voiceprint may include generating the voiceprint based on previous audio signals associated with the individual's speech. For example, this may include detecting a segment in which the individual 5701 speaks alone, and extracting the individual's 5701 voiceprint during the segment.

In some embodiments, obtaining the voiceprint includes retrieving the voiceprint from a database based on the identification of the speaker in at least one of the plurality of images. For example, individual 5701 may be identified by comparing representations or characteristics of individual 5701 in one or more captured images to entries in individual information database 5606. Based on this comparison, the individual 5701's previous voiceprints can be retrieved from the voiceprint database 5607. The extracted voiceprint or the retrieved voiceprint can be used to analyze the received audio signal to separate and process the speech of the individual 5701. In some embodiments, if no previous voiceprints are available, or if the extracted voiceprints are of higher quality than those retrieved from the voiceprint database 5607 (eg, generated on audio captured in quieter areas) voiceprint, etc.), the newly generated voiceprint may be stored in the database in addition to or in place of the previously stored voiceprint. Step 5804 may use the trained model to determine whether the audio signal includes speech associated with a particular voiceprint, or to provide a probability that the audio signal includes speech associated with a particular voiceprint. In some embodiments, only steps 5801 and 5802 may occur so that only lip reading is performed. In some embodiments, only steps 5803 and 5804 may occur so that only voice signature detection is performed. In some embodiments, all steps 5801-5804 may occur to perform both lip reading and speech signature detection.

The method 5800 can include the step 5805 of identifying a first audio signal of the plurality of audio signals associated with the speech of the individual 5701 based on at least one of a voiceprint or a detected lip movement. For example, this may include separating the first audio signal from one or more audio signals not associated with the individual. For example, at step 5805, the processor 5603 may, based on at least one of the voiceprint created or retrieved at step 5804 or the lip movement detected at step 5802, from the data associated with sounds 5704, 5705, and 5706 An audio signal associated with the speech 5707 of the individual 5701 is identified among the plurality of audio signals. In some embodiments, the processor 5603 may identify an audio signal associated with the voice of the individual 5701 from the plurality of audio signals based on a combination of the voiceprint obtained at step 5804 and the lip movement detected at step 5802 . As described above, once the first audio signal is isolated, the processor 5603 may compare the detected particular lip movement to phonemes or other features identified in the first audio signal. In some embodiments, identifying the first audio signal may include determining that the individual is known to the user. Determining that the individual is known to the user may include retrieving information from a database stored in memory that associates the voiceprint with the individual. For example, the information in the database associating one or more voiceprints with one or more individuals may include a mapping table, which may also include an indication of whether the one or more individuals are known to the user 100 and their relationship to the user 100 Information. Processor 5603 may access individual information database 5606 to retrieve individual information stored in memory 5605 and determine whether the individual is known to the user.

Method 5800 may include a step 5806 of processing the first audio signal. In some embodiments, the processing may include selective conditioning, as described throughout this disclosure. For example, at step 5806, the processor 5603 may perform various forms of selective conditioning on the audio signal associated with the individual's 5701 speech. In some embodiments, processing the first audio signal may include amplifying the first audio signal or removing background noise of the first audio signal relative to at least one second audio signal of the plurality of audio signals. For example, the processor 5603 may amplify the audio signal associated with the speech of the individual 5701 relative to at least one of the audio signals associated with the sounds 5705 and 5706. Amplification may be performed by various means, such as operation of a directional microphone, changing one or more parameters associated with the microphone, or digitally processing the audio signal. The processor 5603 may also remove background noise from the audio signal associated with the individual's 5701 speech. In some embodiments, processing the first audio signal may include attenuating at least one second audio signal of the plurality of audio signals relative to the first audio signal or filtering out at least one second audio signal of the plurality of audio signals. For example, processor 5603 may selectively attenuate at least one of the audio signals associated with sounds 5705 and 5706, or filter out at least one of the audio signals associated with sounds 5705 and 5706. In some embodiments, processing the first audio signal may include changing the rate of the recognized speech or introducing one or more pauses between words or sentences of the recognized speech. For example, the processor 5603 may vary the rate of the recognized speech associated with the audio signal associated with the speech of the individual 5701, or the words or sentences of the recognized speech associated with the audio signal associated with the speech of the individual 5701 Introduce one or more pauses in between. In some embodiments, processing the first audio signal may include changing the pitch of the audio signal associated with the individual 5701's speech. In some embodiments, processing the first audio signal may include transcribing the first audio signal.

Method 5800 may include a step 5807 of transmitting the selectively conditioned first audio signal to an auditory interface device configured to provide sound to a user's ear. For example, transceiver 5604 may transmit the conditioned audio signal to an auditory interface device, such as auditory interface device 1710 , which may provide user 100 with sounds corresponding to the audio signal associated with individual 5701's speech. In some embodiments, the auditory interface device may include a speaker associated with the earpiece. For example, an auditory interface device may be inserted at least partially into a user's ear for providing audio to the user. The auditory interface device may also be external to the ear, such as a behind-the-ear hearing device, one or more earphones, small portable speakers, and the like. In some embodiments, the auditory interface device may include a bone conduction headset 1711 (such as the bone conduction headset 1711 discussed above) configured to provide audio signals to the user through vibrations of the user's skull. Such devices may be placed in external contact with the user's skin, or may be surgically implanted and attached to the user's bone.

In some embodiments, memory 5605 may include a non-transitory computer-readable storage medium storing program instructions for execution by processor 5603 to perform method 5800 as described above.

The foregoing description has been presented for purposes of illustration. It is not exhaustive and is not limited to the precise forms or embodiments disclosed. Modifications and adaptations will be apparent to those skilled in the art from consideration of the specification and practice of the disclosed embodiments. Additionally, although aspects of the disclosed embodiments are described as being stored in memory, those skilled in the art will appreciate that these aspects may also be stored on other types of computer-readable media, such as secondary storage devices, such as hard disks or CD ROM, or other forms of RAM or ROM, USB media, DVD, Blu-ray, Ultra HD Blu-ray, or other optical drive media.

Computer programs based on written descriptions and disclosed methods are within the skill of an experienced developer. The various programs or program modules may be created using any technique known to those skilled in the art, or may be designed in conjunction with existing software. For example, program parts or program modules can be written in .NET Framework, .NET Compact Framework (and related languages such as Visual Basic, C, etc.), Java, C++, Objective-C, HTML, HTML/Ajax combination, XML or HTML and their Comes with a Java applet to design.

Furthermore, although illustrative embodiments have been described herein, those skilled in the art will appreciate having equivalent elements, modifications, omissions, combinations (eg, across aspects of the various embodiments), adaptations, and/or changes based on this disclosure the scope of any and all embodiments. The limitations in the claims should be construed broadly based on the language used in the claims, and not limited to the examples described in this specification or the examples described during execution of the application. These examples are to be construed as non-exclusive. Furthermore, the steps of the disclosed methods may be modified in any way, including by reordering steps and/or inserting or deleting steps. Accordingly, this specification and examples are to be regarded as illustrative only, the true scope and spirit of which is to be indicated by the following claims along with their full scope of equivalents.

Claims (250)

1. A hearing aid system for selectively adjusting sound, the hearing aid system comprising:

a wearable camera configured to capture a plurality of images from an environment of a user;

at least one microphone configured to capture sound from the environment of the user; and

at least one processor programmed to:

Receiving the plurality of images captured by the camera;

receiving a plurality of audio signals representing sound captured by the at least one microphone from the environment of the user;

operating in a first mode to cause a first selective adjustment of a first audio signal of the plurality of audio signals;

determining, based on an analysis of at least one of the plurality of images or the plurality of audio signals, to switch to a second mode to cause a second selective adjustment to the first audio signal, the second selective adjustment differing in at least one aspect relative to the first selective adjustment; and

causing the first audio signal selectively adjusted in the second mode to be transmitted to an auditory interface device configured to provide sound to an ear of the user.

2. The hearing aid system according to claim 1, wherein determining to switch to the second mode comprises identifying that an individual in the environment of the user is speaking based on the analysis of the plurality of images.

3. The hearing aid system according to claim 1, wherein the first selective adjustment comprises an amplification of the first audio signal.

4. The hearing aid system according to claim 1, wherein the second selective adjustment comprises an attenuation of at least one second audio signal of the plurality of audio signals relative to the first audio signal.

5. The hearing aid system according to claim 4, wherein the first audio signal is associated with a first individual's voice and the second audio signal is associated with a second individual's voice.

6. The hearing aid system according to claim 1, wherein the determination to switch to the second mode is further based on a context associated with the at least one of the plurality of images or the plurality of audio signals.

7. The hearing aid system according to claim 6, wherein the context indicates that the user enters a room.

8. The hearing aid system according to claim 6, wherein the context indicates that the user is out of the room.

9. The hearing aid system according to claim 1, wherein the at least one processor is further programmed to cause the first audio signal selectively adjusted in the first mode to be transmitted to an auditory interface device configured to provide sound to the user's ear.

10. A hearing aid system for selectively adjusting sound, the hearing aid system comprising:

a wearable camera configured to capture a plurality of images from an environment of a user;

at least one microphone configured to capture sound from the environment of the user; and

at least one processor programmed to:

receiving the plurality of images captured by the camera;

receiving a plurality of audio signals representing sound captured by the at least one microphone from the environment of the user;

operating in a plurality of modes, wherein the plurality of modes includes a first mode and a second mode, wherein operating in the first mode causes a first selective adjustment to at least one of the plurality of audio signals, and wherein operating in the second mode causes a second selective adjustment to the at least one audio signal, the second selective adjustment being different in at least one respect relative to the first selective adjustment;

selecting the first mode or the second mode based on an analysis of at least one of the plurality of images or the plurality of audio signals; and

causing the at least one audio signal selectively adjusted based on the selected mode to be transmitted to an auditory interface device configured to provide sound to an ear of the user.

11. The hearing aid system according to claim 10, wherein selecting the first mode or the second mode comprises identifying that an individual in the environment of the user is speaking based on the analysis of the plurality of images.

12. The hearing aid system according to claim 10, wherein the first selective adjustment comprises amplification of the at least one audio signal.

13. The hearing aid system according to claim 10, wherein the first selective adjustment comprises changing an amplitude of the at least one audio signal.

14. The hearing aid system according to claim 10, wherein the first selective adjustment comprises changing a pitch of the at least one audio signal.

15. The hearing aid system according to claim 10, wherein the first selective adjustment comprises time stretching the at least one audio signal.

16. The hearing aid system according to claim 10, wherein selecting the first mode or the second mode is based on a context associated with the at least one of the plurality of images or the plurality of audio signals.

17. The hearing aid system according to claim 10, wherein the context indicates that the user is participating in a speech.

18. The hearing aid system according to claim 10, wherein the context indicates that the user is attending a social event.

19. The hearing aid system according to claim 10, wherein the context indicates that the user is entering a room.

20. The hearing aid system according to claim 10, wherein the context indicates that the user is leaving a room.

21. The hearing aid system according to claim 10, wherein selecting the first mode or the second mode is based on a selection received from the user.

22. The hearing aid system according to claim 10, wherein the selection is received from at least one of a smartphone or a laptop computer.

23. The hearing aid system according to claim 10, wherein the background indicates that the user is entering a lecture room, wherein the at least one audio signal comprises speech of a lecturer, and wherein the first selective adjustment comprises amplification of a first at least one audio signal and attenuation of at least another audio signal of the plurality of audio signals comprising background noise.

24. The hearing aid system according to claim 10, wherein the context indicates that the user is leaving a lecture room, wherein the at least one audio signal comprises speech of an active speaker outside the lecture room, and wherein the first selective adjustment comprises amplification of a first at least one audio signal.

25. The hearing aid system according to claim 10, wherein the context indicates that the user is within a predetermined distance of only one individual, and wherein the at least one audio signal comprises speech of the person, and wherein the first selective adjustment comprises amplification of a first at least one audio signal.

26. The hearing aid system according to claim 25 wherein the predetermined distance is five meters or less.

27. A hearing aid system for selectively adjusting sound, the hearing aid system comprising:

a wearable camera configured to capture a plurality of images from an environment of a user;

at least one microphone configured to capture a plurality of audio signals from the environment of the user; and

at least one processor programmed to:

receiving the plurality of images captured by the camera;

receiving a plurality of audio signals representing sound captured by the at least one microphone from the environment of the user;

identifying an individual identified by at least one of the plurality of images or represented by at least one of the plurality of audio signals;

retrieving from memory a regulation profile associated with the at least one recognized individual;

Causing a selective adjustment to a first audio signal of the plurality of audio signals associated with the at least one recognized individual, the selective adjustment determined based on the adjustment profile; and

causing the adjusted first audio signal to be transmitted to an auditory interface device configured to provide sound to an ear of the user.

28. The hearing aid system according to claim 26 wherein the memory is located within the same housing as the wearable camera.

29. The hearing aid system according to claim 26, wherein the at least one processor is further programmed to:

determining at least one modification to the selective adjustment associated with the identified individual; and

updating the adjustment profile based on the modification.

30. The hearing aid system according to claim 29, wherein the at least one processor is further programmed to apply the at least one modification.

31. The hearing aid system according to claim 29, wherein the at least one modification comprises an adjustment made by the user.

32. The hearing aid system according to claim 26, wherein the at least one modification is determined based on an indication of hearing difficulty from the user.

33. The hearing aid system according to claim 32, wherein the indication of difficulty is determined via one of an audio signal from the user or an input from the user received via an interface of the hearing aid system.

34. The hearing aid system according to claim 33, wherein the at least one modification comprises one of an amplification of the first audio signal or a modification of a frequency spectrum of the first audio signal.

35. The hearing aid system according to claim 26, wherein the selective adjustment comprises an attenuation of at least one second audio signal of the plurality of audio signals not associated with the identified individual.

36. The hearing aid system according to claim 26, wherein the adjustment profile comprises a predefined filter for selectively adjusting the first audio signal based on at least one of a frequency rate or an amplitude of the first audio signal.

37. The hearing aid system of claim 26, wherein the at least one processor is further programmed to:

receiving instructions from the user for modification of the selective adjustment; and

Updating the adjustment profile based on the instructions.

38. The hearing aid system according to claim 26, wherein causing selective adjustment of the first audio signal comprises changing at least one of an accent or a pitch of a speech associated with the recognized individual.

39. The hearing aid system of claim 26 wherein the processor is further programmed to:

identifying another recognized individual represented by at least one of the plurality of images or by at least one of the plurality of audio signals;

retrieving from the memory another adjustment profile associated with the other recognized individual;

causing another selective adjustment to a second audio signal of the plurality of audio signals associated with the other recognized individual, the other selective adjustment determined based on the other adjustment profile; and

causing the adjusted second audio signal to be transmitted to the auditory interface device configured to provide sound to the ear of the user.

40. A hearing aid system for selectively adjusting sound, the hearing aid system comprising:

a wearable camera configured to capture a plurality of images from an environment of a user;

At least one microphone configured to capture a plurality of audio signals from the environment of the user; and

at least one processor programmed to:

receiving the plurality of images captured by the camera;

receiving a plurality of audio signals representing sound captured by the at least one microphone from the environment of the user;

identifying a group of individuals represented by at least one of the plurality of images or by at least one of the plurality of audio signals;

retrieving from memory a conditioning profile associated with the group of individuals;

causing selective adjustment of a first audio signal of the plurality of audio signals associated with the group of individuals, the selective adjustment determined based on the adjustment profile; and

causing the adjusted first audio signal to be transmitted to an auditory interface device configured to provide sound to an ear of the user.

41. The hearing aid system according to claim 40 wherein the memory is located within the same housing as the wearable camera.

42. The hearing aid system according to claim 40, wherein the at least one processor is further programmed to:

Determining at least one modification to the selective adjustment associated with the identified individual; and

updating the adjustment profile based on the modification.

43. The hearing aid system according to claim 42, wherein the at least one processor is further programmed to apply the at least one modification.

44. The hearing aid system according to claim 42, wherein the at least one modification comprises an adjustment made by the user.

45. The hearing aid system according to claim 40, wherein the at least one modification is determined based on an indication of hearing difficulty from the user.

46. The hearing aid system according to claim 45, wherein the indication of difficulty is determined via one of an audio signal from the user or an input from the user received via an interface of the hearing aid system.

47. The hearing aid system according to claim 45, wherein the at least one modification comprises one of an amplification of the first audio signal or a modification of a frequency spectrum of the first audio signal.

48. The hearing aid system of claim 40, causing selective adjustment of the first audio signal comprises changing at least one of an accent or a pitch of speech associated with the recognized individual.

49. A hearing aid system for selectively amplifying sound, the hearing aid system comprising:

a wearable camera configured to capture a plurality of images from an environment of a user;

at least one microphone configured to capture sound from the environment of the user; and

at least one processor programmed to:

receiving the plurality of images captured by the wearable camera;

receiving at least one audio signal representing sound captured by the at least one microphone from the environment of the user;

identifying at least one action of the user based on an analysis of at least one of the plurality of images or the at least one audio signal;

causing selective adjustment of the at least one audio signal received by the at least one microphone based on the identified action; and

causing transmission of at least one conditioned audio signal to an auditory interface device configured to provide sound to an ear of the user.

50. The hearing aid system according to claim 49, wherein the at least one action comprises speech of the user.

51. The hearing aid system according to claim 50 wherein identifying the at least one action comprises detecting words spoken by the user based on an analysis of the at least one audio signal.

52. The hearing aid system according to claim 51 wherein the words indicate that the user is not clearly audible.

53. The hearing aid system according to claim 49, wherein the at least one processor is further programmed to:

storing a portion of the at least one audio signal from the user prior to the at least one action; and

based on the at least one action, causing the stored portion of the at least one audio signal to be transmitted to the auditory interface device.

54. The hearing aid system according to claim 49, wherein identifying the at least one action comprises detecting a motion of the user based on an analysis of the plurality of images.

55. The hearing aid system according to claim 54 wherein the motion comprises hand movement of the user.

56. The hearing aid system according to claim 54, wherein the motion comprises a tilting motion of the user.

57. The hearing aid system according to claim 56, wherein causing selective adjustment of the at least one audio signal comprises amplifying the at least one audio signal received from the direction of the tilt motion.

58. A method for selectively amplifying sound, the method comprising:

capturing, by a wearable camera, a plurality of images from an environment of a user;

capturing sound from the environment of the user through at least one microphone;

receiving the plurality of images captured by the wearable camera;

receiving at least one audio signal representing sound captured by the at least one microphone from the environment of the user;

identifying at least one action of the user based on an analysis of at least one of the plurality of images or the at least one audio signal;

cause selective adjustment of the at least one audio signal received by the at least one microphone based on the identified action; and

causing transmission of at least one conditioned audio signal to an auditory interface device configured to provide sound to an ear of the user.

59. The method of claim 58, wherein the at least one action comprises a speaking of the user.

60. The method of claim 59, wherein identifying the at least one action comprises detecting words spoken by the user based on an analysis of the at least one audio signal.

61. The method of claim 60, wherein the words indicate that the user is not audible enough.

62. The method of claim 58, further comprising:

storing a portion of the at least one audio signal from the user prior to the at least one action; and

based on the at least one action, causing the stored portion of the at least one audio signal to be transmitted to the auditory interface device.

63. The method of claim 58, wherein identifying the at least one action comprises detecting motion of the user based on an analysis of the plurality of images.

64. The method of claim 63, wherein the motion comprises a hand movement of the user.

65. The method of claim 63, wherein the motion comprises a tilting motion of the user.

66. The method of claim 65, wherein causing selective adjustment of the at least one audio signal comprises amplifying the at least one audio signal received from the direction of the tilting motion.

67. A non-transitory computer-readable medium storing a set of instructions executable by at least one processor of a computing device to cause the computing device to perform a method for selectively amplifying sound, the method comprising:

Capturing, by a wearable camera, a plurality of images from an environment of a user;

capturing sound from the environment of the user through at least one microphone;

receiving the plurality of images captured by the wearable camera;

receiving at least one audio signal representing sound captured by the at least one microphone from the environment of the user;

identifying at least one action of the user based on an analysis of at least one of the plurality of images or the at least one audio signal;

causing selective adjustment of the at least one audio signal received by the at least one microphone based on the identified action; and

causing transmission of at least one conditioned audio signal to an auditory interface device configured to provide sound to an ear of the user.

68. The non-transitory computer-readable medium of claim 67, wherein the set of instructions is executable by the at least one processor of the computing device to cause the computing device to further perform:

storing a portion of the at least one audio signal from the user prior to the at least one action; and

based on the at least one action, causing the stored portion of the at least one audio signal to be transmitted to the auditory interface device.

69. A hearing aid system for selectively amplifying sound, the hearing aid system comprising:

a wearable camera configured to capture a plurality of images from an environment of a user;

at least one microphone configured to capture sound from the environment of the user; and

at least one processor programmed to:

receiving the plurality of images captured by the camera;

identifying a representation of a first individual in at least one of the plurality of images;

receiving an audio signal representing sound captured by the at least one microphone;

identifying, based on the analysis of the audio signals, a first audio signal associated with a first voice associated with the first individual and a second audio signal associated with a second voice of a second individual;

causing selective adjustment of the first audio signal based on the at least one processor determining that the first audio signal is associated with a higher priority than a priority of the second audio signal; and

causing the selectively adjusted first audio signal to be transmitted to an auditory interface device configured to provide sound to an ear of the user.

70. The hearing aid system of claim 69 wherein determining that the first audio signal is associated with a higher priority comprises determining that the at least one individual is closer to the user than the source of the second voice.

71. The hearing aid system of claim 69 wherein determining that the first audio signal is associated with a higher priority comprises receiving at least one setting configured by the user.

72. The hearing aid system according to claim 71, wherein the at least one setting is received from a paired mobile device.

73. The hearing aid system according to claim 72 wherein the at least one setting is input by the user through a graphical user interface on the mobile device.

74. The hearing aid system according to claim 69, wherein the selective adjustment of the first audio signal comprises amplification of the first audio signal.

75. The hearing aid system according to claim 69, further comprising attenuating the second audio signal.

76. The hearing aid system of claim 69 wherein determining that the first audio signal is associated with a higher priority comprises identifying at least one action based on analysis of at least one of the plurality of images.

77. The hearing aid system according to claim 76, wherein the at least one action comprises a line of sight direction towards an individual associated with the first audio signal.

78. The hearing aid system according to claim 76, wherein the at least one action comprises a pointing action by the user towards an individual associated with the first audio signal.

79. A method for selectively amplifying sound, the method comprising:

capturing, by a wearable camera, a plurality of images from an environment of a user;

capturing sound from the environment of the user through at least one microphone;

receiving the plurality of images captured by the camera;

identifying a representation of a first individual in at least one of the plurality of images;

receiving an audio signal representing sound captured by the at least one microphone;

identifying, based on the analysis of the audio signals, a first audio signal associated with a first voice associated with the first individual and a second audio signal associated with a second voice of a second individual;

causing selective adjustment of the first audio signal based on the at least one processor determining that the first audio signal is associated with a higher priority than a priority of the second audio signal; and

causing the selectively adjusted first audio signal to be transmitted to an auditory interface device configured to provide sound to an ear of the user.

80. The method of claim 79, wherein determining that the first audio signal is associated with a higher priority comprises determining that the at least one individual is closer to the user than a source of the second speech.

81. The method of claim 79, wherein determining that the first audio signal is associated with a higher priority comprises receiving at least one setting configured by the user.

82. The method of claim 81, wherein the at least one setting is received from a paired mobile device.

83. The method of claim 82, wherein the at least one setting is input by the user through a graphical user interface on the mobile device.

84. The method of claim 79, wherein the selective adjustment of the first audio signal comprises amplifying the first audio signal.

85. The method of claim 79, further comprising attenuating the second audio signal.

86. The method of claim 79, wherein determining that the first audio signal is associated with a higher priority comprises identifying at least one action based on an analysis of at least one of the plurality of images.

87. The method of claim 86, wherein the at least one action comprises a direction of a line of sight toward an individual associated with the first audio signal.

88. A non-transitory computer-readable medium storing a set of instructions executable by at least one processor of a computing device to cause the computing device to perform a method for selectively amplifying sound, the method comprising:

capturing, by a wearable camera, a plurality of images from an environment of a user;

capturing sound from the environment of the user through at least one microphone;

receiving the plurality of images captured by the camera;

identifying a representation of a first individual in at least one of the plurality of images;

receiving an audio signal representing sound captured by the at least one microphone;

identifying, based on the analysis of the audio signals, a first audio signal associated with a first voice associated with the first individual and a second audio signal associated with a second voice of a second individual;

causing selective adjustment of the first audio signal based on the at least one processor determining that the first audio signal is associated with a higher priority than a priority of the second audio signal; and

Causing the selectively adjusted first audio signal to be transmitted to an auditory interface device configured to provide sound to an ear of the user.

89. A hearing aid system for selectively amplifying sound, the hearing aid system comprising:

a wearable camera device, the wearable camera device comprising:

at least one camera configured to capture a plurality of images from an environment of a user;

at least one microphone configured to capture sound from the environment of the user; and

at least one first processor programmed to selectively condition audio signals received from the at least one microphone representative of the sound captured by the at least one microphone; and a hearing aid device, the hearing aid device comprising:

at least one speaker configured to provide sound to an ear of the user; and

at least one second processor programmed to:

cause one or more instructions to be transmitted to the wearable camera device;

receiving an adjusted audio signal from the wearable camera device; and

providing sound to the user's ear using the at least one speaker based on the adjusted audio signal.

90. The hearing aid system of claim 89, further comprising a mobile device paired with at least one of the hearing aid device or the wearable camera device.

91. The hearing aid system according to claim 90 wherein the mobile device comprises a user interface for receiving input from the user.

92. The hearing aid system of claim 91 wherein the at least one second processor is further configured to determine the one or more instructions based on the input from the user.

93. The hearing aid system of claim 89 further comprising a range finding device paired with at least one of the hearing aid device or the wearable camera device.

94. The hearing aid system according to claim 90 wherein the mobile device comprises a user interface for providing an output to the user.

95. The hearing aid system of claim 89 wherein the wearable camera device is configured to capture the plurality of images and the sound based on the one or more instructions.

96. The hearing aid system of claim 92 wherein the at least one first processor is programmed to selectively adjust an audio signal based on the one or more instructions.

97. The hearing aid system of claim 89 wherein the conditioned audio signal comprises at least one of an amplified signal or an attenuated signal.

98. The hearing aid system according to claim 90, wherein the mobile device is wirelessly paired with at least one of the hearing aid device or the wearable camera device.

99. A method for amplifying sound in a hearing aid system, the method comprising:

capturing a plurality of images from an environment of a wearable user using at least one camera of a wearable camera device;

capturing sound from the environment of the user using at least one microphone of the wearable camera device;

selectively conditioning, using at least one first processor, audio signals received from the at least one microphone that are representative of the sound captured by the at least one microphone; and

providing sound to the ear of the user using at least one speaker of the hearing aid device by using at least one second processor of the hearing aid device, the at least one second processor programmed to:

cause one or more instructions to be transmitted to the wearable camera device;

Receiving an adjusted audio signal from the wearable camera device; and

providing sound to the user's ear using the at least one speaker based on the adjusted audio signal.

100. The method of claim 99, further comprising pairing a mobile device with at least one of the hearing aid device or the wearable camera device.

101. The method of claim 100, further comprising receiving input from the user via a user interface of the mobile device.

102. The method of claim 101, wherein the at least one second processor is further programmed to determine the one or more instructions based on the input from the user.

103. The method of claim 99, further comprising determining a range using a ranging device paired with at least one of the hearing aid device or the wearable camera device.

104. The method of claim 100, wherein the mobile device includes a user interface for providing output to the user.

105. The method of claim 102, wherein the wearable camera device is configured to capture the plurality of images and the sound based on the one or more instructions.

106. The method of claim 102, wherein the at least one first processor is programmed to selectively adjust an audio signal based on the one or more instructions.

107. The method of claim 99, wherein the conditioned audio signal comprises at least one of an amplified signal or an attenuated signal.

108. The method of claim 100, wherein the mobile device is wirelessly paired with at least one of the hearing aid device or the wearable camera device.

109. A hearing aid system for selectively amplifying sound, the hearing aid system comprising:

a wearable camera configured to capture a plurality of images from an environment of a user, the wearable camera having image capture parameters;

at least one microphone configured to capture sound from the environment of the user; and

at least one processor programmed to perform the steps of:

receiving the plurality of images captured by the camera;

receiving an audio signal representing sound captured by the at least one microphone;

identifying a representation of at least one individual in at least one image of the plurality of images;

Detecting a speech rate associated with the at least one individual based on at least one of the plurality of images or the audio signal; and

cause an adjustment to the image capture parameters of the wearable camera based on the detected speech rate.

110. The hearing aid system according to claim 109 wherein the image capture parameter is a frame rate of the camera.

111. The hearing aid system of claim 109 wherein detecting the speech rate comprises:

identifying at least one lip movement associated with the mouth of the at least one individual based on the analysis of the plurality of images; and

determining the speech rate based on the lip movement.

112. The hearing aid system of claim 109 wherein detecting the speech rate comprises:

identifying a plurality of words spoken by the individual based on the analysis of the audio signal; and

determining the speech rate based on the plurality of words.

113. The hearing aid system of claim 112 wherein identifying the plurality of words comprises using a speech recognition algorithm.

114. The hearing aid system of claim 112 wherein causing the adjustment to the image capture parameters of the wearable camera comprises:

Increasing a frame rate of the camera based on determining that a speed of speech is greater than a threshold.

115. The hearing aid system of claim 112 wherein causing the adjustment to the image capture parameters of the wearable camera comprises:

reducing a frame rate of the camera based on determining that a speed of speech is less than a threshold.

116. The hearing aid system according to claim 109, wherein the image capturing parameter is a resolution of a captured image or a compression of the captured image.

117. The hearing aid system according to claim 109, wherein the at least one processor is further programmed to perform the steps of:

cause an adjustment to the image capture parameters of the wearable camera based on the detected light level.

118. The hearing aid system according to claim 109, wherein the at least one processor is further programmed to perform the steps of:

cause adjustment of the image capture parameters of the wearable camera based on location information.

119. The hearing aid system of claim 109 wherein the at least one microphone has audio capture parameters and the at least one processor is further programmed to perform the steps of:

Causing an adjustment to the audio capture parameters of the at least one microphone based on the detected speech rate.

120. A method for amplifying sound in a hearing aid system, the method comprising:

receiving the plurality of images captured by the wearable camera;

receiving an audio signal representing sound captured by at least one microphone;

identifying a representation of at least one individual in at least one image of the plurality of images;

detecting a speech rate associated with the at least one individual based on at least one of the plurality of images or the audio signal; and

cause an adjustment to an image capture parameter of the wearable camera based on the detected speech rate.

121. The method of claim 120, wherein the image capture parameter is a frame rate of the camera.

122. The method of claim 120, wherein detecting the speech rate comprises:

identifying at least one lip movement associated with a mouth of the at least one individual based on the analysis of the plurality of images; and

determining the speech rate based on the lip movement.

123. The method of claim 120, wherein detecting the speech rate comprises:

Identifying a plurality of words spoken by the individual based on the analysis of the audio signal; and

determining the speech rate based on the plurality of words.

124. The method of claim 123, wherein identifying the plurality of words includes using a speech recognition algorithm.

125. The method of claim 123, wherein causing the adjustment to the image capture parameters of the wearable camera comprises:

increasing a frame rate of the camera based on determining that a speed of speech is greater than a threshold.

126. The method of claim 123, wherein causing the adjustment to the image capture parameters of the wearable camera comprises:

reducing a frame rate of the camera based on determining that a speed of speech is less than a threshold.

127. The method of claim 123, the image capture parameter being a resolution of a captured image or a compression of the captured image.

128. The method of claim 120, wherein the steps further comprise:

cause an adjustment to the image capture parameters of the wearable camera based on the detected light level.

129. The method of claim 120, wherein the steps further comprise:

Cause adjustment of the image capture parameters of the wearable camera based on location information.

130. The method of claim 120, wherein the steps further comprise:

causing an adjustment to an audio capture parameter of the at least one microphone based on the detected speech rate.

131. A hearing aid system for selectively amplifying an audio signal, the hearing aid system comprising:

at least one microphone configured to capture sound from the user's environment; and

at least one processor programmed to:

receiving an audio signal representative of the sound captured by the at least one microphone;

receiving an indication of a time delay associated with processing the audio signal;

storing a plurality of audio samples representing a portion of the audio signal in a buffer; and

processing a first audio sample of the plurality of audio samples to produce a processed first audio sample, wherein processing the first audio sample comprises analyzing a second audio sample of the plurality of audio samples, the second audio sample being represented in the audio signal after the first audio sample and having a length defined by the time delay, wherein an audio quality of the processed first audio sample depends on the length of the second audio sample.

132. The system of claim 131, wherein the time delay is defined by a user through a user interface.

133. The system of claim 132, wherein the user interface is included on a device engaged with the hearing aid system.

134. The system of claim 133, wherein the device is a mobile phone.

135. The system of claim 133, wherein the device is at least one of a desktop computer, a laptop computer, or a tablet computer.

136. The system of claim 131, wherein the set point defining the time delay is stored on a remote storage device.

137. The system according to claim 131, wherein the time delay is determined by the hearing aid system based on feedback from the user regarding previously processed audio signals.

138. The system of claim 131, wherein the longer second audio samples are associated with higher audio quality.

139. A method for selectively amplifying an audio signal, the method comprising:

receiving an audio signal representing sound received by at least one microphone from an environment of a user;

Receiving an indication of a time delay associated with processing the audio signal;

storing a plurality of audio samples representing a portion of the audio signal in a buffer; and

processing a first audio sample of the plurality of audio samples to produce a processed first audio sample, wherein processing the first audio sample comprises analyzing a second audio sample, the second audio sample being represented in the audio signal after the first audio sample and having a length defined by the time delay, wherein an audio quality of the processed first audio sample depends on the length of the second audio sample.

140. The method of claim 139, wherein the time delay is defined by a user through a user interface.

141. The method according to claim 140, wherein the user interface is included on a device engaged with the hearing aid system.

142. The method of claim 141, wherein the device is a mobile phone.

143. The method of claim 139, wherein the set point defining the time delay is stored on a remote storage device.

144. The method according to claim 139, wherein the time delay is determined by the hearing aid system based on feedback from the user regarding previously processed audio signals.

145. The method of claim 139, wherein longer second audio samples are associated with higher audio quality.

146. A hearing aid system for selectively amplifying an audio signal, the hearing aid system comprising:

at least one microphone configured to capture sound from the user's environment;

at least one processor programmed to:

receiving an audio signal representative of the sound captured by the at least one microphone;

processing the audio signal using a first value of at least one aspect to generate a first processed audio signal;

processing the audio signal using a second value of the at least one aspect to generate a second processed audio signal, the second value being different from the first value;

comparing the first processed audio signal and the second processed audio signal to select the first processed audio signal or the second processed audio signal based on a tradeoff between the at least one aspect and audio quality of the first processed audio signal and the second processed audio signal; and

transmitting the selected processed audio signal to an auditory interface device of the user.

147. The system of claim 146, wherein the processor is further programmed to compare the first processed audio signal and the second processed audio signal at predetermined time intervals.

148. The system of claim 146, wherein the at least one aspect includes a time delay for processing the audio signal.

149. The system claimed in claim 146 and wherein said at least one aspect includes the number of microphones used to capture said audio signals.

150. The system of claim 146, wherein the hearing aid system further comprises a wearable camera configured to capture a plurality of images from the user's environment, and the at least one aspect includes using the camera.

151. The system of claim 146, wherein comparing the first processed audio signal and the second processed audio signal includes determining a difference in energy level between the first processed audio signal and the second processed audio signal, and wherein selecting the first processed audio signal or the second processed audio signal includes determining that the difference in energy level is below a predetermined threshold and that an energy level of the selected processed audio signal is below an energy level of an unselected processed audio signal.

152. The system according to claim 146, wherein the at least one processor is further programmed to transmit the unselected processed audio signals to the auditory interface device of the user.

153. The system of claim 152, wherein the unselected processed audio signal is associated with a shorter time delay, and wherein at least one processor is further programmed to determine whether to transmit the unselected processed audio signal based on the detected speech of the user in the audio signal.

154. The system of claim 146, wherein the at least one aspect includes a time delay, and wherein the first processed audio signal or the second processed audio signal is processed to minimize the time delay.

155. A method for selectively amplifying an audio signal, the method comprising:

receiving an audio signal representing sound received by at least one microphone from an environment of a user;

processing the audio signal using a first value of at least one aspect to generate a first processed audio signal;

processing the audio signal using a second value of the at least one aspect to generate a second processed audio signal, the second value being different from the first value;

Comparing the first processed audio signal and the second processed audio signal to select the first processed audio signal or the second processed audio signal based on a tradeoff between the at least one aspect and audio quality of the first processed audio signal and the second processed audio signal; and

transmitting the selected processed audio signal to an auditory interface device of the user.

156. The method of claim 155, wherein the method further comprises comparing the first processed audio signal and the second processed audio signal at predetermined time intervals.

157. The method of claim 155, wherein the at least one aspect comprises a time delay for processing the audio signal.

158. The method of claim 155, wherein the at least one aspect includes a number of microphones used to capture the audio signal.

159. The method of claim 155, wherein the method further comprises receiving at least one image captured from the environment of a user from a wearable camera, and wherein the at least one aspect includes whether the at least one image was processed.

160. The method of claim 155, wherein comparing the first processed audio signal and the second processed audio signal comprises determining a difference in energy level between the first processed audio signal and the second processed audio signal, and wherein selecting the first processed audio signal or the second processed audio signal comprises determining that the difference in energy level is below a predetermined threshold and that the energy level of the selected processed audio signal is below the energy level of the unselected processed audio signal.

161. The method of claim 155, wherein the at least one aspect includes a time delay, and wherein the first processed audio signal or the second processed audio signal is processed to minimize the time delay.

162. A hearing aid system for selectively replacing an audio signal, the hearing aid system comprising:

a wearable camera configured to capture a plurality of images from an environment of a user;

at least one microphone configured to capture sound from the environment of the user; and

at least one processor programmed to:

receiving the plurality of images captured by the camera;

Receiving a plurality of audio signals representing the sound captured by the at least one microphone;

identifying, from the plurality of audio signals, an audio signal associated with a sound emitting object in the environment of the user based on an analysis of the plurality of images or the plurality of audio signals;

predicting, based on the plurality of audio signals, a sound to be received at the user's ear from the environment of the user;

generating a cancellation audio signal configured to cancel at least the predicted sound at the ear of the user;

generating a selectively adjusted audio signal based on the identified audio signal; and

transmitting the cancellation audio signal and the selectively adjusted audio signal to a hearing aid interface device configured to provide sound to the ear of the user.

163. The hearing aid system of claim 162 wherein the at least one processor is further programmed to:

determining a time delay between when the plurality of audio signals are received and when the predicted sound is to be received at the user's ear; and

transmitting the cancellation audio signal at a time based on the time delay.

164. The hearing aid system of claim 163 wherein the time delay is determined based at least in part on a speed of sound propagation in air.

165. The hearing aid system of claim 163 wherein the time delay is determined based at least in part on a positioning of the sound emitting object relative to the at least one microphone and the hearing aid interface device.

166. The hearing aid system of claim 163 wherein the localization of the sound emitting object is determined based on the plurality of images.

167. The hearing aid system of claim 163 wherein the time delay is determined based at least in part on input from a user.

168. The hearing aid system of claim 167 wherein the input is received through a user interface of an external device.

169. The hearing aid system of claim 162 wherein the selective adjustment comprises amplifying the identified audio signal relative to additional audio signals of the plurality of audio signals.

170. The hearing aid system of claim 162 wherein the sound emitting object is an individual.

171. The hearing aid system of claim 162 wherein the cancellation audio signal is further configured to cancel at least one additional sound from the environment of the user.

172. A method for selectively replacing an audio signal, the method comprising:

receiving a plurality of images captured by a wearable camera from an environment of a user;

receiving a plurality of audio signals representing sound captured by at least one microphone from the environment of the user;

identifying, from the plurality of audio signals, an audio signal associated with a sound emitting object in the environment of the user based on an analysis of the plurality of images or the plurality of audio signals;

predicting, based on the plurality of audio signals, a sound to be received from the environment of the user at an ear of the user;

generating a cancellation audio signal configured to cancel at least the predicted sound at the ear of the user;

generating a selectively adjusted audio signal based on the identified audio signal; and

transmitting the cancellation audio signal and the selectively adjusted audio signal to a hearing aid interface device configured to provide sound to the ear of the user.

173. The method of claim 172, wherein the method further comprises:

determining a time delay between when the plurality of audio signals are received and when the predicted sound is to be received at the user's ear; and

transmitting the cancellation audio signal at a time based on the time delay.

174. The method of claim 173, wherein the time delay is determined based at least in part on a speed of sound traveling in air.

175. The method of claim 173, wherein the time delay is determined based at least in part on a positioning of the sound emitting object relative to the at least one microphone and the hearing aid interface device.

176. The method of claim 175, wherein the location of the sound emitting object is determined based on the plurality of images.

177. The method of claim 173, wherein the time delay is determined based at least in part on input from a user.

178. The method of claim 177, wherein the input is received through a user interface of an external device.

179. The method of claim 172, wherein the selective adjustment includes amplifying the identified audio signal relative to additional audio signals of the plurality of audio signals.

180. The method of claim 172, wherein the sound emitting object is an individual.

181. The method of claim 172, wherein the cancellation audio signal is further configured to cancel at least one additional sound from the environment of the user.

182. A hearing aid system for selectively amplifying sound, the hearing aid system comprising:

a wearable camera configured to capture a plurality of images from an environment of a user;

at least one microphone configured to capture sound from the environment of the user; and

at least one processor programmed to:

receiving the plurality of images captured by the camera;

receiving an audio signal representing sound captured by the at least one microphone;

determining a location of a sound emitting object based on an analysis of at least one of the plurality of images or the audio signal;

generating a stereo representation based on the position of the sound emitting object, the stereo representation comprising a first audio signal and a second audio signal, the first audio signal differing from the second audio signal in at least one aspect to simulate the position of the object relative to the user; and

Causing the stereo representation to be transmitted to a hearing aid interface device configured to provide sound based on the first audio signal to a first ear of the user,

and providing sound based on the second audio signal to a second ear of the user.

183. The hearing aid system of claim 182 wherein sound associated with the sound emitting object is attenuated in the first audio signal relative to the second audio signal.

184. The hearing aid system of claim 183 wherein an attenuation level is determined based on the location of the sound emitting object.

185. The hearing aid system of claim 182 wherein sound associated with the sound emitting object is delayed in the first audio signal relative to the second audio signal.

186. The hearing aid system of claim 185 wherein the sound associated with the sound emitting object is delayed in the first audio signal by a delay duration based on the distance of the sound emitting object from the user.

187. The hearing aid system of claim 185, wherein the at least one microphone comprises a first microphone, the hearing aid system further comprising a second microphone, and wherein the delay duration is based on a difference between an arrival time of the sound associated with the sound emitting object at the first microphone and an arrival time of the sound associated with the sound emitting object at the second microphone.

188. The hearing aid system of claim 182 wherein the sound associated with the sound emitting object is delayed in the first audio signal by a delay duration based on a direction of the sound emitting object relative to the user.

189. The hearing aid system of claim 182 wherein generating the first audio signal and the second audio signal comprises selectively adjusting at least one audio signal associated with the sound emitting object.

190. The hearing aid system of claim 182 wherein the at least one processor is further programmed to identify the representation of the sound emitting object in at least one of the plurality of images by identifying at least one moving object in the environment of the user.

191. The hearing aid system of claim 190 wherein determining the location of the sound emitting object further comprises:

determining a distance of a first moving object to the user;

determining a distance of a second moving object to the user; and

selecting one of the first moving object and the second moving object as the sound emission object based on the determined distance.

192. A method for selectively amplifying sound, the method comprising:

receiving a plurality of images from an environment of a user, the images captured by a wearable camera;

receiving an audio signal representing sound from the environment of the user, the sound captured by at least one microphone;

determining a location of a sound emitting object based on an analysis of at least one of the plurality of images or the audio signal;

generating a stereo representation based on the position of the sound emitting object, the stereo representation comprising a first audio signal and a second audio signal, the first audio signal differing from the second audio signal in at least one aspect to simulate the position of the object relative to the user; and

causing the stereo representation to be transmitted to a hearing aid interface device configured to provide sound based on the first audio signal to a first ear of the user and to provide sound based on the second audio signal to a second ear of the user.

193. The method of claim 192, wherein sound associated with the sound emitting object is attenuated in the first audio signal relative to the second audio signal.

194. The method of claim 193, wherein a degree of attenuation is determined based on the location of the sound emitting object.

195. The method of claim 192, wherein sound associated with the sound emitting object is delayed in the first audio signal relative to the second audio signal.

196. The method of claim 195, wherein the sound associated with the sound-emitting object is delayed in the first audio signal by a delay duration based on a distance of the sound-emitting object from the user.

197. The method of claim 195, wherein the at least one microphone comprises a first microphone, the hearing aid system further comprises a second microphone, and wherein the delay duration is based on a difference between an arrival time of the sound associated with the sound emitting object at the first microphone and an arrival time of the sound associated with the sound emitting object at the second microphone.

198. The method of claim 195, wherein the sound associated with the sound-emitting object is delayed in the first audio signal by a delay duration based on a direction of the sound-emitting object relative to the user.

199. The method of claim 192, further comprising identifying the representation of the sound emitting object in at least one of the plurality of images by identifying at least one moving object in the environment of the user.

200. The method of claim 199, wherein determining the location of the sound emitting object further comprises:

determining a distance of a first moving object to the user;

determining a distance of a second moving object to the user; and

selecting one of the first moving object and the second moving object as the sound emission object based on the determined distance.

201. A non-transitory computer-readable storage medium that, when executed on one or more processors, causes the one or more processors to perform operations comprising:

receiving a plurality of images from an environment of a user, the images captured by a wearable camera;

receiving an audio signal representing sound from the environment of the user, the sound captured by at least one microphone;

identifying a representation of a sound emitting object in at least one of the plurality of images;

determining a location of the sound emitting object based on the analysis of the plurality of images;

Generating a stereo representation based on the position of the sound emitting object, the stereo representation comprising a first audio signal and a second audio signal, the first audio signal differing from the second audio signal in at least one aspect to simulate the position of the object relative to the user; and

causing the stereo representation to be transmitted to a hearing aid interface device configured to provide sound based on the first audio signal to a first ear of the user and to provide sound based on the second audio signal to a second ear of the user.

202. A hearing aid system for selectively amplifying sound, the hearing aid system comprising:

at least one microphone configured to capture sound from an environment of a user; and

at least one processor programmed to:

receiving a plurality of audio signals representing sound captured by the at least one microphone;

identifying a first audio signal of the plurality of audio signals, the first audio signal being associated with an individual;

processing the first audio signal to selectively adjust at least one speech feature of the individual; and

Causing the processed first audio signal to be transmitted to an auditory interface device configured to provide sound to an ear of the user.

203. The hearing aid system of claim 202 wherein the at least one speech characteristic comprises an accent of the individual and the selective adjustment comprises altering the accent.

204. The hearing aid system of claim 202 wherein the at least one voice characteristic includes a cloudiness of the individual, and the selective adjustment includes removal of the cloudiness.

205. The hearing aid system of claim 202 wherein the at least one speech feature comprises a pronunciation of a word and the selective adjustment comprises changing the pronunciation of the word.

206. The hearing aid system of claim 202 wherein the selective adjustment comprises altering the individual's voice to simulate a second individual's voice.

207. The hearing aid system of claim 202 wherein the selective adjustment comprises changing a pitch of the speech of the individual.

208. The hearing aid system of claim 202 wherein the selective adjustment comprises a change in the individual's speech rate.

209. The hearing aid system of claim 208 wherein varying the speech rate of the individual comprises varying the duration of a silence period between words.

210. The hearing aid system of claim 202 wherein the at least one processor is further programmed to:

identifying a voice signature in the first audio signal;

determining an identity of the individual based on the voice signature; and

accessing a memory to determine the at least one speech feature, the at least one speech feature being stored in the memory in association with the individual.

211. The hearing aid system of claim 210 wherein the voice signature comprises at least one of a speech disturbance, a mispronunciation, a speech rate, or an accent.

212. The hearing aid system of claim 202 further comprising:

a wearable camera configured to capture a plurality of images from the environment of the user; and

the at least one processor is further programmed to:

receiving the plurality of images captured by the camera;

identifying a representation of the individual in at least one of the plurality of images;

determining an identity of the individual based on the representation; and

Accessing a memory to determine the at least one voice feature, the at least one voice feature stored in the memory in association with the identity.

213. The hearing aid system of claim 202 wherein processing the first audio signal to selectively adjust at least one voice characteristic of the individual comprises:

identifying words in the first audio signal;

accessing a database to determine near homophones for the words;

determining a difference between the word and the near homophone; and

increasing at least one of a volume or a duration of a phoneme corresponding to the difference.

214. A method for selectively amplifying sound, comprising:

receiving a plurality of audio signals representing sound captured by at least one microphone configured to capture sound from a user's environment;

identifying a first audio signal of the plurality of audio signals, the first audio signal being associated with an individual;

processing the first audio signal to selectively adjust at least one speech feature of the individual; and

causing the processed first audio signal to be transmitted to an auditory interface device configured to provide sound to an ear of the user.

215. The method of claim 214, wherein the at least one speech feature includes an accent of the individual, and the selectively adjusting includes changing the accent.

216. The method of claim 214, wherein the at least one speech feature includes orolingness of the individual, and the selective adjusting includes removing the orolingness.

217. The method of claim 214, wherein the at least one speech feature includes a pronunciation of a word, and the selective adjustment includes changing the pronunciation of the word.

218. The method of claim 214, wherein the selectively adjusting includes altering the individual's voice to simulate a second individual's voice.

219. The method of claim 214, wherein said selectively adjusting includes changing a pitch of the voice of the individual.

220. The hearing aid system of claim 214 wherein the selective adjustment includes changing a rate of speech of the individual.

221. The hearing aid system of claim 220 wherein varying the speech rate of the individual comprises varying the duration of a silence period between words.

222. The method of claim 214, further comprising:

identifying a voice signature in the first audio signal;

determining an identity of the individual based on the voice signature; and

accessing a memory to determine the at least one speech feature, the at least one speech feature being stored in the memory in association with the individual.

223. The method of claim 214, wherein the voice signature includes at least one of a speech disturbance, a pronunciation error, or an accent.

224. The method of claim 214, further comprising:

receiving a plurality of images captured by a wearable camera;

identifying a representation of the individual in at least one image of the plurality of images;

determining an identity of the individual based on the representation; and

accessing a memory to determine the at least one voice feature, the at least one voice feature stored in the memory in association with the identity.

225. A non-transitory computer-readable storage medium that, when executed on one or more processors, causes the one or more processors to perform operations comprising:

receiving a plurality of audio signals representing sound captured by at least one microphone configured to capture sound from an environment of a user;

Identifying a first audio signal of the plurality of audio signals, the first audio signal being associated with an individual;

processing the first audio signal to selectively adjust at least one speech feature of the individual; and

causing the processed first audio signal to be transmitted to an auditory interface device configured to provide sound to an ear of the user.

226. A hearing aid system for selectively adjusting sound, the hearing aid system comprising:

a wearable camera configured to capture a plurality of images from an environment of a user, the wearable camera having an image capture rate;

at least one microphone configured to capture sound from the environment of the user; and

at least one processor programmed to:

receiving the plurality of images captured by the camera;

receiving a plurality of audio signals representing sound captured by the at least one microphone;

obtaining a voiceprint associated with an individual in the environment of the user, the individual identified based on at least one of the plurality of images or the plurality of audio signals;

detecting at least one lip movement associated with the mouth of the individual based on the analysis of the plurality of images;

Identifying a first audio signal of the plurality of audio signals that is associated with the individual's voice based on at least one of the voiceprint or the detected lip movement;

processing the first audio signal; and

causing the selectively adjusted first audio signal to be transmitted to an auditory interface device configured to provide sound to an ear of the user.

227. The hearing aid system of claim 226 wherein obtaining the voiceprint comprises generating the voiceprint based on a previous audio signal associated with the individual's utterance.

228. The hearing aid system of claim 226 wherein obtaining the voiceprint comprises retrieving the voiceprint from a database based on an identification of a speaker in at least one of the plurality of images.

229. The hearing aid system of claim 226 wherein identifying the first audio signal comprises identifying the first audio signal based on a combination of the voiceprint and the detected lip movement.

230. The hearing aid system of claim 226 wherein processing the first audio signal comprises:

amplifying the first audio signal relative to at least one second audio signal of the plurality of audio signals; or

Removing background noise of the first audio signal.

231. The hearing aid system of claim 226 wherein causing processing of the first audio signal comprises:

attenuating at least one second audio signal of the plurality of audio signals relative to the first audio signal; or

Filtering out the at least one second audio signal of the plurality of audio signals.

232. The hearing aid system of claim 226 wherein processing the first audio signal comprises at least one of:

changing a speech rate within the first audio signal; or

One or more pauses are introduced between words or sentences within the first audio signal.

233. The hearing aid system of claim 226 wherein processing the first audio signal comprises transcribing the first audio signal.

234. The hearing aid system of claim 226 wherein identifying the first audio signal includes determining that the individual is known to the user.

235. The hearing aid system of claim 234 wherein determining that the individual is known to the user comprises retrieving information from a database stored in memory, the database associating the voiceprint with the individual.

236. A computer-implemented method for selectively adjusting sound in a hearing aid system, the method comprising:

receiving a plurality of images from an environment of a user, the plurality of images captured by a wearable camera;

receiving a plurality of audio signals representing sound captured by at least one microphone; and

obtaining a voiceprint associated with an individual in the environment of the user, the individual identified based on at least one of the plurality of images or the plurality of audio signals;

detecting at least one lip movement associated with the mouth of the individual based on the analysis of the plurality of images;

identifying a first audio signal of the plurality of audio signals that is associated with the individual's voice based on at least one of the voiceprint or the detected lip movement;

processing the first audio signal; and

causing the selectively adjusted first audio signal to be transmitted to an auditory interface device configured to provide sound to an ear of the user.

237. The method of claim 236, wherein obtaining the voiceprint comprises generating the voiceprint based on a previous audio signal associated with the individual's utterance.

238. The method of claim 236, wherein obtaining the voiceprint comprises retrieving the voiceprint from a database based on recognition of a speaker in at least one of the plurality of images.

239. The method of claim 236, wherein identifying the first audio signal comprises identifying the first audio signal based on a combination of the identified speech and the detected lip movement.

240. The method of claim 236, wherein processing the first audio signal comprises:

amplifying the first audio signal relative to at least one second audio signal of the plurality of audio signals; or

Removing background noise of the first audio signal.

241. The method of claim 236, wherein processing the first audio signal comprises:

attenuating at least one second audio signal of the plurality of audio signals relative to the first audio signal; or

Filtering out the at least one second audio signal of the plurality of audio signals.

242. The method of claim 236, wherein processing the first audio signal comprises:

changing a rate of the recognized speech; or alternatively

One or more pauses are introduced between words or sentences of the recognized speech.

243. The method of claim 236, wherein identifying the first audio signal includes determining that the individual is known to the user.

244. The method of claim 243, wherein determining that the individual is known to the user comprises retrieving information from a database stored in memory, the database associating the voiceprint with the individual.

245. A non-transitory computer readable storage medium storing program instructions for execution by at least one processor to perform:

receiving a plurality of images from an environment of a user, the plurality of images captured by a wearable camera;

receiving a plurality of audio signals representing sound captured by at least one microphone;

obtaining a voiceprint associated with an individual in the environment of the user, the individual identified based on at least one of the plurality of images or the plurality of audio signals;

detecting at least one lip movement associated with the mouth of the individual based on the analysis of the plurality of images;

identifying a first audio signal of the plurality of audio signals that is associated with the individual's voice based on at least one of the voice print or the detected lip movement;

Processing the first audio signal; and

causing the selectively adjusted first audio signal to be transmitted to an auditory interface device configured to provide sound to an ear of the user.

246. The non-transitory computer readable storage medium of claim 245, wherein identifying the first audio signal comprises identifying the first audio signal based on a combination of the identified speech and the detected lip movement.

247. The non-transitory computer readable storage medium of claim 245, wherein processing the first audio signal comprises:

amplifying the first audio signal relative to at least one second audio signal of the plurality of audio signals; or

Removing background noise of the first audio signal.

248. The non-transitory computer readable storage medium of claim 245, wherein processing the first audio signal comprises:

attenuating at least one second audio signal of the plurality of audio signals relative to the first audio signal; or

Filtering out the at least one second audio signal of the plurality of audio signals.

249. The non-transitory computer readable storage medium of claim 245, wherein processing the first audio signal comprises:

Changing a rate of the recognized speech; or

One or more pauses are introduced between words or sentences of the recognized speech.

250. The non-transitory computer readable storage medium of claim 245, wherein identifying the first audio signal comprises:

determining that the individual is known to the user, wherein the determining further comprises retrieving information from a database stored in memory, the database associating the voiceprint with the individual.

Images