KR20250069593A - Head-tracking segmentation rendering and head-related transfer function personalization - Google Patents

Abstract

Systems, methods, and computer program products for direction-of-arrival (DOA) based head tracking segmented rendering and head-related transfer function (HRTF) personalization are described. Head tracking audio rendering is split between two devices. A first device receives a main bitstream representation of encoded audio. A second device tracks head pose information. The first device decodes the main bitstream using a main decoder and encodes the decoded bitstream into pre-rendered binaural signals and post-rendering metadata. The second device decodes the pre-rendered binaural signals and the post-renderer metadata from the intermediate bitstream and provides the decoded pre-rendered binaural signals and the post-renderer metadata to a lightweight renderer. The lightweight renderer renders the pre-rendered binaural signals into binaural audio based on the post-renderer metadata, the head pose information, the generic HRTF, and the personalized HRTF.

Description

Head-tracking segmentation rendering and head-related transfer function personalization

Cross-reference to related applications

This application claims the benefit of U.S. Provisional Application No. 63/405,538, filed September 12, 2022, and U.S. Provisional Application No. 63/422,331, filed November 3, 2022, each of which is hereby incorporated by reference in its entirety.

Field of invention

This disclosure relates to audio processing. In particular, this disclosure relates to audio rendering.

Extended Reality (XR) (AR/MR/VR) will increasingly rely on very power-constrained end devices. Augmented Reality (AR) glasses are a prime example. In order to be as lightweight as possible, they cannot be equipped with heavy batteries. Therefore, to enable reasonable operating times, the processors contained within them are limited to very complex numerical computations. Immersive audio, on the other hand, is an essential media component of XR services. Such services can typically support adjusting the presented immersive audio/visual scene in response to 3DoF or 6DoF user (head) movements. Performing high-quality corresponding immersive audio renditions typically requires high numerical complexity.

One potential solution to this problem is to perform the rendering not on the device itself, but on some entity in the mobile/wireless network to which the end device is connected, or on a powerful mobile User Equipment (UE) to which the end device is tethered. In such a case, the end device would only receive audio that has already been binaurally rendered, for example. The 3DoF/6DoF head pose information (head tracking metadata) would have to be transmitted to the rendering entity (network entity/UE). The problem with this approach is the transmission latency between the end device and the network entity/UE, which can be on the order of 100 ms or more. Performing the rendering on the network entity/UE would therefore mean relying on outdated head tracking metadata, and the binauralized audio played by the end rendering device would not match the actual head pose of the head/end device. This latency is referred to as motion-to-sound latency. If this latency is too large, end users will perceive it as a degradation in quality.

For the video component of immersive media rendering, this problem is being addressed by a split render approach, where an approximate part of the video scene is rendered by the network entity/UE and the final video scene manipulation is performed on the end device. For audio, this area is currently less studied.

In various audio services, for example, immersive voice and audio services (IVAS), it is desirable to track the user's head movements during audio rendering and adjust the audio accordingly, so as to provide an immersive audio experience to the user. This requires immersive audio decoding and binaural rendering using a set of head related transfer functions (HRTFs), where the choice of a particular HRTF may depend on the properties of the immersive audio signal and the user's head movements (or head poses). Depending on the immersive audio format, decoding and head-tracked binaural rendering can be computationally complex operations. For example, scene-based audio (e.g., higher-order Ambisonics), channel-based audio (e.g., with a 7.1.4 channel layout) or object-based audio with many objects may each rely on a very large number of constituent audio components, which makes decoding and rendering computationally complex. This means that decoding the bitstream and binaurally rendering it in response to the user’s head movements requires a lot of computational processing. The computational complexity requires power and generates heat, which can be a problem for small, portable devices such as AR glasses.

An object of the present invention is to solve the problems described herein and to provide split rendering in which head pose specific processing can be performed on a second device.

According to some implementations, these and other objects are achieved by a method according to claim 1 or claim 14. According to other implementations, these and other objects are achieved by a user-held device according to claim 22.

Techniques for direction of arrival (DOA) based head tracking segmented rendering and head related transfer function (HRTF) personalization are described. Head tracking audio decoding and binaural rendering can be split between two or more devices. In some examples, a first device can coordinate the segmented decoding and rendering operations with a second device. A first device, e.g., a smartphone, receives a main bitstream representation of encoded audio. The first device decodes and renders the main bitstream into pre-rendered binaural signals using a main decoder and a binaural renderer, and encodes post-rendering metadata including information about the pre-rendered binaural signals and HRTFs associated with the binaural rendering. The first device provides the pre-rendered binaural signals and the post-renderer metadata to the second device as a multiplexed intermediate bitstream. The second device, e.g., headphones, AR glasses, or earbuds, tracks current head pose information. The second device decodes the pre-rendered binaural signals and the post-renderer metadata from the intermediate bitstream, and provides the decoded pre-rendered binaural signals and the post-renderer metadata to the lightweight renderer. The lightweight renderer renders the pre-rendered binaural signals into binaural audio based on the post-renderer metadata, the current head pose information, the generic HRTF, and optionally the personalized HRTF.

The post-render metadata includes at least an indication of the pre-render HRTF used for binaural pre-rendering. The pre-render HRTF is associated with the direction of arrival (DOA) (typically two angles) of the dominant directional component of the audio content with respect to the assumed head pose. The indication of the pre-render HRTF can be the DOA or some kind of index, which allows the user's handheld device to identify the correct HRTF.

In some implementations, the pre-rendered HRTF instructions also include one or more parameters that can be personalized.

The rendering may include applying an HRTF compensation operation configured to compensate for the effect of the pre-rendering HRTF to the binaural audio signal to compute a compensated stereo audio signal, and applying the post-rendering HRTF to the compensated stereo signal to compute a binaural output signal. These steps may be performed in a single operation. The HRTF compensation operation may include an inverse of the pre-rendering HRTF, obtained, for example, by accessing a lookup table. Other methods of achieving compensation of the pre-rendering HRTF are also possible.

Binauralization is described herein as being performed using head-related transfer functions (HRTFs), but can equally well be performed using binaural room impulse responses (BRIRs). It should also be noted that all HRTF processing must be performed for each time frame and each frequency band, often represented as time/frequency tiles.

In some applications, the assumed head pose is also included in the metadata. In another implementation, the user portable device is configured to transmit the current head pose to the main device. Optionally, the second device encodes at least a portion of the head pose information into a head pose bitstream and provides the bitstream to the first device. The first device decodes the head pose bitstream to obtain the head pose information, and subsequently applies the head pose information to the main decoder and the pre-renderer. The main decoder/pre-renderer decodes and pre-renders the main bitstream based on the received head pose information (also referred to as the assumed head pose) and the generic HRTF. In this case, the user portable device can estimate the assumed head pose based on the expected transmission delay.

Information about the assumed head pose is transmitted to the second device together with other information, unless the second device derives the assumed head pose from prior knowledge, which may be based on (head pose) information previously transmitted to the first device or on an assumed head pose previously agreed upon between the two devices.

Additionally, the present disclosure relates to further inventive concepts including techniques for DOA-based head tracking segmentation rendering using suitable prototype signals and optional diffusion signals. A first device decodes a main bitstream using a main decoder, and renders the decoded bitstream as a dominant directional component, referred to as a prototype signal, and zero or more diffusion signals, and post-rendering metadata. The first device then encodes the prototype signal and the zero or more diffusion signals (or parameters indicative thereof) and the post-renderer metadata and provides them as a multiplexed intermediate bitstream to a second device. The second device decodes the prototype signal and the zero or more diffusion signals and the post-renderer metadata from the intermediate bitstream, and provides the decoded prototype signal and the zero or more diffusion signals and the post-renderer metadata to a lightweight renderer. The lightweight renderer renders the prototype signal and zero or more diffuse signals into binaural audio based on post-render metadata, information about head pose, generic HRTFs, and optionally personalized HRTFs.

The techniques described herein can achieve various technical advantages over conventional rendering techniques. Splitting the processing between the two devices prolongs battery life by reducing processing on the wearable device. The wearable device can perform lightweight rendering based on the user's current head pose without having to rely solely on binaural rendition by a powerful rendering device that only has access to delayed/outdated head pose information, thereby reducing motion-to-sound latency due to potential use of outdated head pose information during rendering. For example, the allocation of processing between the first device can be flexible by adjusting the amount of head pose information transmitted from the second device to the first device, from none to full transmission, thereby allowing matching with various wearable devices having different processing capabilities. Other advantages, features and benefits, not explicitly described above, will be apparent from consideration of the detailed description and associated drawings described below.

This Summary is provided to introduce selected concepts in a simplified form and is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter. For example, the term "technology" may refer to system(s), method(s), computer-readable instructions, module(s), algorithms, hardware logic, and/or operation(s), as permitted by the context described above and throughout this document.

Figure 1 is a block diagram of an exemplary system implementing head tracking segmented rendering.
Figure 2 is a flowchart illustrating processing in a first or main device.
Figure 3 is a flowchart illustrating processing on a second or user portable device.
Figure 4 illustrates an exemplary technique for DOA-based segmentation rendering using pre-rendered binaural signals.
Figure 5 illustrates an exemplary technique for HRTF personalization.
Figure 6 illustrates an exemplary technique for DOA-based segmented rendering using prototype signals.

In the following detailed description, reference is made to the accompanying drawings, which form a part of this specification and in which are shown by way of illustration specific exemplary configurations in which the concepts may be practiced. These configurations are described in sufficient detail to enable those skilled in the art to practice the technology disclosed herein, and it is to be understood that other configurations may be utilized and other changes may be made without departing from the spirit or scope of the concepts disclosed. The following detailed description is, therefore, not to be taken in a limiting sense, and the scope of the concepts disclosed is defined only by the appended claims.

The systems and methods disclosed in this application may be implemented in software, firmware, hardware, or a combination thereof. In a hardware implementation, the division of tasks does not necessarily correspond to division into physical units; instead, a single physical component may have multiple functions, and a single task may be performed by multiple cooperating physical components.

The computer hardware may be, for example, a server computer, a client computer, a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a cellular telephone, a smart phone, a web device, a network router, a switch or bridge, or any machine capable of executing instructions (sequential or otherwise) specifying actions to be taken by said computer hardware. Furthermore, the present disclosure relates to any collection of computer hardware that individually or jointly executes instructions to perform any one or more of the concepts discussed herein.

Certain or all of the components contain computer-readable (also called machine-readable) code comprising a set of instructions that, when executed by one or more of the processors, perform at least one of the methods described herein. Can be implemented by one or more processors. Any processor capable of executing a set of instructions (sequential or otherwise) specifying actions to be taken is included. Thus, an example is a typical processing system (e.g., computer hardware) comprising one or more processors. Each processor may include one or more of a CPU, a graphics processing unit, and a programmable DSP unit. The processing system may further include a memory subsystem comprising a hard drive, an SSD, RAM, and/or ROM. A bus subsystem may be included for communication between the components. The software may reside in the memory subsystem and/or within the processor during its execution by the computer system.

One or more of the processors may operate as standalone devices or may be connected to other processor(s), for example, in a network. Such a network may be built on a variety of different network protocols and may be the Internet, a Wide Area Network (WAN), a Local Area Network (LAN), or any combination thereof.

The software may be distributed across a computer-readable medium, which may include computer storage media (or non-transitory media) and communication media (or transitory media). As is well known to those skilled in the art, the term computer storage media includes both volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules or other data. Computer storage media includes, but is not limited to, various forms of physical (non-transitory) storage media, such as ROM, PROM, EPROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by a computer. Additionally, it is well known to those skilled in the art that communication media (transitory) typically embodies computer readable instructions, data structures, program modules or other data in the form of modulated data signals, such as carrier waves or other transport mechanisms, and includes any information delivery media.

The present disclosure assumes that an immersive audio codec, such as an IVAS codec, is being used in some extended reality (XR) application. The main decoding and pre-rendering can be performed by a first device (a user equipment (UE)) of the assumed 5G system or by an edge or other network node. The second device includes a post decoder and a (lightweight) post renderer. Thus, the overall operations can be divided into operations of multiple devices. The first device (the main device) can be a mobile device, such as a laptop, tablet, or smartphone, or a stationary device, such as a workstation or a server. The first device can also be a combination of multiple processing devices. The second device can be a user-held (e.g., wearable) device, such as a pair of augmented reality (AR) glasses.

One fundamental assumption and creative insight applied in the field of discrete rendering is that, for each time-frequency tile, audio consists of a dominant directional component and a diffuse (omnidirectional) component. The directional component is assumed to be a prototype signal arriving from a particular direction of arrival (DOA), while the diffuse component is a decorrelated version of that prototype signal. This concept has proven to be very powerful in spatial audio coding approaches such as DirAC or metadata assisted spatial audio (MASA) coding.

Based on at least these assumptions, various exemplary implementations may include the following steps:

1. The pre-renderer binauralizes the decoded immersive audio using a set of generic HRTFs (or BRIRs) given a head pose (P'), which may be transmitted from a lightweight device equipped with a head tracker or may be a preset value that does not necessarily correspond to any actual head pose of the user but may be a reasonable default, such as a forward-looking head pose. Applying the HRTFs during the binaural pre-rendering operation can be done using HRTFs specifically chosen for each time/frequency tile. The HRTFs are chosen based on the direction of arrival (DOA) of the dominant component of the immersive audio content, relative to the assumed head pose.

2. The first or main device encodes and transmits the binaural audio channels and an indication of the used HRTF and/or DOA angles, and the assumed head pose (P').

3. The post-renderer of the second device aims to adjust the received left binaural signal and right binaural signal based on the current head pose (P) (if the current head pose (P) deviates from P').

4. Given the HRTF or DOA angle and head pose applied by the pre-renderer, the left HRTF compensated signal and the right HRTF compensated signal are computed by the second device by basically inverse HRTF filtering of the left audio channel and the right audio channel and optionally linearly combining them.

5. The HRTF compensated signal is filtered by a second device using the correct HRTF corresponding to the correct head posture.

6. Potential errors in the diffusion component are mitigated by appropriately choosing the weights of the mentioned linear combinations.

The main idea is that while generic HRTFs are used by the pre-renderer, the post-renderer compensates for these generic HRTFs and subsequently applies personalized HRTFs, which can also be applied to HRTF personalization.

Below, exemplary implementations of the novel concepts described in this specification are illustrated with reference to FIGS. 1 to 3.

FIG. 1 is a block diagram of an exemplary system implementing head tracking segmented rendering arranged in accordance with various aspects of the present invention. The exemplary system includes a first device (10) and a second device (20). The first device (10) may also be referred to as a main device, while the second device (20) may also be referred to as a mobile or user-held device.

In Fig. 1, the first or main device (10) comprises a decoder/renderer (11), an optional head pose decoder (12), an encoder (13 and 14), and a multiplexer (15). The decoder/renderer (11), for example an IVAS decoder, receives a main bitstream (b ₁ ) containing encoded immersive audio content (step (S1)), decodes the immersive audio content (step (S2)), and performs binaural rendering of the decoded audio content using HRTFs associated with a direction of arrival (DOA) based on an assumed head pose (P') of the user (step (S3)). The HRTFs used are typically a set of generic HRTFs (for different directions of arrival (DOAs)) ) is taken from. This processing is typically too computationally complex to be performed on a mobile or user-held (lightweight) device. The assumed head pose (P') can be a suitable baseline head pose or can be an actual user head pose received from the user-held device, which can optionally be decoded by a head pose decoder (12) (step (S21)). This decoded user head pose may represent a recent head pose rather than the current head pose of the user. The renderer (11) outputs the binaural signals (L ₁ , R ₁ ) as well as post-rendering metadata (M). The post-rendering metadata (M) includes, for example, an indication of the direction of arrival (DOA) of the dominant directional component of the immersive audio content expressed relative to the assumed head pose (P') or an indication of the used HRTF, expressed as an index of the used HRTF. The post-rendering metadata (M) may also include an indication of the head pose (P') associated with the binaural rendering. The encoders (13, 14) are arranged to encode the binaural signals (L ₁ , R ₁ ) and the post-rendering metadata (M) into encoded signals (b ₁₁ and b ₁₂ ), respectively (step (S4)). The multiplexer (15) is arranged to multiplex or combine the encoded binaural signals (b ₁₁ ) and the encoded metadata (b ₁₂ ) into an intermediate bitstream (b ₂ ) (step (S5)), which intermediate stream (b ₂ ) is transmitted to the second device (20) (step (S6)).

A second device (20), which may be a user portable device (20), includes a demuxer (21), decoders (22 and 23), an encoder (25), a renderer (26), and a head tracker (24). The user portable device (20) receives an intermediate bitstream (b ₂ ) (step (S11)), the demuxer (21) separates the intermediate bitstream (b ₂ ) into encoded signals (b ₂₁ and b ₂₂ ); these encoded signals are received by corresponding decoders (22 and 23). The decoders (22 and 23) respond by decoding the encoded signals (b ₂₁ and b ₂₂ ) (step (S12)) to obtain decoded binaural signals (L ₂ , R ₂ ) and decoded metadata (M'). As mentioned, the metadata (M') contains an indication of the HRTF used, for example indicated by the direction of arrival relative to the index or the head pose looking straight ahead.

A head tracker (124) that may be included in or connected to a user portable device (120) detects a current head pose (P) of the user's head (step (S13)). An encoder (25) may optionally be used to encode the detected head pose (P) as b _P (step (S131)) and transmit the encoded detected head pose (b _P ) to the main device (10).

The metadata (M') may also include an assumed head pose (P') used in the renderer (11). Alternatively, in an implementation where the detected head pose (P) is transmitted to the main device (10), the user's portable device may estimate the assumed head pose based on the expected transmission delay. In principle, the assumed head pose may be assumed to be the head pose detected at a point in time corresponding to the expected transmission delay.

Finally, the renderer (26) receives the decoded binaural audio signal (L ₂ , R ₂ ), the DOA or the used HRTF, the assumed head pose (P') and the current head pose (P), and computes the output binaural signal (L _out , R _out ). This processing includes identifying a post-rendering HRTF corresponding to the detected current head pose (P) (step (S14)), computing a compensated stereo audio signal by applying an HRTF compensation operation configured to compensate the effect of the pre-rendering HRTF to the binaural audio signal (step (S15)), and finally applying the identified post-rendering HRTF (step (S16)). For this processing, which will be discussed below, the renderer (26) typically uses a set of generic HRTFs (for different directions of arrival (DOAs)) ( ) is provided with HRTF data. The renderer (26) also provides a personalized HRTF set ( ) can be provided.

FIG. 2 is a flowchart illustrating processing in a first device (or main device), including steps (S1 to S6) described above and an optional step (S21).

The process includes: receiving a bitstream in step (S1) "receiving a bitstream"; decoding the bitstream by a decoder in step (S2) "decoding" to obtain decoded immersive audio content; optionally, in step (S21) "posture decoding", the process may include receiving and decoding an indication of a current user head pose from a second user portable device, and determining an assumed user head pose based on the current user head pose. In step (S3) "pre-rendering", the process includes binauralizing the immersive audio content by a pre-renderer to generate a pre-rendered binaural signal, wherein the binauralization uses a pre-rendering HRTF from a set of HRTFs and the assumed head pose of the user. In step (S4) "encoding", the process includes encoding the pre-rendered binaural signal and encoding post-rendering metadata, wherein the metadata represents the pre-rendering HRTF. In step (S5) “Combining”, the process comprises combining, in a multiplexer, the encoded binaural audio signal and the encoded post-rendering metadata to form a bitstream comprising a binaural audio representation. In step (S6) “Transmitting”, the process comprises transmitting the bitstream to a second device (or a user portable device).

FIG. 3 is a flowchart illustrating processing in a second device (or a user portable device), including steps (S11 to S16) described above and optional step (S131).

The process comprises, in step (S11) "receiving a bitstream", receiving a bitstream from a first device (or main device) comprising a representation of binaural pre-rendering of immersive audio content. The binaural pre-rendering is obtained based on an assumed head pose (P'). In step (S12) "decoding", the process comprises decoding the bitstream to obtain a binaural audio signal and associated post-rendering metadata. The metadata represents a pre-rendering HRTF used in the binaural pre-rendering, wherein the pre-rendering HRTF is associated with the assumed head pose (P'). In step (S13) "detecting a current pose", the process comprises obtaining user head pose information representing a current head pose (P). In step (131) "encoding a pose", the process may comprise an optional step of encoding the detected head pose (P) using an encoder and transmitting an indication of the current head pose (P) to the main device. As mentioned above, the main device can then use the current pose received from the second user portable device as the assumed pose. Therefore, in this case, the second user portable device can estimate the assumed head pose (P') based on the expected transmission delay (and the previously transmitted current pose). In step (S14) "Post-rendering HRTF identification", the process includes identifying a post-rendering HRTF based on the metadata, the assumed head pose (P') and the current head pose (P). In step (S15) "Compensated Audio Computation", the process includes computing a compensated stereo audio signal by applying an HRTF compensation operation configured to compensate for the effect of the pre-rendering HRTF to the binaural audio signal.

Various exemplary HRTF compensation operations suitable for computing a compensated stereo audio signal are described herein. These operations may include any number of methods that suitably counteract and adjust for the various effects resulting from the pre-rendered HRTF operations. In some examples, the HRTF compensation may include an inverse mathematical operation of the pre-rendered HRTF. In some other examples, the HRTF compensation may be implemented using a lookup table type operation, where - to reduce memory requirements - the lightweight device may rely on a less dense set of HRTFs than those used by the pre-rendered device. Thus, the set of inverse HRTFs available to the lightweight device may include a suitable approximation of the inverses of the HRTFs available in the HRTF set of the pre-rendered device. In another example, the HRTF compensation may be implemented using a numerical approximation method, which may include linear interpolation or non-linear interpolation or a combination thereof. In another example, the HRTF compensation may be implemented using a best fit type of approximation. Combinations of various methods are equally applicable and are considered to be within the scope of the present disclosure.

Returning to Fig. 3, in step (S16) “Apply Post-Rendering HRTF”, this process includes computing a binaural output signal by applying a post-rendering HRTF to the compensated stereo signal. Steps (S15 and S16) can be performed in a single operation.

The processes illustrated by FIGS. 2 and 3 include a collection of blocks representing a series of operations or steps that may be implemented in hardware, software, or a combination thereof. In the context of software, the blocks may represent computer-executable instructions that, when executed by one or more processors, perform the operations noted. Generally, computer-executable instructions may include routines, programs, objects, components, data structures, etc. that perform or implement functions. The order in which the operations are described is not intended to be limited, and any number of the described blocks may be combined in any order to implement the process, separated into additional blocks, and/or operated in parallel.

Approach in pre-renderer (11):

The basic assumption is that, for each time-frequency tile, audio consists of a dominant directional component and a diffuse (omnidirectional) component. The directional component is expressed in azimuth and elevation angles in some room coordinate system. ) arriving from a specific DOA with a prototype signal ( ) is assumed. The diffusion component is the prototype signal ( ) is an inversely correlated version of .

Pre-rendering synthesis is performed here by convolving the directional component (time-frequency tile-wise) with the HRTF corresponding to the DOA and adding the diffusion component. Both components have their own weights. and is added using:

, , class is an inverse correlation coefficient.

Here, is the head pose assumed in the pre-renderer (11). ) are the azimuth and elevation angles of the directional component based on the head pose (P'). If the head pose (P') is expressed in the same indoor coordinate system, at least under certain additional constraint assumptions, And, am.

Approach in post-renderer:

The post-renderer receives the left binaural signal and the right binaural signal based on the current head pose (P) if the current head pose (P) deviates from P'. and ) aims to adjust.

The key insight is that the correct output signal is

,

The point is that it will be.

signal( ) and the anticorrelation signal ( and ) is not available. Instead, and This available signal( and ) will be approximated by a parametric approach. Assuming that the HRTF applied by the pre-renderer is known, the left HRTF compensated signal and the right HRTF compensated signal can be computed. One possibility to obtain these signals is to derive them as a weighted combination of the HRTF compensated left channel signal and the HRTF compensated right channel signal. By this, and is a suitable weighting factor or operator.

The HRTF compensated left channel signal and the HRTF compensated right channel signal are respectively, for the left channel signal and the right channel signal. and am.

The left HRTF compensated signal and the right HRTF compensated signal are then

and

am.

In one simple example, = = 1 (no weight), and therefore

And

am.

Using these signals, the left output signal and right output signal of the post-renderer are obtained as follows:

and .

This approach obtains the correct direction component in the output signal relative to the current head pose, i.e., and Leads to .

However, errors occur in the diffusion component which can be quantified as follows:

and

.

Assuming that HRTFs can be decomposed into delay and gain/shape operations, the associated delay changes in the decorrelated diffusion components may not be perceptually significant, while gain/shape changes may result in timbral deviations or coloration effects. Given a set of relevant HRTFs, , This error can be mitigated by appropriate selection of . In a more general form, , can be a linear or nonlinear operator, such as a (frequency-selective) filter operator or a gain limiter that prevents output samples from exceeding a predetermined numerical range.

weight( , ) is an advantage of adaptively selecting the head pose change when the head pose change is limited to yaw, i.e. only rotation about the z-axis occurs, and therefore the elevation angle of the directional component with respect to the assumed head pose and the current head pose are the same ( ) is illustrated by an example considering the case.

First, consider the case where there is little rotation. Therefore, Is is almost the same as . In this case, the weights are preferably = = 1, which is selected based on the received left binaural signal and right binaural signal ( and ) is printed with almost no modifications. This means that for (sufficiently) small deviations (e.g. less than 20 degrees), the weights = = Setting it to 1 is a good choice.

Secondly, there is a change of 180 degrees, go Consider the case where the weights are almost identical. Now, the weights are preferably = = 0, which is selected based on the received left binaural signal and right binaural signal ( and ) are (virtually) swapped as part of the adjustment process. This results in the following diffusion components for the left and right output channels:

, and

.

Considering that the right ear HRTF can be approximated by the left ear HRTF taken with a 180 degree azimuth offset, and similarly, the left ear HRTF can be approximated by the right ear HRTF taken with a 180 degree azimuth offset, in the above mathematical expression, and It can be seen that can be approximated as 1. This leads to the following approximations of the diffusion components for the left output channel and the right output channel:

, and

.

Therefore, if the difference between the current head posture and the assumed head posture is close to 180 degrees, the weight is = We can conclude that setting it to = 0 is a good choice.

The third case considered is the case where there is a rotation of the axis by 90 degrees, resulting in Is It becomes the same as the help. Now, when constructing the HRTF compensated signal, the available signals are or It can be argued that there is no reason to give priority to either one, since this could potentially result in a solution with asymmetrical behavior between the left and right channels. In the case of a rotation close to 90 degrees, = When = 0.5 is chosen, symmetric behavior is achieved.

This discussion is about the rotation of the decided ( ), i.e., the assumed head posture ( ) and current head posture ( ) in response to the difference between the azimuths of the directional components based on the weight ( , ) leads to a preferred solution that adaptively selects:

.

Note that similar embodiments can be formulated based on the assumed head pose and the roll angle of the current head pose.

Figure 4 illustrates an exemplary technique of DOA-based segmentation rendering using pre-rendered binaural signals. The acoustic wavefront is azimuthally ( , each one, ) is assumed to arrive at the listener's head (30). The pre-renderer is assumed to arrive at the listener's head (30) at an angle ( ) with head posture ( ) is only accessible to the angles ( ) with head posture ( ) is performed using HRTFs corresponding to the left and right ears. One main effect is the interaural time difference (ITD) of the wavefront between the left and right ears. There is also a corresponding interaural level difference (ILD) and spectral difference. The applied HRTF mimics these effects by imposing/imprinting the appropriate ITD, ILD, and spectrum. This figure shows the angle ( ) with actual head posture ( ) is additionally illustrated as an assumed situation in a post-renderer that knows the actual head pose (P) - angle ( instead ) - is shown how to arrive from different DOAs, which in turn leads to different ITDs ( ) results in. In particular, there are other ILDs and spectra corresponding to actual head poses. One of the main concepts of the present disclosure, visualized in Fig. 4, is therefore to first and then compensate ITD by applying From is to change to. Similarly, ILD and spectrum are head posture ( ) and compensate for the corresponding actual head posture ( ) is modified by applying the ILD and spectrum of the HRTF corresponding to the corresponding signal.

The above explanation leads to the following simplified approach (see also Figure 4):

home:

The back-front axis A of the listener (30) defines the x-axis of the right-handed coordinate system. Furthermore, in many relevant cases, the user primarily moved his head around the yaw-axis (z-axis) and most immersive audio content has sound sources close to the horizontal plane. Therefore, the elevation angle of the DOA is relatively close to 0 degrees (e.g., limited to the interval [-20, 20] degrees).

Under these assumptions, and according to the simplified formulation, only the azimuthal component of the DOA produces significant ITD, which further affects the gain and spectral shape.

Limited elevation components (pitch, roll) of DOA affect gain and spectral shape, but not ITD.

HRTF filters can be decomposed into delay and gain/shape operations:

, .

Listener's head posture ( ) is assumed during pre-rendering. The pre-renderer renders according to the azimuth component (α') of the DOA that deviates from the actual azimuth (α) at playback time.

The post-renderer takes into account the listener's head pose at playback time ( ) is rendered according to the azimuth component (α) of the DOA corresponding to the angle.

The time difference (ITD) between the pre-rendered signal and the signal after post-rendering adjustments is calculated as follows:

ΔLR ,

Δ'LR ,

Here is the distance between the two sides, is the speed of sound.

As a result, the post-renderer calculates the ITD of the directional component in a given time-frequency tile. From should be adjusted.

In addition to ITD adjustment, the post-renderer also adjusts the interaural level difference and spectral shape by taking into account the comparison of the actual head pose with the assumed head pose by the pre-renderer.

It is noteworthy that a similar formulation is possible without assuming a limited elevation component of the DOA. Even then, the post-rendering operations can be decomposed into ITD adjustment, inter-bench level difference, and spectral shaping adjustment. In that case, however, the amount of ITD adjustment required depends on the azimuth and elevation angles of the DOA assumed during pre-rendering and valid during post-rendering.

Figure 5 illustrates an exemplary technique of HRTF personalization. DOA angle ( ) is shown to show how the assumed wavefront arriving at the listener's head (30) results in different ITDs depending on the size of the listener's head. Pre-rendering using a generic HRTF is performed using a generic interaural distance ( ) can be assumed to have a listener head size of will result in a corresponding generic ITD and corresponding ILD and spectral shape of the left and right audio signals. The personalized HRTF will be based on (more) accurate listener head dimensions. Therefore, this will result in more accurate and personalized ITD( ) and will result in more accurate corresponding ILD and spectral shapes of the left and right audio signals. The general idea of HRTF personalization is that the post-renderer compensates for the generic HRTF and imposes the effect of the personalized HRTF. The overall concept is very similar to the head pose compensation in the post-renderer described above. Therefore, both concepts are compatible with each other and can be easily combined.

The same description as above applies to the main device (10) having a pre-renderer (11). However, the pre-renderer (11) has a generic HRTF set ( ) while the post-renderer (26) within the user's handheld device (20) renders using a personalized set of HRTFs ( ) to perform the adjustment, the post-renderer knows about the common HRTF used by the pre-renderer.

The post-renderer (26) is based on the current head pose (P) and a personalized HRTF set ( ) received left binaural signal and right binaural signal ( and ) aims to adjust.

The correct output signal is

, would.

signal( ) and the anticorrelation signal ( and ) is not available. Instead, and This available signal( and ) will be approximated parametrically using a pre-renderer. Assuming that the HRTF applied by the pre-renderer is known, the left HRTF compensated signal and the right HRTF compensated signal are computed as linear combinations of the HRTF compensated left channel signal and the HRTF compensated right channel signal:

and

.

Using these signals, the left output signal and right output signal of the post-renderer are obtained as follows:

and .

This approach obtains the correct directional component in the output signal relative to the actual head pose and personalized HRTF, i.e., and Leads to .

However, again, there is an error in the diffusion component which can be quantified as follows:

and

.

Assuming that HRTFs can be decomposed into delay and gain/shape operations, the associated delay changes in the decorrelated diffusion components may not be perceptually significant, while gain/shape changes may result in timbre deviations or coloration effects. Given a set of relevant HRTFs, , This error can be mitigated by appropriate selection of weights. An embodiment that adaptively selects weights in response to deviations in yaw and/or roll between the current head pose and the assumed head pose remains fully applicable.

Specific aspects of the embodiment

The post-renderer receives direction of arrival (DOA) information. This DOA information is the azimuth and elevation angles (DOA angles) of the dominant directional component of the immersive audio content relative to the assumed head pose (P'). ) can be expressed as DOA. It should be noted that DOA is determined per time-frequency tile. The index of the HRTF used is another form of providing DOA information to the post-renderer.

Additionally, the post-renderer assumes the head pose assumed in the pre-renderer ( ) needs to be known. The corresponding information can be transmitted to the post-renderer (i.e. in the metadata). We can also rely on the fact that P' corresponds to the actual head pose at a previous time instant, transmitted from the post-renderer to the pre-renderer. Assuming that the transmission delay from the post-renderer to the pre-renderer is known or can be estimated in advance, this makes the transmission of P' to the post-renderer unnecessary. One way to estimate the transmission delay from the post-renderer to the pre-renderer is to base the round-trip delay measurement from the post-renderer to the pre-renderer and back to the post-renderer, for example, using timestamps.

Parameters , , , are mathematically interconnected. Therefore, it is possible to exploit this interdependence, which allows us to avoid using decorrelators in post-renderers, for example. , It helps to find the appropriate choice of . The advantage of this approach is to avoid post-renderer complexity.

If a decoder is to be used in the post-renderer, a suitable decoder input signal should be one that compensates/removes the directional component.

and therefore becomes:

.

Preferably, pre-rendered binaural channel signals ( , ) is transmitted in the complex-valued quadrature mirror filterbank (CQMF)/frequency domain, which avoids performing forward time-to-CQMF/frequency domain operations in the post-renderer, which is advantageous in terms of complexity and delay.

A notable difference between our approach and prior art techniques is that our approach relies on compensating for the HRTF filter operation of the pre-renderer and applying the HRTF that would ideally be used. In contrast, the alternative technique relies on transforming the binaural output channels using a linear transformation whose coefficients are obtained according to the LMS approach and interpolation.

An example implementation of DOA-based segmentation rendering using prototype signals involves the following steps:

1. A pre-renderer or decoder generates a prototype signal (S). Some exemplary approaches to generating S include:

a. Obtain an Ambisonics W or omni-channel representation from the decoder output using any of the known techniques and use this as S.

b. Obtain a representation of the dominant eigensignal from the decoder output and use it as S.

c. Pre-render the decoded immersive audio using a set of generic HRTFs (or BRIRs), generating S = aL + bR, where L and R are the left and right channels of the pre-rendered bin signal, and a and b are complex or real-only gain factors per time-frequency tile, which can be dynamically computed or statically predetermined, for example, a = 0.5 and b = 0.5.

2. The main device transmits a coded prototype signal (S), and an assumed head pose (P') and/or an assumed DOA angle (or equivalent information) and diffusion parameters.

3. The post-renderer decodes the prototype signal bits and produces S' (S' must be identical to S if the codec used to code S has zero delay and is lossless).

4. The post-renderer aims to generate left binaural signals and right binaural signals based on the current head pose (P) (if the current head pose (P) deviates from P').

4. The post-renderer adjusts the DOA angle transmitted by the main device based on the difference between P and P'. With S', the HRTF in the post-renderer, and the adjusted DOA angle, the post-renderer generates the directional component of the post-rendered binaural signal. The diffusion parameters are used together with the decorrelated S' to fill in the diffuse energy in the post-rendered binaural signal.

Approach in post-renderer:

The post-renderer adjusts the received DOA based on the current head pose (P) if it deviates from P', and then generates a prototype signal (S') and a set of HRTFs that can be personalized or generic ( ) aims to coordinate with the post-renderer. The post-renderer generates the head-tracking binaural signal as follows.

), .

, .

is the diffusion parameter transmitted by the main device. is the directional gain, which can be calculated using DOA and spherical harmonics.

Advantages of this approach:

- This bitrate mode can be achieved by coding only the S channel and sending it to the post-renderer.

- There is no need for HRTF compensation and the error in diffusion compensation can be reduced to 0.

Disadvantages of this approach:

- Pre-rendered binaural audio signals are not immediately available on the user's handheld device, in the potential case where the user's handheld device only outputs the decoded binaural audio signal without any additional processing or post-rendering operations.

Figure 6 illustrates an exemplary technique for DOA-based segmented rendering using prototype signals.

In FIG. 6, the main device (110) includes a decoder/renderer (111), a head pose decoder (112), encoders (113 and 114), and a multiplexer (115). The decoder/renderer (111), for example, an IVAS decoder, receives the main bitstream (b ₁ ) and performs rendering synthesis of a prototype signal (S) having a direction of arrival (DOA) based on an assumed head pose (P') of the user. The assumed head pose (P') may be a suitable baseline head pose or may be an actual user head pose received from a user handheld device, which may optionally be decoded by the head pose decoder (112). This decoded user head pose will represent a recent head pose, not a current head pose of the user. Here, the renderer (111) outputs the prototype signal (S) and metadata (M) including at least the direction of arrival (DOA) of the prototype signal. An encoder (113, 114) encodes a prototype signal (S) and metadata (M), and a multiplexer (115) multiplexes the encoded prototype signal (b ₁₁ ) and the encoded metadata (b ₁₂ ) into one intermediate bitstream (b ₂ ).

The user handheld device (120) includes a demuxer (121), decoders (122, 123), a head tracker (124), an encoder (125), and a post-renderer (126). The demuxer (121) receives the intermediate bitstream and separates it into two encoded signals (b ₂₁ and b ₂₂ ), and the two decoders (122, 123) responsively decode these signals to obtain a decoded prototype signal (S') and decoded metadata (M'), e.g., DOA and (optionally) an assumed head pose (P'), which are used in the renderer (111). The head tracker (124), which may be included in or connected to the user handheld device (120), detects the current head pose (P) of the user's head. The encoder (125) encodes the detected head pose (P) and transmits it to the main device (110). Finally, the post-renderer (126) receives the decoded prototype signal (S'), the DOA, the assumed head pose (P') and the current head pose (P), and computes the output binaural signals (L _out , R _out ). For this processing, which will be discussed below, the post-renderer (126) typically uses a set of generic HRTFs (for different directions of arrival (DOAs)) ) is provided with HRTF data. The renderer (125) also provides a personalized HRTF set ( ) can be provided.

Example use cases that can benefit from split rendering of immersive audio

The techniques described herein can be implemented for a variety of use cases. It is assumed that the basic audio processing/audio signal augmentation and subsequent pre-rendering are performed on some powerful device or network node, while the post-rendering is performed on a lightweight end device such as AR glasses.

Some examples are provided below.

1. AR/MR with audio

Audio Zoom/Magnifier: Similar to a magnifying glass, but for sound. The user can zoom in on sounds of interest.

Overlaying sounds on real world objects: Real world objects/items will be associated with sounds. Useful for but not limited to assistive systems for the visually impaired.

Conversation Enhancement/Smart Ambient Noise Reduction: Helps those with cocktail party issues by raising their active voice above the background noise.

Mood Sound Ambience: Similar to mood lighting, sounds will be associated with real-world environments, items, and personal preferences.

2. Use case characteristics

These use cases will typically rely on audio/visual capture, some scene analysis, and the generation of augmented sound signals. In some scenarios, immersive sound from some network node or far end of the communication may also be overlaid.

Use cases will typically rely on head-tracked audio/visual rendering.

3. Additional non-AR/MR use cases

Immersive voice communications (two-party calls, conferencing) and immersive content streaming using AR glasses as the end device are potential IVAS use cases.

Some of these may rely on head-tracked audio rendering, some may not.

Some use cases may involve one-to-many immersive distribution of head-tracked audio.

Aspects of the system described herein may be implemented in any suitable computer-based sound processing network environment for processing digital or digitized audio files. Parts of the adaptive audio system may include one or more networks comprising any desired number of individual machines, including one or more routers (not shown) that serve to buffer and route data transmitted between the computers. Such networks may be built on a variety of different network protocols, and may be the Internet, a Wide Area Network (WAN), a Local Area Network (LAN), or any combination thereof.

One or more of the components, blocks, processes, or other functional components may be implemented by a computer program that controls the execution of a processor-based computing device of the system. It should also be noted that the various functions disclosed herein, in terms of their behavior, register transfers, logic components, and/or other characteristics, may be described using any number of combinations of hardware, firmware, and/or data and/or instructions embodied in various machine-readable or computer-readable media. The computer-readable media on which such formatted data and/or instructions may be embodied include, but are not limited to, various forms of physical (non-transitory), non-volatile storage media, such as optical, magnetic, or semiconductor storage media.

While one or more implementations have been described by way of example and in terms of specific embodiments, it should be understood that one or more implementations are not limited to the disclosed embodiments. On the contrary, it is intended to cover various modifications and similar arrangements as will be apparent to those skilled in the art. Accordingly, the scope of the appended claims should be interpreted in the broadest manner possible to encompass all such modifications and similar arrangements.

Additional details and embodiments of the present invention can be understood from the following list of enumerated exemplary embodiments (EEE):

EEE1. A method for processing audio,

A step of receiving a main bitstream representation of encoded audio by a first device;

A step of obtaining user head posture information by a second device;

A step of determining downmixed signals including at least one channel and metadata from a main bitstream by a first device;

A step of providing downmixed signals and metadata to a second device by a first device;

A step of rendering downmixed signals based on metadata and user head pose information into output binaural audio by a lightweight renderer of a second device.

A method comprising:

EEE2. A method according to EEE1, wherein the downmixed signals include pre-rendered binaural signals.

EEE3. In EEE2, the steps of determining pre-rendered binaural signals and rendering metadata are:

A step of decoding a main bitstream representation by a main renderer of a first device to generate decoded audio;

A step of binauralizing audio decoded by a pre-renderer of a first device to generate pre-rendered binaural signals and rendering metadata.

, and the pre-renderer is:

General head-related transfer function (HRTF) or binaural room impulse response (BRIR), or

User head posture information

Binauralization is performed using at least one of the user's head posture information.

Head tracker of the second device,

A storage device that stores preset values, or

Assumed Direction of Arrival (DOA) Angle

A method obtained from at least one of:

EEE4. In EEE 3, metadata:

Instructions for the HRTF or BRIR used by the pre-renderer,

The assumed user head pose used by the pre-renderer, or

Assumed DOA angle used by the pre-renderer

A method comprising at least one of:

EEE5. A method according to EEE 4, wherein the step of rendering the pre-rendered binaural signals into output binaural audio comprises the step of adjusting, by a lightweight renderer, left and right channels of the pre-rendered binaural signals based on a current user head pose acquired via a head tracker rather than an assumed user head pose used by the pre-renderer.

EEE6. In any one of EEE2 to EEE5, the step of rendering pre-rendered binaural signals comprises:

A step of inverse HRTF filtering the left and right channels of the pre-rendered binaural signals according to the HRTF used by the pre-renderer or the assumed DOA angle; and

Step of linearly combining the inverse HRTF filtered signals

A method comprising:

EEE7. In EEE6, the method comprises the step of inverse HRTF filtering comprising the step of correcting the HRTF used in the pre-renderer using a current user head pose acquired via a head tracker of the second device.

EEE8. A method according to EEE6 or EEE7, wherein the step of linearly combining the inverse HRTF filtered signals comprises the step of alleviating errors in diffusion components by selecting weights of the linear combination.

EEE9. A method according to any one of claims 2 to 8, comprising the step of applying HRTF personalization, wherein the pre-renderer applies a generic HRTF, and the lightweight renderer compensates for the generic HRTF and subsequently applies the personalized HRTF.

EEE10. A method according to EEE1, wherein the downmixed signals include a prototype signal.

EEE11. A method according to EEE10, wherein the prototype signal comprises a single channel.

EEE12. In EEE10 or EEE11, the steps for computing a prototype signal are:

A step of decoding a main bitstream representation by a main decoder of a first device to generate decoded audio; and

A step of applying gains to decoded audio and adding the decoded audio with the gains applied to the decoded audio.

A method comprising:

EEE13. In any one of EEE10 to EEE12,

Step of calculating prototype signals and DOA angles based on assumed head pose (P') and diffusivity parameters;

A step of transmitting the assumed head pose (P'), DOA angles for the assumed head pose (P'), diffusion parameters and prototype signal to a post-renderer device;

In the post-rendering device, a step of adjusting the DOA angles based on the actual head pose (P);

Step of computing directional components using prototype signal and HRTF set and adjusted DOA angles;

Computing diffusion components using the diffusion parameters and the decorrelated version of the prototype signal; and

Step to generate post-rendered binaural output by adding directional components and diffusion components.

A method comprising:

EEE14. A method according to any one of EEE1 to EEE13, wherein the first device comprises a smartphone, the second device comprises a wearable audio, visual, or AR device, and the main bitstream comprises an immersive audio and video services (IVAS) bitstream.

EEE15. A system, comprising one or more processors configured to perform any one of the operations of EEE1 to EEE14.

EEE16. A computer program product, configured to cause one or more processors to perform any one of the operations of EEE1 to EEE14.

Claims (28)

A method for processing audio on a user's portable processing device, comprising:
A step of receiving from a main device a bitstream comprising a representation of binaural pre-rendering of immersive audio content, wherein the binaural pre-rendering is obtained based on an assumed head pose (P');
A step of decoding said bitstream to obtain a binaural audio signal and associated post-rendering metadata, wherein said metadata is indicative of a pre-rendering HRTF used in said binaural pre-rendering, wherein said pre-rendering HRTF is associated with said assumed head pose (P');
A step of obtaining user head posture information representing the current head posture (P);
A step of identifying a post-rendering HRTF based on the above metadata, the assumed head pose (P') and the current head pose (P);
Computing a compensated stereo audio signal by applying an HRTF compensation operation configured to compensate for the effect of the pre-rendered HRTF to the binaural audio signal; and
A step of calculating a binaural output signal by applying the above post-rendering HRTF to the above compensated stereo signal.
A method comprising: A method according to claim 1, wherein the step of calculating a compensated stereo audio signal and the step of calculating a binaural output signal are performed in one single operation. A method according to claim 1 or 2, wherein the HRTF compensation operation includes an inverse of the pre-rendering HRTF. In the third aspect, the method wherein the inverse of the pre-rendering HRTF is obtained by accessing a look-up in a table containing an approximation to the inverse of the pre-rendering HRTF. In claim 3 or 4, the step of calculating a compensated stereo audio signal comprises:
A method comprising the steps of forming an inverse-filtered left channel by applying an inverse-left HRTF to a left channel of the binaural audio signal, forming an inverse-filtered right channel by applying an inverse-right HRTF to a right channel of the binaural audio signal, and combining the inverse-filtered left channel and the inverse-filtered right channel to form a left channel and a right channel of the compensated stereo audio signal, respectively. A method in accordance with claim 5, wherein the inverse filtered left channel and the inverse filtered right channel are linearly combined, and weights of the linear combination are selected to alleviate errors in diffusion components of the binaural output signal. In the sixth paragraph, the weights are preferably adaptive, responsive to the difference between the assumed head posture (P') and the current head posture (P). A method according to any one of claims 1 to 7, wherein the post-rendering HRTF is personalized for a user of the user's portable processing device. A method according to any one of claims 1 to 8, wherein the post-rendering metadata further comprises an indication of the assumed head pose (P'). A method according to any one of claims 1 to 9, wherein the method further comprises the step of transmitting an indication of the current head pose (P) to the main device and estimating the assumed head pose (P') based on an expected transmission delay. A method according to any one of claims 1 to 10, wherein the user-held processing device comprises a wearable audio, visual or AR device. A method according to any one of claims 1 to 11, wherein the bitstream comprises an immersive audio and video services (IVAS) bitstream. A method according to any one of claims 1 to 12, wherein the metadata comprises a direction of arrival (DOA) associated with a dominant directional component of the immersive audio content, wherein the DOA represents the pre-rendered HRTF. As a method of processing audio,
Step of receiving a bitstream;
A step of decoding the bitstream by a decoder to obtain decoded immersive audio content;
A step of binauralizing said immersive audio content by a pre-renderer to generate a pre-rendered binaural signal, wherein said binauralization uses a pre-rendered HRTF from among a HRTF set and an assumed head pose of the user;
A step of encoding the above pre-rendered binaural signal;
A step of encoding post-rendering metadata, said metadata representing said pre-rendering HRTF;
In a multiplexer, combining the encoded binaural audio signal and the encoded post-rendering metadata to form a bitstream including a binaural audio representation; and
Step of transmitting the above bitstream to a user's portable device
A method comprising: A method in claim 134, wherein the metadata includes a direction of arrival associated with a dominant directional component of the immersive audio content. A method according to claim 14 or 15, wherein the metadata further includes the assumed head pose. In any one of Articles 14 to 16,
A step of receiving an indication of the current user head posture from the user's portable device, and
A step of determining the assumed user head posture based on the current user head posture.
A method further comprising: A method according to any one of claims 14 to 17, wherein the method is performed on a smartphone. A system comprising one or more processors configured to perform the method of any one of claims 1 to 18. A computer program product, configured to cause one or more processors to perform any one of the operations of claims 1 to 18. As a user portable processing device,
A decoder configured to decode a bitstream comprising a representation of binaural pre-rendering of immersive audio content, and to obtain a binaural audio signal and associated post-rendering metadata, wherein the metadata is indicative of a pre-rendering HRTF used in the binaural pre-rendering, wherein the pre-rendering HRTF is associated with the assumed head pose;
A head tracker for obtaining user head pose information representing the current head pose;
Renderer
, and the renderer comprises:
Identifying a post-rendering HRTF based on the above metadata, the assumed head pose (P') and the current head pose (P);
Computing a compensated stereo audio signal by applying a HRTF compensation operation configured to compensate for the effect of the above pre-rendered HRTF to the binaural audio signal;
A device configured to compute a binaural output signal by applying the post-rendering HRTF to the compensated stereo signal. In claim 21, the device comprises an inverse of the pre-rendering HRTF. In claim 22, the device is configured to obtain the inverse of the pre-rendering HRTF by accessing a lookup in a table containing an approximation to the inverse of the pre-rendering HRTF. In claim 22 or 23, the renderer
Applying an inverse left HRTF to the left channel of the binaural audio signal to form an inverse filtered left channel, applying an inverse right HRTF to the right channel of the binaural audio signal to form an inverse filtered right channel, and combining the inverse filtered left channel and the inverse filtered right channel to form left channels and right channels of the compensated stereo audio signal, respectively.
A device configured to compute a stereo audio signal compensated by. A device in accordance with claim 24, wherein the inverse filtered left channel and the inverse filtered right channel are linearly combined, and weights of the linear combination are selected to alleviate errors in diffusion components of the binaural output signal. In the 25th paragraph, the weights are preferably adaptive, responsive to the difference between the assumed head posture (P') and the current head posture (P). A device according to any one of claims 21 to 26, wherein the post-rendering HRTF is personalized for a user of the user portable processing device. A device according to any one of claims 21 to 27, wherein the device is integrated into a wearable audio, visual or AR device.