Neural Ambisonics Encoding Shows Accurate Spatial Audio with Varying Sound Sources

Researchers are tackling the limitations of current spatial audio recording techniques with a novel neural network approach to Ambisonics encoding. Mikko Heikkinen from Nokia, Archontis Politis from Tampere University, Konstantinos Drossos from Nokia, and Tuomas Virtanen from Tampere University, alongside their colleagues, demonstrate a system capable of accurately representing sound from microphone arrays with complex, real-world characteristics. Their work moves beyond traditional methods reliant on simple array geometry, instead utilising directional array transfer functions to achieve more precise spatial audio representations, and significantly outperforms existing digital signal processing and deep learning solutions in challenging reverberant environments. This advancement promises improved immersive audio experiences and more faithful capture of soundscapes for applications ranging from virtual reality to teleconferencing.

Generalising spatial audio encoding via directional transfer functions and cross-attention networks enables robust and efficient sound field reproduction

Scientists have developed a deep neural network capable of encoding microphone array signals into Ambisonics, a spatial audio format, that functions effectively with diverse microphone configurations. This breakthrough addresses a limitation of previous methods which relied heavily on precise array geometry as metadata.
The research team achieved generalisation to arbitrary microphone array configurations with a fixed microphone count, even when microphone locations and frequency-dependent directional characteristics vary. Unlike conventional approaches, the new method utilises directional array transfer functions, enabling accurate characterisation of real-world microphone arrays and overcoming the constraints of relying solely on geometry.

The proposed architecture employs separate encoders to process both audio signals and directional responses, subsequently combining these through a cross-attention mechanism. This innovative approach generates array-independent spatial audio representations, effectively decoupling the encoding process from specific array configurations.

Experiments were conducted on simulated data, encompassing both a complex mobile phone environment with body scattering and a free-field condition, both with varying numbers of sound sources in reverberant environments. Evaluations demonstrate that this new approach consistently outperforms conventional digital signal processing techniques and existing deep neural network solutions in accurately capturing spatial audio.

Furthermore, the study reveals that employing array transfer functions as metadata input, rather than solely relying on geometry, significantly improves accuracy when dealing with realistic, non-ideal microphone arrays. The research establishes a scale-invariant signal-to-distortion ratio (SI-SDR) that is demonstrably higher than existing methods, particularly in complex scattering scenarios.

In free-field conditions, the model achieves the best SI-SDR and performs comparably to existing neural solutions, while also surpassing the static encoder across all Ambisonics signal metrics. Analysis of attention weights provides insights into how the model leverages directivity metadata for precise spatial encoding, opening avenues for improved immersive audio experiences in virtual and augmented reality applications.

Encoding Spatial Audio via Cross-Attention of Signals and Directivity Metadata enables realistic soundstage reproduction

Scientists developed a deep neural network to encode microphone array signals into Ambisonics, achieving generalisation across varying array configurations with a fixed microphone count. Unlike conventional methods relying solely on array geometry, this research harnessed directional array transfer functions to accurately characterise real-world arrays.

The proposed architecture employs separate encoders for both audio and directional responses, integrating them via cross-attention mechanisms to generate spatial audio representations independent of the specific array used. Researchers constructed a system that processes microphone array signals alongside complex-valued directional array transfer functions to predict Ambisonic output, enabling generalisation to previously unseen arrays.

The study pioneered an architecture featuring dedicated signal and directivity metadata encoders, connected through cross-attention, to produce an array-independent latent representation. This latent representation is then decoded into a filter matrix facilitating the transformation from microphone array signals to Ambisonics.

Experiments employed simulated data in two distinct settings: a mobile phone environment exhibiting complex body scattering, and a free-field condition, both incorporating varying numbers of sound sources within reverberant spaces. The team generated array transfer functions to model microphone location, directivity patterns, and device body scattering effects, providing detailed metadata for the network.

Evaluations demonstrated the approach outperforms conventional digital signal processing methods and existing deep neural network solutions, achieving the highest scale-invariant signal-to-distortion-ratio in complex scattering scenarios. Furthermore, the use of array transfer functions as metadata input, rather than geometry alone, improved accuracy on realistic arrays.

In free-field conditions, the model achieved the best SI-SDR, performing comparably to existing neural solutions while consistently surpassing the performance of a static encoder across all Ambisonics signal metrics. Analysis of attention weights revealed how the model utilises directivity metadata for spatial encoding, demonstrating a key innovation in spatial audio processing.

Deep Neural Networks Generalise Spatial Audio Encoding Across Diverse Microphone Arrays effectively

Scientists have developed a deep neural network capable of encoding microphone array signals into Ambisonics, demonstrating generalization across varying microphone array configurations. The research addresses limitations of existing methods by utilizing directional array transfer functions, rather than solely relying on array geometry as metadata.

This approach enables accurate characterization of real-world arrays with fixed microphone counts but differing locations and frequency-dependent directional characteristics. Experiments revealed that the proposed architecture employs separate encoders for audio and directional responses, combining them through cross-attention to generate array-independent spatial audio representations.

The team evaluated the method using simulated data in two distinct settings: a mobile phone environment with complex body scattering and a free-field condition, both incorporating varying numbers of sound sources in reverberant environments. Results demonstrate that this new method outperforms conventional digital signal processing techniques and existing deep neural network solutions in accurately encoding spatial audio.

Measurements confirm that, in complex scattering scenarios, the proposed method achieves the highest scale-invariant signal-to-distortion-ratio (SI-SDR). Specifically, the system consistently surpasses a static encoder across all Ambisonics signal metrics. In free-field conditions, the model attains the best SI-SDR and exhibits comparable performance to existing neural solutions, while still exceeding the performance of the static encoder.

Data shows the model processes microphone array signals and complex-valued directional array transfer functions to predict Ambisonic output, generalizing to unseen arrays. The DNN consists of a signal encoder, a directivity encoder, attention mechanisms, and a decoder. The signal encoder transforms input signals into a feature representation, while the directivity encoder processes array transfer functions.

These are combined using multihead attention, producing an array-independent latent representation that is then decoded into a filter matrix for Ambisonics transformation. Analysis of attention weights provides insights into how the model utilizes directivity metadata for spatial encoding.

Generalising spatial audio rendering via directional transfer function encoding enables portable and personalised 3D sound

Scientists have developed a deep neural network capable of encoding signals from microphone arrays into Ambisonics, a full-sphere surround sound format. This new approach successfully generalises to various microphone array configurations, even with differing microphone locations and frequency-dependent directional characteristics.

Unlike conventional methods relying solely on array geometry, the research utilises directional array transfer functions, allowing for accurate characterisation of real-world arrays. The proposed architecture features separate encoders for both audio and directional responses, which are then combined using cross-attention to create spatial audio representations independent of the specific array used.

Evaluations conducted on simulated data, including complex mobile phone scenarios and free-field conditions with multiple sound sources, demonstrate that this method surpasses both traditional digital signal processing techniques and existing deep learning solutions. Notably, employing array transfer functions as metadata input enhances accuracy when dealing with realistic array setups.

The authors acknowledge a limitation in that perceptual quality has not yet been fully assessed through listening tests, despite the model’s relatively small parameter count and efficient cross-attention mechanism. Future research will concentrate on evaluating perceptual quality using additional computational metrics and subjective listening tests, as well as exploring ways to enhance the model’s learning capacity and further investigate the complex patterns observed within the attention weights. These findings represent a significant advancement in spatial audio processing, offering improved accuracy and generalisation capabilities for Ambisonics encoding.