Stereo Sound Localization Advances With New Video Content Systems

Accurately pinpointing the source of sounds within video content remains a significant challenge in fields ranging from surveillance technology to assistive listening devices. Researchers are now focusing on integrating semantic understanding – the meaning of sounds and visual cues – with spatial audio processing to improve sound event localisation and detection (SELD). Davide Berghi, Philip J. B. Jackson, and colleagues from the Centre for Vision Speech and Signal Processing (CVSSP) at the University of Surrey, U.K., detail their approach in a report titled ‘Spatial and Semantic Embedding Integration for Stereo Sound Event Localization and Detection in Regular Videos’. Their work, submitted to the DCASE2025 Task 3 Challenge, demonstrates improved performance by incorporating pre-trained embeddings, specifically CLAP for audio and OWL-ViT for visual inputs, into a modified Conformer module, alongside autocorrelation-based acoustic features and data augmentation techniques.

The Detection and Classification of Acoustic Scenes and Events (DCASE) challenge has become a key driver of progress in SELD, providing a standardised platform for researchers to evaluate and compare their algorithms. The challenge has evolved from utilising multi-channel audio formats, such as first-order ambisonics, to focusing on more conventional stereo recordings, mirroring the audio found in everyday video content.

Traditional SELD systems rely on spatial cues extracted from multiple microphones to determine sound source locations, but this approach can be limited by data availability and computational complexity. Current research explores the integration of semantic information, leveraging advances in large language models and vision-language models, to improve both the accuracy and robustness of SELD systems. The aim is to enable machines to ‘understand’ the acoustic scene, rather than simply processing raw audio signals, unlocking a new level of contextual awareness.

A key aspect of the current challenge involves estimating not only the direction of a sound source, but also its distance from the recording device, adding another layer of complexity to the task. Furthermore, the inclusion of visual information introduces the possibility of multimodal SELD, where audio and visual cues are combined to achieve more accurate and reliable localisation. This approach acknowledges that real-world scenes are rarely perceived through a single sensory modality, mirroring human perception.

The development of effective SELD systems requires reasoning across spatial, temporal, and semantic dimensions, demanding sophisticated algorithms capable of integrating diverse information sources. Spatial information provides cues about direction and distance, while temporal information tracks the onset, duration, and movement of sound sources. Semantic understanding, facilitated by language models, allows the system to interpret the meaning of sounds and their relationship to the surrounding environment, creating a holistic understanding of the acoustic scene.

Recent work addresses scaling SELD systems beyond limited multichannel datasets by integrating semantic information derived from pre-trained models, effectively augmenting standard architectures. The core innovation lies in the incorporation of embeddings generated by Contrastive Language-Image Pre-training (CLAP) for audio and OWL-ViT, a visual transformer, for visual inputs, providing a rich semantic representation of the scene. CLAP aligns audio and text, enabling the system to understand what sound is occurring, not just that a sound is present. Similarly, OWL-ViT, pre-trained on a vast dataset of images, offers a robust understanding of visual scenes, identifying objects and their spatial relationships.

These embeddings are then fed into a modified Conformer module, termed the Cross-Modal Conformer, designed to facilitate effective multimodal fusion. The Conformer architecture, originally developed for speech recognition, excels at capturing both local and global dependencies within sequential data, making it well-suited for processing the temporal dynamics of sound events.

A key methodological strength resides in the deliberate choice of pre-trained models, enabling the system to generalise beyond specific sounds and environments. Furthermore, the system incorporates autocorrelation-based acoustic features, which provide information about the temporal structure of sound events, improving the accuracy of distance estimation – a critical component of sound event localisation.

To further enhance robustness and generalisation, researchers employed a data augmentation strategy involving left-right channel swapping, artificially increasing the size of the training dataset. The system’s performance was then evaluated on the DCASE 2025 Task 3 challenge, demonstrating substantial improvements over the baseline systems on the development set. Notably, the researchers also employed an ensembling technique, combining the predictions of multiple models to further improve accuracy and robustness.

This work presents a system achieving substantial performance gains in stereo sound event localisation and detection (SELD), as demonstrated by results on the DCASE 2025 challenge. By employing contrastively-aligned pre-trained models – CLAP for audio and OWL-ViT for visual inputs – the system leverages rich semantic embeddings to enhance event understanding, creating a more robust and accurate system.

A key innovation lies in the Cross-Modal Conformer module, a modified Conformer architecture specifically designed for multimodal fusion, effectively combining the semantic embeddings with acoustic features. Furthermore, the incorporation of autocorrelation-based acoustic features demonstrably improves the accuracy of distance estimation, a critical aspect of SELD. The system benefits from pre-training on curated synthetic datasets, augmented by a left-right channel swapping technique, effectively increasing the volume of training data and improving generalisation.

Evaluation on the DCASE 2025 development set confirms the system’s effectiveness, significantly outperforming challenge baselines in both audio-only and audio-visual configurations. Performance gains are further realised through ensembling techniques and a visual post-processing step utilising human keypoint detection, refining spatial accuracy.

Future work will focus on quantifying the individual contributions of each modality, enabling a more nuanced understanding of their respective roles in the overall system performance. Exploration of architectural variants within the Cross-Modal Conformer module promises further refinements and potential improvements in accuracy and efficiency. Investigating the system’s robustness to challenging conditions, such as noisy environments or partial occlusions, remains a priority. Finally, extending the system to operate in real-time presents a valuable avenue for practical application and deployment.