Speech-aware Large Language Models Preserve Understanding Capabilities, Assessed by New C3T Benchmark

The increasing use of speech interfaces demands that artificial intelligence systems not only process spoken words, but also accurately understand their meaning, maintaining the same level of comprehension as with text input. Marek Kubis, Paweł Skórzewski, and Iwona Christop from Adam Mickiewicz University, alongside Mateusz Czyżnikiewicz, Jakub Kubiak, and Łukasz Bondaruk from Samsung R and D Institute Poland, address this critical challenge with a new benchmark called C3T, or Cross-modal Capabilities Conservation Test. This innovative tool rigorously assesses whether large language models retain their understanding abilities when accessed through speech, employing voice cloning technology to generate diverse speech inputs. By quantifying performance across different speakers and comparing text and speech modalities, C3T provides a crucial measure of fairness and robustness, ultimately helping to build more reliable and equitable speech-aware artificial intelligence systems.

Speech-aware large language models represent a growing area of research, and this work introduces a benchmark to quantify the preservation of language understanding capabilities when models are accessed via speech input. The benchmark utilizes textual tasks and a voice cloning text-to-speech model to assess performance across modalities, crucially quantifying the fairness of the model for different categories of speakers and its robustness across both text and speech. This approach allows for a comprehensive evaluation of speech-aware large language models, moving beyond traditional text-based assessments to consider the complexities introduced by speech as an input method.

Speech Understanding Beyond Speech Recognition

This paper introduces a new benchmark designed to evaluate speech-aware Large Language Models (LLMs), arguing that existing benchmarks often focus on foundational capabilities, such as speech recognition, rather than assessing true language understanding when presented via audio. The authors aim to create a benchmark that tests the preservation of language understanding when transitioning from text input to speech input, ensuring the model comprehends meaning, not simply hears words. The core contribution is a benchmark that addresses the limitations of existing evaluations, using automated processes to select suitable language understanding tasks and transform them into audio format. To evaluate performance across different speakers, the authors employ voice cloning, generating audio with a variety of accents and demographics, utilizing datasets like GLOBE. The work addresses a critical gap in evaluation procedures, moving beyond purely textual assessments to quantify performance when models receive speech as input, verifying that language understanding remains consistent regardless of variations in speaker characteristics. The C3T benchmark utilizes tasks originally designed for textual language models, adapting them for speech input through a carefully designed filtering process to ensure plausibility for voice interaction. This approach allows for a direct comparison of performance between textual and speech-aware models, revealing any potential loss of capabilities when transitioning to a speech interface.

The team implemented voice cloning techniques to generate diverse voice samples, creating a dataset that facilitates detailed analysis of model fairness across different demographic groups. Experiments demonstrate the importance of evaluating beyond simple accuracy metrics, as raw accuracy can mask unfair or non-robust behaviour across speakers, identifying instances where a model provides incorrect answers for specific demographic groups. Researchers quantify fairness by aggregating worst-case outcomes, determined by substituting lexical terms indicating gender, dialect, or a person’s name, to model demographic groups through distinct voices, allowing for a quantitative assessment of how consistently a model performs across different voices and demographic characteristics.

Speech Input Reveals Language Model Weaknesses

The researchers developed C3T, a new benchmark designed to assess how well large language models maintain their understanding of language when accessed through speech input. This benchmark moves beyond simply testing a model’s ability to recognise speech or acoustic scenes, instead focusing on whether core language understanding capabilities are preserved when the input is spoken rather than typed, involving automatically selecting and transforming textual tasks, then presenting them to the model via both text and speech, using a voice cloning technique to ensure consistent audio input. Results demonstrate that even high-performing models can exhibit inconsistencies between text and speech modalities, with a notable performance drop observed when comparing the two input types. The team quantified both fairness, assessing performance across different speaker groups, and robustness, measuring consistency between text and speech, revealing a nuanced picture of model behaviour, highlighting that achieving fair performance across speakers does not necessarily guarantee consistent performance across input modalities. The authors acknowledge that the benchmark, like all evaluations, has limitations, and suggest future work could explore expanding the range of tasks and speaker groups to further refine the assessment of speech-aware language models.