Text-Based Football Action Spotting Rivals Video Analysis with LLMs.

Research demonstrates reliable soccer action spotting using only expert commentary and large language models, bypassing computationally intensive video analysis. A system of three LLMs, assessing outcome, excitement and tactics within commentary streams, accurately identifies key match events like goals and substitutions without requiring training data.

The analysis of sporting events traditionally demands substantial computational resources, primarily focused on processing extensive video data to identify key actions. Researchers are now investigating whether the rich textual information contained within expert commentary provides sufficient detail to accurately pinpoint these same events, offering a potentially more efficient and scalable approach. Ritabrata Chakraborty of Manipal University Jaipur, Rajatsubhra Chakraborty from UNC–Charlotte, Avijit Dasgupta of IIIT Hyderabad, and Sandeep Chaurasia, also from Manipal University Jaipur, explore this concept in their work, “Do We Need Large VLMs for Spotting Soccer Actions?”. Their investigation utilises the SoccerNet Echoes dataset, leveraging timestamped commentary and a system of three large language models (LLMs) – sophisticated artificial intelligence systems trained on vast amounts of text – to identify critical match events such as goals, cautions and player substitutions, demonstrating a viable alternative to vision-language models (VLMs) which combine image and text processing.

Researchers present a compelling alternative to conventional video analysis for identifying key moments in football, shifting towards a methodology centred on analysing textual data derived from match commentary. This approach proposes that detailed audio commentary contains sufficient information to accurately pinpoint events such as goals, fouls, and substitutions, offering a computationally efficient solution compared to processing extensive video footage. By focusing on transcribed audio, the study establishes a pathway towards more scalable systems for automated sports analysis and broadcast enhancement.

The research team utilised the SoccerNet-v2 dataset, alongside a newly created resource, SoccerNet-Echoes, which provides timestamped audio commentary. This allowed for a direct comparison between traditional video-based methods and the proposed text-centric approach. They employed Whisper, an automatic speech recognition (ASR) system – a technology that converts spoken language into written text – to transcribe the commentary. This transcription was then processed by a system comprising three large language models (LLMs), each functioning as a specialist judge focusing on distinct aspects of the game: outcome, excitement, and tactics. This modular design enables a nuanced evaluation of the commentary, identifying relevant actions and generating corresponding timestamps with increased accuracy.

Experiments demonstrate the effectiveness of this text-centric approach in detecting critical match events, achieving performance comparable to state-of-the-art methods reliant on complex visual feature extraction, but without the associated computational demands. The system operates in a training-free manner, simplifying implementation and reducing resource requirements, a significant advantage over traditional machine learning approaches that necessitate extensive labelled datasets. This capability facilitates rapid deployment and adaptation to new leagues or broadcasting styles without the need for retraining.

Researchers acknowledge the reliance on accurate speech recognition, noting that errors in transcription directly impact performance, and highlight the crucial role of commentary quality. Sparse or irrelevant descriptions limit the system’s ability to identify key moments. The LLMs require sufficient contextual understanding to correctly interpret the commentary and differentiate between significant and insignificant events, demanding careful prompt engineering and model selection. Prompt engineering refers to the process of designing effective instructions for LLMs to elicit desired responses.

The study contributes to a growing body of work exploring the use of LLMs for video understanding, demonstrating the effectiveness of a text-centric approach and opening up new possibilities for developing efficient and scalable systems for a range of video analysis tasks.

Researchers acknowledge several limitations, including the dependence of system performance on the quality of automatic speech recognition and the completeness of the commentary itself. Errors in transcription or insufficient contextual detail within the commentary can negatively impact accuracy, necessitating further research into robust ASR models and techniques for handling incomplete commentary.

Further research avenues include exploring the generalisability of this approach to other sports and events where detailed commentary is available, potentially expanding the applicability of this methodology beyond football. Investigating the use of different LLM architectures and prompting strategies could also yield performance improvements, optimising the system for specific tasks and datasets. Additionally, quantifying the potential benefits of combining commentary-based action spotting with visual analysis remains a key area for future investigation.

The findings provide a valuable foundation for future work in this area, paving the way for more sophisticated and intelligent video analysis systems. By demonstrating the power of LLMs to extract meaningful information from textual data, this study highlights the potential of natural language processing to revolutionise the field of video analysis.

The study suggests that combining commentary-based action spotting with visual analysis could yield further improvements, and that the approach could be extended to other sports and events.