TriSense: AI Model Integrates Vision, Audio and Speech for Video Understanding.

TriSense, a new large model, enhances video understanding by integrating visual, audio, and speech data. A key component, the Query-Based Connector, dynamically prioritises input modalities, improving performance even with missing data. The model’s development is supported by TriSense-2M, a dataset of over two million curated video samples.

The accurate interpretation of video relies on the brain’s seamless integration of what is seen and heard, a process current artificial intelligence systems often struggle to replicate. Researchers are now developing models capable of more holistic video analysis by simultaneously processing visual, auditory, and speech data. Zinuo Li, Xian Zhang, Yongxin Guo, Mohammed Bennamoun, Farid Boussaid, Girish Dwivedi, Luqi Gong, and Qiuhong Ke detail their work in ‘Watch and Listen: Understanding Audio-Visual-Speech Moments with Multimodal LLM’, presenting TriSense, a large language model designed to improve the temporal understanding of video through the adaptive fusion of these three modalities. The team also introduces TriSense-2M, a dataset of over 2 million samples intended to facilitate broader generalisation in multimodal video analysis.

TriSense: A Multimodal Model for Holistic Video Understanding

TriSense represents an advancement in multimodal video analysis, demonstrating strong performance in understanding video content through the effective integration of visual, audio, and speech information. The model achieves this via a novel ‘Query-Based Connector’ which dynamically adjusts the relative importance of each modality – vision, audio, and speech – based on the specific query. This enables robust performance even with incomplete data streams.

Existing multimodal models often struggle to effectively fuse information from diverse sources, leading to incomplete or inaccurate interpretations of video content. TriSense overcomes these challenges by employing a dynamic weighting mechanism that prioritises the most relevant information for each task, enhancing both accuracy and robustness.

Researchers validated the model’s capabilities using a new dataset, TriSense-2M, comprising over two million curated video samples. This dataset distinguishes itself through its inclusion of long-form videos and diverse combinations of available modalities, facilitating broader generalisation of the model’s understanding.

Experiments across multiple benchmarks confirm TriSense’s effectiveness in tasks such as Audio-Visual Moment Retrieval (AV-MR) and Video and Speech Moment Retrieval (VS-MR), where it accurately locates specific moments within videos corresponding to textual queries. The model consistently outperforms existing approaches in these tasks, demonstrating its superior ability to fuse information from multiple modalities and provide accurate, contextually relevant responses.

Performance evaluations reveal that TriSense frequently matches human accuracy in identifying relevant video segments. For example, the model correctly identifies timestamps corresponding to events like a man jumping accompanied by music, or a person describing a snake, showcasing its ability to process and interpret multimodal data.

A key feature of TriSense is its Query-Based Connector, which dynamically adjusts the weighting of each modality according to the specific query. This adaptive weighting proves crucial, enhancing performance and robustness even when certain modalities are unavailable. Observed variations in modal weights confirm this dynamic adjustment, with vision, speech, or audio taking precedence depending on the query’s focus.

The introduction of TriSense-2M significantly supports the model’s multimodal capabilities, providing extensive training data for robust performance. This dataset, generated through a fine-tuned large language model pipeline, provides long-form videos and diverse combinations of available modalities, facilitating broader generalisation of the model’s understanding.

Researchers validated the model’s performance on several benchmark datasets, demonstrating its superior ability to fuse information from multiple modalities and provide accurate, contextually relevant responses. The model consistently outperforms existing approaches in tasks such as audio-visual event detection and video captioning, showcasing its versatility and adaptability.

The model’s architecture incorporates several key innovations, including a novel attention mechanism that allows it to focus on the most relevant features in each modality. This attention mechanism enables the model to effectively filter out noise and distractions, improving its accuracy and robustness. Furthermore, the model incorporates a sophisticated temporal modeling component that allows it to capture the dynamic relationships between events in a video sequence.

The availability of the TriSense-2M dataset and the model itself will accelerate research and development in the field of multimodal video analysis, enabling researchers to build upon this work and explore new applications. Researchers plan to release the code and data publicly, fostering open science and collaboration.

Future research directions include exploring new architectures for multimodal fusion, developing more robust methods for handling noisy or incomplete data, and extending the model to handle more complex video scenarios. Researchers are also investigating the use of reinforcement learning to train the model to perform more complex tasks, such as video summarization and question answering. The ultimate goal is to create a truly intelligent video understanding system that can seamlessly interact with humans and assist them in a variety of tasks.