Tandem Achieves 0.73 Hate Speech Detection Accuracy with Temporal-Aware Multimodal Analysis

Detecting hate speech within the growing volume of multimodal online content presents a significant challenge for platform moderation! Researchers Girish A. Koushik, Helen Treharne, and Diptesh Kanojia, from the University of Surrey, address this problem with their new framework, TANDEM, which moves beyond simple flagging to provide detailed, temporally-grounded insights into hateful narratives! Their work transforms hate speech detection from a binary classification task into a structured reasoning problem, employing a tandem reinforcement learning strategy to stabilise analysis across extended video and audio sequences! Significantly, TANDEM outperforms existing methods , achieving a 30% improvement in target identification on the HateMM dataset , and offers a blueprint for more transparent and actionable online safety tools, crucial for effective human-in-the-loop moderation.

Fine-tuning was performed using Low-Rank Adaptation (LoRA) with a rank of 8 and an alpha value of 16, utilising a learning rate of (5 \times 10^{-5}) and a batch size of 2 samples per GPU! Sequence length was capped at 384 tokens, with 4 generations per sample, and the debiased Generalized Proximal Policy Optimization (GRPO) loss function from the trl1 library was used, including a KL penalty of (0.05)! Cross-Modal Context History was limited to one chunk due to computational constraints.

Training involved 200 steps for HateMM and 72 hours for MHC, taking approximately 6 days for HateMM and 72 hours for MHC! Candidate annotations were generated for a subset of 100 videos using Qwen3-Omni-30B-A3B-Thinking and strictly filtered against ground truth, retaining only samples where the model correctly predicted classification labels to serve as structural priors for the SFT phase! Temporal localization was assessed using IoU against all ground-truth intervals, assigning the maximum overlap score, with segments spanning the 30-second chunk boundary treated as truncated intervals! The code, data, and instructions will be released upon paper acceptance, with details such as data splits, hyperparameters, and error bars reported. On the HateMM dataset, utilising Qwen2-Audio-7B (A+V), the researchers achieved a classification accuracy of 0.77 ±0.08 and a macro F1 score of 0.78 ±0.06, demonstrating robust performance in correctly identifying hateful content. Furthermore, the system exhibited strong temporal localisation capabilities, achieving an average IoU (Intersection over Union) of 0.18 ±0.03, indicating accurate pinpointing of hateful segments within videos. The system attained a classification accuracy of 0.67 ±0.07 and a macro F1 score of 0.32 ±0.03 with Qwen2-Audio-7B (A+V). The researchers also explored various training configurations, including ablations of self-constrained context rounding (SCCR) and different reinforcement learning (RL) reward structures. Notably, the combination of supervised fine-tuning (SFT), SCCR, and GSPO consistently yielded the best results across all datasets. The authors acknowledge that distinguishing between offensive and genuinely hateful content remains a challenge, due to ambiguities in labelling and imbalances within the datasets used! They also note a performance gap when compared to fully supervised models trained on domain-specific data, indicating the continued need for labelled data to maximise accuracy. Future research could focus on addressing these labelling issues and exploring methods to further reduce the reliance on extensive supervised learning, potentially through self-supervised techniques or more robust data augmentation strategies. This work establishes a blueprint for transparent and actionable online safety tools, moving beyond simple flagging to provide detailed, interpretable evidence for human moderators.