New Dataset of 35k Image-Text Pairs Advances Multimodal Safety Evaluation

The increasing sophistication of multimodal large language models presents growing challenges for ensuring their safe operation in real-world scenarios, yet current methods for building datasets to test this safety struggle to keep pace with this complexity. Jingen Qu, Lijun Li, and Bo Zhang, along with Yichen Yan from Zhejiang University and Jing Shao from the Shanghai Artificial Intelligence Laboratory, address this critical gap by introducing a novel approach to dataset construction that begins with images and automatically generates paired text and guidance responses. This work results in a substantial dataset of 35,000 image-text pairs, and importantly, proposes a standardized evaluation metric for assessing safety capabilities by fine-tuning a safety judge and testing it across multiple datasets. Through extensive experimentation, the researchers demonstrate the scalability and effectiveness of this image-oriented pipeline, offering a significant advancement in how we build and evaluate the safety of increasingly powerful artificial intelligence systems.

This research introduces a novel image-oriented method for constructing datasets specifically designed for real-world multimodal safety scenarios, beginning with images and culminating in the creation of paired text and guidance responses. Employing this method, researchers automatically generate a real-world multimodal safety scenario dataset comprising 35,000 image-text pairs with corresponding guidance responses.

MLLM Safety Evaluation with Real-World Scenarios

This research details the performance of various Multimodal Large Language Models (MLLMs) on a dataset called RMS (Real-world Malicious Scenario), designed to evaluate MLLM safety in realistic situations where users might attempt to elicit harmful responses, focusing on scenarios that could lead to physical harm. The dataset presents scenarios with real-world context, an accompanying image, a safe and helpful guidance response, and the actual responses generated by the tested MLLMs, covering self-harm, dangerous acts involving wildlife, and potentially harmful fantasies. The MLLMs demonstrate varied performance, with some models providing safe and helpful guidance, while others generate unhelpful, encouraging, or potentially dangerous replies. Common issues include encouraging harmful behavior, failing to recognize inherent dangers, ignoring crucial context, and providing inappropriate responses.

Llama-3. 2-B-Vision-Instruct explicitly refuses dangerous prompts, a positive safety feature, while Gemini-1. 5-flash and GPT-4o often provide encouraging responses that can be dangerous in these contexts. Qwen2-VL-7B and Qwen2-VL-72B generally provide cautious and informative responses, emphasizing safety, while Phi-3. 5-Vision-Instruct sometimes fails to understand the context. These results demonstrate that MLLMs are not inherently safe and can be manipulated into generating harmful responses, necessitating robust safety mechanisms, including improved danger detection, reinforced safety training, and stronger filters. Contextual understanding is crucial, and evaluation with datasets like RMS is essential for identifying areas for improvement, requiring continued research and development to create both powerful and safe MLLMs.

Real-World Safety Risks in Multimodal Models

Scientists have developed a new approach to constructing datasets for evaluating the safety of multimodal large language models (MLLMs), addressing limitations in existing methods that often rely on synthetic images and preset risks. Researchers introduced an image-oriented method that automatically generates a challenging Real-World Multimodal Safety Scenario (RMS) dataset, comprising 35,000 image-text pairs with corresponding guidance responses. The team focused on identifying safety risks arising from “information complementarity”, where individually safe information combines to create an unsafe outcome, organizing scenarios into 12 distinct categories to adapt to increasingly complicated real-world situations. Furthermore, scientists introduced a standardized evaluation metric, achieved by fine-tuning a model to act as a ‘safety judge’ and assessing its performance across multiple safety datasets, offering a novel way to measure dataset effectiveness. Experiments demonstrate that this image-oriented approach effectively identifies real-world multimodal safety scenarios, with performance improving as the dataset scale increases. The RMS dataset distinguishes itself from existing benchmarks, such as VLSBench and HADES, by utilizing real-world images and an automatic construction process, delivering a significant advancement in MLLM safety evaluation and providing a valuable resource for developing more robust and reliable artificial intelligence systems.

Realistic Safety Dataset and Evaluation Metric

This research introduces a new approach to building and evaluating datasets for multimodal large language models (MLLMs), focusing on real-world safety scenarios. The team developed a method for automatically generating a dataset of 35,000 image-text pairs with corresponding guidance responses, starting with images to ensure a focus on realistic visual contexts. Crucially, they also propose a standardized evaluation metric, achieved by fine-tuning a model to act as a ‘safety judge’ and assessing its performance across existing safety datasets, demonstrating the effectiveness of this image-oriented pipeline in creating a scalable and relevant safety dataset. Results indicate that while MLLMs can identify some unsafe content, they struggle to consistently avoid generating risky responses. The fine-tuned ‘safety judge’ model shows significant improvements in evaluating safety across multiple datasets, highlighting the value of the new evaluation metric. The authors acknowledge that current safety judgments by MLLMs remain limited and superficial in certain categories, suggesting future work could focus on further refining the dataset and evaluation methods to improve the robustness of MLLM safety.