AI Video Generators Now Tested on Understanding How the World Works

Researchers are increasingly focused on whether generative video models truly understand the underlying principles governing the physical world. Mingxin Liu from Shanghai Jiao Tong University and Tencent Youtu Lab, alongside Shuran Ma and Shibei Meng from Beijing Normal University, et al., have introduced RISE-Video, a new benchmark designed to assess a model’s ability to reason about and adhere to implicit world rules during video generation. This work represents a significant step forward because it moves evaluation beyond simply assessing visual appeal and instead probes the cognitive reasoning capabilities of text-to-video models. Comprising 467 annotated samples and a novel multi-dimensional evaluation protocol, RISE-Video provides a rigorous testbed for gauging intelligence across areas such as commonsense reasoning and spatial dynamics, ultimately offering critical insights to guide the development of more realistic and intelligent generative models.

While current models excel at creating visually realistic videos, their ability to understand and accurately simulate implicit world rules remains largely unexplored.

This work addresses this critical gap by shifting the focus from aesthetic quality to deep cognitive reasoning within video synthesis. RISE-Video comprises a meticulously curated dataset of 467 human-annotated video samples, spanning eight distinct reasoning categories including commonsense, spatial dynamics, and specialised subject domains.
The framework introduces a multi-dimensional evaluation protocol utilising four key metrics: Reasoning Alignment, Temporal Consistency, Physical Rationality, and Visual Quality. This holistic approach ensures generated videos adhere not only to visual plausibility but also to the underlying cognitive and physical constraints dictated by input instructions.

To facilitate scalable assessment, an automated pipeline leveraging Large Multimodal Models (LMMs) has been developed to emulate human-centric evaluation, guided by reasoning-aware questions and prompts. Extensive experiments conducted on 11 state-of-the-art text-to-video models reveal pervasive deficiencies in simulating complex scenarios governed by implicit constraints.

These findings offer crucial insights for advancing the development of future generative models capable of more accurately simulating the world. Validation confirms a high degree of alignment between the LMM-based evaluation pipeline and human judgements, suggesting its potential as a reliable and cost-effective alternative to large-scale human assessment.

The benchmark is organised into eight reasoning dimensions encompassing experiential, commonsense, temporal, societal, perceptual, spatial, subject-specific, and logical reasoning. This taxonomy provides comprehensive coverage of the reasoning landscape in video synthesis, ranging from low-level perceptual cues to high-level abstract inferences. The research demonstrates that current systems struggle with fundamental reasoning tasks, highlighting a clear need for improved rule-aware evaluation and model development.

RISE-Video dataset construction and reasoning category definitions

A meticulously human-annotated dataset of 467 samples underpins this work, designed to rigorously evaluate reasoning abilities in Text-Image-to-Video (TI2V) synthesis models. The dataset, termed RISE-Video, is partitioned into eight distinct categories of reasoning knowledge, each targeting a specific aspect of understanding and generating videos with structured constraints.

These categories encompass Commonsense Knowledge, Subject Knowledge, Perceptual Knowledge, Societal Knowledge, Logical Capability, Experiential Knowledge, and Spatial Knowledge, providing a comprehensive testbed for model intelligence. Within Commonsense Knowledge, the research assesses understanding of everyday physics, biological responses, and health practices, evaluating models on aspects like footprint formation, skin reactions to mosquito bites, and the development of dental decay.

Subject Knowledge is evaluated across Physics, Chemistry, Geography, and Sports, probing understanding of principles ranging from electricity and chemical reactions to river formations and soccer shooting techniques. Perceptual Knowledge is assessed through manipulation of size, colour, count, position, and occlusion, testing the robustness of visual grounding in generated videos.

Societal Knowledge evaluation focuses on emotion recognition from facial expressions, adherence to social rules like proper waste disposal, and reflection of cultural customs such as dietary traditions. Logical Capability is challenged through game actions, puzzle solving, and geometric reasoning, demanding structured, constraint-based inference.

Experiential Knowledge is probed by assessing the ability to infer intentions from cues, identify individuals, understand procedural sequences, and apply contextual knowledge. Finally, Spatial Knowledge is evaluated through viewpoint transformations, object arrangement, and structural inference, mirroring the importance of three-dimensional understanding in video generation.

To facilitate scalable evaluation, an automated pipeline leveraging Large Multimodal Models (LMMs) was implemented to emulate human-centric assessment, utilising four metrics: Reasoning Alignment, Temporal Consistency, Rationality, and Visual Quality. This framework allows for extensive experimentation on 11 state-of-the-art TI2V models, revealing pervasive deficiencies in simulating complex scenarios under implicit constraints and offering critical insights for advancing world-simulating generative models.

Reasoning performance across diverse video generation scenarios

The RISE-Video benchmark comprises 467 meticulously human-annotated samples spanning eight reasoning categories. These categories encompass diverse scenarios, providing a structured testbed for evaluating model intelligence across dimensions such as commonsense and spatial dynamics. The framework introduces a multi-dimensional evaluation protocol consisting of Reasoning Alignment, Temporal Consistency, Rationality, and Visual Quality.

This approach ensures generated videos adhere to cognitive and physical constraints mandated by input instructions. To facilitate scalable evaluation, an automated pipeline leveraging Large Multimodal Models was developed to emulate human-centric assessment. Experiments conducted on 11 state-of-the-art Text-Image-to-Video models revealed pervasive deficiencies in simulating complex scenarios under implicit constraints.

Logical Capability accounted for 83 samples within the benchmark, while Commonsense Knowledge comprised 50 samples, demonstrating a focus on core reasoning abilities. Spatial Knowledge was represented by 33 samples, and Societal Knowledge by 78, indicating broad coverage of reasoning types. The data distribution further detailed Experiential Knowledge with 23 samples, Perceptual Knowledge with one sample, and Temporal Knowledge with three samples.

Puzzle Solving comprised 17 samples, and Geometric Reasoning accounted for 14, highlighting the inclusion of more complex cognitive tasks. Analysis of video length revealed a distribution of Medium length videos at 19 samples, Short videos at 15 samples, and Long videos at 14 samples. The proposed evaluation pipeline exhibits a high degree of alignment with human judgments, suggesting LMM-based evaluation can serve as a reliable and cost-effective alternative to large-scale human assessment.

Reasoning deficits limit complex scenario generation in text-to-video models

RISE-Video, a new benchmark, systematically evaluates the reasoning capabilities of text-to-video generation models, moving beyond simple visual fidelity assessments. This benchmark comprises 467 meticulously annotated video samples across eight categories designed to test diverse reasoning skills, including commonsense understanding, spatial awareness, and specialised knowledge.

Evaluation employs four metrics, Reasoning Alignment, Temporal Consistency, Rationality, and Visual Quality, to provide a holistic assessment of generated videos. A key component of this work is an automated evaluation pipeline utilising large multimodal models to mimic human judgement, enabling scalable and detailed analysis.

Extensive testing of eleven state-of-the-art text-to-video models revealed consistent weaknesses in simulating complex scenarios governed by implicit rules, despite generally strong visual quality. The authors acknowledge a potential bias in the automated evaluation pipeline related to tolerance rates, which may overestimate the quality of near-perfect outputs and hinder accurate differentiation between high-quality and flawed videos.

These findings highlight a significant gap between achieving visually realistic video and ensuring consistency with underlying world rules in current generative models. The development of RISE-Video facilitates more rigorous evaluation of text-to-video systems and is intended to encourage future research focused on designing and training models that prioritise reasoning abilities alongside visual fidelity. Further research may explore expanding the benchmark with more complex reasoning scenarios and refining the automated evaluation pipeline to address identified biases.