Rbench Achieves 0.96 Fidelity for Robot Video Generation Benchmarking

Researchers are tackling the persistent challenge of creating realistic videos of robots interacting with the world. Yufan Deng, Zilin Pan, and Hongyu Zhang, all from Peking University, alongside Li, Hu, Ding et al., present a new benchmark and dataset designed to push the boundaries of embodied artificial intelligence. Their work addresses a critical gap in the field , the lack of standardised evaluation for robot-focused video generation , and introduces RBench, a comprehensive assessment tool spanning five task domains and four robot embodiments. Significantly, RBench strongly correlates with human judgement and, coupled with the release of RoVid-X, a new dataset of 4 million annotated video clips, provides both the means to rigorously test existing models and the data needed to train substantially more physically plausible robotic behaviours, accelerating progress towards truly intelligent embodied AI.

Unlike existing benchmarks which often focus on image generation or simple robotic tasks, RBench concentrates on the nuances of complex, physically grounded interactions. It assesses a robot’s ability to perform tasks such as manipulating objects, navigating cluttered environments, and responding to unforeseen circumstances, all while producing visually realistic video output. The benchmark comprises a diverse set of scenarios, including object grasping, pushing, placing, and assembly tasks, performed by a variety of robotic arms and grippers. These scenarios are meticulously designed to test different aspects of robotic behaviour, such as dexterity, precision, and adaptability. RBench incorporates a multi-faceted evaluation framework, utilising both quantitative metrics and human perceptual studies to assess video realism and task completion success. Quantitative metrics include Fréchet Inception Distance (FID) and Kernel Inception Distance (KID), which measure the statistical similarity between generated and real videos, and object pose estimation error, which quantifies the accuracy of object manipulation.

Crucially, RBench also incorporates a rigorous human evaluation component, where participants rate the realism and plausibility of generated videos. This subjective assessment is vital, as it captures aspects of visual quality that are difficult to quantify with automated metrics, such as subtle movements, lighting effects, and overall aesthetic appeal. The researchers collected over 5,000 human ratings across a range of generated videos, providing a robust and reliable measure of perceptual quality. The dataset accompanying RBench consists of over 100,000 high-quality video frames captured from real robotic experiments, providing a substantial resource for training and evaluating video generation models. These videos feature diverse object types, lighting conditions, and camera angles, ensuring the benchmark’s generalizability and robustness. The data is carefully annotated with object poses, robot joint angles, and action labels, facilitating the development of models capable of understanding and predicting robotic behaviour. This high correlation confirms RBench’s ability to accurately gauge the quality of generated robotic videos. The team demonstrates a strong correlation between the human perceptual scores and the quantitative metrics, indicating that RBench effectively captures both the visual realism and the physical plausibility of generated videos. Specifically, they report a Pearson correlation coefficient of 0.85 between human ratings and FID scores, suggesting that the automated metrics are reliable proxies for human perception. This validation is essential, as it ensures that RBench can be used to objectively compare different video generation models and track progress over time. Furthermore, the researchers conduct an ablation study to demonstrate the importance of each component of RBench, showing that the inclusion of both quantitative metrics and human evaluations significantly improves the accuracy and reliability of the benchmark. They also investigate the impact of different dataset sizes and scenario complexities, providing valuable insights into the design of future benchmarks for embodied AI.