Robots Gain Near-Real-Time Planning with New, Streamlined ‘world model’ Technology

Researchers are tackling a key challenge in robotics: creating world models capable of efficient autonomous planning. Shicheng Yin, Kaixuan Yin, and Weixing Chen, all from Sun Yat-sen University, alongside Yang Liu, Guanbin Li, and Liang Lin, present DDP-WM, a novel approach to disentangled dynamics prediction. This work is significant because it addresses the computational burden of current dense Transformer-based models, hindering their real-time application. By decomposing latent state evolution into primary dynamics and background updates, and employing efficient historical processing, DDP-WM achieves substantial speedups, approximately nine times faster on the Push-T task, and improved success rates, paving the way for more practical and high-fidelity world models for robotic systems.

Decomposing Scene Dynamics for Efficient Robotic Planning enables robust and adaptable behavior

Scientists have developed a new world model, DDP-WM, that significantly boosts efficiency in robotic planning and control systems. Addressing a critical bottleneck in existing dense Transformer-based models, this research introduces Disentangled Dynamics Prediction (DDP) to optimise computational resources.

The core hypothesis underpinning DDP-WM is that changes within observed scenes are not uniform; latent state evolution can be separated into sparse primary dynamics, driven by physical interactions, and secondary, context-driven background updates. This decomposition is realised through an architecture integrating efficient historical processing with dynamic localisation, effectively isolating the primary dynamics.

DDP-WM employs a cross-attention mechanism for background updates, intelligently allocating resources and creating a smoother optimisation landscape for robotic planners. Extensive experiments across diverse tasks, including navigation, precise tabletop manipulation, and complex interactions involving deformable or multi-body systems, demonstrate substantial improvements in both speed and performance.

Notably, on the challenging Push-T task, DDP-WM achieves an approximate 9× inference speedup and elevates the Model Predictive Control (MPC) success rate from 90% to 98% when compared to current state-of-the-art dense models. Analysis of internal feature evolution within existing models, visualised using Principal Component Analysis (PCA), revealed significant computational redundancy in processing static background regions.

This confirmed that a large proportion of processing power was being wasted on areas with minimal change. Further investigation of feature differences between consecutive frames highlighted the inherent sparsity of physical dynamics, with only a small fraction of features undergoing significant alteration.

Based on these insights, the researchers designed DDP-WM to allocate computational effort in proportion to the actual dynamics occurring within a scene. The framework’s key innovation, the Low-Rank Correction Module (LRM), utilises a unidirectional, causal cross-attention mechanism to efficiently capture background dynamics at a minimal computational cost, ensuring feature-space consistency.

This work establishes a promising pathway for developing efficient, high-fidelity world models, paving the way for more responsive and capable robotic systems. Code for DDP-WM will be made available at https://github.com/HCPLabSYSU/DDP-WM.

Disentangling primary dynamics and background updates with localised cross-attention offers improved spatiotemporal representation learning

A dynamic localization network underpins the DDP-WM architecture, identifying regions where primary dynamics occur and concentrating computational resources accordingly. This network isolates action-driven changes, enabling a powerful primary predictor to model these critical interactions. Simultaneously, an efficient Low-Rank Correction Module (LRM) manages context-driven background updates induced by the primary dynamics at a reduced computational cost.

This module employs a unidirectional, causal cross-attention mechanism to capture background dynamics while maintaining feature-space consistency. The study addresses the computational bottleneck of dense Transformer-based world models by introducing the Disentangled Dynamics Prediction (DDP) paradigm.

This posits that scene dynamics can be separated into sparse “primary dynamics” and broader “context-driven background updates”. DDP-WM instantiates this paradigm, leveraging the LRM to optimise resource allocation and create a smoother optimisation landscape for planners. The framework operates within a Partially Observable Markov Decision Process (POMDP) and utilises a pre-trained observation model, gφ, to map high-dimensional observations into a latent space.

Experiments were conducted using the challenging Push-T benchmark, where DDP-WM improved the success rate from 90% to 98% and achieved an approximately 9times inference speedup compared to state-of-the-art dense models. This closed-loop success is attributed to the tractable optimisation landscape provided by the method for the planner.

The research team will release code, models, and supplementary materials to facilitate reproducibility of the results. Figure 2 summarises the comprehensive performance gains across key benchmarks, demonstrating the method’s efficiency and accuracy.

Dynamic World Modelling enables accelerated robotic control through sparse feature processing and efficient prediction

On the challenging Push-T task, DDP-WM achieved an approximately 9× inference speedup compared to state-of-the-art dense models. Furthermore, the success rate improved from 90% to 98% using Model Predictive Control (MPC) with DDP-WM. Analysis of a dense model revealed that background regions remain largely static, indicating computational redundancy in existing architectures.

Principal Component Analysis (PCA) of internal feature evolution showed minimal feature change in background regions after processing through multiple layers of self-attention. Feature difference visualization confirmed the inherent sparsity of physical dynamics, with significant changes occurring in only a small portion of the image features.

DDP-WM’s framework identifies sparse regions of primary dynamics using a dynamic localization network, focusing computational resources accordingly. The Low-Rank Correction Module (LRM) handles context-driven background updates with low computational cost, optimising resource allocation. Experiments across diverse tasks, including navigation, tabletop manipulation, and deformable object interaction, demonstrate significant efficiency and performance gains.

Figure 2 presents an overview of the method’s performance on key benchmarks, plotting success rates directly and normalising Chamfer Distance (CD) values. The research introduces the Disentangled Dynamics Prediction (DDP) paradigm, decoupling scene dynamics into sparse primary dynamics and context-driven background updates. DDP-WM’s key innovation is the LRM, leveraging unidirectional causal cross-attention for efficient background updates.

Disentangled dynamics prediction enhances robotic planning efficiency and success rates by enabling more accurate and adaptable strategies

Researchers have developed DDP-WM, a novel world model designed to improve the efficiency of autonomous robotic planning. This model centres on Disentangled Dynamics Prediction, a principle which separates latent state evolution into sparse primary dynamics driven by physical interactions and secondary, context-driven background updates.

By isolating these primary dynamics through efficient historical processing and dynamic localisation, DDP-WM optimises resource allocation and creates a smoother optimisation landscape for planners. Extensive testing across navigation, tabletop manipulation, and complex interactions demonstrated significant improvements in both speed and performance.

Specifically, on the challenging Push-T task, DDP-WM achieved a nine-fold increase in inference speed and raised the success rate from 90% to 98% when compared to existing dense models. The authors attribute this success to a Low-Rank Correction Module which maintains feature-space consistency, addressing the typically uneven optimisation landscapes associated with sparse prediction approaches.

The authors acknowledge that sparse approaches can introduce challenges in creating smooth optimisation landscapes, and their LRM is designed to mitigate this. They highlight that the synergy between sparse prediction and landscape-smoothing correction is key to DDP-WM’s performance. Future research could explore the application of this framework to even more complex scenarios and investigate further refinements to the LRM to enhance its effectiveness and generalisability, establishing a promising path for developing efficient, high-fidelity world models.

.