Vision Language Models As Closed-Loop Symbolic Planners Improve Robotic Control through Control-Theoretic Insights

Recent advances in artificial intelligence have seen Vision Language Models (VLMs) increasingly applied to complex tasks such as robotic planning, but a crucial question remains unanswered: how can these models be reliably used in real-time, closed-loop systems? Hao Wang, Sathwik Karnik, Bea Lim, and Somil Bansal, from the University of Southern California and Stanford University, address this challenge by investigating the use of VLMs as symbolic planners from a control-theoretic perspective. Their work demonstrates how key parameters, such as the planning horizon and initial conditions, significantly impact performance, offering valuable insights into building more robust and predictable robotic systems. By treating VLMs not simply as ‘black boxes’ but as components within a control framework, the team provides a pathway towards deploying these powerful models in safety-critical applications where consistent, reliable performance is paramount.

This work investigates how VLMs can function as closed-loop symbolic planners for robots, adopting a control-theoretic perspective to address the risks associated with their unpredictable behaviour. 1-mini, Gemini-2. 5-flash, and Llama-4-Maverick-17B, across robotic manipulation tasks of increasing complexity. Results demonstrate that Gemini-2. 5-flash consistently outperforms the other models, particularly on challenging tasks, achieving high Goal Achievement Rate and Task Completion Rate.

GPT-4. 1-mini performs reasonably well on simpler tasks but struggles with increased complexity, while Llama-4-Maverick-17B consistently exhibits the lowest performance. The difficulty of the task significantly impacts performance, with success rates decreasing as complexity increases.

Closed-Loop Planning Boosts Robotic Success Rates

This research demonstrates that closed-loop planning consistently improves robotic performance compared to open-loop methods, even in static environments. Experiments investigated the impact of control horizon and the frequency of replanning on performance, revealing that a shorter control horizon does not always guarantee superior results. While the shortest control horizon achieved the best Task Completion Rate in many scenarios, these improvements were not always statistically significant. Through controlled experiments, researchers investigated how factors such as control horizon and warm-starting influence the performance of these models as planners, focusing on both geometric and logical reasoning capabilities. Results indicate that closed-loop planning consistently outperforms open-loop methods, likely due to its ability to adapt to environmental changes during robot interaction. The study introduces a suite of metrics to evaluate planner performance, including Task Completion Rate, Goal Achieved Rate, and measures of logical correction, allowing for detailed analysis of both overall success and specific reasoning strengths.