AI Sees and Reasons with Images More Reliably

Researchers are tackling the challenge of improving reasoning capabilities in vision-language models (VLMs), where effectively combining visual and textual information remains difficult. Mingjia Shi, Yinhan He, Yaochen Zhu, and Jundong Li, all from the University of Virginia, present a new approach to address the tendency of these models to become overly reliant on text during reasoning, potentially leading to errors and hallucinations. Their work introduces Saliency-Aware Principle (SAP) selection, a method that operates on high-level reasoning principles to provide stable control and allow for revisiting visual evidence as needed. This is significant because SAP is both training-free and model-agnostic, offering a practical solution to enhance VLM performance, reduce object hallucination, and improve reasoning stability and speed compared to conventional methods.

Imagine explaining a complex picture to someone over a crackly phone line, misunderstandings quickly build up. Current computer vision systems struggle with similar problems when interpreting images alongside text, losing track of what’s actually in the picture during lengthy reasoning processes. This new approach allows these systems to revisit visual details as needed, maintaining a clearer understanding throughout complex tasks.

Scientists have observed that textual reasoning in vision-language models (VLMs) can become increasingly text-dominated during autoregressive generation, leading to the accumulation of early visual grounding errors. Conventional visual grounding guidance during inference is often coarse and noisy, hindering effective steering of reasoning over extended texts.

To address these challenges, researchers propose Saliency-Aware Principle (SAP) selection. SAP operates on high-level reasoning principles rather than token-level trajectories, enabling stable control over discrete generation under noisy feedback while allowing later reasoning steps to re-consult visual evidence when renewed grounding is required. Additionally, SAP supports multi-route inference, facilitating parallel exploration of diverse reasoning behaviours.

SAP is model-agnostic and data-free, requiring no additional training. Empirical results demonstrate that SAP achieves competitive performance, particularly in reducing object hallucination, under comparable token-generation budgets, while yielding more stable reasoning and lower response latency than chain-of-thought (CoT)-style long sequential reasoning.

Vision-language models (VLMs) aim to solve multimodal reasoning tasks by jointly processing visual and textual inputs. Recent advances in large language models (LLMs) have shown that allocating additional inference-time computation, such as generating longer reasoning sequences or exploring multiple reasoning routes, can improve reasoning quality. This capability, referred to as inference-time scaling, has become a central mechanism for enhancing LLM-based reasoning systems.

Nevertheless, whether similar inference-time scaling can be achieved in VLMs remains an open question, despite growing interest in multimodal chain-of-thought (COT) reasoning research. Traditional inference-time scaling relies on the ability of LLMs to iteratively refine reasoning over long inference horizons, where intermediate reasoning states are revisited, corrected, and extended as generation proceeds.

In the multimodal setting, this requires not only producing longer textual reasoning chains but also continuously incorporating and re-evaluating visual evidence throughout inference. In contrast to language-only reasoning, vision-language reasoning generally requires repeatedly aligning textual reasoning states with visual grounding. As a result, effective scaling critically depends on revisiting visual information rather than a one-time visual summary at the beginning of the reasoning process.

Achieving high-quality visual grounding is difficult due to a systematic discrepancy between the textual and visual modalities during the autoregressive generation process. Textual representations are explicitly generated and updated at every decoding step, whereas visual information is typically incorporated through fixed encodings or limited cross-modal interactions.

As generation length increases, this discrepancy causes the reasoning process to become increasingly text-dominated, a phenomenon empirically linked to hallucination and biased visual reasoning. While summarizing visual content early and relying on the text afterwards may appear sufficient, such summarization is inherently lossy, as omissions or misinterpretations introduced during early summarization cannot be corrected by later reasoning steps.

As a result, inference-time scaling amplifies early visual grounding errors, leading to a reasoning route that drifts away from the underlying visual evidence. A general approach to mitigating this issue is to introduce supervision signals during inference that encourage the model to attend to underutilized vision modalities. However, providing effective guidance in vision-language reasoning presents fundamental challenges.

In multimodal tasks, supervision signals often reflect inconsistent and implicit evaluation principles regarding how visual evidence should be used, leading to noisy and difficult-to-calibrate feedback. Also, inference-time reasoning unfolds through discrete textual generation processes, where guidance signals are inherently coarse, making them difficult to propagate to intermediate multimodal reasoning trajectories.

Together, these factors render stable and general inference-time optimisation particularly challenging in the multimodal setting. To address these challenges, researchers introduce Saliency-Aware Principle Selection (SAP) for vision-language reasoning. SAP adopts a model-agnostic, black-box formulation of test-time scaling that leverages visual saliency as a modality-aware guidance signal, enabling consistent and high-quality utilisation of informative visual evidence throughout the inference process rather than relying on a one-time visual summary.

By operating on high-level reasoning principles instead of token-level trajectories, SAP mitigates the accumulation of early visual grounding errors and prevents long-horizon reasoning from drifting toward text-only states. Performing inference-time optimisation in a discrete principle space allows SAP to remain strong to noisy feedback. Initial visual attention during reasoning rapidly diminishes as textual generation proceeds, dropping from 25.31% to 2.71% and further to 0.02% as the sequence length increased.

This substantial decline in visual focus contributes to text-dominated reasoning and, as a result, object hallucination. Work presented here demonstrates a method to counteract this effect, maintaining greater visual grounding throughout the reasoning process. SAP enables parallel exploration of diverse reasoning behaviours. Specifically, SAP supports multi-route inference, allowing the model to consider multiple potential paths simultaneously.

Measurements on the OCRVQA and POPE-recall benchmarks show improved perception, indicating a reduction in object hallucination. Performance on perception-intensive tasks was in particular better with SAP integration. By employing visual information fitness as a guiding principle, SAP identifies and prioritizes salient objects within the image. Results indicate that SAP yields lower response latency than CoT-style long sequential reasoning.

Also, the research shows that SAP achieves competitive performance under comparable token-generation budgets, offering a balance between reasoning quality and computational cost. The method’s ability to maintain visual grounding throughout the reasoning process addresses a key limitation of traditional approaches. For years, artificial intelligence systems have struggled to truly “see” and reason about images, often falling back on textual shortcuts and making basic errors of perception.

A new approach shifts the balance, compelling vision-language models to repeatedly check their understanding against the visual evidence itself. Unlike previous methods that attempt to refine reasoning step-by-step, this work focuses on guiding the overall principles of thought, allowing for more stable and reliable conclusions. It’s a subtle but important distinction, akin to teaching someone how to think rather than simply providing them with more facts. Once a model begins to rely heavily on its internal textual representations, errors can quickly accumulate, particularly when dealing with complex scenes.