Iterative RAG Achieves Superior Performance to Gold Context in 11 LLMs

Scientists are increasingly exploring Retrieval-Augmented Generation (RAG) to enhance large language models, but a critical question remains: when does repeatedly retrieving and reasoning actually improve performance over simply providing all relevant information at once? Mahdi Astaraki, Mohammad Arshi Saloot, and Ali Shiraee Kasmaee, from BASF Canada Inc. and McMaster University, alongside Hamidreza Mahyar and Soheila Samiee, address this challenge in a new diagnostic study. Their research demonstrates that an iterative RAG approach consistently surpasses even an ‘ideal’ static RAG system , one supplied with perfect evidence , on the complex, multi-hop reasoning tasks found within scientific domains like chemistry. This finding, based on analysis of eleven state-of-the-art LLMs and the ChemKGMultiHopQA dataset, is significant because it reveals that how information is retrieved and processed can be more impactful than having all the information, offering crucial guidance for building more reliable and controllable RAG systems for specialised scientific applications.

and., 2020). Yet most evaluations treat retrieval as a static preprocessing step, followed by one-shot generation over a fixed context. Recent work argues that advanced RAG algorithms, such as iterative or dynamic RAG, can outperform static pipelines by progressively focusing the evidence set and correcting course mid-chain (Gao et al., 2025). This strategy supports multi-hop QA in two complementary ways: (i) reasoning-augmented retrieval and (ii) retrieval-augmented reasoning. Prior work on reasoning-augmented retrieval typically assumes that the “ideal evidence”, Gold Context supplied by dataset annotators, defines an upper bound, and thus evaluates how far improved retrieval can approach that bound without surpassing it (Nahid & Rafiei, 2025).

Meanwhile, another group of studies (Xu et al., 2024; Li et al., 2025b; Wu et al., 2025) focus on enhancing retrieval-augmented reasoning, showing superior performance over direct reasoning without retrieval and over standard RAG pipelines. However, most existing comparisons use one-step retrieval as the only baseline. This makes results highly sensitive to parsing, chunking, embedding, and re-ranking design choices, and obscures whether improvements stem from the algorithm or simply from retrieval configuration variance. Moreover, the Gold Context baseline (commonly included in reasoning-augmented retrieval studies) is often absent in evaluations of retrieval-augmented reasoning, making it difficult to form a complete picture.
Although Gold Context is not guaranteed to be an operational upper bound, since it may be distracting for long chains with many internal hops, misaligned with a model’s reasoning trajectory, or insufficient for compositional synthesis (Nahid & Rafiei, 2025; Chen et al., 2025), its inclusion remains essential for understanding the limits of static RAG. Taken together, prior work tends to evaluate only one side of the retrieval, reasoning interaction, either retrieval-enhanced reasoning or reasoning-enhanced retrieval, and rarely examines how both critical baselines (No Context and Gold Context) jointly shape conclusions. Moreover, most studies evaluate only a small set of language models (typically fewer than five), which limits any systematic assessment of how model architecture influences observed performance. Existing survey papers primarily summarise reported results without offering deeper diagnostic insights, and direct, mechanism-level evaluation of retrieval-augmented reasoning in scientific multi-hop QA remains largely unexplored.

Their goal is to jointly evaluate both aspects of potential enhancements within a single controlled framework, and to determine whether iterative retrieval can support reasoning strongly enough to surpass an idealised static evidence condition. To achieve this, they evaluate three regimes: (i) No Context (parametric memory only), (ii) Gold Context (Oracle evidence is supplied to the generator as a paragraph for each hop), and (iii) Iterative RAG (a controlled retrieval – i. e., reasoning loop with explicit step allocation and stopping). Their study focuses on chemistry QA, a domain where general-purpose training provides limited coverage and where retrieval is genuinely required to bridge knowledge gaps. They begin with a No Context screen to remove questions answerable from internal memory and concentrate the analysis on retrieval-dependent cases.

They structure the investigation around four questions: (1) Accuracy: Under what conditions does iterative RAG outperform Gold Context, and how does this vary across model families and hop depths? (2) Utilisation Dynamics: How do models use the retrieval loop to self-correct (e. g., anchor propagation), allocate steps across the chain, and calibrate stopping? (3) Failure Modes: What are the dominant sources of error in scientific multi-hop QA (e. g., coverage gaps in the final hop, composition failures despite sufficient evidence,0.64 percentage point. (2025). Research on multi-hop question answering has been driven by the release of several datasets designed to test reasoning beyond simple fact retrieval. HotpotQA Yang et al. (2018) is perhaps the most widely used example, with questions that force models to draw connections between different Wikipedia passages. Importantly, it also provides supporting sentences, which means systems can be assessed not just on correctness but also on whether their reasoning steps are traceable. A related resource, WikiMultihopQA Ho et al. (2020), pushes this further by linking structured information from Wikidata with unstructured text, and by explicitly annotating inference paths.

Iterative RAG boosts complex chemistry question answering

Scientists achieved a significant breakthrough in retrieval-augmented generation (RAG) demonstrating that iterative retrieval-reasoning loops can surpass static RAG performance, even when provided with idealised evidence. This demonstrates a substantial improvement in multi-hop question answering within a complex scientific domain. Experiments meticulously measured model reliance on parametric memory, comparing performance across three regimes: No Context, Gold Context, and Iterative RAG. The team isolated questions genuinely requiring retrieval to analyse behaviour through diagnostics encompassing retrieval coverage gaps, anchor-carry drop, query quality, composition fidelity, and control calibration.

Data shows that staged retrieval effectively reduces late-hop failures, mitigates context overload, and dynamically corrects early hypothesis drift, benefits unattainable with static evidence provision. Results demonstrate that the iterative process of retrieval is often more influential than simply having access to perfect evidence. Specifically, the study recorded improvements in handling sparse domain knowledge and heterogeneous evidence, crucial for scientific reasoning. Analysis of retrieval dynamics revealed how models utilise the iterative loop for self-correction, step allocation across reasoning chains, and calibration of stopping criteria.

The team quantified sufficiency and coverage at each hop, providing a detailed mechanism-level understanding of the process. Further investigation identified limiting failure modes, including incomplete hop coverage, distractor latch trajectories, early stopping miscalibration, and high composition failure rates, even when retrieval was perfect. Despite these challenges, the work establishes a foundation for developing more reliable and controllable iterative retrieval-reasoning frameworks. The research provides practical guidance for deploying and diagnosing RAG systems in specialised scientific settings, offering a pathway towards enhanced performance in complex knowledge domains. The code and evaluation results are publicly available, facilitating further research and development in this area.

Iterative RAG excels in complex chemistry questions

Scientists have demonstrated that iterative retrieval-augmented generation (RAG) consistently outperforms a static RAG approach, even when provided with ideal, complete evidence, termed ‘Gold Context’, in the complex domain of chemistry question answering. The study employed the ChemKGMultiHopQA dataset, focusing on questions demanding genuine information retrieval and multi-hop reasoning, and assessed performance across various conditions: no external context, complete ‘Gold Context’, and iterative RAG. The findings establish that staged retrieval significantly reduces failures at later reasoning steps, alleviates issues caused by excessive context, and dynamically corrects initial errors in hypothesis formation. Gains of up to 25.6 percentage points were observed, particularly in LLMs fine-tuned for reasoning tasks, indicating that the process of iterative retrieval can be more impactful than simply having access to perfect information. However, the authors acknowledge limitations including incomplete coverage of relevant information during retrieval, susceptibility to irrelevant ‘distractor’ information, miscalibration of stopping criteria for the iterative process, and challenges in accurately composing information from multiple sources. Future research should focus on addressing these remaining failure modes and developing more robust and controllable iterative retrieval-reasoning frameworks, as highlighted by the team.