Int Achieves 14% Reasoning Improvement in LLMs with Self-Proposed Interventions

Researchers are tackling the critical problem of credit assignment within large language models (LLMs) used for complex reasoning tasks. Matthew Y. R. Yang (Carnegie Mellon University), Hao Bai (University of Illinois Urbana-Champaign), and Ian Wu, alongside Gene Yang, Amrith Setlur, and Aviral Kumar et al, present a novel approach called Intervention Training (InT) that allows LLMs to self-assess and correct their reasoning processes. This innovative technique moves beyond simply rewarding final answers, instead enabling the model to pinpoint and rectify errors within its own reasoning chain , a significant step towards more robust and reliable AI. By proposing targeted corrections based on readily available reference solutions, InT not only improves accuracy, achieving a nearly 14% boost on the IMO-AnswerBench benchmark with a 4B-parameter model, but also outperforms larger open-source LLMs like gpt-oss-20b.

This breakthrough reveals a method for fine-grained credit assignment, moving beyond traditional RL approaches that penalize entire reasoning chains for a single incorrect final answer or uniformly reinforce all steps in a successful trace. Experiments show that InT localizes errors to specific steps, upweighting the likelihood of corrective interventions and fostering more effective learning during both supervised fine-tuning and subsequent RL training. The research establishes that standard outcome-reward RL often discourages correct intermediate steps in failed reasoning and inadvertently reinforces spurious steps in successful ones, leading to undesirable behaviours such as increased verbosity or premature shifts in the reasoning process.

This innovative approach allows the model to generate counterfactual reasoning traces that succeed where the original failed, providing valuable learning signals even in challenging scenarios where no correct rollout is initially produced. The resulting model serves as a far better initialization for RL training, leading to substantial performance gains on complex mathematical reasoning benchmarks. After implementing InT and subsequent RL fine-tuning, the team improved accuracy by nearly 14% over a 4B-parameter base model on the IMO-AnswerBench, even outperforming larger open-source models such as gpt-oss-20b. This work opens exciting possibilities for developing more robust and reliable LLMs capable of tackling increasingly complex reasoning tasks, with potential applications in fields ranging from scientific discovery to automated problem-solving. Furthermore, the research demonstrates that InT makes particularly effective use of reference solutions, especially when combined with a strong base model and a relatively small supervised fine-tuning dataset.

LLM Self-Correction via Reasoning Intervention Training improves model

The study addresses the limitations of standard outcome-reward RL, which uniformly reinforces or penalises entire reasoning traces based solely on the final answer’s correctness. Researchers engineered a system where the LLM proactively identifies errors within its own generated reasoning and proposes single-step corrective interventions to improve trajectories. This approach leverages the asymmetry in difficulty between generating complete solutions and verifying individual steps, utilising textual comparison against reference solutions. The core of InT involves instructing a base LLM to analyse discrepancies between its previously generated, incorrect reasoning trace and a corresponding reference solution.

This targeted SFT selectively reduces the likelihood of incorrect reasoning steps, guiding the model towards more favourable alternatives. The team harnessed readily available reference solutions from mathematical reasoning datasets to facilitate this process. The study pioneered a method that avoids branched rollouts, explicit value-function training, or modifications to the RL objective, maintaining computational efficiency. Throughout the procedure, the research did not rely on a larger model, instead exploiting the inherent differences in task difficulty, instruction-following, verification, and generation, within the same model to achieve credit assignment.

InT improves LLM reasoning via credit assignment

The research addresses a key limitation of standard reinforcement learning (RL), its tendency to assign credit solely based on the final answer, potentially discouraging correct intermediate steps in failed reasoning traces and reinforcing spurious steps in successful ones. Experiments revealed that InT enables LLMs to perform fine-grained credit assignment on their own reasoning traces by proposing targeted corrections to steer trajectories toward higher rewards. The team measured performance using mathematical reasoning datasets, leveraging the ease of verifying generated solutions compared to generating correct ones from scratch. Results demonstrate that this process serves as a significantly better initialization for RL training, improving accuracy by nearly 14% over a 4B-parameter base model on the IMO-AnswerBench.

After online RL, the team recorded an average performance improvement of nearly 10% across four challenging mathematical reasoning benchmarks. The most significant gain was observed on the IMO-AnswerBench, achieving a ∼14% improvement on problems curated by former International Mathematical Olympiad medalists. These measurements confirm that InT is a simple yet effective paradigm for enhancing LLM reasoning through improved credit assignment. Data shows that the work focuses on assigning credit within incorrect rollouts by pinpointing the steps that derail the solution, assuming access to reference solutions commonly found in open-source math datasets. Analysis of Olympiad-level math problems revealed that over 80% of rollout groups contained no successful trajectories at the start of training, highlighting the potential of extracting learning signals from these failed attempts. The research further indicates that failed attempts average 10,000 to 15,000 more tokens than successful ones, making credit assignment particularly challenging in these longer trajectories.

Targeted Error Correction Boosts LLM Reasoning abilities

InT enables LLMs to perform fine-grained credit assignment by identifying errors within their own reasoning traces and proposing targeted, single-step corrections to steer towards correct solutions. The core of InT lies in exploiting the asymmetry between generating solutions and verifying existing steps; base models are often more reliable at identifying flaws in reasoning when compared to a known correct solution than at creating solutions from scratch. Supervised fine-tuning is then applied to the rollout up to the error point, combined with the intervention, effectively localising the mistake. This process results in a significantly better initialisation for subsequent reinforcement learning training. The authors acknowledge that the effectiveness of identifying appropriate interventions relies on the quality of reference solutions and the base model’s ability to perform self-verification. Future research could explore extending this approach to more complex tasks and investigating methods for automatically generating high-quality reference solutions.