Generative Adversarial Reasoner Advances LLM Performance through Joint Training

Large language models demonstrate impressive capabilities in areas like mathematical reasoning, yet frequently stumble due to process errors including flawed calculations and illogical steps. Qihao Liu, Luoxin Ye, Wufei Ma, Yu-Cheng Chou, and Alan Yuille from Johns Hopkins University address this challenge with a new framework called the Generative Adversarial Reasoner. Their method enhances reasoning abilities by simultaneously training a language model to generate solutions and another to act as a critical discriminator, using a process inspired by adversarial reinforcement learning. This innovative approach delivers substantial improvements in accuracy across several mathematical benchmarks, notably boosting performance on AIME24 with both DeepSeek-R1-Distill-Qwen-7B and DeepSeek-R1-Distill-Llama-8B, and also offers a flexible system for refining reasoning processes towards specific goals such as aligning with expert knowledge or verifying mathematical proofs.

Model Reasoning Steps and Correctness Judgments

This research investigates how large language models approach problem-solving, specifically in mathematical and logical reasoning. The team analyzed the detailed steps a model takes to reach a solution, revealing insights into its capabilities and limitations. Each problem-solving attempt is broken down into a series of reasoning steps, followed by a judgment of whether that step is correct or incorrect, and a detailed explanation of the evaluation. This analysis demonstrates that the model doesn’t simply provide answers, but attempts to articulate its thought process, which is crucial for understanding its reasoning and identifying potential errors.

The model also exhibits a degree of self-awareness, as it can assess the quality of its own reasoning. Detailed error analysis reveals common pitfalls, such as making unsupported assumptions, failing to fully analyze the problem, overlooking contradictory evidence, or lacking a systematic approach. The research highlights the model’s ability to cross-validate solutions using multiple approaches, demonstrating a sound problem-solving strategy. This approach co-evolves a language model with a discriminator through adversarial reinforcement learning, creating a system where both components improve through interaction. To manage computational demands, the team implemented a method to partition reasoning chains into logically complete segments of comparable length. The discriminator then evaluates each segment for logical soundness, providing a concise rationale for its judgment. This generates well-calibrated rewards, supplementing sparse signals and improving the model’s ability to learn from its mistakes.

The language model is rewarded for logically consistent steps leading to correct answers, while the discriminator is rewarded for accurately detecting errors. This co-adaptation dynamically aligns the reward signal with the model’s evolving capabilities, reducing the need for extensive manual annotation. Experiments on benchmarks including AIME24 demonstrated significant improvements in performance, boosting scores for different language models. Further testing on various datasets consistently demonstrated gains over strong baseline models. The modular design of the discriminator also allows for flexible reward shaping, enabling applications such as teacher distillation and alignment with human preferences. Researchers addressed the persistent problem of process errors, incorrect calculations, and flawed logic in LLMs designed for mathematical reasoning. The team developed a system where an LLM “reasoner” and an LLM-based “discriminator” co-evolve through adversarial reinforcement learning, leading to substantial improvements in performance. The core of GAR involves partitioning complex reasoning chains into logically complete segments, allowing the discriminator to evaluate each segment for soundness and provide focused feedback.

This simplifies the task for the discriminator and delivers interpretable justifications for its assessments. The reasoner is then rewarded for producing logically consistent steps that lead to correct answers, while the discriminator receives rewards for accurately identifying errors and distinguishing valid reasoning from flawed attempts. Experiments demonstrate that this co-adaptation dynamically aligns the reward signal with the model’s evolving capabilities, reducing the need for costly, fine-grained annotations. Across several mathematical benchmarks, GAR consistently improved performance over strong baseline models. Specifically, on the AIME24 benchmark, the team boosted the performance of different language models by significant margins. These results highlight the potential of GAR to deliver more reliable and accurate LLMs for complex problem-solving.

Adversarial Reasoning Improves Mathematical Problem Solving

The research team developed a novel method to enhance the reasoning capabilities of large language models, addressing the common issue of process errors in mathematical problem-solving. The results demonstrate consistent improvements in performance across various mathematical benchmarks, notably increasing scores on the AIME24 dataset when applied to different language models. Importantly, the team also observed that these gains in accuracy were achieved without a reduction in model entropy, indicating that the method preserves the model’s ability to explore diverse solutions and avoid overconfidence.

Analysis reveals a mechanism where the system encourages deterministic reasoning on predictable segments while maintaining exploration on more complex decision points. The authors acknowledge that further research is needed to explore the full potential of this approach and to investigate its effectiveness with different language model architectures. The modular design of the discriminator also allows for flexible reward shaping, potentially enabling applications such as teacher distillation and alignment with human preferences.