AI Learns to Reason: New Method Boosts Diversity and Accuracy

Reinforcement Learning with Verifiable Rewards is proving to be a powerful technique for enhancing the reasoning abilities of large language models, but current methods often sacrifice the diversity of generated responses in pursuit of simple accuracy. Xiao Liang and Yelong Shen from the University of California, Los Angeles, working with Zhongzhi Li from the Chinese Academy of Sciences, Yeyun Gong and Weizhu Chen from Microsoft, and Zhijiang Guo from Hong Kong University of Science and Technology, address this limitation with a new training strategy. Their research demonstrates that actively updating the problems used for training helps maintain a balance between accuracy and diversity, preventing the model from becoming overly focused on a single solution. The team’s Self-play with Variational problem Synthesis method achieves substantial improvements in reasoning performance, delivering gains of over 18% and 22% on challenging benchmarks and consistently demonstrating robust generalizability across a range of model sizes.

For post-training Large Language Models (LLMs), reinforcement learning from human feedback is frequently employed, particularly for complex reasoning tasks. However, conventional reinforcement learning often improves performance at the expense of reducing the diversity of generated responses, ultimately limiting the potential for achieving high accuracy. Researchers have discovered that augmenting the training process with new, varied problems can effectively counteract this decline in diversity, maintaining a broader exploration of potential solutions.

Self-play Boosts Diversity in Language Models

Recent advances in training large language models focus on reinforcement learning techniques, particularly for tasks demanding complex reasoning abilities. This work introduces a novel strategy called Self-play with Variational problem Synthesis (SvS), which dynamically generates new training problems based on the model’s existing correct solutions. Crucially, this process doesn’t require external labeling or guidance, allowing the model to self-improve through continuous refinement. By synthesizing variations of problems the model already solves correctly, SvS maintains stable policy entropy, a measure of response diversity, throughout the training process, preventing the narrowing of reasoning pathways observed in traditional methods.

This represents a significant leap forward, as standard reinforcement learning methods often plateau at lower levels of accuracy on these demanding tasks. The research highlights a clear link between training data diversity and sustained learning, demonstrating that continuously updating the training set with new, varied problems not only improves performance but also fosters a more robust and adaptable reasoning capability in large language models. This approach offers a promising pathway towards unlocking the full potential of these powerful systems, enabling them to tackle increasingly complex challenges with greater accuracy and creativity.

Diverse Problem Synthesis Boosts Language Model Reasoning

This research introduces a new strategy, Self-play with Variational problem Synthesis (SvS), to improve reinforcement learning with verifiable rewards for large language models. The team observed that standard reinforcement learning can reduce the diversity of generated solutions, limiting overall reasoning performance. SvS addresses this by generating new training problems based on the model’s existing solutions, effectively augmenting the training data and maintaining a broader range of responses. The results demonstrate that SvS consistently outperforms standard reinforcement learning across various model sizes and reasoning benchmarks, notably improving performance on challenging tasks where standard methods show limited gains.

By generating diverse yet verifiable problems without requiring external annotations, SvS sustains exploration and enhances the model’s reasoning capabilities through self-improvement. The authors acknowledge that the method relies on the quality of the initial policy and that further research could explore the optimal balance between problem variation and semantic alignment. Future work may also investigate the application of SvS to other reinforcement learning paradigms and different types of reasoning tasks.