Brorl Scales Reinforcement Learning with Broadened Exploration, Yielding Gains Beyond Thousands of Steps Via Hundreds of Rollouts

Reinforcement learning techniques are increasingly vital for developing complex artificial intelligence systems, and researchers continually seek ways to improve their performance and scalability. Jian Hu, Mingjie Liu, Ximing Lu, and colleagues demonstrate a significant advance in this field with their development of BroRL, a novel approach to broadening exploration during reinforcement learning. Their work addresses a key limitation of existing methods, where performance gains plateau after a certain amount of training, by dramatically increasing the number of rollouts examined per example. This strategy, informed by a detailed analysis of probability mass changes during learning, ensures continuous improvement even beyond the saturation point observed in previous techniques, and ultimately achieves state-of-the-art results on challenging benchmarks for large language models. By exhaustively exploring a wider range of possibilities, BroRL revives performance after existing methods reach their limits, paving the way for more powerful and capable AI systems.

Diverse Rollouts Improve Language Model Reasoning

This research introduces BroRL, a new reinforcement learning method that enhances the reasoning abilities of large language models. The core principle behind BroRL is that increasing the diversity of experiences during training is more effective than simply extending the training duration. Researchers achieve this diversity by significantly increasing the number of samples, known as rollouts, generated from each prompt during the learning process. This approach consistently outperforms existing methods, which often reach performance plateaus. Reinforcement learning trains an agent to make decisions within an environment to maximize rewards.

A larger rollout size allows the model to consider a wider range of possibilities during each training update. Results consistently demonstrate that BroRL outperforms existing methods across various tasks, achieving statistically significant improvements and avoiding performance plateaus. The research demonstrates that a larger rollout size is crucial for improving the performance of large language models on complex reasoning tasks. BroRL achieves comparable or better results with a similar number of total generated samples while requiring fewer GPU hours, highlighting its efficiency.

BroRL Scales Reinforcement Learning Through Broad Exploration

This study pioneers BroRL, a method for scaling reinforcement learning by dramatically increasing the number of rollouts per example. This approach addresses performance plateaus often encountered when simply increasing training steps. Researchers grounded BroRL in a detailed analysis of how probability mass shifts between correct and incorrect tokens during reinforcement learning, revealing that rollouts consistently contribute to expanding the probability of correct answers. To validate this theoretical framework, scientists developed a token-level simulator mirroring the per-token update analysis.

Experiments demonstrate that increasing the rollout size minimizes the influence of unsampled actions, leading to more stable policy updates and faster accumulation of probability mass for correct tokens. Results show that larger rollout sizes accelerate the growth of correct probability mass and increase the proportion of correct tokens that improve at each step. Further studies applied BroRL to large language models, achieving state-of-the-art results across diverse benchmarks. This work establishes a principled method for robust and continuous improvement in reinforcement learning, translating theoretical guarantees into practical gains for complex reasoning tasks.

BroRL Revives Reinforcement Learning Performance Gains

Scientists achieved continuous performance gains in reinforcement learning by systematically increasing the number of rollouts per example, a technique termed BroRL. This research addresses the performance plateaus observed when scaling reinforcement learning solely through increased training steps, demonstrating that performance can be revived and continuously improved beyond established checkpoints. The team’s approach is grounded in a mass balance equation analysis, which demonstrates that increasing the number of rollouts ensures overall expansion of probability mass for correct tokens. Simulations confirm that a sufficiently large rollout size guarantees an increase in the probability mass of all correct tokens, effectively eliminating knowledge shrinkage. BroRL achieves state-of-the-art results across diverse benchmarks and is both more data- and compute-efficient, highlighting the practicality of BroRL for real-world deployment.

Rollout Size Stabilises Language Model Learning

This work establishes rollout size as a critical factor in scaling reinforcement learning for large language models, demonstrating that increasing the number of rollouts sampled per prompt consistently improves performance beyond the limitations encountered when simply increasing training steps. Researchers discovered that performance plateaus arise not from fundamental limits of the learning process, but from instabilities caused by insufficient exploration of possible solutions. Through a mass balance equation analysis, they identified an “unsampled coupling” term that contributes to these instabilities and proved that increasing rollout size systematically mitigates its effect, ensuring a more reliable learning signal. Importantly, this improvement was achieved with increased computational efficiency, shifting the bottleneck from memory to compute. This research offers a pathway to continuous learning and improved reasoning capabilities in large language models.