Arenabencher Evolves Benchmarks Via Multi-model Evaluation, Preserving Comparability and Exposing Shared Weaknesses in Test Cases

The validity of benchmarks, essential tools for measuring and improving artificial intelligence, faces a growing challenge from data leakage, where models simply memorise training data rather than demonstrating genuine understanding. Qin Liu from University of California, Davis, Jacob Dineen and Yuxi Huang from Arizona State University, along with colleagues, address this problem by introducing ArenaBencher, a new framework that automatically evolves benchmarks to ensure they accurately assess a model’s capabilities. ArenaBencher works by identifying the core skills each benchmark question tests, then generating new, challenging questions that preserve the original intent, and rigorously verifying their correctness. This iterative process, guided by large language models, produces updated benchmarks that reveal previously hidden weaknesses, increase difficulty, and provide a fairer, more reliable measure of progress in artificial intelligence.

Automated Benchmark Generation for Large Language Models

This research introduces a system for automatically creating benchmarks used to evaluate large language models, streamlining a process traditionally performed manually. The system analyzes existing benchmark questions to understand the underlying skills being tested, such as reasoning, safety awareness, or factual knowledge, and identifies the specific concepts and difficulty level involved. It then generates new test cases that assess the same capabilities, ensuring they maintain a similar level of challenge while using different wording to avoid simple memorization. This automation allows for the creation of a much larger and more diverse set of benchmarks, targeting specific skills and mitigating the risk of models simply memorizing existing test questions.

The core of the system relies on prompts sent to a powerful language model, instructing it to analyze existing benchmarks and generate new ones. These prompts guide the model to extract key information about the test’s purpose, identify the skills being assessed, and create new questions that maintain the original intent while varying the phrasing. The system outputs the generated test cases in a structured format, making them easy to integrate into automated evaluation pipelines. The detailed analysis of test targets also provides valuable insights into the strengths and weaknesses of different language models.

Evolving Robust Language Model Benchmarks Automatically

Researchers have pioneered ArenaBencher, a novel framework that automatically evolves benchmarks used to assess language models, tackling the critical problem of data leakage from pretraining data. Recognizing that models can sometimes memorize training data rather than demonstrate genuine understanding, the team developed a system to create more reliable evaluations. ArenaBencher begins by determining the core ability each test case intends to measure, such as multi-step arithmetic or identifying potentially harmful actions. This understanding then guides the generation of new question-answer pairs, introducing controlled variations while preserving the original task’s objective.

To ensure the quality of these new test cases, the team employs a language model as a judge, verifying the correctness of the answers and their alignment with the intended ability. Candidate test cases are then evaluated across a diverse set of language models, and their performance is assessed using aggregated metrics like loss values or behavioral failures. This multi-model evaluation helps to reduce biases and identify challenges that expose shared weaknesses across different systems. The framework further refines test cases through iterative improvement, retaining the strongest candidates as examples to guide subsequent generation, amplifying signals of common failure patterns while maintaining alignment with the original task intent.

Evolving Benchmarks, Mitigating Data Leakage in LLMs

This research presents ArenaBencher, a new framework designed to automatically evolve benchmarks for large language models while maintaining comparability. Researchers addressed the critical issue of data leakage from pretraining data, which can inflate benchmark scores and distort progress measurement. The core of ArenaBencher involves iteratively updating test cases, ensuring that new questions preserve the original objective while increasing difficulty and exposing model weaknesses. This process utilizes a language model as a judge to verify correctness and intent, and aggregates feedback from multiple models to select challenging and diagnostic cases.

Experiments demonstrate that ArenaBencher significantly improves benchmark quality across three diverse tasks: mathematical reasoning, commonsense reasoning, and safety. On the mathematical reasoning dataset, the framework achieved an 8. 6 percent increase in accuracy while simultaneously increasing difficulty. In the safety domain, the attack success rate on the original benchmark decreased from 76. 4 percent to 68.

2 percent on the updated version, demonstrating improved robustness. For commonsense reasoning, the updated benchmark showed an accuracy of 60. 6 percent, a decrease of 19. 7 percent from the original, indicating increased challenge. The evaluation metrics reveal substantial improvements in benchmark characteristics, including fairness and separability, consistently exceeding 85 percent and 10 percent respectively. These results demonstrate that ArenaBencher effectively generates benchmarks that are more challenging, fairer, and better aligned with the intended evaluation objectives, providing a scalable path to continuously assess and improve large language models.

Evolving Benchmarks To Combat Data Leakage

ArenaBencher represents a significant advance in the evaluation of large language models, addressing the critical issue of data leakage from training data that compromises benchmark validity. Researchers developed a framework capable of automatically evolving benchmarks by generating updated test cases while maintaining comparability to the originals. This process infers the core abilities tested by each question, then creates new variations using language models, verifying both correctness and intended meaning with an independent judging system. Through iterative refinement guided by challenging examples, ArenaBencher consistently identifies new failure modes and increases difficulty without altering the fundamental skills being assessed.

Experiments across mathematics, commonsense reasoning, and safety domains demonstrate that ArenaBencher successfully enhances benchmark difficulty, preserves alignment with original objectives, and largely maintains the ability to differentiate between models. The framework offers a scalable pathway towards continuous benchmark evolution, keeping pace with the rapid advancements in foundation models and mitigating the effects of data contamination. While acknowledging this work as a first step, the authors identify future research directions including expanding the scope to multimodal settings and strengthening validity checks through more sophisticated constraints and calibrated judging ensembles.