GPT-5.4 Repairs 48.8% of Qiskit Bugs Across Six Versions

GPT-5.4 successfully repaired 48.8% of bugs across six versions of Qiskit, a leading quantum software development kit, achieving the highest pass@10 metric among the models tested, including GPT-5.4-mini (47.3%), GPT-5o-mini (30.3%), and GPT-4o-mini (22.6%). Researchers, led by Saumya Brahmbhatt, evaluated the LLMs using Bugs4Q, a benchmark of 67 unique real Qiskit defects, and discovered an issue with benchmark reliability: 64% of successful repairs addressed code already flagged as invalid by the quantum software environment. This finding highlights that correctness depends on the specific pairing of benchmark and Qiskit version, rather than the benchmark alone. The team’s work introduces a version-pinned validation protocol and a re-validated Bugs4Q benchmark, demonstrating that benchmark validation must precede repair evaluation.

Bugs4Q Benchmark & Defect Collection for Qiskit Repair

Nearly half of attempted quantum code repairs using advanced artificial intelligence are, in effect, fixing code that was already broken by the testing environment itself, according to new research into the reliability of quantum benchmarks. Scientists have detailed significant inconsistencies in the Bugs4Q benchmark, a widely used collection of real quantum defects, revealing that its validity is heavily dependent on the specific version of the Qiskit quantum software framework used for testing. The researchers constructed a deduplicated benchmark of 67 unique Qiskit defects, sourced from Bugs4Q-Framework and Bugs4Q-NA, and authored executable tests for cases where they were originally missing. This process revealed a surprising degree of instability; the team found that entries become invalid across different Qiskit versions.

This demonstrates that benchmark labels are not inherently stable and that correctness is tied to the specific (benchmark, version) pairing. The team reports that validity is not stable across releases, emphasizing the need for rigorous validation before assessing repair performance. Evaluating four large language models, GPT-4o-mini, GPT-5o-mini, GPT-5.4, and GPT-5.4-mini, the researchers found that GPT-5.4 achieves the highest pass@10 (48.8%), followed by GPT-5.4-mini (47.3%). However, performance across all models peaked on Qiskit version 0.45.0 and declined after the transition to Qiskit 1.0. This sensitivity to the underlying software environment is a key finding, suggesting that LLM-based repair is not simply a matter of logical correction but is also heavily influenced by API compatibility. Most surprisingly, the analysis revealed that 64% of observed passes occur on invalid benchmark entries.

This means that in a significant number of cases, the LLMs were successfully “repairing” code that was already failing due to environmental issues, rather than correcting genuine logical errors. As the researchers state, much of the invalidity is “silent,” with buggy programs sometimes passing the test outright, obscuring the true effectiveness of the repair. The team has released the re-validated benchmark and associated artifacts for reproducibility, hoping to establish a more reliable foundation for evaluating and advancing LLM-driven quantum code repair.

Version-Pinned Validation Protocol Across Qiskit Releases

The pursuit of reliable quantum software is increasingly focused on automated program repair, leveraging the capabilities of large language models (LLMs). However, a fundamental challenge has emerged regarding the stability of benchmarks used to evaluate these repair tools; the very definition of a “buggy” program can shift with changes to the underlying quantum software environment. Their investigation, detailed in a paper with a license date of July 10, 2026, reveals a surprising degree of instability in the widely used Bugs4Q benchmark suite. This process involved careful review to ensure tests accurately reflected the reported issue, rather than incidental output. The core of their methodology lies in testing each bug-fix pair across six distinct Qiskit releases (0. 25. 0, 0. 45. 0, 1. 0. 0, 1. 1. 1, 2. 0. 0, and 2. 3. 1).

A program deemed faulty under one Qiskit version might run perfectly well, or fail to run at all, in another, rendering the benchmark invalid for evaluation in that environment. This instability isn’t merely a matter of broken tests; a significant portion of these failures are “silent,” meaning the buggy program passes the test outright, indicating the original fault no longer reproduces. In many (case, version) pairs, entries become invalid. The researchers found that the LLMs tested, GPT-4o-mini, GPT-5o-mini, GPT-5. 4, and GPT-5. 4-mini, performed best on Qiskit version 0. 45. 0, with GPT-5. 4 achieving the highest pass@10 (48.8%), followed closely by GPT-5. 4-mini at 47. 3%. Performance declined after the transition to Qiskit 1. 0. Further analysis revealed that environment incompatibility accounts for a substantial portion of repair failures, ranging from 13 to 56% depending on the model.

LLM Repair Evaluation: GPT-4o-mini to GPT-5.4 Performance

Researchers are meticulously examining the reliability of large language models in automatically correcting errors within quantum software, a field demanding increasing precision as quantum computing scales. Saumya Brahmbhatt and colleagues focused on Bugs4Q, a benchmark containing real defects in Qiskit, the popular open-source quantum computing framework, and discovered a surprising fragility in the very foundations of automated repair evaluation. Their work reveals that the validity of benchmark tests isn’t inherent to the bug itself, but rather a complex interplay between the bug, the attempted fix, and the specific version of Qiskit used for testing. They meticulously constructed executable tests for cases lacking them, and then ran both the original buggy code and proposed fixes across six pinned Qiskit releases, ranging from version 0.25.0 to 2.3.1, to establish a controlled evaluation environment.

Evaluating four LLMs, GPT-4o-mini, GPT-5o-mini, GPT-5.4, and GPT-5.4-mini, the researchers found that GPT-5.4 achieves the highest pass@10 (48.8%), followed by GPT-5.4-mini (47.3%), GPT-5o-mini (30.3%), and GPT-4o-mini (22.6%). However, this performance was heavily influenced by the Qiskit version; all models peaked on Qiskit 0.45.0. A substantial portion of failures weren’t due to incorrect repairs, but rather to deprecated or incompatible APIs within the Qiskit environment itself. This finding underscores the critical need for rigorous benchmark validation before evaluating repair capabilities, and the team has released a re-validated, version-pinned Bugs4Q benchmark to facilitate more reliable future assessments. The researchers emphasize that benchmark validity is a property of the (case, version) pair rather than the benchmark alone, a crucial insight for the development of trustworthy LLM-driven quantum software repair tools.

Impact of Qiskit Version Transitions on Repair Success

The increasing reliance on automated program repair, driven by large language models, is encountering an unexpected obstacle in the realm of quantum computing: the instability of benchmark tests themselves. Recent research demonstrates that the very foundations used to evaluate the success of these repair algorithms are surprisingly fragile, shifting with changes in the underlying quantum software environment. This poses a significant challenge to confidently assessing progress in LLM-driven quantum code correction and highlights the need for rigorous, version-controlled validation procedures. Researchers evaluated four LLMs, GPT-4o-mini, GPT-5o-mini, GPT-5.4, and GPT-5.4-mini, on their ability to repair 67 real-world Qiskit defects, meticulously tracking performance across six distinct Qiskit releases, ranging from version 0.25.0 to 2.3.1. The study revealed a peak in performance for all models on Qiskit version 0.45.0, with GPT-5.4 achieving the highest pass@10 (48.8%), followed by GPT-5.4-mini (47.3%), GPT-5o-mini (30.3%), and GPT-4o-mini (22.6%). However, performance declined after the Qiskit 1.0 transition.

This sensitivity to the software environment is underscored by the discovery that a substantial portion of seemingly successful repairs are occurring on benchmark entries that are already invalid under the target Qiskit version. This suggests that reported repair rates may be inflated, masking the true capabilities of these models. The root of this instability lies in the evolving nature of Qiskit’s API. Changes between releases can render previously valid tests ineffective, either by causing the buggy program to unexpectedly pass or by preventing the reference fix from executing altogether. They introduced a version-pinned approach, meticulously verifying the validity of each benchmark case against each Qiskit release before evaluating repair attempts, and released this re-validated benchmark for wider use. This work underscores that assessing the efficacy of LLM-driven quantum code repair requires not only evaluating the models themselves but also ensuring the stability and reliability of the benchmarks used to measure their performance.

Executable Test Oracle Construction & Validation Criteria

The assumption that software benchmarks represent stable, reliable measures of program correctness is increasingly challenged, particularly in the rapidly evolving field of quantum computing. While classical software benefits from mature testing infrastructure, quantum programs face unique hurdles stemming from the inherent complexities of quantum mechanics and the immaturity of supporting tools. Recent research demonstrates that even established benchmarks like Bugs4Q are surprisingly fragile, their validity inextricably linked to the specific software environment in which they are executed. Across Qiskit releases 0.25.0 to 2.3.1, revealing a disconcerting trend: quantum benchmarks can suffer from silent label inversion, where entries become invalid without errors when reference fixes stop executing or buggy programs no longer reproduce failures. This instability has significant implications for evaluating automated program repair (APR) tools, particularly those leveraging large language models (LLMs).

The team’s work evaluated GPT-4o-mini, GPT-5o-mini, GPT-5.4, and GPT-5.4-mini, generating up to ten repair candidates per defect. GPT-5.4 achieves the highest pass@10 (48.8%), and the researchers discovered a troubling phenomenon. Performance declined after the transition to Qiskit 1.0. This sensitivity underscores the critical need for version-pinned evaluation protocols and the inherent difficulty of creating stable, long-lived benchmarks for quantum software.