A significant 64.5% of quantum code shared alongside published research papers fails to run successfully in a new computing environment, according to a new analysis by Dominik Köster and colleagues. The team’s combined manual and automated assessment of nearly 5000 quantum computing papers reveals a substantial barrier to verifying claims within the rapidly developing field. Their manual evaluation of 127 papers found that only 24.4% actually provide code artifacts, and approximately one-third of those with accessible code lack machine-readable environment specifications. These findings suggest that “reproducibility is not yet consistently achieved in quantum computing research,” and point to unrecorded development environment details as a key obstacle.
A substantial 64.5% of papers included publicly accessible code. A focused manual evaluation of 127 papers further illuminated the problem, finding that just 24.4% actually provide code artifacts. As the authors note, these failures often occur independently of long-term dependency changes, and suggest that some artifacts may never have been fully reproducible outside the original development setup. This lack of reproducibility isn’t simply a matter of outdated software, but frequently stems from incomplete or implicit environment specifications. The researchers observed patterns of unrecorded dependencies and undocumented configuration steps, creating a barrier to independent verification.
While tools like virtual environments and containerization can mitigate some of these issues, they often prove insufficient to capture the full execution context. The study highlights the potential of declarative environment specifications, using languages like Nix and devenv.sh, to create explicit and reproducible environment definitions that reduce reliance on undocumented assumptions and ensure long-term reproducibility. The authors emphasize that providing code is generally highly commendable and a necessary precondition for reproducibility in quantum computing.
The current state of quantum computing research reveals a significant gap between the aspiration of reproducible science and the practical realities of software-driven experimentation. While the field increasingly relies on complex codebases to demonstrate advancements, a substantial proportion of published work lacks the necessary components for independent verification. Researchers are finding that simply having code available is insufficient; its functionality must be readily demonstrable. A recent analysis indicates that 64.5% of quantum code accompanying published papers fails to execute successfully in a clean environment, despite being made accessible, which is a critical barrier to confirming reported results. The team’s manual evaluation of 127 papers revealed that only 24.4% actually provide code artifacts, a surprisingly low rate given the field’s growing software complexity. The researchers advocate for declarative environment specifications, citing languages like Nix and devenv.sh.
Researchers at Leibniz Universität Hannover are meticulously examining the practical hurdles to verifying results in quantum computing, going beyond simply assessing code availability to pinpoint the reasons why shared code often fails to run. Dominik Köster and colleagues discovered a significant disconnect between code sharing and actual reproducibility, finding that 64.5% of publicly available quantum code fails to execute successfully in a fresh environment. This isn’t merely a matter of outdated dependencies; the team’s analysis suggests deeper issues with how these quantum software projects are initially developed and documented. The investigation revealed a prevalence of implicit assumptions about the development environment, such as unreferenced locally installed dependencies or undocumented configuration steps. These unrecorded details create a substantial barrier for researchers attempting to independently validate published findings.
The team manually evaluated 127 papers, and their large-scale automated screening of nearly 5000 quantum computing papers corroborated the manual findings, showing a consistent code availability rate of 26.8%. Approximately one-third of papers with accessible code lacked machine-readable environment specifications, further compounding the problem. The team advocates for declarative environment specifications, highlighting languages and tools like Nix and devenv.sh that enable explicit and reproducible environment definitions. These approaches, they argue, are crucial for reducing implicit assumptions and ensuring long-term reproducibility, even as dependencies and hardware evolve. The study’s findings underscore that simply publishing code is insufficient; a complete and precisely defined environment is essential for truly verifiable quantum computing research.
The frustrating experience of failing to rerun published research is a common hurdle in science, but particularly acute in quantum computing where complex software stacks are the norm. A recent analysis reveals that 64.5% of quantum code made available alongside publications fails to execute in a fresh environment, highlighting a significant impediment to verifying claims and building upon existing work. This isn’t simply a matter of code errors; the research points to deeper issues with how these computational experiments are packaged and shared. This lack of precise environmental documentation leads to a situation where results may appear to work for the original researchers, but prove impossible to reproduce elsewhere. The researchers suggest using systems like devenv.sh to address these shortcomings, which enable explicit and reproducible environment definitions, allowing researchers to precisely specify all software dependencies and configurations.
Source: https://arxiv.org/abs/2607.08348
