Quantum software testing at QCE26 met the reproducibility gap

This guide follows 22 talks from IEEE Quantum Week 2026 in Toronto on how to test quantum code and how to rerun quantum work. The tools grew sharper, and the records behind them stayed thin.

On the last afternoon of IEEE Quantum Week 2026, in Room 713B, Nicole Hoess of the Technical University of Applied Sciences Regensburg reported what happened when their group tried to rerun other people’s quantum experiments. Of the 31 published code packages they could fetch, 11 would start. The paper took its title from an old programmers’ excuse, works on my machine, and swapped the machine for a QPU, a quantum processing unit. For quantum software testing the joke has an edge. The excuse every programmer has used now hides results that nobody else can check.

Testing ordinary software rests on a simple move, running a program on an input and comparing its output with the answer you expected, and quantum programs break it. Their output is random, so a test must collect many runs, called shots, and compare the spread of results with statistics. Looking inside a running program destroys the state being examined, so no debugger can pause halfway. On real hardware, noise blurs the result until a correct program and a slightly wrong one look alike, and often the expected answer itself, which testers call the oracle, is simply unknown.

The field has borrowed decades of tools from classical software to cope, from planted bugs to proofs checked by machine. Reproducibility is the other half of trust, and it depends on far more than source code, because the compiler makes random choices fixed by a seed and the chip’s error rates drift between calibrations. Taken together, the 22 talks followed here point to an uncomfortable conclusion. The weakest link in trusting a quantum result is bookkeeping, the record of versions, seeds and settings. Cleverer tests will count for little until the field can rerun its own work.

The case for the clever tools is strong, and the speakers who made it had results to show. Masaomi Yamaguchi of Fujitsu checked chemistry runs against quantities the physics must conserve, such as particle number, and so tested programs whose right answer nobody knew. Ning Ma of Polytechnique Montréal trained a graph model to clean noisy outputs before judging them. José Campos and João Pascoal Faria of Porto made mutation testing far cheaper. At the formal end, Stavan Sunil Jain of UW Madison built a small error-correcting code in Lean, a language whose proofs a machine checks, so that a reader need not take the author’s word.

Each of those tools stops at a stated edge. Nearly every quantum software testing result here came from simulators, many of them at seven qubits or fewer, where an ordinary laptop can still work out the right answer by brute force. Jain left the code’s distance unproved. A Fordham and Stevens team checked the control loop of VQE, a hybrid chemistry method, without checking its numbers. The input generator from Maryse Ernzer and their co-authors still needed a classical simulation to judge its outputs. The tools work best where the answer can already be computed another way.

The argument turns in Finland, at the University of Jyväskylä. The group around Vlad Stirbu appeared five times in Toronto, and four of those talks carried one message. Record the circuit, the compiler settings, the software versions and the seed. Otso Kinanen argued and Jonny Peltonen showed that an unset seed in the transpiler, the tool that rewrites a circuit for a given chip, can change the compiled circuit with no change to the source. Valter Uotila described a run that scored far below its classical rival after it asked for four qubits instead of three and pulled values from an older experiment, errors a logged setting would have caught. None of this needs a proof assistant. It needs the habits of a careful laboratory.

Testing quantum programs when the answer is unknown

The hardest problem in quantum software testing is the oracle, the known right answer that a test compares against, because for the programs that matter most nobody can compute it in advance. Three talks found ways round it, and a fourth asked how many shots a judgement needs.

Fujitsu checks chemistry runs against conserved quantities

Masaomi Yamaguchi of Fujitsu went after programs whose right answer cannot be worked out on a normal computer, which is just where a quantum computer is meant to help. In their hydrogen example a bug adds an extra term to the energy formula. Both runs settle smoothly. Only the known exact energy exposes the fault.

Their tests use symmetry operators, quantities such as particle number or total spin that should stay fixed and can be read off the problem without solving it. A health check runs at each step of the optimiser, and an acceptance check on the final state measures how much the conserved quantity varies. Yamaguchi planted five faults in a hydrogen workflow. The acceptance check caught all five. The work was with Fuyuki Ishikawa of Japan’s National Institute of Informatics.

Used as a filter on 1,000 hydrogen runs, the check threw out 73.7 percent of those that missed the accuracy chemists need and none of those that reached it. Yamaguchi did not call it a proof. Passing both checks does not make a run correct, Yamaguchi said, and the tests used only simulated noise.

Random test inputs that look truly random

Maryse Ernzer, Seung Yeob Shin, Fabrizio Pastore and Domenico Bianculli asked how to choose test inputs for a quantum program, where each input is itself a quantum state. Three features matter, the size of each amplitude, its phase and the entanglement between qubits. The team defined nine scores of diversity. Each feature got three.

They proposed a brick circuit, rows of small rotating and entangling blocks laid out like brickwork, and compared it with three earlier designs on three, five and seven qubits. The brick circuit looked statistically random with the fewest gates, and its lead grew with qubit count, though a plain random string of gates did better on one entanglement score. The work is simulation only. Asked how such inputs can be judged, the speaker agreed that classical simulation is the oracle here.

Montréal cleans noisy outputs before judging them

Ning Ma of Polytechnique Montréal, with Jun Dai and Heng Li, began with a three-qubit circuit that returns 010 on all 1,024 shots on an ideal simulator, but only 932 times on a simulated IBM device. A tester comparing the two might call a correct program buggy. So the noise has to come out first.

Their tool, Q-Bridge, reads the compiled circuit as a map of gates, tags each gate with the error rates of its qubits and uses a neural network built for such maps to predict the clean output. Across nine circuit families on 23 simulated IBM devices, Ma reported a 53.6 percent average cut in distance from the ideal, against the strongest earlier method. After cleaning, tests caught about 83 percent of truly faulty variants. That earlier method caught under 10 percent. Every result came from simulated devices.

StableShots stops sampling once the output settles

Giuseppe Bisicchia of the University of Pisa and colleagues found that a quantum job which stops once its output settles can meet its target every time with fewer shots. In 3,600 held-out runs it never missed, at a median of 7,650 shots. The others were Alessandro Bocci and Antonio Brogi of Pisa and Ernesto Pimentel of the University of Malaga. Textbook bounds are little help. For one 14-qubit circuit, one bound asks for 2.27 million shots and another for 179.8 billion, while the real need is about 20,000 at the median.

StableShots runs shots in batches and stops once the running output changes by less than a set amount for several checks in a row. Tests used 180 traces from QSimBench, six circuit families of 4 to 14 qubits and five noisy simulated IBM backends, tuned on four and tested on the fifth. Fixed budgets of 5,000, 10,000 and 18,000 shots hit the target 54.1, 66.7 and 83.3 percent of the time. StableShots also logs why it stopped. A fixed budget cannot.

Mutation testing, made cheaper

Mutation testing plants small deliberate bugs, called mutants, in a program and counts how many of them the tests catch, so it grades the tests themselves. It is costly, because every mutant must be run. Two talks from Porto made it cheaper.

Porto skips mutants that change nothing

José Campos of the University of Porto went after equivalent mutants, the edits that change nothing at all, so that no test can ever catch them and every minute spent on them is wasted. Their filter, TBE, compiles the original and the mutant the same way. Then it compares the text Qiskit prints. Matching text marks a mutant as equivalent. The work was with Andriy Miranskyy of Toronto Metropolitan University.

From 348,299 surviving mutants of 375 circuits, a full check of the quantum states marked 92,011 as doing nothing new. TBE found 29,536 of them, about a third, with no false alarms, and used 30 percent less time and 63 percent less memory. The filter is cautious by design. TBE misses about two thirds of such mutants but never throws away a live one.

SAMT mutates the compiled circuit

João Pascoal Faria presented work with first author Amizito Rajabo and with Rui Abreu, from the University of Porto, INESC TEC and INESC-ID, on a tool called SAMT. It edits the compiled circuit in place of the source code. It also stops firing shots as soon as the evidence that a mutant differs is clear. That saves most of the runs.

On 12 programs from a standard benchmark set, run on five simulated qubits, Faria reported circuit-level mutation about 14,000 times faster than editing source code. Generating the mutants took about 8 thousandths of a second. The source-level tool took 27 to 35 seconds. SAMT covered all 12 programs, where a tool that edits source code reached 6, and it used 88 to 89 percent fewer shots with no false kills.

Proofs and model checking

Formal verification goes further than testing. It proves a program correct, and it gives certainty at a price, because proofs take expert effort and the states to explore double with every qubit. Four talks tried it. A fifth asked what it costs to match two circuits.

Lean comes to quantum error correction

Stavan Sunil Jain, a PhD student at UW Madison, opened a two-part Sunday tutorial on Lean, a proof assistant whose small core checks every proof against the claim it states, line by line. Jain pointed out that AI models now write Lean proofs, and that the core can check them. The programme lists three more names. They are Aws Albarghouthi and Swamit Tannu of UW Madison and Nithin Raveendran of the University of Arizona.

Jain cited Google DeepMind’s AlphaProof reaching a silver-medal mark at the 2024 International Mathematical Olympiad, and the startup Harmonic reaching gold in 2025, both with proofs set down in Lean. The live demo proved small claims and built custom types for playing cards, the same tools that would later define quantum operators. Jain gave one warning. The core checks a proof against a statement, so a reader must still check that the statement says what the paper claims.

The Steane code, built and checked in Lean

In the afternoon Jain reached error correction. Their library, QEC Lean, defines Pauli operators as strings of the letters I, X, Y and Z with a phase, and a code as a set of such strings that all commute. Jain built the Steane code, which stores one logical qubit in seven physical ones, writing out its six checks live. Distance was left unproved.

Distance, the smallest number of errors that can corrupt the stored qubit, is the hard part, since brute-force search fails past distance three, Jain said, though solvers that emit certificates Lean can check may help. Jain had also asked Claude to help find a new code. It offered shorter proofs for known ones. Nothing came of it that Jain wanted to publish.

Kyushu splits the state to model-check larger circuits

Qiming Li of Kyushu University presented the work, with Shangzhou Xia and Jianjun Zhao on the programme. Model checking tests whether a program meets a stated property, and for quantum programs the cost lies in the state itself, which doubles with each qubit. Li’s method keeps only the part of the state the property needs. It splits the qubits into small overlapping units.

A gate on one unit updates only that unit, while a gate across units merges them, applies the gate and splits them again. Li said the method beat full simulation in time and memory from about 13 or 14 qubits, matched all 34 tests and finished runs from 50 to 200 qubits. Random circuits are hard to split. Shor’s algorithm is hard or impossible, and every model was built by hand.

Fordham and Stevens model-check the VQE loop

A joint team from Fordham University and the Stevens Institute of Technology, with Tom Wray, Ying Wang and Juntao Chen among its authors, spoke in the closing session of the Q-Sec security workshop on the Sunday. Their subject was VQE, which passes measurements and new settings back and forth between a quantum and a classical computer. They treated that loop as a protocol to check.

The team reduced standard VQE code to a finite-state model and wrote 41 properties, covering the order of phases, fresh data in each round, stopping and the handling of failures. A model checker called NuXmv found all 41 satisfied. Planted faults, such as ignoring an error from the quantum machine, showed up as violations. The work does not check numerical accuracy, the optimiser or noise.

Taiwan sorts which circuit matches a quantum computer can find

Chih-Hsiang Chuang of National Taiwan University showed which kinds of matching between two black-box quantum circuits a quantum computer can settle efficiently. Five kinds remain open. The paper is by Chuang, Tian-Fu Chen, Jie-Hong Roland Jiang, Bo-Hung Chen and Dah-Wei Chiou. Boolean matching is a standard task in chip design. Two circuits are promised to be equal once their inputs or outputs are flipped with X gates, reordered, or both, and the task is to find those flips and orders with as few queries as possible.

With four options on each side there are 16 types. Flips on the input side alone can be found with a swap test, which compares two quantum states, and reordering the inputs alone also has an efficient quantum test. Types with changes on both sides inherit hardness from a known result on reversible circuits. The output side is harder to pin down, Chuang explained, because a black-box circuit can hide output changes from any fixed test input, and random inputs fail since random quantum states tend to look alike.

Records, seeds and the software stack

Reproducibility depends on far more than source code. Seven talks, four of them from Finland’s University of Jyväskylä, asked what has to be written down.

Jyväskylä logs a quantum reservoir in MLflow

Otso Kinanen of the University of Jyväskylä presented work with Valter Uotila and Vlad Stirbu. Their point was that tuning a quantum program by trial and error should rest on logged data, because results vary from run to run and the hardware keeps changing under the programmer’s feet. Kinanen used QProv. It is a 2021 checklist from the University of Stuttgart that lists what to record about a circuit, a machine and a run.

The test case was a quantum reservoir fed a standard chaotic signal, with 875 circuits on 5 simulated qubits, and every stage wrote its settings and results into MLflow. MLflow is a tracking tool built for machine learning. The tracking worked, within limits. Kinanen said no tracking tool built for quantum work exists yet, and one experiment reached 500 megabytes.

A tutorial on tracking every run

Vlad Stirbu opened a Monday tutorial on one question, how a given result was obtained, and said most projects answer it with records made up as they go along. Their test is whether, six months on, a result can be rerun, compared fairly and its change explained. Stirbu had seen that test fail. Colleagues’ notebooks broke because Qiskit had been installed without a fixed version.

Kinanen then ran a notebook in MLflow. Each run stores its settings, such as shot count, seed and backend, its scores, such as depth or fidelity, and its files, such as circuit text and snapshots of the chip’s state. A compiler test showed why the seed matters. A nine-qubit QFT kept about 300 two-qubit gates at the lowest setting and about 130 at the top two. Two of the compiler’s choices are random, so logging the seed turns an anecdote into a benchmark.

A reservoir run fails, and a log would have caught it

Valter Uotila, of the University of Helsinki and now TU Delft, forecast a chaotic series with a quantum reservoir that lets old input fade through a setting called gamma. A good run scored a test R2, a measure of fit where 1 is perfect, of 0.972. Uotila’s lesson came from a run that failed.

One run scored 0.704 against 0.970 for a classical rival. It had asked for four qubits where three were intended, and a hand-written key had pulled values from an older experiment into the new one. Uotila had not logged that project in MLflow. Uotila used the run to show what an unlogged project costs.

Venturus keeps a reservoir’s one win as a local result

Erico Teixeira of Venturus showed QRC-Lab, a set of open notebooks for quantum reservoir models, on behalf of its author, Anderson Santos. In such a model a fixed quantum system reshapes a time series, and only a classical readout is trained. Teixeira listed the usual faults in papers on quantum machine learning. Data sets go poorly described. Splits leak future data into the past, the prep work does the real job and the classical baselines are weak.

The worked example forecasts wind speed one to seven days ahead from a public daily climate series. Training, validation and test blocks stay in time order, so no future point trains a past one. The reservoir model beat its classical baselines once, and Teixeira kept that win as a local result with no claim of quantum advantage. The code is on GitHub. Next come a larger dataset and stronger baselines.

Jyväskylä scores drift in compiled circuits

Jonny Peltonen, a doctoral student at the University of Jyväskylä, worked with Otso Kinanen and Vlad Stirbu on a plain worry. The same source program does not run as written. It can change with the toolkit version, the target’s native gates, its wiring, the optimisation level and the random seed before it reaches any hardware.

Their drift score compares a compiled circuit’s depth and gate counts against a stored baseline, with a warning at 0.1 and a failure at 0.5. On a 12-qubit circuit, an unset seed left IonQ unchanged, since its qubits are all connected, but moved IQM to 0.160 and Rigetti to 0.140. Upgrading Qiskit from 2.1.2 to 2.4.2 raised a 10-qubit QFT’s depth from 213 to 253. The check ran automatically under GitHub Actions.

Jyväskylä maps the gravity wells in a software stack

Ronja Heikkinen of Jyväskylä worried about the choices a team makes before it runs anything at all, such as how to join a supercomputer to a quantum machine, because tools locked in that early limit what comes later. The work was with Vlad Stirbu and with Santiago Núñez-Corrales of NCSA. No single firm covers every layer well, Heikkinen said.

Their central idea is the gravity well, a densely connected part of a system that pulls other choices towards it. Heikkinen compared three ways to link supercomputers and quantum machines, two built on Kubernetes and one on the Slurm scheduler alone, and found wells in both camps. Heikkinen proposed three scores to measure them. For now, Heikkinen said, the framework is subjective.

Regensburg finds 11 of 31 shared packages will start

Nicole Hoess and Dominik Köster, both in Wolfgang Mauerer’s group at Regensburg, asked whether published quantum results can be rebuilt from the code their authors share. Hoess started from about 5,000 studies from the last five years and found 127 that were peer-reviewed experiments, of which 31 had a working link to their code. The programme also lists Maja Franz, Benjamin Zec and Ralf Ramsauer.

Only 11 of those 31 packages would start. Hoess put the failure rate at about 60 percent and found that only about a quarter of the wider set of studies shared any code at all, so most results could not even be tried. Köster told the room to pin every version, keep paths relative and let a workflow rebuild results on every push. Code sent only on request, Köster said, does not count.

The quantum software ecosystem, measured

Testing asks whether one program works. Five talks stepped back to measure the people, projects and methods behind such programs.

Castilla-La Mancha finds 82 percent of quantum projects rest on one person

A team from the Universidad de Castilla-La Mancha found that 82 percent of open-source quantum software projects on GitHub have a bus factor of one. That means a single person holds more than half the commits and pull requests. The study, ENTAGLE, began as one student’s final degree project, and the team calls it preliminary, though its snapshot covers more than 1,500 repositories and almost 3,000 developers over ten years. It is a first map.

The network splits cleanly into 752 groups, with a modularity of 0.96 on a scale from zero to one, which the team asked might mean silos. Some 2,387 people bridge them. Four big hubs, Qiskit, Cirq, PennyLane and Braket, raise questions of how much rests on a few firms. Private projects are missing, questioners noted, which may tilt the picture toward large companies. The team’s own card asked whether that means poor resilience.

Regensburg traces code moving between compiler projects

Vincent Gierisch and Nicole Hoess of the Technical University of Applied Sciences Regensburg found that almost no quantum compilation project stands alone. People who commit to more than one project tend to commit the most. The work is with Ralf Ramsauer and Wolfgang Mauerer, of Regensburg and Siemens. To see this, the team mined each project’s main-branch history, matched people across projects by name, email and GitHub handle, and built a graph that links each person to each project they touch.

Hoess traced one developer from QIRO, an academic compiler whose commits fall sharply after its preprint, to Catalyst and JeffMLIR, adding closely similar code to all three. Other routes run from XACC through QCOR to CUDA-Q. The team treats these links as clues that need checking in the code. Its request to researchers was the one this guide keeps meeting, to ship reusable code and a reproducibility package.

Lorenz says a slow workflow wipes out a quantum gain

Jeanette Miriam Lorenz of Fraunhofer IKS and LMU Munich argued that a quantum advantage in one subroutine is lost if the classical workflow around it runs badly. Lorenz treats the quantum computer as one subroutine in a much larger classical job. Which benchmark matters depends on who is asking, from Shor factoring and Q-Score down to coherence times, and for an application the worst part sets the final value.

Lorenz expected developers soon to program without thinking about gates and wiring, and pointed to a decision tree from the group that recommends tools for the upper software stack with no code. A second piece of work adds the sampling overhead for a given algorithm. Its plot of shots against problem size shows a boundary climbing past 10 to the 42, beyond which the group marks a region of quantum disadvantage.

QuARC keeps rules in charge and lets a language model fill in detail

Ricardo Pérez-Castillo of the University of Castilla-La Mancha proposed QuARC, which lets fixed rules set a program’s shape and lets a language model fill in only the detail. Five co-authors joined the work. QuARC starts from goals written in the i-star notation. It moves to a model of variants in UVL, then to a UML design with a quantum profile, and last of all to code.

The running example, Q-TradeX, compares classical and quantum machine learning on Bitcoin and Tether prices. Pérez-Castillo called it an early proposal. Three problems are open. The models must be kept in step, the rules must adapt to new technologies, and language models are hard to make give the same answer twice. The route back from edited code to the model is not yet built.

A VQE program turns out 75 percent classical by function points

Samuel Richard, an undergraduate at the École de technologie supérieure in Montréal, measured an example VQE program at 91 COSMIC function points, of which 68 are classical and 23 quantum. COSMIC, the ISO 19761 standard for functional size, counts one point for each movement of data into or out of a function. So nearly 75 percent of the program’s functionality is classical. The work is with Tuna Hacaloglu, Hassan Soubra of ECE Lyon, Pierre Bourque and Alain Abran.

A function counts as quantum when its requirements can only be stated in quantum terms, such as states or circuits, and as classical otherwise, so each firm draws that line for itself. Richard warned that this is one program solving one problem. Size will turn into staffing only with logged data on build times.

What it means for quantum software testing

The practical fix costs almost nothing. Anyone who funds, reviews or publishes quantum software work can ask for pinned versions, logged seeds, raw shots and a script that rebuilds every figure, as Dominik Köster urged on Friday. Treat code sent only on request as unpublished. Ask who maintains it, too, since a Castilla-La Mancha study found 82 percent of open-source quantum projects resting on one person. The quantum software testing tools in this guide will mature faster in a field that keeps such records.

None of this makes clever oracles less useful. Fujitsu’s symmetry checks, Montréal’s noise cleaner and Porto’s cheaper mutants will all do the most good once the runs they judge can be repeated by someone else on another day. For another approach, see our report on checking quantum programs while they run.

Works on my machine was a programmer’s excuse long before anyone built a QPU. Until the records improve, too many quantum results will work only on their authors’ machines.