While Schrodinger’s cat gets all the attention, few people even notice his growing bug collection. Consider this: quantum control systems, compilers, languages, frameworks, and user interfaces are all written in classical computer code. Have you even encountered bug-free computer code? No, you have not.
“Bugs in Quantum Computing Platforms: An Empirical Study” by Matteo Paltenghi and Michael Pradel from the University of Stuttgart, Germany, reveals the discovery of 223 bugs across 18 open-source quantum computing platforms, including the most popular ones, such as Qiskit, Cirq, and Q#. Of these 223 bugs, 92 resulted in easy-to-detect crashes. Alarmingly, however, another 89 of these bugs — roughly 40% — were quantum-specific, and they manifested themselves in the worst possible way: in the results. This means that everything looks like it’s working, but if you double-check the results, they’re wrong.
I’ve personally encountered some obvious bugs: APIs that didn’t work, libraries renaming/relocating their functions, and those sorts of things. For example, a while back I tried to use Qiskit’s portfolio optimization module. The first problem I encountered was that none of the APIs for accessing data worked. Each API had to be troubleshooted with their respective providers; at least one was outright discontinued. Once the data finally started flowing, the sample code had to be troubleshooted. There were obvious errors throughout and the module turned out to be more of a suggested starting point for an algorithm than an actual ready-to-use solution.
This “Bugs” paper makes me wonder deeply about the quiet errors and their effects, though, perhaps on the accuracy of QASM editors and transpilers. They’re written in classical code. Does every algorithm really get executed as designed? Are bad results always the result of bad circuits, or do novel circuits get inadvertently transpiled into bad circuits? For that matter, even with good circuits, how perfect/imperfect is the control software actually executing those jobs?
Furthermore, how about Variational Quantum Algorithms? Also known as hybrid classical-quantum algorithms, execution on real devices and quantum computing simulators are just a fraction of what these algorithms do. Think about all the code that goes into classical pre-processing and post-processing. You’re not just calling your own functions, you’re calling functions from large libraries with open issues on GitHub.
Referencing this “Bugs” paper and continuing this research, “A Comprehensive Study of Bug Fixes in Quantum Programs” by Junjie Luo, Pengzhan Zhao, Zhongtao Miao, Shuhan Lan, and Jianjun Zhao from Kyushu University, Japan, reveals the discovery of 96 bugs with 77 of them — roughly 80% — being quantum-specific. This means that you need considerable domain-specific knowledge to even identify these bugs. It’s also worth noting that the authors searched for bugs in different areas than the “Bugs” authors, so these might be 96 on top of the previous 223, and not duplicative.
Bugs in code surprise no computer scientist. However, it’s easy to become complacent when code generates no errors and the results usually look the way you’d expect them to. But, does that really mean that everything is executing perfectly? Of all the places bugs could be hiding between your laptop screen and your qubits, have they all been sprayed?
With NISQ qubits, we’re expecting to see a lot of noise anyway. We all know our qubits need error-correction, but now we can be mindful that all the classical code leading to and from our qubits needs error correction, as well.
