Automated Bug Classification Improves Quantum Software Quality and Reliability.

An automated rule-based framework accurately classifies software bugs in Qiskit repositories, achieving up to 85.21% accuracy. Analysis of 12,910 issues reveals classical bugs dominate (67.2%), with circuit-level problems prevalent among the 27.3% specific to the framework. Severity classification requires further refinement.

The increasing complexity of quantum software necessitates robust methods for identifying and categorising defects, a challenge addressed by a novel automated framework detailed in recent research. Accurate bug classification is paramount to software quality, yet manual approaches are time-consuming and prone to inconsistency. Researchers at the National Institute of Technology Srinagar, Mir Mohammad Yousuf and Shabir Ahmad Sofi, present a rule-based system designed to classify software issues by type, category, severity, and impact on quality attributes, with a specific focus on defects inherent in quantum systems. Their work, entitled ‘Bug Classification in Quantum Software: A Rule-Based Framework and Its Evaluation’, details the development and assessment of this framework using a substantial dataset of over 12,900 issues sourced from 36 Qiskit repositories, achieving accuracy levels up to 85.21% and demonstrating substantial agreement with manual classifications.
A functional, rule-based framework successfully classifies software bugs within the quantum computing domain, utilising the Qiskit ecosystem as a primary case study. Researchers achieve up to 85.21% accuracy on a dataset of nearly 13,000 issues by employing keyword and heuristic-based techniques to determine bug type, category, severity, and the impacted quality attributes. Statistical analysis, including paired t-tests and Cohen’s Kappa, demonstrates substantial to almost perfect agreement between automated classifications and manually verified data for most categories, validating the framework’s reliability and establishing a foundation for automated bug triage and quality assurance.

The developed system categorises issues reported in software repositories by analysing keywords and applying heuristic techniques to discern patterns and assign classifications. Validation against a manually classified dataset of 4,984 issues, drawn from a larger collection of 12,910 issues across 36 Qiskit repositories, demonstrates high accuracy, reaching 85.21%. F1-scores, a measure of precision and recall, range from 0.7075 for severity to 0.8393 for quality attribute classification, indicating the system’s effectiveness. Statistical analysis utilising paired t-tests and Cohen’s Kappa confirms substantial to almost perfect agreement between automated classifications and ground truth data for bug type (κ = 0.696), category (κ = 0.826), quality attribute (κ = 0.818), and quantum-specific bug type (κ = 0.712). Cohen’s Kappa measures inter-rater reliability, with values closer to 1 indicating stronger agreement.

Large-scale analysis reveals that classical software bugs constitute the majority of reported issues (67.2%), while quantum-specific bugs account for 27.3%, highlighting the continued relevance of traditional software engineering principles. Compatibility, functional, and quantum-specific defects represent the most frequent bug categories, indicating areas requiring focused attention. Usability, maintainability, and interoperability emerge as the most impacted quality attributes, emphasising the need for improved software design and engineering practices. A detailed examination of 1,550 quantum-specific bugs identifies circuit-level problems as the most prevalent, followed by gate errors and hardware-related issues, providing valuable insights for targeted debugging and optimisation.

This research builds upon existing work in software bug classification, issue tracker analysis, and hybrid rule-based/machine learning systems, bridging the gap between general software engineering and the unique challenges of ensuring reliability in quantum computing platforms. Researchers meticulously analysed reported bugs, identifying key keywords and patterns to distinguish different bug types and severity levels, enabling the creation of a robust classification framework. Data from issue trackers informed the development of effective classification rules, ensuring the framework’s practicality and relevance.

While the framework demonstrates strong overall performance, severity assessment remains a challenge, exhibiting only slight agreement (κ = 0.162). Researchers acknowledge the difficulties in accurately assessing bug severity, as it often requires a deeper understanding of the bug’s impact on system functionality and user experience, which is difficult to capture through automated analysis alone. Future work will focus on incorporating more sophisticated techniques, such as natural language processing and machine learning, to improve the accuracy of severity assessment and provide more nuanced classifications, involving training models on larger datasets and incorporating contextual information.

The framework’s ability to accurately classify these bugs represents a step towards improving the reliability and quality of quantum computing applications, fostering greater confidence in the technology. This work establishes a foundation for automated bug triage and quality assurance within the rapidly evolving field of quantum software engineering, paving the way for more efficient and reliable software development processes.