Hybrid Quantum and Classical Workload Management with Graph-based Scheduling

Researchers at Lawrence Livermore National Laboratory have developed Fluence, a new Kubernetes scheduler plugin that reduces the cost of utilizing quantum computing resources. By intelligently pinning workloads to the cheapest or shortest-queue device available on AWS Braket, Fluence achieves a roughly 70-fold reduction in mean per-run cost. This advancement addresses a key challenge in hybrid quantum-classical workflows: coordinating tasks between classical and remote quantum processors introduces significant delays. Fluence incorporates a synchronization primitive that reduces worker idle time by approximately five times under short queues, and by several orders of magnitude when device queues extend to hours, enabling faster results and more efficient resource allocation.

A roughly 70-fold reduction in mean per-run cost represents a significant improvement in accessibility, made possible through innovations in workload management. Researchers have developed Fluence, a Kubernetes scheduler plugin underpinned by the Fluxion graph-based scheduler, designed to intelligently orchestrate hybrid quantum-classical workflows. A core challenge addressed by Fluence is what they term the two-queue problem, the synchronization difficulty arising from remote quantum devices introducing an external queue alongside the traditional scheduler’s own. This is not merely about speed, but about reclaiming valuable computing time otherwise lost to waiting. Fluence’s atomic gang placement largely eliminates wasted node-time under node contention, contrasting with the inefficiencies of a default scheduler. This capability is crucial because it ensures resources are fully utilized, preventing partially placed gangs from idling. The researchers state that quantum-awareness can be added to a cloud-native scheduler without modifying user containers, streamlining integration and maximizing resource efficiency for complex scientific applications.

The integration of quantum processing units into existing high performance computing infrastructure is rapidly evolving, shifting from experimental setups to practical applications tackling previously intractable optimization problems. However, this expansion introduces significant orchestration challenges, particularly concerning the management of remote quantum devices and the resulting synchronization issues. Researchers are now addressing the two-queue problem, the need to coordinate scheduling across both conventional cluster queues and the external queues of quantum service providers. This improvement stems from gating classical workers until the quantum task is nearing completion, reclaiming valuable resources otherwise lost to queuing delays. Beyond synchronization, Fluence also optimizes cost and efficiency. Fluence’s atomic gang placement largely eliminates wasted node-time under node contention, a common issue with default schedulers. The researchers note that whether the quantum devices are local, remote, or owned by an institution is irrelevant, but owned devices are often more directly under control and thus scheduling transparency is improved, highlighting the adaptability of their approach.

The Two-Queue Problem in Quantum Resource Management

Vanessa Sochat and Daniel Milroy at Lawrence Livermore National Laboratory are tackling a fundamental challenge in the growing field of hybrid quantum-classical computing: efficiently managing workloads that span both conventional and quantum resources. Their work centers on a problem arising from the need to schedule tasks across a traditional cluster queue and a separate, external queue for remote quantum processing units. This synchronization issue is particularly acute when accessing quantum devices via services like Amazon Braket or IBM Quantum Cloud. They’ve developed a synchronization primitive for the two-queue problem where a single producer submits a shared quantum task while consumers are held scheduling-gated, demonstrably reducing worker idle time by approximately five times under short device queues and by several orders of magnitude when a real device queue stretched to hours.

This is not simply about faster execution; it’s about preventing valuable classical computing resources from sitting unused while awaiting quantum task completion. Fluence enables quantum-awareness to be added to a cloud-native scheduler without requiring modifications to user containers, streamlining the adoption of quantum resources within existing HPC environments.

Beyond simply accessing quantum hardware, a new approach to workload management is actively minimizing computational waste through intelligent resource allocation. This is not about accelerating calculations; it’s about ensuring that allocated computing resources aren’t left idle while waiting for dependent quantum tasks to complete. The core of this efficiency lies in how Fluence handles “gangs” of tasks, groups of jobs intended to run concurrently. Unlike a default scheduler which may partially place these gangs, leading to underutilized nodes, Fluence’s atomic gang placement largely eliminates the wasted node-time that a default scheduler accrues by partially placing gangs. This is particularly impactful when classical computations are stalled awaiting results from a remote quantum processing unit. The researchers report cutting mean per-run cost by roughly 70-fold and time-to-result from hours to under a minute, demonstrating a substantial improvement in both efficiency and affordability for hybrid quantum-classical workflows.

Beyond simply accessing quantum hardware, efficient orchestration is proving critical to realizing the potential of hybrid quantum-classical computing. Researchers are now focusing on minimizing wasted resources, a challenge particularly acute when integrating remote quantum processing units (QPUs) into existing high-performance computing (HPC) infrastructure. The team behind Fluence, a Kubernetes scheduler plugin, identified a significant source of inefficiency they term the two-queue problem, where classical computing resources remain idle while awaiting the completion of tasks submitted to an external quantum queue. The impact on cost is substantial.

Fluence, a Kubernetes scheduler plugin, demonstrably minimizes the financial and temporal costs associated with hybrid quantum-classical computation by intelligently selecting the optimal backend for each workload. Beyond simply utilizing available quantum devices, the system actively seeks the cheapest or shortest-queue AWS Braket device, a strategy that yields substantial savings. This is not merely about accelerating research; it’s about maximizing the return on investment for increasingly expensive quantum resources. The core of Fluence’s efficiency lies in its ability to address a unique challenge introduced by remote quantum processing units. The system implements a mechanism that largely eliminates the wasted node-time that a default scheduler accrues by partially placing gangs. It also introduces a synchronization primitive for the two-queue problem, reducing worker idle time by approximately five times under short device queues and by several orders of magnitude when a real device queue stretched to hours. Cost- and queue-aware backend selection cuts mean per-run cost by roughly 70-fold and time-to-result from hours to under a minute. Together, these results show that quantum-awareness can be added to a cloud-native scheduler without modifying user containers.

The expanding landscape of high-performance computing now routinely incorporates quantum processing units, yet integrating these QPUs into existing cloud-native workflow managers presents unique challenges. A primary hurdle lies in the two-queue problem arising from remote quantum devices accessed through application programming interfaces like those offered by Amazon Braket and IBM Quantum Cloud. Unlike traditional HPC resources, these services introduce an external, vendor-owned queue alongside the scheduler’s own, creating synchronization difficulties. Quantum-awareness can be added to a cloud-native scheduler without modifying user containers. A key innovation is a synchronization primitive designed to minimize worker idle time. Beyond queue management, Fluence also optimizes cost and resource allocation.