Multiverse Computing tests quantum neural networks in a 30B LLM

Researchers at Multiverse Computing and the University of Toronto have demonstrated a multi-head quantum neural network (QNN) capable of replacing transformer blocks within a 30-billion-parameter large language model. The work shows a single QNN, containing 145,281 trainable parameters, can displace 16.33% of the model’s parameters while maintaining accuracy. This approach uses noisy intermediate-scale quantum (NISQ) devices and angle encoding to overcome limitations in scaling quantum computations for large models. The team successfully ran these quantum modules on a 156-qubit IBM Heron processor, showing compatibility with existing hardware.

Multi-head Quantum Neural Networks for LLM Integration

Encoding 30-billion-parameter language model activation vectors into quantum states is enabled by the architecture of a multi-head quantum neural network, potentially offering a pathway to more efficient large language model processing. Researchers demonstrated this by integrating the network into Qwen3-30B-A3B-Instruct-2507, a 48-layer mixture-of-experts model with a hidden dimension of 2048. This integration focused on replacing standard transformer blocks, which consist of self-attention and a sparse MoE feed-forward network, with the quantum modules, all while maintaining residual connections.

The resulting system aims to process information using multilinear trigonometric polynomials of encoded features, each with a learned frequency. The design addresses a critical challenge in quantum machine learning: efficiently encoding data into a quantum state. Amplitude encoding, while qubit-efficient, requires a number of qubits proportional to the logarithm of the input dimension, creating scaling issues for production-scale LLMs. Similarly, angle encoding faces limitations in qubit count.

Instead, the multi-head QNN partitions the hidden space into k disjoint heads, encoding each into its own $n_q$-qubit register, and processes these heads in parallel. These parallel processes are then coupled through constrained classical linear layers, termed “mixers,” to facilitate communication between the quantum heads. Successful execution of the quantum heads was demonstrated on a 156-qubit IBM Heron processor for the initial prompt token, and then for N=8 at every decoding step of two 15-token generations.

The remainder of the model’s processing occurred on GPUs, highlighting a hybrid quantum-classical approach. The team’s experiments showed that a single multi-head QNN could replace a span of eight transformer blocks, displacing 16.33% of the model’s parameters without significant loss of accuracy.

The researchers detail that the total module is defined as where Θ collects all trainable parameters. The work builds on previous efforts in quantum-assisted LLMs, including fine-tuning, distillation and natural-language generation, as well as proposals for quantum self-attention layers in smaller models. Complete data and protocols are available in the appendices, allowing for independent verification and further exploration of this approach to quantum-enhanced LLMs.

Qwen3-30B-A3B-Instruct-2507 as a Testbed for Quantum Modules

The Qwen3-30B-A3B-Instruct-2507 large language model served as the platform for testing a new approach to integrating quantum processing into existing artificial intelligence infrastructure. This modular design aims to circumvent limitations in scaling quantum encoding methods for large-scale language models, specifically addressing the challenges posed by both amplitude and angle encoding. The resulting module was initially trained using exact statevector simulation before being deployed on a 156-qubit IBM Heron processor. The approach uses the inherent redundancy within trained transformer blocks, a characteristic exploited in techniques like pruning and quantization.

The premise is that a significantly smaller quantum module can approximate the function of a displaced block without substantial loss of accuracy. Ablation studies revealed that the quantum heads, encompassing the circuits and learned readout layers, are primarily responsible for recovering the displaced function when replacing a single block. However, when multiple blocks are fused, the classical mixers alone can achieve performance within 1-3% of the full quantum module in terms of perplexity.

Researchers found that the width of the host model is absorbed by the number of heads, rather than the size of any single quantum register. This design choice addresses a key obstacle in quantum machine learning: encoding a model’s hidden neurons into a qubit register. The team’s work builds on the observation that a trained transformer tolerates the removal of entire blocks with minimal performance degradation, suggesting that lightweight modules can effectively substitute for more complex layers.

Constrained Classical Mixers in Multi-head QNN Architecture

Partitioning the hidden layer of a large language model into disjoint heads is central to a new approach for integrating quantum computation, allowing for angle encoding into individual qubit registers and parallel processing of these heads. Experiments performed on the Qwen3-30B-A3B-Instruct-2507 model demonstrate that a single multi-head quantum neural network can effectively replace eight contiguous transformer blocks.

This substitution results in a displacement of 16.33% of the model’s parameters, suggesting a potential pathway for model compression through quantum integration. Ablations show that, for a single replaced block, the quantum heads (circuits followed by a learned readout), and not the classical mixers alone, recover the displaced function, whereas for fused spans the mixers alone come within 1-3% in perplexity.

145,281 Trainable Parameters Replacing Eight Transformer Blocks

33%. This substitution was performed as a test of integrating quantum computing into existing AI infrastructure at a substantial scale, demonstrating a potential pathway for future hybrid systems. The multi-head quantum neural network (QNN) achieved this parameter reduction while maintaining performance within approximately 1% of the original model’s perplexity on both WikiText-2 and C4 datasets, indicating minimal loss in accuracy. The team’s tests demonstrate the module’s ability to function within the existing framework of a 30-billion-parameter LLM, a significant step toward practical quantum-enhanced language processing.

The study concludes that a quantum module can be integrated into a production-scale language model, trained to emulate the function of transformer blocks, and evaluated against the original model, all while being executed on currently available hardware. The team’s findings suggest that this approach could be a viable strategy for incorporating quantum computing into the field of large language models, potentially clearing the way for future advancements in AI.

NISQ-Scale Execution on a 156-Qubit IBM Heron Processor

The team executed the quantum heads of five fusion checkpoints on a 156-qubit IBM Heron processor, to verify compilation and runtime behavior. The circuits, as compiled, fit onto the 156-qubit processor at a two-qubit depth of 13 with no routing swaps required, a critical factor for near-term quantum applications. Each execution processed 14 heads simultaneously using 112 qubits, indicating efficient resource use within the constraints of the NISQ architecture.

While larger circuits would exceed the capacity for classical simulation, the researchers acknowledge that trainability may degrade as circuit complexity increases, potentially limiting the scalability of this method. They note that structural restrictions designed to improve trainability could also preserve classical simulability, creating a trade-off between quantum advantage and computational feasibility.

Analysis of the learned readout revealed its importance; removing the quantum heads at a sensitive depth resulted in a 3.5-perplexity point cost, though a product-state or frozen circuit combined with the learned readout recovered at least that much ground. The ability to compile these trained circuits onto a 156-qubit Heron processor represents a step toward realizing practical quantum-enhanced language models, even within the limitations of NISQ-era devices.

The team’s work suggests that partitioning the model’s hidden space into multiple heads, rather than scaling a single quantum register, offers a viable strategy for managing qubit requirements and enabling execution on near-term quantum hardware.

Parameter Efficiency of Quantum Circuits vs. Classical Layers

The ability to achieve comparable performance to classical layers with fewer parameters is now demonstrated within a 30-billion-parameter language model. This parameter efficiency stems from the way information is stored; a circuit layer on n_q qubits stores O(n_q) parameters while operating on a space. This means a quantum circuit can potentially deliver the function of a transformer block with significantly fewer stored weights than its classical counterpart.

The researchers emphasize this is not a claim of quantum advantage, but rather a demonstration that NISQ-scale circuits can perform a portion of a production model’s function. The approach differs from previous quantum language model work, which either adapted classical layers for quantum execution or added quantum components to smaller models.

More like this

Quantum AlgorithmsSymmetry in quantum neural networks boosts accuracy on image tasks

Quantum Machine LearningResearchers map qubit states to probability distributions in phase space

High Performance ComputingGerman funds QUDORA’s quantum machine learning project for energy

Quantum Machine LearningQuantum Data Proposes DYRK1A Compounds Differing from Tabular Models