Quantum-inspired Reinforcement Learning with PEPS Enhances Coherence in LLM Reasoning Traces

Large Language Models frequently falter when tasked with complex, multi-step reasoning, often producing illogical or inconsistent outputs. To address this critical limitation, Venkat Margapuri, Garik Kazanjian, and Naren Kosaraju from Villanova University present a novel reinforcement learning technique inspired by the principles of quantum physics. Their method incorporates a reward system based on Projected Pair States, which assesses the structural consistency of the model’s reasoning process, guiding it towards more coherent conclusions. This approach, unlike existing methods relying on direct instruction or comparison, fosters global coherence within the generated reasoning traces and demonstrably improves performance on challenging datasets spanning arithmetic, intuitive, and logical reasoning tasks.

Specifically, they utilize quantum-inspired tensor network techniques to enhance LLMs’ capacity for complex reasoning tasks, such as mathematical problem-solving and natural language inference. This work aims to improve the ability of LLMs to not only provide answers, but also to explain their reasoning process. The research demonstrates that quantum-inspired tensor networks allow for efficient adaptation of LLMs with reduced computational cost, improving performance on reasoning tasks.

The team demonstrates improved reasoning performance on benchmarks including mathematical problem-solving, natural language inference, and tasks requiring explanation generation, positioning these techniques as a potentially superior alternative to traditional fine-tuning methods. The methodology involves using reinforcement learning to train LLMs to optimize for specific reasoning objectives, employing techniques like Direct Preference Optimization and Constitutional AI. Quantum-inspired tensor networks serve as the core technical contribution, representing and manipulating the parameters of the LLM during fine-tuning. This research demonstrates that leveraging quantum computing principles can improve LLM performance, offering a practical application of quantum-inspired techniques in natural language processing.

Reasoning Trace Fidelity Using Tensor Networks

Scientists have developed a novel approach to enhance the coherence of multi-step reasoning in large language models by drawing inspiration from quantum physics. By contracting this tensor network, the team computes a fidelity score, quantifying the global consistency of the reasoning trace. Specifically, Proximal Policy Optimization was selected as the policy optimization algorithm due to its robustness and effectiveness in fine-tuning large language models.

PPO enables the model to iteratively refine its generation policy based on the PEPS-guided feedback, represented by the fidelity score, while maintaining training stability. This allows for direct optimization of logical consistency beyond traditional token-level objectives. The team introduced a reasoning-aware evaluation framework combining structural metrics with semantic similarity measures, assessing both the internal coherence and the validity of the generated reasoning traces.

Quantum-Inspired Reasoning Improves Language Model Coherence

Scientists have developed a novel approach to enhance the coherence of multi-step reasoning in large language models (LLMs) by drawing inspiration from quantum physics. The work introduces a method that utilizes Projected Entangled Pair States (PEPS), a concept from quantum many-body physics, to model reasoning traces as structured tensor networks, allowing the system to capture relationships between individual reasoning steps. The PEPS model computes a fidelity score, reflecting the global consistency of the reasoning trace, which serves as a reward signal for the Proximal Policy Optimization (PPO) algorithm. This allows the LLM to adapt its generation policy, prioritizing logically sound arguments. Results show that the quantum-inspired approach outperforms supervised, contrastive, and pretrained baseline approaches, achieving improved coherence scoring through the use of trace-level fidelity. The team successfully trained a compact language model, TinyLLaMA-1. 1B, using this quantum-inspired method, achieving enhanced coherence in generated reasoning traces without requiring extensive computational resources. Future work will focus on exploring the potential of this approach with larger language models and investigating the transferability of the learned structural coherence to different reasoning tasks.