MIRAGE Remaps Model Parameters to Accelerate Large Language Model Inference

Large language models demand ever-increasing computational resources, and a key bottleneck lies in accelerating their inference speed without exhausting available memory, a challenge Ruihao Li, Shagnik Pal, and Vineeth Narayan Pullu, from The University of Texas at Austin, and their colleagues address in new research. The team introduces MIRAGE, a novel approach to optimising key-value (KV) cache, a technique used to speed up language model responses, by cleverly repurposing memory normally allocated to model parameters. Unlike the constantly changing KV cache, model parameters remain static during operation, allowing MIRAGE to dynamically reallocate this memory for KV cache storage and avoid slow data transfers between the computer’s main memory and processing units. This parameter remapping proves particularly effective in multi-tenant environments, where memory used by inactive models can be reclaimed, and experiments demonstrate substantial performance gains, including significant reductions in response time and increased throughput, compared to existing state-of-the-art solutions.

LLM Inference Bottlenecks and KV Cache Limits

Large Language Models (LLMs) are rapidly increasing in size, creating a significant bottleneck for efficient operation due to escalating memory demands. As models grow, the need for memory often exceeds the available resources on modern graphics processing units (GPUs), hindering their ability to process information quickly and efficiently. A key technique to improve performance, known as KV caching, stores previously computed information to avoid redundant calculations, but this also consumes substantial memory. When the KV cache reaches the GPU’s limit, the system must recompute data, significantly increasing processing time and reducing performance.

Current approaches often involve extending GPU memory by utilizing CPU memory, a process called CPU-offloading. This allows for a larger effective memory capacity, but introduces latency due to the time required to transfer data between the CPU and GPU. Frequent data swapping between these two components creates synchronization overhead, hindering overall system throughput and increasing response times. Recent advancements in hardware, such as the Grace Hopper superchip, have dramatically increased the bandwidth between CPUs and GPUs, offering a potential solution to this challenge. However, simply increasing bandwidth isn’t enough; the limited processing capabilities of CPUs can become a new bottleneck.

Researchers have begun exploring methods to leverage this increased bandwidth more effectively, such as offloading future layers of the KV cache to CPU memory and swapping them back as needed. Despite these improvements, the constant need to synchronize data during swapping continues to limit performance. Instead of swapping, a new approach focuses on repurposing existing GPU memory. This involves dynamically remapping memory originally allocated to the model’s parameters, the core components that define the model’s knowledge, to expand the KV cache. Because model parameters remain constant during operation, this remapping process introduces minimal synchronization overhead, offering a significant advantage over traditional swapping techniques.

This is particularly beneficial in multi-tenant environments, where memory allocated to inactive models can be proactively reused to support active models, maximizing resource utilization and improving overall system efficiency. To realize this potential, researchers have developed systems like MIRAGE, a dynamic remapping engine that automatically adjusts KV cache size at a layer-by-layer granularity based on real-time demands. This adaptive approach allows for flexible integration into any LLM inference serving system and enables efficient resource management, ultimately reducing latency and improving throughput for large language model applications.

Repurposing Model Memory for Faster Inference

To accelerate large language model (LLM) inference, researchers developed a novel approach called MIRAGE, addressing the computational bottleneck of managing the KV cache, the memory storing past interactions crucial for generating text. MIRAGE circumvents the issue of data swapping by cleverly repurposing memory originally allocated to the model’s parameters, the core components defining its knowledge, for use as KV cache. The methodology leverages the high-speed connection between CPUs and GPUs found in modern hardware, such as the Grace Hopper Superchip, to ensure efficient data transfer and minimize latency. The team rigorously tested MIRAGE in various configurations, including scenarios with shared and isolated GPUs, and under different workloads.

Results consistently showed substantial improvements in tail latency, the delay experienced by the slowest requests, and overall throughput, the rate at which requests are processed. Compared to methods that rely on KV cache swapping, MIRAGE achieves comparable performance in initial response times but dramatically reduces the delay for subsequent tokens and increases the overall processing rate. Furthermore, MIRAGE offers greater flexibility in dynamic resource allocation, allowing memory to be efficiently reclaimed from inactive models and assigned to the KV cache of active models. This adaptability is a key advantage over traditional swapping methods, which are limited by their inability to reclaim memory from models not currently in use. The methodology’s effectiveness stems from its ability to maximize CPU-GPU bandwidth through unidirectional data transfer, streamlining the process and minimizing bottlenecks.

Dynamic GPU Memory Reallocation for LLMs

Large Language Models (LLMs) are rapidly growing in size, creating a significant challenge for efficient operation as memory demands quickly outpace available GPU capacity. A key technique to improve performance, known as KV caching, stores previously computed information to avoid redundant calculations, but this also increases memory requirements. Current approaches often rely on extending GPU memory with CPU memory, a process called CPU-offloading, but frequent data transfers between the two create bottlenecks and increase latency. Researchers have now developed a new system, MIRAGE, that avoids this constant swapping by intelligently repurposing existing GPU memory.

Instead of moving data back and forth, MIRAGE dynamically reallocates memory originally assigned to the model’s parameters, which remain constant during operation, to expand the KV cache. This approach significantly reduces synchronization overhead, as it involves a one-time, unidirectional data transfer, unlike the continuous back-and-forth of traditional CPU-offloading. The results demonstrate substantial performance gains compared to state-of-the-art systems like vLLM. MIRAGE achieves reductions of 44.8% to 82.5% in tail time-between-token latency, and 20.7% to 99.3% in tail time-to-first-token latency.

Furthermore, the system delivers 6.6% to 86.7% higher throughput, meaning it can process more requests simultaneously. This approach is particularly beneficial in multi-tenant environments, where multiple LLMs share the same hardware. MIRAGE can proactively reclaim memory allocated to inactive models and repurpose it as KV cache for active models, maximizing resource utilization. The system intelligently manages this reallocation at a layer-by-layer granularity, adapting to changing demands and ensuring optimal performance even under varying workloads. By eliminating the need for constant data swapping, MIRAGE offers a significant advancement in LLM inference serving, paving the way for faster, more efficient, and more scalable AI applications.