Augserve: Adaptive Request Scheduling Achieves 33.1x Throughput, Improves LLM Inference Serving with 96.3% SLO Compliance

The increasing popularity of augmented large language models (LLMs) in modern web applications demands significant improvements in inference serving efficiency, a challenge that Ying Wang, Zhen Jin, and colleagues from Zhejiang University and Alibaba Group now address with their innovative framework, AugServe. Existing systems often struggle with delays caused by inefficient request scheduling and inflexible batch processing, limiting their ability to handle requests within acceptable timeframes. AugServe overcomes these limitations through a two-stage adaptive request scheduling strategy, intelligently prioritising requests based on their characteristics and current system capabilities. This approach, combined with dynamic adjustment of token batching, demonstrably enhances throughput, achieving up to 33. 1times greater performance than existing systems like vLLM and InferCept, while also significantly reducing initial response times.

Service-level objectives (SLOs) are critical for enhancing user experience. To achieve this, inference systems must maximize request handling within latency constraints, referred to as increasing effective throughput. However, existing systems face two major challenges: reliance on first-come, first-served (FCFS) scheduling causes severe head-of-line blocking, leading to queuing delays exceeding the SLOs for many requests, and a static batch token limit, which fails to adapt to fluctuating loads and hardware conditions. Both of these factors degrade effective throughput and service quality. This research presents AugServe, an efficient inference framework designed to reduce queuing latency.

AugServe, a High-Throughput LLM Inference System

The document presents a comprehensive evaluation of a new inference system called AugServe, designed to improve the performance of Large Language Models (LLMs). Experiments consistently demonstrate that AugServe outperforms existing systems like vLLM and InferCept in terms of throughput, latency, resource utilization, and overall efficiency. AugServe achieves this through a combination of dynamic scheduling, batch processing, and intelligent context handling. The research team focused on addressing limitations in current systems that hinder performance and scalability. AugServe consistently achieves higher throughput and lower latency compared to vLLM and InferCept across various models and hardware configurations.

It also maintains its performance advantages as the load increases, demonstrating better scalability. Furthermore, AugServe utilizes computational resources more efficiently, leading to lower costs and improved sustainability. The system maintains low tail latency even under high load, providing a better user experience, and optimizes time-to-first-token while maintaining excellent performance.

AugServe Dramatically Boosts LLM Service Throughput

AugServe, a novel inference framework, demonstrably enhances the efficiency of augmented Large Language Model (LLM) services, addressing critical limitations in request handling and throughput. The research team developed a two-stage adaptive request scheduling strategy and a dynamic token batching mechanism to minimize queuing delays and maximize the number of completed requests within specified latency constraints. Experiments reveal that AugServe achieves substantial performance gains compared to existing systems, vLLM and InferCept. Specifically, the team measured effective throughput improvements ranging from 4.

7 to 33. 1times higher with AugServe, and 3. 3 to 13. 2times higher than InferCept. These gains were achieved by intelligently prioritizing requests and adapting to fluctuating workloads.

Furthermore, AugServe significantly reduced time-to-first-token, decreasing it by up to 96. 3% compared to vLLM and 95. 0% compared to InferCept. These results demonstrate a substantial reduction in the delay experienced by users. The research identified that conventional first-come, first-served scheduling causes significant head-of-line blocking, leading to excessive queuing delays.

AugServe’s adaptive scheduling strategy addresses this by considering the characteristics of each request, particularly the duration of external calls, and dynamically adjusting the processing order. Additionally, the team discovered that static batch token limits hinder performance, as they fail to accommodate varying request lengths and system loads. AugServe’s dynamic token batching mechanism resolves this by adjusting the batch size based on real-time conditions, optimizing resource utilization and throughput. Measurements show that 50% of requests processed by InferCept at a load of 4. 0 requests per second experienced queuing delays exceeding 343 seconds, far exceeding typical service level objectives. AugServe effectively mitigates these delays, delivering a more responsive and efficient user experience.

AugServe Boosts LLM Inference Throughput Adaptively

This research presents AugServe, a novel framework designed to significantly improve the efficiency of serving augmented large language model inferences. The team addressed critical limitations in existing systems, namely substantial queuing delays and inflexible batch processing, by introducing an adaptive request scheduling strategy and dynamic token batching. AugServe employs a two-stage scheduling process that considers request characteristics and system capabilities, continuously refining decisions to optimise performance. Furthermore, the system dynamically adjusts the number of tokens processed in each batch, responding to workload variations and hardware status.

Experimental results demonstrate that AugServe achieves substantial gains in effective throughput, exceeding the performance of current state-of-the-art systems by a factor of 3. 3 to 13. 2. The framework also reduces time-to-first-token, delivering responses more quickly to users.