Vision encoding within multimodal AI can require hundreds of milliseconds, creating a bottleneck that impacts overall response times. To address this, NVIDIA researchers Jesse Gu, Ryan McCormick and Akshatha Kamath have detailed how separating the vision encoder stage from the prefill and decode phases, a technique called Encode-Prefill-Decode (EPD) disaggregation, can deliver significant performance gains, the company says.
Utilizing EPD disaggregation with NVIDIA Dynamo achieves up to 5x faster time to first token and 7x faster end-to-end response time, according to the company. This optimization is particularly effective for image-heavy prompts and quantized mixture-of-experts models, allowing for independently scalable stages.
Encode-Prefill-Decode Disaggregation Accelerates Multimodal AI Serving
Efficiently serving multimodal AI requests demands careful allocation of computational resources, and a new technique called Encode-Prefill-Decode (EPD) disaggregation offers substantial gains in speed when implemented with NVIDIA Dynamo. By routing both request types to a prefill worker but invoking a separate encoder pool only for multimodal requests needing embeddings, EPD allows text-only prompts to proceed immediately. Jesse Gu, a summer systems software engineering intern on the Dynamo team, explains that this separation is most effective for image-heavy prompts, short-to-medium outputs, and quantized mixture-of-experts (MoE) models.
Hardware configuration plays an important role in maximizing the benefits of EPD disaggregation; the vision encoder was in BF16 precision during testing, while NVFP4 accelerated LLM prefill and decode. This configuration allows the stages to batch, schedule, and scale independently.
Three encoder placement options exist: aggregated serving, co-located encoder and prefill/decode (PD) workers, and a fully disaggregated encoder utilizing a separate, lower-cost GPU tier. In a controlled environment, two NVIDIA RTX 6000D GPUs ran the encoder workers, while four NVIDIA GB200 GPUs ran the PD workers. This placement keeps the lighter encoder workload on the RTX GPUs and reserves the GB200 GPUs for the more compute- and memory-intensive LLM workload.
This analysis does not consider the disaggregated setup with homogenous GPUs as it always underperforms the colocated encoder setup. The effectiveness of EPD disaggregation is directly tied to the division of work between vision encoding, LLM prefill, and decode; it proves most valuable when vision encoding consumes a significant portion of the overall request processing time or becomes a throughput bottleneck.
Production workloads frequently combine text-only and multimodal requests, and the Dynamo framework supports scheduling these diverse request types efficiently with EPD. “Dedicated encoder workers can improve batching, memory efficiency, and overall throughput,” the researchers report, highlighting the framework’s ability to scale these stages independently.
Dynamo Framework Separates Encoding for Scalable Multimodal Inference
Dynamo, an open source inference framework from NVIDIA, implements a separation of tasks called encode-prefill-decode (EPD) disaggregation to improve the scalability of multimodal AI serving. This approach allows vision encoding, the initial processing of image or video data, to operate independently from the language model’s prefill and decode stages, a distinction that unlocks performance gains under specific conditions. The framework’s design enables each stage to batch, schedule, and scale resources without being constrained by the others, a departure from traditional aggregated serving models.
In a conventional setup, where all processing occurs on a single worker, text-only requests may wait behind multimodal requests even though they do not require vision encoding. Dynamo addresses this by routing multimodal requests to dedicated encoder workers, which produce vision embeddings, while prefill and decode operations are handled by separate PD workers.
This separation is facilitated by NVIDIA Inference Transfer Library (NIXL), enabling efficient communication between the encoder and PD workers, the firm reports. A comparison of aggregated and EPD serving illustrates how the separation allows text-only requests to bypass the vision encoding stage entirely.
The implementation of EPD serving within Dynamo involves separating the encoder and PD worker roles without fixing their hardware placement, allowing for dynamic resource allocation based on demand, NVIDIA reports. This flexibility is important for handling diverse workloads and maximizing the utilization of available resources. The researchers’ work highlights the importance of a nuanced approach to inference optimization, recognizing that a one-size-fits-all solution is rarely effective in the rapidly evolving landscape of AI model serving.
Colocated vs. Disaggregated Encoder Topologies with NVIDIA GPUs
The efficiency of multimodal AI serving hinges on how encoding tasks are distributed across available hardware; NVIDIA’s experiments with differing topologies reveal significant performance variations depending on cluster composition. Specifically, a colocated encoder setup generally outperforms disaggregated approaches when utilizing a homogeneous cluster of GPUs, as reserving dedicated hardware for vision encoding can lead to underutilization of resources.
This contrasts with a disaggregated encoder configuration, which becomes advantageous when a cluster incorporates a lower-cost GPU tier specifically suited for encoder workloads, while reserving more powerful GPUs for the computationally intensive language model prefill and decode stages. Dynamo, NVIDIA’s framework, facilitates the transfer of vision embeddings generated by the encoder tier to the processing of the language model using NIXL, achieving a peak data transfer rate of 20 Gbps in testing.
In a test environment, two NVIDIA RTX 6000D GPUs ran the encoder workers, while four NVIDIA GB200 GPUs ran the PD workers, by the company’s account. This configuration strategically assigns the lighter encoder workload to the RTX GPUs, maximizing the utilization of the GB200 GPUs for tasks demanding greater computational power and memory capacity.
The impact of mixed workloads, combining text-only and multimodal requests, further highlights the benefits of EPD disaggregation; isolating encoder work allows text requests to wait behind multimodal requests rather than being forced to wait while vision transformer (ViT) processing completes. Benchmarks using the Qwen3. 5 122B A10B NVFP4 model, run on a cluster of four GB200 GPUs alongside additional RTX 6000D GPUs in the disaggregated setup, demonstrated these differences.
The team measured performance against a “goodput SLO” of inter-token latency under 100 milliseconds, a key metric for responsive AI applications. A test involving ten images per request, capped at 256 tokens per image with an output sequence length (OSL) of 1024, revealed how different topologies handle moderately heavy visual workloads. Performance degradation in the aggregated encoder scenario was observed as the visual token load increased, while both disaggregated topologies maintained relatively consistent performance.
Further analysis, varying image counts from five to fifty with token budgets of 128 and 256, confirmed this trend. Varying the OSL from 128 to 2,048 while keeping the image load fixed at five images showed that while TTFT remained largely unchanged, decode time increasingly dominated end-to-end latency.
Consequently, the performance gain of heterogeneous EPD over aggregated serving narrowed from 20. 3% to 5. 2% at higher OSLs. The colocated encoder, however, experienced a shift from an 11. 8% gain to a 2. 5% regression, as its additional encoder workers shared GPU resources with the prefill and decode workers. In scenarios with high OSL and low image load, disaggregating the encoder compromised performance, but across the full range of tested parameters, from 5 to 50 images and varying OSL, both topologies delivered TTFT improvements ranging from 25 to 93%.
The heaviest tested workload, 50 images at 128 tokens each, achieved gains of 92-93%, with TTFT remaining unaffected by output length. “Both topologies deliver 25-93% TTFT improvement across the full grid,” the researchers state, emphasizing the consistent benefits of disaggregation in demanding scenarios.
Workload Factors Impacting EPD Performance Gains
The benefits of employing Encode-Prefill-Decode (EPD) disaggregation are not uniform; performance gains are heavily influenced by the interplay between image load and output sequence length (OSL), according to recent NVIDIA analysis. Experiments utilizing the Qwen3. 5 122B A10B NVFP4 model revealed that disaggregation delivers improvements ranging from 25 to 93 percent in time to first token (TTFT) across a comprehensive set of tested configurations. These gains, however, are not guaranteed and require careful consideration of workload characteristics to maximize their effectiveness.
This suggests that separating the encoder workload can buffer against performance degradation when processing prompts with a high volume of visual information. The architecture of the serving infrastructure is important in realizing EPD benefits. This configuration, using NIXL over UCX RC/TCP Ethernet for vision embedding transfer with a peak throughput of 20 Gbps, highlights the importance of efficient data transfer between tiers.
As model size increases, the proportion of serving workload attributable to the language model grows, while the vision encoder remains relatively constant in size. This dynamic impacts the effectiveness of EPD, as a larger LLM workload provides more opportunity for the PD workers to offset any overhead introduced by disaggregating the encoder. Mixed traffic, consisting of both text-only and multimodal requests, presents a unique challenge.
EPD isolates the encoder work, enabling text-only requests to bypass waiting behind multimodal requests even though they do not require vision encoding, which can significantly improve responsiveness for text-only prompts. In mixed workloads, the ability to schedule prefill batches independently of encoder processing can further enhance overall throughput.
Colocated EPD exhibited a slight end-to-end regression of 2. 5 percent at 5 x 128 images with an OSL of 2048, where embedding transfer overhead was not fully compensated by encoder capacity savings.
Image-Heavy Requests Demonstrate 5x Faster TTFT with EPD
Serving multimodal AI requests benefits significantly when vision encoding is isolated, with NVIDIA Dynamo experiments revealing up to a 5x faster time to first token (TTFT) and 7x faster end-to-end response time utilizing encode-prefill-decode (EPD) disaggregation under mixed traffic conditions. The performance gains are particularly pronounced in mixed workloads consisting of both text and image requests, where a media-heavy request can delay its own prefill and contend with concurrent prefill and decode work.
EPD disaggregation routes requests to dedicated encoder workers, enabling concurrent processing and reducing delays for text-only prompts. The impact of image load and OSL was further investigated using requests containing between five and fifty images, each with a token budget of 128 or 256, while maintaining a constant OSL.
In contrast, aggregated serving experienced a degradation in performance as image load increased, highlighting the limitations of a single worker handling both vision and language tasks. “EPD provides the most value when vision encoding represents a significant part of request processing,” the researchers state, emphasizing the optimization’s effectiveness with media-heavy inputs and shorter output sequences. This is because these models, while powerful, can be particularly sensitive to latency in the encoding stage.
The team’s findings suggest that the optimal encoder placement depends on where request time is spent, with EPD offering the greatest advantage when vision encoding constitutes a substantial portion of the overall processing time. The analysis revealed that EPD disaggregation delivers gains even with moderately heavy visual workloads, though TTFT gains hold while E2E gains shrink with longer output sequences. This decoupling of tasks is key to achieving faster response times and improved resource utilization.
The team’s work provides a clear guide for optimizing multimodal AI serving, demonstrating that strategic disaggregation of the encoding stage can significantly reduce latency and improve the overall user experience, particularly in scenarios with mixed traffic and image-heavy prompts, NVIDIA claims. The results underscore the importance of carefully considering workload characteristics and model architecture when selecting the best inference optimization techniques.
NVIDIA, a public company listed on Nasdaq as NVDA and headquartered in Santa Clara, United States, has been a significant technology provider since 1993. While widely recognised for graphics processing units, NVIDIA has expanded into high-performance computing and, more recently, quantum computing.
Through its CUDA Quantum platform and DGX Quantum systems, NVIDIA provides both the infrastructure and services for quantum computation, partnering with companies including Quantinuum, QuEra Computing and IQM. These collaborations, such as the 2025 integration of Quantinuum’s Helios processor with NVIDIA GPUs via NVQLink, demonstrate a commitment to linking classical and quantum processing capabilities.
This focus on efficient computation underpins recent developments, including the launch of cudaq-algorithms in May 2026, an accelerated library for quantum-classical computing built on CUDA-Q. NVIDIA’s activity extends beyond quantum, with reports from September 2026 detailing improvements to AI performance in areas like portrait relighting, security layers for AI agents, and financial clustering, as well as adaptations to vehicle design through its Omniverse NuRec platform.
The company has also pledged continued open access to AI building tools via Hugging Face. Ten patent families and four publications in the last twelve months, updated as of 5 September 2026, reflect ongoing research and development.
See today’s quantum computing news on Quantum Zeitgeist for the latest breakthroughs in qubits, hardware, algorithms, and industry deals.
