NVIDIA is streamlining generative AI workflows by enabling a single TensorRT network to execute across multiple GPUs, the company says. The company’s TensorRT 11.0 now supports multi-device inference using NCCL-backed distributed collectives, and NVIDIA Dynamo-Triton release 26.07 manages the complex setup for users. This integration allows organizations to reduce request latency by trading GPU resources. By handling GPU coordination internally, Dynamo-Triton accelerates review-and-refine cycles in latency-sensitive applications like video generation, demonstrated with the NVIDIA Cosmos 3 Nano model, where Dynamo-Triton serves the 44,160 video tokens across as many as eight NVIDIA GPUs.
TensorRT Multi-Device Inference with Dynamo-Triton Integration
NVIDIA Cosmos 3 Nano video generation uses NVIDIA Dynamo-Triton, which serves the 44,160 video tokens across as many as eight NVIDIA GPUs. This integration addresses the increasing computational and memory demands of generative AI models, which often exceed the capacity of a single graphics processing unit. The distributed Ulysses graph, central to this approach, is compiled into each TensorRT plan prior to deployment, streamlining the process for developers and reducing the complexity of multi-GPU setups.
This compilation allows Dynamo-Triton to load versioned plans and create execution states without requiring user intervention in GPU coordination. Dynamo-Triton’s role extends beyond simply distributing the workload; it presents a unified gRPC model endpoint to the client, abstracting away the underlying multi-GPU infrastructure. Clients submit transformer requests to this single endpoint, eliminating the need to manage individual GPU ranks or communicators.
This simplification is particularly impactful for latency-sensitive applications like Cosmos 3 Nano, where the transformer stage accounts for 93.4% of single-GPU generation time, making it the primary target for acceleration. Each of the 35 denoising steps within Cosmos 3 Nano requires two sequential Triton calls, resulting in 70 transformer requests per generated frame, all managed transparently through the unified Dynamo-Triton interface. The benefits of this integration are demonstrated through Cosmos 3 Nano, a long-sequence workload that highlights the potential for scaling AI inference.
The system delivers noise patches to the application workflow after processing prepared tensors, with Dynamo-Triton handling the underlying TensorRT multi-device inference, according to NVIDIA. “One Triton KIND_MODEL instance can own multiple GPUs, create per-rank TensorRT execution contexts, CUDA streams, and NCCL communicators, and launch the ranks together for each request,” explains the development team in their published findings.
This architecture allows for a reduction in user wait time and accelerates review-and-refine cycles, important for generative media workflows where rapid iteration is essential. The team observed diminishing returns as more GPUs were utilized. While this benchmark focuses on latency reduction, the team acknowledges that further evaluation is needed to assess concurrent request throughput, cost per generated video, and total cost of ownership against specific service level objectives.
NVIDIA’s investment in multi-device inference aligns with its broader strategy in accelerated computing and quantum technologies. The company’s NVQLink architecture, launched in 2026, facilitates low-latency integration of quantum processors with GPU supercomputers, and a quantum calibration model was reported on 2026-07-28 functioning across six qubit modalities. This commitment to both classical and quantum computing underscores NVIDIA’s position as a key player in the future of AI and high-performance computing.
Multi-GPU Performance Gains with Cosmos 3 Nano Generation
Dynamo-Triton now serves the 36-layer denoising transformer, responsible for 93.4% of single-GPU generation time in Cosmos 3 Nano, across as many as eight NVIDIA GPUs using Ulysses context parallelism. This distribution compiles into a distributed Ulysses graph, then loads into a versioned TensorRT plan accessible via a single gRPC model endpoint, shielding the client from GPU coordination complexities.
A complete Cosmos 3 video generation, previously requiring over two and a half minutes, now finishes in approximately 34 seconds with this configuration, all while maintaining a conventional model-serving interface. Speedup versus ideal linear scaling indicates diminishing returns as more GPUs were utilized.
Specifically, the transformer RPC share of generation time decreased from 93.4% on a single GPU to 70.2% on eight GPUs, as client-side overhead, prompt processing, scheduler updates, VAE decoding, and post-processing, became a more significant factor. The team validated generated output by sampling frames and comparing context-parallel results with single-device outputs, ensuring consistency across configurations. The observed performance gains are quantified in a comparison of single-GPU (SD) and multi-GPU (CP2, CP4, CP8) Cosmos 3 runs, measuring both end-to-end latency and Triton transformer RPC latency.
CP8 achieved a 4.58x speedup in end-to-end latency and a 6.09x speedup in transformer RPC latency compared to the single-GPU baseline. Evaluating these additional metrics remains important for organizations determining the optimal resource-for-latency trade-off for their specific deployments. To replicate these results, developers can download NVIDIA Dynamo-Triton 26.
The team provides detailed documentation outlining the process, enabling wider adoption of multi-device inference for generative AI workloads, the firm reports. These events happened on 2026-09-17, with the company reporting a 6.4x speedup with NVIDIA in the MLPerf Edge Agentic benchmark, and a new chip delivering leading performance in MLPerf, further demonstrating its commitment to pushing the boundaries of AI inference.
Validation of Output Quality Across GPU Configurations
NVIDIA’s implementation of multi-device inference with TensorRT 11.0 and Dynamo-Triton 26.07 doesn’t merely accelerate processing; validation procedures confirm maintained output quality even as generative AI tasks scale across multiple GPUs. The team rigorously assessed generated content, ensuring consistent results despite distributing the computational load, a critical step beyond simply measuring latency gains. Every run utilized a 1280×720 output, processing 189 frames at 24 frames per second with 35 denoising steps, followed by five complete generations after a single warm-up period.
Timing encompassed prompt processing, 70 Dynamo-Triton calls, configuration updates, VAE decoding, and frame post-processing, excluding model loading and final MP4 encoding. To confirm visual fidelity, the researchers employed a comparative validation process, sampling frames 0, 47, 94, 141, and 188 from each configuration, single GPU, and configurations utilizing two, four, and eight GPUs, and comparing them against a single-device baseline.
This assessment focused on both format consistency and temporal variation, establishing thresholds of mean absolute error (MAE) ≤ 25 and peak signal-to-noise ratio (PSNR) ≥ 18 dB. The team notes that pixel-identical replication wasn’t the goal, but rather maintaining a coherent and acceptable visual output. CP2 and CP4 configurations achieved a measured MAE of 12.759 and a PSNR of 21.111 dB, while the eight-GPU CP8 setup yielded an MAE of 16.316 and a PSNR of 19.400 dB, all falling within the defined parameters.
Visual inspection of a contact sheet further confirmed consistent action, a robot arm cleaning a plate, across all tested configurations. The significance of this validation extends beyond aesthetic quality; it addresses a core challenge in distributed inference.
Speedup versus ideal linear scaling implies diminishing returns as transformer RPCs account for a smaller share of total generation time with increased GPU utilization, falling from 93.4% on a single GPU to 70.2% on eight, other computational bottlenecks become more prominent. The team observed that time spent on tasks outside the measured RPC path remained relatively constant, between 10.2 and 10.5 seconds, across all configurations.
This indicates that prompt processing, scheduler updates, VAE decoding, post-processing, and client-side overhead increasingly influence overall performance as GPU parallelism is maximized. Consequently, optimizing these areas becomes important for realizing further gains. This focus on comprehensive performance, encompassing both speed and output integrity, reflects NVIDIA’s broader strategy in accelerated computing. The practical implication, as the team notes, is a viable option for product teams prioritizing response time over sheer GPU allocation, enabling a complete Cosmos 3 generation in approximately 34 seconds, a substantial reduction from the initial two-and-a-half-minute timeframe.
