NVIDIA NVLink Delivers 260 TB/s Rack Bandwidth for AI Factories

NVIDIA’s sixth-generation NVLink provides 260 terabytes per second of rack-level bandwidth, a figure that significantly surpasses the performance of conventional Ethernet solutions for demanding artificial intelligence workloads. NVIDIA has demonstrated the impact of this approach with its latest NVLink iteration, delivering up to 2.3 times the decode throughput compared to leading off-the-shelf Ethernet in models such as DeepSeek-R1 and Qwen 235B, as well as a simulated 2T parameter LLM, achieving 3.6 TB/s bandwidth per accelerator. This scale-up networking fabric determines how efficiently data moves between processing units, impacting both performance and deployment risk. “The scale-up fabric is what enables accelerators to work as a single unit of compute,” NVIDIA explains, highlighting NVLink’s role in optimizing collective operations and maximizing throughput for trillion-parameter models and beyond. Through co-design of hardware and software, including NVIDIA Dynamo and TensorRT-LLM, the platform has achieved a 50X improvement in tokens per watt from the NVIDIA Hopper to NVIDIA Blackwell architectures.

NVLink 6: 3.6 TB/s Scale-Up Networking for AI Factories

NVLink 6 delivers 3.6 terabytes per second of bandwidth per GPU, defining high-end data transfer speeds within artificial intelligence infrastructure. This leap in connectivity enables a new architectural approach to AI, centered around data center-scale systems designed to continuously convert data and energy into intelligence. These factories demand more than peak accelerator performance; they require a cohesive system where numerous accelerators function as a unified computational unit. The scale-up fabric is critical for this unified operation, determining how efficiently tokens move between processing units and impacting both training times and inference throughput.

NVLink 6 achieves this with a rack-level bandwidth of 260 terabytes per second, significantly outpacing conventional off-the-shelf Ethernet solutions, particularly when handling the intensive communication patterns of modern AI workloads like mixture-of-experts models. Efficient inference implementations, for example, rely on expert parallelism, distributing tasks across GPUs and necessitating rapid data exchange; a slow fabric can negate the benefits of this parallelization. This efficiency boost translates to reduced energy consumption and lower operational costs for AI factories. The platform’s design prioritizes resilience, incorporating features like control plane resilience, hot-swappable switch trays, and dynamic routing to minimize downtime and ensure continuous operation. Beyond immediate performance gains, NVLink is engineered for future expansion. NVLink-C2C supports CPU-GPU coherence, while NVLink Fusion enables seamless integration of custom XPUs, ensuring the platform can accommodate diverse acceleration technologies beyond NVIDIA’s own GPUs.

This forward-looking approach, coupled with a proven supply chain, aims to deliver sustained, dependable return on investment for production AI factories, acknowledging that scaling AI requires both technological innovation and a reliable infrastructure to support it. NVIDIA asserts that delivered performance only matters if the factory is operational, emphasizing the importance of a robust and dependable system. Evaluating scale-up technologies requires considering not just bandwidth numbers, but also end-to-end latency, in-network compute capabilities, and a mature software stack.

The current landscape of AI infrastructure demands a holistic approach to performance, moving beyond simply faster processors to consider the intricate interplay between hardware and software. Unlike scale-out networks that connect servers, NVLink focuses on maximizing communication within a domain of accelerators, enabling them to function as a unified compute resource. This co-design philosophy, integrating chips, systems, fabric, and software, is now in its sixth generation and is demonstrably impacting efficiency. A key element of this integrated design is the optimization of software alongside the NVLink hardware. NVIDIA highlights the importance of tools like Dynamo, TensorRT-LLM, NCCL, and NIXL, which work with NVLink to facilitate features such as disaggregated inference and expert parallelism. If communication is hampered by a low-bandwidth or high-latency fabric, the benefits of parallel processing are diminished. Beyond performance, the NVLink platform prioritizes operational robustness.

Scale-Up vs. Scale-Out Networking for AI Workloads

NVIDIA is refining the architecture of AI infrastructure with its sixth-generation NVLink, a networking fabric designed to address the escalating demands of artificial intelligence workloads. As AI models grow in complexity and scale, increasing peak accelerator performance is insufficient; modern systems require efficient communication between numerous accelerators operating as a unified compute resource. This shift has positioned scale-up networking as a critical architectural decision for AI factories, data center-scale systems dedicated to converting data and energy into intelligence. The fundamental distinction lies in how these networks function. Scale-out fabrics, exemplified by NVIDIA Quantum InfiniBand and Spectrum-X Ethernet, excel at connecting servers across vast data centers, ideal for large, data-parallel AI training and scientific simulations. Conversely, scale-up networks, like NVLink, focus on interconnecting accelerators within a single domain, prioritizing high bandwidth, low latency, and shared memory access.

This is particularly evident in inference tasks utilizing mixture-of-experts models, where efficient dispatching of tokens to selected experts demands intensive all-to-all communication between GPUs. A slow fabric can negate the benefits of this parallelization. NVIDIA has demonstrated the impact of this approach with its latest NVLink iteration, delivering up to 2.3 times the decode throughput compared to off-the-shelf Ethernet in a 72-accelerator domain achieving 3.6 TB/s bandwidth per accelerator for models such as DeepSeek-R1 and Qwen 235B, as well as a simulated 2T parameter LLM.

The escalating demands of artificial intelligence are reshaping data center architecture, pushing beyond traditional limitations of compute power to focus on efficient data transfer. Modern AI factories, designed to continuously refine data into intelligence, now prioritize how quickly information moves between processing units. The architecture’s impact is particularly pronounced when deploying mixture-of-experts models, a technique that distributes processing across multiple specialized “experts” housed on different GPUs. Efficient inference implementations use expert parallelism to distribute experts across GPUs, and they use large batch sizes to maximize factory throughput. For models such as DeepSeek-R1 and Qwen 235B, as well as a simulated 2T parameter LLM, NVLink delivers up to 2.3X the decode throughput compared to leading off-the-shelf Ethernet, both in a 72 accelerator scale-up domain at 3.6 TB/s bandwidth per accelerator. “All of the GPU-to-GPU communication must happen in parallel,” NVIDIA explains, highlighting the critical need for low latency and high bandwidth. 6 TB/s bandwidth per accelerator. Results are based on simulation, but the implications for scaling AI inference are substantial.

Resiliency and Operational Features for AI Factory Uptime

While peak accelerator performance receives considerable attention, the ability to sustain operations, monitor system health, and perform maintenance without disrupting the entire system is now a defining characteristic of successful AI infrastructure. NVIDIA’s sixth-generation NVLink provides purpose-built, high-bandwidth, low-latency scale-up networking for AI factories, addressing this need with a comprehensive suite of resiliency and operational features designed to maximize uptime and return on investment. Beyond simply accelerating data transfer, the NVLink platform incorporates control plane resilience, ensuring continued operation even in the event of component failure. Hot-swappable switch trays allow for component replacement without system shutdown, minimizing downtime and maintaining continuous throughput. Dynamic routing capabilities adapt to changing network conditions, circumventing potential bottlenecks and maintaining consistent performance. These features, combined with in-service update capabilities, allow for software and firmware updates without interrupting production workloads, a critical advantage for rapidly iterating AI models.

Fine-grained telemetry provides detailed insights into system health, enabling proactive identification and resolution of potential issues before they escalate into full-scale outages. This level of observability is essential for managing the complexity of large-scale AI deployments and ensuring consistent performance. The maturity of the NVLink ecosystem, built upon over a decade of investment and deployments with leading hyperscalers and supercomputing centers, further reinforces its reliability. A proven supply chain mitigates the risks associated with sourcing critical components, ensuring consistent availability and predictable lead times. This contrasts sharply with adopting unproven technologies that could introduce unforeseen delays or disruptions. NVIDIA emphasizes that scaling AI factories is an incredibly complex challenge with many dependencies, and operators seek to minimize risk by leveraging established technologies with a demonstrated track record. The platform’s architecture is designed for future expansion, with NVLink-C2C supporting CPU-GPU coherence and NVLink Fusion enabling seamless integration of custom XPUs.

This adaptability ensures the platform can accommodate evolving hardware configurations and emerging acceleration technologies, safeguarding long-term investment and preventing technological lock-in. The combination of these features delivers not just performance, but sustained, dependable ROI over the lifetime of the AI factory.