Keeping pace with rapidly evolving AI infrastructure has become a core challenge for data scientists, and NVIDIA is addressing a surprisingly basic bottleneck: accurately mapping cluster connections. The company’s new Topograph toolkit discovers and normalizes GPU and network relationships, publishing the data in formats compatible with widely-used cluster managers like Kubernetes and Slurm, the company says.
“A scheduler can only act on the topology it observes,” explain NVIDIA’s Dmitry Shmulevich and Rob Esker, and Topograph ensures that schedulers work with current data, avoiding performance losses from fragmented topology and congested links. By providing a dynamic, up-to-date cluster map, Topograph aims to unlock greater efficiency from power-hungry AI factories.
NVIDIA Topograph Resolves Cluster Topology for AI Workloads
NVIDIA Topograph generates scheduler-ready outputs in three distinct formats, Kubernetes node labels, Slurm topology configuration, and topology YAML, demonstrating a deliberate design for integration with established cluster management systems rather than requiring a complete infrastructure overhaul. This adaptability addresses a critical bottleneck in AI workload optimization: maintaining an accurate, current view of GPU and network relationships, a challenge where practical implementation often falls short of theoretical efficiency.
The toolkit discovers cluster topology from cloud application programming interfaces or on-premises fabric systems, then normalizes this information into a standardized model before publishing it for use by schedulers. The core problem Topograph addresses stems from the continuous data exchange between GPUs during both training and inference, meaning distributed workloads benefit significantly from communication locality.
NVIDIA NVLink and NVLink Switch provide 1.8 terabytes per second of bidirectional bandwidth per GPU in its fifth generation (NVIDIA Blackwell, such as GB200/GB300) and 3.6 terabytes per second per GPU in its sixth generation (Vera Rubin), through a dedicated NVIDIA NVLink Switch fabric. Topograph maps how these cluster hardware connections function, allowing schedulers to prioritize nearby resources and minimize bottlenecks; the system visualizes the network as a road system, where GPUs within the same locality domain have short, high-bandwidth paths, and traffic between domains encounters more shared links and switches.
Spreading a tightly coupled workload across distant domains can increase contention and latency, a scenario Topograph aims to prevent through optimized placement. Topograph’s architecture centers around two key concepts: providers and engines. The system accepts a generation request, then discovers and normalizes topology from a selected cloud or network fabric provider, before publishing the scheduler-ready outputs.
The API server exposes five service endpoints, including a POST request to initiate an asynchronous topology generation, returning a unique ID with an HTTP 202 status code, according to Kubernetes. Installation typically occurs on Linux bare-metal servers or virtual machines, utilizing native package managers with Debian and RPM build targets, and the resulting package installs the service without automatically starting it, allowing for configuration review and editing via the /etc/topograph/topograph-config. yaml file.
Slurm clusters benefit from Topograph’s ability to regenerate topology views on request and upon detected cluster changes, ensuring the scheduler operates with current data rather than relying on potentially outdated snapshots. The toolkit supports three distinct output modes for Slurm: cluster-wide tree, cluster-wide block, or topology YAML, offering flexibility based on specific cluster configurations. For Slinky deployments, a narrower provider option exists for Multi-Node NVLink systems, supporting block topology only (DRA block topology only).
Dynamic Slurm nodes can also be annotated with the current topology specification through an optional mode, providing further adaptability to changing cluster environments. This deliberate strategy recognizes that communication needs differ at various scales, and the toolkit is engineered to accommodate these variations. The system’s open-source nature further encourages community contributions and customization, potentially accelerating the development of even more sophisticated topology-aware scheduling solutions for AI factories.
NVLink and Spectrum-X Enhance GPU Interconnect Bandwidth
NVIDIA’s Topograph toolkit utilizes the differing capabilities of NVLink and Spectrum-X Ethernet to optimize data flow within and between GPU clusters, a critical step toward maximizing the efficiency of increasingly power-hungry artificial intelligence factories. While NVLink and its Switch variant deliver all-to-all connectivity within a rack, providing up to 1.8 terabytes per second of bidirectional bandwidth per GPU in its fifth generation (NVIDIA Blackwell, such as GB200/GB300) and 3.6 terabytes per second per GPU in its sixth generation (Vera Rubin), through a dedicated NVIDIA NVLink Switch fabric, Spectrum-X Ethernet extends that connectivity across systems and racks, acknowledging that communication demands vary significantly depending on scale.
Topograph Provider & Engine Architecture for Topology Normalization
This design prioritizes adaptability, acknowledging that diverse infrastructure configurations require tailored discovery mechanisms; the system currently supports InfiniBand integration via ibnetdiscover, and NetQ for Spectrum-X or Multi-Node NVLink domains when deployed on-premises. The flexibility extends to output formats, translating normalized topology into configurations usable by established cluster management systems like Slurm, Kubernetes, Slinky ConfigMaps, and Node Feature Discovery resources, the firm reports. The engine component of Topograph isn’t limited to a single environment; it supports Kubernetes, Slurm, and graph-based representations, offering options for different scheduling paradigms.
Current upstream support, as of September 16, 2026, details specific provider-to-engine output combinations, with requirements varying based on Topograph version, environment, and provider configuration. The Crusoe provider operates within Kubernetes, reading fabric and accelerator-domain labels from Crusoe Managed Kubernetes nodes, while the Slurm engine requires a writable volume for its configured topology configuration file.
The NFD engine, in turn, necessitates the alpha NodeFeatureGroupAPI feature gate for functionality. Kubernetes deployments benefit from runtime helpers that react to cluster changes and collect per-node data, enabling dynamic adjustments to workload placement, the company states. For testing and development, simulation models allow the creation of virtual Kubernetes nodes using the kwok-nodes utility and Kind/KWOK helpers, providing a hardware-independent environment. The system’s approach to Kubernetes deployments addresses a key limitation of the default scheduler, which lacks awareness of physical interconnect hierarchy.
Topograph resolves this by publishing provider-reported topology as node labels, allowing native affinity and topology-aware schedulers to optimize workload distribution. Deploying Topograph via Helm simplifies the process, with a readily available chart and example values files.
Operators can customize the Kubernetes engine’s fabricLabels array and acceleratorLabel parameter to use custom keys, though tiers beyond that array are not labeled, and the sub-domain key remains fixed. Verification of label application is achieved via kubectl get nodes –show-labels | grep -E ‘fabric\. topograph\. run|accelerator\. run’. The system offers three distinct mechanisms for dynamic Slurm nodes: cluster-wide tree, cluster-wide block, and topology YAML, allowing operators to choose the mode best suited to their Slinky configuration. Highlighting the importance of accurate provider data for optimal performance.
Supported Environments: Kubernetes, Slurm, and Cloud Integration
Topograph’s architecture centers around providers and engines, with providers responsible for discovering and normalizing topology from diverse sources, and engines translating that standardized model into formats consumable by various workload managers. The system’s API server facilitates topology generation through asynchronous requests, returning a status code 202 with the request ID, and dynamically updates the cluster map upon detected changes. These collaborations, alongside patents in related fields, underscore NVIDIA’s commitment to pushing the boundaries of high-performance computing.
The benefits of this dynamic topology mapping extend beyond simply improving scheduling accuracy; Topograph aims to enhance AI factory efficiency, tokens per watt, and overall cost. Through integration with KAI Scheduler, Kueue, and native Kubernetes, the system provides a current, provider-reported map of the physical network, ensuring consistent topology-aware decisions across both cloud and on-premises environments.
This eliminates the need for manual maintenance, a historically significant bottleneck in AI workload optimization, and allows for more effective resource allocation. The system is available from the dsx-ai-factory/topograph GitHub repository, offering developers a readily accessible tool for optimizing their AI infrastructure.
Topograph API and Components Maintain Current Cluster Topology Views
This adaptability stems from the toolkit’s core design, which decouples topology discovery from workload management, allowing schedulers to utilize current hardware connections for optimized resource allocation. The node observer continuously monitors Kubernetes nodes or Pod changes, triggering topology regeneration upon detection of alterations, ensuring the scheduler operates with the most current information. This proactive approach contrasts with static topology maps, which quickly become inaccurate in dynamic AI environments.
The node data broker then collects per-node attributes, storing them as annotations for efficient access and analysis. Topograph’s flexibility extends to its provider and engine structure, allowing it to adapt to various infrastructure configurations.
The choice of engine dictates the output format; the Slurm engine, when running within Kubernetes, requires a writable volume to store its configured topology file, while the NFD engine leverages the alpha NodeFeatureGroupAPI feature gate. This modularity enables seamless integration with existing workflows and minimizes disruption during deployment. This design prioritizes responsiveness and efficiency, especially in large-scale AI factories where frequent topology updates are common. yaml, allows administrators to specify the provider and request aggregation delay, tailoring the system to their specific environment.
For dynamic Slurm nodes, an optional mode annotates Kubernetes nodes with the current Slurm topology specification, providing a consistent view of resources across both platforms, by the company’s account. “Topograph regenerates and updates the ConfigMap when selected slurmd Pods change,” according to the documentation, ensuring continuous synchronization between the cluster’s physical topology and the scheduler’s view.
See today’s quantum computing news on Quantum Zeitgeist for the latest breakthroughs in qubits, hardware, algorithms, and industry deals.
