NVIDIA has released NodeWright, an open-source project designed to update entire fleets of GPU nodes without interrupting active training runs, a feat currently often managed with “a spreadsheet, a maintenance window, and an engineer watching a terminal at 3 a.m.” Unlike standard Kubernetes node management, NodeWright addresses the unique challenges of GPU infrastructure where hardware scarcity prevents simple node replacement and rescheduling of long-running jobs.
The project, previously used internally at NVIDIA as Skyhook, is part of NVIDIA DSX OS, a portfolio of open-source tools including NVIDIA GPU Operator and Topograph, aiming to treat entire AI facilities as single, cohesive systems.
NVIDIA NodeWright: Addressing Kubernetes Node Management Challenges
NodeWright distinguishes itself from existing configuration tools by prioritizing fleet-level changes over individual machine management, a distinction important for large-scale AI infrastructure. Traditional systems like Ansible and Puppet were not designed to handle the complexities of actively running, sensitive workloads within a Kubernetes cluster; updating a kernel parameter across 200 GPU nodes requires more than simply executing a script.
NVIDIA GPU Operator, NVIDIA Network Operator, Topograph, DRA Driver for NVIDIA GPUs, NodeWright, NVIDIA Cluster Readiness Engine (NVCRE), and NVSentinel all contribute to this AI-ready foundation, working in concert to deliver a cohesive system, the company says. NodeWright itself functions as a Kubernetes-native package manager specifically designed for modifying and maintaining host infrastructure at scale, offering a declarative approach to node configuration and updates. NodeWright’s internal deployment as Skyhook provided valuable real-world testing before its public release, allowing NVIDIA to refine the system within a production environment.
The operator component is the central controller, monitoring NodeWright custom resources and orchestrating the lifecycle of changes across the entire node fleet. This operator meticulously tracks the state and semantic version of each package installed on every node, enabling it to differentiate between initial installations, upgrades, and downgrades, and to manage dependencies to ensure correct execution order. Validation is integrated throughout the process, with checks verifying the expected node state at each stage, apply, configuration, upgrade, uninstall, and post-interrupt, and surfacing failures directly through the Kubernetes API.
The system’s risk tolerance is configurable, allowing administrators to initiate fleet-wide updates starting with a single canary node. Successful verification on that initial node triggers automatic acceleration of the rollout, while any detected issues halt the process, preventing cascading failures.
Error reporting is similarly integrated, with NodeWright marking failed Jobs, and adding labels and conditions to affected nodes for rapid identification and triage via the Kubernetes API. This granular control is particularly important given that GPU nodes cannot simply be discarded and spun up like other Kubernetes nodes due to hardware scarcity and the difficulty of rescheduling long-running training jobs.
NodeWright packages are defined as Custom Resources, seamlessly integrating with existing Kubernetes deployment methods such as kubectl, Helm, Argo CD, and Flux. The public package repository offers modular components for tasks like executing shell commands, managing bind mounts, and establishing kernel crash dump collectors.
A specialized package addresses Google Kubernetes Engine (GKE) nodes running Container-Optimized OS, providing necessary tuning when the standard stack is unavailable. Dedicated packages automate bootstrap steps for specific cloud and accelerator combinations, handling kernel version management and Elastic Fabric Adapter (EFA) driver installation for Amazon Elastic Kubernetes Service (Amazon EKS) clusters equipped with NVIDIA Hopper or Blackwell GPUs. These packages are part of a larger initiative integrating NodeWright with NVIDIA AI Cluster Runtime (AICR).
Because packages require root-level privileges for host modification, NodeWright uses native Kubernetes primitives for security and consistency; fine-grained RBAC controls user permissions, admission controllers validate specifications, and integrated validation checks ensure state consistency throughout the entire lifecycle. “You set the risk tolerance,” explains the documentation, highlighting the system’s focus on controlled, reliable updates in complex AI environments.
DSX OS and NodeWright: Declarative Configuration for GPU Fleets
NodeWright distinguishes itself through a focus on host-level configuration, extending beyond Kubernetes’ orchestration of containerized applications to manage underlying operating system elements like kernel settings and system packages. This granular control addresses a critical gap in traditional configuration management systems, which often lack awareness of Kubernetes’ operational needs and can disrupt ongoing workloads during updates. NVIDIA’s approach centers on managing fleets of nodes as a unit, rather than individual machines, streamlining updates and reducing the risk of inconsistencies across a cluster.
The system’s design prioritizes maintaining workload continuity, a feature absent in many existing solutions that rely on manual intervention and potentially lengthy downtime. The development of NodeWright stemmed from internal needs at NVIDIA, initially known as Skyhook, to reliably update GPU node fleets without interrupting critical training runs.
NodeWright packages, the core components of the system, are designed to perform a wide range of host-level operations, including setting system parameters, configuring crash dump collection, and installing security agents, all without requiring node recycling. These packages can declare dependencies, ensuring correct execution order and simplifying complex configuration sequences. Validation is integral to the NodeWright lifecycle, with each operation, application, configuration, upgrade, and uninstallation, paired with checks that verify the expected node state.
This proactive approach allows the system to detect failures and provide immediate visibility into affected nodes, rather than relying on workload failures to signal a problem, according to NVIDIA. For instance, a package designed to remediate a Common Vulnerabilities and Exposures (CVE) can confirm whether a vulnerable kernel module remains loaded and flag the package as failed if the issue persists.
This capability extends to cluster scaling, where newly provisioned nodes can be automatically tainted until required configuration and tuning are completed, preventing workloads from being scheduled on incompletely prepared hardware. The project’s architecture is modular, allowing organizations to adopt individual components or combine them into a comprehensive platform. “Adopt one project, integrate several, or compose them into a platform,” states NVIDIA, emphasizing the flexibility of DSX OS and NodeWright.
This open design contrasts with monolithic stacks, offering users greater control and customization options. Packages within NodeWright are defined as Custom Resources, so they deploy the way everything else in your cluster does: through kubectl, Helm, Argo CD, Flux, or whatever GitOps tooling you already run. The public package repository provides pre-built modules for common tasks, such as executing shell commands and managing bind mounts, while also supporting the creation of custom packages tailored to specific environments.
NodeWright is licensed under Apache 2. 0 and is a component of DSX OS, a broader portfolio of open-source projects focused on building an AI-ready infrastructure. The company encourages community involvement, providing access to the project repository, issue tracker, and documentation to facilitate collaboration and further development, the company says. The emphasis on open interfaces and independent adoption underscores a shift towards open-source solutions, allowing organizations to benefit from a collaborative ecosystem and tailor the system to their specific needs.
NodeWright Architecture: Operator, Custom Resources, and Packages
NodeWright employs Kubernetes Custom Resources to define modifications to node fleets, moving beyond traditional scripting for system-level changes. These resources detail desired states, such as specific kernel settings or package versions, and are applied through a dedicated Kubernetes operator. This operator monitors for these custom resources and orchestrates a controlled sequence of actions on targeted nodes, ensuring changes are implemented consistently across the infrastructure. The system’s architecture prioritizes respecting existing Kubernetes primitives like PodDisruptionBudgets, node selectors, and taints, integrating seamlessly with established orchestration workflows.
The operator begins by cordoning a node before initiating any modifications, preventing new pods from scheduling onto it during the process, and only uncordons the node upon successful completion. This careful sequencing is exemplified in a sample Custom Resource definition, which includes a podNonInterruptLabels field specifying workloads that should be paused during the operation.
Packages themselves are container images containing the necessary scripts, configurations, and binaries for the desired changes, and include verification scripts to detect and halt rollouts if modifications fail. This built-in validation extends to multiple stages of the package lifecycle, including application, configuration, upgrades, uninstallation, and post-interrupt procedures, all verified through Kubernetes mechanisms. Packages can also declare dependencies on each other, allowing NodeWright to intelligently determine the correct execution order, ensuring that prerequisites are met before subsequent modifications are applied.
This granular tracking and dependency management are critical for maintaining a stable and predictable environment, particularly in complex AI workloads. The project integrates with existing deployment tools through Helm, deploying as an OCI artifact without requiring the addition of external repositories.
NVIDIA provides a public repository for both the NodeWright project itself, including source code, issue tracking, and discussion forums, and for pre-built packages, fostering community contributions and extending the system’s functionality, the company states. The company actively encourages external involvement, seeking packages tailored to hardware and cloud combinations not currently covered by the catalog, as well as tuning profiles from operators running unique configurations, the company’s account states.
This collaborative approach reflects a broader shift towards open-source solutions, as NodeWright is licensed under Apache 2. The value proposition, they assert, lies in open interfaces, independent adoption, and a coherent lifecycle, rather than a monolithic, tightly coupled stack. Operating at scale has revealed early failure modes, and NVIDIA shares these observations with the community.
NodeWright complements existing NVIDIA tools like the GPU Operator and Network Operator, managing the host operating system layer beneath them, rather than replacing them. The team views Kubernetes as having transformed workload management, and NodeWright aims to bring the same declarative, automated, and safe approach to the underlying host layer, particularly for GPU nodes supporting demanding AI applications, NVIDIA claims. Project references and package repositories are publicly available, inviting further exploration and contribution.
Safe, Scalable Updates with NodeWright’s Deployment Policies
This granular awareness allows for precise control over the update process, a capability previously difficult to achieve at scale, and is a core component of NVIDIA DSX OS, a portfolio designed to provide an AI-ready foundation for Kubernetes infrastructure. The system’s architecture is built around three key components: an operator, custom resources, and packages, working in concert to automate host-level changes throughout their lifecycle. Packages can perform many host-level operations that would normally require root access without recycling nodes.
They can set sysctl and GRUB parameters, configure crash dump collection, create logical volumes, install security agents, remediate CVEs, and perform other system-level configuration tasks. Safe rollouts are achieved through the NodeWright DeploymentPolicy resource, which provides progressive rollout strategies tailored to the specific risk tolerance of the operator; a fleet-wide kernel update, for example, can begin with a single canary node, automatically accelerating the rollout only after successful verification.
Packages within NodeWright are declarative definitions that include dependencies, enabling the system to determine the correct execution order; this ensures that updates are applied in a logical sequence, preventing conflicts and maintaining system stability, NVIDIA says. “You set the risk tolerance,” explains NVIDIA, highlighting the flexibility of the system and its ability to adapt to diverse operational requirements. The ability to manage dependencies and execution order is particularly important in complex AI infrastructure, where multiple software components must interact seamlessly to deliver optimal performance.
NodeWright’s design acknowledges the unique challenges of GPU node fleets, where hardware scarcity and long-running training jobs preclude simple discard-and-recreate strategies; unlike stateless applications, interrupting a training run can be costly and time-consuming, necessitating a more nuanced approach to updates. The system integrates seamlessly with existing Kubernetes primitives, PodDisruptionBudgets, node selectors, taints, and tolerations, respecting established workload management policies and minimizing disruption.
See today’s quantum computing news on Quantum Zeitgeist for the latest breakthroughs in qubits, hardware, algorithms, and industry deals.
