NVIDIA FLARE 2.8 supports Docker and Kubernetes deployment, and FLARE 2.9 adds Slurm support, extending its existing compatibility with Docker and Kubernetes to allow federated learning projects to span sites with diverse infrastructure. The update addresses a key challenge for collaborative studies: operating shared infrastructure as projects scale beyond a single server and dataset.
FLARE’s two-layer architecture specifically separates persistent federation services from job execution, meaning organizations don’t need to standardize infrastructure to participate. This allows each site to utilize its preferred execution backend, Docker, Kubernetes, or Slurm, while retaining local control over data and compute policies.
FLARE 2.9 Integrates Slurm with Docker and Kubernetes
NVIDIA FLARE 2. This addition addresses a critical hurdle in scaling federated learning beyond initial, simplified setups, where a single server and limited datasets at each location were sufficient. As projects mature and incorporate more participants, the need for flexible infrastructure management becomes paramount, and FLARE’s architecture is designed to accommodate this growth without mandating platform standardization.
The core of this expanded capability lies in FLARE’s two-layer design, which distinctly separates the persistent services that maintain the federation from the transient job execution processes. This architectural choice allows each participating site to use its preferred execution environment, whether a Docker host, a Kubernetes cluster, or a Slurm-managed high-performance computing (HPC) environment, while still contributing to a unified federated learning workflow.
Each site maintains independent control over its compute resources, datasets, images, secrets, and scheduling policies, an important factor for organizations with stringent data governance or security requirements, NVIDIA says. The system’s ability to dynamically launch job workers through each site’s native runtime ensures resource availability without monopolizing GPUs needed for ongoing federation services. When a data scientist submits a job through FLARE, the system initiates a worker process on each participating site, utilizing the locally configured execution platform.
This worker receives the job instructions, requests the necessary resources, executes the task, and returns the results, exiting once the work is complete. Importantly, the job description separates resource intent, the desired GPUs, CPU units, and memory, from the platform-specific details of how those resources are allocated. A site’s launcher then translates these requirements into the appropriate commands for its environment, whether creating a Docker container, a Kubernetes pod, or a Slurm allocation.
This abstraction layer is key to interoperability, allowing a single job submission to trigger actions across a heterogeneous landscape. Docker provides a practical solution for single-host environments, such as workstations or edge systems. FLARE utilizes parent containers for persistent server and client processes, and dynamically launches separate job containers for each submitted task. Sites can expose requested GPUs to these containers through the NVIDIA Container Toolkit, customizing settings for shared memory, mounts, networking, and other host-specific needs.
Separating parent and job images streamlines updates; infrastructure owners can maintain a stable parent environment while researchers independently update their training environments, subject to site-specific image and code approval policies. Kubernetes deployments use Helm charts to install persistent FLARE server and client processes as parent pods.
For each job, the parent pod creates a separate job pod, which is then scheduled by the Kubernetes scheduler based on CPU, memory, GPU, storage, and placement requirements. This integration allows FLARE jobs to benefit from familiar Kubernetes capabilities, including namespaces, service accounts, Secrets, persistent volumes, node selectors, tolerations, and admission policies. For example, a study could direct certain pods to H100 nodes while others utilize a different node pool, optimizing resource allocation for specific workloads.
Platform operators retain full control over storage classes, registry credentials, network policy, GPU enablement, and role-based access control (RBAC) within their clusters, ensuring security and compliance, according to NVIDIA. Many research institutions and enterprise computing environments rely on Slurm to manage shared GPU clusters, and FLARE 2.9 adds Slurm support.
The FLARE Slurm launcher submits each server or client job worker as a batch job to Slurm, which then selects the appropriate compute nodes and enforces the requested resources, including GPUs, CPU, memory, partition, account, quality of service (QoS), and time limits. Jobs can request either a single node or multiple nodes, providing scalability for complex tasks. FLARE monitors the job’s state, propagates completion or failure notifications, and cancels the allocation when necessary.
A site-owned study_runtime. yaml file further customizes the environment, mapping a study to locally available resources. This configuration file can define environment variables, secret-backed environment variables and file mounts, a site-approved default job image, Kubernetes pod templates, Docker runtime settings, and Slurm sandbox, partition, account, and QoS policies. A data scientist initiates a study-scoped session, and any jobs submitted from that session automatically inherit the study’s context, ensuring consistent resource allocation and configuration.
Organizations requiring separate PKI, infrastructure administrators, or failure isolation should deploy separate FLARE instances. Consider a pathology study connecting hospitals and a university, utilizing GPU workers and each site’s approved image datasets. In this scenario, the university might override the requested GPU count, while the hospital server overrides the GPU count to zero. Both sites, however, inherit the same CPU and memory requirements. Each launcher then translates these resolved values into native settings; Docker converts CPU units to nano_cpus and GiB to byte-valued memory limits, applying the GPU device request.
Kubernetes sets matching CPU and memory requests and limits, adding the GPU request. Slurm emits corresponding command-line arguments for CPUs, memory, and GPUs. The process is streamlined for the data scientist, who submits the job once to the FLARE server.
FLARE then deploys it to the study’s participants, with each site’s configured launcher applying the study’s local runtime defaults and creating the worker. FLARE coordinates the federated workflow and reports job status, while Docker, Kubernetes, and Slurm manage local resources. The use of launcher_spec allows for backend-specific topology and policy definitions, such as a Slurm node layout or runtime-specific image configurations. This layered approach ensures that federated learning infrastructure does not require uniformity across participants.
FLARE Studies Enable Multi-Tenant Research Isolation
FLARE 2.9 adds Slurm support for federated learning projects alongside existing Docker and Kubernetes support, enabling collaborative research across organizations with radically different infrastructure. This addition addresses a critical need for operational flexibility as federated learning initiatives move beyond initial, simplified configurations and scale to encompass more complex, real-world deployments. The ability to accommodate Slurm is particularly relevant for institutions, universities and research centers that already utilize the workload manager for their GPU clusters, eliminating a significant barrier to participation.
FLARE’s architecture is designed to maintain the availability of core federation services while dynamically provisioning job workers at each participating site, a separation that allows infrastructure owners to maintain stability in their parent environments even as researchers independently update their training environments, the company says. The system supports bare execution, Pyxis/Enroot containers, and Apptainer containers within the Slurm environment, ensuring compatibility with existing cluster management practices.
Slurm remains the authority for resource allocation, with its established accounts, associations, partitions, quality of service rules, and security protocols fully enforced. Beyond simply supporting diverse execution backends, FLARE 2.9 introduces “studies” as a mechanism for logical multi-tenancy within a single deployment, providing a secure boundary between different research teams and their associated data. Each study defines its participating client sites and authorized admin users, limiting job visibility, client status, and deployment maps to members of that specific study.
This isolation extends to the local level at each site, preventing jobs from accessing arbitrary dataset paths or supplying unverified secret values. The FLARE server verifies study membership before a job reaches a site, while the site operator maintains control over the mapping between the study and local resources. Organizations needing separate PKI, infrastructure administration, or failure domains should deploy separate FLARE instances.
This tiered approach to isolation allows for a balance between collaboration and security, enabling researchers to share resources and expertise while maintaining control over their own data and intellectual property, NVIDIA reports. Both studies can coexist within the same federation, sharing persistent services but operating with distinct configurations and access controls.
This allows researchers to specify their resource needs in a standardized format, which is then translated by each site’s launcher into native settings appropriate for its execution backend, the company’s account states. The separation of persistent federation services from dynamically launched job workers allows each site to use its existing infrastructure and operational expertise, reducing the cost and complexity of participation.
This is a significant departure from earlier approaches to federated learning, which often required organizations to adopt a common platform or invest in significant infrastructure upgrades. The documentation explains that FLARE provides participant authentication, secure communication, and authorization, while each site’s execution platform enforces local resource and workload policies. Site operators configure host and cluster security, approved images, secrets, and access controls, and validate workload isolation for their chosen runtime.
The benefits of this approach extend beyond simply reducing infrastructure costs. By allowing organizations to maintain control over their own data and compute resources, FLARE also addresses important security and privacy concerns. Data remains within the control of the organization that owns it, and access is strictly controlled through the study-based isolation mechanism. This is particularly important for sensitive data, such as patient medical records or financial information.
The system’s ability to integrate with existing security protocols and access control mechanisms further enhances its security posture. To facilitate adoption, NVIDIA provides comprehensive documentation and a publicly accessible GitHub repository for FLARE, enabling users to explore the system’s capabilities and contribute to its ongoing development, the company claims. The documentation details the configuration options for each supported execution backend, Docker, Kubernetes, and Slurm, and provides examples of how to set up and manage studies.
The GitHub repository contains the source code for FLARE, as well as tools and scripts for building, deploying, and testing the system. This open-source approach fosters collaboration and innovation, allowing researchers and developers to contribute to the advancement of federated learning technology. Peter Cnudde, an author of the project, notes that the team works with customers, partners, and research teams to enable real-world federated learning with NVIDIA FLARE, building on his more than 30 years of experience building distributed systems.
See today’s quantum computing news on Quantum Zeitgeist for the latest breakthroughs in qubits, hardware, algorithms, and industry deals.
