NVIDIA is tackling a core challenge in scaling large language models: efficiency. Rather than processing every piece of data through every layer, mixture-of-experts (MoE) architectures activate only a select few subnetworks for each task. This approach promises greater capacity, but fragmented computation has historically limited gains; NVIDIA Transformer Engine (TE) addresses these bottlenecks with optimized primitives for grouped expert computation, kernel fusion, and low-precision training. As biological foundation models grow in complexity, TE, along with techniques like the MXFP8 format and GroupedMLP kernel, offers a pathway to expand model capacity while improving GPU efficiency.
GroupedLinear Optimizes Expert Computation in MoE Models
NVIDIA’s Transformer Engine (TE) implements GroupedLinear to address inefficiencies stemming from fragmented expert kernels within mixture-of-experts (MoE) models, a common bottleneck in scaling large language models. A naive implementation of MoE models often involves iterating through each expert network in a Python loop, triggering a separate kernel launch for each, which introduces substantial overhead. TE circumvents this by enabling grouped execution, preserving individual expert matrices but submitting their computations together as a single operation, markedly improving GPU utilization, NVIDIA says.
This approach contrasts with the Hugging Face baseline, which processes each expert sequentially, creating a performance disparity. The benefits of grouped execution are further amplified when combined with NVIDIA’s MXFP8 quantization and a specialized kernel called GroupedMLP.
This fusion streamlines the computational process, reducing framework overhead and eliminating the need to materialize intermediate results, a key factor in accelerating training for biological foundation models. The TE Sequential API identifies and replaces the standard GroupedLinear, ScaledSwiGLU, and GroupedLinear sequence with the ForwardGroupedMLP_CuTeGEMMSwiGLU_MXFP8 object for the forward pass, and a corresponding fused operation for the backward pass. Beyond computational efficiency, TE tackles the challenge of large model size and activation memory requirements inherent in MoE architectures, particularly when applied to genomics workloads with long sequences.
MoE models, by design, increase total parameter capacity, which can strain activation memory during training. The API incorporates dequantization directly into the fused computational path, further optimizing memory usage. ScaledSwiGLU combines routing probabilities, often referred to as “scales,” with the expert feed-forward network computations, streamlining the process and reducing the memory footprint.
The implementation of these techniques is demonstrated through the NVIDIA BioNeMo MoE recipe, providing a practical reference for efficiently training MoE-based biological foundation models. The result is a system that not only accelerates training but also expands the capacity of these models, enabling more sophisticated and accurate biological research.
MXFP8 Precision Reduces Memory Footprint for Biological Models
Mixture-of-experts (MoE) architectures achieve greater efficiency by activating only a small subset of subnetworks, or “experts,” for each input token, a marked departure from dense transformers where every token traverses every layer. This selective activation, however, introduces computational challenges; fragmented expert computation can limit GPU utilization, and routing between experts adds communication overhead. The BioNeMo recipe uses TE to support both FP8 and MXFP8 training, significantly reducing memory requirements during model development.
Both formats utilize 8 bits to represent model weights and activations, compared to the 16 bits used by BF16, but MXFP8 employs a unique scaling granularity. Specifically, MXFP8 assigns a scaling factor to each block of 32 consecutive values, a technique designed to preserve numerical range and accuracy, critical for maintaining model performance. On NVIDIA Blackwell GPUs, MXFP8 operations are hardware-accelerated, allowing MXFP8 general matrix multiplications (GEMMs) to utilize specialized Tensor Core instructions for increased speed.
While low-precision computation offers speed and memory benefits, the need to convert between 8-bit and 16-bit formats introduces overhead. Standard training frameworks often perform quantization and dequantization as separate operations, adding computational steps. To mitigate this, TE incorporates a fused MLP path, streamlining the process by combining quantization, the SwiGLU activation function, and routing-weight scaling into a single kernel operation. This fusion eliminates the need for intermediate data transfers and reduces overall computational load.
BioNeMo Recipe Achieves 2.21x Throughput on NVIDIA B200 GPUs
The BioNeMo recipe, using NVIDIA’s Transformer Engine, achieved a 2.21x throughput increase during training on eight NVIDIA B200 Tensor Core GPUs compared to a standard Hugging Face baseline. This performance gain stems from optimizations targeting the specific challenges of mixture-of-experts (MoE) models, where only a portion of the network processes each input. The recipe’s implementation details are now publicly available, offering a practical guide for researchers and developers working with large biological foundation models.
Central to this improvement is the GroupedLinear component, which enhances expert computation within the MoE architecture. Beyond this, the BioNeMo recipe utilizes MXFP8 precision, a technique that reduces memory consumption by representing model weights and activations with 8 bits instead of the conventional 16 bits used in BF16. This reduction in memory footprint is important for scaling MoE models, which often require substantial computational resources.
The recipe’s code demonstrates how to implement this fused MLP using the Transformer Engine Sequential API, chaining together the necessary operations for efficient execution. Validation of the recipe’s effectiveness begins with a two-GPU configuration, designed to confirm the proper functioning of expert parallelism and the training environment. Once validated, users can scale to the full Mixtral-8x7B configuration, using expert parallelism with a setting of EP=8 and MXFP8 precision across eight GPUs.
The recipe README provides detailed launch, checkpoint, and benchmark commands, facilitating reproducibility and performance evaluation. Researchers are instructed to adjust data-parallel and expert-parallel sizes to match their total GPU count, optimizing resource utilization.
Sudhakar Singh US, Varun Thumbe US, Santosh Santosh US, Timur Rvachov US, and Chris Hoge US contributed to the development of the BioNeMo recipe, furthering NVIDIA’s position as a leader in accelerated computing for scientific discovery, according to the company. Peter, a machine learning engineer on the BioNeMo team, is focused on accelerating digital biology workflows in the protein and genomics space, highlighting the practical applications of this research.
NVIDIA’s cuQuantum SDK v25.11 delivers up to a 4,000x speedup in quantum simulation, utilizing new Pauli propagation and stabilizer primitives, with GB200 NVL72 systems achieving an 800x speedup over CPU implementations, the firm reports. This demonstrates NVIDIA’s broader commitment to accelerating computationally intensive workloads across diverse fields, including quantum computing and biological modeling.
Source: https://developer.nvidia.com/blog/efficient-moe-training-for-biological-foundation-models/
