Nemotron 3.5 Lightning illustrates that large language models do not necessarily require activating all parameters with each processing step; the 30 billion-parameter model functions by activating just 3 billion parameters per token. This is achieved through a Mixture-of-Experts (MoE) architecture, which selectively routes incoming text to a subset of its networks, the company says. “How a model organizes its parameters matters as much as how many it has,” highlighting that parameter organization impacts throughput and memory cost more than sheer parameter count. Like engines with varying cylinder activation, the approach prioritizes efficiency without sacrificing capacity.
MoE Architecture Activates Fewer Parameters in Nemotron 3.5 Lightning
Nemotron 3.5 Lightning activates only three billion parameters per token despite possessing a total of 30 billion parameters, a demonstration of efficient scaling within Mixture-of-Experts architectures. This selective activation contrasts sharply with dense models where every parameter engages with each input, influencing both computational demands and achievable throughput. The architecture’s design prioritizes processing speed by strategically routing tokens to a focused subset of its extensive parameter set.
This approach to parameter management directly impacts serving throughput, as evidenced by a comparison with other models; Gemma 4 31B and Nemotron 3.5 Lightning both possess approximately 30 billion total parameters, yet their output speed ranges differ significantly across NVIDIA GPU providers. Artificial Analysis Intelligence Index data, retrieved August 31, 2026, reveals Lightning achieves between 235.7 and 494.2 tokens per second, while Gemma 4 operates between 30 and 36.9 tokens per second.
The difference isn’t solely attributable to the MoE architecture; Lightning also incorporates Mamba-2 layers and speculative decoding techniques that independently contribute to performance gains. This efficiency extends to cost as well, with Lightning delivering output at $0.22 per million tokens, a substantial reduction compared to Qwen3.8-27B’s $3.00 and Mistral Small 4’s $0.60, according to the same Artificial Analysis data. While Lightning’s general capability scores less favorably than some dense models, the trade-off between speed, cost, and overall performance highlights the nuanced considerations in selecting an AI architecture.
The internal routing process within the MoE model directs incoming tokens through a limited selection of feed-forward network blocks, known as experts, rather than engaging all parameters. While tokens navigate only a subset of the FFN blocks, they still pass through the full attention mechanism, ensuring comprehensive contextual understanding.
Modern inference frameworks facilitate this routing without discarding tokens, though all expert networks must reside simultaneously in GPU memory. This memory requirement is a key consideration, as it can limit the space available for the KV cache, a component that helps maintain context during extended interactions. The choice between dense and MoE architectures, therefore, hinges on deployment constraints and the balance between capacity, throughput, and resource management.
The concept resembles two engines with equivalent displacement; one activates all cylinders with each cycle, while the other selectively fires only those necessary for the current load. A model card specifies encompasses both the attention and embedding weights for each token, alongside the weights of the selected FFN blocks. Variants of MoE exist, including the NVIDIA model card for Nemotron 3.5 Lightning, which details a Mamba-2 + MoE + Attention hybrid configuration.
This hybrid approach demonstrates that combining different architectural elements can further optimize performance and efficiency. The data from Artificial Analysis shows that Lightning delivers four to five times Qwen3.8-27B’s output speed at approximately one-fourteenth of the price, illustrating the potential benefits of this combined strategy.
Despite achieving higher throughput and lower costs, Lightning’s general capability scores are lower, suggesting that the optimal architecture depends on the specific application and desired balance of performance characteristics. The observed differences in VRAM usage, Lightning requiring approximately 60 GB compared to Gemma 4’s 61 GB and Qwen3.8-27B’s 56 GB, further underscore the complex interplay between model architecture, memory footprint, and performance.
Dense vs. MoE: Parameter Usage and Performance Differences
The ability to achieve substantial performance with a fraction of activated parameters defines a key distinction between dense and Mixture-of-Experts (MoE) model architectures. Nemotron 3.5 Lightning illustrates this principle by activating approximately three billion parameters within its total thirty billion, a strategy that allows for increased throughput without necessarily maximizing computational cost for each token processed, according to Mixture-of-Experts. The choice between these approaches hinges on balancing computational efficiency with deployment simplicity and the capacity to manage increased memory requirements.
Dense models, characterized by their uniform parameter activation, offer a straightforward deployment process and predictable latency. Each of the twenty-seven billion parameters in a twenty-seven billion parameter model is utilized for every token, channeled through a single feed-forward network block per decoder layer. MoE models, conversely, replace this singular network with multiple expert blocks, routing incoming tokens through a carefully selected subset.
This decoupling of memory, impacting VRAM, from compute, influencing FLOPs per token, is a fundamental difference. The source details a comparison of model architecture, total parameters, active parameters, VRAM usage in both native and 4-bit quantization, and performance metrics including output speed and cost per million outputs. The internal routing mechanism within an MoE model is a critical component, susceptible to minor perturbations that can alter routing decisions.
The “recipes” within the model prioritize the router, embeddings, and output head, while hybrid-attention models rely on recurrent projections to influence routing. Quantized Qwen3.8-27B, for example, maintains its linear-attention block in BF16 precision specifically to stabilize these routing decisions. This highlights the nuanced considerations involved in optimizing MoE models, where seemingly small adjustments can have a significant impact on performance and stability.
Ultimately, the selection between dense and MoE architectures represents a trade-off between capability per parameter and compute per token. Dense models prioritize simplicity and full parameter utilization, easing fine-tuning and serving processes. MoE models, however, offer the potential for greater capacity and throughput when memory and serving complexity can be effectively managed. The choice, therefore, is dictated by specific deployment constraints and the desired balance between efficiency, complexity, and capability. This level of detail underscores the intricate engineering involved in optimizing these architectures for specific workloads and hardware configurations.
MoE Routing Decouples Memory from Compute for Increased Throughput
MoE routing fundamentally alters the relationship between a large language model’s memory requirements and computational demands, enabling increased throughput by activating a surprisingly limited number of parameters per token. This selective activation isn’t merely a theoretical optimization; it’s a functional characteristic of the architecture, illustrated by the performance of Lightning and similar models. This decoupling stems from the core design of Mixture-of-Experts (MoE) models, which maintain multiple expert networks but strategically route each token through only a select few.
While dense models see all parameters engaged with every token, MoE models effectively distribute the computational load, reducing the per-token processing requirements. The result is a shift from variable per-token compute, where the cost scales with the number of activated parameters, to fixed memory costs, as idle experts require storage but no active processing.
The source material states, “When every expert loads into memory the compute is paid per token and scales only with the experts that fire.” This is particularly advantageous at a batch size of one, where memory availability, rather than computational power, often becomes the limiting factor. The benefits of this approach extend beyond simple speed gains; MoE architectures offer a distinct advantage in managing resource allocation.
A dense model’s hosting and inference costs scale proportionally, meaning increased capacity directly translates to increased computational demands. MoE models, however, break this link, allowing for a larger overall model capacity without a corresponding increase in per-token processing. As batch sizes increase, the advantage narrows as more experts become engaged, but the reduction in work per token persists, maintaining a throughput benefit across various workloads. The impact of MoE routing is particularly visible when comparing models of equivalent total parameter size.
Gemma 4 31B and Nemotron 3.5 Lightning, both around 30 billion parameters, exhibit significantly different serving throughputs, highlighting the practical implications of sparse activation. The source material notes that these models’ output-speed ranges do not overlap, demonstrating the tangible performance gains achievable through MoE. However, achieving optimal performance requires careful attention to detail; the router, responsible for directing tokens to the appropriate experts, is a critical component.
The nuanced engineering involved in optimizing MoE models is further illustrated by the example of quantized Qwen3. Recurrent projections within hybrid-attention models also influence routing, underscoring the interconnectedness of various architectural elements.
See today’s quantum computing news on Quantum Zeitgeist for the latest breakthroughs in qubits, hardware, algorithms, and industry deals.
