The volume of user history data generated daily, reaching terabytes or petabytes at industry scale, is creating a critical bottleneck for recommender systems, exceeding the capacity of even the most advanced GPU high-bandwidth memory. This challenge arises as companies shift toward generative recommenders, which model user preferences by predicting the next item in a sequence, much like large language models.

“Unlike modalities such as text or images, user histories involve a mix of categorical and continuous features that change frequently over time,” explain Elijah Soba, Santosh Bhavani, Shijie Liu and Burak Yoldemir. NVIDIA is addressing these production hurdles with new tools designed for this evolving landscape.

Generative Recommenders Reframing RecSys with Sequence Modeling

This reframing aims to model the probability distribution of the next item given user history, expressed as P(next_item | user_history), and allows for the potential unification of retrieval and ranking within a single model. Two primary approaches are currently driving this architectural shift: Hierarchical Sequential Transduction Units (HSTU) and Semantic IDs. HSTU, introduced by Meta in 2024, establishes a generative objective for RecSys and introduces innovations for efficient production-scale training and serving; it represents user data as a sequence of items and actions ordered by timestamp.

Semantic ID GR architectures, used in systems like TIGER, PLUM, and OneRec v1/v2, utilize Semantic IDs as a foundation for scalable, autoregressive recommendation, directly generating recommendations instead of searching embedding spaces. This direct generation enables methodologies like beam search to produce multiple Semantic IDs in a single forward pass, potentially selecting niche items within a cluster and improving throughput.

NVIDIA is addressing the production challenges introduced by this shift with the recsys-examples repository, a collection of examples demonstrating best practices for training and deploying generative recommenders on NVIDIA GPUs using PyTorch, the company says. Unlike LLM workloads which may tolerate autoregressive decoding latency, RecSys models require frequent retrieval and ranking of thousands of candidate items within milliseconds, a demand that necessitates specialized tools. The nv-embedding-cache library complements recsys-examples, specifically designed to accelerate the retrieval of embeddings critical to generative recommendation performance.

An example of how to perform inference with these two libraries is available for review, offering quick-start guides and detailed information for developers. This move beyond traditional embedding-similarity-based objectives is not merely architectural; it represents a fundamental change in how recommendations are generated and served, potentially unlocking new levels of personalization and efficiency in consumer internet applications.

HSTU Architecture: Efficient Attention for User History

The sparse nature of user-item interaction data presents a significant hurdle for modern recommender systems, as the sheer volume of items often far exceeds the number of users, limiting the signal available for training on niche products. This means that despite representing genuine user preferences, a vast majority of items receive insufficient training data, hindering the accuracy of predictions for less popular choices. New users and items further exacerbate this issue, lacking initial interaction history needed to generate quality embeddings.

Meta’s introduction of Hierarchical Sequential Transduction Units (HSTU) in 2024 offered a new approach by framing recommender systems as a generative objective, enabling efficient training and deployment at scale. This formulation draws a parallel to next-token prediction in large language models, allowing for meaningful learning signals both within and across training batches.

The HSTU architecture modifies standard Transformer attention by replacing softmax normalization with SiLU-based weighting, incorporating relative attention bias, and applying elementwise gating before output projection. NVIDIA’s recsys-examples repository provides optimized implementations of both HSTU and semantic ID models, covering the entire workflow from training to inference, and consolidates modular components for dynamic embedding management and efficient CUDA operations, according to the company. Traditional embedding tables, which assume a fixed vocabulary, struggle to accommodate the continuous influx of new users and items, quickly exceeding the capacity of even high-bandwidth memory.

To address this, the repository incorporates DynamicEmb, a system for managing embedding layers with dynamic capacity and caching, alongside a KV cache and storage manager specifically tailored for recommender systems. During training, the integration of TorchRec, DynamicEmb, and Megatron-Core enables coordinated sharding and parallelism across both embedding and dense modules.

The pipeline also incorporates dynamic shuffling to balance workloads, overlaps embedding communication with computation, and includes a HSTU layer with fused operations and optimized attention kernels for NVIDIA Ampere, Hopper, and Blackwell GPUs. According to NVIDIA, these optimizations improved end-to-end Model FLOP Utilization (MFU) to 31.40% from 7.65% on two DGX H100 nodes, demonstrating substantial gains in training efficiency. The modular design of the library allows for plug-and-play compatibility with custom architectures, offering flexibility for researchers and developers. Prefetching techniques further enhance performance by keeping frequently accessed embeddings resident in high-bandwidth memory for rapid access.

Semantic IDs Enable Scalable Autoregressive Recommendation

Generative recommenders are shifting away from reliance on geometric similarity, instead modeling user-item preference as a sequence prediction problem akin to large language models, a change that introduces unique serving challenges. This demand for speed necessitates new approaches to serving these models efficiently. Traditional RecSys architectures often struggle with the long user context and large beam widths characteristic of Semantic ID-based generative recommenders. A single request can involve thousands of historical tokens, yet require decoding only two or three Semantic ID tokens, coupled with beam widths of 128 or 256 to enhance recommendation diversity.

Existing LLM serving systems, optimized for multi-user chat with paged KV cache and dynamic batching, are not ideally suited to this pattern. NVIDIA addresses this mismatch with tools designed to complement existing HSTU and embedding components, specifically targeting autoregressive Semantic ID generation with strict latency requirements.

Semantic ID-based generative recommenders differ fundamentally from traditional approaches by directly generating recommendations through autoregressive decoding, rather than searching within an embedding space. “HSTU and DynamicEmb address production-scale training and embedding-heavy inference, while the Semantic ID-GR inference path targets autoregressive Semantic ID generation with long context, large beam search, and strict recommender-system latency requirements,” according to NVIDIA documentation.

Strict latency requirements further complicate matters, as even small increases in processing time can negatively impact user engagement. These new architectures, however, are not without their challenges. Deriving user preferences from a smaller set of features can negatively impact recommendation trajectories, and requires careful consideration of feature engineering and model design. The interplay between efficient training, embedding management, and autoregressive decoding is essential for realizing the full potential of generative recommenders at scale.

DynamicEmb: GPU-Optimized Embedding Tables for RecSys

This creates a critical bottleneck as models struggle to represent the long-tail of niche items, despite these representing genuine user preferences, and training data offers limited signal for them. A core difficulty arises from skewed interaction data, where popular items dominate, leaving sparse information for the majority of the catalog. NVIDIA addresses this through DynamicEmb, a GPU-optimized scored hash table replacing traditional static embedding tables, the firm reports.

Unlike fixed-vocabulary approaches, DynamicEmb allocates rows only for feature IDs the model actively uses, allowing the table to expand beyond a single GPU’s capacity by utilizing both HBM and pinned host memory. This on-demand allocation avoids the waste of over-provisioning or the performance degradation of under-provisioning, both common issues with static tables in rapidly evolving recommender systems. The recsys-examples repository integrates DynamicEmb with TorchRec for managing item, user, action, and contextual embedding tables.

For dense layers, the HSTU backbone utilizes Megatron-Core, enabling single training runs to use data, tensor, sequence, and pipeline parallelism. Sharding across devices is handled via CUDA virtual memory, permitting a single logical embedding table to span multiple GPUs or nodes. According to NVIDIA documentation, and a lockless invalidate-and-commit protocol ensures concurrent lookup and cache modifications on the GPU without stalling the lookup stream. The nv-embedding-cache (NVE) SDK further accelerates large-scale embedding table lookups and operations, offering NVEmbedding and NVEmbeddingBag as drop-in replacements for standard PyTorch embedding layers.

This seamless integration simplifies adoption within existing PyTorch ecosystems. Hierarchical memory look-up within NVE-supported embedding tables optimizes performance for complex recommendation scenarios.

NVIDIA recsys-examples: Training and Deployment Stack for GRs

NVIDIA’s recsys-examples repository offers a solution by integrating tools designed to manage this influx and facilitate the shift toward generative recommenders. These systems, unlike traditional approaches, directly generate recommendations through autoregressive decoding of Semantic IDs rather than searching embedding spaces, and naturally rank outputs via logits. For inference, the stack is engineered to meet stringent low-latency requirements, supporting PyTorch AOTInductor to execute models in the Torch C++ runtime while maintaining compatibility with NVIDIA Triton Inference Server.

The nv-embedding-cache (NVE) SDK further optimizes performance by keeping frequently accessed embeddings close to the GPU, and computation is reduced through a customized FlexKV-enabled KV cache that distributes entries across multiple memory tiers. Because these layers function as standard nn. Module components, they integrate into existing recommender models with minimal graph changes. Recsys-examples DynamicEmb tables are also supported in NVE, simplifying the transition between training and inference.

Source: https://developer.nvidia.com/blog/how-generative-recommenders-are-redefining-recsys-at-scale/

More like this