Faster Text Generation via Gaussian Diffusion and Discrete Consistency.

Researchers present Duo, a novel approach enhancing uniform-state discrete diffusion models for faster text generation. By recognising the connection to Gaussian diffusion, they employ curriculum learning to accelerate training and Discrete Consistency Distillation to achieve two orders of magnitude faster sampling, surpassing autoregressive models on several benchmarks.

The efficient generation of coherent text remains a central challenge in artificial intelligence, with current methods often balancing speed and quality. Researchers are continually seeking to refine diffusion models, a class of generative models that learn to create data by progressively reversing a process of adding noise. A new approach, detailed in work by Sekhar Sahoo, Deschenaux, Gokaslan, Wang, Chiu, and Kuleshov, addresses limitations in discrete diffusion models, specifically their tendency to underperform autoregressive and masked diffusion techniques. Their research, entitled ‘The Diffusion Duality’, demonstrates how insights from continuous Gaussian diffusion – a related but distinct methodology – can be transferred to improve both the training and sampling processes of discrete diffusion models, resulting in faster generation and enhanced performance on established language benchmarks.

Recent research details a notable enhancement in uniform-state discrete diffusion models for text generation, substantially reducing the performance disparity previously observed when contrasted with established autoregressive and masked diffusion models. Researchers achieve this through Duo, a novel methodology that exploits the relationship between uniform-state and Gaussian diffusion processes, and demonstrate a marked improvement in the model’s capacity to predict and generate coherent text even without prior exposure to specific training data. The central tenet of this work involves adapting established techniques from Gaussian diffusion to refine both the training and sampling stages.

Discrete diffusion models, which operate on distinct, separate data points rather than continuous values, have historically underperformed their continuous counterparts. The team introduces a curriculum learning strategy, informed by the principles of Gaussian diffusion, which demonstrably doubles training speed by minimising variance during the learning process and achieves improved zero-shot perplexity—a measure of how well a probability model predicts a sample—on three of seven benchmark datasets. Perplexity is lower for better models. Furthermore, the researchers present Discrete Consistency Distillation, an algorithm that adapts consistency distillation—a technique originally developed for continuous data—to the discrete setting. Consistency distillation improves the efficiency of diffusion models by training them to produce consistent outputs across different noise levels. This adaptation unlocks few-step generation within diffusion models and accelerates the sampling process by two orders of magnitude, significantly reducing computational cost.

The research acknowledges that uniform-state discrete diffusion is not fundamentally distinct from Gaussian diffusion, but rather represents a specific instance of it. This realisation allows for the transfer of established techniques to enhance both training and sampling procedures. The team validates these advancements through rigorous experimentation and provides all code and model checkpoints publicly, facilitating reproducibility and further research in the field of discrete diffusion modelling. These combined improvements position discrete diffusion models as a more competitive and efficient approach to text generation, narrowing the gap with established autoregressive techniques. The successful implementation of these techniques highlights the benefits of bridging the gap between continuous and discrete diffusion modelling approaches, and by leveraging insights from Gaussian diffusion, the researchers have not only improved the performance of uniform-state models but also paved the way for further exploration of hybrid approaches.