Diffusion AI: Quantization Boosts Edge Device Performance

Diffusion-based large language models, or dLLMs, represent a new approach to natural language generation, but their substantial size hinders deployment on everyday devices. Haokun Liu and colleagues at the University of Science and Technology of China, along with collaborators, address this challenge by systematically investigating post-training quantization, a compression technique commonly used for other types of large language models. The team discovered that dLLMs exhibit unique activation outliers, extreme values that complicate the quantization process and threaten accuracy, and they evaluated several state-of-the-art quantization methods to overcome this issue. This comprehensive study, which considers different bit-widths, tasks, and model variants, provides crucial insights into effectively compressing dLLMs, paving the way for more efficient and accessible natural language processing applications.

Diffusion-based language models represent an alternative to autoregressive models for generating text, utilizing full attention and a decoding strategy based on removing noise. However, deploying these models on devices with limited resources remains challenging due to their substantial size and computational demands. Post-training quantization (PTQ), a technique for compressing models without further training, has proven effective for conventional language models, but its application to diffusion-based models has received limited attention. This research presents a systematic study of quantizing diffusion-based language models, beginning by identifying the presence of activation outliers, unusually large activation values that dominate the model’s dynamic range, which pose a significant challenge to effective quantization.

Post-Training Quantization Accuracy Improvements

Recent research focuses heavily on reducing the size and increasing the speed of large language models, with quantization being a dominant theme. Researchers are investigating various quantization techniques, including post-training quantization and quantization-aware training, and combining them with other techniques like pruning and knowledge distillation to achieve greater efficiency gains. A strong emphasis is placed on practicality and efficiency, making models smaller, faster, and more deployable, especially for resource-constrained environments. Mitigating accuracy loss during quantization is a major challenge, and significant effort is devoted to making diffusion models faster and more memory-efficient.

Several key trends emerge from this body of work, including exploring discrete diffusion processes as an alternative to continuous diffusion for faster inference. Techniques that consider the distribution of weights and activations improve quantization accuracy, and preference optimization techniques improve the quality of generated content. Researchers are also combining language with other modalities, such as vision, leading to multimodal models, and reducing the number of tokens processed to improve efficiency. Specific papers of note include SmoothQuant, Llama 2, Vidit-Q, MixDQ, Fast-DLLM, and LLADA 1. 5. Future research areas include automated quantization, hardware-aware quantization, combining quantization with sparsity, improving the robustness of quantized models, scaling diffusion models, and developing more effective compression techniques for multimodal models.

Diffusion Models Compress Effectively with Quantization

Recent research demonstrates that diffusion-based large language models (dLLMs) can be effectively compressed using post-training quantization (PTQ). This is particularly important because dLLMs, while promising, currently demand significant computational resources. The study systematically investigates how well existing PTQ methods, originally developed for conventional LLMs, translate to the unique characteristics of dLLMs. A key finding is the presence of “activation outliers” within dLLMs, where certain activation values are disproportionately large and dominate the dynamic range, hindering the ability to accurately represent the majority of values.

Researchers evaluated a range of PTQ methods, discovering that GPTQ consistently outperforms AWQ across most tasks. For methods that quantize both weights and activations, rotation-based approaches like DuQuant and QuaRot show clear advantages over SmoothQuant. The study also reveals that the performance of these quantization methods varies depending on the complexity of the task, with instruction-tuned LLaDA models demonstrating greater robustness to quantization compared to their base counterparts. Overall, this research provides valuable guidance for deploying efficient and practical dLLMs, paving the way for their wider adoption in resource-constrained environments.

Quantization Improves Diffusion Language Model Efficiency

This work presents the first comprehensive investigation into applying post-training quantization to diffusion-based language models, offering valuable insights into compressing these models for more efficient deployment. Researchers identified that activation outliers pose a significant challenge to low-bit quantization. Through extensive evaluation of various quantization methods, the study demonstrates that GPTQ and DuQuant perform particularly well under constrained conditions. The findings also reveal that quantization behaviour is not uniform, varying depending on the specific task and model type, with instruct-tuned models exhibiting greater robustness.

These results provide practical guidance for designing effective quantization strategies tailored to different diffusion language models and applications. While acknowledging the challenges posed by activation outliers, the authors suggest future research should explore step-aware quantization levels and investigate different remasking strategies to further optimise performance. The team intends to release their code and implementation details to facilitate continued development and deployment of quantized diffusion language models in resource-constrained environments.