MicroMix Quantization and Kernel Optimisation Unlock Blackwell’s FP4 Tensor Core Speedup

Large language models are transforming artificial intelligence, but their immense size demands increasingly efficient processing techniques. Researchers, including Wenyuan Liu, Haoqian Meng, and Yilun Luo from Tianjin University, alongside colleagues, address this challenge with MicroMix, a novel approach to quantization that significantly accelerates model inference. The team’s work centres on reducing the precision of the numbers used within these models, allowing for faster calculations without substantial loss of accuracy. MicroMix introduces a flexible system using custom ‘microscaling’ formats, specifically designed to leverage the capabilities of new hardware architectures, and delivers at least 20% faster execution than existing methods on both consumer and server-grade graphics cards. This advancement promises to make large language models more accessible and practical for a wider range of applications, from everyday devices to large-scale data centres.

Weights and activations are increasingly converted to the INT4 format. Although the new FP4 Tensor Cores in NVIDIA’s Blackwell architecture offer up to 4× speedup over FP16, existing INT4-based kernels fail to fully exploit this capability due to mismatched data formats. To bridge this gap, researchers propose MicroMix, a co-designed mixed-precision quantisation algorithm and matrix multiplication kernel based on Microscaling (MX) data formats. Tailored for the Blackwell architecture, the MicroMix kernel supports arbitrary combinations of MXFP4, MXFP6, and MXFP8 channels, and produces BFloat16 outputs. To achieve a favourable trade-off between accuracy and efficiency for each linear layer, the team introduces a novel quantisation approach.

Microscaling and Mixed-Precision Quantization for LLMs

This research details quantization techniques for large language models, focusing on microscaling and mixed-precision to optimize performance and reduce memory usage. Microscaling uses a combination of different precisions, such as FP4, FP6, and FP8, within the model to fine-tune the quantization process, while mixed-precision quantization uses different precisions for different parts of the model. The research demonstrates that MicroMix achieves competitive or superior results compared to existing quantization methods. The research explains concepts like symmetric and asymmetric quantization, and how to evaluate performance using metrics like zero-shot accuracy and perplexity.

Techniques like Hadamard transformation, activation sorting, and reorder-and-quantize are also discussed as methods to improve quantization accuracy. A theoretical analysis of quantization error and its impact on model accuracy is provided, followed by a description of the experimental setup, models used, datasets, and evaluation metrics. The results section presents a comparison of Microscaling with other quantization methods. The key findings demonstrate that Microscaling achieves competitive or superior results, and that the combination of different precisions is crucial for optimal performance. The research provides insights into the trade-offs between accuracy and performance, and offers detailed experimental results and implementation details. The potential implications of this work include reduced memory footprint, accelerated computation, improved energy efficiency, and wider adoption of large language models.

MicroMix Accelerates LLMs with Mixed Precision

Researchers have developed MicroMix, a new quantization technique that significantly accelerates the performance of large language models while maintaining accuracy. Quantization reduces the precision of the numbers used in calculations, allowing for faster processing and reduced memory usage. MicroMix addresses the challenge of accuracy loss by intelligently balancing precision levels within the model, achieving substantial speedups without compromising results. The method focuses on efficiently utilizing the advanced FP4 Tensor Cores, unlocking their full potential for large language model inference.

MicroMix introduces a mixed-precision approach, dynamically assigning different levels of precision, 4, 6, or 8 bits, to various parts of the model based on their sensitivity to quantization error. Unlike existing methods that apply a fixed precision allocation, MicroMix adapts to the unique characteristics of each layer, ensuring critical components retain sufficient precision for accurate calculations. The system identifies activation elements prone to significant error with lower precision formats and automatically allocates higher precision channels to preserve accuracy. A key innovation is the integration of a reorder step directly into the quantization process, enabling high-throughput calculations even with varying precision levels.

Testing on both consumer and server-grade graphics cards demonstrates substantial performance gains, with kernel-level computation accelerated by up to 46% compared to existing methods like TensorRT-FP8. Across various large language models, including Llama and Qwen models, MicroMix consistently improves both prefill latency and memory efficiency. Layer-wise execution speed increases by up to 29%, and end-to-end throughput improves by up to 9%, translating to faster response times and reduced computational costs.

MicroMix Accelerates LLMs with Mixed Precision

MicroMix presents a co-designed algorithm and kernel for quantizing large language models, achieving accelerated inference through mixed-precision techniques. The method intelligently combines MXFP4, MXFP6, and MXFP8 formats, selectively applying higher precision where necessary to minimise accuracy loss during the quantization process. Results demonstrate that MicroMix consistently improves both prefill latency and memory efficiency across various Llama and Qwen models, and delivers at least 20% faster execution compared to existing TensorRT-FP8 baselines on both consumer and server-grade GPUs. While the algorithm effectively balances accuracy and efficiency, future work could explore automated calibration strategies or investigate the application of MicroMix to other model architectures, further expanding its potential impact on the field of efficient deep learning.