Block Rotation Achieves Accurate W4A4 Quantization for MXFP4, Enabling Efficient Large Language Model Deployment

The increasing size of large language models presents significant challenges for efficient deployment, demanding innovations in model compression techniques. Yuantian Shao, Peisong Wang, and Yuanteng Chen, along with colleagues, address this issue by investigating post-training quantization, a method for reducing model precision. Their work focuses on the emerging MXFP4 format, a low-precision standard gaining hardware support from companies like AMD and Intel, and reveals a fundamental incompatibility between existing rotation-based quantization methods and this new format. The team identifies that this conflict stems from a mismatch between MXFP4’s scaling properties and the way rotation methods handle outlier data, and they propose a simple block rotation strategy that successfully adapts these methods for use with MXFP4. This advancement leads to substantial improvements in accuracy across a range of large language models, offering practical guidance for developers and establishing a foundation for future research into low-precision model quantization.

Quantization Benchmarks Across Large Language Models

This document details experimental results comparing several quantization methods, BINT4, GPTQ, SpinQuant, and BRQ, against a baseline, across multiple Large Language Models (LLMs): Mistral 7B, Qwen2. 5 (1. 5B, 3B, 7B), and Llama2/Llama3 (7B, 13B). Performance was evaluated using benchmarks including Wiki, WG, PIQA, OBQA, ARC-E, and ARC-C. The analysis reveals key performance trends and insights.

Overall, BRQ consistently delivers strong performance, frequently achieving the best or near-best results across models and benchmarks. SpinQuant also proves to be a strong contender, consistently showing good performance, often comparable to BRQ, and frequently exceeding the baseline. GPTQ and SmoothQuant generally perform competitively, often surpassing the baseline, but typically falling slightly behind BRQ and SpinQuant. BINT4 and RTN generally underperform, consistently showing the lowest accuracy and may be suitable only for extremely resource-constrained environments. Specifically, BRQ and SpinQuant consistently outperform all other methods on Mistral 7B.

Across all sizes of the Qwen2. 5 model, BRQ and SpinQuant also consistently perform well, with a more pronounced performance gap between the best and worst methods in the smaller models. Similarly, BRQ and SpinQuant consistently demonstrate the best performance on Llama2/Llama3 (7B, 13B). BRQ and SpinQuant consistently excel on Wiki and WG, while generally outperforming other methods on PIQA and OBQA, though with smaller differences. They consistently achieve the best performance on ARC-E and ARC-C, indicating their effectiveness in reasoning tasks.

For optimal performance, BRQ and SpinQuant are the recommended quantization methods, consistently delivering the best results across various models and benchmarks. GPTQ and SmoothQuant are good alternatives if BRQ and SpinQuant are too complex or resource-intensive. BINT4 and RTN should be avoided unless extremely limited resources are available. The choice of quantization method may depend on the specific model and application, so evaluating performance on the target model and benchmarks is crucial.

MXFP4 Quantization Benchmarking and Rotation Incompatibility

Scientists established a comprehensive benchmark for evaluating post-training quantization (PTQ) methods under the MXFP4 format, a new FP4 format gaining hardware support from NVIDIA, AMD, and Intel. The study systematically categorized existing PTQ techniques into compensation-based, transformation-based, and optimization-based groups, then rigorously evaluated representative methods within each category to assess their performance with MXFP4 quantization. This evaluation revealed significant variations in accuracy and highlighted the incompatibility of rotation-based approaches when applied to the MXFP4 format. Researchers traced this incompatibility to a fundamental mismatch between MXFP4’s power-of-two block scaling and the energy redistribution inherent in global rotation methods.

MXFP4 employs shared block-scale factors to effectively manage outliers, while rotation techniques attempt to mitigate outliers by dispersing their energy across all channels, creating a conflict when combined. Building on this insight, the team proposed a novel block rotation strategy designed to adapt rotation-based methods specifically for use with MXFP4. This new strategy, easily integrated into existing rotation schemes, groups rotations to align with MXFP4’s block scaling, substantially improving PTQ accuracy across diverse large language models. Experiments demonstrated that the proposed block rotation strategy effectively addresses the limitations of traditional rotation methods under MXFP4, delivering significant improvements in quantization performance. This work provides clear guidance for practitioners selecting effective quantization methods and establishes a foundation for future research focused on optimizing PTQ under emerging low-precision formats like MXFP4.

MXFP4 Quantization Reveals Rotation Method Limitations

This work presents a comprehensive benchmark of post-training quantization (PTQ) methods applied to the MXFP4 format, a new FP4 format gaining hardware support from companies like AMD and Intel. Researchers systematically evaluated existing PTQ techniques, categorizing them into compensation-based, transformation-based, and optimization-based approaches, to determine their effectiveness under MXFP4. Results demonstrate that GPTQ consistently delivers strong performance, while rotation-based methods suffer significant incompatibility with MXFP4. Investigations revealed a fundamental mismatch between MXFP4’s power-of-two block scaling and the outlier energy redistribution attempted by global rotation methods.

Specifically, MXFP4 suppresses outliers using shared block scales, while rotation aims to distribute their energy across all channels, leading to performance collapse when combined. To address this, scientists proposed a block-wise rotation strategy, adapting rotation-based methods to MXFP4 and achieving substantial accuracy improvements across diverse large language models. The team established a W4A4 quantization benchmark, systematically categorizing existing PTQ methods and evaluating their limitations under the new MXFP4 format. Experiments showed that methods like SmoothQuant effectively redistribute data to reduce the impact of extreme values, while GPTQ consistently performed well. Furthermore, the block-wise rotation strategy, integrated into existing rotation schemes, substantially improved PTQ accuracy across multiple models and tasks, offering clear guidance for practitioners and establishing a direction for future optimization efforts. This research provides a foundation for advancing PTQ research under emerging low-precision formats like MXFP4.

Block Rotation Improves MXFP4 Quantization Performance

This research presents a comprehensive evaluation of post-training quantization methods when applied to a new FP4 format, MXFP4, designed for efficient deployment of large language models. Through systematic benchmarking, scientists discovered that existing rotation-based quantization techniques, widely used for model compression, perform poorly with MXFP4 due to a fundamental incompatibility between the format’s block-wise quantization and the way rotation redistributes energy within the model. The team traced this issue to the power-of-two scaling inherent in MXFP4, which amplifies the conflict. To address this limitation, researchers developed a block rotation strategy, termed BRQ, which adapts rotation-based methods for compatibility with MXFP4. Results demonstrate that BRQ significantly improves quantization performance across diverse large language models, while also reducing the additional computational cost associated with rotation during inference. This work provides clear guidance for practitioners seeking to deploy these models on emerging low-precision hardware.