Quantization in Machine Learning: Beyond Accuracy to Feature Enhancement

On April 18, 2025, Weizhi Lu, Mingrui Chen, and Weiyu Li published The Binary and Ternary Quantization Can Improve Feature Discrimination, challenging conventional wisdom by demonstrating that binary and ternary quantization enhance feature discrimination across images, speech, and text.

The study investigates how quantisation affects classification performance, challenging the assumption that higher quantisation errors always reduce accuracy. While current research focuses on quantization errors, this paper proposes evaluating feature discrimination of quantized data instead. Surprisingly, binary and ternary quantization methods improve feature separation compared to non-quantized data, despite high quantization errors. This finding is supported by classification experiments across images, speech, and text datasets, demonstrating that low-bit quantization can enhance performance rather than degrade it.

In the dynamic field of machine learning, researchers are constantly exploring methods to optimize models for efficiency and performance. Recent findings indicate that reducing the precision of numerical representations within these models can not only save computational resources but also enhance accuracy under specific conditions. This discovery challenges the conventional belief that higher precision always leads to better results.

The study investigates how quantizing features—reducing their bit depth from 32-bit floating-point numbers to fewer bits, such as binary or ternary representations—can improve classification tasks across various datasets and classifiers. By simplifying the numerical representation of data points, models can achieve unexpected gains in accuracy while maintaining efficiency.

The research focuses on quantizing feature vectors by setting a threshold based on their average magnitude. Features exceeding this threshold are assigned binary values (1 or -1) for binary quantization or extended to include a middle value (0) for ternary quantization. This approach was tested across multiple datasets, including ImageNet, CIFAR-10, MNIST, and others, using classifiers such as K-Nearest Neighbors (KNN), Support Vector Machines (SVM), Decision Trees, Random Forests, and AdaBoost.

The results were striking: quantization not only reduced computational demands but also improved accuracy in many cases. This was particularly evident when dealing with sparse or high-variance datasets, where the simplified representations helped models focus on significant features while discarding noise.

The findings suggest that quantization can be a powerful tool for optimizing machine learning models without compromising performance. The technique proved effective across different classifiers, indicating its broad applicability. This is especially promising for resource-constrained environments, where efficiency is crucial.

Moreover, the study highlights that quantization’s benefits extend beyond mere computational savings. By simplifying data representations, it can lead to more robust models capable of handling noisy or complex datasets with greater accuracy.