Feature Projection Learning Boosts Image Classification

Researchers are tackling the challenge of efficiently adapting powerful vision-language pre-trained models, such as CLIP, to new tasks. Yi Zhang, Weicheng Lin, and Liang-Jie Zhang, all from Shenzhen University, present a novel approach called Feature Projection Learning (FPL) which overcomes limitations in performance, parameter count, and training time seen in existing methods. Their work reframes image classification as a feature projection problem, learning to project class prototypes directly into image feature space and reconstruct image features , significantly boosting accuracy and achieving state-of-the-art results. This innovative technique promises to unlock the full potential of these models for a wider range of applications.

By pooling features from multiple images within the same class into a unified representation, the team created robust “class prototype features” that capture essential visual characteristics. For each query image, the model attempts to reconstruct the image’s feature map using these projected prototypes, with the reconstruction error serving as a key indicator of class similarity. This approach cleverly leverages the inherent structure of the feature space, making it easier to distinguish between images belonging to different classes and improving overall classification accuracy.

Experiments demonstrate that FPL delivers superior accuracy, surpassing current state-of-the-art methods by a substantial margin, achieving up to a 5.1% improvement in few-shot learning and a 4.3% increase in domain generalization performance. This streamlined approach not only accelerates the adaptation process but also reduces computational costs, making it a practical solution for real-world applications. Furthermore, the team’s innovative use of feature map reconstruction, incorporating spatial details and eliminating irrelevant location-specific information, enhances the model’s ability to generalise to unseen domains. By assessing the reconstruction of the entire query image feature map, FPL captures a more holistic representation of visual information, leading to more robust and accurate predictions. The final output of FPL is a refined prediction derived from a combination of the projection model’s output and the original, pre-trained CLIP model, ensuring a balance between learned adaptation and preserved foundational knowledge. This work opens exciting possibilities for deploying advanced vision-language models in diverse applications with limited supervision and resources.

Class Prototype Feature Projection for Adaptation improves domain

This reconstruction process leverages spatial details within the feature map and intentionally disregards irrelevant location-specific information, enhancing robustness. Experiments employed a ridge regression approach to solve the feature map projection, achieving an efficient closed-form solution with a single learned constraint. This technique bypasses the need for iterative optimization, significantly reducing training time and computational demands. The study pioneered a method for calculating this solution directly, avoiding complex parameter tuning and streamlining the adaptation process. Furthermore, the final prediction combines the output of this projection model with the original, pre-trained CLIP model, leveraging the strengths of both approaches. Comprehensive evaluations demonstrate that FPL delivers superior accuracy, substantially surpassing the performance of current state-of-the-art methods. Results demonstrate that reconstructing images from the same class is demonstrably simpler, due to shared embeddings, while images from different classes present a greater reconstruction challenge, leading to larger errors. Assessing the reconstruction of the entire feature map preserves crucial spatial details and eliminates irrelevant location-specific information, enhancing the model’s discriminatory power. Measurements confirm that FPL outperforms existing methods by up to 5.1% on few-shot learning benchmarks.

Furthermore, the study recorded a 4.3% improvement in domain generalization capabilities, showcasing the model’s ability to adapt to unseen data distributions. The breakthrough delivers a parameter-efficient solution, employing a ridge regression task to compute a closed-form solution with a single learned constraint. This eliminates the need for extensive parameter tuning or increased computational demands, making FPL a practical and scalable approach. Scientists innovatively approached feature map projection as a ridge regression task, enabling efficient computation of solutions. The final output of FPL is a combined prediction from both the projection model and the original pre-trained CLIP, leveraging the strengths of both approaches. Tests prove that this combination consistently yields higher accuracy and robustness. Empirical evaluations demonstrate that FPL achieves state-of-the-art performance on eleven few-shot classification datasets and four domain generalisation datasets, significantly surpassing existing methods. Ablation studies revealed that both ridge regression penalty and projection orthogonality loss enhance model performance, with the ridge regression penalty having a greater overall impact and the orthogonality loss proving more effective in few-shot scenarios. The authors acknowledge that the performance gains from projection orthogonality loss are more pronounced when dealing with limited data. Future research could explore the application of FPL to other vision-language tasks and investigate methods for further optimising the projection and reconstruction processes.