Hexformer Achieves Enhanced Image Classification with Novel Exponential Map Aggregation

Researchers are increasingly recognising the limitations of Euclidean geometry when modelling complex, hierarchical data found in images and other modalities. Haya Alyoussef, Ahmad Bdeir, and Diego Coello de Portugal Mecke, from Hildesheim University’s Information Systems and Machine Learning Lab (ISMLL), alongside et al., present HexFormer, a novel vision transformer operating within hyperbolic space to address this challenge. This work introduces a system which utilises exponential map aggregation to create more accurate and stable representations, demonstrably improving performance on image classification tasks compared to existing Euclidean and hyperbolic models. Crucially, their findings also reveal that hyperbolic models exhibit enhanced gradient stability during training, potentially offering a more robust and efficient approach to deep learning architecture design.

This breakthrough research introduces a transformer architecture entirely formulated within the Lorentz model of hyperbolic space, incorporating a new attention mechanism based on exponential map aggregation. The team achieved consistent performance improvements over Euclidean baselines and existing hyperbolic ViTs across multiple datasets, demonstrating the efficacy of their approach. HexFormer’s innovative attention mechanism yields more accurate and stable aggregated representations compared to standard centroid-based averaging, proving that simpler methods can still deliver competitive results.

The study explores two designs: a hyperbolic ViT (HexFormer) and a hybrid variant (HexFormer-Hybrid) combining a hyperbolic encoder with a Euclidean linear classification head. Experiments reveal that the HexFormer-Hybrid variant consistently achieves the strongest overall performance, highlighting the benefits of this combined approach. Crucially, the research also provides a detailed analysis of gradient stability in hyperbolic transformers, demonstrating that these models exhibit more stable gradients and reduced sensitivity to warmup strategies compared to their Euclidean counterparts. This improved stability translates to more robust and efficient training, reducing the need for extensive hyperparameter tuning.
This work establishes that hyperbolic geometry can significantly enhance vision transformer architectures by improving both gradient stability and accuracy. The novel exponential map aggregation within the attention mechanism provides a simple yet effective way to aggregate features in hyperbolic space, avoiding distortions common in centroid-based methods. Furthermore, the analysis of training dynamics reveals that hyperbolic ViTs are less susceptible to the challenges of warmup schedules, offering a more streamlined training process. The researchers demonstrate consistent improvements across various datasets, activation functions, and model scales, solidifying the potential of hyperbolic representations for computer vision tasks.

Beyond the architectural innovations, the study unveils a deeper understanding of how hyperbolic models behave during training. The findings indicate that the inherent properties of hyperbolic space contribute to more stable gradients, allowing for faster convergence and potentially enabling the training of larger, more complex models. The team’s HexFormer and HexFormer-Hybrid models consistently outperform prior hyperbolic ViTs, such as HVT and LViT, showcasing the practical benefits of their design choices. This research opens new avenues for exploring hyperbolic deep learning and its application to a wider range of computer vision problems, promising more accurate and efficient image classification systems.

Hyperbolic Vision Transformers with Exponential Map Aggregation achieve

Scientists. Experiments consistently demonstrate performance improvements over Euclidean baselines and previous hyperbolic ViTs, with the HexFormer-Hybrid achieving the strongest overall results across multiple datasets. This breakthrough delivers a novel approach to image classification by leveraging the benefits of hyperbolic geometry for representing hierarchical data structures. Results demonstrate that HexFormer’s novel attention mechanism, based on exponential map aggregation, yields more accurate and stable aggregated representations compared to standard centroid-based averaging. Measurements confirm that this simpler approach retains competitive merit, suggesting that complex mechanisms aren’t always necessary for achieving significant gains.

The study meticulously analysed gradient stability in hyperbolic transformers, revealing that hyperbolic models exhibit more stable gradients and reduced sensitivity to warmup strategies when compared to Euclidean architectures. This finding highlights the robustness and efficiency of hyperbolic models during the training process, potentially reducing the need for extensive hyperparameter tuning. Tests prove that the HexFormer-Hybrid model consistently outperforms both Euclidean ViTs and prior hyperbolic ViTs across various datasets, activation functions, and model scales. The research team meticulously measured performance gains, establishing a clear advantage for hyperbolic representations in capturing complex hierarchical structures within images.

Furthermore, the analysis of training dynamics revealed that hyperbolic ViTs achieve improved gradient stability, allowing for more efficient and reliable training. Data shows that this improved stability reduces the need for extensive fine-tuning, streamlining the development process and lowering computational costs. Scientists recorded that the exponential map aggregation technique provides a strong practical benefit, offering a simple yet effective method for feature aggregation within the hyperbolic attention mechanism. The work introduces a hyperbolic Vision Transformer entirely formulated in the Lorentz model of hyperbolic space, pushing the boundaries of current vision transformer architectures.

Overall, these findings indicate that hyperbolic geometry can enhance vision transformer architectures by improving gradient stability and accuracy, paving the way for more robust and efficient image classification systems. Code is publicly available to facilitate further research and development in this exciting area. HexFormer boosts image.