How Neural Networks Encode Spurious Correlations: Insights from XAI Analysis

On April 22, 2025, researchers Phuong Quynh Le, Jörg Schlötterer, and Christin Seifert published An XAI-based Analysis of Shortcut Learning in Neural Networks. Their study explores how neural networks encode spurious correlations through a novel diagnostic measure. It compares CNNs and ViTs, revealing varying degrees of disentanglement and highlighting the need for improved mitigation strategies to enhance AI safety.

The paper introduces the neuron spurious score, an XAI-based diagnostic tool to quantify neural networks’ reliance on spurious features—non-causal correlations with target labels. By analyzing convolutional neural networks (CNNs) and vision transformers (ViTs), the study reveals that while spurious features are partially disentangled, the degree varies across architectures. Current mitigation methods fail due to incomplete assumptions about how models encode these correlations. The findings provide a foundation for developing safer AI systems by effectively addressing spurious correlations.

In recent years, machine learning models have achieved remarkable success across various tasks. However, their lack of transparency has led researchers to develop post-hoc explainability methods, such as the s-score, to provide insights into model behavior.

The s-score quantifies the alignment between a model’s attention mechanism and human-defined features in an input. It is calculated using post-hoc explanation techniques like Grad-CAM, which generates heatmaps to visualize where the model focuses within an input. These heatmaps highlight regions contributing most to predictions, allowing the s-score to measure overlap with predefined features.

Despite its utility, the s-score’s reliance on post-hoc methods introduces limitations. For instance, in image classification tasks, a model might focus on edges rather than centers of patches, as observed in specific examples. This discrepancy highlights how heatmaps can misrepresent nuances of model attention.

Two main challenges arise: overemphasis on local features and potential misinterpretation of focus regions. Heatmaps often highlight localized areas, potentially overlooking broader contextual cues influencing decisions. Additionally, they may not always align with human intuition about relevant features, leading to misleading conclusions.

To address these challenges, researchers must develop more sophisticated techniques for evaluating model attention, ensuring explanations are both precise and meaningful. By understanding the limitations of current approaches, we can build more transparent and trustworthy machine learning systems.

In conclusion, while the s-score is a valuable tool for assessing model attention alignment with human-defined features, its reliance on post-hoc methods introduces limitations. Future research should enhance evaluation techniques to ensure accurate and interpretable explanations.