Autonomous Driving Achieves Generalization with Foundation Models and 90% Variance Capture

The challenge of creating truly robust and generalisable autonomous driving systems remains a significant hurdle in the field of artificial intelligence. Amir Mallak, Erfan Aasi, Shiva Sreeram, et al. from the University of Haifa and other institutions, have investigated how to improve performance in unpredictable, real-world conditions. Their research demonstrates that current end-to-end approaches, while showing promise, often rely on redundant information within the visual data processed by foundation models, leading to overfitting and reduced performance when faced with unfamiliar scenarios. The team introduces Stochastic-Patch-Selection (SPS), a novel technique that randomly masks portions of visual input, forcing the autonomous system to focus on invariant features and ultimately enhancing its ability to navigate diverse and challenging environments. Through extensive simulations and real-world testing in a physical vehicle, they prove SPS significantly outperforms existing state-of-the-art methods, representing a crucial step towards reliable and adaptable self-driving technology.

Recent advances in end-to-end autonomous driving demonstrate that policies trained on patch-aligned features extracted from foundation models generalise better to Out-Of-Distribution (OOD) scenarios. This research investigates the hypothesis that the self-attention mechanism within these models causes each patch feature to implicitly embed information from all other patches, albeit in a different representation. To explore this, the team proposes a Sparse Patch Selection (SPS) approach, a subset of descriptors designed to force the policy to learn from diverse subsets with reduced correlation. This methodology aims to improve computational efficiency, achieving a 2.4x speedup, and enhance generalisation performance by 6.2%, offering a flexible architecture for autonomous systems.

Patch Descriptor Redundancy and Dimensionality Reduction

End-to-end autonomous driving approaches are becoming increasingly effective, streamlining perception and control into a single model. Utilising patch-aligned features from foundation models improves a system’s ability to generalise to unseen scenarios. These patch features are created through masked attention mechanisms applied to a layer within the BLIP2 Q-Former, generating a descriptor for each image patch within a shared vision-language space. However, analysis reveals significant redundancy and correlation within these patch descriptors. Principal Component Analysis (PCA) showed that 90% of the variance in these features is captured by only 17 out of 64 principal components, indicating substantial overlap.

This redundancy can lead to overfitting on spurious correlations, hindering robustness in new situations. To address this, the researchers developed Stochastic-Patch-Selection (SPS), a method designed to learn more robust and generalisable policies by randomly masking a portion of the patch descriptors for each frame, while maintaining spatial arrangement. This process encourages the policy to rely on features invariant to specific patches, consistently outperforming state-of-the-art methods across OOD scenarios, achieving an average improvement of 6.2% and up to 20.4% in closed-loop simulations. Ablation studies across nine systems showed that eight surpassed previous performance, and the learned policy successfully transferred to a real-world vehicle without further adjustments.

Patch Feature Redundancy in Foundation Models

Recent advances in autonomous driving have shown improved generalisation when utilising patch-aligned features extracted from foundation models. Researchers hypothesised that these patch features contain redundant information due to the self-attention mechanisms within the foundation models, implicitly embedding data from all other patches. Principal Component Analysis (PCA) and cross-patch similarity measurements confirmed this, revealing that 90% of the variance in feature data is captured by only 17 of 64 principal components. Experiments demonstrated pervasive correlations between patch descriptors, visualised through correlation heatmaps and similarity overlays.

These overlays showed widespread high similarity extending beyond the initial patch, indicating a global entanglement of information. The study quantified this by demonstrating that the principal r-dimensional subspace of the feature data is preserved even when randomly selecting a subset of patches, provided the subset meets a specific index threshold, confirming that crucial scene semantics are maintained despite removing redundant information. To address spurious correlations, scientists developed Stochastic-Patch-Selection (SPS), randomly masking a fraction of patch descriptors during training. This forces the policy to base decisions on features invariant to specific token survival, promoting robustness and generalisation.

Closed-loop simulations across diverse Out-of-Distribution (OOD) scenarios showed that SPS outperforms the state-of-the-art, achieving an average improvement of 6% and up to 10% in challenging simulations. Rigorous ablation studies, involving nine trained systems, revealed that eight surpassed prior performance. The learned policy successfully transferred to a physical car without further tuning. Real-world testing, conducted with a 2019 Lexus RX 450H equipped with an NVIDIA RTX 4070 Ti GPU, confirmed the policy’s ability to navigate safely, measured by a reduction in safety driver interventions. Models were trained on a cloud cluster utilising four A100 GPUs, completing the process in approximately four days.

Robust Policies Via Stochastic Feature Masking

This work demonstrates that patch-aligned features, extracted from vision-language models for autonomous driving policies, contain inherent correlations and redundancy. Researchers addressed this by introducing Stochastic-Patch-Selection (SPS), randomly masking a portion of patch descriptors during training, encouraging the policy to rely on more robust and invariant features. Through this approach, the driving policy learns to base decisions on features that remain reliable regardless of which specific patches are present. Extensive closed-loop simulations across varied driving scenarios, including shifts in weather, lighting, and geographic location, showed that SPS improves out-of-distribution performance by an average of 6.2%, with gains reaching 20.4% in the most difficult situations.

The method also accelerates inference speed by a factor of 2.4 and enables data augmentation in the latent patch space. Importantly, policies trained with SPS successfully transferred to a real-world autonomous vehicle without further adjustments. Future research will focus on developing more sophisticated selection techniques, potentially learning a state-dependent sampling policy or employing coreset selection to identify and remove demonstrably redundant patches, further enhancing efficiency and robustness.