Tubular Riemannian Laplace Approximation Advances Bayesian Inference for Complex Neural Networks

Bayesian neural networks offer a powerful approach to quantifying uncertainty in machine learning, but calculating true Bayesian predictions often proves computationally intractable, necessitating approximations. Rodrigo Pereira David, from the National Institute of Metrology, Technology and Quality (Inmetro), and colleagues introduce a new method, the Tubular Riemannian Laplace (TRL) approximation, to address limitations in existing techniques. The team’s innovation explicitly models the posterior probability as a probabilistic tube aligned with low-loss valleys within the network’s complex landscape, effectively distinguishing between uncertainty arising from limited prior knowledge and that driven by the data itself. This approach achieves a significant breakthrough, delivering calibration comparable to, or better than, complex ensemble methods while dramatically reducing computational cost, requiring only one-fifth the training time, and therefore bridging the gap between efficient single-model predictions and highly reliable ensemble performance.

Researchers recognised that standard techniques struggle with the highly curved loss surfaces and symmetry groups inherent in contemporary neural network architectures, motivating the development of a geometrically informed approach.

The team engineered TRL to explicitly model the posterior probability distribution as a probabilistic tube, aligning with low-loss valleys created by functional symmetries within the network’s weight space. This innovative method employs a Fisher/Gauss-Newton metric, separating uncertainty into tangential and transverse components, effectively distinguishing between directions influenced by the prior distribution and those driven by the data. The approach leverages implicit curvature estimates to operate efficiently in high-dimensional parameter spaces, enabling scalable Bayesian inference.

Experiments involving ResNet-18 networks trained on image datasets demonstrate that TRL achieves excellent calibration, matching or exceeding the reliability of Deep Ensembles, a benchmark for uncertainty quantification. The work centers on accurately representing the posterior probability distribution, which describes the uncertainty in a model’s parameters after training, and achieving this efficiently.

TRL explicitly models this posterior as a probabilistic tube, aligning with natural symmetries present in the data and model, and separating uncertainty into tangential and transverse components. This approach leverages a Fisher/Gauss-Newton metric to distinguish between uncertainty driven by the prior and that dictated by the data. Experiments demonstrate that TRL achieves excellent calibration, matching or exceeding the reliability of ensemble methods while requiring significantly less computational cost, only one-fifth the training time.

This represents a substantial step towards bridging the efficiency of single models with the robust performance typically associated with ensembles, offering tangible gains in performance and reliability. The authors acknowledge that while TRL improves calibration, it does not address inherent biases present in the training data, a crucial consideration for responsible application.