• Corpus ID: 235658031

Laplace Redux - Effortless Bayesian Deep Learning

@article{Daxberger2021LaplaceR,
  title={Laplace Redux - Effortless Bayesian Deep Learning},
  author={Erik A. Daxberger and Agustinus Kristiadi and Alexander Immer and Runa Eschenhagen and Matthias Bauer and Philipp Hennig},
  journal={ArXiv},
  year={2021},
  volume={abs/2106.14806}
}
Bayesian formulations of deep learning have been shown to have compelling theoretical properties and offer practical functional benefits, such as improved predictive uncertainty quantification and model selection. The Laplace approximation (LA) is a classic, and arguably the simplest family of approximations for the intractable posteriors of deep neural networks. Yet, despite its simplicity, the LA is not as popular as alternatives like variational Bayes or deep ensembles. This may be due to… 
Bayesian Deep Learning via Subnetwork Inference
TLDR
This work shows that it suffices to perform inference over a small subset of model weights in order to obtain accurate predictive posteriors, and proposes a subnetwork selection strategy that aims to maximally preserve the model’s predictive uncertainty.
A Survey on Evidential Deep Learning For Single-Pass Uncertainty Estimation
TLDR
This survey aims to familiarize the reader with an alternative class of models based on the concept of Evidential Deep Learning, which allow uncertainty estimation in a single model and forward pass by parameterizing distributions over distributions.
Mixtures of Laplace Approximations for Improved Post-Hoc Uncertainty in Deep Learning
TLDR
This work proposes to predict with a Gaussian mixture model posterior that consists of a weighted sum of Laplace approximations of independently trained deep neural networks that mitigates overconfidence “far away” from the training data.
Efficient Online Bayesian Inference for Neural Bandits
TLDR
A new algorithm for online (sequential) inference in Bayesian neural networks, and its suitability for tackling contextual bandit problems is shown, showing good results on the “Deep Bayesian Bandit Showdown” benchmark, as well as MNIST and a recommender system.
Probing as Quantifying the Inductive Bias of Pre-trained Representations
TLDR
This work presents a novel framework for probing where the goal is to evaluate the inductive bias of representations for a particular task, and provides a practical avenue to do this using Bayesian inference.

References

SHOWING 1-10 OF 106 REFERENCES
Bayesian Deep Learning and a Probabilistic Perspective of Generalization
TLDR
It is shown that deep ensembles provide an effective mechanism for approximate Bayesian marginalization, and a related approach is proposed that further improves the predictive distribution by marginalizing within basins of attraction, without significant overhead.
Practical Deep Learning with Bayesian Principles
TLDR
This work enables practical deep learning while preserving benefits of Bayesian principles, and applies techniques such as batch normalisation, data augmentation, and distributed training to achieve similar performance in about the same number of epochs as the Adam optimiser.
What Are Bayesian Neural Network Posteriors Really Like?
TLDR
It is shown that BNNs can achieve significant performance gains over standard training and deep ensembles, and a single long HMC chain can provide a comparable representation of the posterior to multiple shorter chains, and posterior tempering is not needed for near-optimal performance.
Bayesian Dark Knowledge
We consider the problem of Bayesian parameter estimation for deep neural networks, which is important in problem settings where we may have little data, and/ or where we need accurate posterior
Bayesian Deep Learning via Subnetwork Inference
TLDR
This work shows that it suffices to perform inference over a small subset of model weights in order to obtain accurate predictive posteriors, and proposes a subnetwork selection strategy that aims to maximally preserve the model’s predictive uncertainty.
Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning
TLDR
A new theoretical framework is developed casting dropout training in deep neural networks (NNs) as approximate Bayesian inference in deep Gaussian processes, which mitigates the problem of representing uncertainty in deep learning without sacrificing either computational complexity or test accuracy.
Scalable Bayesian Optimization Using Deep Neural Networks
TLDR
This work shows that performing adaptive basis function regression with a neural network as the parametric form performs competitively with state-of-the-art GP-based approaches, but scales linearly with the number of data rather than cubically, which allows for a previously intractable degree of parallelism.
Bayesian Optimization Meets Laplace Approximation for Robotic Introspection
TLDR
This paper introduces a scalable Laplace Approximation technique to make Deep Neural Networks (DNNs) more introspective, i.e. to enable them to provide accurate assessments of their failure probability for unseen test data.
Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles
TLDR
This work proposes an alternative to Bayesian NNs that is simple to implement, readily parallelizable, requires very little hyperparameter tuning, and yields high quality predictive uncertainty estimates.
A Simple Baseline for Bayesian Uncertainty in Deep Learning
TLDR
It is demonstrated that SWAG performs well on a wide variety of tasks, including out of sample detection, calibration, and transfer learning, in comparison to many popular alternatives including MC dropout, KFAC Laplace, SGLD, and temperature scaling.
...
1
2
3
4
5
...