• Corpus ID: 225067132

Scalable Bayesian neural networks by layer-wise input augmentation

@article{Trinh2020ScalableBN,
  title={Scalable Bayesian neural networks by layer-wise input augmentation},
  author={Trung Trinh and Samuel Kaski and Markus Heinonen},
  journal={ArXiv},
  year={2020},
  volume={abs/2010.13498}
}
We introduce implicit Bayesian neural networks, a simple and scalable approach for uncertainty representation in deep learning. Standard Bayesian approach to deep learning requires the impractical inference of the posterior distribution over millions of parameters. Instead, we propose to induce a distribution that captures the uncertainty over neural networks by augmenting each layer's inputs with latent variables. We present appropriate input distributions and demonstrate state-of-the-art… 
View With Semantic Reader
2 Citations
Background Citations
1
Methods Citations
1

2 Citations

Citation Type

Citation Type

Tackling covariate shift with node-based Bayesian neural networks
TLDR
This work interprets latent noise variables of node-based BNNs as implicit representations of simple and domain-agnostic data perturbations during training, and proposes a straightforward approach to increase the entropy of these variables during training.
Continuous-Time Model-Based Reinforcement Learning
TLDR
This work proposes a continuous-time MBRL framework based on a novel actor-critic method that infers the unknown state evolution differentials with Bayesian neural ordinary differential equations (ODE) to account for epistemic uncertainty.

References

SHOWING 1-10 OF 30 REFERENCES
Multiplicative Normalizing Flows for Variational Bayesian Neural Networks
We reinterpret multiplicative noise in neural networks as auxiliary random variables that augment the approximate posterior in a variational setting for Bayesian neural networks. We show that through…
Evaluating Scalable Bayesian Deep Learning Methods for Robust Computer Vision
TLDR
This work proposes a comprehensive evaluation framework for scalable epistemic uncertainty estimation methods in deep learning and applies this framework to provide the first properly extensive and conclusive comparison of the two current state-of-the- art scalable methods: ensembling and MC-dropout.
Projected BNNs: Avoiding weight-space pathologies by learning latent representations of neural network weights
TLDR
A novel variational inference framework for Bayesian neural networks that encodes complex distributions in highdimensional parameter space with representations in a low-dimensional latent space, and performs inference efficiently on the lowdimensional representations.
Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adam
TLDR
New natural-gradient algorithms to reduce efforts for Gaussian mean-field VI by perturbing the network weights during gradient evaluations, and uncertainty estimates can be cheaply obtained by using the vector that adapts the learning rate.
Latent Projection BNNs: Avoiding weight-space pathologies by learning latent representations of neural network weights
TLDR
A novel variational inference framework for Bayesian neural networks that encodes complex distributions in high-dimensional parameter space with representations in a low-dimensional latent space, and performs inference efficiently on the low- dimensional representations is introduced.
Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles
TLDR
This work proposes an alternative to Bayesian NNs that is simple to implement, readily parallelizable, requires very little hyperparameter tuning, and yields high quality predictive uncertainty estimates.
Probabilistic inference of Bayesian neural networks with generalized expectation propagation
The Case for Bayesian Deep Learning
TLDR
The key distinguishing property of a Bayesian approach is marginalization instead of optimization, not the prior, or Bayes rule, which reflects the inductive biases of neural networks that help them generalize.
Functional Variational Bayesian Neural Networks
TLDR
Functional variational Bayesian neural networks (fBNNs), which maximize an Evidence Lower BOund defined directly on stochastic processes, are introduced and it is proved that the KL divergence between stoChastic processes equals the supremum of marginal KL divergences over all finite sets of inputs.
Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning
TLDR
A new theoretical framework is developed casting dropout training in deep neural networks (NNs) as approximate Bayesian inference in deep Gaussian processes, which mitigates the problem of representing uncertainty in deep learning without sacrificing either computational complexity or test accuracy.
...
...