# Black-Box Alpha Divergence Minimization

@inproceedings{HernndezLobato2016BlackBoxAD, title={Black-Box Alpha Divergence Minimization}, author={Jos{\'e} Miguel Hern{\'a}ndez-Lobato and Yingzhen Li and Mark Rowland and Thang D. Bui and Daniel Hern{\'a}ndez-Lobato and Richard E. Turner}, booktitle={ICML}, year={2016} }

Black-box alpha (BB-$\alpha$) is a new approximate inference method based on the minimization of $\alpha$-divergences. BB-$\alpha$ scales to large datasets because it can be implemented using stochastic gradient descent. BB-$\alpha$ can be applied to complex probabilistic models with little effort since it only requires as input the likelihood function and its gradients. These gradients can be easily obtained using automatic differentiation. By changing the divergence parameter $\alpha$, the… Expand

#### Supplemental Code

#### Figures, Tables, and Topics from this paper

#### 76 Citations

Perturbative Black Box Variational Inference

- Mathematics, Computer Science
- NIPS
- 2017

This paper views BBVI with generalized divergences as a form of estimating the marginal likelihood via biased importance sampling, and builds a family of new variational bounds that captures the standard KL bound for $K=1, and converges to the exact marginal likelihood as $K\to\infty$. Expand

Generalizing expectation propagation with mixtures of exponential family distributions and an application to Bayesian logistic regression

- Computer Science, Mathematics
- Neurocomputing
- 2019

A generalized version of EP called generalized EP (GEP), which is a new method based on the minimization of KL divergence for approximate inference, which provides faster convergence and better performance than other state-of-the-art approaches. Expand

Nonlinear Kalman Filtering With Divergence Minimization

- Mathematics, Computer Science
- IEEE Transactions on Signal Processing
- 2017

Novel algorithms are proposed to approximate this posterior by optimizing the forward and reverse forms of the KL divergence, as well as the <inline-formula>-divergence that contains these two as limiting cases, which use Monte Carlo techniques to derive unbiased algorithms for direct optimization. Expand

Alpha divergence minimization in multi-class Gaussian process classification

- Computer Science
- Neurocomputing
- 2020

The results obtained show that one can do better than EP and VB by considering intermediate values of α, and the generalization properties of each of the proposed methods for different values of the α parameter are analyzed. Expand

Probabilistic inference of Bayesian neural networks with generalized expectation propagation

- Computer Science
- Neurocomputing
- 2020

The generalized version of expectation propagation (GEP) is instantiated to provide an effective and efficient approximate inference method for BNNs including fully connected neural networks and convolutional neural networks on multiple benchmark datasets and shows a better performance than some state-of-the-art approximate inference methods. Expand

Variational Inference based on Robust Divergences

- Computer Science, Mathematics
- AISTATS
- 2018

This paper proposes an outlier-robust pseudo-Bayesian variational method by replacing the Kullback-Leibler divergence used for data fitting to a robust divergence such as the beta- and gamma-divergences, and theoretically proves that the influence function in the proposed method is bounded, while it is unbounded in the ordinary variational inference. Expand

Dropout Inference in Bayesian Neural Networks with Alpha-divergences

- Computer Science, Mathematics
- ICML
- 2017

A reparametrisation of the alpha-divergence objectives is proposed, deriving a simple inference technique which, together with dropout, can be easily implemented with existing models by simply changing the loss of the model. Expand

Advances in Variational Inference

- Mathematics, Computer Science
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- 2019

An overview of recent trends in variational inference is given and a summary of promising future research directions is provided. Expand

Pairing an arbitrary regressor with an artificial neural network estimating aleatoric uncertainty

- Computer Science
- Neurocomputing
- 2019

We suggest a general approach to quantification of different forms of aleatoric uncertainty in regression tasks performed by artificial neural networks. It is based on the simultaneous training of… Expand

Truncated Variational Sampling for 'Black Box' Optimization of Generative Models

- Computer Science, Mathematics
- LVA/ICA
- 2018

This work investigates the optimization of two generative models with binary hidden variables using a novel variational EM approach that is applicable without any further analytical steps and efficiently as well as effectively increases the variational free-energy objective. Expand

#### References

SHOWING 1-10 OF 30 REFERENCES

Black Box Variational Inference

- Mathematics, Computer Science
- AISTATS
- 2014

This paper presents a "black box" variational inference algorithm, one that can be quickly applied to many models with little additional derivation, based on a stochastic optimization of the variational objective where the noisy gradient is computed from Monte Carlo samples from the Variational distribution. Expand

ABC-EP: Expectation Propagation for Likelihoodfree Bayesian Computation

- Computer Science
- ICML
- 2011

It is shown that Expectation Propagation, a widely successful approximate inference technique, can be adapted to the likelihood-free context and the resulting algorithm does not require summary statistics, is an order of magnitude faster than existing techniques, and remains usable when prior information is vague. Expand

Stochastic Expectation Propagation

- Computer Science, Mathematics
- NIPS
- 2015

Stochastic expectation propagation is presented, called SEP, that maintains a global posterior approximation but updates it in a local way (like EP), and is ideally suited to performing approximate Bayesian learning in the large model, large dataset setting. Expand

Expectation Propagation in the large-data limit

- Mathematics
- 2015

Expectation Propagation (Minka, 2001) is a widely successful algorithm for variational inference. EP is an iterative algorithm used to approximate complicated distributions, typically to find a… Expand

Practical Bayesian Optimization of Machine Learning Algorithms

- Computer Science, Mathematics
- NIPS
- 2012

This work describes new algorithms that take into account the variable cost of learning algorithm experiments and that can leverage the presence of multiple cores for parallel experimentation and shows that these proposed algorithms improve on previous automatic procedures and can reach or surpass human expert-level optimization for many algorithms. Expand

Distributed Bayesian Learning with Stochastic Natural Gradient Expectation Propagation and the Posterior Server

- Computer Science, Mathematics
- J. Mach. Learn. Res.
- 2017

Stochastic natural gradient expectation propagation is proposed, a novel alternative to expectation propagation, a popular variational inference algorithm, and a novel architecture for distributed Bayesian learning which is called the posterior server. Expand

Adam: A Method for Stochastic Optimization

- Computer Science, Mathematics
- ICLR
- 2015

This work introduces Adam, an algorithm for first-order gradient-based optimization of stochastic objective functions, based on adaptive estimates of lower-order moments, and provides a regret bound on the convergence rate that is comparable to the best known results under the online convex optimization framework. Expand

Auto-Encoding Variational Bayes

- Mathematics, Computer Science
- ICLR
- 2014

A stochastic variational inference and learning algorithm that scales to large datasets and, under some mild differentiability conditions, even works in the intractable case is introduced. Expand

A family of algorithms for approximate Bayesian inference

- Mathematics, Computer Science
- 2001

This thesis presents an approximation technique that can perform Bayesian inference faster and more accurately than previously possible, and is found to be convincingly better than rival approximation techniques: Monte Carlo, Laplace's method, and variational Bayes. Expand

Expectation Propagation for approximate Bayesian inference

- Computer Science, Mathematics
- UAI
- 2001

Expectation Propagation approximates the belief states by only retaining expectations, such as mean and varitmce, and iterates until these expectations are consistent throughout the network, which makes it applicable to hybrid networks with discrete and continuous nodes. Expand