Corpus ID: 38342008

Black-Box Alpha Divergence Minimization

@inproceedings{HernndezLobato2016BlackBoxAD,
title={Black-Box Alpha Divergence Minimization},
author={Jos{\'e} Miguel Hern{\'a}ndez-Lobato and Yingzhen Li and Mark Rowland and Thang D. Bui and Daniel Hern{\'a}ndez-Lobato and Richard E. Turner},
booktitle={ICML},
year={2016}
}
Black-box alpha (BB-$\alpha$) is a new approximate inference method based on the minimization of $\alpha$-divergences. BB-$\alpha$ scales to large datasets because it can be implemented using stochastic gradient descent. BB-$\alpha$ can be applied to complex probabilistic models with little effort since it only requires as input the likelihood function and its gradients. These gradients can be easily obtained using automatic differentiation. By changing the divergence parameter $\alpha$, the… Expand
Perturbative Black Box Variational Inference
• Mathematics, Computer Science
• NIPS
• 2017
This paper views BBVI with generalized divergences as a form of estimating the marginal likelihood via biased importance sampling, and builds a family of new variational bounds that captures the standard KL bound for $K=1, and converges to the exact marginal likelihood as$K\to\infty\$. Expand
Generalizing expectation propagation with mixtures of exponential family distributions and an application to Bayesian logistic regression
• Computer Science, Mathematics
• Neurocomputing
• 2019
A generalized version of EP called generalized EP (GEP), which is a new method based on the minimization of KL divergence for approximate inference, which provides faster convergence and better performance than other state-of-the-art approaches. Expand
Nonlinear Kalman Filtering With Divergence Minimization
• Mathematics, Computer Science
• IEEE Transactions on Signal Processing
• 2017
Novel algorithms are proposed to approximate this posterior by optimizing the forward and reverse forms of the KL divergence, as well as the <inline-formula>-divergence that contains these two as limiting cases, which use Monte Carlo techniques to derive unbiased algorithms for direct optimization. Expand
Alpha divergence minimization in multi-class Gaussian process classification
• Computer Science
• Neurocomputing
• 2020
The results obtained show that one can do better than EP and VB by considering intermediate values of α, and the generalization properties of each of the proposed methods for different values of the α parameter are analyzed. Expand
Probabilistic inference of Bayesian neural networks with generalized expectation propagation
• Computer Science
• Neurocomputing
• 2020
The generalized version of expectation propagation (GEP) is instantiated to provide an effective and efficient approximate inference method for BNNs including fully connected neural networks and convolutional neural networks on multiple benchmark datasets and shows a better performance than some state-of-the-art approximate inference methods. Expand
Variational Inference based on Robust Divergences
• Computer Science, Mathematics
• AISTATS
• 2018
This paper proposes an outlier-robust pseudo-Bayesian variational method by replacing the Kullback-Leibler divergence used for data fitting to a robust divergence such as the beta- and gamma-divergences, and theoretically proves that the influence function in the proposed method is bounded, while it is unbounded in the ordinary variational inference. Expand
Dropout Inference in Bayesian Neural Networks with Alpha-divergences
• Computer Science, Mathematics
• ICML
• 2017
A reparametrisation of the alpha-divergence objectives is proposed, deriving a simple inference technique which, together with dropout, can be easily implemented with existing models by simply changing the loss of the model. Expand
Advances in Variational Inference
• Mathematics, Computer Science
• IEEE Transactions on Pattern Analysis and Machine Intelligence
• 2019
An overview of recent trends in variational inference is given and a summary of promising future research directions is provided. Expand
Pairing an arbitrary regressor with an artificial neural network estimating aleatoric uncertainty
• Computer Science
• Neurocomputing
• 2019
We suggest a general approach to quantification of different forms of aleatoric uncertainty in regression tasks performed by artificial neural networks. It is based on the simultaneous training ofExpand
Truncated Variational Sampling for 'Black Box' Optimization of Generative Models
• Computer Science, Mathematics
• LVA/ICA
• 2018
This work investigates the optimization of two generative models with binary hidden variables using a novel variational EM approach that is applicable without any further analytical steps and efficiently as well as effectively increases the variational free-energy objective. Expand

References

SHOWING 1-10 OF 30 REFERENCES
Black Box Variational Inference
• Mathematics, Computer Science
• AISTATS
• 2014
This paper presents a "black box" variational inference algorithm, one that can be quickly applied to many models with little additional derivation, based on a stochastic optimization of the variational objective where the noisy gradient is computed from Monte Carlo samples from the Variational distribution. Expand
ABC-EP: Expectation Propagation for Likelihoodfree Bayesian Computation
• Computer Science
• ICML
• 2011
It is shown that Expectation Propagation, a widely successful approximate inference technique, can be adapted to the likelihood-free context and the resulting algorithm does not require summary statistics, is an order of magnitude faster than existing techniques, and remains usable when prior information is vague. Expand
Stochastic Expectation Propagation
• Computer Science, Mathematics
• NIPS
• 2015
Stochastic expectation propagation is presented, called SEP, that maintains a global posterior approximation but updates it in a local way (like EP), and is ideally suited to performing approximate Bayesian learning in the large model, large dataset setting. Expand
Expectation Propagation in the large-data limit
• Mathematics
• 2015
Expectation Propagation (Minka, 2001) is a widely successful algorithm for variational inference. EP is an iterative algorithm used to approximate complicated distributions, typically to find aExpand
Practical Bayesian Optimization of Machine Learning Algorithms
• Computer Science, Mathematics
• NIPS
• 2012
This work describes new algorithms that take into account the variable cost of learning algorithm experiments and that can leverage the presence of multiple cores for parallel experimentation and shows that these proposed algorithms improve on previous automatic procedures and can reach or surpass human expert-level optimization for many algorithms. Expand
Distributed Bayesian Learning with Stochastic Natural Gradient Expectation Propagation and the Posterior Server
Stochastic natural gradient expectation propagation is proposed, a novel alternative to expectation propagation, a popular variational inference algorithm, and a novel architecture for distributed Bayesian learning which is called the posterior server. Expand
Adam: A Method for Stochastic Optimization
• Computer Science, Mathematics
• ICLR
• 2015
This work introduces Adam, an algorithm for first-order gradient-based optimization of stochastic objective functions, based on adaptive estimates of lower-order moments, and provides a regret bound on the convergence rate that is comparable to the best known results under the online convex optimization framework. Expand
Auto-Encoding Variational Bayes
• Mathematics, Computer Science
• ICLR
• 2014
A stochastic variational inference and learning algorithm that scales to large datasets and, under some mild differentiability conditions, even works in the intractable case is introduced. Expand
A family of algorithms for approximate Bayesian inference
• T. Minka
• Mathematics, Computer Science
• 2001
This thesis presents an approximation technique that can perform Bayesian inference faster and more accurately than previously possible, and is found to be convincingly better than rival approximation techniques: Monte Carlo, Laplace's method, and variational Bayes. Expand
Expectation Propagation for approximate Bayesian inference
• T. Minka
• Computer Science, Mathematics
• UAI
• 2001
Expectation Propagation approximates the belief states by only retaining expectations, such as mean and varitmce, and iterates until these expectations are consistent throughout the network, which makes it applicable to hybrid networks with discrete and continuous nodes. Expand