• Corpus ID: 216078090

Auto-Encoding Variational Bayes

@article{Kingma2014AutoEncodingVB,
  title={Auto-Encoding Variational Bayes},
  author={Diederik P. Kingma and Max Welling},
  journal={CoRR},
  year={2014},
  volume={abs/1312.6114}
}
Abstract: How can we perform efficient inference and learning in directed probabilistic models, in the presence of continuous latent variables with intractable posterior distributions, and large datasets. [] Key Method First, we show that a reparameterization of the variational lower bound yields a lower bound estimator that can be straightforwardly optimized using standard stochastic gradient methods. Second, we show that for i.i.d.

Figures from this paper

Likelihood Almost Free Inference Networks

TLDR
It is shown that the proposed approach is essentially optimizing a probabilistic mixture of ELBOs, thus enriching modeling capacity and enhancing robustness, and outperforms state-of-the-art methods in the experiments on several density estimation tasks.

Variational Gaussian Process

TLDR
The variational Gaussian process is constructed, a Bayesian nonparametric model which adapts its shape to match complex posterior distributions, and is proved a universal approximation theorem for the VGP, demonstrating its representative power for learning any model.

Asymmetric Variational Autoencoders.

TLDR
It can be shown that the actual Variational posterior of the proposed approach is essentially modeling a rich probabilistic mixture of simple variational posterior indexed by auxiliary variables, thus a flexible inference model can be built.

Trust Region Sequential Variational Inference

TLDR
This work presents a new algorithm for stochastic variational inference of sequential models which trades off bias for variance to tackle the challenge of handling high-dimensional data and models with non-differentiable densities caused by, for instance, the use of discrete latent variables.

Advances in Variational Inference

TLDR
An overview of recent trends in variational inference is given and a summary of promising future research directions is provided.

Boosted Stochastic Backpropagation for Variational Inference

TLDR
A mixture-based non-parametric variational inference algorithm is proposed that proves a convergence to the true posterior in O(1/t) where t is the number of mixture components and boosted stochastic backpropagation is proposed.

Fixing a Broken ELBO

TLDR
This framework derives variational lower and upper bounds on the mutual information between the input and the latent variable, and uses these bounds to derive a rate-distortion curve that characterizes the tradeoff between compression and reconstruction accuracy.

Sampling-Free Variational Inference of Bayesian Neural Networks by Variance Backpropagation

We propose a new Bayesian Neural Net formulation that affords variational inference for which the evidence lower bound is analytically tractable subject to a tight approximation. We achieve this

Variational Bayes on Monte Carlo Steroids

TLDR
A new class of bounds on the marginal log-likelihood of directed latent variable models is proposed, which relies on random projections to simplify the posterior, and empirical improvements on benchmark datasets in vision and language for sigmoid belief networks are demonstrated.

Structured Dropout Variational Inference for Bayesian Neural Networks

TLDR
This work focuses on the inflexibility of the factorized structure in Dropout posterior and proposes an improved method called Variational Structured Dropout (VSD), which employs an orthogonal transformation to learn a structured representation on the variational noise and consequently induces statistical dependencies in the approximate posterior.
...

References

SHOWING 1-10 OF 24 REFERENCES

Black Box Variational Inference

TLDR
This paper presents a "black box" variational inference algorithm, one that can be quickly applied to many models with little additional derivation, based on a stochastic optimization of the variational objective where the noisy gradient is computed from Monte Carlo samples from the Variational distribution.

Stochastic Back-propagation and Variational Inference in Deep Latent Gaussian Models

We marry ideas from deep neural networks and approximate Bayesian inference to derive a generalised class of deep, directed generative models, endowed with a new algorithm for scalable inference and

Variational Bayesian Inference with Stochastic Search

TLDR
This work presents an alternative algorithm based on stochastic optimization that allows for direct optimization of the variational lower bound and demonstrates the approach on two non-conjugate models: logistic regression and an approximation to the HDP.

Stochastic variational inference

TLDR
Stochastic variational inference lets us apply complex Bayesian models to massive data sets, and it is shown that the Bayesian nonparametric topic model outperforms its parametric counterpart.

Stochastic Backpropagation and Approximate Inference in Deep Generative Models

We marry ideas from deep neural networks and approximate Bayesian inference to derive a generalised class of deep, directed generative models, endowed with a new algorithm for scalable inference and

Fixed-Form Variational Posterior Approximation through Stochastic Linear Regression

TLDR
A general algorithm for approximating nonstandard Bayesian posterior distributions that minimizes the Kullback-Leibler divergence of an approximating distribution to the intractable posterior distribu- tion.

Adaptive Subgradient Methods for Online Learning and Stochastic Optimization

TLDR
This work describes and analyze an apparatus for adaptively modifying the proximal function, which significantly simplifies setting a learning rate and results in regret guarantees that are provably as good as the best proximal functions that can be chosen in hindsight.

Efficient Learning of Deep Boltzmann Machines

We present a new approximate inference algorithm for Deep Boltzmann Machines (DBM’s), a generative model with many layers of hidden variables. The algorithm learns a separate “recognition” model that

Practical Variational Inference for Neural Networks

TLDR
This paper introduces an easy-to-implement stochastic variational method (or equivalently, minimum description length loss function) that can be applied to most neural networks and revisits several common regularisers from a variational perspective.

Deep AutoRegressive Networks

TLDR
An efficient approximate parameter estimation method based on the minimum description length (MDL) principle is derived, which can be seen as maximising a variational lower bound on the log-likelihood, with a feedforward neural network implementing approximate inference.