# Auto-Encoding Variational Bayes

@article{Kingma2014AutoEncodingVB, title={Auto-Encoding Variational Bayes}, author={Diederik P. Kingma and Max Welling}, journal={CoRR}, year={2014}, volume={abs/1312.6114} }

Abstract: How can we perform efficient inference and learning in directed probabilistic models, in the presence of continuous latent variables with intractable posterior distributions, and large datasets. [... ] Key Method First, we show that a reparameterization of the variational lower bound yields a lower bound estimator that can be straightforwardly optimized using standard stochastic gradient methods. Second, we show that for i.i.d. Expand

## 18,285 Citations

### Likelihood Almost Free Inference Networks

- Computer ScienceArXiv
- 2017

It is shown that the proposed approach is essentially optimizing a probabilistic mixture of ELBOs, thus enriching modeling capacity and enhancing robustness, and outperforms state-of-the-art methods in the experiments on several density estimation tasks.

### Variational Gaussian Process

- Computer ScienceICLR
- 2016

The variational Gaussian process is constructed, a Bayesian nonparametric model which adapts its shape to match complex posterior distributions, and is proved a universal approximation theorem for the VGP, demonstrating its representative power for learning any model.

### Asymmetric Variational Autoencoders.

- Computer Science
- 2017

It can be shown that the actual Variational posterior of the proposed approach is essentially modeling a rich probabilistic mixture of simple variational posterior indexed by auxiliary variables, thus a flexible inference model can be built.

### Trust Region Sequential Variational Inference

- Computer Science, MathematicsACML
- 2019

This work presents a new algorithm for stochastic variational inference of sequential models which trades off bias for variance to tackle the challenge of handling high-dimensional data and models with non-differentiable densities caused by, for instance, the use of discrete latent variables.

### Advances in Variational Inference

- Computer ScienceIEEE Transactions on Pattern Analysis and Machine Intelligence
- 2019

An overview of recent trends in variational inference is given and a summary of promising future research directions is provided.

### Boosted Stochastic Backpropagation for Variational Inference

- Computer Science
- 2017

A mixture-based non-parametric variational inference algorithm is proposed that proves a convergence to the true posterior in O(1/t) where t is the number of mixture components and boosted stochastic backpropagation is proposed.

### Fixing a Broken ELBO

- Computer ScienceICML
- 2018

This framework derives variational lower and upper bounds on the mutual information between the input and the latent variable, and uses these bounds to derive a rate-distortion curve that characterizes the tradeoff between compression and reconstruction accuracy.

### Sampling-Free Variational Inference of Bayesian Neural Networks by Variance Backpropagation

- Computer ScienceUAI
- 2019

We propose a new Bayesian Neural Net formulation that affords variational inference for which the evidence lower bound is analytically tractable subject to a tight approximation. We achieve this…

### Variational Bayes on Monte Carlo Steroids

- Computer ScienceNIPS
- 2016

A new class of bounds on the marginal log-likelihood of directed latent variable models is proposed, which relies on random projections to simplify the posterior, and empirical improvements on benchmark datasets in vision and language for sigmoid belief networks are demonstrated.

### Structured Dropout Variational Inference for Bayesian Neural Networks

- Computer ScienceNeurIPS
- 2021

This work focuses on the inflexibility of the factorized structure in Dropout posterior and proposes an improved method called Variational Structured Dropout (VSD), which employs an orthogonal transformation to learn a structured representation on the variational noise and consequently induces statistical dependencies in the approximate posterior.

## References

SHOWING 1-10 OF 24 REFERENCES

### Black Box Variational Inference

- Computer ScienceAISTATS
- 2014

This paper presents a "black box" variational inference algorithm, one that can be quickly applied to many models with little additional derivation, based on a stochastic optimization of the variational objective where the noisy gradient is computed from Monte Carlo samples from the Variational distribution.

### Stochastic Back-propagation and Variational Inference in Deep Latent Gaussian Models

- Computer ScienceArXiv
- 2014

We marry ideas from deep neural networks and approximate Bayesian inference to derive a generalised class of deep, directed generative models, endowed with a new algorithm for scalable inference and…

### Variational Bayesian Inference with Stochastic Search

- Computer ScienceICML
- 2012

This work presents an alternative algorithm based on stochastic optimization that allows for direct optimization of the variational lower bound and demonstrates the approach on two non-conjugate models: logistic regression and an approximation to the HDP.

### Stochastic variational inference

- Computer ScienceJ. Mach. Learn. Res.
- 2013

Stochastic variational inference lets us apply complex Bayesian models to massive data sets, and it is shown that the Bayesian nonparametric topic model outperforms its parametric counterpart.

### Stochastic Backpropagation and Approximate Inference in Deep Generative Models

- Computer ScienceICML
- 2014

We marry ideas from deep neural networks and approximate Bayesian inference to derive a generalised class of deep, directed generative models, endowed with a new algorithm for scalable inference and…

### Fixed-Form Variational Posterior Approximation through Stochastic Linear Regression

- Computer ScienceArXiv
- 2012

A general algorithm for approximating nonstandard Bayesian posterior distributions that minimizes the Kullback-Leibler divergence of an approximating distribution to the intractable posterior distribu- tion.

### Adaptive Subgradient Methods for Online Learning and Stochastic Optimization

- Computer ScienceJ. Mach. Learn. Res.
- 2011

This work describes and analyze an apparatus for adaptively modifying the proximal function, which significantly simplifies setting a learning rate and results in regret guarantees that are provably as good as the best proximal functions that can be chosen in hindsight.

### Efficient Learning of Deep Boltzmann Machines

- Computer ScienceAISTATS
- 2010

We present a new approximate inference algorithm for Deep Boltzmann Machines (DBM’s), a generative model with many layers of hidden variables. The algorithm learns a separate “recognition” model that…

### Practical Variational Inference for Neural Networks

- Computer ScienceNIPS
- 2011

This paper introduces an easy-to-implement stochastic variational method (or equivalently, minimum description length loss function) that can be applied to most neural networks and revisits several common regularisers from a variational perspective.

### Deep AutoRegressive Networks

- Computer ScienceICML
- 2014

An efficient approximate parameter estimation method based on the minimum description length (MDL) principle is derived, which can be seen as maximising a variational lower bound on the log-likelihood, with a feedforward neural network implementing approximate inference.