Corpus ID: 216078090

Auto-Encoding Variational Bayes

@article{Kingma2014AutoEncodingVB,
  title={Auto-Encoding Variational Bayes},
  author={Diederik P. Kingma and Max Welling},
  journal={CoRR},
  year={2014},
  volume={abs/1312.6114}
}
Abstract: How can we perform efficient inference and learning in directed probabilistic models, in the presence of continuous latent variables with intractable posterior distributions, and large datasets. [...] Key Method First, we show that a reparameterization of the variational lower bound yields a lower bound estimator that can be straightforwardly optimized using standard stochastic gradient methods. Second, we show that for i.i.d.Expand
Likelihood Almost Free Inference Networks
TLDR
It is shown that the proposed approach is essentially optimizing a probabilistic mixture of ELBOs, thus enriching modeling capacity and enhancing robustness, and outperforms state-of-the-art methods in the experiments on several density estimation tasks. Expand
Variational Gaussian Process
TLDR
The variational Gaussian process is constructed, a Bayesian nonparametric model which adapts its shape to match complex posterior distributions, and is proved a universal approximation theorem for the VGP, demonstrating its representative power for learning any model. Expand
Asymmetric Variational Autoencoders.
Variational inference for latent variable models is prevalent in various machine learning problems, typically solved by maximizing the Evidence Lower Bound (ELBO) of the true data likelihood withExpand
Trust Region Sequential Variational Inference
TLDR
This work presents a new algorithm for stochastic variational inference of sequential models which trades off bias for variance to tackle the challenge of handling high-dimensional data and models with non-differentiable densities caused by, for instance, the use of discrete latent variables. Expand
Advances in Variational Inference
TLDR
An overview of recent trends in variational inference is given and a summary of promising future research directions is provided. Expand
Sampling-Free Variational Inference of Bayesian Neural Networks by Variance Backpropagation
  • Melih Kandemir, Manuel Haussmann, Fred A. Hamprecht
  • Computer Science, Mathematics
  • UAI
  • 2019
We propose a new Bayesian Neural Net formulation that affords variational inference for which the evidence lower bound is analytically tractable subject to a tight approximation. We achieve thisExpand
Fixing a Broken ELBO
TLDR
This framework derives variational lower and upper bounds on the mutual information between the input and the latent variable, and uses these bounds to derive a rate-distortion curve that characterizes the tradeoff between compression and reconstruction accuracy. Expand
Variational Bayes on Monte Carlo Steroids
TLDR
A new class of bounds on the marginal log-likelihood of directed latent variable models is proposed, which relies on random projections to simplify the posterior, and empirical improvements on benchmark datasets in vision and language for sigmoid belief networks are demonstrated. Expand
Tutorial: Deriving the Standard Variational Autoencoder (VAE) Loss Function
TLDR
This tutorial derives the variational lower bound loss function of the standard variational autoencoder in the instance of a gaussian latent prior and gaussian approximate posterior, under which assumptions the Kullback-Leibler term in the variations lower bound has a closed form solution. Expand
Neural Variational Inference and Learning in Belief Networks
TLDR
This work proposes a fast non-iterative approximate inference method that uses a feedforward network to implement efficient exact sampling from the variational posterior and shows that it outperforms the wake-sleep algorithm on MNIST and achieves state-of-the-art results on the Reuters RCV1 document dataset. Expand
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 25 REFERENCES
Black Box Variational Inference
TLDR
This paper presents a "black box" variational inference algorithm, one that can be quickly applied to many models with little additional derivation, based on a stochastic optimization of the variational objective where the noisy gradient is computed from Monte Carlo samples from the Variational distribution. Expand
Stochastic Back-propagation and Variational Inference in Deep Latent Gaussian Models
We marry ideas from deep neural networks and approximate Bayesian inference to derive a generalised class of deep, directed generative models, endowed with a new algorithm for scalable inference andExpand
Variational Bayesian Inference with Stochastic Search
TLDR
This work presents an alternative algorithm based on stochastic optimization that allows for direct optimization of the variational lower bound and demonstrates the approach on two non-conjugate models: logistic regression and an approximation to the HDP. Expand
Stochastic variational inference
TLDR
Stochastic variational inference lets us apply complex Bayesian models to massive data sets, and it is shown that the Bayesian nonparametric topic model outperforms its parametric counterpart. Expand
Stochastic Backpropagation and Approximate Inference in Deep Generative Models
We marry ideas from deep neural networks and approximate Bayesian inference to derive a generalised class of deep, directed generative models, endowed with a new algorithm for scalable inference andExpand
Deep Generative Stochastic Networks Trainable by Backprop
TLDR
Theorems that generalize recent work on the probabilistic interpretation of denoising autoencoders are provided and obtain along the way an interesting justification for dependency networks and generalized pseudolikelihood. Expand
Fixed-Form Variational Posterior Approximation through Stochastic Linear Regression
TLDR
A general algorithm for approximating nonstandard Bayesian posterior distributions that minimizes the Kullback-Leibler divergence of an approximating distribution to the intractable posterior distribu- tion. Expand
Adaptive Subgradient Methods for Online Learning and Stochastic Optimization
TLDR
This work describes and analyze an apparatus for adaptively modifying the proximal function, which significantly simplifies setting a learning rate and results in regret guarantees that are provably as good as the best proximal functions that can be chosen in hindsight. Expand
Efficient Learning of Deep Boltzmann Machines
We present a new approximate inference algorithm for Deep Boltzmann Machines (DBM’s), a generative model with many layers of hidden variables. The algorithm learns a separate “recognition” model thatExpand
Practical Variational Inference for Neural Networks
  • A. Graves
  • Computer Science, Mathematics
  • NIPS
  • 2011
TLDR
This paper introduces an easy-to-implement stochastic variational method (or equivalently, minimum description length loss function) that can be applied to most neural networks and revisits several common regularisers from a variational perspective. Expand
...
1
2
3
...