• Corpus ID: 16895865

Stochastic Backpropagation and Approximate Inference in Deep Generative Models

@inproceedings{JimenezRezende2014StochasticBA,
  title={Stochastic Backpropagation and Approximate Inference in Deep Generative Models},
  author={Danilo Jimenez Rezende and Shakir Mohamed and Daan Wierstra},
  booktitle={ICML},
  year={2014}
}
We marry ideas from deep neural networks and approximate Bayesian inference to derive a generalised class of deep, directed generative models, endowed with a new algorithm for scalable inference and learning. [] Key Method We develop stochastic back-propagation -- rules for back-propagation through stochastic variables -- and use this to develop an algorithm that allows for joint optimisation of the parameters of both the generative and recognition model. We demonstrate on several real-world data sets that…
View Paper
jmlr.csail.mit.edu
3,814 Citations
Highly Influential Citations
643
Background Citations
1,875
Methods Citations
1,759
Results Citations
26

Figures and Tables from this paper

3,814 Citations

Citation Type

Citation Type

Towards Local Learning and MCMC Inference in Biologically Plausible Deep Generative Networks
TLDR
An ansatz for approximate Markov Chain Monte Carlo inference and learning in multi-layered (deep) generative models that exhibit biologically plausible dynamics is proposed and provides new perspectives on how biologically plausible learning algorithms could be reconciled with statistical learning theory.
Learning Deep Generative Models with Doubly Stochastic MCMC
TLDR
This work presents doubly stochastic gradient MCMC, a simple and generic method for (approximate) Bayesian inference of deep generative models (DGMs) in a collapsed continuous parameter space that outperforms many state-of-the-art competitors.
Structured Inference Networks for Nonlinear State Space Models
TLDR
A unified algorithm is introduced to efficiently learn a broad class of linear and non-linear state space models, including variants where the emission and transition distributions are modeled by deep neural networks.
Neural Variational Inference and Learning in Undirected Graphical Models
TLDR
This work proposes black-box learning and inference algorithms for undirected models that optimize a variational approximation to the log-likelihood of the model via a unified variational inference framework and empirically demonstrates the effectiveness of the method on several popular generative modeling datasets.
Asymptotically exact inference in differentiable generative models
TLDR
A method for performing efficient MCMC inference in generative models when conditioning on observations of the model output is presented, using a constrained variant of Hamiltonian Monte Carlo which leverages the smooth geometry of the manifold to coherently move between inputs exactly consistent with observations.
Inference in Deep Networks in High Dimensions
  • A. Fletcher, S. Rangan
  • Computer Science
    2018 IEEE International Symposium on Information Theory (ISIT)
  • 2018
TLDR
The main contribution shows that the mean-squared error (MSE) of ML-VAMP can be exactly predicted in a certain large system limit and matches the Bayes optimal value recently postulated by Reeves when certain fixed point equations have unique solutions.
Structured VAEs: Composing Probabilistic Graphical Models and Variational Autoencoders
TLDR
A new framework for unsupervised learning is developed that composes probabilistic graphical models with deep learning methods and combines their respective strengths to learn flexible feature models and bottom-up recognition networks.
Variational Gaussian Process
TLDR
The variational Gaussian process is constructed, a Bayesian nonparametric model which adapts its shape to match complex posterior distributions, and is proved a universal approximation theorem for the VGP, demonstrating its representative power for learning any model.
Composing graphical models with neural networks for structured representations and fast inference
TLDR
A general modeling and inference framework that composes probabilistic graphical models with deep learning methods and combines their respective strengths is proposed, giving a scalable algorithm that leverages stochastic variational inference, natural gradients, graphical model message passing, and the reparameterization trick.
Probabilistic Models with Deep Neural Networks
TLDR
An overview of the main concepts, methods, and tools needed to use deep neural networks within a probabilistic modeling framework is provided.
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 44 REFERENCES
Deep AutoRegressive Networks
TLDR
An efficient approximate parameter estimation method based on the minimum description length (MDL) principle is derived, which can be seen as maximising a variational lower bound on the log-likelihood, with a feedforward neural network implementing approximate inference.
Auto-Encoding Variational Bayes
TLDR
A stochastic variational inference and learning algorithm that scales to large datasets and, under some mild differentiability conditions, even works in the intractable case is introduced.
Deep Generative Stochastic Networks Trainable by Backprop
TLDR
Theorems that generalize recent work on the probabilistic interpretation of denoising autoencoders are provided and obtain along the way an interesting justification for dependency networks and generalized pseudolikelihood.
Stochastic variational inference
TLDR
Stochastic variational inference lets us apply complex Bayesian models to massive data sets, and it is shown that the Bayesian nonparametric topic model outperforms its parametric counterpart.
Bayesian Learning via Stochastic Gradient Langevin Dynamics
In this paper we propose a new framework for learning from large scale datasets based on iterative learning from small mini-batches. By adding the right amount of noise to a standard stochastic…
Deep Gaussian Processes
TLDR
Deep Gaussian process (GP) models are introduced and model selection by the variational bound shows that a five layer hierarchy is justified even when modelling a digit data set containing only 150 examples.
A Deep and Tractable Density Estimator
TLDR
This work introduces an efficient procedure to simultaneously train a NADE model for each possible ordering of the variables, by sharing parameters across all these models.
Practical Variational Inference for Neural Networks
TLDR
This paper introduces an easy-to-implement stochastic variational method (or equivalently, minimum description length loss function) that can be applied to most neural networks and revisits several common regularisers from a variational perspective.
Variational Bayesian Inference with Stochastic Search
TLDR
This work presents an alternative algorithm based on stochastic optimization that allows for direct optimization of the variational lower bound and demonstrates the approach on two non-conjugate models: logistic regression and an approximation to the HDP.
Unsupervised Variational Bayesian Learning of Nonlinear Models
TLDR
This paper presents a framework for using multi-layer perceptron (MLP) networks in nonlinear generative models trained by variational Bayesian learning using a Gauss-Hermite quadrature at the hidden neurons for accurate approximation for cases of large posterior variance.
...
1
2
3
4
5
...