Corpus ID: 49412311

Stochastic natural gradient descent draws posterior samples in function space

@article{Smith2018StochasticNG,
  title={Stochastic natural gradient descent draws posterior samples in function space},
  author={S. L. Smith and Daniel Duckworth and Quoc V. Le and Jascha Sohl-Dickstein},
  journal={ArXiv},
  year={2018},
  volume={abs/1806.09597}
}

S. L. Smith, Daniel Duckworth, +1 author Jascha Sohl-Dickstein

Published 2018

Computer Science, Mathematics

ArXiv

Recent work has argued that stochastic gradient descent can approximate the Bayesian uncertainty in model parameters near local minima. In this work we develop a similar correspondence for minibatch natural gradient descent (NGD). We prove that for sufficiently small learning rates, if the model predictions on the training set approach the true conditional distribution of labels given inputs, the stationary distribution of minibatch NGD approaches a Bayesian posterior near local minima. The… CONTINUE READING

View PDF on arXiv

Save to Library

Create Alert

Cite

Launch Research Feed