Corpus ID: 10940950

On the importance of initialization and momentum in deep learning

@inproceedings{Sutskever2013OnTI,
  title={On the importance of initialization and momentum in deep learning},
  author={Ilya Sutskever and J. Martens and G. Dahl and Geoffrey E. Hinton},
  booktitle={ICML},
  year={2013}
}

Ilya Sutskever, J. Martens, +1 author Geoffrey E. Hinton

Published in ICML 2013

Computer Science

Deep and recurrent neural networks (DNNs and RNNs respectively) are powerful models that were considered to be almost impossible to train using stochastic gradient descent with momentum. In this paper, we show that when stochastic gradient descent with momentum uses a well-designed random initialization and a particular type of slowly increasing schedule for the momentum parameter, it can train both DNNs and RNNs (on datasets with long-term dependencies) to levels of performance that were… CONTINUE READING

View Paper

cs.toronto.edu

Save to Library

Create Alert

Cite

Launch Research Feed