Corpus ID: 9440787

Equilibrated adaptive learning rates for non-convex optimization

@inproceedings{Dauphin2015EquilibratedAL,
  title={Equilibrated adaptive learning rates for non-convex optimization},
  author={Yann Dauphin and H. D. Vries and Yoshua Bengio},
  booktitle={NIPS},
  year={2015}
}

Yann Dauphin, H. D. Vries, Yoshua Bengio

Published in NIPS 2015

Computer Science, Mathematics

Parameter-specific adaptive learning rate methods are computationally efficient ways to reduce the ill-conditioning problems encountered when training large deep networks. Following recent work that strongly suggests that most of the critical points encountered when training such networks are saddle points, we find how considering the presence of negative eigenvalues of the Hessian could help us design better suited adaptive learning rate schemes. We show that the popular Jacobi preconditioner… CONTINUE READING

View PDF on arXiv

Save to Library

Create Alert

Cite

Launch Research Feed