Practical Gauss-Newton Optimisation for Deep Learning

@inproceedings{Botev2017PracticalGO,
  title={Practical Gauss-Newton Optimisation for Deep Learning},
  author={Aleksandar Botev and Hippolyt Ritter and David Barber},
  booktitle={ICML},
  year={2017}
}
We present an efficient block-diagonal ap- proximation to the Gauss-Newton matrix for feedforward neural networks. Our result- ing algorithm is competitive against state- of-the-art first order optimisation methods, with sometimes significant improvement in optimisation performance. Unlike first-order methods, for which hyperparameter tuning of the optimisation parameters is often a labo- rious process, our approach can provide good performance even when used with default set- tings. A side… CONTINUE READING

Citations

Publications citing this paper.
SHOWING 1-10 OF 25 CITATIONS

Efficient Neural Architecture Search for Language Modeling

VIEW 5 EXCERPTS
CITES BACKGROUND & METHODS
HIGHLY INFLUENCED