Corpus ID: 6318468

Krylov Subspace Descent for Deep Learning

@inproceedings{Vinyals2012KrylovSD,
  title={Krylov Subspace Descent for Deep Learning},
  author={Oriol Vinyals and D. Povey},
  booktitle={AISTATS},
  year={2012}
}

Oriol Vinyals, D. Povey

Published in AISTATS 2012

Computer Science, Mathematics

In this paper, we propose a second order optimization method to learn models where both the dimensionality of the parameter space and the number of training samples is high. In our method, we construct on each iteration a Krylov subspace formed by the gradient and an approximation to the Hessian matrix, and then use a subset of the training data samples to optimize over this subspace. As with the Hessian Free (HF) method of Martens (2010), the Hessian matrix is never explicitly constructed, and… CONTINUE READING

View PDF on arXiv

Save to Library

Create Alert

Cite

Launch Research Feed