SGDR: Stochastic Gradient Descent with Restarts

@article{Loshchilov2016SGDRSG,
  title={SGDR: Stochastic Gradient Descent with Restarts},
  author={Ilya Loshchilov and Frank Hutter},
  journal={CoRR},
  year={2016},
  volume={abs/1608.03983}
}
Restart techniques are common in gradient-free optimization to deal with multimodal functions. Partial restarts are also gaining popularity in gradient-based optimization to improve the rate of convergence in accelerated gradient schemes to deal with ill-conditioned functions. In this paper, we propose a simple restart technique for stochastic gradient descent to improve its anytime performance when training deep neural networks. We empirically study its performance on CIFAR10 and CIFAR-100… CONTINUE READING
Highly Influential
This paper has highly influenced 17 other papers. REVIEW HIGHLY INFLUENTIAL CITATIONS
Highly Cited
This paper has 115 citations. REVIEW CITATIONS
Recent Discussions
This paper has been referenced on Twitter 203 times over the past 90 days. VIEW TWEETS

Citations

Publications citing this paper.
Showing 1-10 of 80 extracted citations

116 Citations

050201620172018
Citations per Year
Semantic Scholar estimates that this publication has 116 citations based on the available data.

See our FAQ for additional information.

Similar Papers

Loading similar papers…