SGDR: Stochastic Gradient Descent with Restarts

  title={SGDR: Stochastic Gradient Descent with Restarts},
  author={Ilya Loshchilov and Frank Hutter},
Restart techniques are common in gradient-free optimization to deal with multimodal functions. Partial restarts are also gaining popularity in gradient-based optimization to improve the rate of convergence in accelerated gradient schemes to deal with ill-conditioned functions. In this paper, we propose a simple restart technique for stochastic gradient descent to improve its anytime performance when training deep neural networks. We empirically study its performance on CIFAR10 and CIFAR-100… CONTINUE READING
Highly Influential
This paper has highly influenced 17 other papers. REVIEW HIGHLY INFLUENTIAL CITATIONS
Highly Cited
This paper has 115 citations. REVIEW CITATIONS
Recent Discussions
This paper has been referenced on Twitter 203 times over the past 90 days. VIEW TWEETS


Publications citing this paper.
Showing 1-10 of 80 extracted citations

116 Citations

Citations per Year
Semantic Scholar estimates that this publication has 116 citations based on the available data.

See our FAQ for additional information.

Similar Papers

Loading similar papers…