Demystifying Learning Rate Policies for High Accuracy Training of Deep Neural Networks

@article{Wu2019DemystifyingLR,
  title={Demystifying Learning Rate Policies for High Accuracy Training of Deep Neural Networks},
  author={Yanzhao Wu and Ling Liu and Juhyun Bae and Ka-Ho Chow and Arun Iyengar and Calton Pu and Wenqi Wei and Lei Yu and Qi Zhang},
  journal={2019 IEEE International Conference on Big Data (Big Data)},
  year={2019},
  pages={1971-1980}
}
  • Yanzhao Wu, Ling Liu, +6 authors Qi Zhang
  • Published in
    IEEE International Conference…
    2019
  • Computer Science, Mathematics
  • Learning Rate (LR) is an important hyper-parameter to tune for effective training of deep neural networks (DNNs). Even for the baseline of a constant learning rate, it is non-trivial to choose a good constant value for training a DNN. Dynamic learning rates involve multi-step tuning of LR values at various stages of the training process and offer high accuracy and fast convergence. However, they are much harder to tune. In this paper, we present a comprehensive study of 13 learning rate… CONTINUE READING

    Create an AI-powered research feed to stay up to date with new papers like this posted to ArXiv

    Citations

    Publications citing this paper.

    Effects of Differential Privacy and Data Skewness on Membership Inference Vulnerability

    VIEW 1 EXCERPT
    CITES BACKGROUND

    References

    Publications referenced by this paper.
    SHOWING 1-10 OF 31 REFERENCES

    Channel Pruning for Accelerating Very Deep Neural Networks

    • Yihui He, Xiangyu Zhang, Jian Sun
    • Computer Science
    • 2017 IEEE International Conference on Computer Vision (ICCV)
    • 2017
    VIEW 12 EXCERPTS
    HIGHLY INFLUENTIAL

    Cyclical Learning Rates for Training Neural Networks

    • Leslie N. Smith
    • Computer Science
    • 2017 IEEE Winter Conference on Applications of Computer Vision (WACV)
    • 2015
    VIEW 5 EXCERPTS
    HIGHLY INFLUENTIAL

    Deep Residual Learning for Image Recognition

    VIEW 5 EXCERPTS
    HIGHLY INFLUENTIAL

    hyperopt — distributed asynchronous hyperparameter optimization in python

    • Hyperopt Developers
    • http://hyperopt.github.io/hyperopt/, 2019, [Online; accessed 13-Aug-2019].
    • 2019
    VIEW 2 EXCERPTS

    Benchmarking Deep Learning Frameworks: Design Considerations, Metrics and Beyond

    • Ling Liu, Yanzhao Wu, +3 authors Qi Zhang
    • Computer Science
    • 2018 IEEE 38th International Conference on Distributed Computing Systems (ICDCS)
    • 2018

    Cost-Sensitive Learning of Deep Feature Representations From Imbalanced Data

    VIEW 1 EXCERPT

    Experimental Characterizations and Analysis of Deep Learning Frameworks