• DOI:10.1007/978-3-030-46147-8_27
Corpus ID: 52909328

Learning with Random Learning Rates

@article{Blier2019LearningWR,
  title={Learning with Random Learning Rates},
  author={L. Blier and Pierre Wolinski and Y. Ollivier},
  journal={ArXiv},
  year={2019},
  volume={abs/1810.01322}
}

• L. Blier, Pierre Wolinski, Y. Ollivier
• Published 2019
• Computer Science, Mathematics
• ArXiv

Hyperparameter tuning is a bothersome step in the training of deep learning models. One of the most sensitive hyperparameters is the learning rate of the gradient descent. We present the 'All Learning Rates At Once' (Alrao) optimization method for neural networks: each unit or feature in the network gets its own learning rate sampled from a random distribution spanning several orders of magnitude. This comes at practically no computational cost. Perhaps surprisingly, stochastic gradient descent… Expand

View PDF on arXiv

Save to Library

Create Alert

Cite

Launch Research Feed