Corpus ID: 211069156

On the distance between two neural networks and the stability of learning

@article{Bernstein2020OnTD,
  title={On the distance between two neural networks and the stability of learning},
  author={J. Bernstein and Arash Vahdat and Yisong Yue and MingYu Liu},
  journal={ArXiv},
  year={2020},
  volume={abs/2002.03432}
}

J. Bernstein, Arash Vahdat, +1 author MingYu Liu

Published 2020

Computer Science, Mathematics

ArXiv

View PDF on arXiv

Save to Library

Create Alert

Cite

Launch Research Feed

This paper relates parameter distance to gradient breakdown for a broad class of nonlinear compositional functions. The analysis leads to a new distance function called deep relative trust and a descent lemma for neural networks. Since the resulting learning rule seems not to require learning rate grid search, it may unlock a simpler workflow for training deeper and more complex neural networks. Please find the Python code used in this paper here: this https URL.