• DOI:10.48550/arXiv.2210.17216
• Corpus ID: 253237810

Symmetries, flat minima, and the conserved quantities of gradient flow

@article{Zhao2022SymmetriesFM,
  title={Symmetries, flat minima, and the conserved quantities of gradient flow},
  author={Bo Zhao and Iordan Ganev and Robin Walters and Rose Yu and Nima Dehmamy},
  journal={ArXiv},
  year={2022},
  volume={abs/2210.17216}
}

• Bo Zhao, I. Ganev, +2 authors Nima Dehmamy
• Published 31 October 2022
• Computer Science
• ArXiv

Empirical studies of the loss landscape of deep networks have revealed that many local minima are connected through low-loss valleys. Yet, little is known about the theoretical origin of such valleys. We present a general framework for finding continuous symmetries in the parameter space, which carve out low-loss valleys. Our framework uses equivariances of the activation functions and can be applied to different layer architectures. To generalize this framework to nonlinear neural networks, we introduce a novel set of nonlinear, data-dependent symmetries. These symmetries can transform a trained model such that it performs similarly on new samples, which allows ensemble building that improves robustness under certain adversarial attacks. We then show that conserved quantities associated with linear symmetries can be used to define coordinates along low-loss valleys. The conserved quantities help reveal that using common initialization methods, gradient flow only explores a small part of the global minimum. By relating conserved quantities to convergence rate and sharpness of the minimum, we provide insights on how initialization impacts convergence and generalizability.