Corpus ID: 211259030

The Early Phase of Neural Network Training

@article{Frankle2020TheEP,
  title={The Early Phase of Neural Network Training},
  author={Jonathan Frankle and D. Schwab and Ari S. Morcos},
  journal={ArXiv},
  year={2020},
  volume={abs/2002.10365}
}
Recent studies have shown that many important aspects of neural network learning take place within the very earliest iterations or epochs of training. For example, sparse, trainable sub-networks emerge (Frankle et al., 2019), gradient descent moves into a small subspace (Gur-Ari et al., 2018), and the network undergoes a critical period (Achille et al., 2019). Here we examine the changes that deep neural networks undergo during this early phase of training. We perform extensive measurements of… Expand
FICIENT TRAINING OF DEEP NETWORKS
Catastrophic Fisher Explosion: Early Phase Fisher Matrix Impacts Generalization
Emerging Paradigms of Neural Network Pruning
Understanding the Role of Training Regimes in Continual Learning
Speeding-up pruning for Artificial Neural Networks: Introducing Accelerated Iterative Magnitude Pruning
The large learning rate phase of deep learning: the catapult mechanism
Roulette: A Pruning Framework to Train a Sparse Neural Network From Scratch
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 23 REFERENCES
Learning both Weights and Connections for Efficient Neural Network
Understanding deep learning requires rethinking generalization
Wide Residual Networks
Why Does Unsupervised Pre-training Help Deep Learning?
Rethinking the Value of Network Pruning
Critical Learning Periods in Deep Neural Networks
Deep Residual Learning for Image Recognition
Empirical Analysis of the Hessian of Over-Parametrized Neural Networks
...
1
2
3
...