Corpus ID: 233864730

A Geometric Analysis of Neural Collapse with Unconstrained Features

@article{Zhu2021AGA,
  title={A Geometric Analysis of Neural Collapse with Unconstrained Features},
  author={Zhihui Zhu and Tianyu Ding and Jinxin Zhou and Xiao Li and Chong You and Jeremias Sulam and Qing Qu},
  journal={ArXiv},
  year={2021},
  volume={abs/2105.02375}
}
We provide the first global optimization landscape analysis of Neural Collapse – an intriguing empirical phenomenon that arises in the last-layer classifiers and features of neural networks during the terminal phase of training. As recently reported in [1], this phenomenon implies that (i) the class means and the last-layer classifiers all collapse to the vertices of a Simplex Equiangular Tight Frame (ETF) up to scaling, and (ii) cross-example within-class variability of last-layer activations… Expand

Figures and Tables from this paper

Neural Collapse Under MSE Loss: Proximity to and Dynamics on the Central Path

References

SHOWING 1-10 OF 127 REFERENCES
Neural Collapse with Cross-Entropy Loss
Optimizationmethods for large-scalemachine learning
  • Siam Review,
  • 2018
Mnist handwritten digit database
  • at&t labs,
  • 2010
ImageNet: A large-scale hierarchical image database
Prevalence of neural collapse during the terminal phase of deep learning training
Densely Connected Convolutional Networks
Adam: A Method for Stochastic Optimization
Layer-Peeled Model: Toward Understanding Well-Trained Deep Neural Networks
Nonconvex optimizationmeets low-rankmatrix factorization: An overview
  • IEEE Transactions on Signal Processing,
  • 2019
The Global Optimization Geometry of Shallow Linear Neural Networks
...
1
2
3
4
5
...