Singularities Affect Dynamics of Learning in Neuromanifolds

  title={Singularities Affect Dynamics of Learning in Neuromanifolds},
  author={Shun-ichi Amari and Hyeyoung Park and Tomoko Ozeki},
  journal={Neural Computation},
The parameter spaces of hierarchical systems such as multilayer perceptrons include singularities due to the symmetry and degeneration of hidden units. A parameter space forms a geometrical manifold, called the neuromanifold in the case of neural networks. Such a model is identified with a statistical model, and a Riemannian metric is given by the Fisher information matrix. However, the matrix degenerates at singularities. Such a singular structure is ubiquitous not only in multilayer… CONTINUE READING
Highly Cited
This paper has 134 citations. REVIEW CITATIONS


Publications citing this paper.
Showing 1-10 of 49 extracted citations

Parameter Identifiability in Statistical Machine Learning: A Review

Neural Computation • 2017
View 20 Excerpts
Highly Influenced

Skip Connections as Effective Symmetry-Breaking

ArXiv • 2017
View 7 Excerpts
Highly Influenced

Learning Dynamics of a Single Polar Variable Complex-Valued Neuron

Neural Computation • 2015
View 4 Excerpts
Highly Influenced

Numerical Analysis near Singularities in RBF Networks

Journal of Machine Learning Research • 2018
View 4 Excerpts
Highly Influenced

134 Citations

Citations per Year
Semantic Scholar estimates that this publication has 134 citations based on the available data.

See our FAQ for additional information.


Publications referenced by this paper.
Showing 1-10 of 55 references

New consideration on criteria of model selection

S. Amari
(Proceedings of the Sixth International Conference on Neural Networks and Soft Computing) • 2003
View 6 Excerpts
Highly Influenced

A failure of likelihood asymptotics for normal mixtures

J. A. Hartigan
Proc. Barkeley Conf. in Honor of J. Neyman and J. Kiefer, • 1985
View 8 Excerpts
Highly Influenced

Asymptotics for likelihood ratio tests under loss of identifiability

X. Liu, Y. Shao
Annals of Statistics, • 2003
View 3 Excerpts
Highly Influenced

Differential and algebraic geometry of multilayer perceptrons

S. Amari, T. Ozeki
IEICE Trans., • 2001
View 5 Excerpts
Highly Influenced

Natural gradient descent for on-line learning

M. Rattray, D. Saad, S. Amari
Physical Review Letters, • 1998
View 4 Excerpts
Highly Influenced

Similar Papers

Loading similar papers…