Exponential expressivity in deep neural networks through transient chaos

@article{Poole2016ExponentialEI,
  title={Exponential expressivity in deep neural networks through transient chaos},
  author={Ben Poole and Subhaneil Lahiri and Maithra Raghu and Jascha Sohl-Dickstein and Surya Ganguli},
  journal={ArXiv},
  year={2016},
  volume={abs/1606.05340}
}
We combine Riemannian geometry with the mean field theory of high dimensional chaos to study the nature of signal propagation in deep neural networks with random weights. Our results reveal a phase transition in the expressivity of random deep networks, with networks in the chaotic phase computing nonlinear functions whose global curvature grows exponentially with depth, but not with width. We prove that this generic class of random functions cannot be efficiently computed by any shallow… CONTINUE READING

Citations

Publications citing this paper.
SHOWING 1-10 OF 141 CITATIONS

Sample Variance Decay in Randomly Initialized ReLU Networks

VIEW 4 EXCERPTS
CITES METHODS & BACKGROUND
HIGHLY INFLUENCED

DEEP MEAN FIELD THEORY: LAYERWISE VARIANCE

  • 2018
VIEW 14 EXCERPTS
CITES BACKGROUND & METHODS
HIGHLY INFLUENCED

ON RANDOM DEEP AUTOENCODERS: EXACT ASYMP-

  • 2018
VIEW 10 EXCERPTS
CITES BACKGROUND
HIGHLY INFLUENCED

FILTER CITATIONS BY YEAR

2016
2019

CITATION STATISTICS

  • 31 Highly Influenced Citations

  • Averaged 45 Citations per year from 2017 through 2019

References

Publications referenced by this paper.
SHOWING 1-10 OF 19 REFERENCES

McIntosh , Niru Maheswaranathan , Aran Nayebi , Surya Ganguli , and Stephen A . Baccus . Deep learning models of the retinal response to natural scenes

Yoshua Bengio, Aaron Courville, Pascal Vincent
  • Advances in Neural Information Processing Systems
  • 2016

Deep learning

Shubho Sengupta, Adam Coates, Yann LeCun, Yoshua Bengio
  • Nature
  • 2015