# Exponential expressivity in deep neural networks through transient chaos

@article{Poole2016ExponentialEI, title={Exponential expressivity in deep neural networks through transient chaos}, author={Ben Poole and Subhaneil Lahiri and Maithra Raghu and Jascha Sohl-Dickstein and Surya Ganguli}, journal={ArXiv}, year={2016}, volume={abs/1606.05340} }

- Published in NIPS 2016

We combine Riemannian geometry with the mean field theory of high dimensional chaos to study the nature of signal propagation in deep neural networks with random weights. Our results reveal a phase transition in the expressivity of random deep networks, with networks in the chaotic phase computing nonlinear functions whose global curvature grows exponentially with depth, but not with width. We prove that this generic class of random functions cannot be efficiently computed by any shallow… CONTINUE READING

#### Citations

##### Publications citing this paper.

SHOWING 1-10 OF 141 CITATIONS

## Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10, 000-Layer Vanilla Convolutional Neural Networks

VIEW 11 EXCERPTS

CITES BACKGROUND & METHODS

## Fast generalization error bound of deep learning without scale invariance of activation functions

VIEW 7 EXCERPTS

CITES METHODS & BACKGROUND

HIGHLY INFLUENCED

## On Random Deep Weight-Tied Autoencoders: Exact Asymptotic Analysis, Phase Transitions, and Implications to Training

VIEW 10 EXCERPTS

CITES BACKGROUND

HIGHLY INFLUENCED

## Sample Variance Decay in Randomly Initialized ReLU Networks

VIEW 4 EXCERPTS

CITES METHODS & BACKGROUND

HIGHLY INFLUENCED

## Variance-Preserving Initialization Schemes Improve Deep Network Training: But Which Variance is Preserved?

VIEW 5 EXCERPTS

CITES METHODS & BACKGROUND

HIGHLY INFLUENCED

## DEEP MEAN FIELD THEORY: LAYERWISE VARIANCE

VIEW 14 EXCERPTS

CITES BACKGROUND & METHODS

HIGHLY INFLUENCED

## Dynamical Isometry and a Mean Field Theory of RNNs: Gating Enables Signal Propagation in Recurrent Neural Networks

VIEW 14 EXCERPTS

CITES BACKGROUND & METHODS

HIGHLY INFLUENCED

## Neural-net-induced Gaussian process regression for function approximation and PDE solution

VIEW 12 EXCERPTS

CITES BACKGROUND

HIGHLY INFLUENCED

## ON RANDOM DEEP AUTOENCODERS: EXACT ASYMP-

VIEW 10 EXCERPTS

CITES BACKGROUND

HIGHLY INFLUENCED

## On the Selection of Initialization and Activation Function for Deep Neural Networks

VIEW 4 EXCERPTS

CITES RESULTS & METHODS

HIGHLY INFLUENCED

### FILTER CITATIONS BY YEAR

### CITATION STATISTICS

**31**Highly Influenced Citations**Averaged 45 Citations**per year from 2017 through 2019

#### References

##### Publications referenced by this paper.

SHOWING 1-10 OF 19 REFERENCES

## Riemannian Manifolds: An Introduction to Curvature

VIEW 5 EXCERPTS

HIGHLY INFLUENTIAL

## McIntosh , Niru Maheswaranathan , Aran Nayebi , Surya Ganguli , and Stephen A . Baccus . Deep learning models of the retinal response to natural scenes

## On the expressive power of deep neural networks

VIEW 1 EXCERPT