Dynamical Isometry and a Mean Field Theory of RNNs: Gating Enables Signal Propagation in Recurrent Neural Networks

@article{Chen2018DynamicalIA,
  title={Dynamical Isometry and a Mean Field Theory of RNNs: Gating Enables Signal Propagation in Recurrent Neural Networks},
  author={Minmin Chen and Jeffrey Pennington and Samuel S. Schoenholz},
  journal={ArXiv},
  year={2018},
  volume={abs/1806.05394}
}
Recurrent neural networks have gained widespread use in modeling sequence data across various domains. While many successful recurrent architectures employ a notion of gating, the exact mechanism that enables such remarkable performance is not well understood. We develop a theory for signal propagation in recurrent networks after random initialization using a combination of mean field theory and random matrix theory. To simplify our discussion, we introduce a new RNN cell with a simple gating… CONTINUE READING

Citations

Publications citing this paper.
SHOWING 1-10 OF 33 CITATIONS

Dynamical Isometry and a Mean Field Theory of LSTMs and GRUs

  • ArXiv
  • 2019
VIEW 18 EXCERPTS
CITES METHODS & BACKGROUND

ANTISYMMETRICRNN: A DYNAMICAL SYSTEM VIEW

  • 2018
VIEW 5 EXCERPTS
CITES METHODS & BACKGROUND
HIGHLY INFLUENCED

Almost Surely Asymptotic Freeness for Jacobian Spectrum of Deep Network

Tomohiro Hayase
  • ArXiv
  • 2019
VIEW 1 EXCERPT
CITES BACKGROUND

References

Publications referenced by this paper.
SHOWING 1-10 OF 50 REFERENCES