Corpus ID: 220845704

Theory of gating in recurrent neural networks

@article{Krishnamurthy2020TheoryOG,
  title={Theory of gating in recurrent neural networks},
  author={K. Krishnamurthy and T. Can and D. Schwab},
  journal={ArXiv},
  year={2020},
  volume={abs/2007.14823}
}
Recurrent neural networks (RNNs) are powerful dynamical models, widely used in machine learning (ML) for processing sequential data, and also in neuroscience, to understand the emergent properties of networks of real neurons. Prior theoretical work in understanding the properties of RNNs has focused on models with additive interactions. However, real neurons can have gating -- i.e. multiplicative -- interactions, and gating is also a central feature of the best performing RNNs in machine… Expand
Learning to represent continuous variables in heterogeneous neural networks
TLDR
The work suggests that the functional properties of manifold attractors in the brain can be inferred from the overlooked asymmetries in connectivity and in the low-dimensional representation of the encoded variable. Expand
Dynamics of continuous-time gated recurrent neural networks
We study how gates shape the dynamics of a continuous-time gated recurrent network, closely related to the Gated Recurrent Unit (GRU). As a function of the initialization hyperparameters, we map outExpand
Reverse engineering learned optimizers reveals known and novel mechanisms
TLDR
This work studies learned optimizers trained from scratch on three disparate tasks, and discovers that they have learned interpretable mechanisms, including: momentum, gradient clipping, learning rate schedules, and a new form of learning rate adaptation. Expand
Searching for long time scales without fine tuning
TLDR
It is argued that a single long time scale can emerge generically from realistic constraints, but a full spectrum of slow modes requires more tuning, as well as the constraints required for long time scales to become generic. Expand

References

SHOWING 1-10 OF 74 REFERENCES
Physical review letters 110
  • 118101
  • 2013
Physical Review X 8
  • 041029
  • 2018
I and i
TLDR
There is, I think, something ethereal about i —the square root of minus one, which seems an odd beast at that time—an intruder hovering on the edge of reality. Expand
Physical Review Letters 63
  • 1989
A and V
Physical Review Research 2
  • 013111
  • 2020
and D
  • J. Schwab, arXiv preprint arXiv:2002.00025
  • 2020
and F
  • Krzakala, arXiv preprint arXiv:2006.07310
  • 2020
and L
  • Abbott, arXiv preprint arXiv:2006.02427
  • 2020
and P
  • Koumoutsakos, Neural Networks
  • 2020
...
1
2
3
4
5
...