# Theory of gating in recurrent neural networks

@article{Krishnamurthy2020TheoryOG, title={Theory of gating in recurrent neural networks}, author={K. Krishnamurthy and T. Can and D. Schwab}, journal={ArXiv}, year={2020}, volume={abs/2007.14823} }

Recurrent neural networks (RNNs) are powerful dynamical models, widely used in machine learning (ML) for processing sequential data, and also in neuroscience, to understand the emergent properties of networks of real neurons. Prior theoretical work in understanding the properties of RNNs has focused on models with additive interactions. However, real neurons can have gating -- i.e. multiplicative -- interactions, and gating is also a central feature of the best performing RNNs in machine… Expand

#### Figures and Topics from this paper

#### 4 Citations

Learning to represent continuous variables in heterogeneous neural networks

- Biology
- 2021

The work suggests that the functional properties of manifold attractors in the brain can be inferred from the overlooked asymmetries in connectivity and in the low-dimensional representation of the encoded variable. Expand

Dynamics of continuous-time gated recurrent neural networks

- 2020

We study how gates shape the dynamics of a continuous-time gated recurrent network, closely related to the Gated Recurrent Unit (GRU). As a function of the initialization hyperparameters, we map out… Expand

Reverse engineering learned optimizers reveals known and novel mechanisms

- Computer Science, Mathematics
- ArXiv
- 2020

This work studies learned optimizers trained from scratch on three disparate tasks, and discovers that they have learned interpretable mechanisms, including: momentum, gradient clipping, learning rate schedules, and a new form of learning rate adaptation. Expand

Searching for long time scales without fine tuning

- Computer Science, Physics
- 2020

It is argued that a single long time scale can emerge generically from realistic constraints, but a full spectrum of slow modes requires more tuning, as well as the constraints required for long time scales to become generic. Expand

#### References

SHOWING 1-10 OF 74 REFERENCES

Physical review letters 110

- 118101
- 2013

Physical Review X 8

- 041029
- 2018

I and i

- Computer Science
- BMJ : British Medical Journal
- 2001

There is, I think, something ethereal about i —the square root of minus one, which seems an odd beast at that time—an intruder hovering on the edge of reality. Expand

Physical Review Research 2

- 013111
- 2020

and D

- J. Schwab, arXiv preprint arXiv:2002.00025
- 2020

and F

- Krzakala, arXiv preprint arXiv:2006.07310
- 2020

and L

- Abbott, arXiv preprint arXiv:2006.02427
- 2020

and P

- Koumoutsakos, Neural Networks
- 2020