Capacity and Trainability in Recurrent Neural Networks

  title={Capacity and Trainability in Recurrent Neural Networks},
  author={Jasmine Collins and Jascha Sohl-Dickstein and David Sussillo},
Two potential bottlenecks on the expressiveness of recurrent neural networks (RNNs) are their ability to store information about the task in their parameters, and to store information about the input history in their units. We show experimentally that all common RNN architectures achieve nearly the same per-task and per-unit capacity bounds with careful training, for a variety of tasks and stacking depths. They can store an amount of task information which is linear in the number of parameters… CONTINUE READING
Highly Cited
This paper has 58 citations. REVIEW CITATIONS
Related Discussions
This paper has been referenced on Twitter 78 times. VIEW TWEETS


Publications citing this paper.

59 Citations

Citations per Year
Semantic Scholar estimates that this publication has 59 citations based on the available data.

See our FAQ for additional information.


Publications referenced by this paper.
Showing 1-10 of 39 references

Vapnik-Chervonenkis Dimension of Recurrent Neural Networks

Discrete Applied Mathematics • 1998
View 2 Excerpts
Highly Influenced

LSTM: A Search Space Odyssey

IEEE Transactions on Neural Networks and Learning Systems • 2017
View 2 Excerpts

Deep Residual Learning for Image Recognition

2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) • 2016
View 2 Excerpts

Large text compression benchmark: About the test data, 2011. URL http://mattmahoney. net/dc/textdata. [Online; accessed 15-November-2016

Matt Mahoney

Similar Papers

Loading similar papers…