Corpus ID: 12775860

Memory-Efficient Backpropagation Through Time

@inproceedings{Gruslys2016MemoryEfficientBT,
  title={Memory-Efficient Backpropagation Through Time},
  author={A. Gruslys and R. Munos and Ivo Danihelka and Marc Lanctot and A. Graves},
  booktitle={NIPS},
  year={2016}
}
We propose a novel approach to reduce memory consumption of the backpropagation through time (BPTT) algorithm when training recurrent neural networks (RNNs). Our approach uses dynamic programming to balance a trade-off between caching of intermediate results and recomputation. The algorithm is capable of tightly fitting within almost any user-set memory budget while finding an optimal execution policy minimizing the computational cost. Computational devices have limited memory capacity and… Expand
100 Citations
Memory-Efficient Backpropagation for Recurrent Neural Networks
Backpropagation for long sequences: beyond memory constraints with constant overheads
  • 2
  • PDF
A Graph Theoretic Framework of Recomputation Algorithms for Memory-Efficient Backpropagation
  • 8
  • PDF
A Practical Sparse Approximation for Real Time Recurrent Learning
  • 3
  • PDF
Backprop with Approximate Activations for Memory-efficient Network Training
  • 4
  • PDF
Reversible Recurrent Neural Networks
  • 23
  • PDF
Optimal memory-aware backpropagation of deep join networks
  • 6
  • PDF
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 18 REFERENCES
Training Deep Nets with Sublinear Memory Cost
  • 280
  • Highly Influential
  • PDF
Long Short-Term Memory
  • 36,733
  • PDF
Hybrid computing using a neural network with dynamic external memory
  • 966
  • PDF
Asynchronous Methods for Deep Reinforcement Learning
  • 3,891
  • PDF
Speech recognition with deep recurrent neural networks
  • 5,784
  • PDF
Backpropagation Through Time: What It Does and How to Do It
  • 2,488
  • PDF
Learning to Transduce with Unbounded Memory
  • 228
  • PDF
Generating Text with Recurrent Neural Networks
  • 1,082
  • PDF
Neural Turing Machines
  • 1,403
  • PDF
Deep Attention Recurrent Q-Network
  • 72
  • PDF
...
1
2
...