Recurrent Highway Networks
@article{Zilly2017RecurrentHN, title={Recurrent Highway Networks}, author={Julian G. Zilly and R. Srivastava and J. Koutn{\'i}k and J. Schmidhuber}, journal={ArXiv}, year={2017}, volume={abs/1607.03474} }
Many sequential processing tasks require complex nonlinear transition functions from one step to the next. However, recurrent neural networks with 'deep' transition functions remain difficult to train, even when using Long Short-Term Memory (LSTM) networks. We introduce a novel theoretical analysis of recurrent networks based on Gersgorin's circle theorem that illuminates several modeling and optimization issues and improves our understanding of the LSTM cell. Based on this analysis we propose… CONTINUE READING
Supplemental Code
Github Repo
Via Papers with Code
Recurrent Highway Networks - Implementations for Tensorflow, Torch7, Theano and Brainstorm
Figures, Tables, and Topics from this paper
312 Citations
Recurrent Highway Networks With Grouped Auxiliary Memory
- Computer Science
- IEEE Access
- 2019
- 5
- Highly Influenced
Residual Recurrent Highway Networks for Learning Deep Sequence Prediction Models
- Computer Science
- Journal of Grid Computing
- 2018
- 8
- Highly Influenced
Highway State Gating for Recurrent Highway Networks: Improving Information Flow Through Time
- Computer Science, Mathematics
- CSCML
- 2018
- PDF
Highway-LSTM and Recurrent Highway Networks for Speech Recognition
- Computer Science
- INTERSPEECH
- 2017
- 21
- Highly Influenced
- PDF
References
SHOWING 1-10 OF 75 REFERENCES
Highway long short-term memory RNNS for distant speech recognition
- Computer Science, Engineering
- 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
- 2016
- 225
- PDF
Context dependent recurrent neural network language model
- Computer Science
- 2012 IEEE Spoken Language Technology Workshop (SLT)
- 2012
- 477
- PDF
LSTM: A Search Space Odyssey
- Computer Science, Medicine
- IEEE Transactions on Neural Networks and Learning Systems
- 2017
- 2,334
- PDF