Learning Complex, Extended Sequences Using the Principle of History Compression

@article{Schmidhuber1992LearningCE,
  title={Learning Complex, Extended Sequences Using the Principle of History Compression},
  author={J. Schmidhuber},
  journal={Neural Computation},
  year={1992},
  volume={4},
  pages={234-242}
}
Previous neural network learning algorithms for sequence processing are computationally expensive and perform poorly when it comes to long time lags. This paper first introduces a simple principle for reducing the descriptions of event sequences without loss of information. A consequence of this principle is that only unexpected inputs can be relevant. This insight leads to the construction of neural architectures that learn to divide and conquer by recursively decomposing sequences. I describe… Expand
380 Citations
Continuous history compression
  • 18
  • PDF
Learning Unambiguous Reduced Sequence Descriptions
  • 44
  • PDF
Variable Computation in Recurrent Neural Networks
  • 39
  • PDF
Decoupling Hierarchical Recurrent Neural Networks With Locally Computable Losses
  • Highly Influenced
  • PDF
A Clockwork RNN
  • 331
  • PDF
The Vanishing Gradient Problem During Learning Recurrent Neural Nets and Problem Solutions
  • S. Hochreiter
  • Mathematics, Computer Science
  • Int. J. Uncertain. Fuzziness Knowl. Based Syst.
  • 1998
  • 1,090
Hierarchical Recurrent Neural Networks for Long-Term Dependencies
  • 287
  • PDF
Eigenvalue Normalized Recurrent Neural Networks for Short Term Memory
  • PDF
Hierarchical Conflict Propagation: Sequence Learning in a Recurrent Deep Neural Network
  • PDF
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 20 REFERENCES
Learning to Control Fast-weight Memories: an Alternative to Dynamic Recurrent Networks
  • 46
  • Highly Influential
Adaptive Decomposition Of Time
  • 14
  • Highly Influential
  • PDF
Experimental Analysis of the Real-time Recurrent Learning Algorithm
  • 330
Learning Factorial Codes by Predictability Minimization
  • 144
Incremental Development of Complex Behaviors
  • 17
  • PDF
Learning with Delayed Reinforcement Through Attention-Driven Buffering
  • C. Myers
  • Computer Science
  • Int. J. Neural Syst.
  • 1991
  • 10
...
1
2
...