Learning to Forget: Continual Prediction with LSTM

@article{Gers2000LearningTF,
  title={Learning to Forget: Continual Prediction with LSTM},
  author={F. Gers and J. Schmidhuber and Fred A. Cummins},
  journal={Neural Computation},
  year={2000},
  volume={12},
  pages={2451-2471}
}
  • F. Gers, J. Schmidhuber, Fred A. Cummins
  • Published 2000
  • Mathematics, Computer Science, Medicine
  • Neural Computation
    • Long short-term memory (LSTM; Hochreiter & Schmidhuber, 1997) can solve numerous tasks not solvable by previous learning algorithms for recurrent neural networks (RNNs). We identify a weakness of LSTM networks processing continual input streams that are not a priori segmented into subsequences with explicitly marked ends at which the network's internal state could be reset. Without resets, the state may grow indefinitely and eventually cause the network to break down. Our remedy is a novel… CONTINUE READING
    View on MIT Press
    citeseerx.ist.psu.edu
    Top 3 of 1,573 Citations
    A generalized LSTM-like training algorithm for second-order recurrent neural networks
    Derek Monner, J. ReggiaNeural Networks2012
    52
    Learning Precise Timing with LSTM Recurrent Networks
    F. Gers, N. N. Schraudolph, J. SchmidhuberJ. Mach. Learn. Res.2002
    935
    Learning compact recurrent neural networks
    Zhiyun Lu, V. Sindhwani, T. Sainath2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)2016
    64

    Supplemental Presentations

    PRESENTATION SLIDES
    Learning to Forget: Continual Prediction with LSTM
    Explore Further
    Discover more papers related to the topics discussed in this paper
    1,573 Citations
    Fields of Study

    Fields of Study

    Citation Type

    Citation Type

    A generalized LSTM-like training algorithm for second-order recurrent neural networks
    52
    Learning Precise Timing with LSTM Recurrent Networks
    935
    Learning compact recurrent neural networks
    64
    EXTRAPOLATED INPUT NETWORK SIMPLIFICATION
    • 2019
    Radically Simplifying Gated Recurrent Architectures Without Loss of Performance
    Gated Orthogonal Recurrent Units: On Learning to Forget
    55
    Training Recurrent Networks by Evolino
    224
    The Performance of LSTM and BiLSTM in Forecasting Time Series
    7
    A review on the long short-term memory model
    Understanding LSTM - a tutorial into Long Short-Term Memory Recurrent Neural Networks
    8

    References

    Publications referenced by this paper.
    SHOWING 1-10 OF 57 REFERENCES
    Learning to forget: continual prediction with LSTM
    1198
    Learning to Forget: Continual Prediction with Lstm Learning to Forget: Continual Prediction with Lstm
    11
    Long Short-Term Memory
    29817
    LSTM recurrent networks learn simple context-free and context-sensitive languages
    412
    Encoding sequential structure: experience with the real-time recurrent learning algorithm
    41
    Learning long-term dependencies with gradient descent is difficult
    4408
    Learning long-term dependencies in NARX recurrent neural networks
    502
    Gradient calculations for dynamic recurrent neural networks: a survey
    549
    The Recurrent Cascade-Correlation Architecture
    169
    Finite State Automata and Simple Recurrent Networks
    410