• Corpus ID: 15467150

The''echo state''approach to analysing and training recurrent neural networks

@inproceedings{Jaeger2001TheechoST,
  title={The''echo state''approach to analysing and training recurrent neural networks},
  author={Herbert Jaeger},
  year={2001}
}
The report introduces a constructive learning algorithm for recurrent neural networks, which modifies only the weights to output units in order to achieve the learning task. key words: recurrent neural networks, supervised learning Zusammenfassung. Der Report führt ein konstruktives Lernverfahren für rekurrente neuronale Netze ein, welches zum Erreichen des Lernzieles lediglich die Gewichte der zu den Ausgabeneuronen führenden Verbindungen modifiziert. Stichwörter: rekurrente neuronale Netze… 
minds.jacobs-university.de
1,887 Citations
Highly Influential Citations
433
Background Citations
1,059
Methods Citations
624
Results Citations
45

Figures from this paper

1,887 Citations

Citation Type

Citation Type

Erratum note for the techreport, The "echo state" approach to analysing and training recurrent neural networks
TLDR
In the technical report The “echo state” approach to analysing and training recurrent neural networks from 2001, a number of equivalent conditions for the echo state property were given but one of them is too weak and not equivalent to the others.
Training Echo State Networks with Neuroscale
  • Tzai-Der Wang, C. Fyfe
  • Computer Science
    2011 International Conference on Technologies and Applications of Artificial Intelligence
  • 2011
TLDR
An artificial neural network is created which is a version of echo state machines, ESNs, which is optimal for projecting multivariate time series data onto a low dimensional manifold so that the structure in the time series can be identified by eye.
Modeling neural plasticity in echo state networks for classification and regression
Recurrent neural networks: methods and applications to non-linear predictions
TLDR
A novel approach to address the main problem in training recurrent neural networks, the so-called vanishing gradient problem, is developed, which allows us to train a very simple recurrent neural network, making the gradient not to vanish even after many time-steps.
Temporal overdrive recurrent neural network
TLDR
This work presents a novel recurrent neural network architecture designed to model systems characterized by multiple characteristic timescales in their dynamics, composed by several recurrent groups of neurons that are trained to separately adapt to each timescale.
Reservoir computing approaches to recurrent neural network training
A Practical Guide to Applying Echo State Networks
TLDR
Practical techniques and recommendations for successfully applying Echo State Network, as well as some more advanced application-specific modifications are presented.
Gated Echo State Networks: a preliminary study
TLDR
It is observed that the use of randomized gates by itself can increase the predictive accuracy of a ESN, but this increase is not meaningful when compared with other techniques.
Learning Input and Recurrent Weight Matrices in Echo State Networks
TLDR
This proposed method exploits linearity of activation function in the output units to formulate the relationships amongst the various matrices in an RNN, which results in the gradient of the cost function having an analytical form and being more accurate.
Recent advances in efficient learning of recurrent networks
TLDR
This tutorial gives an overview of this recent developments in efficient, biologically plausible recurrent informa- tion processing.
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 18 REFERENCES
New results on recurrent network training: unifying the algorithms and accelerating convergence
TLDR
An on-line version of the proposed algorithm, which is based on approximating the error gradient, has lower computational complexity in computing the weight update than the competing techniques for most typical problems and reaches the error minimum in a much smaller number of iterations.
Gradient calculations for dynamic recurrent neural networks: a survey
TLDR
The author discusses advantages and disadvantages of temporally continuous neural networks in contrast to clocked ones and presents some "tricks of the trade" for training, using, and simulating continuous time and recurrent neural networks.
Learning to Forget: Continual Prediction with LSTM
TLDR
This work identifies a weakness of LSTM networks processing continual input streams that are not a priori segmented into subsequences with explicitly marked ends at which the network's internal state could be reset, and proposes a novel, adaptive forget gate that enables an LSTm cell to learn to reset itself at appropriate times, thus releasing internal resources.
Real-Time Computing Without Stable States: A New Framework for Neural Computation Based on Perturbations
TLDR
A new computational model for real-time computing on time-varying input that provides an alternative to paradigms based on Turing machines or attractor neural networks, based on principles of high-dimensional dynamical systems in combination with statistical learning theory and can be implemented on generic evolved or found recurrent circuitry.
Learning dynamical systems by recurrent neural networks from orbits
Applying LSTM to Time Series Predictable through Time-Window Approaches
TLDR
It is found that LSTM''s superiority does not carry over to certain simpler time series prediction tasks solvable by time window approaches: the Mackey-Glass series and the Santa Fe FIR laser emission series.
Local Modeling Optimization for Time Series Prediction
TLDR
A method of optimizing parameters of local models so as to minimize the leave-one-out cross-validation error is described, which reduces the burden on the user to pick appropriate values and improves the prediction accuracy.
Adaptive control using neural networks and approximate models
TLDR
A case is made in this paper that such approximate input-output models warrant a detailed study in their own right in view of their mathematical tractability as well as their success in simulation studies.
Synaptic plasticity: taming the beast
TLDR
This work reviews three Hebbian forms of plasticity—synaptic scaling, spike-timing dependent plasticity and synaptic redistribution—and discusses their functional implications.
Innovations in local modeling for time series prediction
TLDR
New optimization algorithms are introduced that improve the model accuracy by adjusting the initial parameter values provided by the user in this work, which take advantage of local models’ ability to efficiently calculate the leave-one-out cross-validation error.
...
1
2
...