A Practical Guide to Applying Echo State Networks

@inproceedings{Lukoeviius2012APG,
  title={A Practical Guide to Applying Echo State Networks},
  author={Mantas Luko{\vs}evi{\vc}ius},
  booktitle={Neural Networks: Tricks of the Trade},
  year={2012}
}
Reservoir computing has emerged in the last decade as an alternative to gradient descent methods for training recurrent neural networks. Echo State Network (ESN) is one of the key reservoir computing “flavors”. While being practical, conceptually simple, and easy to implement, ESNs require some experience and insight to achieve the hailed good performance in many tasks. Here we present practical techniques and recommendations for successfully applying ESNs, as well as some more advanced… 
Integer Echo State Networks: Hyperdimensional Reservoir Computing
TLDR
The intESN architecture is verified with typical tasks in reservoir computing: memorizing of a sequence of inputs; classifying time-series; learning dynamic processes.
Performance optimization of echo state networks through principal neuron reinforcement
TLDR
A neuroplasticity-inspired algorithm was proposed in this study to alter the strength of internal synapses within the reservoir towards the goal of optimizing the neuronal dynamics of the ESN pertaining to the specific problem to be solved.
On the Statistical Challenges of Echo State Networks and Some Potential Remedies
TLDR
It is found ESN can track in short term for most dataset, but it collapses in the long run, so this work aggregates many of ESNs into an ensemble to lower the variance and stabilize the system by stochastic replications and bootstrapping of input data.
Restricted Echo State Networks
TLDR
It is shown that loops and cycles can replicate each other, while discrete time is a necessity for the suitability of the reservoir, and the potential limitation of energy conservation is equivalent to limiting the spectral radius.
A Review of Designs and Applications of Echo State Networks
TLDR
The ESN-based methods are categorized to basic ESNs, DeepESNs and combinations, then analyzed them from the perspective of theoretical studies, network designs and specific applications to discuss the challenges and opportunities of ESNs.
Regular echo state networks: simple and accurate reservoir models to real-world applications
TLDR
The results revealed that some problems can be considerably benefited from some level of organization in the reservoir, such as those provided by regular or small-world network models; and that the non-linear support vector machine classifier achieved the best predictive performance, although it was statistically comparable with the k-nearest neighbors one, which has much smaller time complexity.
Growing Echo-State Network With Multiple Subreservoirs
TLDR
Simulation results show that the proposed GESN has better prediction performance and faster leaning speed than some ESNs with fixed sizes and topologies.
Evolutionary Echo State Network: evolving reservoirs in the Fourier space
TLDR
A new computational model of the ESN type, that represents the reservoir weights in the Fourier space and performs a fine-tuning of these weights applying genetic algorithms in the frequency domain is proposed, thus providing a dimensionality reduction transformation of the initial method.
Computational analysis of memory capacity in echo state networks
...
...

References

SHOWING 1-10 OF 72 REFERENCES
Reservoir Computing Trends
TLDR
A brief introduction into basic concepts, methods, insights, current developments, and some applications of RC are given.
Reservoir computing approaches to recurrent neural network training
Echo State Networks with Trained Feedbacks
TLDR
This report explores possible directions in which the theoretical findings could be applied to increase the computational power of Echo State Networks and proposes a modification of ESNs called Layered ESNs.
Echo State Gaussian Process
TLDR
A novel Bayesian approach toward ESNs is introduced, the echo state Gaussian process (ESGP), which combines the merits of ESNs and Gaussian processes to provide a more robust alternative to conventional reservoir computing networks while also offering a measure of confidence on the generated predictions.
Recurrent Kernel Machines: Computing with Infinite Echo State Networks
TLDR
The concept of ESNs is extended to infinite-sized recurrent neural networks, which can be considered recursive kernels that subsequently can be used to create recursive support vector machines.
Long Short-Term Memory in Echo State Networks: Details of a Simulation Study
TLDR
This report exposes ESNs to a series of synthetic benchmark tasks that have been used in the literature to study the learnability of long-range temporal dependencies and provides all the detail necessary to replicate these experiments.
Adaptive Nonlinear System Identification with Echo State Networks
TLDR
An online adaptation scheme based on the RLS algorithm known from adaptive linear systems is described, as an example, a 10-th order NARMA system is adaptively identified.
The copula echo state network
...
...