# Training Recurrent Networks by Evolino

@article{Schmidhuber2007TrainingRN, title={Training Recurrent Networks by Evolino}, author={J{\"u}rgen Schmidhuber and Daan Wierstra and Matteo Gagliolo and Faustino J. Gomez}, journal={Neural Computation}, year={2007}, volume={19}, pages={757-779} }

In recent years, gradient-based LSTM recurrent neural networks (RNNs) solved many previously RNN-unlearnable tasks. Sometimes, however, gradient information is of little use for training RNNs, due to numerous local minima. For such cases, we present a novel method: EVOlution of systems with LINear Outputs (Evolino). Evolino evolves weights to the nonlinear, hidden nodes of RNNs while computing optimal linear mappings from hidden state to output, using methods such as pseudo-inverse-based linear…

## 256 Citations

### Supervised and Evolutionary Learning of Echo State Networks

- Computer SciencePPSN
- 2008

This paper proposes to apply CMA-ES, the state-of-the-art method in evolutionary continuous parameter optimization, to the evolutionary learning of ESN parameters, and shows that the evolutionary ESN obtain results that are comparable with those of the best topology-learning methods.

### Evolutionary Echo State Network: evolving reservoirs in the Fourier space

- Computer Science2022 International Joint Conference on Neural Networks (IJCNN)
- 2022

A new computational model of the ESN type, that represents the reservoir weights in the Fourier space and performs a fine-tuning of these weights applying genetic algorithms in the frequency domain is proposed, thus providing a dimensionality reduction transformation of the initial method.

### Evolving reservoir weights in the frequency domain

- Computer ScienceGECCO Companion
- 2021

This work introduces an evolutionary method for adjusting the reservoir non-null weights, called EvoESN (Evolutionary ESN), which combines an evolutionary search in the Fourier space with supervised learning for the readout weights.

### Knowledge-based recurrent neural networks in Reinforcement Learning

- Computer Science
- 2007

Several methods which have the potential of transferring of knowledge in RL using RNN are presented: Directed Transfer, Cascade-Correlation, Mixture of Expert Systems, and Two-Level Architecture.

### The Power of Linear Recurrent Neural Networks.

- Computer Science
- 2018

P predictive neural networks outperform the previous state-of-the-art for the MSO task with a minimal number of units and can approximate any time-dependent function f(t) given by a number of function values by simply solving a linear equation system.

### The Power of Linear Recurrent Neural Networks - Predictive Neural Networks

- Computer Science
- 2018

Predictive neural networks outperform the previous state-of-the-art for the MSO task with a minimal number of units and can effectively be learned by simply solving a linear equation system; no backpropagation or similar methods are needed.

### Predictive Neural Networks

- Computer ScienceArXiv
- 2018

Predictive neural networks outperform the previous state-of-the-art for the MSO task with a minimal number of units and can be reduced by taking only the most relevant components of the network.

### Re-visiting Reservoir Computing architectures optimized by Evolutionary Algorithms

- Computer ScienceArXiv
- 2022

A systematic brief survey about applications of the EAs on the domain of the recurrent NNs named Reservoir Computing, where EAs are helpful tools to bring out optimal RC architectures.

### An EM Based Training Algorithm for Recurrent Neural Networks

- Computer ScienceICANN
- 2009

This work considers recurrent networks as nonlinear state space models and suggests a training algorithm based on Expectation-Maximization and investigates the use of a Particle Smoother to approximate the E-step and simultaneously estimate the expectations required in the M-step.

## References

SHOWING 1-10 OF 65 REFERENCES

### Evolino: Hybrid Neuroevolution/Optimal Linear Search for Sequence Learning

- Computer ScienceIJCAI
- 2005

A general framework for sequence learning, EVOlution of recurrent systems with LINear outputs (Evolino), which uses evolution to discover good RNN hidden node weights, while using methods such as linear regression or quadratic programming to compute optimal linear mappings from hidden state to output.

### Modeling systems with internal state using evolino

- Computer ScienceGECCO '05
- 2005

This work uses the general framework for sequence learning, EVOlution of recurrent systems with LINear Outputs (Evolino), to discover good RNN hidden node weights through evolution, while using linear regression to compute an optimal linear mapping from hidden state to output.

### Co-evolving recurrent neurons learn deep memory POMDPs

- Computer Science, BiologyGECCO '05
- 2005

A new neuroevolution algorithm called Hierarchical Enforced SubPopulations that simultaneously evolves networks at two levels of granularity: full networks and network components or neurons is introduced.

### Evolino for recurrent support vector machines

- Computer ScienceESANN
- 2006

This work introduces a new class of recurrent, truly sequential SVM-like devices with internal adaptive states, trained by a novel method called EVOlution of systems with KErnel-based outputs (Evoke), an instance of the recent Evolino class of methods.

### Learning to Forget: Continual Prediction with LSTM

- Computer ScienceNeural Computation
- 2000

This work identifies a weakness of LSTM networks processing continual input streams that are not a priori segmented into subsequences with explicitly marked ends at which the network's internal state could be reset, and proposes a novel, adaptive forget gate that enables an LSTm cell to learn to reset itself at appropriate times, thus releasing internal resources.

### Gradient calculations for dynamic recurrent neural networks: a survey

- Computer ScienceIEEE Trans. Neural Networks
- 1995

The author discusses advantages and disadvantages of temporally continuous neural networks in contrast to clocked ones and presents some "tricks of the trade" for training, using, and simulating continuous time and recurrent neural networks.

### Optimization and applications of echo state networks with leaky- integrator neurons

- Computer ScienceNeural Networks
- 2007

### Sequential Behavior and Learning in Evolved Dynamical Neural Networks

- Computer ScienceAdapt. Behav.
- 1994

This article explores the use of a real-valued modular genetic algorithm to evolve continuous-time recurrent neural networks capable of sequential behavior and learning and utilizes concepts from dynamical systems theory to understand the operation of some of these evolved networks.

### Kalman filters improve LSTM network performance in problems unsolvable by traditional recurrent nets

- Computer ScienceNeural Networks
- 2003

### Learning Precise Timing with LSTM Recurrent Networks

- Computer ScienceJ. Mach. Learn. Res.
- 2002

This work finds that LSTM augmented by "peephole connections" from its internal cells to its multiplicative gates can learn the fine distinction between sequences of spikes spaced either 50 or 49 time steps apart without the help of any short training exemplars.