# A procedure for training recurrent networks

@article{Phan2013APF, title={A procedure for training recurrent networks}, author={Manh Cong Phan and Mark H. Beale and Martin T. Hagan}, journal={The 2013 International Joint Conference on Neural Networks (IJCNN)}, year={2013}, pages={1-8} }

In this paper, we introduce a new procedure for efficient training of recurrent neural networks. The new procedure uses a batch training method based on a modified version of the Levenberg-Marquardt algorithm. The information of gradients of individual sequences is used to mitigate the effect of spurious valleys in the error surface of recurrent networks. The method is tested on the modeling and control of several physical systems.

## Figures and Tables from this paper

## 8 Citations

Enhanced recurrent network training

- Computer Science2015 International Joint Conference on Neural Networks (IJCNN)
- 2015

New, more efficient, methods for training recurrent neural networks (RNNs) based on a new understanding of the error surfaces of RNNs, which increase the prediction horizons in a principled way that enables the search algorithms to avoid the spurious valleys.

Application of new training methods for neural model reference control

- Computer ScienceEng. Appl. Artif. Intell.
- 2018

A novel recurrent neural network soft sensor via a differential evolution training algorithm for the tire contact patch

- Computer ScienceNeurocomputing
- 2017

Pronóstico del área de contacto de los neumáticos de un vehículo vía redes neuronales recurrentes

- Computer ScienceRes. Comput. Sci.
- 2016

In this paper, a predictive soft sensor via a Recurrent Neural Network is proposed for sensing the area between the tires and the terrain, which is one of the most important issues in the automotive…

Actuator failure‐tolerant control of an all‐thruster satellite in coupled translational and rotational motion using neural networks

- EngineeringInternational Journal of Adaptive Control and Signal Processing
- 2018

The nonlinear model predictive control (MPC) approach is used to control the coupled translational‐rotational motion of an all‐thruster spacecraft when one of the actuators fails. In order to model…

Wind Estimation Using Quadcopter Motion: A Machine Learning Approach

- Environmental ScienceArXiv
- 2019

Optimization of engine performance parameters and exhaust emissions in compression ignition engine fueled with biodiesel-alcohol blends using taguchi method, multiple regression and artificial neural network

- Environmental Science
- 2020

Estimating Wind Velocity with a Neural Network using Quadcopter Trajectories

- Computer ScienceAIAA Scitech 2019 Forum
- 2019

## References

SHOWING 1-10 OF 12 REFERENCES

Error Surface of Recurrent Neural Networks

- Computer ScienceIEEE Transactions on Neural Networks and Learning Systems
- 2013

Two types of spurious valleys that appear in the error surfaces of recurrent networks are described, which are not affected by the desired network output or by the problem that the network is trying to solve.

Backpropagation Algorithms for a Broad Class of Dynamic Networks

- Computer ScienceIEEE Transactions on Neural Networks
- 2007

It is demonstrated that the BPTT algorithm is more efficient for gradient calculations, but the RTRL algorithm isMore efficient for Jacobian calculations.

New results on recurrent network training: unifying the algorithms and accelerating convergence

- Computer ScienceIEEE Trans. Neural Networks Learn. Syst.
- 2000

An on-line version of the proposed algorithm, which is based on approximating the error gradient, has lower computational complexity in computing the weight update than the competing techniques for most typical problems and reaches the error minimum in a much smaller number of iterations.

Training feedforward networks with the Marquardt algorithm

- Computer ScienceIEEE Trans. Neural Networks
- 1994

The Marquardt algorithm for nonlinear least squares is presented and is incorporated into the backpropagation algorithm for training feedforward neural networks and is found to be much more efficient than either of the other techniques when the network contains no more than a few hundred weights.

Recent advances in efficient learning of recurrent networks

- Computer ScienceESANN
- 2009

This tutorial gives an overview of this recent developments in efficient, biologically plausible recurrent informa- tion processing.

Spurious Valleys in the Error Surface of Recurrent Networks—Analysis and Avoidance

- Computer ScienceIEEE Transactions on Neural Networks
- 2009

It is shown that these error surfaces contain many spurious valleys, and it is demonstrated that the principle mechanism can be understood through the analysis of the roots of random polynomials.

Identification and control of dynamical systems using neural networks

- EngineeringIEEE Trans. Neural Networks
- 1990

It is demonstrated that neural networks can be used effectively for the identification and control of nonlinear dynamical systems and the models introduced are practically feasible.

Neural network design

- Computer Science
- 1995

This book, by the authors of the Neural Network Toolbox for MATLAB, provides a clear and detailed coverage of fundamental neural network architectures and learning rules, as well as methods for training them and their applications to practical problems.

Learning Recurrent Neural Networks with Hessian-Free Optimization

- Computer ScienceICML
- 2011

This work solves the long-outstanding problem of how to effectively train recurrent neural networks on complex and difficult sequence modeling problems which may contain long-term data dependencies and offers a new interpretation of the generalized Gauss-Newton matrix of Schraudolph which is used within the HF approach of Martens.

An introduction to the use of neural networks in control systems

- Computer Science
- 2002

The multilayer perceptron neural network is introduced and how it can be used for function approximation is described and several techniques for improving generalization are discussed.