Corpus ID: 14385179

A guide to recurrent neural networks and backpropagation

@inproceedings{Bodn2001AGT,
  title={A guide to recurrent neural networks and backpropagation},
  author={Mikael Bod{\'e}n},
  year={2001}
}
This paper provides guidance to some of the concepts surrounding recurrent neural networks. Contrary to feedforward networks, recurrent networks can be sensitive, and be adapted to past inputs. Backpropagation learning is described for feedforward networks, adapted to suit our (probabilistic) modeling needs, and extended to cover recurrent networks. The aim of this brief paper is to set the scene for applying and understanding recurrent neural networks. 
Extensions of recurrent neural network language model
TLDR
Several modifications of the original recurrent neural network language model are presented, showing approaches that lead to more than 15 times speedup for both training and testing phases and possibilities how to reduce the amount of parameters in the model. Expand
Recurrent Neural Networks and Related Models
TLDR
This chapter presents the state-space formulation of the basic RNN as a nonlinear dynamical system, where the recurrent matrix governing the system dynamics is largely unstructured and analyzes the RNNAs a bottom-up, discriminative, dynamic system model against the top-down, generative counterpart of dynamic system. Expand
A Resource Efficient Localized Recurrent Neural Network Architecture and Learning Algorithm
TLDR
This thesis introduces TRTRL, an e¢ cient, low-complexity online learning algorithm for recurrent neural networks based on the real-time recurrent learning (RTRL) algorithm, whereby the sensitivity set of each neuron is reduced to weights associated either with its input or output links. Expand
Problem and Strategy: Overfitting in Recurrent Cycles of Internal Symmetry Networks by Back Propagation
  • Guanzhong Li
  • Computer Science
  • 2009 International Conference on Computational Intelligence and Natural Computing
  • 2009
TLDR
Overfitting in recurrent cycles of Internal Symmetry Networks is analyzed and back propagation is trained for an image processing task. Expand
Artificial neural networks in artificial time series prediction benchmark
artificial neural networks. The characteristic samples of artificial neural network types were selected to be compared in numerous simulations, while influences of key parameters are studied. TheExpand
Fully Bayesian Recurrent Neural Networks for Safe Reinforcement Learning
TLDR
This work builds on Probabilistic Backpropagation to introduce a fully Bayesian Recurrent Neural Network architecture, and demonstrates that the proposed method significantly outperforms a popular approach for obtaining model uncertainties in collision avoidance tasks. Expand
Generating weights and generating vectors to map complex functions with artificial neural networks
  • Richard Neville, S. Holland
  • Computer Science
  • 2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence)
  • 2008
TLDR
A set of vectors are generated from the transformed derived nets that are then used to train an ANN to map one-to-many tasks and the associated rotational symmetries performed are specified. Expand
Large scale recurrent neural network on GPU
TLDR
The experiment results of the Microsoft Research Sentence Completion Challenge demonstrate that the large scale recurrent network without class layer is able to beat the traditional class-based modest-size recurrent network and achieve an accuracy of 47%, the best result achieved by a single recurrent neural network on the same dataset. Expand
Recurrent neural network based language model
TLDR
Results indicate that it is possible to obtain around 50% reduction of perplexity by using mixture of several RNN LMs, compared to a state of the art backoff language model. Expand
rnn : Recurrent Library for Torch
The rnn package provides components for implementing a wide range of Recurrent Neural Networks. It is built withing the framework of the Torch distribution for use with the nn package. The componentsExpand
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 20 REFERENCES
A Learning Algorithm for Continually Running Fully Recurrent Neural Networks
The exact form of a gradient-following learning algorithm for completely recurrent networks running in continually sampled time is derived and used as the basis for practical algorithms for temporalExpand
Learning long-term dependencies with gradient descent is difficult
TLDR
This work shows why gradient based learning algorithms face an increasingly difficult problem as the duration of the dependencies to be captured increases, and exposes a trade-off between efficient learning by gradient descent and latching on information for long periods. Expand
Context-free and context-sensitive dynamics in recurrent neural networks
TLDR
The dynamics in recurrent neural networks that process context-free languages can also be employed in processing some context-sensitive languages, and this continuity of mechanism between language classes contributes to the understanding of neural networks in modelling language learning and processing. Expand
The Dynamics of Discrete-Time Computation, with Application to Recurrent Neural Networks and Finite State Machine Extraction
TLDR
It is shown that an RNN performing a finite state computation must organize its state space to mimic the states in the minimal deterministic finite state machine that can perform that computation, and a precise description of the attractor structure of such systems is given. Expand
A Recurrent Neural Network that Learns to Count
TLDR
This research employs standard backpropagation training techniques for a recurrent neural network in the task of learning to predict the next character in a simple deterministic CFL (DCFL), and shows that an RNN can learn to recognize the structure of a simple DCFL. Expand
Backpropagation: the basic theory
TLDR
Since the publication of the PDP volumes in 1986, learning by backpropagation has become the most popular method of training neural networks because of the underlying simplicity and relative power of the algorithm. Expand
Fool's Gold: Extracting Finite State Machines from Recurrent Network Dynamics
TLDR
How sensitivity to initial conditions and discrete measurements can trick these extraction methods to return illusory finite state descriptions is described. Expand
Finite State Automata and Simple Recurrent Networks
TLDR
A network architecture introduced by Elman (1988) for predicting successive elements of a sequence and shows that long distance sequential contingencies can be encoded by the network even if only subtle statistical properties of embedded strings depend on the early information. Expand
Learning and Extracting Finite State Automata with Second-Order Recurrent Neural Networks
TLDR
It is shown that a recurrent, second-order neural network using a real-time, forward training algorithm readily learns to infer small regular grammars from positive and negative string training samples, and many of the neural net state machines are dynamically stable, that is, they correctly classify many long unseen strings. Expand
Learning to predict a context-free language: analysis of dynamics in recurrent hidden units
TLDR
It is concluded that any gradient-based learning method will experience difficulty in learning the language due to the nature of the space, and that a more promising approach to improving learning performance may be to make weight changes in a non-independent manner. Expand
...
1
2
...