Diagrammatic Derivation of Gradient Algorithms for Neural Networks

@article{Wan1996DiagrammaticDO,
  title={Diagrammatic Derivation of Gradient Algorithms for Neural Networks},
  author={Eric A. Wan and Françoise Beaufays},
  journal={Neural Computation},
  year={1996},
  volume={8},
  pages={182-201}
}
Deriving gradient algorithms for time-dependent neural network structures typically requires numerous chain rule expansions, diligent bookkeeping, and careful manipulation of terms. In this paper, we show how to derive such algorithms via a set of simple block diagram manipulation rules. The approach provides a common framework to derive popular algorithms including backpropagation and backpropagation-through-time without a single chain rule expansion. Additional examples are provided for a… 
A Diagrammatic Approach to GradientDerivations for Neural
TLDR
It is shown that an eecient gradient descent algorithm may be formulated for any network structure with virtually no eeort using a set of simple block diagram manipulation rules.
Diagrammatic Methods for Deriving and Relating Temporal Neural Network Algorithms
In this paper, we present an approach to derive gradient algorithms for time-dependent neural networks that is based on a set of simple block diagram manipulation rules. We further introduce a
Dynamical systems learning by a circuit theoretic approach
  • P. Campolucci, A. Uncini, F. Piazza
  • Computer Science, Mathematics
    ISCAS '98. Proceedings of the 1998 IEEE International Symposium on Circuits and Systems (Cat. No.98CH36187)
  • 1998
TLDR
A new general method is derived for both on-line and off-line backward gradient computation of a system output, or cost function, with respect to system parameters, using a circuit theoretic approach.
Showing the equivalence of two training algorithms. I
  • M. Koch, I. Fischer, M. Berthold
  • Computer Science
    1998 IEEE International Joint Conference on Neural Networks Proceedings. IEEE World Congress on Computational Intelligence (Cat. No.98CH36227)
  • 1998
TLDR
Graph transformations are used to show the equivalence of two training algorithms for recurrent neural networks; back Propagation through time, and a variant of real-time backpropagation.
Specifying intrinsically adaptive architectures
  • S. Lucas
  • Computer Science
    2000 IEEE Symposium on Combinations of Evolutionary Computation and Neural Networks. Proceedings of the First IEEE Symposium on Combinations of Evolutionary Computation and Neural Networks (Cat. No.00
  • 2000
TLDR
The paper describes a method for specifying (and evolving) intrinsically adaptive neural architectures that have back-propagation style gradient descent behavior built into them at a cellular level and can be seamlessly integrated with any method for evolving neural network architectures.
Formalizing Neural Networks using Graph
In this paper a unifying framework for the formal-ization of diierent types of Neural Networks and the corresponding algorithms for computation and training is presented. The used Graph
A general approach to gradient based learning in multirate systems and neural networks
  • F. Rosati, P. Campolucci, F. Piazza
  • Computer Science
    Proceedings of the IEEE-INNS-ENNS International Joint Conference on Neural Networks. IJCNN 2000. Neural Computing: New Challenges and Perspectives for the New Millennium
  • 2000
TLDR
A new algorithm to estimate the derivative of the output with respect to an internal parameter have been proposed in the literature for discrete-time systems and can be employed for gradient-based learning of general multirate circuits, such as the new "multirate" neural networks.
Formalizing neural networks using graph transformations
  • M. Berthold, I. Fischer
  • Computer Science
    Proceedings of International Conference on Neural Networks (ICNN'97)
  • 1997
In this paper a unifying framework for the formalization of different types of neural networks and the corresponding algorithms for computation and training is presented. The used graph
Formalizing Neural Networks
TLDR
This work describes different kinds of training algorithms within this theoretical methodology, from simple parameter adapting algorithms, such as Error Backpropagation, to more complex algorithms that also change the network's topology.
A Signal-Flow-Graph Approach to On-line Gradient Calculation
TLDR
A method for both on-line and batch-backward gradient computation of a system output or cost function with respect to system parameters is derived by the SFG representation theory and its known properties.
...
...

References

SHOWING 1-10 OF 55 REFERENCES
Network Reciprocity: A Simple Approach to Derive Gradient Algorithms for Arbitrary Neural Network St
TLDR
This paper shows how to use the principle of Network Reciprocity to derive backpropagation algorithms via a set of block diagram manipulation rules for time-dependent neural network structures, including feedforward and feedback systems.
A Learning Algorithm for Continually Running Fully Recurrent Neural Networks
The exact form of a gradient-following learning algorithm for completely recurrent networks running in continually sampled time is derived and used as the basis for practical algorithms for temporal
Learning of neural networks using their adjoint systems
  • K. Matsuoka
  • Computer Science
    Systems and Computers in Japan
  • 1991
TLDR
By employing the proposed method, it is possible to realize the network with connection coefficients with a spatial structure, to determine the initial state of the network and to execute the learning for the input, not only for the connection coefficients.
Backpropagation Through Time: What It Does and How to Do It
TLDR
This paper first reviews basic backpropagation, a simple method which is now being widely used in areas like pattern recognition and fault diagnosis, and describes further extensions of this method, to deal with systems other than neural networks, systems involving simultaneous equations or true recurrent networks, and other practical issues which arise with this method.
Relating Real-Time Backpropagation and Backpropagation-Through-Time: An Application of Flow Graph Interreciprocity
TLDR
It is shown that signal flow graph theory provides a simple way to relate two popular algorithms used for adapting dynamic neural networks, real-time backpropagation and back Propagation-through-time, with a theoretical argument to verify that both flow graphs implement the same overall weight update.
Neural Networks and Nonlinear Adaptive Filtering: Unifying Concepts and New Algorithms
TLDR
It is shown that neural networks can be considered as general nonlinear filters that can be trained adaptively, that is, that can undergo continual training with a possibly infinite number of time-ordered examples.
FIR and IIR Synapses, a New Neural Network Architecture for Time Series Modeling
TLDR
It is shown that the local-recurrent global-feedforward model performs better than the local/local recurrent global feedforward model and the learning rule minimizing a mean square error criterion is derived.
Finite impulse response neural networks with applications in time series prediction
TLDR
A dynamic network is proposed which uses Finite Impulse Response (FIR) linear filters to model the processes of axonal transport, synaptic modulation, and membrane charge dissipation, and a unifying principle called Network Reciprocity is introduced.
Backpropagation Applied to Handwritten Zip Code Recognition
TLDR
This paper demonstrates how constraints from the task domain can be integrated into a backpropagation network through the architecture of the network, successfully applied to the recognition of handwritten zip code digits provided by the U.S. Postal Service.
...
...