# Appendix G: Thirty Years of Adaptive Neural Networks: Perceptron, Madaline, and Backpropagation

@inproceedings{Widrow2008AppendixGT, title={Appendix G: Thirty Years of Adaptive Neural Networks: Perceptron, Madaline, and Backpropagation}, author={Bernard Widrow and Eugene Walach}, year={2008} }

This chapter contains sections titled: Introduction Fundamental Concepts Adaptation ??????-?????? The Minimal Disturbance Principle Error Correction Rules ??????-?????? Single Threshold Element Error Correction Rules ??????-?????? Multi-Element Networks Steepest-Descent Rules ??????-?????? Single Threshold Element Steepest-Descent Rules ??????-?????? Multi-Element Networks Summary Acknowledgments Bibliography

## Figures and Topics from this paper

## 31 Citations

A Training Methodology for Neural Networks Noise-Filtering when no Training Sets are available for Supervised Learning

- Computer Science2006 IEEE International Conference on Computational Intelligence for Measurement Systems and Applications
- 2006

A training methodology combining data simulation, modular neural networks and an interval-splitting strategy for noise-filtering where training data sets are not necessary is proposed.

Hopfield neural networks: a survey

- Computer Science
- 2007

The Hopfield neural network is surveyed, introduction of which rekindled interest in the neural networks through the work of Hopfield and others, and some interesting variations of the network or neuron model are noted.

An optimized recursive learning algorithm for three-layer feedforward neural networks for mimo nonlinear system identifications

- Computer ScienceIntell. Autom. Soft Comput.
- 2011

An optimized recursive algorithm is presented for online learning based on matrix operation and optimization methods analytically, which can avoid the trouble to select a proper learning rate for the gradient method.

An Adaptive Spiking Neural Network with

- Computer Science
- 2011

A new computational approach that uses neurogenesis and synaptogenesis to continually learn and grow and uses a combination of homeostasis of synapse weights, spike timing, and stochastic forgetting to achieve stable and efficient learning.

Hardware-Amenable Structural Learning for Spike-Based Pattern Classification Using a Simple Model of Active Dendrites

- Computer Science, BiologyNeural Computation
- 2015

This letter presents a spike-based model that employs neurons with functionally distinct dendritic compartments for classifying high-dimensional binary patterns, and its performance is compared against that achieved using support vector machine and extreme learning machine techniques.

Discovering the Computational Relevance of Brain Network Organization

- Medicine, Computer ScienceTrends in Cognitive Sciences
- 2020

A new framework for understanding the role of connectivity in cognition is offered: network coding (encoding/decoding) models, which utilize connectivity to specify the transfer of information via neural activity flow processes, successfully predicting the formation of cognitive representations in empirical neural data.

A neurally inspired pattern recognition approach with latency-phase encoding and precise-spike-driven rule in spiking neural network

- Computer Science2017 IEEE International Conference on Cybernetics and Intelligent Systems (CIS) and IEEE Conference on Robotics, Automation and Mechatronics (RAM)
- 2017

The empirical results show that the proposed approach can perform quite competitive in comparison to other commonly used encoding and learning strategies in SNNs, when evaluating on the gray OCR dataset and the Address-Event-Representation motion dataset for image classification.

Fault Tolerant Neural Predictors for Compression of Sensor Telemetry Data

- Computer ScienceJ. Univers. Comput. Sci.
- 2006

Overall, it is found that even small low- complexity neural networks are more resilient to faults due to the characteristics of their parallel architecture and distributed storage/processing characteristics.

Efficient Sparse Coding in Early Sensory Processing: Lessons from Signal Recovery

- Computer Science, MedicinePLoS Comput. Biol.
- 2012

It is argued that higher level overcompleteness becomes computationally tractable by imposing sparsity on synaptic activity and it is shown that such structural sparsity can be facilitated by statistics based decomposition of the stimuli into typical and atypical parts prior to sparse coding.

An adaptive spiking neural network with Hebbian learning

- Computer Science2011 IEEE Workshop on Evolving and Adaptive Intelligent Systems (EAIS)
- 2011

A numerical approach to simulating biologically-plausible spiking neural networks with realistic models for the neurons (Hodgkin-Huxley) based on Hebbian approach based on spike timing dependent plasticity (STDP).

## References

SHOWING 1-10 OF 155 REFERENCES

What Size Net Gives Valid Generalization?

- Mathematics, Computer ScienceNeural Computation
- 1989

It is shown that if m O(W/ ∊ log N/∊) random examples can be loaded on a feedforward network of linear threshold functions with N nodes and W weights, so that at least a fraction 1 ∊/2 of the examples are correctly classified, then one has confidence approaching certainty that the network will correctly classify a fraction 2 ∊ of future test examples drawn from the same distribution.

Backpropagation separates when perceptrons do

- Mathematics, Computer ScienceInternational 1989 Joint Conference on Neural Networks
- 1989

The authors show that there are no nonglobal minima in the behavior of the least-squares problem that arises when one attempts to train a feedforward net with no hidden neurons.

Universal approximation using feedforward networks with non-sigmoid hidden layer activation functions

- MathematicsInternational 1989 Joint Conference on Neural Networks
- 1989

K.M. Hornik, M. Stinchcombe, and H. White (Univ. of California at San Diego, Dept. of Economics Discussion Paper, June 1988; to appear in Neural Networks) showed that multilayer feedforward networks…

Backpropagation Can Give Rise to Spurious Local Minima Even for Networks without Hidden Layers

- Mathematics, Computer ScienceComplex Syst.
- 1989

We give an example of a neural net without hidden layers and with a sigmoid transfer function, together with a training set of binary vectors, for which the sum of the squared errors, regarded as a…

ART 3: Hierarchical search using chemical transmitters in self-organizing pattern recognition architectures

- Computer ScienceNeural Networks
- 1990

A model to implement parallel search of compressed or distributed pattern recognition codes in a neural network hierarchy is introduced and is a form of hypothesis testing capable of discovering appropriate representations of a nonstationary input environment.

Accelerated Learning in Layered Neural Networks

- Computer ScienceComplex Syst.
- 1988

This work investigates the merits of using this error function over t he traditional quad ratic function for gradient descent for conrf guity problem and explains the characteristic steepness of the landscape defined by the error function in configuration space.

Improving the learning speed of 2-layer neural networks by choosing initial values of the adaptive weights

- Computer Science1990 IJCNN International Joint Conference on Neural Networks
- 1990

The authors describe how a two-layer neural network can approximate any nonlinear function by forming a union of piecewise linear segments. A method is given for picking initial weights for the…

A Learning Algorithm for Continually Running Fully Recurrent Neural Networks

- Computer ScienceNeural Computation
- 1989

The exact form of a gradient-following learning algorithm for completely recurrent networks running in continually sampled time is derived and used as the basis for practical algorithms for temporal…

Neuronlike adaptive elements that can solve difficult learning control problems

- Computer ScienceIEEE Transactions on Systems, Man, and Cybernetics
- 1983

It is shown how a system consisting of two neuronlike adaptive elements can solve a difficult learning control problem and the relation of this work to classical and instrumental conditioning in animal learning studies and its possible implications for research in the neurosciences.

Neurons with graded response have collective computational properties like those of two-state neurons.

- Computer Science, MathematicsProceedings of the National Academy of Sciences of the United States of America
- 1984

A model for a large network of "neurons" with a graded response (or sigmoid input-output relation) is studied and collective properties in very close correspondence with the earlier stochastic model based on McCulloch - Pitts neurons are studied.