Appendix G: Thirty Years of Adaptive Neural Networks: Perceptron, Madaline, and Backpropagation

@inproceedings{Widrow2008AppendixGT,
  title={Appendix G: Thirty Years of Adaptive Neural Networks: Perceptron, Madaline, and Backpropagation},
  author={Bernard Widrow and Eugene Walach},
  year={2008}
}
This chapter contains sections titled: Introduction Fundamental Concepts Adaptation ??????-?????? The Minimal Disturbance Principle Error Correction Rules ??????-?????? Single Threshold Element Error Correction Rules ??????-?????? Multi-Element Networks Steepest-Descent Rules ??????-?????? Single Threshold Element Steepest-Descent Rules ??????-?????? Multi-Element Networks Summary Acknowledgments Bibliography 
A Training Methodology for Neural Networks Noise-Filtering when no Training Sets are available for Supervised Learning
  • M. M. Luaces
  • Computer Science
    2006 IEEE International Conference on Computational Intelligence for Measurement Systems and Applications
  • 2006
TLDR
A training methodology combining data simulation, modular neural networks and an interval-splitting strategy for noise-filtering where training data sets are not necessary is proposed.
Hopfield neural networks: a survey
TLDR
The Hopfield neural network is surveyed, introduction of which rekindled interest in the neural networks through the work of Hopfield and others, and some interesting variations of the network or neuron model are noted.
An optimized recursive learning algorithm for three-layer feedforward neural networks for mimo nonlinear system identifications
TLDR
An optimized recursive algorithm is presented for online learning based on matrix operation and optimization methods analytically, which can avoid the trouble to select a proper learning rate for the gradient method.
An Adaptive Spiking Neural Network with
TLDR
A new computational approach that uses neurogenesis and synaptogenesis to continually learn and grow and uses a combination of homeostasis of synapse weights, spike timing, and stochastic forgetting to achieve stable and efficient learning.
Hardware-Amenable Structural Learning for Spike-Based Pattern Classification Using a Simple Model of Active Dendrites
TLDR
This letter presents a spike-based model that employs neurons with functionally distinct dendritic compartments for classifying high-dimensional binary patterns, and its performance is compared against that achieved using support vector machine and extreme learning machine techniques.
Discovering the Computational Relevance of Brain Network Organization
TLDR
A new framework for understanding the role of connectivity in cognition is offered: network coding (encoding/decoding) models, which utilize connectivity to specify the transfer of information via neural activity flow processes, successfully predicting the formation of cognitive representations in empirical neural data.
A neurally inspired pattern recognition approach with latency-phase encoding and precise-spike-driven rule in spiking neural network
  • Na Guo, Rong Xiao, Shaobing Gao, Huajin Tang
  • Computer Science
    2017 IEEE International Conference on Cybernetics and Intelligent Systems (CIS) and IEEE Conference on Robotics, Automation and Mechatronics (RAM)
  • 2017
TLDR
The empirical results show that the proposed approach can perform quite competitive in comparison to other commonly used encoding and learning strategies in SNNs, when evaluating on the gray OCR dataset and the Address-Event-Representation motion dataset for image classification.
Fault Tolerant Neural Predictors for Compression of Sensor Telemetry Data
TLDR
Overall, it is found that even small low- complexity neural networks are more resilient to faults due to the characteristics of their parallel architecture and distributed storage/processing characteristics.
Efficient Sparse Coding in Early Sensory Processing: Lessons from Signal Recovery
TLDR
It is argued that higher level overcompleteness becomes computationally tractable by imposing sparsity on synaptic activity and it is shown that such structural sparsity can be facilitated by statistics based decomposition of the stimuli into typical and atypical parts prior to sparse coding.
An adaptive spiking neural network with Hebbian learning
  • L. Long
  • Computer Science
    2011 IEEE Workshop on Evolving and Adaptive Intelligent Systems (EAIS)
  • 2011
TLDR
A numerical approach to simulating biologically-plausible spiking neural networks with realistic models for the neurons (Hodgkin-Huxley) based on Hebbian approach based on spike timing dependent plasticity (STDP).
...
1
2
3
4
...

References

SHOWING 1-10 OF 155 REFERENCES
What Size Net Gives Valid Generalization?
TLDR
It is shown that if m O(W/ ∊ log N/∊) random examples can be loaded on a feedforward network of linear threshold functions with N nodes and W weights, so that at least a fraction 1 ∊/2 of the examples are correctly classified, then one has confidence approaching certainty that the network will correctly classify a fraction 2 ∊ of future test examples drawn from the same distribution.
Backpropagation separates when perceptrons do
TLDR
The authors show that there are no nonglobal minima in the behavior of the least-squares problem that arises when one attempts to train a feedforward net with no hidden neurons.
Universal approximation using feedforward networks with non-sigmoid hidden layer activation functions
K.M. Hornik, M. Stinchcombe, and H. White (Univ. of California at San Diego, Dept. of Economics Discussion Paper, June 1988; to appear in Neural Networks) showed that multilayer feedforward networks
Backpropagation Can Give Rise to Spurious Local Minima Even for Networks without Hidden Layers
We give an example of a neural net without hidden layers and with a sigmoid transfer function, together with a training set of binary vectors, for which the sum of the squared errors, regarded as a
ART 3: Hierarchical search using chemical transmitters in self-organizing pattern recognition architectures
TLDR
A model to implement parallel search of compressed or distributed pattern recognition codes in a neural network hierarchy is introduced and is a form of hypothesis testing capable of discovering appropriate representations of a nonstationary input environment.
Accelerated Learning in Layered Neural Networks
TLDR
This work investigates the merits of using this error function over t he traditional quad ratic function for gradient descent for conrf­ guity problem and explains the characteristic steepness of the landscape defined by the error function in configuration space.
Improving the learning speed of 2-layer neural networks by choosing initial values of the adaptive weights
  • D. Nguyen, B. Widrow
  • Computer Science
    1990 IJCNN International Joint Conference on Neural Networks
  • 1990
The authors describe how a two-layer neural network can approximate any nonlinear function by forming a union of piecewise linear segments. A method is given for picking initial weights for the
A Learning Algorithm for Continually Running Fully Recurrent Neural Networks
The exact form of a gradient-following learning algorithm for completely recurrent networks running in continually sampled time is derived and used as the basis for practical algorithms for temporal
Neuronlike adaptive elements that can solve difficult learning control problems
TLDR
It is shown how a system consisting of two neuronlike adaptive elements can solve a difficult learning control problem and the relation of this work to classical and instrumental conditioning in animal learning studies and its possible implications for research in the neurosciences.
Neurons with graded response have collective computational properties like those of two-state neurons.
  • J. Hopfield
  • Computer Science, Mathematics
    Proceedings of the National Academy of Sciences of the United States of America
  • 1984
TLDR
A model for a large network of "neurons" with a graded response (or sigmoid input-output relation) is studied and collective properties in very close correspondence with the earlier stochastic model based on McCulloch - Pitts neurons are studied.
...
1
2
3
4
5
...