# The Softmax Nonlinearity: Derivation Using Statistical Mechanics and Useful Properties as a Multiterminal Analog Circuit Element

@inproceedings{Elfadel1993TheSN, title={The Softmax Nonlinearity: Derivation Using Statistical Mechanics and Useful Properties as a Multiterminal Analog Circuit Element}, author={I. Elfadel and J. Wyatt}, booktitle={NIPS}, year={1993} }

We use mean-field theory methods from Statistical Mechanics to derive the "softmax" nonlinearity from the discontinuous winner-take-all (WTA) mapping. We give two simple ways of implementing "softmax" as a multiterminal network element. One of these has a number of important network-theoretic properties. It is a reciprocal, passive, incrementally passive, nonlinear, resistive multiterminal element with a content function having the form of information-theoretic entropy. These properties should… Expand

#### 22 Citations

Convex Potentials and their Conjugates in Analog Mean-Field Optimization

- Mathematics, Computer Science
- Neural Computation
- 1995

Bio-inspired feedback-circuit implementation of discrete, free energy optimizing, winner-take-all computations

- Computer Science, Medicine
- Biological Cybernetics
- 2016

Competitive learning with floating-gate circuits

- Computer Science, Medicine
- IEEE Trans. Neural Networks
- 2002

Contrasting Advantages of Learning With Random Weights and Backpropagation in Non-Volatile Memory Neural Networks

- Computer Science
- IEEE Access
- 2019

Equilibria of Iterative Softmax and Critical Temperatures for Intermittent Search in Self-Organizing Neural Networks

- Mathematics, Computer Science
- Neural Computation
- 2007

Softmax Is Not an Artificial Trick: An Information-Theoretic View of Softmax in Neural Networks

- Mathematics, Computer Science
- ArXiv
- 2019

Optimization via Intermittency with a Self-Organizing Neural Network

- Mathematics, Computer Science
- Neural Computation
- 2005

Feedforward Inhibition and Synaptic Scaling – Two Sides of the Same Coin?

- Computer Science, Medicine
- PLoS Comput. Biol.
- 2012

Optical Rectifying Linear Units for Back-Propagation Learning in a Deep Holographic Convolutional Neural Network

- Computer Science
- IEEE Journal of Selected Topics in Quantum Electronics
- 2020

#### References

SHOWING 1-10 OF 17 REFERENCES

Neurons with graded response have collective computational properties like those of two-state neurons.

- Computer Science, Mathematics
- Proceedings of the National Academy of Sciences of the United States of America
- 1984

A New Method for Mapping Optimization Problems Onto Neural Networks

- Mathematics, Computer Science
- Int. J. Neural Syst.
- 1989

Constrained Nets for Graph Matching and Other Quadratic Assignment Problems

- Mathematics, Medicine
- Neural Comput.
- 1991

The invisible hand algorithm: Solving the assignment problem with statistical physics

- Mathematics, Computer Science
- Neural Networks
- 1994

Parallel distributed processing: explorations in the microstructure of cognition, vol. 1: foundations

- Computer Science
- 1986

Analog neural networks with local competition. I. Dynamics and stability.

- Mathematics, Medicine
- Physical review. E, Statistical physics, plasmas, fluids, and related interdisciplinary topics
- 1993

2The concepts of reciprocity, passivity, content, and co-content are fundamental to nonlinear circuit theory. They are carefully developed in

- Nonlinearity
- 1992