Corpus ID: 8985801

The Softmax Nonlinearity: Derivation Using Statistical Mechanics and Useful Properties as a Multiterminal Analog Circuit Element

  title={The Softmax Nonlinearity: Derivation Using Statistical Mechanics and Useful Properties as a Multiterminal Analog Circuit Element},
  author={I. Elfadel and J. Wyatt},
We use mean-field theory methods from Statistical Mechanics to derive the "softmax" nonlinearity from the discontinuous winner-take-all (WTA) mapping. We give two simple ways of implementing "softmax" as a multiterminal network element. One of these has a number of important network-theoretic properties. It is a reciprocal, passive, incrementally passive, nonlinear, resistive multiterminal element with a content function having the form of information-theoretic entropy. These properties should… Expand
Convex Potentials and their Conjugates in Analog Mean-Field Optimization
  • I. Elfadel
  • Mathematics, Computer Science
  • Neural Computation
  • 1995
Bio-inspired feedback-circuit implementation of discrete, free energy optimizing, winner-take-all computations
Competitive learning with floating-gate circuits
Adaptive CMOS: from biological inspiration to systems-on-a-chip
Equilibria of Iterative Softmax and Critical Temperatures for Intermittent Search in Self-Organizing Neural Networks
  • P. Tiňo
  • Mathematics, Computer Science
  • Neural Computation
  • 2007
Softmax Is Not an Artificial Trick: An Information-Theoretic View of Softmax in Neural Networks
Optimization via Intermittency with a Self-Organizing Neural Network
Optical Rectifying Linear Units for Back-Propagation Learning in a Deep Holographic Convolutional Neural Network
  • K. Wagner, S. McComb
  • Computer Science
  • IEEE Journal of Selected Topics in Quantum Electronics
  • 2020


Neurons with graded response have collective computational properties like those of two-state neurons.
  • J. Hopfield
  • Computer Science, Mathematics
  • Proceedings of the National Academy of Sciences of the United States of America
  • 1984
Analog VLSI and neural systems
A New Method for Mapping Optimization Problems Onto Neural Networks
Constrained Nets for Graph Matching and Other Quadratic Assignment Problems
The invisible hand algorithm: Solving the assignment problem with statistical physics
Global dynamics of winner-take-all networks
  • I. Elfadel
  • Mathematics, Engineering
  • Optics & Photonics
  • 1993
Analog neural networks with local competition. I. Dynamics and stability.
  • Waugh, Westervelt
  • Mathematics, Medicine
  • Physical review. E, Statistical physics, plasmas, fluids, and related interdisciplinary topics
  • 1993
2The concepts of reciprocity, passivity, content, and co-content are fundamental to nonlinear circuit theory. They are carefully developed in
  • Nonlinearity
  • 1992