Corpus ID: 231855708

Formalising the Use of the Activation Function in Neural Inference

  title={Formalising the Use of the Activation Function in Neural Inference},
  author={Dalton A R Sakthivadivel},
We investigate how activation functions can be used to describe neural firing in an abstract way, and in turn, why they work well in artificial neural networks. We discuss how a spike in a biological neurone belongs to a particular universality class of phase transitions in statistical physics. We then show that the artificial neurone is, mathematically, a mean field model of biological neural membrane dynamics, which arises from modelling spiking as a phase transition. This allows us to treat… Expand

Figures from this paper

Characterising the Non-Equilibrium Dynamics of a Neural Cell
This work combines nonequilibrium statistical mechanics and dynamical systems theory to characterise the nature of the neural resting state, and its relationship to firing, which provides a multiscale model of the Neural cell, from molecules to spikes, and explains various phenomena in a unified manner. Expand
A Multiscale Model of Traumatic Brain Injury Suggests Possible Mechanisms of Post-Traumatic Psychosis
A possible mechanism for post-traumatic psychosis is evaluated, shedding light on the anomalous nature of psychosis as over-activity and brain injury as destruction, and a proof-of-principle of advanced multiscale modelling methods in computational psychiatry and neuromedicine is found. Expand
Evaluating the Performance of Extreme Learning Machine Technique for Ore Grade Estimation
ELM-Sigmoid can be used by field practitioners as a reliable alternative ore grade estimation technique and compared with the state-of-the-art benchmark methods of Backpropagation Neural Network (BPNN) and Ordinary Kriging (OK). Expand


The Dynamic Brain: From Spiking Neurons to Neural Masses and Cortical Fields
It is argued that elaborating principled and informed models is a prerequisite for grounding empirical neuroscience in a cogent theoretical framework, commensurate with the achievements in the physical sciences. Expand
Biologically Realistic Mean-Field Models of Conductance-Based Networks of Spiking Neurons with Adaptation
It is concluded that mean-field models based on networks of adaptive exponential integrate-and-fire excitatory and inhibitory neurons are biologically realistic in the sense that they can capture both spontaneous and evoked activity and naturally appear as candidates to build very large-scale models involving multiple brain areas. Expand
In vitro neural networks minimise variational free energy
This work stimulated an in vitro cortical cell culture with stimulus trains that had a known statistical structure to address the neuronal encoding problem from a Bayesian perspective and found evidence for functional specialisation and segregation in the in vitro neural network that reproduced in silico learning via free energy minimisation. Expand
A logical calculus of the ideas immanent in nervous activity
It is shown that many particular choices among possible neurophysiological assumptions are equivalent, in the sense that for every net behaving under one assumption, there exists another net which behaves under the other and gives the same results, although perhaps not in the same time. Expand
On the Impact of the Activation Function on Deep Neural Networks Training
A comprehensive theoretical analysis of the Edge of Chaos is given and it is shown that one can indeed tune the initialization parameters and the activation function in order to accelerate the training and improve the performance. Expand
Criticality in the brain: A synthesis of neurobiology, models and cognition
The core nature of criticality, the evidence supporting its role in neural systems and its explanatory potential in brain health and disease are reviewed. Expand
Mean Field Residual Networks: On the Edge of Chaos
It is shown, theoretically as well as empirically, that common initializations such as the Xavier or the He schemes are not optimal for residual networks, because the optimal initialization variances depend on the depth. Expand
Cortical Circuit Dynamics Are Homeostatically Tuned to Criticality In Vivo
The results suggest that criticality in excitatory networks is established by inhibitory plasticity and architecture, and suggest a key role for homeostatic regulation of inhibition in visual cortical circuits. Expand
Excitatory and inhibitory interactions in localized populations of model neurons.
It is proved that the existence of limit cycle dynamics in response to one class of stimuli implies theexistence of multiple stable states and hysteresis in responseTo this work, coupled nonlinear differential equations are derived for the dynamics of spatially localized populations containing both excitatory and inhibitory model neurons. Expand
(Physio)logical circuits: the intellectual origins of the McCulloch-Pitts neural networks.
  • Tara H. Abraham
  • Sociology, Medicine
  • Journal of the history of the behavioral sciences
  • 2002
Examining the intellectual backgrounds of McCulloch and Pitts as individuals will show that besides being an important event in the history of cybernetics proper, the McCulloch-Pitts collaboration was an important result of early twentieth-century efforts to apply mathematics to neurological phenomena. Expand