Subdominant Dense Clusters Allow for Simple Learning and High Computational Performance in Neural Networks with Discrete Synapses.

@article{Baldassi2015SubdominantDC,
  title={Subdominant Dense Clusters Allow for Simple Learning and High Computational Performance in Neural Networks with Discrete Synapses.},
  author={Carlo Baldassi and Alessandro Ingrosso and Carlo Lucibello and Luca Saglietti and Riccardo Zecchina},
  journal={Physical review letters},
  year={2015},
  volume={115 12},
  pages={
          128101
        }
}
We show that discrete synaptic weights can be efficiently used for learning in large scale neural systems, and lead to unanticipated computational performance. We focus on the representative case of learning random patterns with binary synapses in single layer networks. The standard statistical analysis shows that this problem is exponentially dominated by isolated solutions that are extremely hard to find algorithmically. Here, we introduce a novel method that allows us to find analytical… Expand
Learning may need only a few bits of synaptic precision.
Learning in neural networks poses peculiar challenges when using discretized rather then continuous synaptic states. The choice of discrete synapses is motivated by biological reasoning andExpand
On the role of synaptic stochasticity in training low-precision neural networks
TLDR
It is shown that a neural network model with stochastic binary weights naturally gives prominence to exponentially rare dense regions of solutions with a number of desirable properties such as robustness and good generalization performance, while typical solutions are isolated and hard to find. Expand
Unreasonable effectiveness of learning neural networks: From accessible states and robust ensembles to basic algorithmic schemes
TLDR
It is shown that there are regions of the optimization landscape that are both robust and accessible and that their existence is crucial to achieve good performance on a class of particularly difficult learning problems, and an explanation of this good performance is proposed in terms of a nonequilibrium statistical physics framework. Expand
Learning through atypical "phase transitions" in overparameterized neural networks
TLDR
Methods from statistical physics of disordered systems are used to analytically study the computational fallout of overparameterization in nonconvex neural network models and find that there exist a gap between the SAT/UNSAT interpolation transition where solutions begin to exist and the point where algorithms start to find solutions, i.e. where accessible solutions appear. Expand
Mean-field inference methods for neural networks
TLDR
A selection of classical mean-field methods and recent progress relevant for inference in neural networks are reviewed, and the principles of derivations of high-temperature expansions, the replica method and message passing algorithms are reminded, highlighting their equivalences and complementarities. Expand
Binary perceptron: efficient algorithms can find solutions in a rare well-connected cluster
It was recently shown that almost all solutions in the symmetric binary perceptron are isolated, even at low constraint densities, suggesting that finding typical solutions is hard. In contrast, someExpand
Understanding the computational difficulty of a binary-weight perceptron and the advantage of input sparseness
TLDR
A perceptron model, which associates binary input patterns with outputs using binary (0 or 1) weights, modeling a single neuron receiving excitatory inputs is studied, highlighting the heterogeneity of learning dynamics of weights. Expand
Statistical physics of neural systems
TLDR
This work represents learning as an optimization problem, actually implementing a local search, in the synaptic space, of specific configurations, known as solutions and making a neural network able to accomplish a series of different tasks. Expand
CORSO DI LAUREA MAGISTRALE IN FISICA OUT-OF-EQUILIBRIUM ANALYSIS OF SIMPLE NEURAL NETWORKS
We consider a novel approach to learning in neural networks with discrete synapses [1, 2, 3] and discuss its possible extensions to simple continuous neural networks. The problem of learning isExpand
Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data
TLDR
By optimizing the PAC-Bayes bound directly, Langford and Caruana (2001) are able to extend their approach and obtain nonvacuous generalization bounds for deep stochastic neural network classifiers with millions of parameters trained on only tens of thousands of examples. Expand
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 30 REFERENCES
Efficient supervised learning in networks with binary synapses
TLDR
A neurobiologically plausible on-line learning algorithm that derives from belief propagation algorithms that performs remarkably well in a model neuron with binary synapses, and a finite number of “hidden” states per synapse, that has to learn a random classification task. Expand
Origin of the computational hardness for learning with binary synapses
  • H. Huang, Y. Kabashima
  • Mathematics, Medicine
  • Physical review. E, Statistical, nonlinear, and soft matter physics
  • 2014
TLDR
This work analytically derive the Franz-Parisi potential for the binary perceptron problem by starting from an equilibrium solution of weights and exploring the weight space structure around it, which reveals the geometrical organization of theWeight space is composed of isolated solutions, rather than clusters of exponentially many close-by solutions. Expand
Learning from examples in large neural networks.
TLDR
Numerical results on training in layered neural networks indicate that the generalization error improves gradually in some cases, and sharply in others, and statistical mechanics is used to study generalization curves in large layered networks. Expand
Generalization Learning in a Perceptron with Binary Synapses
AbstractWe consider the generalization problem for a perceptron with binary synapses, implementing the Stochastic Belief-Propagation-Inspired (SBPI) learning algorithm which we proposed earlier, andExpand
A Max-Sum algorithm for training discrete neural networks
TLDR
The algorithm is a variant of the so-called Max-Sum algorithm that performs as well as BP on binary perceptron learning problems, and may be better suited to address the problem on fully-connected two-layer networks, since inherent symmetries in two layer networks are naturally broken using the MS approach. Expand
Entropy landscape of solutions in the binary perceptron problem
TLDR
The statistical picture of the solution space for a binary perceptron and the geometrical organization is elucidated by the entropy landscape from a reference configuration and of solution-pairs separated by a given Hamming distance in the solutions space. Expand
A Fast Learning Algorithm for Deep Belief Nets
TLDR
A fast, greedy algorithm is derived that can learn deep, directed belief networks one layer at a time, provided the top two layers form an undirected associative memory. Expand
Dynamics of learning for the binary perceptron problem
A polynomial learning algorithm for a perceptron with binary bonds and random patterns is investigated within dynamic mean field theory. A discontinuous freezing transition is found at a temperatureExpand
Storage capacity of memory networks with binary couplings
We study the number p of unbiased random patterns which can be stored in a neural network of N neurons used as an associative memory, in the case where the synaptic efficacies are constrained to takeExpand
Hippocampal Spine Head Sizes Are Highly Precise
TLDR
In an electron microscopic reconstruction of hippocampal neuropil, single axons making two or more synaptic contacts onto the same dendrites which would have shared histories of presynaptic and postsynaptic activity were found. Expand
...
1
2
3
...