# Unreasonable effectiveness of learning neural networks: From accessible states and robust ensembles to basic algorithmic schemes

@article{Baldassi2016UnreasonableEO, title={Unreasonable effectiveness of learning neural networks: From accessible states and robust ensembles to basic algorithmic schemes}, author={Carlo Baldassi and C. Borgs and J. Chayes and A. Ingrosso and Carlo Lucibello and Luca Saglietti and R. Zecchina}, journal={Proceedings of the National Academy of Sciences}, year={2016}, volume={113}, pages={E7655 - E7662} }

Significance Artificial neural networks are some of the most widely used tools in data science. Learning is, in principle, a hard problem in these systems, but in practice heuristic algorithms often find solutions with good generalization properties. We propose an explanation of this good performance in terms of a nonequilibrium statistical physics framework: We show that there are regions of the optimization landscape that are both robust and accessible and that their existence is crucial to… CONTINUE READING

#### Supplemental Code

GITHUB REPO

Replicated Stochastic Gradient Descent on Commitee machines with binary weights

77 Citations

Shaping the learning landscape in neural networks around wide flat minima

- Mathematics, Computer Science
- 2019

6

Energy–entropy competition and the effectiveness of stochastic gradient descent in machine learning

- Chemistry, Computer Science
- 2018

18

Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data

- Computer Science, Mathematics
- 2017

238

Reinforced backpropagation improves test performance of deep networks: a toy-model study

- Computer Science
- 2017

How neural networks find generalizable solutions: Self-tuned annealing in deep learning

- Computer Science, Mathematics
- 2020

#### References

##### Publications referenced by this paper.

SHOWING 1-10 OF 56 REFERENCES

Subdominant Dense Clusters Allow for Simple Learning and High Computational Performance in Neural Networks with Discrete Synapses.

- Physics, Medicine
- 2015

68

Origin of the computational hardness for learning with binary synapses

- Computer Science, Physics
- 2014

27

Local entropy as a measure for sampling solutions in constraint satisfaction problems

- Mathematics, Physics
- 2016

28