Corpus ID: 81981745

On-line learning dynamics of ReLU neural networks using statistical physics techniques

@article{Straat2019OnlineLD,
  title={On-line learning dynamics of ReLU neural networks using statistical physics techniques},
  author={M. Straat and Michael Biehl},
  journal={ArXiv},
  year={2019},
  volume={abs/1903.07378}
}
  • M. Straat, Michael Biehl
  • Published 2019
  • Computer Science, Physics, Mathematics
  • ArXiv
  • We introduce exact macroscopic on-line learning dynamics of two-layer neural networks with ReLU units in the form of a system of differential equations, using techniques borrowed from statistical physics. For the first experiments, numerical solutions reveal similar behavior compared to sigmoidal activation researched in earlier work. In these experiments the theoretical results show good correspondence with simulations. In ove-rrealizable and unrealizable learning scenarios, the learning… CONTINUE READING
    4 Citations
    Hidden Unit Specialization in Layered Neural Networks: ReLU vs. Sigmoidal Activation
    • 1
    • PDF
    Supervised Learning in the Presence of Concept Drift: A modelling framework

    References

    SHOWING 1-10 OF 11 REFERENCES
    Transient dynamics of on-line learning in two-layered neural networks
    • 36
    Learning by on-line gradient descent
    • 93
    Statistical Mechanics of On-Line Learning Under Concept Drift
    • 12
    • PDF
    Phase transitions in soft-committee machines
    • 16
    • PDF
    Dynamics and Generalization Ability of LVQ Algorithms
    • 105
    • PDF
    SELF-AVERAGING AND ON-LINE LEARNING
    • 46
    • PDF
    Deep Sparse Rectifier Neural Networks
    • 4,272
    • PDF
    Exact solution for on-line learning in multilayer neural networks.
    • Y Saad, Solla
    • Physics, Medicine
    • Physical review letters
    • 1995
    • 110