Corpus ID: 211989398

Rethinking Parameter Counting in Deep Models: Effective Dimensionality Revisited

@article{Maddox2020RethinkingPC,
  title={Rethinking Parameter Counting in Deep Models: Effective Dimensionality Revisited},
  author={Wesley J. Maddox and Gregory M. Benton and Andrew Gordon Wilson},
  journal={ArXiv},
  year={2020},
  volume={abs/2003.02139}
}
  • Wesley J. Maddox, Gregory M. Benton, Andrew Gordon Wilson
  • Published in ArXiv 2020
  • Computer Science, Mathematics
  • Neural networks appear to have mysterious generalization properties when using parameter counting as a proxy for complexity. Indeed, neural networks often have many more parameters than there are data points, yet still provide good generalization performance. Moreover, when we measure generalization as a function of parameters, we see double descent behaviour, where the test error decreases, increases, and then again decreases. We show that many of these properties become understandable when… CONTINUE READING

    References

    Publications referenced by this paper.
    SHOWING 1-10 OF 63 REFERENCES

    Benign Overfitting in Linear Regression

    VIEW 5 EXCERPTS
    HIGHLY INFLUENTIAL

    Pyro: Deep Universal Probabilistic Programming

    VIEW 4 EXCERPTS
    HIGHLY INFLUENTIAL

    Subspace Inference for Bayesian Deep Learning

    VIEW 5 EXCERPTS

    Neural Networks and the Bias/Variance Dilemma

    VIEW 4 EXCERPTS
    HIGHLY INFLUENTIAL

    Reconciling modern machine-learning practice and the classical bias-variance trade-off.

    VIEW 7 EXCERPTS
    HIGHLY INFLUENTIAL

    Surprises in High-Dimensional Ridgeless

    • T. Hastie, A. Montanari, S. Rosset, R. J. Tibshirani
    • Least Squares Interpolation
    • 2019
    VIEW 4 EXCERPTS
    HIGHLY INFLUENTIAL

    Two models of double descent for weak features

    VIEW 6 EXCERPTS
    HIGHLY INFLUENTIAL