Just Interpolate: Kernel "Ridgeless" Regression Can Generalize

@article{Liang2018JustIK,
  title={Just Interpolate: Kernel "Ridgeless" Regression Can Generalize},
  author={Tengyuan Liang and A. Rakhlin},
  journal={ArXiv},
  year={2018},
  volume={abs/1808.00387}
}
  • Tengyuan Liang, A. Rakhlin
  • Published 2018
  • Mathematics, Computer Science
  • ArXiv
  • In the absence of explicit regularization, Kernel "Ridgeless" Regression with nonlinear kernels has the potential to fit the training data perfectly. It has been observed empirically, however, that such interpolated solutions can still generalize well on test data. We isolate a phenomenon of implicit regularization for minimum-norm interpolated solutions which is due to a combination of high dimensionality of the input data, curvature of the kernel function, and favorable geometric properties… CONTINUE READING
    118 Citations

    Figures, Tables, and Topics from this paper.

    Harmless interpolation of noisy data in regression
    • 47
    • PDF
    Asymptotics of Ridge(less) Regression under General Source Condition
    • 2
    • PDF
    The Neural Tangent Kernel in High Dimensions: Triple Descent and a Multi-Scale Theory of Generalization
    • 4
    • PDF
    Benign overfitting in ridge regression
    • 1
    • PDF
    Consistency of Interpolation with Laplace Kernels is a High-Dimensional Phenomenon
    • 29
    • PDF
    Interpolation and Learning with Scale Dependent Kernels
    • 1
    • Highly Influenced
    • PDF
    On the Inductive Bias of Neural Tangent Kernels
    • 53
    • PDF
    On the Multiple Descent of Minimum-Norm Interpolants and Restricted Lower Isometry of Kernels
    • 24
    • PDF
    Harmless Interpolation of Noisy Data in Regression
    • 1

    References

    SHOWING 1-10 OF 36 REFERENCES
    Overfitting or perfect fitting? Risk bounds for classification and regression rules that interpolate
    • 104
    • Highly Influential
    • PDF
    To understand deep learning we need to understand kernel learning
    • 161
    • Highly Influential
    • PDF
    Approximation beats concentration? An approximation view on inference with smooth radial kernels
    • M. Belkin
    • Computer Science, Mathematics
    • COLT
    • 2018
    • 25
    • Highly Influential
    • PDF
    Optimal Rates for the Regularized Least-Squares Algorithm
    • 432
    • Highly Influential
    • PDF
    On Early Stopping in Gradient Descent Learning
    • 441
    • PDF
    Model Selection for Regularized Least-Squares Algorithm in Learning Theory
    • 162
    • PDF
    Generalized cross-validation as a method for choosing a good ridge parameter
    • 2,350
    • PDF
    The spectrum of kernel random matrices
    • 122
    • Highly Influential
    • PDF
    Kernel Ridge Regression
    • V. Vovk
    • Mathematics, Computer Science
    • Empirical Inference
    • 2013
    • 81
    • PDF