Over-parameterization: A Necessary Condition for Models that Extrapolate

@article{Yousefzadeh2022OverparameterizationAN,
  title={Over-parameterization: A Necessary Condition for Models that Extrapolate},
  author={Roozbeh Yousefzadeh},
  journal={ArXiv},
  year={2022},
  volume={abs/2203.10447}
}
In this work, we study over-parameterization as a necessary condition for having the ability for the models to extrapolate outside the convex hull of training set. We specifically, consider classification models, e.g., image classification and other applications of deep learning. Such models are classification functions that partition their domain and assign a class to each partition [10]. Partitions are defined by decision boundaries and so is the classification model/function. Convex hull of… 

References

SHOWING 1-10 OF 17 REFERENCES

Deep Learning Generalization and the Convex Hull of Training Sets

This work investigates the decision boundaries of a neural network, with various degrees of parameters, inside and outside the convex hull of its training set, and uses a polynomial decision boundary to study the necessity of over-parameterization and the influence of training regime in shaping its extensions outside the conveyed hull of training set.

Decision boundaries and convex hulls in the feature space that deep learning functions learn from images

The partitioning of the domain in feature space is studied, regions guaranteed to have certain classifications are identified, and its implications for the pixel space are investigated, providing insights about adversarial vulnerabilities, image morphing, extrapolation, ambiguity, and the mathematical understanding of image classi fication models.

A Farewell to the Bias-Variance Tradeoff? An Overview of the Theory of Overparameterized Machine Learning

This paper provides a succinct overview of this emerging theory of overparameterized ML (henceforth abbreviated as TOPML) that explains these recent findings through a statistical signal processing perspective and emphasizes the unique aspects that define the TOPML research area as a subfield of modern ML theory.

Learning in High Dimension Always Amounts to Extrapolation

It is demonstrated that on any high-dimensional (>100) dataset, interpolation almost surely never happens and the validity of the current interpolation/extrapolation definition as an indicator of generalization performances is challenged.

Extrapolation Frameworks in Cognitive Psychology Suitable for Study of Image Classification Models

The proposed extrapolation framework can provide novel answers to open research problems about deep learning including their over-parameterization, their training regime, out-of-distribution detection, etc.

Empirical Study of the Topology and Geometry of Deep Networks

It is shown that state-of-the-art deep nets learn connected classification regions, and that the decision boundary in the vicinity of datapoints is flat along most directions, and an essential connection is drawn between two seemingly unrelated properties of deep networks: their sensitivity to additive perturbations of the inputs, and the curvature of their decision boundary.

Mad Max: Affine Spline Insights Into Deep Learning

A rigorous bridge between deep networks (DNs) and approximation theory via spline functions and operators is built and a simple penalty term is proposed that can be added to the cost function of any DN learning algorithm to force the templates to be orthogonal with each other.

Understanding deep learning (still) requires rethinking generalization

These experiments establish that state-of-the-art convolutional networks for image classification trained with stochastic gradient methods easily fit a random labeling of the training data, and corroborate these experimental findings with a theoretical construction showing that simple depth two neural networks already have perfect finite sample expressivity.

Understanding deep learning requires rethinking generalization

These experiments establish that state-of-the-art convolutional networks for image classification trained with stochastic gradient methods easily fit a random labeling of the training data, and confirm that simple depth two neural networks already have perfect finite sample expressivity.

Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolation

Just as a physical prism separates colours mixed within a ray of light, the figurative prism of interpolation helps to disentangle generalization and optimization properties within the complex picture of modern machine learning.