• Corpus ID: 235358945

Evaluating State-of-the-Art Classification Models Against Bayes Optimality

@inproceedings{Theisen2021EvaluatingSC,
  title={Evaluating State-of-the-Art Classification Models Against Bayes Optimality},
  author={Ryan Theisen and Huan Wang and Lav R. Varshney and Caiming Xiong and Richard Socher},
  booktitle={NeurIPS},
  year={2021}
}
Evaluating the inherent difficulty of a given data-driven classification problem is important for establishing absolute benchmarks and evaluating progress in the field. To this end, a natural quantity to consider is the Bayes error, which measures the optimal classification error theoretically achievable for a given data distribution. While generally an intractable quantity, we show that we can compute the exact Bayes error of generative models learned using normalizing flows. Our technique… 

Figures and Tables from this paper

Is the Performance of My Deep Network Too Good to Be True? A Direct Approach to Estimating the Bayes Error in Binary Classification

TLDR
A simple and direct Bayes error estimator, where the mean of the labels that show uncertainty of the classes is taken, which has no hyperparameters and gives a more accurate estimate of the Bayeserror than classi-er-based baselines.

SpinalNet: Deep Neural Network with Gradual Input

TLDR
The human somatosensory system is studied and the SpinalNet is proposed to achieve higher accuracy with less computational resources and the vanishing gradient problem does not exist.

References

SHOWING 1-10 OF 40 REFERENCES

Learning to Benchmark: Determining Best Achievable Misclassification Error from Training Data

TLDR
This work proposes a benchmark learner based on an ensemble of $\epsilon$-ball estimators and Chebyshev approximation that achieves an optimal (parametric) mean squared error (MSE) rate of $O(N^{-1})$, where $N$ is the number of samples.

Meta learning of bounds on the Bayes classifier error

TLDR
This work estimates multiple bounds on the Bayes error using an estimator that applies meta learning to slowly converging plug-in estimators to obtain the parametric convergence rate.

Generalized Bhattacharyya and Chernoff upper bounds on Bayes error using quasi-arithmetic means

  • F. Nielsen
  • Computer Science
    Pattern Recognit. Lett.
  • 2014

Understanding the Limitations of Conditional Generative Models

TLDR
The theoretical result reveals that it is impossible to guarantee detectability of adversarially-perturbed inputs even for near-optimal generative classifiers, and the results indicate that likelihood-based conditional generative models may are surprisingly ineffective for robust classification.

Sharpness-Aware Minimization for Efficiently Improving Generalization

TLDR
This work introduces a novel, effective procedure for simultaneously minimizing loss value and loss sharpness, Sharpness-Aware Minimization (SAM), which improves model generalization across a variety of benchmark datasets and models, yielding novel state-of-the-art performance for several.

Multivariate f-divergence Estimation With Confidence

TLDR
This work establishes the asymptotic normality of a recently proposed ensemble estimator of f-divergence between two distributions from a finite number of samples, which has MSE convergence rate of O (1/T), is simple to implement, and performs well in high dimensions.

Ensemble Estimation of Information Divergence †

TLDR
An empirical estimator of Rényi-α divergence is proposed that greatly outperforms the standard kernel density plug-in estimator in terms of mean squared error, especially in high dimensions and is shown to be robust to the choice of tuning parameters.

DIME: An Information-Theoretic Difficulty Measure for AI Datasets

TLDR
This work proposes DIME, an information-theoretic DIfficulty MEasure for datasets, based on Fano’s inequality and a neural network estimation of the conditional entropy of the sample-label distribution, which can be decomposed into components attributable to the data distribution and the number of samples.

Normalizing Flows for Probabilistic Modeling and Inference

TLDR
This review places special emphasis on the fundamental principles of flow design, and discusses foundational topics such as expressive power and computational trade-offs, and summarizes the use of flows for tasks such as generative modeling, approximate inference, and supervised learning.

Nonparametric Divergence Estimation with Applications to Machine Learning on Distributions

TLDR
Estimation algorithms are presented, how to apply them for machine learning tasks on distributions are described, and empirical results on synthetic data, real word images, and astronomical data sets are shown.