• Corpus ID: 235390419

Understanding the Under-Coverage Bias in Uncertainty Estimation

  title={Understanding the Under-Coverage Bias in Uncertainty Estimation},
  author={Yu Bai and Song Mei and Huan Wang and Caiming Xiong},
Estimating the data uncertainty in regression tasks is often done by learning a quantile function or a prediction interval of the true label conditioned on the input. It is frequently observed that quantile regression—a vanilla algorithm for learning quantiles with asymptotic guarantees—tends to under-cover than the desired coverage level in reality. While various fixes have been proposed, a more fundamental understanding of why this under-coverage bias happens in the first place remains… 

Figures and Tables from this paper

Efficient and Differentiable Conformal Prediction with General Function Classes
This meta-algorithm generalizes existing conformal prediction algorithms, and it achieves approximate valid population coverage and near-optimal efficiency within class, whenever the function class in the conformalization step is low-capacity in a certain sense.
Theoretical characterization of uncertainty in high-dimensional linear classification
This manuscript characterise uncertainty for learning from limited number of samples of high-dimensional Gaussian input data and labels generated by the probit model, and provides a closed-form formula for the joint statistics between the logistic classifier, the uncertainty of the statistically optimal Bayesian classifier and the ground-truth probit uncertainty.
Quantifying Epistemic Uncertainty in Deep Learning
A theoretical framework to dissect the uncertainty, especially the epistemic component, in deep learning into procedural variability and data variability is provided, which is the first such attempt in the literature to the authors' best knowledge.


Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift
A large-scale benchmark of existing state-of-the-art methods on classification problems and the effect of dataset shift on accuracy and calibration is presented, finding that traditional post-hoc calibration does indeed fall short, as do several other previous methods.
Robust Validation: Confident Predictions Even When Distributions Shift
A method is presented that produces prediction sets (almost exactly) giving the right coverage level for any test distribution in an $f$-divergence ball around the training population, and achieves (nearly) valid coverage in finite samples.
Nonparametric Quantile Estimation
A nonparametric version of a quantile estimator is presented, which can be obtained by solving a simple quadratic programming problem and uniform convergence statements and bounds on the quantile property of the estimator are provided.
Adaptive, Distribution-Free Prediction Intervals for Deep Networks
A neural network is proposed that outputs three values instead of a single point estimate and optimizes a loss function motivated by the standard quantile regression loss and provides two prediction interval methods with finite sample coverage guarantees solely under the assumption that the observations are independent and identically distributed.
Distribution-Free, Risk-Controlling Prediction Sets
This work shows how to generate set-valued predictions from a black-box predictor that controls the expected loss on future test points at a user-specified level, and provides explicit finite-sample guarantees for any dataset by using a holdout set to calibrate the size of the prediction sets.
Estimating conditional quantiles with the help of the pinball loss
This work establishes inequalities that describe how close approximate pinball risk minimizers are to the corresponding conditional quantile, and uses them to establish an oracle inequality for support vector machines that use the pinball loss.
Knowing what You Know: valid and validated confidence sets in multiclass and multilabel prediction
To address the potential challenge of exponentially large confidence sets in multilabel prediction, this work builds tree-structured classifiers that efficiently account for interactions between labels that can be bolted on top of any classification model to guarantee its validity.
Distribution-Free Predictive Inference for Regression
A general framework for distribution-free predictive inference in regression, using conformal inference, which allows for the construction of a prediction band for the response variable using any estimator of the regression function, and a model-free notion of variable importance, called leave-one-covariate-out or LOCO inference.
The limits of distribution-free conditional predictive inference
This work aims to explore the space in between exact conditional inference guarantees and what types of relaxations of the conditional coverage property would alleviate some of the practical concerns with marginal coverage guarantees while still being possible to achieve in a distribution-free setting.
Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles
This work proposes an alternative to Bayesian NNs that is simple to implement, readily parallelizable, requires very little hyperparameter tuning, and yields high quality predictive uncertainty estimates.