• Corpus ID: 235266057

Adaptive Conformal Inference Under Distribution Shift

@inproceedings{Gibbs2021AdaptiveCI,
  title={Adaptive Conformal Inference Under Distribution Shift},
  author={Isaac Gibbs and Emmanuel J. Cand{\`e}s},
  booktitle={NeurIPS},
  year={2021}
}
We develop methods for forming prediction sets in an online setting where the data generating distribution is allowed to vary over time in an unknown fashion. Our framework builds on ideas from conformal inference to provide a general wrapper that can be combined with any black box method that produces point predictions of the unseen label or estimated quantiles of its distribution. While previous conformal inference methods rely on the assumption that the data points are exchangeable, our… 

Figures from this paper

Robust Flow-based Conformal Inference (FCI) with Statistical Guarantee
TLDR
This paper develops a series of conformal inference methods, including building predictive sets and inferring outliers for complex and high-dimensional data, and leverages ideas from adversarial learning to transfer the input data to a random vector with known distributions, which enable them to construct a non-conformity score for uncertainty quantification.
Conformal prediction beyond exchangeability
TLDR
These algorithms are provably robust, with substantially less loss of coverage when exchangeability is violated due to distribution drift or other challenging features of real data, while also achieving the same coverage guarantees as existing conformal prediction methods if the data points are in fact exchangeable.
Doubly Robust Calibration of Prediction Sets under Covariate Shift
Conformal prediction has received tremendous attention in recent years and has offered new solu-tions to problems in missing data and causal inference; yet these advances have not leveraged modern
Conformalized Online Learning: Online Calibration Without a Holdout Set
We develop a framework for constructing uncertainty sets with a valid coverage guarantee in an online setting, in which the underlying data distribution can drastically—and even adversarially—shift
A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification
TLDR
This hands-on introduction is aimed at a reader interested in the practical implementation of distribution-free UQ who is not necessarily a statistician, allowing them to rigorously quantify algorithmic uncertainty with one self-contained document.
Conformal prediction for dynamic time-series
TLDR
A computationally efficient algorithm called EnbPI is introduced that wraps around ensemble predictors, which is closely related to conformal prediction (CP) but does not require data exchangeability.
PAC Prediction Sets Under Covariate Shift
TLDR
This work proposes a novel approach that addresses this challenge by constructing probably approximately correct (PAC) prediction sets in the presence of covariate shift by constructing prediction sets with the smallest average normalized size among approaches that always satisfy the PAC constraint.
Split Conformal Prediction for Dependent Data
TLDR
It is shown that coverage guarantees from split CP can be extended to dependent processes, such as the class of stationary β -mixing processes, by adding a small coverage penalty, and that the empirical coverage bounds for some β - Mixing processes match the order of the bounds under exchangeability.
A general framework for multi-step ahead adaptive conformal heteroscedastic time series forecasting
TLDR
A new method is proposed called AEnbMIMOCQR (Adaptive ensemble batch multi-input multi-output conformalized quantile regression), which produces asymptotic valid PIs and is appropriate for heteroscedastic time series forecasting.
Practical Adversarial Multivalid Conformal Prediction
TLDR
A simple, generic conformal prediction method for sequential prediction that achieves target empirical coverage guarantees against adversarially chosen data and is computationally lightweight but does not require having a held-out validation set.
...
...

References

SHOWING 1-10 OF 38 REFERENCES
Distribution-Free Predictive Inference for Regression
TLDR
A general framework for distribution-free predictive inference in regression, using conformal inference, which allows for the construction of a prediction band for the response variable using any estimator of the regression function, and a model-free notion of variable importance, called leave-one-covariate-out or LOCO inference.
Conformal Prediction Under Covariate Shift
TLDR
It is shown that a weighted version of conformal prediction can be used to compute distribution-free prediction intervals for problems in which the test and training covariate distributions differ, but the likelihood ratio between these two distributions is known.
Robust Validation: Confident Predictions Even When Distributions Shift
TLDR
A method is presented that produces prediction sets (almost exactly) giving the right coverage level for any test distribution in an $f$-divergence ball around the training population, and achieves (nearly) valid coverage in finite samples.
Classification with Valid and Adaptive Coverage
TLDR
A novel conformity score is developed, which is explicitly demonstrate to be powerful and intuitive for classification problems, but whose underlying principle is potentially far more general.
Adaptive, Distribution-Free Prediction Intervals for Deep Networks
TLDR
A neural network is proposed that outputs three values instead of a single point estimate and optimizes a loss function motivated by the standard quantile regression loss and provides two prediction interval methods with finite sample coverage guarantees solely under the assumption that the observations are independent and identically distributed.
Conformalized Quantile Regression
TLDR
This paper proposes a new method that is fully adaptive to heteroscedasticity, which combines conformal prediction with classical quantile regression, inheriting the advantages of both.
A tutorial on conformal prediction
TLDR
This tutorial presents a self-contained account of the theory of conformal prediction and works through several numerical examples of how the model under which successive examples are sampled independently from the same distribution can be applied to any method for producing ŷ.
The limits of distribution-free conditional predictive inference
TLDR
This work aims to explore the space in between exact conditional inference guarantees and what types of relaxations of the conditional coverage property would alleviate some of the practical concerns with marginal coverage guarantees while still being possible to achieve in a distribution-free setting.
Conformal inference of counterfactuals and individual treatment effects
  • Lihua Lei, E. Candès
  • Mathematics
    Journal of the Royal Statistical Society: Series B (Statistical Methodology)
  • 2021
TLDR
This work proposes a conformal inference-based approach that can produce reliable interval estimates for counterfactuals and individual treatment effects under the potential outcome framework and achieves the desired coverage with reasonably short intervals.
Knowing what You Know: valid and validated confidence sets in multiclass and multilabel prediction
TLDR
To address the potential challenge of exponentially large confidence sets in multilabel prediction, this work builds tree-structured classifiers that efficiently account for interactions between labels that can be bolted on top of any classification model to guarantee its validity.
...
...