Online Multivalid Learning: Means, Moments, and Prediction Intervals

@inproceedings{Gupta2022OnlineML,
  title={Online Multivalid Learning: Means, Moments, and Prediction Intervals},
  author={Varun Gupta and Christopher Jung and Georgy Noarov and Mallesh M. Pai and Aaron Roth},
  booktitle={ITCS},
  year={2022}
}
We present a general, efficient technique for providing contextual predictions that are “multivalid” in various senses, against an online sequence of adversarially chosen examples (x, y). This means that the resulting estimates correctly predict various statistics of the labels y not just marginally – as averaged over the sequence of examples – but also conditionally on x ∈ G for any G belonging to an arbitrary intersecting collection of groups G. We provide three instantiations of this… 

Figures from this paper

Practical Adversarial Multivalid Conformal Prediction
TLDR
A simple, generic conformal prediction method for sequential prediction that achieves target empirical coverage guarantees against adversarially chosen data and is computationally lightweight but does not require having a held-out validation set.
Private Prediction Sets
TLDR
This work develops a method that takes any pre-trained predictive model and outputs differentially private prediction sets, and follows the general approach of split conformal prediction; it uses holdout data to calibrate the size of the prediction sets but preserves privacy by using a privatized quantile subroutine.
Multi-group Agnostic PAC Learnability
TLDR
This work unify and extend previous positive and negative results from the multi-group fairness literature, which applied for specific loss functions, to study “multi-group agnostic PAC learnability”.
Low-Degree Multicalibration
TLDR
This work presents compelling evidence that low-degree multicalibration represents a sweet spot, pairing computational and sample efficiency with strong fairness and accuracy guarantees.
Beyond Bernoulli: Generating Random Outcomes that cannot be Distinguished from Nature
TLDR
An equivalence is found between Outcome Indistinguishability and Multicalibration that is more subtle than in the binary case and sheds light on the techniques employed by Jung et al. to obtain Moment MulticalIBration.
Online Multiobjective Minimax Optimization and Applications
We introduce a simple but general online learning framework, in which at every round, an adaptive adversary introduces a new game, consisting of an action space for the learner, an action space for
Lexicographically Fair Learning: Algorithms and Generalization
TLDR
A notion of approximate lexifairness is given that avoids this issue, and oracle-efficient algorithms for finding approximately lexif air solutions in a very general setting are derived, which are provably efficient even in the worst case.
A User Friendly Power Tool for Deriving Online Learning Algorithms (Invited Talk)
TLDR
This talk overviews a simple and user friendly framework that can be used to derive online learning algorithms in a number of settings, and uses it to derive a new variant of Blackwell’s Approachability Theorem, which is term “Fast Polytope Approachability”.
A Unified Approach to Fair Online Learning via Blackwell Approachability
TLDR
A setting and a general approach to fair online learning with stochastic sensitive and non-sensitive contexts and Adapting Blackwell’s approachability theory to handle the case of an unknown contexts’ distribution is provided.
Scaffolding Sets
TLDR
This work begins the study of the construction of scaffolding sets, a small collection S of sets with the property that multi-calibration with respect to S ensures correctness, and not just calibration, of the predictor.
...
...

References

SHOWING 1-10 OF 41 REFERENCES
Stronger calibration lower bounds via sidestepping
TLDR
An Ω(T0.528) bound on the calibration error is proved, which is the first super-√T lower bound for this setting to the best of the authors' knowledge, and an abstraction of the prediction setting is proposed, termed the Sign-Preservation game, which may be of independent interest.
Moment Multicalibration for Uncertainty Estimation
We show how to achieve the notion of "multicalibration" from Hebert-Johnson et al. [2018] not just for means, but also for variances and other higher moments. Informally, it means that we can find
Outcome indistinguishability
TLDR
This hardness result provides the first scientific grounds for the political argument that, when inspecting algorithmic risk prediction instruments, auditors should be granted oracle access to the algorithm, not simply historical predictions.
Multicalibration: Calibration for the (Computationally-Identifiable) Masses
We develop and study multicalibration as a new measure of fairness in machine learning that aims to mitigate inadvertent or malicious discrimination that is introduced at training time (even from
Distribution-free binary classification: prediction sets, confidence intervals and calibration
TLDR
A 'tripod' of theorems is established that connects three notions of uncertainty quantification---calibration, confidence intervals and prediction sets---for binary classification in the distribution-free setting, that is without making any distributional assumptions on the data.
A tutorial on conformal prediction
TLDR
This tutorial presents a self-contained account of the theory of conformal prediction and works through several numerical examples of how the model under which successive examples are sampled independently from the same distribution can be applied to any method for producing ŷ.
The limits of distribution-free conditional predictive inference
TLDR
This work aims to explore the space in between exact conditional inference guarantees and what types of relaxations of the conditional coverage property would alleviate some of the practical concerns with marginal coverage guarantees while still being possible to achieve in a distribution-free setting.
Multiaccuracy: Black-Box Post-Processing for Fairness in Classification
TLDR
It is proved that MULTIACCURACY-BOOST converges efficiently and it is shown that if the initial model is accurate on an identifiable subgroup, then the post-processed model will be also.
An Empirical Study of Rich Subgroup Fairness for Machine Learning
TLDR
In general, the Kearns et al. algorithm converges quickly, large gains in fairness can be obtained with mild costs to accuracy, and that optimizing accuracy subject only to marginal fairness leads to classifiers with substantial subgroup unfairness.
Online Multiobjective Minimax Optimization and Applications
We introduce a simple but general online learning framework, in which at every round, an adaptive adversary introduces a new game, consisting of an action space for the learner, an action space for
...
...