# Online Multivalid Learning: Means, Moments, and Prediction Intervals

@inproceedings{Gupta2022OnlineML, title={Online Multivalid Learning: Means, Moments, and Prediction Intervals}, author={Varun Gupta and Christopher Jung and Georgy Noarov and Mallesh M. Pai and Aaron Roth}, booktitle={ITCS}, year={2022} }

We present a general, efficient technique for providing contextual predictions that are “multivalid” in various senses, against an online sequence of adversarially chosen examples (x, y). This means that the resulting estimates correctly predict various statistics of the labels y not just marginally – as averaged over the sequence of examples – but also conditionally on x ∈ G for any G belonging to an arbitrary intersecting collection of groups G. We provide three instantiations of this…

## Figures from this paper

## 15 Citations

Practical Adversarial Multivalid Conformal Prediction

- Computer ScienceArXiv
- 2022

A simple, generic conformal prediction method for sequential prediction that achieves target empirical coverage guarantees against adversarially chosen data and is computationally lightweight but does not require having a held-out validation set.

Private Prediction Sets

- Computer ScienceArXiv
- 2021

This work develops a method that takes any pre-trained predictive model and outputs differentially private prediction sets, and follows the general approach of split conformal prediction; it uses holdout data to calibrate the size of the prediction sets but preserves privacy by using a privatized quantile subroutine.

Multi-group Agnostic PAC Learnability

- Computer ScienceICML
- 2021

This work unify and extend previous positive and negative results from the multi-group fairness literature, which applied for specific loss functions, to study “multi-group agnostic PAC learnability”.

Low-Degree Multicalibration

- Computer ScienceCOLT
- 2022

This work presents compelling evidence that low-degree multicalibration represents a sweet spot, pairing computational and sample eﬃciency with strong fairness and accuracy guarantees.

Beyond Bernoulli: Generating Random Outcomes that cannot be Distinguished from Nature

- Computer ScienceALT
- 2022

An equivalence is found between Outcome Indistinguishability and Multicalibration that is more subtle than in the binary case and sheds light on the techniques employed by Jung et al. to obtain Moment MulticalIBration.

Online Multiobjective Minimax Optimization and Applications

- Computer ScienceArXiv
- 2021

We introduce a simple but general online learning framework, in which at every round, an adaptive adversary introduces a new game, consisting of an action space for the learner, an action space for…

Lexicographically Fair Learning: Algorithms and Generalization

- Computer ScienceFORC
- 2021

A notion of approximate lexifairness is given that avoids this issue, and oracle-efficient algorithms for finding approximately lexif air solutions in a very general setting are derived, which are provably efficient even in the worst case.

A User Friendly Power Tool for Deriving Online Learning Algorithms (Invited Talk)

- Computer ScienceESA
- 2021

This talk overviews a simple and user friendly framework that can be used to derive online learning algorithms in a number of settings, and uses it to derive a new variant of Blackwell’s Approachability Theorem, which is term “Fast Polytope Approachability”.

A Unified Approach to Fair Online Learning via Blackwell Approachability

- Computer ScienceNeurIPS
- 2021

A setting and a general approach to fair online learning with stochastic sensitive and non-sensitive contexts and Adapting Blackwell’s approachability theory to handle the case of an unknown contexts’ distribution is provided.

Scaffolding Sets

- Computer ScienceArXiv
- 2021

This work begins the study of the construction of scaffolding sets, a small collection S of sets with the property that multi-calibration with respect to S ensures correctness, and not just calibration, of the predictor.

## References

SHOWING 1-10 OF 41 REFERENCES

Stronger calibration lower bounds via sidestepping

- Computer ScienceSTOC
- 2021

An Ω(T0.528) bound on the calibration error is proved, which is the first super-√T lower bound for this setting to the best of the authors' knowledge, and an abstraction of the prediction setting is proposed, termed the Sign-Preservation game, which may be of independent interest.

Moment Multicalibration for Uncertainty Estimation

- MathematicsCOLT
- 2021

We show how to achieve the notion of "multicalibration" from Hebert-Johnson et al. [2018] not just for means, but also for variances and other higher moments. Informally, it means that we can find…

Outcome indistinguishability

- Computer ScienceElectron. Colloquium Comput. Complex.
- 2020

This hardness result provides the first scientific grounds for the political argument that, when inspecting algorithmic risk prediction instruments, auditors should be granted oracle access to the algorithm, not simply historical predictions.

Multicalibration: Calibration for the (Computationally-Identifiable) Masses

- Computer ScienceICML
- 2018

We develop and study multicalibration as a new measure of fairness in machine learning that aims to mitigate inadvertent or malicious discrimination that is introduced at training time (even from…

Distribution-free binary classification: prediction sets, confidence intervals and calibration

- Mathematics, Computer ScienceNeurIPS
- 2020

A 'tripod' of theorems is established that connects three notions of uncertainty quantification---calibration, confidence intervals and prediction sets---for binary classification in the distribution-free setting, that is without making any distributional assumptions on the data.

A tutorial on conformal prediction

- Computer ScienceJ. Mach. Learn. Res.
- 2008

This tutorial presents a self-contained account of the theory of conformal prediction and works through several numerical examples of how the model under which successive examples are sampled independently from the same distribution can be applied to any method for producing ŷ.

The limits of distribution-free conditional predictive inference

- Computer Science, MathematicsInformation and Inference: A Journal of the IMA
- 2020

This work aims to explore the space in between exact conditional inference guarantees and what types of relaxations of the conditional coverage property would alleviate some of the practical concerns with marginal coverage guarantees while still being possible to achieve in a distribution-free setting.

Multiaccuracy: Black-Box Post-Processing for Fairness in Classification

- Computer ScienceAIES
- 2019

It is proved that MULTIACCURACY-BOOST converges efficiently and it is shown that if the initial model is accurate on an identifiable subgroup, then the post-processed model will be also.

An Empirical Study of Rich Subgroup Fairness for Machine Learning

- Computer ScienceFAT
- 2019

In general, the Kearns et al. algorithm converges quickly, large gains in fairness can be obtained with mild costs to accuracy, and that optimizing accuracy subject only to marginal fairness leads to classifiers with substantial subgroup unfairness.

Online Multiobjective Minimax Optimization and Applications

- Computer ScienceArXiv
- 2021

We introduce a simple but general online learning framework, in which at every round, an adaptive adversary introduces a new game, consisting of an action space for the learner, an action space for…