The Scope of Multicalibration: Characterizing Multicalibration via Property Elicitation

@article{Noarov2023TheSO,
  title={The Scope of Multicalibration: Characterizing Multicalibration via Property Elicitation},
  author={Georgy Noarov and Aaron Roth},
  journal={ArXiv},
  year={2023},
  volume={abs/2302.08507}
}
We make a connection between multicalibration and property elicitation and show that (under mild technical conditions) it is possible to produce a multicalibrated predictor for a continuous scalar distributional property $\Gamma$ if and only if $\Gamma$ is elicitable. On the negative side, we show that for non-elicitable continuous properties there exist simple data distributions on which even the true distributional predictor is not calibrated. On the positive side, for elicitable $\Gamma$, we… 
[PDF] Semantic Reader

32 References

Online Multivalid Learning: Means, Moments, and Prediction Intervals

This paper presents a general, efficient technique for providing contextual predictions that are "multivalid" in various senses, against an online sequence of adversarially chosen examples, and defines a new notion of prediction interval multivalidity, and gives an algorithm for finding prediction intervals which satisfy it.

Low-Degree Multicalibration

This work defines and initiates the study of Low-Degree Multicalibration, a hierarchy of increasingly-powerful multi-group fairness notions that spans multiaccuracy and the original formulation of multicalIBration at the extremes, and shows that low-degree multicalsibration can be significantly more efficient than full multICALibration.

Elicitation complexity of statistical properties

This work lays the foundation for a general theory of elicitation complexity, including several basic results about how elicit complexity behaves, and the complexity of standard properties of interest.

Online Minimax Multiobjective Optimization: Multicalibeating and Other Applications

A simple but general online learning framework in which a learner plays against an adversary in a vector-valued game that changes every round, and gives a simple algorithm that can compete with the setting in which the adversary must announce their action first, with optimally diminishing regret.

Moment Multicalibration for Uncertainty Estimation

We show how to achieve the notion of "multicalibration" from Hebert-Johnson et al. [2018] not just for means, but also for variances and other higher moments. Informally, it means that we can find…

Eliciting properties of probability distributions

We investigate the problem of truthfully eliciting an expert's assessment of a property of a probability distribution, where a property is any real-valued function of the distribution such as mean or…

Multicalibration: Calibration for the (Computationally-Identifiable) Masses

We develop and study multicalibration as a new measure of fairness in machine learning that aims to mitigate inadvertent or malicious discrimination that is introduced at training time (even from…

Convex Elicitation of Continuous Properties

In this paper, in a finite-outcome setting, it is shown that in fact every elicitable real-valued property can be elicited by a convex loss function.

Batch Multivalid Conformal Prediction

Two fast distribution-free conformal prediction algorithms for obtaining multivalid coverage on exchangeable data in the batch setting are developed: BatchGCP and BatchMVP, which give the full guarantees ofMultivalid conformal Prediction: prediction sets that are valid conditionally both on group membership and non-conformity threshold.

Outcome indistinguishability

This hardness result provides the first scientific grounds for the political argument that, when inspecting algorithmic risk prediction instruments, auditors should be granted oracle access to the algorithm, not simply historical predictions.