• Corpus ID: 15070342

Venn-Abers Predictors

@inproceedings{Vovk2014VennAbersP,
  title={Venn-Abers Predictors},
  author={Vladimir Vovk and Ivan Petej},
  booktitle={UAI},
  year={2014}
}
This paper continues study, both theoretical and empirical, of the method of Venn prediction, concentrating on binary prediction problems. Venn predictors produce probability-type predictions for the labels of test objects which are guaranteed to be well calibrated under the standard assumption that the observations are generated independently from the same distribution. We give a simple formalization and proof of this property. We also introduce Venn-Abers predictors, a new class of Venn… 
View PDF on arXiv
34 Citations
Highly Influential Citations
2
Background Citations
10
Methods Citations
15

Tables from this paper

34 Citations

Inductive Venn-Abers predictive distribution
TLDR
This paper shows how Venn Predictors can be applied on top of any regression method to obtain calibrated predictive distributions, without requiring assumptions beyond i.i.d. of calibration and test sets.
Calibration of Natural Language Understanding Models with Venn--ABERS Predictors
TLDR
This paper proposes to build several inductive Venn–ABERS predictors (IVAP), which are guaranteed to be well calibrated under minimal assumptions, based on a selection of pre-trained transformers, and shows that they are capable of producing well-calibrated probabilistic predictions that are uniformly spread over the [0,1] interval.
Large-scale probabilistic prediction with and without validity guarantees
This paper studies theoretically and empirically a method of turning machinelearning algorithms into probabilistic predictors that automatically enjoys a property of validity (perfect calibration)…
Large-scale probabilistic predictors with and without guarantees of validity
This paper studies theoretically and empirically a method of turning machine-learning algorithms into probabilistic predictors that automatically enjoys a property of validity (perfect calibration)…
Efficient Venn predictors using random forests
TLDR
The empirical investigation, using 22 publicly available data sets, showed that all four versions of the Venn predictors were better calibrated than both the raw estimates from the random forest, and the standard techniques Platt scaling and isotonic regression.
Small and large scale probabilistic classifiers with guarantees of validity
TLDR
A new method of probabilistic prediction which is based on conformal prediction a machine learning method for generating prediction sets that are guaranteed to have a specified coverage probability and a new class of Venn–Abers predictors, which are based on the idea of isotonic regression are introduced.
Multi-class probabilistic classification using inductive and cross Venn-Abers predictors
TLDR
This work presents a new approach to multi-class probability estimation by turning IVAPs and CVAPs into multiclass probabilistic predictors, which are experimentally more accurate than both uncalibrated predictors and existing calibration methods.
Are Traditional Neural Networks Well-Calibrated?
TLDR
It is demonstrated, using 25 publicly available two-class data sets, that both single multilayers perceptrons and ensembles of multilayer perceptrons in fact often are poorly calibrated and that it can be significantly improved by using either Platt scaling or Venn-Abers predictors.
Distribution-free binary classification: prediction sets, confidence intervals and calibration
TLDR
A 'tripod' of theorems is established that connects three notions of uncertainty quantification---calibration, confidence intervals and prediction sets---for binary classification in the distribution-free setting, that is without making any distributional assumptions on the data.
Is distribution-free inference possible for binary regression?
For a regression problem with a binary label response, we examine the problem of constructing confidence intervals for the label probability conditional on the features. In a setting where we do not…
...
...

References

SHOWING 1-10 OF 21 REFERENCES
Reliable Probability Estimates Based on Support Vector Machines for Large Multiclass Datasets
TLDR
This work develops an IVP with a taxonomy derived from a multiclass Support Vector Machine (SVM) and compares the method with other probabilistic methods for SVMs, namely Platt’s method, SVM Binning, and SVM with Isotonic Regression.
Self-calibrating Probability Forecasting
TLDR
It turns out that, although no probability forecasting algorithm is automatically well-calibrated in the authors' sense, there exists a wide class of algorithms for "multiprobability forecasting" which satisfy this property; the algorithms in this class are called "Venn probability machines".
Obtaining Calibrated Probabilities from Boosting
TLDR
After calibration, boosted full decision trees predict better probabilities than other learning methods such as SVMs, neural nets, bagged decision trees, and KNNs, even after these methods are calibrated.
Verification of Probabilistic Predictions: A Brief Review
TLDR
An artificial example of comparative verification in which different measures appear to yield contradictory results is used to demonstrate the importance of, and need for, a careful analysis of the evaluation process.
Transforming classifier scores into accurate multiclass probability estimates
TLDR
This work shows how to obtain accurate probability estimates for multiclass problems by combining calibrated binary probability estimates, and proposes a new method for obtaining calibrated two-class probability estimates that can be applied to any classifier that produces a ranking of examples.
Smooth Isotonic Regression: A New Method to Calibrate Predictive Models
TLDR
A novel method that combines parametric and non-parametric approaches is introduced that calibrates probabilistic outputs smoothly and shows better generalization ability than its ancestors in simulated as well as real world biomedical data sets.
Testing Statistical Hypotheses
This classic textbook, now available from Springer, summarizes developments in the field of hypotheses testing. Optimality considerations continue to provide the organizing principle. However, they…
Statistical inference under order restrictions : the theory and application of isotonic regression
Abstract : ;Contents: Isotonic regression; Estimation under order restrictions; Testing the equality of ordered means--likelihood ratio tests in the normal case; Testing the equality of ordered…
The WEKA data mining software: an update
TLDR
This paper provides an introduction to the WEKA workbench, reviews the history of the project, and, in light of the recent 3.6 stable release, briefly discusses what has been added since the last stable version (Weka 3.4) released in 2003.
Algorithmic Learning in a Random World
TLDR
A selection of books about type systems in programming languages, information theory, and machine learning that takes the randomness of the world into account, and verification of real time systems.
...
...