On Calibration of Ensemble-Based Credal Predictors

  title={On Calibration of Ensemble-Based Credal Predictors},
  author={Thomas Mortier and Viktor Bengs and Eyke H{\"u}llermeier and Stijn Luca and Willem Waegeman},
In recent years, several classification methods that intend to quantify epistemic uncertainty have been proposed, either by producing predictions in the form of second-order distributions or sets of probability distributions. In this work, we focus on the latter, also called credal predictors, and address the question of how to evaluate them: What does it mean that a credal predictor represents epistemic uncertainty in a faithful manner? To answer this question, we refer to the notion of… 

Figures and Tables from this paper


Credal ensembles of classifiers
Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift
A large-scale benchmark of existing state-of-the-art methods on classification problems and the effect of dataset shift on accuracy and calibration is presented, finding that traditional post-hoc calibration does indeed fall short, as do several other previous methods.
On the Difficulty of Epistemic Uncertainty Quantification in Machine Learning: The Case of Direct Uncertainty Estimation through Loss Minimisation
It is shown that loss minimisation does not work for secondorder predictors: the loss functions proposed for inducing such predictors do not incentivise the learner to represent its epistemic uncertainty in a faithful way.
Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles
This work proposes an alternative to Bayesian NNs that is simple to implement, readily parallelizable, requires very little hyperparameter tuning, and yields high quality predictive uncertainty estimates.
Aleatoric and Epistemic Uncertainty with Random Forests
It is shown how two general approaches for measuring the learner's aleatoric and epistemic uncertainty in a prediction can be instantiated with decision trees and random forests as learning algorithms in a classification setting.
Evidential Deep Learning to Quantify Classification Uncertainty
This work treats predictions of a neural net as subjective opinions and learns the function that collects the evidence leading to these opinions by a deterministic neural net from data, which achieves unprecedented success on detection of out-of-distribution queries and endurance against adversarial perturbations.
Posterior Network: Uncertainty Estimation without OOD Samples via Density-Based Pseudo-Counts
The Posterior Network (PostNet) is proposed, which uses Normalizing Flows to predict an individual closed-form posterior distribution over predicted probabilites for any input sample, and achieves state-of-the art results in OOD detection and in uncertainty calibration under dataset shifts.
Deep Evidential Regression
This paper proposes a novel method for training deterministic NNs to not only estimate the desired target but also the associated evidence in support of that target by placing evidential priors over the original Gaussian likelihood function and training the NN to infer the hyperparameters of the authors' evidential distribution.
On Calibration of Modern Neural Networks
It is discovered that modern neural networks, unlike those from a decade ago, are poorly calibrated, and on most datasets, temperature scaling -- a single-parameter variant of Platt Scaling -- is surprisingly effective at calibrating predictions.
Predictive Uncertainty Estimation via Prior Networks
This work proposes a new framework for modeling predictive uncertainty called Prior Networks (PNs) which explicitly models distributional uncertainty by parameterizing a prior distribution over predictive distributions and evaluates PNs on the tasks of identifying out-of-distribution samples and detecting misclassification on the MNIST dataset, where they are found to outperform previous methods.