Confidence-based Out-of-Distribution Detection: A Comparative Study and Analysis

  title={Confidence-based Out-of-Distribution Detection: A Comparative Study and Analysis},
  author={Christoph Berger and Magdalini Paschali and Ben Glocker and Konstantinos Kamnitsas},
Image classification models deployed in the real world may receive inputs outside the intended data distribution. For critical applications such as clinical decision making, it is important that a model can detect such out-of-distribution (OOD) inputs and express its uncertainty. In this work, we assess the capability of various state-of-the-art approaches for confidence-based OOD detection through a comparative study and in-depth analysis. First, we leverage a computer vision benchmark to… 

Figures and Tables from this paper

Robust, Primitive, and Unsupervised Quality Estimation for Segmentation Ensembles
A multitude of image-based machine learning segmentation and classification algorithms has recently been proposed, offering diagnostic decision support for the identification and characterization of
CertainNet: Sampling-Free Uncertainty Estimation for Object Detection
This work proposes a novel sampling-free uncertainty estimation method for object detection, and it is the first to provide separate uncertainties for each output signal: objectness, class, location and size.
Test for non-negligible adverse shifts
This work proposes a framework to detect adverse shifts based on outlier scores, D-SOS, which is uniquely tailored to serve as a robust metric for model monitoring and data validation.


Analyzing the Quality and Challenges of Uncertainty Estimations for Brain Tumor Segmentation
An analysis of the most commonly used uncertainty estimation methods in regards to benefits and challenges for brain tumor segmentation, and shows that the uncertainty methods are typically well-calibrated when evaluated at the dataset level.
Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks
The proposed ODIN method, based on the observation that using temperature scaling and adding small perturbations to the input can separate the softmax score distributions between in- and out-of-distribution images, allowing for more effective detection, consistently outperforms the baseline approach by a large margin.
CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison
A labeler is designed to automatically detect the presence of 14 observations in radiology reports, capturing uncertainties inherent in radiograph interpretation, in CheXpert, a large dataset that contains 224,316 chest radiographs of 65,240 patients.
Autoencoders for Unsupervised Anomaly Segmentation in Brain MR Images: A Comparative Study
A ranking of the methods in the field of Unsupervised Anomaly Detection in brain MRI by utilizing a single architecture, a single resolution and the same dataset(s) is provided, to establish comparability among recent methods.
Exploring Uncertainty Measures in Deep Networks for Multiple Sclerosis Lesion Detection and Segmentation
A 3D MS lesion segmentation CNN is developed, augmented to provide four different voxel-based uncertainty measures based on Monte Carlo (MC) dropout, and empirical evidence suggests that uncertainty measures consistently allow us to choose superior operating points compared only using the network’s sigmoid output as a probability.
Unsupervised Lesion Detection via Image Restoration with a Normative Prior
A probabilistic model that uses a network-based prior as the normative distribution and detect lesions pixel-wise using MAP estimation is proposed and outperforms the state-of-the-art unsupervised methods by a substantial margin.
Exploring uncertainty measures in deep networks for Multiple sclerosis lesion detection and segmentation
A 3D MS lesion segmentation CNN is developed, augmented to provide four different voxel-based uncertainty measures based on MC dropout, and the first exploration of multiple uncertainty estimates based on Monte Carlo (MC) dropout is presented in the context of deep networks for lesion detection and segmentation in medical images.
Bayesian Image Quality Transfer with CNNs: Exploring Uncertainty in dMRI Super-Resolution
This work proposes to account for intrinsic uncertainty through a per-patch heteroscedastic noise model and for parameter uncertainty through approximate Bayesian inference in the form of variational dropout and shows that the combined benefits lead to the state-of-the-art performance SR of diffusion MR brain images in terms of errors compared to ground truth.
Unsupervised Anomaly Detection with Generative Adversarial Networks to Guide Marker Discovery
AnoGAN, a deep convolutional generative adversarial network is proposed to learn a manifold of normal anatomical variability, accompanying a novel anomaly scoring scheme based on the mapping from image space to a latent space.
A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks
A simple baseline that utilizes probabilities from softmax distributions is presented, showing the effectiveness of this baseline across all computer vision, natural language processing, and automatic speech recognition, and it is shown the baseline can sometimes be surpassed.