• Corpus ID: 227407829

Scaling Out-of-Distribution Detection for Real-World Settings

@inproceedings{Hendrycks2022ScalingOD,
  title={Scaling Out-of-Distribution Detection for Real-World Settings},
  author={Dan Hendrycks and Steven Basart and Mantas Mazeika and Mohammadreza Mostajabi and Jacob Steinhardt and Dawn Xiaodong Song},
  booktitle={ICML},
  year={2022}
}
Detecting out-of-distribution examples is important for safety-critical machine learning applications such as medical screening and self-driving cars. However, existing research mainly focuses on simple small-scale settings. To set the stage for more realistic out-of-distribution detection, we depart from small-scale settings and explore large-scale multiclass and multi-label settings with high-resolution images and hundreds of classes. To make future work in real-world settings possible, we… 
WILDS: A Benchmark of in-the-Wild Distribution Shifts
TLDR
WILDS is presented, a benchmark of in-the-wild distribution shifts spanning diverse data modalities and applications, and is hoped to encourage the development of general-purpose methods that are anchored to real-world distribution shifts and that work well across different applications and problem settings.
Augmenting Softmax Information for Selective Classification with Out-of-Distribution Data
TLDR
A novel method is proposed, Softmax Information Retaining Combination (SIRC), that augments softmax-based con-�dence scores with feature-agnostic information such that their ability to identify OOD samples is improved without sacrificing separation between correct and incorrect ID predictions.
DenseHybrid: Hybrid Anomaly Detection for Dense Open-set Recognition
TLDR
A novel hybrid algorithm based on reinterpreting discriminative logits as a logarithm of the unnormalized joint distribution which allows dense open-set recognition on large natural images and outperforms each of the two components.
Hyperdimensional Feature Fusion for Out-Of-Distribution Detection
TLDR
It is shown that the hyperdimensional fusion of multiple network layers is critical to achieve best general performance and expressive classspecific descriptor vectors for all in-distribution classes are created.
Consensus Synergizes with Memory: A Simple Approach for Anomaly Segmentation in Urban Scenes
TLDR
This paper proposes a novel and simple approach named Consensus Synergizes with Memory (CosMe), inspired by the psychology finding that groups perform better than individuals on memory tasks, and shows that CosMe outperforms previous approaches by large margins.
Deep Metric Learning for Open World Semantic Segmentation
TLDR
This paper adopts the Deep Metric Learning Network (DMLNet) with contrastive clustering to implement open-set semantic segmentation, which behaves like a human being, which is able to identify OOD objects and gradually learn them with corresponding supervision.
Standardized Max Logits: A Simple yet Effective Approach for Identifying Unexpected Road Obstacles in Urban-Scene Segmentation
TLDR
This work proposes a simple yet effective approach that standardizes themax logits in order to align the different distributions and reflect the relative meanings of max logits within each predicted class, and considers the local regions from two different perspectives based on the intuition that neighboring pixels share similar semantic information.
Task-Driven Detection of Distribution Shifts with Statistical Guarantees for Robot Learning
TLDR
The resulting approach provides guaranteed confidence bounds on OOD detection including bounds on both the false positive and false negative rates of the detector and is task-driven and sensitive only to changes that impact the robot’s performance.
SegmentMeIfYouCan: A Benchmark for Anomaly Segmentation
TLDR
The “SegmentMeIfYouCan” benchmark is introduced, which addresses two tasks: Anomalous object segmentation, which considers any previously-unseen object category; and road obstacle segmentation), which focuses on any object on the road, may it be known or unknown.
Out-of-Distribution Detection in Time-Series Domain: A Novel Seasonal Ratio Scoring Approach
TLDR
Experiments on diverse real-world benchmarks demonstrate that the SRS method is well-suited for time-series OOD detection when compared to baseline methods.
...
...

References

SHOWING 1-10 OF 50 REFERENCES
Discriminative out-of-distribution detection for semantic segmentation
TLDR
The proposed approach to discriminative detection of OOD pixels in input data succeeds to identify out-of-distribution pixels while outperforming previous work by a wide margin.
Learning Confidence for Out-of-Distribution Detection in Neural Networks
TLDR
This work proposes a method of learning confidence estimates for neural networks that is simple to implement and produces intuitively interpretable outputs, and addresses the problem of calibrating out-of-distribution detectors.
Deep Anomaly Detection with Outlier Exposure
TLDR
In extensive experiments on natural language processing and small- and large-scale vision tasks, it is found that Outlier Exposure significantly improves detection performance and that cutting-edge generative models trained on CIFar-10 may assign higher likelihoods to SVHN images than to CIFAR-10 images; OE is used to mitigate this issue.
Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks
TLDR
The proposed ODIN method, based on the observation that using temperature scaling and adding small perturbations to the input can separate the softmax score distributions between in- and out-of-distribution images, allowing for more effective detection, consistently outperforms the baseline approach by a large margin.
A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks
TLDR
This paper proposes a simple yet effective method for detecting any abnormal samples, which is applicable to any pre-trained softmax neural classifier, and obtains the class conditional Gaussian distributions with respect to (low- and upper-level) features of the deep models under Gaussian discriminant analysis.
The Cityscapes Dataset for Semantic Urban Scene Understanding
TLDR
This work introduces Cityscapes, a benchmark suite and large-scale dataset to train and test approaches for pixel-level and instance-level semantic labeling, and exceeds previous attempts in terms of dataset size, annotation richness, scene variability, and complexity.
A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks
TLDR
A simple baseline that utilizes probabilities from softmax distributions is presented, showing the effectiveness of this baseline across all computer vision, natural language processing, and automatic speech recognition, and it is shown the baseline can sometimes be surpassed.
Deep Autoencoding Models for Unsupervised Anomaly Segmentation in Brain MR Images
TLDR
This work shows that deep spatial autoencoding models can be efficiently used to capture normal anatomical variability of entire 2D brain MR images and shows that constraints on the latent space and adversarial training can further improve the segmentation performance over standard deep representation learning.
Using Self-Supervised Learning Can Improve Model Robustness and Uncertainty
TLDR
This work finds that self-supervision can benefit robustness in a variety of ways, including robustness to adversarial examples, label corruption, and common input corruptions, and greatly benefits out-of-distribution detection on difficult, near-dist distribution outliers.
Towards Pixel-Level OOD Detection for Semantic Segmentation
TLDR
This thesis adapts state-of-the-art methods for detecting out of distribution images for image classification to the new task of detectingOut of distribution pixels, which can localise the unusual objects.
...
...