• Corpus ID: 237364121

Towards Out-Of-Distribution Generalization: A Survey

@article{Shen2021TowardsOG,
  title={Towards Out-Of-Distribution Generalization: A Survey},
  author={Zheyan Shen and Jiashuo Liu and Yue He and Xingxuan Zhang and Renzhe Xu and Han Yu and Peng Cui},
  journal={ArXiv},
  year={2021},
  volume={abs/2108.13624}
}
Classic machine learning methods are built on the i.i.d. assumption that training and testing data are independent and identically distributed. However, in real scenarios, the i.i.d. assumption can hardly be satisfied, rendering the sharp drop of classic machine learning algorithms’ performances under distributional shifts, which indicates the significance of investigating the Out-of-Distribution generalization problem. Out-of-Distribution (OOD) generalization problem addresses the challenging… 

Figures and Tables from this paper

OoD-Bench: Benchmarking and Understanding Out-of-Distribution Generalization Datasets and Algorithms

This work identifies and measure two distinct kinds of distribution shifts that are ubiquitous in various datasets and compares OoD generalization algorithms across two groups of benchmarks, revealing their strengths on one distribution shift as well as limitations on the other distribution shift.

OoD-Bench: Quantifying and Understanding Two Dimensions of Out-of-Distribution Generalization

  • Nanyang YeKaican Li Jun Zhu
  • Computer Science
    2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
  • 2022
This work first identifies and measure two distinct kinds of distribution shifts that are ubiquitous in various datasets, and compares OoD generalization algorithms across two groups of benchmarks, revealing their strengths on one shift as well as limitations on the other shift.

Out-Of-Distribution Generalization on Graphs: A Survey

This paper is the first systematic and comprehensive review of OOD generalization on graphs and categorizes existing methods into three classes from conceptually different perspectives, i.e., data, model, and learning strategy, based on their positions in the graph machine learning pipeline.

General Greedy De-bias Learning

This paper proposes a General Greedy De-bias learning framework (GGD), which greedily trains the biased models and base model and introduces the Curriculum Regularization inspired by curriculum learning, which achieves a good trade-off between in-dist distribution and out-of-distribution performance.

An Empirical Study on Distribution Shift Robustness From the Perspective of Pre-Training and Data Augmentation

This empirical study provides a comprehensive baseline for a wide range of pre-training models tuned with data augmentation, which potentially inspires research exploiting the power ofPre-training andData augmentation in the future of distribution shift study.

Towards Domain Generalization in Object Detection

This paper formulate the DGOD problem and propose a comprehensive evaluation benchmark and proposes a novel method named Region Aware Proposal reweighTing (RAPT) to eliminate dependence within RoI features and demonstrates that this method outperforms other state-of-the-art counterparts.

Uncertainty Modeling for Out-of-Distribution Generalization

This paper hypothesizes that the feature statistic, after considering the potential uncertainties, follows a multivariate Gaussian distribution, so that each feature statistic is no longer a deterministic value, but a probabilistic point with diverse distribution possibilities.

GOOD: A Graph Out-of-Distribution Benchmark

This work aims at developing an OOD benchmark, known as GOOD, for graphs specifically, and explicitly makes distinctions between covariate and concept shifts and design data splits that accurately reflect different shifts.

Domain Decorrelation with Potential Energy Ranking

PoER is proposed using Po tential E nergy R anking (PoER) to decouple the object feature and the domain feature in given images, promoting the learning of label-discriminative features while limiting the irrelevant correlations between the objects and the background.

Generalizing to Unseen Domains: A Survey on Domain Generalization

This paper provides a formal definition of domain generalization and discusses several related fields, and categorizes recent algorithms into three classes and present them in detail: data manipulation, representation learning, and learning strategy, each of which contains several popular algorithms.
...

References

SHOWING 1-10 OF 238 REFERENCES

Towards a Theoretical Framework of Out-of-Distribution Generalization

This work takes the first step towards rigorous and quantitative definitions of 1) what is OOD; and 2) what does it mean by saying an OOD problem is learnable; and introduces a new concept of expansion function, which characterizes to what extent the variance is amplified in the test domains over the training domains, and therefore give a quantitative meaning of invariant features.

Deep Stable Learning for Out-Of-Distribution Generalization

This paper proposes to address the dependencies between features via learning weights for training samples, which helps deep models get rid of spurious correlations and, in turn, concentrate more on the true connection between discriminative features and labels.

OoD-Bench: Benchmarking and Understanding Out-of-Distribution Generalization Datasets and Algorithms

This work identifies and measure two distinct kinds of distribution shifts that are ubiquitous in various datasets and compares OoD generalization algorithms across two groups of benchmarks, revealing their strengths on one distribution shift as well as limitations on the other distribution shift.

Domain Generalization: A Survey

A comprehensive literature review in DG is provided to summarize the developments over the past decade and cover the background by formally defining DG and relating it to other relevant fields like domain adaptation and transfer learning.

Out-of-Distribution Generalization Analysis via Influence Function

Influence Function is introduced, a classical tool from robust statistics, into the OOD generalization problem and the variance of influence function is suggested to monitor the stability of a model on training domains to help discern whether OOD algorithms are needed and whether a model achieves good OODgeneralization.

A Survey on Transfer Learning

The relationship between transfer learning and other related machine learning techniques such as domain adaptation, multitask learning and sample selection bias, as well as covariate shift are discussed.

Out-of-Distribution Generalization via Risk Extrapolation (REx)

This work introduces the principle of Risk Extrapolation (REx), and shows conceptually how this principle enables extrapolation, and demonstrates the effectiveness and scalability of instantiations of REx on various OoD generalization tasks.

Out-of-Distribution Generalization with Maximal Invariant Predictor

The basic results of probability are used to prove maximal Invariant Predictor condition, a theoretical result that can be used to identify the OOD optimal solution and the superiority of IGA over previous methods on both the original and the extended version of Colored-MNIST.

Stable Prediction across Unknown Environments

This paper proposes a novel Deep Global Balancing Regression (DGBR) algorithm to jointly optimize a deep auto-encoder model for feature selection and a global balancing model for stable prediction across unknown environments, and demonstrates that the algorithm outperforms the state-of-the-art methods for stable predictions acrossunknown environments.
...