• Corpus ID: 225062556

Coping with Label Shift via Distributionally Robust Optimisation

@article{Zhang2020CopingWL,
  title={Coping with Label Shift via Distributionally Robust Optimisation},
  author={J. Zhang and Aditya Krishna Menon and Andreas Veit and Srinadh Bhojanapalli and Sanjiv Kumar and Suvrit Sra},
  journal={ArXiv},
  year={2020},
  volume={abs/2010.12230}
}
The label shift problem refers to the supervised learning setting where the train and test label distributions do not match. Existing work addressing label shift usually assumes access to an \emph{unlabelled} test sample. This sample may be used to estimate the test label distribution, and to then train a suitably re-weighted classifier. While approaches using this idea have proven effective, their scope is limited as it is not always feasible to access the target domain; further, they require… 

Figures and Tables from this paper

Robust Generalization despite Distribution Shift via Minimum Discriminating Information

This paper introduces a modeling framework where, in addition to training data, the author has partial structural knowledge of the shifted test distribution, and employs the principle of minimum discriminating information to embed the available prior knowledge.

Just Train Twice: Improving Group Robustness without Training Group Information

This paper proposes a simple two-stage approach, JTT, that minimizes the loss over a reweighted dataset where the authors upweight training examples that are misclassified at the end of a few steps of standard training, leading to improved worst-group performance.

Optimal algorithms for group distributionally robust optimization and beyond

Stochastic algorithms for a class of DRO problems including group DRO, subpopulation fairness, and empirical conditional value at risk (CVaR) optimization are devised.

Spuriosity Rankings: Sorting Data for Spurious Correlation Robustness

A framework for ranking images within their class based on the strength of spurious cues present and a second metric to assess feature reliability, finding that spurious features are generally less reliable than non-spurious ones, though again, spurious features can be more reliable for certain classes.

Learning with Noisy Labels over Imbalanced Subpopulations

A novel LNL method to simultaneously deal with noisy labels and imbalanced subpopulations is proposed, which leverages sample correlation to estimate samples’ clean probabilities for label correction and then utilizes corrected labels for Distributionally Robust Optimization (DRO) to further improve the robustness.

RLS BENCH : A Large-Scale Empirical Study of Domain Adaptation Under Relaxed Label Shift

  • Computer Science
  • 2022
This paper introduces RLS BENCH, a large-scale benchmark for such relaxed label shift settings, and develops an effective meta-algorithm, compatible with most deep domain adaptation heuristics, that consists of the following two steps: pseudo-balance the data at each epoch and adjust the final class with an estimate of a target label distribution.

Just Mix Once: Worst-group Generalization by Group Interpolation

This work proposes to unify and generalize self-supervision and oversampling using a class-conditional variant of mixup tailored for worst-group generalization, and shows that this approach, Just Mix Once (JM1), performs on par or better than the state-of-the-art on worst- group generalization.

Wild-Time: A Benchmark of in-the-Wild Distribution Shift over Time

Wild-Time is curate, a benchmark of 7 datasets that reflect temporal distribution shifts arising in a variety of real-world applications that demonstrate that existing methods are limited in tackling temporal distribution shift.

Explicit Tradeoffs between Adversarial and Natural Distributional Robustness

It is shown that adversarially trained classifiers rely on backgrounds more than their standardly trained counterparts, validating the theoretical results and revealing that previous claims that adversarial vulnerability is rooted in spurious correlations are incomplete.

References

SHOWING 1-10 OF 52 REFERENCES

Regularized Learning for Domain Adaptation under Label Shifts

We propose Regularized Learning under Label shifts (RLLS), a principled and a practical domain-adaptation algorithm to correct for shifts in the label distribution between a source and a target

An introduction to domain adaptation and transfer learning

In machine learning, if the training data is an unbiased sample of an underlying distribution, then the learned classification function will make accurate predictions for new samples. However, if the

Mixture Regression for Covariate Shift

The main advantages of this new formulation over previous models for covariate shift are that the test and training densities are known, the regression and density estimation are combined into a single procedure, and previous methods are reproduced as special cases of this procedure, shedding light on the implicit assumptions the methods are making.

Domain Adaptation with Conditional Distribution Matching and Generalized Label Shift

A recent upper-bound on the performance of adversarial domain adaptation is extended to multi-class classification and more general discriminators and generalized label shift (GLS) is proposed as a way to improve robustness against mismatched label distributions.

Does Distributionally Robust Supervised Learning Give Robust Classifiers?

This paper proves that the DRSL just ends up giving a classifier that exactly fits the given training distribution, which is too pessimistic, and proposes simple D RSL that overcomes this pessimism and empirically demonstrate its effectiveness.

A Unified View of Label Shift Estimation

A unified view of the two methods and the first theoretical characterization of the likelihood-based estimator is presented, attributing BBSE's statistical inefficiency to a loss of information due to coarse calibration.

Detecting and Correcting for Label Shift with Black Box Predictors

Black Box Shift Estimation (BBSE) is proposed to estimate the test distribution of p(y) and it is proved BBSE works even when predictors are biased, inaccurate, or uncalibrated, so long as their confusion matrices are invertible.

Domain Adaptation under Target and Conditional Shift

This work considers domain adaptation under three possible scenarios, kernel embedding of conditional as well as marginal distributions, and proposes to estimate the weights or transformations by reweighting or transforming training data to reproduce the covariate distribution on the test domain.

Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss

A theoretically-principled label-distribution-aware margin (LDAM) loss motivated by minimizing a margin-based generalization bound is proposed that replaces the standard cross-entropy objective during training and can be applied with prior strategies for training with class-imbalance such as re-weighting or re-sampling.

Decoupling Representation and Classifier for Long-Tailed Recognition

It is shown that it is possible to outperform carefully designed losses, sampling strategies, even complex modules with memory, by using a straightforward approach that decouples representation and classification.
...