• Corpus ID: 220525799

Long-tail learning via logit adjustment

@article{Menon2021LongtailLV,
  title={Long-tail learning via logit adjustment},
  author={Aditya Krishna Menon and Sadeep Jayasumana and Ankit Singh Rawat and Himanshu Jain and Andreas Veit and Sanjiv Kumar},
  journal={ArXiv},
  year={2021},
  volume={abs/2007.07314}
}
Real-world classification problems typically exhibit an imbalanced or long-tailed label distribution, wherein many labels are associated with only a few samples. This poses a challenge for generalisation on such labels, and also makes naive learning biased towards dominant labels. In this paper, we present two simple modifications of standard softmax cross-entropy training to cope with these challenges. Our techniques revisit the classic idea of logit adjustment based on the label frequencies… 
Improving Long-Tailed Classification from Instance Level
TLDR
This paper revisits the long-tailed problem from the instance level and proposes two instancelevel components to improve long-tails classification, including an Adaptive Logit Adjustment (ALA) loss, which applies an adaptive adjusting term to the logit.
Label-Imbalanced and Group-Sensitive Classification under Overparameterization
TLDR
It is proved that for all linear classifiers trained in TPT, it is necessary to introduce multiplicative, rather than additive, logit adjustments so that the interclass margins change appropriately, and it is shown that additive adjustments can speed up convergence by countering the initial negative effect of the multiplicative weights.
Disentangling Sampling and Labeling Bias for Learning in Large-Output Spaces
TLDR
This paper shows that different negative sampling schemes implicitly trade-off performance on dominant versus rare labels, and provides a unified means to explicitly tackle both sampling bias and labeling bias, which is inherent to the data due to label imbalance.
Balanced Knowledge Distillation for Long-tailed Learning
TLDR
This work explores knowledge distillation in long-tailed scenarios and proposes a novel distillation framework, named Balanced Knowledge Distillation (BKD), to disentangle the contradiction between the two goals of longtailed learning, i.e., learning generalizable representations and facilitating learning for tail classes.
CReST: A Class-Rebalancing Self-Training Framework for Imbalanced Semi-Supervised Learning
TLDR
The proposed Class-Rebalancing Self-Training (CReST), a simple yet effective framework to improve existing semi-supervised learning methods on class-imbalanced data, and a progressive distribution alignment to adaptively adjust the rebalancing strength dubbed CReST+.
Energy Aligning for Biased Models
TLDR
A simple and effective method named Energy Aligning is proposed by merely adding the calculated shift scalars onto the output logits during inference, which does not require to modify the network architectures, intervene the standard learning paradigm, or perform two-stage training.
Overparameterisation and worst-case generalisation: friend or foe?
TLDR
It is shown that in certain settings, overparameterised models’ performance on under-represented subgroups may be improved via post-hoc processing, and two post- hoc correction techniques to mitigate this bias are detailed.
Sample Selection with Uncertainty of Losses for Learning with Noisy Labels
TLDR
This paper incorporates the uncertainty of losses by adopting interval estimation instead of point estimation of losses, where lower bounds of the con dence intervals of losses derived from distribution-free concentration inequalities, but not losses themselves, are used for sample selection.
Class-Balanced Distillation for Long-Tailed Visual Recognition
TLDR
This work introduces a new training method, referred to as Class-Balanced Distillation (CBD), that leverages knowledge distillation to enhance feature representations and consistently outperforms the state of the art on long-tailed recognition benchmarks such as ImageNet-LT, iNaturalist17 and i naturalist18.
mGEV: Extension of the GEV Activation to Multiclass Classification
TLDR
This work conducts experiments on X-ray images, with three classes, showing much-improved performance over the commonly used softmax activation of the generalized extreme value (GEV) activation.
...
...

References

SHOWING 1-10 OF 83 REFERENCES
Decoupling Representation and Classifier for Long-Tailed Recognition
TLDR
It is shown that it is possible to outperform carefully designed losses, sampling strategies, even complex modules with memory, by using a straightforward approach that decouples representation and classification.
Striking the Right Balance With Uncertainty
TLDR
This paper demonstrates that the Bayesian uncertainty estimates directly correlate with the rarity of classes and the difficulty level of individual samples, and presents a novel framework for uncertainty based class imbalance learning that efficiently utilizes sample and class uncertainty information to learn robust features and more generalizable classifiers.
To Balance or Not to Balance: A Simple-yet-Effective Approach for Learning with Long-Tailed Distributions
TLDR
To promote the awareness of tail-classes, a class-balanced sampling scheme is utilised for training both the classifier and the feature extractor, and an auxiliary training task is introduced, which is to train a classifier under the regular random sampling scheme.
Reviving Threshold-Moving: a Simple Plug-in Bagging Ensemble for Binary and Multiclass Imbalanced Data
TLDR
Contrary to usual rebalancing practice, the proposed probability thresholding bagging (PT-bagging), a versatile plug-in method that preserves the natural class distribution of the data resulting in well calibrated posterior probabilities, is presented.
Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss
TLDR
A theoretically-principled label-distribution-aware margin (LDAM) loss motivated by minimizing a margin-based generalization bound is proposed that replaces the standard cross-entropy objective during training and can be applied with prior strategies for training with class-imbalance such as re-weighting or re-sampling.
Large-Scale Long-Tailed Recognition in an Open World
TLDR
An integrated OLTR algorithm is developed that maps an image to a feature space such that visual concepts can easily relate to each other based on a learned metric that respects the closed-world classification while acknowledging the novelty of the open world.
Training Cost-Sensitive Neural Networks with Methods Addressing the Class Imbalance Problem
TLDR
The empirical study suggests that some methods that have been believed to be effective in addressing the class imbalance problem may, in fact, only be effective on learning with imbalanced two-class data sets.
Apportioned Margin Approach for Cost Sensitive Large Margin Classifiers
TLDR
A framework to address the problem of cost sensitive multiclass classification, where the sensitivity of an important class at the expense of a less important one is increased, and an efficient margin shift between classes that share the same boundary is adopted.
Class Imbalance, Redux
TLDR
This work approaches the problem of imbalance from a probabilistic perspective, and from this vantage identify dataset characteristics that exacerbate the problem and advocates the approach of bagging an ensemble of classifiers induced over balanced bootstrap training samples, arguing that this strategy will often succeed where others fail.
The Balanced Accuracy and Its Posterior Distribution
TLDR
It is shown that both problems can be overcome by replacing the conventional point estimate of accuracy by an estimate of the posterior distribution of the balanced accuracy.
...
...