Corpus ID: 236428710

Improve Unsupervised Pretraining for Few-label Transfer

  title={Improve Unsupervised Pretraining for Few-label Transfer},
  author={Suichan Li and Dongdong Chen and Yinpeng Chen and Lu Yuan and Lei Zhang and Qi Chu and Bin Liu and Nenghai Yu},
  • Suichan Li, Dongdong Chen, +5 authors Nenghai Yu
  • Published 2021
  • Computer Science
  • ArXiv
Unsupervised pretraining has achieved great success and many recent works have shown unsupervised pretraining can achieve comparable or even slightly better transfer performance than supervised pretraining on downstream target datasets. But in this paper, we find this conclusion may not hold when the target dataset has very few labeled samples for finetuning, i.e., few-label transfer. We analyze the possible reason from the clustering perspective: 1) The clustering quality of target samples is… Expand

Figures and Tables from this paper


A Closer Look at Few-shot Classification
The results reveal that reducing intra-class variation is an important factor when the feature backbone is shallow, but not as critical when using deeper backbones, and a baseline method with a standard fine-tuning practice compares favorably against other state-of-the-art few-shot learning algorithms. Expand
Optimization as a Model for Few-Shot Learning
Rethinking ImageNet Pre-Training
Experiments show that ImageNet pre-training speeds up convergence early in training, but does not necessarily provide regularization or improve final target task accuracy, and these discoveries will encourage people to rethink the current de facto paradigm of `pre-training and fine-tuning' in computer vision. Expand
Unsupervised Feature Learning via Non-parametric Instance Discrimination
This work forms this intuition as a non-parametric classification problem at the instance-level, and uses noise-contrastive estimation to tackle the computational challenges imposed by the large number of instance classes. Expand
Big Transfer (BiT): General Visual Representation Learning
By combining a few carefully selected components, and transferring using a simple heuristic, Big Transfer achieves strong performance on over 20 datasets and performs well across a surprisingly wide range of data regimes -- from 1 example per class to 1M total examples. Expand
Density-Aware Graph for Deep Semi-Supervised Visual Recognition
This paper proposes a new Density-aware Neighborhood Aggregation module to learn more discriminative features by incorporating the neighborhood information in a density-aware manner and a novel D density-ascending Path based Label Propagation module to generate the pseudo-labels for unlabeled samples more efficiently according to the feature distribution characterized by density. Expand
Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
We propose an algorithm for meta-learning that is model-agnostic, in the sense that it is compatible with any model trained with gradient descent and applicable to a variety of different learningExpand
ReMixMatch: Semi-Supervised Learning with Distribution Alignment and Augmentation Anchoring
A variant of AutoAugment which learns the augmentation policy while the model is being trained, and is significantly more data-efficient than prior work, requiring between $5\times and $16\times less data to reach the same accuracy. Expand
Deep Residual Learning for Image Recognition
This work presents a residual learning framework to ease the training of networks that are substantially deeper than those used previously, and provides comprehensive empirical evidence showing that these residual networks are easier to optimize, and can gain accuracy from considerably increased depth. Expand
Link-based Active Learning
Different ways of adapting existing active learning work to network data are proposed while utilizing links to select better examples to label, to improve the quality of the learned models. Expand