Pseudo-Labeling and Confirmation Bias in Deep Semi-Supervised Learning

@article{Arazo2020PseudoLabelingAC,
  title={Pseudo-Labeling and Confirmation Bias in Deep Semi-Supervised Learning},
  author={Eric Arazo and Diego Ortego and Paul Albert and Noel E. O'Connor and Kevin McGuinness},
  journal={2020 International Joint Conference on Neural Networks (IJCNN)},
  year={2020},
  pages={1-8}
}
Semi-supervised learning, i.e. jointly learning from labeled an unlabeled samples, is an active research topic due to its key role on relaxing human annotation constraints. [...] Key Method We show that a naive pseudo-labeling overfits to incorrect pseudo-labels due to the so-called confirmation bias and demonstrate that label noise and mixup augmentation are effective regularization techniques for reducing it. The proposed approach achieves state-of-the-art results in CIFAR-10/100 and Mini-Imaget despite being…Expand
Distribution Aligning Refinery of Pseudo-label for Imbalanced Semi-supervised Learning
TLDR
A convex optimization problem to softly refine the pseudo-labels generated from the biased model is formulated, and a simple algorithm is developed, named Distribution Aligning Refinery of Pseudo-label (DARP) that solves it provably and efficiently. Expand
Self-Supervised Wasserstein Pseudo-Labeling for Semi-Supervised Image Classification
The goal is to use Wasserstein metric to provide pseudo labels for the unlabeled images to train a Convolutional Neural Networks (CNN) in a Semi-Supervised Learning (SSL) manner for theExpand
ReLaB: Reliable Label Bootstrapping for Semi-Supervised Learning
TLDR
Reliable Label Bootstrapping (ReLaB) is proposed, an unsupervisedpreprossessing algorithm that paves the way for semi-supervised learning solutions, enabling them to work with much lowersupervision. Expand
Transfer of Pretrained Model Weights Substantially Improves Semi-supervised Image Classification
TLDR
It is found that transfer learning always substantially improves the model’s accuracy when few labeled examples are available, regardless of the type of loss used for training the neural network. Expand
Semi-Supervised Few-Shot Learning with Pseudo Label Refinement
  • Pan Li, Guile Wu, S. Gong, Xu Lan
  • Computer Science
  • 2021 IEEE International Conference on Multimedia and Expo (ICME)
  • 2021
TLDR
A new baseline method for semi-supervised few-shot learning by iterative pseudo label refinement to reduce noise is introduced and the label noise propagation problem is investigated and improved with a denoising network to learn distributions of clean and noisy pseudo-labelled examples via a mixture model. Expand
Self-Tuning for Data-Efficient Deep Learning
TLDR
SelfTuning is presented to enable data-efficient deep learning by unifying the exploration of labeled and unlabeled data and the transfer of a pre-trained model, as well as a Pseudo Group Contrast (PGC) mechanism to mitigate the reliance on pseudo-labels and boost the tolerance to false labels. Expand
Ask-n-Learn: Active Learning via Reliable Gradient Representations for Image Classification
TLDR
Ask-n-Learn, an active learning approach based on gradient embeddings obtained using the pesudo-labels estimated in each iteration of the algorithm, is proposed and significant improvements over state-of-the-art baselines, including the recently proposed BADGE algorithm are demonstrated. Expand
Streaming Self-Training via Domain-Agnostic Unlabeled Images
We present streaming self-training (SST) that aims to democratize the process of learning visual recognition models such that a non-expert user can define a new task depending on their needs via aExpand
FMixCutMatch for semi-supervised deep learning
  • Xiang Wei, Xiaotao Wei, Xiangyuan Kong, Siyang Lu, Weiwei Xing, Wei Lu
  • Medicine, Computer Science
  • Neural Networks
  • 2021
TLDR
The proposed FMix cutMatch achieves state-of-the-art performance on CIFAR-10/100, SVHN and Mini-Imagenet across a variety of SSL conditions with the CNN-13, WRN-28-2 and ResNet-18 networks. Expand
Semi-supervised Text Classification with Temporal Ensembling
  • Rong Xiang, Shiqun Yin
  • 2021 International Conference on Computer Communication and Artificial Intelligence (CCAI)
  • 2021
Supervised learning algorithms require a lot of labeled data to train the model while obtaining labeled data for large datasets is costly and time consuming. Semi-supervised learning algorithms canExpand
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 54 REFERENCES
Label Propagation for Deep Semi-Supervised Learning
TLDR
This work employs a transductive label propagation method that is based on the manifold assumption to make predictions on the entire dataset and use these predictions to generate pseudo-labels for the unlabeled data and train a deep neural network. Expand
Realistic Evaluation of Deep Semi-Supervised Learning Algorithms
TLDR
This work creates a unified reimplemention and evaluation platform of various widely-used SSL techniques and finds that the performance of simple baselines which do not use unlabeled data is often underreported, that SSL methods differ in sensitivity to the amount of labeled and unlabeling data, and that performance can degrade substantially when the unlabelED dataset contains out-of-class examples. Expand
Certainty-Driven Consistency Loss for Semi-supervised Learning
TLDR
A novel certainty-driven consistency loss (CCL) to dynamically select data samples that have relatively low uncertainty to guide the student learn more meaningful and certain/reliable targets, and hence improve the quality of the gradients backpropagated to the student. Expand
Regularization With Stochastic Transformations and Perturbations for Deep Semi-Supervised Learning
TLDR
An unsupervised loss function is proposed that takes advantage of the stochastic nature of these methods and minimizes the difference between the predictions of multiple passes of a training sample through the network. Expand
Smooth Neighbors on Teacher Graphs for Semi-Supervised Learning
TLDR
A novel method, called Smooth Neighbors on Teacher Graphs (SNTG), which serves as a similarity measure with respect to which the representations of "similar" neighboring points are learned to be smooth on the low-dimensional manifold and achieves state-of-the-art results on semi-supervised learning benchmarks. Expand
Temporal Ensembling for Semi-Supervised Learning
TLDR
Self-ensembling is introduced, where it is shown that this ensemble prediction can be expected to be a better predictor for the unknown labels than the output of the network at the most recent training epoch, and can thus be used as a target for training. Expand
Exploiting Unlabeled Data in CNNs by Self-Supervised Learning to Rank
TLDR
The results show that networks trained to regress to the ground truth targets for labeled data and to simultaneously learn to rank unlabeled data obtain significantly better, state-of-the-art results for both IQA and crowd counting. Expand
Unsupervised label noise modeling and loss correction
TLDR
A suitable two-component mixture model is suggested as an unsupervised generative model of sample loss values during training to allow online estimation of the probability that a sample is mislabelled and correct the loss by relying on the network prediction. Expand
Deep Co-Training for Semi-Supervised Image Recognition
TLDR
This paper presents Deep Co-Training, a deep learning based method inspired by the Co- Training framework, which outperforms the previous state-of-the-art methods by a large margin in semi-supervised image recognition. Expand
Decoupled Certainty-Driven Consistency Loss for Semi-supervised Learning
TLDR
A novel Certainty-driven Consistency Loss (CCL) is proposed that exploits the predictive uncertainty in the consistency loss to let the student dynamically learn from reliable targets. Expand
...
1
2
3
4
5
...