• Corpus ID: 13748870

Realistic Evaluation of Deep Semi-Supervised Learning Algorithms

@inproceedings{Oliver2018RealisticEO,
  title={Realistic Evaluation of Deep Semi-Supervised Learning Algorithms},
  author={Avital Oliver and Augustus Odena and Colin Raffel and Ekin Dogus Cubuk and Ian J. Goodfellow},
  booktitle={NeurIPS},
  year={2018}
}
Semi-supervised learning (SSL) provides a powerful framework for leveraging unlabeled data when labels are limited or expensive to obtain. [...] Key Method After creating a unified reimplementation of various widely-used SSL techniques, we test them in a suite of experiments designed to address these issues. We find that the performance of simple baselines which do not use unlabeled data is often underreported, that SSL methods differ in sensitivity to the amount of labeled and unlabeled data, and that…Expand
RealMix: Towards Realistic Semi-Supervised Deep Learning Algorithms
TLDR
RealMix demonstrates how SSL can be used in real world situations with limited access to both data and compute and guides further research in SSL with practical applicability in mind.
Matching Distributions via Optimal Transport for Semi-Supervised Learning
TLDR
This work proposes a new approach that adopts an Optimal Transport technique serving as a metric of similarity between discrete empirical probability measures to provide pseudo-labels for the unlabeled data, which can then be used in conjunction with the initial labeled data to train the CNN model in an SSL manner.
Semi-supervised Learning with Contrastive Predicative Coding
TLDR
This paper explores recently developed contrastive predictive coding technique to improve discriminative power of deep learning models when a large portion of labels are absent, and proposes two models, cpc-SSL and a class conditional variant~.
On The Consistency Training for Open-Set Semi-Supervised Learning
TLDR
This work thoroughly study how OOD samples affect DNN training in both low and high-dimensional spaces, where two fundamental SSL methods are considered: Pseudo Labeling (PL) and Data Augmentation based Consistency Training (DACT).
Unsupervised Data Augmentation for Consistency Training
TLDR
A new perspective on how to effectively noise unlabeled examples is presented and it is argued that the quality of noising, specifically those produced by advanced data augmentation methods, plays a crucial role in semi-supervised learning.
A Realistic Evaluation of Semi-Supervised Learning for Fine-Grained Classification
  • Jong-Chyi Su, Zezhou Cheng, Subhransu Maji
  • Computer Science
    2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
  • 2021
TLDR
This work suggests that semi-supervised learning with experts on realistic datasets may require different strategies than those currently prevalent in the literature, and suggests that standard fine-tuning followed by distillation-based self-training is the most robust.
Semi-supervised Active Learning for Semi-supervised Models: Exploit Adversarial Examples with Graph-based Virtual Labels
The performance of computer vision models significantly improves with more labeled data. However, the acquisition of labeled data is limited by the high cost. To mitigate the reliance on large
Safe Deep Semi-Supervised Learning for Unseen-Class Unlabeled Data
TLDR
A simple and effective safe deep SSL method to alleviate the harm caused by class distribution mismatch and it is theoretically guaranteed that its generalization approaches the optimal in the order O( √ d ln(n)/n), even faster than the convergence rate in supervised learning associated with massive parameters.
Distribution Aligning Refinery of Pseudo-label for Imbalanced Semi-supervised Learning
TLDR
A convex optimization problem to softly refine the pseudo-labels generated from the biased model is formulated, and a simple algorithm is developed, named Distribution Aligning Refinery of Pseudo-label (DARP) that solves it provably and efficiently.
An Empirical Study and Analysis on Open-Set Semi-Supervised Learning
TLDR
Style Disturbance is proposed to improve traditional SSL methods on open-set SSL and experimentally shows the approach can achieve state-of-the-art results on various datasets by utilizing OOD samples properly.
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 61 REFERENCES
Regularization With Stochastic Transformations and Perturbations for Deep Semi-Supervised Learning
TLDR
An unsupervised loss function is proposed that takes advantage of the stochastic nature of these methods and minimizes the difference between the predictions of multiple passes of a training sample through the network.
Temporal Ensembling for Semi-Supervised Learning
TLDR
Self-ensembling is introduced, where it is shown that this ensemble prediction can be expected to be a better predictor for the unknown labels than the output of the network at the most recent training epoch, and can thus be used as a target for training.
Semi-supervised Learning with Deep Generative Models
TLDR
It is shown that deep generative models and approximate Bayesian inference exploiting recent advances in variational methods can be used to provide significant improvements, making generative approaches highly competitive for semi-supervised learning.
Mutual exclusivity loss for semi-supervised deep learning
TLDR
An unsupervised regularization term is proposed that explicitly forces the classifier's prediction for multiple classes to be mutually-exclusive and effectively guides the decision boundary to lie on the low density space between the manifolds corresponding to different classes of data.
Neural Simpletrons: Learning in the Limit of Few Labels with Directed Generative Networks
TLDR
It is found that the network studied here can be applied to numbers of few labels where no other system has been reported to operate so far, and can be scaled using standard deep learning tools for parallelized GPU implementation.
Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results
TLDR
The recently proposed Temporal Ensembling has achieved state-of-the-art results in several semi-supervised learning benchmarks, but it becomes unwieldy when learning large datasets, so Mean Teacher, a method that averages model weights instead of label predictions, is proposed.
Semi-supervised Learning by Entropy Minimization
TLDR
This framework, which motivates minimum entropy regularization, enables to incorporate unlabeled data in the standard supervised learning, and includes other approaches to the semi-supervised problem as particular or limiting cases.
Semi-Supervised Learning
TLDR
This first comprehensive overview of semi-supervised learning presents state-of-the-art algorithms, a taxonomy of the field, selected applications, benchmark experiments, and perspectives on ongoing and future research.
Improved Techniques for Training GANs
TLDR
This work focuses on two applications of GANs: semi-supervised learning, and the generation of images that humans find visually realistic, and presents ImageNet samples with unprecedented resolution and shows that the methods enable the model to learn recognizable features of ImageNet classes.
Improved Regularization of Convolutional Neural Networks with Cutout
TLDR
This paper shows that the simple regularization technique of randomly masking out square regions of input during training, which is called cutout, can be used to improve the robustness and overall performance of convolutional neural networks.
...
1
2
3
4
5
...