Corpus ID: 195873898

Unsupervised Data Augmentation for Consistency Training

@article{Xie2020UnsupervisedDA,
  title={Unsupervised Data Augmentation for Consistency Training},
  author={Qizhe Xie and Zihang Dai and Eduard H. Hovy and Minh-Thang Luong and Quoc V. Le},
  journal={arXiv: Learning},
  year={2020}
}
Semi-supervised learning lately has shown much promise in improving deep learning models when labeled data is scarce. Common among recent approaches is the use of consistency training on a large amount of unlabeled data to constrain model predictions to be invariant to input noise. In this work, we present a new perspective on how to effectively noise unlabeled examples and argue that the quality of noising, specifically those produced by advanced data augmentation methods, plays a crucial role… Expand
Hybrid Consistency Training with Prototype Adaptation for Few-Shot Learning
TLDR
This work introduces Hybrid Consistency Training to jointly leverage interpolation consistency, including interpolating hidden features, that imposes linear behavior locally and data augmentation consistency that learns robust embeddings against sample variations. Expand
On The Consistency Training for Open-Set Semi-Supervised Learning
TLDR
This work thoroughly study how OOD samples affect DNN training in both low and high-dimensional spaces, where two fundamental SSL methods are considered: Pseudo Labeling (PL) and Data Augmentation based Consistency Training (DACT). Expand
Pseudo-Representation Labeling Semi-Supervised Learning
TLDR
The pseudo-representation labeling is a simple and flexible framework that utilizes pseudo-labeling techniques to iteratively label a small amount of unlabeled data and use them as training data and outperforms the current state-of-the-art semi-supervised learning methods in industrial types of classification problems such as the WM-811K wafer map and the MIT-BIH Arrhythmia dataset. Expand
Dash: Semi-Supervised Learning with Dynamic Thresholding
TLDR
The proposed approach, Dash, enjoys its adaptivity in terms of unlabeled data selection and its theoretical guarantee, and theoretically establishes the convergence rate of Dash from the view of non-convex optimization. Expand
Unsupervised Data Augmentation with Naive Augmentation and without Unlabeled Data
TLDR
This paper re-examine UDA and finds that applying its consistency loss affords meaningful gains without any unlabeled data at all, i.e., in a standard supervised setting, and does not require complex data augmentation to be effective. Expand
Boosting the Performance of Semi-Supervised Learning with Unsupervised Clustering
TLDR
It is shown that ignoring the labels altogether for whole epochs intermittently during training can significantly improve performance in the small sample regime, and the method's efficacy in boosting several state-of-the-art SSL algorithms is demonstrated. Expand
Does Data Augmentation Benefit from Split BatchNorms
TLDR
A recently proposed training paradigm is explored using an auxiliary BatchNorm for the potentially out-of-distribution, strongly augmented images, and this method significantly improves the performance of common image classification benchmarks such as CIFar-10, CIFAR-100, and ImageNet. Expand
Progressive Representative Labeling for Deep Semi-Supervised Learning
TLDR
A graph neural network (GNN) labeler is designed to label only the most representative samples to expand the labeled set and achieves 72.1% top-1 accuracy, surpassing the previous best result by 3.3%, on the challenging ImageNet benchmark with only 10% labeled data. Expand
DP-SSL: Towards Robust Semi-supervised Learning with A Few Labeled Samples
  • Yi Xu, Jiandong Ding, Lu Zhang, Shuigeng Zhou
  • Computer Science
  • ArXiv
  • 2021
TLDR
A new SSL method called DP-SSL is proposed that adopts an innovative data programming (DP) scheme to generate probabilistic labels for unlabeled data and achieves better classification performance on test sets than existing SSL methods, especially when only a small number of labeled samples are available. Expand
GuidedMix-Net: Learning to Improve Pseudo Masks Using Labeled Images as Reference
TLDR
A novel method for semi-supervised semantic segmentation named GuidedMix-Net is proposed, by leveraging labeled information to guide the learning of unlabeled instances, which achieves competitive segmentation accuracy and significantly improves the mIoU by +7% compared to previous state-of-the-art approaches. Expand
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 84 REFERENCES
Realistic Evaluation of Deep Semi-Supervised Learning Algorithms
TLDR
This work creates a unified reimplemention and evaluation platform of various widely-used SSL techniques and finds that the performance of simple baselines which do not use unlabeled data is often underreported, that SSL methods differ in sensitivity to the amount of labeled and unlabeling data, and that performance can degrade substantially when the unlabelED dataset contains out-of-class examples. Expand
Temporal Ensembling for Semi-Supervised Learning
TLDR
Self-ensembling is introduced, where it is shown that this ensemble prediction can be expected to be a better predictor for the unknown labels than the output of the network at the most recent training epoch, and can thus be used as a target for training. Expand
RandAugment: Practical data augmentation with no separate search
TLDR
RandAugment can be used uniformly across different tasks and datasets and works out of the box, matching or surpassing all previous learned augmentation approaches on CIFAR-10, CIFar-100, SVHN, and ImageNet. Expand
Regularization With Stochastic Transformations and Perturbations for Deep Semi-Supervised Learning
TLDR
An unsupervised loss function is proposed that takes advantage of the stochastic nature of these methods and minimizes the difference between the predictions of multiple passes of a training sample through the network. Expand
Unifying semi-supervised and robust learning by mixup
TLDR
It is suggested that semi-supervised learning can outperform robust learning with noisy labels and a training strategy for mixing mixup techniques to learn from bi-quality data effectively is proposed. Expand
There Are Many Consistent Explanations of Unlabeled Data: Why You Should Average
TLDR
It is shown that SGD struggles to converge on the consistency loss and continues to make large steps that lead to changes in predictions on the test data, and proposes to train consistency-based methods with Stochastic Weight Averaging (SWA), a recent approach which averages weights along the trajectory of SGD with a modified learning rate schedule. Expand
Smooth Neighbors on Teacher Graphs for Semi-Supervised Learning
TLDR
A novel method, called Smooth Neighbors on Teacher Graphs (SNTG), which serves as a similarity measure with respect to which the representations of "similar" neighboring points are learned to be smooth on the low-dimensional manifold and achieves state-of-the-art results on semi-supervised learning benchmarks. Expand
Semi-Supervised Sequence Modeling with Cross-View Training
TLDR
Cross-View Training (CVT), a semi-supervised learning algorithm that improves the representations of a Bi-LSTM sentence encoder using a mix of labeled and unlabeled data, is proposed and evaluated, achieving state-of-the-art results. Expand
Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results
TLDR
The recently proposed Temporal Ensembling has achieved state-of-the-art results in several semi-supervised learning benchmarks, but it becomes unwieldy when learning large datasets, so Mean Teacher, a method that averages model weights instead of label predictions, is proposed. Expand
Are Labels Required for Improving Adversarial Robustness?
TLDR
Theoretically, it is shown that in a simple statistical setting, the sample complexity for learning an adversarially robust model from unlabeled data matches the fully supervised case up to constant factors, and this finding extends as well to the more realistic case where unlabeling data is also uncurated, therefore opening a new avenue for improving adversarial training. Expand
...
1
2
3
4
5
...