• Corpus ID: 195750576

Using Self-Supervised Learning Can Improve Model Robustness and Uncertainty

@inproceedings{Hendrycks2019UsingSL,
  title={Using Self-Supervised Learning Can Improve Model Robustness and Uncertainty},
  author={Dan Hendrycks and Mantas Mazeika and Saurav Kadavath and Dawn Xiaodong Song},
  booktitle={NeurIPS},
  year={2019}
}
Self-supervision provides effective representations for downstream tasks without requiring labels. However, existing approaches lag behind fully supervised training and are often not thought beneficial beyond obviating the need for annotations. We find that self-supervision can benefit robustness in a variety of ways, including robustness to adversarial examples, label corruption, and common input corruptions. Additionally, self-supervision greatly benefits out-of-distribution detection on… 
Improving Model Robustness via Automatically Incorporating Self-supervision Tasks
Robust deep learning models are of great practical importance. Hendrycks et al. [12] have recently shown that one can train a robust deep learning model by incorporating a self-supervision loss
On visual self-supervision and its effect on model robustness
TLDR
An empirical study to understand more precisely in what way can self-supervised learning as a pre-training technique or part of adversarial training affects model robustness to l2 and l∞ adversarial perturbations and natural image corruptions.
Evaluating the Robustness of Self-Supervised Learning in Medical Imaging
TLDR
It is demonstrated that networks trained via self-super supervised learning have superior robustness and generalizability compared to fully-supervised learning in the context of medical imaging.
Adversarial Robustness: From Self-Supervised Pre-Training to Fine-Tuning
TLDR
This work introduces adversarial training into self-supervision, to provide general-purpose robust Pretrained models for the first time, and finds these robust pretrained models can benefit the subsequent fine-tuning in two ways: i) boosting final model robustness; ii) saving the computation cost, if proceeding towards adversarial fine- Tuning.
Data Transformation Insights in Self-supervision with Clustering Tasks
TLDR
It is shown theoretically and empirically that certain set of transformations are helpful in convergence of self-supervised clustering and faster convergence rate with valid transformations for convex as well as certain family of non-convex objectives along with the proof of convergence to the original set of optima.
Improving out-of-distribution generalization via multi-task self-supervised pretraining
TLDR
It is shown that features obtained using self-supervised learning are comparable to, or better than, supervised learning for domain generalization in computer vision, and that multi-task learning of compatible pretext tasks improvesdomain generalization performance as compared to training individual tasks alone.
Self-Supervised Dynamic Networks for Covariate Shift Robustness
TLDR
This work proposes Self-Supervised Dynamic Networks (SSDN): an input-dependent mechanism, inspired by dynamic networks, that allows a self-supervised network to predict the weights of the main network, and thus directly handle covariate shifts at test-time.
Rethinking Data Augmentation: Self-Supervision and Self-Distillation
TLDR
This work proposes a simple yet effective idea of learning the joint distribution of the original and self-supervised labels of augmented samples, and enables an aggregated inference combining the predictions from different augmented samples for improving the performance.
Self-Supervised Learning for Generalizable Out-of-Distribution Detection
TLDR
This work proposes a new technique relying on self-supervision for generalizable out-of-distribution (OOD) feature learning and rejecting those samples at the inference time, which does not need to pre-know the distribution of targeted OOD samples and incur no extra overheads compared to other methods.
Self-Supervised Representation Learning: Introduction, Advances and Challenges
TLDR
This article introduces this vibrant area including key concepts, the four main families of approach and associated state of the art, and how self-supervised methods are applied to diverse modalities of data, including workflows, representation transferability, and compute cost.
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 46 REFERENCES
Using Pre-Training Can Improve Model Robustness and Uncertainty
TLDR
It is shown that although pre-training may not improve performance on traditional classification metrics, it improves model robustness and uncertainty estimates and surpasses the state-of-the-art in adversarial robustness.
Adversarially Robust Generalization Requires More Data
TLDR
It is shown that already in a simple natural data model, the sample complexity of robust learning can be significantly larger than that of "standard" learning.
Using Trusted Data to Train Deep Networks on Labels Corrupted by Severe Noise
TLDR
It is demonstrated that robustness to label noise up to severe strengths can be achieved by using a set of trusted data with clean labels, and a loss correction that utilizes trusted examples in a data-efficient manner to mitigate the effects of label noise on deep neural network classifiers is proposed.
Learning deep representations by mutual information estimation and maximization
TLDR
It is shown that structure matters: incorporating knowledge about locality in the input into the objective can significantly improve a representation’s suitability for downstream tasks and is an important step towards flexible formulations of representation learning objectives for specific end-goals.
Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels
TLDR
A theoretically grounded set of noise-robust loss functions that can be seen as a generalization of MAE and CCE are presented and can be readily applied with any existing DNN architecture and algorithm, while yielding good performance in a wide range of noisy label scenarios.
Representation Learning with Contrastive Predictive Coding
TLDR
This work proposes a universal unsupervised learning approach to extract useful representations from high-dimensional data, which it calls Contrastive Predictive Coding, and demonstrates that the approach is able to learn useful representations achieving strong performance on four distinct domains: speech, images, text and reinforcement learning in 3D environments.
Training Confidence-calibrated Classifiers for Detecting Out-of-Distribution Samples
TLDR
A novel training method for classifiers so that such inference algorithms can work better, and it is demonstrated its effectiveness using deep convolutional neural networks on various popular image datasets.
LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop
TLDR
This work proposes to amplify human effort through a partially automated labeling scheme, leveraging deep learning with humans in the loop, and constructs a new image dataset, LSUN, which contains around one million labeled images for each of 10 scene categories and 20 object categories.
Training Convolutional Networks with Noisy Labels
TLDR
An extra noise layer is introduced into the network which adapts the network outputs to match the noisy label distribution and can be estimated as part of the training process and involve simple modifications to current training infrastructures for deep networks.
Making Deep Neural Networks Robust to Label Noise: A Loss Correction Approach
TLDR
It is proved that, when ReLU is the only non-linearity, the loss curvature is immune to class-dependent label noise, and it is shown how one can estimate these probabilities, adapting a recent technique for noise estimation to the multi-class setting, and providing an end-to-end framework.
...
1
2
3
4
5
...