Corpus ID: 225067777

Robust Pre-Training by Adversarial Contrastive Learning

@article{Jiang2020RobustPB,
  title={Robust Pre-Training by Adversarial Contrastive Learning},
  author={Ziyu Jiang and Tianlong Chen and Ting Chen and Zhangyang Wang},
  journal={ArXiv},
  year={2020},
  volume={abs/2010.13337}
}
Recent work has shown that, when integrated with adversarial training, self-supervised pre-training can lead to state-of-the-art robustness In this work, we improve robustness-aware self-supervised pre-training by learning representations that are consistent under both data augmentations and adversarial perturbations. Our approach leverages a recent contrastive learning framework, which learns representations by maximizing feature consistency under differently augmented views. This fits… Expand

Figures and Tables from this paper

Adversarial Momentum-Contrastive Pre-Training
TLDR
A novel adversarial momentum-contrastive (AMOC) pre-training approach, which designs two dynamic memory banks to maintain the historical clean and adversarial representations respectively, so as to exploit the discriminative representations that are consistent in a long period. Expand
Understanding and Achieving Efficient Robustness with Adversarial Supervised Contrastive Learning
TLDR
This paper proposes an effective and efficient supervised contrastive learning to achieve model robustness against adversarial attacks and proposes a new sample selection strategy that optimizes the positive/negative sets by removing redundancy and improving correlation with the anchor. Expand
Understanding and Achieving Efficient Robustness with Adversarial Contrastive Learning
TLDR
Experiments show that the Adversarial Supervised Contrastive Learning (ASCL) approach outperforms the state-of-the-art defenses by 2.6% in terms of the robust accuracy, whilst the ASCL with the proposed selection strategy can further gain 1.4% improvement. Expand
KATANA: Simple Post-Training Robustness Using Test Time Augmentations
TLDR
This work proposes a new simple and easy-to-use technique, KATANA, for robustifying an existing pretrained DNN without modifying its weights, and achieves state-of-the-art adversarial robustness on diverse attacks with minimal compromise on the natural images’ classification. Expand
When Does Contrastive Learning Preserve Adversarial Robustness from Pretraining to Finetuning?
TLDR
This paper revisits and advances CL principles through the lens of robustness enhancement and shows that the design of contrastive views matters, and proposes ADVCL, a novel adversarial contrastive pretraining framework that is able to enhance cross-task robustness transferability without loss of model accuracy and finetuning efficiency. Expand
Demystifying Adversarial Training via A Unified Probabilistic Framework
Adversarial Training (AT) is known as an effective approach to enhance the robustness of deep neural networks. Recently researchers notice that robust models with AT have good generative ability andExpand
AugMax: Adversarial Composition of Random Augmentations for Robust Training
TLDR
A disentangled normalization module, termed DuBIN (Dual-Batch-and-Instance Normalization), is designed that disentangles the instance-wise feature heterogeneity arising from AugMax, a stronger form of data augmentation that leads to a significantly augmented input distribution which makes model training more challenging. Expand
Robust Overfitting may be mitigated by properly learned smoothening
TLDR
Two empirical means to inject more learned smoothening during adversarially robust training of deep networks are investigated: one leveraging knowledge distillation and self-training to smooth the logits, the other performing stochastic weight averaging (Izmailov et al., 2018) to Smooth the weights. Expand
Sample Efficient Detection and Classification of Adversarial Attacks via Self-Supervised Embeddings
TLDR
A self-supervised method to detect adversarial attacks and classify them to their respective threat models, based on a linear model operating on the embeddings from a pre-trained self- supervised encoder, called SimCat. Expand
THAT: Two Head Adversarial Training for Improving Robustness at Scale
TLDR
This paper proposes Two Head Adversarial Training (THAT), a two-stream adversarial learning network that is designed to handle the large-scale manyclass ImageNet dataset and demonstrates that the proposed framework outperforms alternative methods under both standard and “free” adversarial training settings. Expand
...
1
2
3
4
...

References

SHOWING 1-10 OF 60 REFERENCES
Adversarial Robustness: From Self-Supervised Pre-Training to Fine-Tuning
TLDR
This work introduces adversarial training into self-supervision, to provide general-purpose robust Pretrained models for the first time, and finds these robust pretrained models can benefit the subsequent fine-tuning in two ways: i) boosting final model robustness; ii) saving the computation cost, if proceeding towards adversarial fine- Tuning. Expand
Are Labels Required for Improving Adversarial Robustness?
TLDR
Theoretically, it is shown that in a simple statistical setting, the sample complexity for learning an adversarially robust model from unlabeled data matches the fully supervised case up to constant factors, and this finding extends as well to the more realistic case where unlabeling data is also uncurated, therefore opening a new avenue for improving adversarial training. Expand
Overfitting in adversarially robust deep learning
TLDR
It is found that overfitting to the training set does in fact harm robust performance to a very large degree in adversarially robust training across multiple datasets (SVHN, CifAR-10, CIFAR-100, and ImageNet) and perturbation models. Expand
Intriguing Properties of Adversarial Training at Scale
TLDR
This paper provides the first rigorous study on diagnosing elements of adversarial training, which reveals two intriguing properties of normalization and the role of network capacity, and finds that enforcing BNs to behave consistently at training and testing can further enhance robustness. Expand
Intriguing properties of adversarial training
TLDR
This paper provides the first rigorous study on diagnosing elements of adversarial training, which reveals two intriguing properties of normalization and the role of network capacity, and finds that enforcing BNs to behave consistently at training and testing can further enhance robustness. Expand
Adversarially Robust Generalization Just Requires More Unlabeled Data
TLDR
It is proved that for a specific Gaussian mixture problem illustrated by [35], adversarially robust generalization can be almost as easy as the standard generalization in supervised learning if a sufficiently large amount of unlabeled data is provided. Expand
Fast is better than free: Revisiting adversarial training
TLDR
It is made the surprising discovery that it is possible to train empirically robust models using a much weaker and cheaper adversary, an approach that was previously believed to be ineffective, rendering the method no more costly than standard training in practice. Expand
Unlabeled Data Improves Adversarial Robustness
TLDR
It is proved that unlabeled data bridges the complexity gap between standard and robust classification: a simple semisupervised learning procedure (self-training) achieves high robust accuracy using the same number of labels required for achieving high standard accuracy. Expand
Stochastic Activation Pruning for Robust Adversarial Defense
TLDR
Stochastic Activation Pruning (SAP) is proposed, a mixed strategy for adversarial defense that prunes a random subset of activations (preferentially pruning those with smaller magnitude) and scales up the survivors to compensate. Expand
Where is the Bottleneck of Adversarial Learning with Unlabeled Data?
TLDR
This paper believes that the quality of pseudo labels is the bottleneck of adversarial learning with unlabeled data, and proposes robust co-training (RCT), which trains two deep networks and encourages two networks diverged by exploiting peer's adversarial examples. Expand
...
1
2
3
4
5
...