• Corpus ID: 210164926

# Fast is better than free: Revisiting adversarial training

@article{Wong2020FastIB,
title={Fast is better than free: Revisiting adversarial training},
author={Eric Wong and Leslie Rice and J. Zico Kolter},
journal={ArXiv},
year={2020},
volume={abs/2001.03994}
}
• Published 12 January 2020
• Computer Science
• ArXiv
Adversarial training, a method for learning robust deep networks, is typically assumed to be more expensive than traditional training due to the necessity of constructing adversarial examples via a first-order method like projected gradient decent (PGD). In this paper, we make the surprising discovery that it is possible to train empirically robust models using a much weaker and cheaper adversary, an approach that was previously believed to be ineffective, rendering the method no more costly…
600 Citations

## Figures and Tables from this paper

### Towards Rapid and Robust Adversarial Training with One-Step Attacks.

• Computer Science
• 2020
Noise injection is added to the initial data point of the FGSM attack, which creates a wider variety of adversaries, thus prohibiting overfitting to one particular perturbation bound, and a learnable regularization step prior to the neural network, which is called Pixelwise Noise Injection Layer (PNIL).

### Towards Fast and Robust Adversarial Training for Image Classification

• Computer Science
ACCV
• 2020
This paper utilized a re-constructor to enforce the classifier to learn the important features under perturbations and employed the enhanced FGSM to generate adversarial examples effectively to improve the robustness and efficiency of the adversarial training.

### 𝓁∞-Robustness and Beyond: Unleashing Efficient Adversarial Training

• Computer Science
ECCV
• 2022
By leveraging the theory of coreset selection, it is shown how selecting a small subset of training data provides a general, more principled approach toward reducing the time complexity of robust training.

### RUSH: Robust Contrastive Learning via Randomized Smoothing

• Computer Science
ArXiv
• 2022
It is shown that contrastive pre-training has an interesting yet implicit connection with robustness, and such natural robustness in the pre-trained representation enables this work to design a powerful robust algorithm against adversarial attacks, R USH, that boosts both standard accuracy and robust accuracy, and reduces training costs as compared with adversarial training.

### Fast and Stable Adversarial Training through Noise Injection

• Computer Science
ArXiv
• 2020
Noise injection is added to the initial data point of the FGSM attack, which creates a wider variety of stronger adversaries, and a learnable regularization step prior to the neural network called Stochastic Augmentation Layer (SAL).

### Boosting Fast Adversarial Training With Learnable Adversarial Initialization

• Computer Science
IEEE Transactions on Image Processing
• 2022
Experimental evaluations on four benchmark databases demonstrate the superiority of the proposed method over state-of-the-art fast AT methods, as well as comparable robustness to advanced multi-step AT methods.

### Overfitting in adversarially robust deep learning

• Computer Science
ICML
• 2020
It is found that overfitting to the training set does in fact harm robust performance to a very large degree in adversarially robust training across multiple datasets (SVHN, CifAR-10, CIFAR-100, and ImageNet) and perturbation models.

### Initializing Perturbations in Multiple Directions for Fast Adversarial Training

• Computer Science
ArXiv
• 2020
The Diversified Initialized Perturbations Adversarial Training (DIP-FAT) which involves seeking the initialization of the perturbation via enlarging the output distances of the target model in a random directions and achieving the best banlance among clean-data, perturbed-data and efficiency.

### $\ell_\infty$-Robustness and Beyond: Unleashing Efficient Adversarial Training

• Computer Science
• 2021
By leveraging the theory of coreset selection, it is shown how selecting a small subset of training data provides a general, more principled approach toward reducing the time complexity of robust training.

### Adversarial Coreset Selection for Efficient Robust Training

• Computer Science
ArXiv
• 2022
This work shows how selecting a small subset of training data provides a principled approach to reducing the time complexity of robust training, and provides convergence guarantees for adversarial coreset selection.

## References

SHOWING 1-10 OF 52 REFERENCES

### Adversarial Training for Free!

• Computer Science
NeurIPS
• 2019
This work presents an algorithm that eliminates the overhead cost of generating adversarial examples by recycling the gradient information computed when updating model parameters, and achieves comparable robustness to PGD adversarial training on the CIFAR-10 and CIFar-100 datasets at negligible additional cost compared to natural training.

### Bilateral Adversarial Training: Towards Fast Training of More Robust Models Against Adversarial Attacks

• Jianyu Wang
• Computer Science
2019 IEEE/CVF International Conference on Computer Vision (ICCV)
• 2019
The experiment on the very (computationally) challenging ImageNet dataset further demonstrates the effectiveness of the fast method, which shows that random start and the most confusing target attack effectively prevent the label leaking and gradient masking problem.

### You Only Propagate Once: Painless Adversarial Training Using Maximal Principle

• Computer Science
• 2019
This work fully exploits structure of deep neural networks and proposes a novel strategy to decouple the adversary update with the gradient back propagation, which avoids forward and backward propagating the data too many times in one iteration, and restricts core descent directions computation to the first layer of the network, thus speeding up every iteration significantly.

### You Only Propagate Once: Accelerating Adversarial Training via Maximal Principle

• Computer Science
NeurIPS
• 2019
It is shown that adversarial training can be cast as a discrete time differential game, and the proposed algorithm YOPO (You Only Propagate Once) can achieve comparable defense accuracy with approximately 1/5 ~ 1/4 GPU time of the projected gradient descent (PGD) algorithm.

### Scaling provable adversarial defenses

• Computer Science
NeurIPS
• 2018
This paper presents a technique for extending these training procedures to much more general networks, with skip connections and general nonlinearities, and shows how to further improve robust error through cascade models.

### Provable defenses against adversarial examples via the convex outer adversarial polytope

• Computer Science
ICML
• 2018
A method to learn deep ReLU-based classifiers that are provably robust against norm-bounded adversarial perturbations, and it is shown that the dual problem to this linear program can be represented itself as a deep network similar to the backpropagation network, leading to very efficient optimization approaches that produce guaranteed bounds on the robust loss.

### Adversarial Robustness Against the Union of Multiple Perturbation Models

• Computer Science
ICML
• 2020
This work shows that it is indeed possible to adversarially train a robust model against a union of norm-bounded attacks, by using a natural generalization of the standard PGD-based procedure for adversarial training to multiple threat models.

### Ensemble Adversarial Training: Attacks and Defenses

• Computer Science
ICLR
• 2018
This work finds that adversarial training remains vulnerable to black-box attacks, where perturbations computed on undefended models are transferred to a powerful novel single-step attack that escapes the non-smooth vicinity of the input data via a small random step.

### On the Effectiveness of Interval Bound Propagation for Training Verifiably Robust Models

• Computer Science
ArXiv
• 2018
This work shows how a simple bounding technique, interval bound propagation (IBP), can be exploited to train large provably robust neural networks that beat the state-of-the-art in verified accuracy and allows the largest model to be verified beyond vacuous bounds on a downscaled version of ImageNet.

### Towards Deep Learning Models Resistant to Adversarial Attacks

• Computer Science
ICLR
• 2018
This work studies the adversarial robustness of neural networks through the lens of robust optimization, and suggests the notion of security against a first-order adversary as a natural and broad security guarantee.