Removing Batch Normalization Boosts Adversarial Training

  title={Removing Batch Normalization Boosts Adversarial Training},
  author={Haotao Wang and Aston Zhang and Shuai Zheng and Xingjian Shi and Mu Li and Zhangyang Wang},
  booktitle={International Conference on Machine Learning},
Adversarial training (AT) defends deep neural networks against adversarial attacks. One challenge that limits its practical application is the performance degradation on clean samples. A major bottleneck identified by previous works is the widely used batch normalization (BN), which struggles to model the different statistics of clean and adversarial training samples in AT. Although the dominant approach is to extend BN to capture this mixture of distribution, we propose to completely elimi-nate… 

Revisiting adapters with adversarial training

It is shown that using the classification token of a Vision Transformer (VIT) as an adapter is enough to match the classification performance of dual normalization layers, while using significantly less additional parameters, and that training with adapters enables model soups through linear combinations of the clean and adversarial tokens.


  • Computer Science
  • 2022
This work provides a counterexample to a corollary of Danskin’s Theorem presented in the seminal paper of Madry et al. (2018) which states that a solution of the inner maximization problem can yield a descent direction for the adversarially robust loss.

Dynamical Isometry for Residual Networks

A random initialization scheme, R ISOTTO, is proposed that achieves perfect dynamical isometry for residual networks with ReLU activation functions even for depth and width and outperforms initialization schemes proposed to make Batch Normalization obsolete, including Fixup and SkipInit, and facilitates stable training.



Intriguing properties of adversarial training

This paper provides the first rigorous study on diagnosing elements of adversarial training, which reveals two intriguing properties of normalization and the role of network capacity, and finds that enforcing BNs to behave consistently at training and testing can further enhance robustness.

Towards an Adversarially Robust Normalization Approach

This paper investigates how BatchNorm causes adversarial vulnerability and proposed new normalization that is robust to adversarial attacks, and proposes Robust Normalization (RobustNorm); an adversarially robust version of Batch Norm.

How benign is benign overfitting?

This work identifies label noise as one of the causes for adversarial vulnerability, and provides theoretical and empirical evidence in support of this and conjecture that in part the need for complex decision boundaries arises from sub-optimal representation learning.

Towards Deep Learning Models Resistant to Adversarial Attacks

This work studies the adversarial robustness of neural networks through the lens of robust optimization, and suggests the notion of security against a first-order adversary as a natural and broad security guarantee.

Once-for-All Adversarial Training: In-Situ Tradeoff between Robustness and Accuracy for Free

This paper proposes a Once-for-all Adversarial Training framework, built on an innovative model-conditional training framework, with a controlling hyper-parameter as the input, that allows for the joint trade-off among accuracy, robustness and runtime efficiency.

Fast is better than free: Revisiting adversarial training

It is made the surprising discovery that it is possible to train empirically robust models using a much weaker and cheaper adversary, an approach that was previously believed to be ineffective, rendering the method no more costly than standard training in practice.

Adversarial Training for Free!

This work presents an algorithm that eliminates the overhead cost of generating adversarial examples by recycling the gradient information computed when updating model parameters, and achieves comparable robustness to PGD adversarial training on the CIFAR-10 and CIFar-100 datasets at negligible additional cost compared to natural training.

Attacks Which Do Not Kill Training Make Adversarial Learning Stronger

A novel approach of friendly adversarial training (FAT) is proposed: rather than employing most adversarial data maximizing the loss, it is proposed to search for least adversarial Data Minimizing the Loss, among the adversarialData that are confidently misclassified.

Adversarial Examples Improve Image Recognition

This work proposes AdvProp, an enhanced adversarial training scheme which treats adversarial examples as additional examples, to prevent overfitting, and shows that AdvProp improves a wide range of models on various image recognition tasks and performs better when the models are bigger.

Improving Adversarial Robustness Requires Revisiting Misclassified Examples

This paper proposes a new defense algorithm called MART, which explicitly differentiates the misclassified and correctly classified examples during the training, and shows that MART and its variant could significantly improve the state-of-the-art adversarial robustness.