• Corpus ID: 56657912

Benchmarking Neural Network Robustness to Common Corruptions and Perturbations

@article{Hendrycks2019BenchmarkingNN,
  title={Benchmarking Neural Network Robustness to Common Corruptions and Perturbations},
  author={Dan Hendrycks and Thomas G. Dietterich},
  journal={ArXiv},
  year={2019},
  volume={abs/1903.12261}
}
In this paper we establish rigorous benchmarks for image classifier robustness. [] Key Result Together our benchmarks may aid future work toward networks that robustly generalize.
View PDF on arXiv
1,141 Citations
Highly Influential Citations
288
Background Citations
642
Methods Citations
542
Results Citations
24

Figures and Tables from this paper

1,141 Citations

Citation Type

Citation Type

Improving Robustness of DNNs against Common Corruptions via Gaussian Adversarial Training
TLDR
This work proposes to train neural networks with adversarial examples where the perturbations are Gaussian-distributed, and shows that the proposed GAT can improve neural networks’ robustness to noise corruptions more than other baseline methods.
Are Adversarial Robustness and Common Perturbation Robustness Independent Attributes ?
TLDR
It is shown that increasing the robustness to carefully selected common perturbations, can make neural networks more robust to unseen common perturgations and it is proved that adversarial robustness and robustness of neural networks to common perturations are independent.
Improving Corruption and Adversarial Robustness by Enhancing Weak Subnets
TLDR
It is shown that the proposed novel robust training method, EWS, greatly improves the robustness against corrupted images as well as the accuracy on clean data, and is complementary to many state-of-the-art data augmentation approaches.
Defending Against Image Corruptions Through Adversarial Augmentations
TLDR
This work proposes AdversarialAugment, a technique which optimizes the parameters of image-to-image models to generate adversarially corrupted augmented images and improves worst-case performance against `p-norm bounded perturbations on both CIFAR-10 and IMAGENET.
How Does Frequency Bias Affect the Robustness of Neural Image Classifiers against Common Corruption and Adversarial Perturbations?
TLDR
This work proposes Jacobian frequency regularization for models’ Jacobians to have a larger ratio of low-frequency components and shows that biasing classifiers towards low (high)-frequency components can bring performance gain against high (low)-frequency corruption and adversarial perturbation, albeit with a tradeoff in performance for low ( high-frequency corruption.
NoisyMix: Boosting Model Robustness to Common Corruptions
TLDR
NoisyMix is a novel training scheme that promotes stability as well as leverages noisy augmentations in input and feature space to improve both model robustness and in-domain accuracy and provides theory to understand implicit regularization and robustness of NoisyMix.
DEFENDING AGAINST IMAGE CORRUPTIONS THROUGH ADVERSARIAL AUGMENTATIONS
TLDR
Classifiers trained using the proposed AdversarialAugment method in conjunction with prior methods improve upon the state-of-the-art on common image corruption benchmarks conducted in expectation on CIFAR-10-C and also improve worst-case performance against `p-norm bounded perturbations on both CIFar-10 and IMAGENET.
Adversarial amplitude swap towards robust image classifiers
TLDR
Results showed that adversarial amplitude images can serve as a better data augmentation method to achieve general robustness against both common corruptions and adversarial perturbations even in an adversarial training setup, and contributed to the understanding and the training of truly robust classifiers.
Improving robustness against common corruptions by covariate shift adaptation
TLDR
It is argued that results with adapted statistics should be included whenever reporting scores in corruption benchmarks and other out-of-distribution generalization settings, and 32 samples are sufficient to improve the current state of the art for a ResNet-50 architecture.
ImageNet-Patch: A Dataset for Benchmarking Machine Learning Robustness against Adversarial Patches
TLDR
This work proposes ImageNet-Patch, a dataset to benchmark machinelearning models against adversarial patches, a set of patches, optimized to generalize across different models, and readily applicable to ImageNet data after preprocessing them with affine transformations, enabling an approximate yet faster robustness evaluation.
...
...

References

SHOWING 1-10 OF 75 REFERENCES
Measuring Neural Net Robustness with Constraints
TLDR
This work proposes metrics for measuring the robustness of a neural net and devise a novel algorithm for approximating these metrics based on an encoding of robustness as a linear program and generates more informative estimates of robusts metrics compared to estimates based on existing algorithms.
On Detecting Adversarial Perturbations
TLDR
It is shown empirically that adversarial perturbations can be detected surprisingly well even though they are quasi-imperceptible to humans.
Adversarially Robust Generalization Requires More Data
TLDR
It is shown that already in a simple natural data model, the sample complexity of robust learning can be significantly larger than that of "standard" learning.
Towards Evaluating the Robustness of Neural Networks
TLDR
It is demonstrated that defensive distillation does not significantly increase the robustness of neural networks, and three new attack algorithms are introduced that are successful on both distilled and undistilled neural networks with 100% probability are introduced.
Using Trusted Data to Train Deep Networks on Labels Corrupted by Severe Noise
TLDR
It is demonstrated that robustness to label noise up to severe strengths can be achieved by using a set of trusted data with clean labels, and a loss correction that utilizes trusted examples in a data-efficient manner to mitigate the effects of label noise on deep neural network classifiers is proposed.
Improving the Robustness of Deep Neural Networks via Stability Training
TLDR
This paper presents a general stability training method to stabilize deep networks against small input distortions that result from various types of common image processing, such as compression, rescaling, and cropping.
Ground-Truth Adversarial Examples
TLDR
Ground truths are constructed: adversarial examples with a provably-minimal distance from a given input point that can serve to assess the effectiveness of attack techniques and also of defense techniques, by computing the distance to the ground truths before and after the defense is applied, and measuring the improvement.
Robust Physical-World Attacks on Deep Learning Models
TLDR
This work proposes a general attack algorithm,Robust Physical Perturbations (RP2), to generate robust visual adversarial perturbations under different physical conditions and shows that adversarial examples generated using RP2 achieve high targeted misclassification rates against standard-architecture road sign classifiers in the physical world under various environmental conditions, including viewpoints.
Adversarial Examples Are Not Easily Detected: Bypassing Ten Detection Methods
TLDR
It is concluded that adversarialExamples are significantly harder to detect than previously appreciated, and the properties believed to be intrinsic to adversarial examples are in fact not.
Distillation as a Defense to Adversarial Perturbations Against Deep Neural Networks
TLDR
The study shows that defensive distillation can reduce effectiveness of sample creation from 95% to less than 0.5% on a studied DNN, and analytically investigates the generalizability and robustness properties granted by the use of defensive Distillation when training DNNs.
...
...