• Corpus ID: 51894279

# Data augmentation instead of explicit regularization

@article{HernndezGarca2018DataAI,
title={Data augmentation instead of explicit regularization},
author={Alex Hern{\'a}ndez-Garc{\'i}a and Peter K{\"o}nig},
journal={ArXiv},
year={2018},
volume={abs/1806.03852}
}
• Published 15 February 2018
• Computer Science
• ArXiv
Contrary to most machine learning models, modern deep artificial neural networks typically include multiple components that contribute to regularization. Despite the fact that some (explicit) regularization techniques, such as weight decay and dropout, require costly fine-tuning of sensitive hyperparameters, the interplay between them and other elements that provide implicit regularization is not well understood yet. Shedding light upon these interactions is key to efficiently using…
66 Citations

## Figures, Tables, and Topics from this paper

ON THE CONSISTENCY LOSS FOR LEVERAGING AUGMENTED DATA TO LEARN ROBUST AND INVARIANT REPRESENTATIONS
• 2020
Data augmentation is one of the most popular techniques for improving the robustness of neural networks. In addition to directly training the model with original samples and augmented samples, a
Squared 𝓁2 Norm as Consistency Loss for Leveraging Augmented Data to Learn Robust and Invariant Representations
• Computer Science
ArXiv
• 2020
The analysis suggests the ideal choices of regularization correspond to various assumptions, and shows that the generic approach identified (squared $\ell_2$ norm regularized augmentation) outperforms several recent methods, which are each specially designed for one task and significantly more complicated than the authors', over three different tasks.
Unsupervised Data Augmentation for Consistency Training
• Computer Science, Mathematics
NeurIPS
• 2020
A new perspective on how to effectively noise unlabeled examples is presented and it is argued that the quality of noising, specifically those produced by advanced data augmentation methods, plays a crucial role in semi-supervised learning.
Deep Active Learning with Augmentation-based Consistency Estimation
• Computer Science
ArXiv
• 2020
It is shown that the augmentation-based regularizer can lead to improved performance on the training step of active learning, while that same approach can be effectively combined with the uncertainty measurement metrics proposed so far.
Improving Deep Learning Models via Constraint-Based Domain Knowledge: a Brief Survey
• Computer Science, Mathematics
ArXiv
• 2020
A first survey of the approaches devised to integrate domain knowledge, expressed in the form of constraints, in DL learning models to improve their performance, in particular targeting deep neural networks is presented.
An Empirical Analysis of the Impact of Data Augmentation on Knowledge Distillation
• Computer Science
ArXiv
• 2020
A novel Class-Discrimination metric is presented to quantitatively measure this dichotomy in performance and link it to the discriminative capacity induced by the different strategies on a network's latent space.
Tradeoffs in Data Augmentation: An Empirical Study
• Computer Science
ICLR
• 2021
An empirical study to quantify how data augmentation improves model generalization is conducted and introduces two interpretable and easy-to-compute measures: Affinity and Diversity.
Learning robust visual representations using data augmentation invariance
• Computer Science
2019 Conference on Cognitive Computational Neuroscience
• 2019
The results show that the proposed data augmentation invariance approach is a simple, yet effective and efficient (10 % increase in training time) way of increasing the invariance of the models while obtaining similar categorization performance.
Safe Augmentation: Learning Task-Specific Transformations from Data
• Computer Science
ArXiv
• 2019
This work proposes a simple novel method that can automatically learn task-specific data augmentation techniques called safe augmentations that do not break the data distribution and can be used to improve model performance.
MATIONS FOR DATA AUGMENTATION
Data augmentation (DA) is fundamental against overfitting in large convolutional neural networks, especially with a limited training dataset. In images, DA is usually based on heuristic

## References

SHOWING 1-10 OF 87 REFERENCES
Further Advantages of Data Augmentation on Convolutional Neural Networks
• Computer Science
ICANN
• 2018
Data augmentation is a popular technique largely used to enhance the training of convolutional neural networks. Although many of its benefits are well known by deep learning researchers and
Do deep nets really need weight decay and dropout?
• Computer Science
ArXiv
• 2018
An ablation study is carried out that concludes that weight decay and dropout may not be necessary for object recognition if enough data augmentation is introduced.
Surprising properties of dropout in deep networks
• Computer Science, Mathematics
COLT
• 2017
This work uncovers new properties of dropout, extends the understanding of why dropout succeeds, and lays the foundation for further progress on how dropout is insensitive to various re-scalings of the input features, outputs, and network weights.
Understanding deep learning requires rethinking generalization
• Computer Science
ICLR
• 2017
These experiments establish that state-of-the-art convolutional networks for image classification trained with stochastic gradient methods easily fit a random labeling of the training data, and confirm that simple depth two neural networks already have perfect finite sample expressivity.
Invariance reduces Variance: Understanding Data Augmentation in Deep Learning and Beyond
• Mathematics, Computer Science
ArXiv
• 2019
A theoretical framework to start to shed light on how data augmentation could be used in problems with symmetry where other approaches are prevalent, such as in cryo-electron microscopy (cryo-EM).
Sensitivity and Generalization in Neural Networks: an Empirical Study
• Computer Science, Mathematics
ICLR
• 2018
It is found that trained neural networks are more robust to input perturbations in the vicinity of the training data manifold, as measured by the norm of the input-output Jacobian of the network, and that it correlates well with generalization.
Improved Regularization of Convolutional Neural Networks with Cutout
• Computer Science
ArXiv
• 2017
This paper shows that the simple regularization technique of randomly masking out square regions of input during training, which is called cutout, can be used to improve the robustness and overall performance of convolutional neural networks.