# Data augmentation instead of explicit regularization

@article{HernndezGarca2018DataAI, title={Data augmentation instead of explicit regularization}, author={Alex Hern{\'a}ndez-Garc{\'i}a and Peter K{\"o}nig}, journal={ArXiv}, year={2018}, volume={abs/1806.03852} }

Contrary to most machine learning models, modern deep artificial neural networks typically include multiple components that contribute to regularization. Despite the fact that some (explicit) regularization techniques, such as weight decay and dropout, require costly fine-tuning of sensitive hyperparameters, the interplay between them and other elements that provide implicit regularization is not well understood yet. Shedding light upon these interactions is key to efficiently using…

## Figures, Tables, and Topics from this paper

## 66 Citations

ON THE CONSISTENCY LOSS FOR LEVERAGING AUGMENTED DATA TO LEARN ROBUST AND INVARIANT REPRESENTATIONS

- 2020

Data augmentation is one of the most popular techniques for improving the robustness of neural networks. In addition to directly training the model with original samples and augmented samples, a…

Squared 𝓁2 Norm as Consistency Loss for Leveraging Augmented Data to Learn Robust and Invariant Representations

- Computer ScienceArXiv
- 2020

The analysis suggests the ideal choices of regularization correspond to various assumptions, and shows that the generic approach identified (squared $\ell_2$ norm regularized augmentation) outperforms several recent methods, which are each specially designed for one task and significantly more complicated than the authors', over three different tasks.

Unsupervised Data Augmentation for Consistency Training

- Computer Science, MathematicsNeurIPS
- 2020

A new perspective on how to effectively noise unlabeled examples is presented and it is argued that the quality of noising, specifically those produced by advanced data augmentation methods, plays a crucial role in semi-supervised learning.

Deep Active Learning with Augmentation-based Consistency Estimation

- Computer ScienceArXiv
- 2020

It is shown that the augmentation-based regularizer can lead to improved performance on the training step of active learning, while that same approach can be effectively combined with the uncertainty measurement metrics proposed so far.

Improving Deep Learning Models via Constraint-Based Domain Knowledge: a Brief Survey

- Computer Science, MathematicsArXiv
- 2020

A first survey of the approaches devised to integrate domain knowledge, expressed in the form of constraints, in DL learning models to improve their performance, in particular targeting deep neural networks is presented.

An Empirical Analysis of the Impact of Data Augmentation on Knowledge Distillation

- Computer ScienceArXiv
- 2020

A novel Class-Discrimination metric is presented to quantitatively measure this dichotomy in performance and link it to the discriminative capacity induced by the different strategies on a network's latent space.

Tradeoffs in Data Augmentation: An Empirical Study

- Computer ScienceICLR
- 2021

An empirical study to quantify how data augmentation improves model generalization is conducted and introduces two interpretable and easy-to-compute measures: Affinity and Diversity.

Learning robust visual representations using data augmentation invariance

- Computer Science2019 Conference on Cognitive Computational Neuroscience
- 2019

The results show that the proposed data augmentation invariance approach is a simple, yet effective and efficient (10 % increase in training time) way of increasing the invariance of the models while obtaining similar categorization performance.

Safe Augmentation: Learning Task-Specific Transformations from Data

- Computer ScienceArXiv
- 2019

This work proposes a simple novel method that can automatically learn task-specific data augmentation techniques called safe augmentations that do not break the data distribution and can be used to improve model performance.

MATIONS FOR DATA AUGMENTATION

- 2019

Data augmentation (DA) is fundamental against overfitting in large convolutional neural networks, especially with a limited training dataset. In images, DA is usually based on heuristic…

## References

SHOWING 1-10 OF 87 REFERENCES

Further Advantages of Data Augmentation on Convolutional Neural Networks

- Computer ScienceICANN
- 2018

Data augmentation is a popular technique largely used to enhance the training of convolutional neural networks. Although many of its benefits are well known by deep learning researchers and…

Do deep nets really need weight decay and dropout?

- Computer ScienceArXiv
- 2018

An ablation study is carried out that concludes that weight decay and dropout may not be necessary for object recognition if enough data augmentation is introduced.

Surprising properties of dropout in deep networks

- Computer Science, MathematicsCOLT
- 2017

This work uncovers new properties of dropout, extends the understanding of why dropout succeeds, and lays the foundation for further progress on how dropout is insensitive to various re-scalings of the input features, outputs, and network weights.

Understanding deep learning requires rethinking generalization

- Computer ScienceICLR
- 2017

These experiments establish that state-of-the-art convolutional networks for image classification trained with stochastic gradient methods easily fit a random labeling of the training data, and confirm that simple depth two neural networks already have perfect finite sample expressivity.

Invariance reduces Variance: Understanding Data Augmentation in Deep Learning and Beyond

- Mathematics, Computer ScienceArXiv
- 2019

A theoretical framework to start to shed light on how data augmentation could be used in problems with symmetry where other approaches are prevalent, such as in cryo-electron microscopy (cryo-EM).

Sensitivity and Generalization in Neural Networks: an Empirical Study

- Computer Science, MathematicsICLR
- 2018

It is found that trained neural networks are more robust to input perturbations in the vicinity of the training data manifold, as measured by the norm of the input-output Jacobian of the network, and that it correlates well with generalization.

Improved Regularization of Convolutional Neural Networks with Cutout

- Computer ScienceArXiv
- 2017

This paper shows that the simple regularization technique of randomly masking out square regions of input during training, which is called cutout, can be used to improve the robustness and overall performance of convolutional neural networks.

Dropout Training as Adaptive Regularization

- Computer Science, MathematicsNIPS
- 2013

By casting dropout as regularization, this work develops a natural semi-supervised algorithm that uses unlabeled data to create a better adaptive regularizer and consistently boosts the performance of dropout training, improving on state-of-the-art results on the IMDB reviews dataset.

Dropout Training, Data-dependent Regularization, and Generalization Bounds

- Mathematics, Computer ScienceICML
- 2018

By applying this framework to ReLU networks with one hidden layer, a generalization upper bound is derived with no assumptions on the parameter norms or data distribution, with O(1/n) fast rate and adaptivity to geometry of data points being achieved at the same time.

Implicit Regularization in Deep Learning

- Mathematics, Computer ScienceArXiv
- 2017

It is shown that implicit regularization induced by the optimization method is playing a key role in generalization and success of deep learning models, and how different complexity measures can ensure generalization is studied to explain different observed phenomena in deep learning.