• Corpus ID: 219530634

The Penalty Imposed by Ablated Data Augmentation

@article{Liu2020ThePI,
  title={The Penalty Imposed by Ablated Data Augmentation},
  author={Frederick Liu and Amir Najmi and Mukund Sundararajan},
  journal={ArXiv},
  year={2020},
  volume={abs/2006.04769}
}
There is a set of data augmentation techniques that ablate parts of the input at random. These include input dropout, cutout, and random erasing. We term these techniques ablated data augmentation. Though these techniques seems similar in spirit and have shown success in improving model performance in a variety of domains, we do not yet have a mathematical understanding of the differences between these techniques like we do for other regularization techniques like L1 or L2. First, we study a… 

Figures from this paper

How Data Augmentation affects Optimization for Linear Regression
TLDR
This work finds joint schedules for learning rate and data augmentation scheme under which augmented gradient descent provably converges and characterize the resulting minimum, and interprets augmented (S)GD as a stochastic optimization method for a time-varying sequence of proxy losses.
Data augmentation as stochastic optimization
TLDR
A time-varying Monro-Robbins theorem with rates of convergence is proved which gives conditions on the learning rate and augmentation schedule under which augmented gradient descent converges.
O ct 2 02 0 Preprint D ATA AUGMENTATION AS STOCHASTIC OPTIMIZATION
  • 2020

References

SHOWING 1-10 OF 24 REFERENCES
AutoAugment: Learning Augmentation Strategies From Data
TLDR
This paper describes a simple procedure called AutoAugment to automatically search for improved data augmentation policies, which achieves state-of-the-art accuracy on CIFAR-10, CIFar-100, SVHN, and ImageNet (without additional data).
Improved Regularization of Convolutional Neural Networks with Cutout
TLDR
This paper shows that the simple regularization technique of randomly masking out square regions of input during training, which is called cutout, can be used to improve the robustness and overall performance of convolutional neural networks.
Smart Augmentation Learning an Optimal Data Augmentation Strategy
TLDR
Smart augmentation works, by creating a network that learns how to generate augmented data during the training process of a target network in a way that reduces that networks loss, and allows to learn augmentations that minimize the error of that network.
Random Erasing Data Augmentation
TLDR
In training, Random Erasing randomly selects a rectangle region in an image and erases its pixels with random values and yields consistent improvement over strong baselines in image classification, object detection and person re-identification.
SwitchOut: an Efficient Data Augmentation Algorithm for Neural Machine Translation
TLDR
An extremely simple data augmentation strategy for NMT: randomly replacing words in both the source sentence and the target sentence with other random words from their corresponding vocabularies is proposed.
Training with Noise is Equivalent to Tikhonov Regularization
TLDR
This paper shows that for the purposes of network training, the regularization term can be reduced to a positive semi-definite form that involves only first derivatives of the network mapping.
Learning Important Features Through Propagating Activation Differences
TLDR
DeepLIFT (Deep Learning Important FeaTures), a method for decomposing the output prediction of a neural network on a specific input by backpropagating the contributions of all neurons in the network to every feature of the input, is presented.
Dropout: a simple way to prevent neural networks from overfitting
TLDR
It is shown that dropout improves the performance of neural networks on supervised learning tasks in vision, speech recognition, document classification and computational biology, obtaining state-of-the-art results on many benchmark data sets.
Striving for Simplicity: The All Convolutional Net
TLDR
It is found that max-pooling can simply be replaced by a convolutional layer with increased stride without loss in accuracy on several image recognition benchmarks.
A Unified Approach to Interpreting Model Predictions
TLDR
A unified framework for interpreting predictions, SHAP (SHapley Additive exPlanations), which unifies six existing methods and presents new methods that show improved computational performance and/or better consistency with human intuition than previous approaches.
...
1
2
3
...