On Dropout, Overfitting, and Interaction Effects in Deep Neural Networks
@article{Lengerich2020OnDO, title={On Dropout, Overfitting, and Interaction Effects in Deep Neural Networks}, author={Benjamin J. Lengerich and E. Xing and R. Caruana}, journal={ArXiv}, year={2020}, volume={abs/2007.00823} }
We examine Dropout through the perspective of interactions: learned effects that combine multiple input variables. Given $N$ variables, there are $O(N^2)$ possible pairwise interactions, $O(N^3)$ possible 3-way interactions, etc. We show that Dropout implicitly sets a learning rate for interaction effects that decays exponentially with the size of the interaction, corresponding to a regularizer that balances against the hypothesis space which grows exponentially with number of variables in the… CONTINUE READING
Figures, Tables, and Topics from this paper
One Citation
AccessNet: A Three Layered Visual Based Access Authentication System for Restricted Zones
- 2020 21st International Arab Conference on Information Technology (ACIT)
- 2020
- Highly Influenced
References
SHOWING 1-10 OF 47 REFERENCES
Dropout: a simple way to prevent neural networks from overfitting
- Computer Science
- J. Mach. Learn. Res.
- 2014
- 20,930
- PDF