Corpus ID: 220302263

On Dropout, Overfitting, and Interaction Effects in Deep Neural Networks

@article{Lengerich2020OnDO,
  title={On Dropout, Overfitting, and Interaction Effects in Deep Neural Networks},
  author={Benjamin J. Lengerich and E. Xing and R. Caruana},
  journal={ArXiv},
  year={2020},
  volume={abs/2007.00823}
}

Benjamin J. Lengerich, E. Xing, R. Caruana

Published 2020

Computer Science, Mathematics

ArXiv

We examine Dropout through the perspective of interactions: learned effects that combine multiple input variables. Given $N$ variables, there are $O(N^2)$ possible pairwise interactions, $O(N^3)$ possible 3-way interactions, etc. We show that Dropout implicitly sets a learning rate for interaction effects that decays exponentially with the size of the interaction, corresponding to a regularizer that balances against the hypothesis space which grows exponentially with number of variables in the… CONTINUE READING

View PDF on arXiv

Save to Library

Create Alert

Cite

Launch Research Feed