Corpus ID: 43922508

Pushing the bounds of dropout

@article{Melis2018PushingTB,
  title={Pushing the bounds of dropout},
  author={G{\'a}bor Melis and Charles Blundell and Tom{\'a}s Kocisk{\'y} and K. Hermann and Chris Dyer and P. Blunsom},
  journal={ArXiv},
  year={2018},
  volume={abs/1805.09208}
}

Gábor Melis, Charles Blundell, +3 authors P. Blunsom

Published 2018

Computer Science, Mathematics

ArXiv

We show that dropout training is best understood as performing MAP estimation concurrently for a family of conditional models whose objectives are themselves lower bounded by the original dropout objective. This discovery allows us to pick any model from this family after training, which leads to a substantial improvement on regularisation-heavy language modelling. The family includes models that compute a power mean over the sampled dropout masks, and their less stochastic subvariants with… CONTINUE READING

View PDF on arXiv

Save to Library

Create Alert

Cite

Launch Research Feed