• DOI:10.18653/v1/2020.acl-main.740
Corpus ID: 216080466

Don't Stop Pretraining: Adapt Language Models to Domains and Tasks

@article{Gururangan2020DontSP,
  title={Don't Stop Pretraining: Adapt Language Models to Domains and Tasks},
  author={Suchin Gururangan and Ana Marasovi{\'c} and Swabha Swayamdipta and Kyle Lo and Iz Beltagy and Doug Downey and Noah A. Smith},
  journal={ArXiv},
  year={2020},
  volume={abs/2004.10964}
}

• Suchin Gururangan, Ana Marasović, +4 authors Noah A. Smith
• Published 2020
• Computer Science
• ArXiv

Language models pretrained on text from a wide variety of sources form the foundation of today's NLP. In light of the success of these broad-coverage models, we investigate whether it is still helpful to tailor a pretrained model to the domain of a target task. We present a study across four domains (biomedical and computer science publications, news, and reviews) and eight classification tasks, showing that a second phase of pretraining in-domain (domain-adaptive pretraining) leads to… Expand

View on ACL

arxiv.org

Save to Library

Create Alert

Cite

Launch Research Feed