Don't Stop Pretraining: Adapt Language Models to Domains and Tasks

@article{Gururangan2020DontSP,
  title={Don't Stop Pretraining: Adapt Language Models to Domains and Tasks},
  author={Suchin Gururangan and Ana Marasovi{\'c} and Swabha Swayamdipta and Kyle Lo and Iz Beltagy and Doug Downey and Noah A. Smith},
  journal={ArXiv},
  year={2020},
  volume={abs/2004.10964}
}
Language models pretrained on text from a wide variety of sources form the foundation of today's NLP. In light of the success of these broad-coverage models, we investigate whether it is still helpful to tailor a pretrained model to the domain of a target task. We present a study across four domains (biomedical and computer science publications, news, and reviews) and eight classification tasks, showing that a second phase of pretraining in-domain (domain-adaptive pretraining) leads to… Expand
291 Citations
Robust QA with Task-Adaptive Pretraining
  • Jeong-O Jeong
  • 2021
Multi-Phase Adaptive Pretraining on DistilBERT for Compact Domain Adaptation
Domain and Task Adaptive Pretraining for Language Models
Task-specific Objectives of Pre-trained Language Models for Dialogue Adaptation
How to Adapt Your Pretrained Multilingual Model to 1600 Languages
Pretrained Language Models for Biomedical and Clinical Tasks: Understanding and Extending the State-of-the-Art
Predictions For Pre-training Language Models
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 75 REFERENCES
Unsupervised Domain Adaptation of Contextualized Embeddings for Sequence Labeling
SciBERT: A Pretrained Language Model for Scientific Text
Improving Language Understanding by Generative Pre-Training
To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Tasks
Learning and Evaluating General Linguistic Intelligence
...
1
2
3
4
5
...