Don't Stop Pretraining: Adapt Language Models to Domains and Tasks

@article{Gururangan2020DontSP,
  title={Don't Stop Pretraining: Adapt Language Models to Domains and Tasks},
  author={Suchin Gururangan and Ana Marasovi{\'c} and Swabha Swayamdipta and Kyle Lo and Iz Beltagy and Doug Downey and Noah A. Smith},
  journal={ArXiv},
  year={2020},
  volume={abs/2004.10964}
}
Language models pretrained on text from a wide variety of sources form the foundation of today's NLP. In light of the success of these broad-coverage models, we investigate whether it is still helpful to tailor a pretrained model to the domain of a target task. We present a study across four domains (biomedical and computer science publications, news, and reviews) and eight classification tasks, showing that a second phase of pretraining in-domain (domain-adaptive pretraining) leads to… Expand
View on ACL
arxiv.org
291 Citations
Highly Influential Citations
51
Background Citations
156
Methods Citations
107
Results Citations
17

291 Citations

Citation Type

Citation Type

Towards a Robust Question Answering System through Domain-adaptive Pretraining and Data Augmentation
  • Feng Chen
  • 2021
An Empirical Investigation towards Efficient Multi-Domain Language Model Pre-training
Robust QA with Task-Adaptive Pretraining
  • Jeong-O Jeong
  • 2021
Multi-Phase Adaptive Pretraining on DistilBERT for Compact Domain Adaptation
Domain and Task Adaptive Pretraining for Language Models
Task-specific Objectives of Pre-trained Language Models for Dialogue Adaptation
How to Adapt Your Pretrained Multilingual Model to 1600 Languages
Pretrained Language Models for Biomedical and Clinical Tasks: Understanding and Extending the State-of-the-Art
Predictions For Pre-training Language Models
NukeLM: Pre-Trained and Fine-Tuned Language Models for the Nuclear and Energy Domains
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 75 REFERENCES
Unsupervised Domain Adaptation of Contextualized Embeddings for Sequence Labeling
Unsupervised Domain Clusters in Pretrained Language Models
SciBERT: A Pretrained Language Model for Scientific Text
Improving Language Understanding by Generative Pre-Training
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Tasks
What to do about non-standard (or non-canonical) language in NLP
An Embarrassingly Simple Approach for Transfer Learning from Pretrained Language Models
Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Tasks
Learning and Evaluating General Linguistic Intelligence
...
1
2
3
4
5
...