To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Tasks

@inproceedings{Peters2019ToTO,
  title={To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Tasks},
  author={Matthew E. Peters and Sebastian Ruder and Noah A. Smith},
  booktitle={RepL4NLP@ACL},
  year={2019}
}
While most previous work has focused on different pretraining objectives and architectures for transfer learning, we ask how to best adapt the pretrained model to a given target task. We focus on the two most common forms of adaptation, feature extraction (where the pretrained weights are frozen), and directly fine-tuning the pretrained model. Our empirical results across diverse NLP tasks with two state-of-the-art models show that the relative performance of fine-tuning vs. feature extraction… CONTINUE READING
30
Twitter Mentions

Figures, Tables, and Topics from this paper.

Citations

Publications citing this paper.
SHOWING 1-10 OF 16 CITATIONS

To Tune or Not To Tune? How About the Best of Both Worlds?

VIEW 14 EXCERPTS
CITES BACKGROUND & METHODS
HIGHLY INFLUENCED

Neural Word Decomposition Models for Abusive Language Detection

Sravan Babu Bodapati, Spandana Gella, Yaser Al-Onaizan, Amazon
  • 2019
VIEW 4 EXCERPTS
CITES METHODS & BACKGROUND
HIGHLY INFLUENCED

Improving Few-shot Text Classification via Pretrained Language Representations

Ningyu Zhang, Zhanlin Sun, Songqiu Deng, Jiaoyan Chen, Huajun Chen
  • ArXiv
  • 2019
VIEW 2 EXCERPTS
CITES BACKGROUND & METHODS

References

Publications referenced by this paper.