Corpus ID: 211532277

Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers

@article{Li2020TrainLT,
  title={Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers},
  author={Zhuohan Li and Eric Wallace and Sheng Shen and Kevin Lin and K. Keutzer and D. Klein and J. Gonzalez},
  journal={ArXiv},
  year={2020},
  volume={abs/2002.11794}
}

Zhuohan Li, Eric Wallace, +4 authors J. Gonzalez

Published 2020

Computer Science

ArXiv

Since hardware resources are limited, the objective of training deep learning models is typically to maximize accuracy subject to the time and memory constraints of training and inference. We study the impact of model size in this setting, focusing on Transformer models for NLP tasks that are limited by compute: self-supervised pretraining and high-resource machine translation. We first show that even though smaller Transformer models execute faster per iteration, wider and deeper models… CONTINUE READING

View PDF on arXiv

Save to Library

Create Alert

Cite

Launch Research Feed