Corpus ID: 202660670

Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism

@article{Shoeybi2019MegatronLMTM,
  title={Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism},
  author={M. Shoeybi and M. Patwary and R. Puri and P. LeGresley and J. Casper and Bryan Catanzaro},
  journal={ArXiv},
  year={2019},
  volume={abs/1909.08053}
}

M. Shoeybi, M. Patwary, +3 authors Bryan Catanzaro

Published 2019

Computer Science

ArXiv

Recent work in language modeling demonstrates that training large transformer models advances the state of the art in Natural Language Processing applications. However, very large models can be quite difficult to train due to memory constraints. In this work, we present our techniques for training very large transformer models and implement a simple, efficient intra-layer model parallel approach that enables training transformer models with billions of parameters. Our approach does not require… CONTINUE READING

View PDF on arXiv

Save to Library

Create Alert

Cite

Launch Research Feed