Corpus ID: 202888986

ALBERT: A Lite BERT for Self-supervised Learning of Language Representations

@article{Lan2020ALBERTAL,
  title={ALBERT: A Lite BERT for Self-supervised Learning of Language Representations},
  author={Zhenzhong Lan and Mingda Chen and Sebastian Goodman and Kevin Gimpel and Piyush Sharma and Radu Soricut},
  journal={ArXiv},
  year={2020},
  volume={abs/1909.11942}
}

Zhenzhong Lan, Mingda Chen, +3 authors Radu Soricut

Published 2020

Computer Science

ArXiv

Increasing model size when pretraining natural language representations often results in improved performance on downstream tasks. [...] Key Method Comprehensive empirical evidence shows that our proposed methods lead to models that scale much better compared to the original BERT. We also use a self-supervised loss that focuses on modeling inter-sentence coherence, and show it consistently helps downstream tasks with multi-sentence inputs. As a result, our best model establishes new state-of-the-art results on…Expand Abstract

View PDF on arXiv

Save to Library

Create Alert

Cite

Launch Research Feed