Corpus ID: 209832412

On the Comparability of Pre-trained Language Models

@article{Aenmacher2020OnTC,
  title={On the Comparability of Pre-trained Language Models},
  author={Matthias A{\ss}enmacher and C. Heumann},
  journal={ArXiv},
  year={2020},
  volume={abs/2001.00781}
}
  • Matthias Aßenmacher, C. Heumann
  • Published 2020
  • Computer Science, Mathematics
  • ArXiv
  • Recent developments in unsupervised representation learning have successfully established the concept of transfer learning in NLP. Mainly three forces are driving the improvements in this area of research: More elaborated architectures are making better use of contextual information. Instead of simply plugging in static pre-trained representations, these are learned based on surrounding context in end-to-end trainable models with more intelligently designed language modelling objectives. Along… CONTINUE READING

    References

    Publications referenced by this paper.
    SHOWING 1-10 OF 57 REFERENCES
    Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
    387
    Transfer Learning in Natural Language Processing
    40
    Language Models are Unsupervised Multitask Learners
    1539
    Improving Language Understanding by Generative Pre-Training
    1313
    Universal Language Model Fine-tuning for Text Classification
    1046
    ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
    467
    Fine-tuned Language Models for Text Classification
    240
    BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
    8977
    GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
    761
    Regularizing and Optimizing LSTM Language Models
    562