Corpus ID: 49313245

Improving Language Understanding by Generative Pre-Training

@inproceedings{Radford2018ImprovingLU,
  title={Improving Language Understanding by Generative Pre-Training},
  author={A. Radford},
  year={2018}
}
  • A. Radford
  • Published 2018
  • Computer Science
  • Natural language understanding comprises a wide range of diverse tasks such as textual entailment, question answering, semantic similarity assessment, and document classification. Although large unlabeled text corpora are abundant, labeled data for learning these specific tasks is scarce, making it challenging for discriminatively trained models to perform adequately. We demonstrate that large gains on these tasks can be realized by generative pre-training of a language model on a diverse… CONTINUE READING
    Language Models are Unsupervised Multitask Learners
    • 1,724
    • Open Access
    Unsupervised Domain Adaptation of Contextualized Embeddings for Sequence Labeling
    • 21
    • Open Access
    Cross-lingual Language Model Pretraining
    • 483
    • Open Access
    Unified Language Model Pre-training for Natural Language Understanding and Generation
    • 213
    • Highly Influenced
    • Open Access
    Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
    • 477
    • Highly Influenced
    • Open Access
    Language Models are Few-Shot Learners
    • 208
    • Open Access
    Transfer Learning in Natural Language Processing
    • 44
    • Highly Influenced
    • Open Access
    Multi-Task Deep Neural Networks for Natural Language Understanding
    • 296
    • Open Access
    Language Models as Knowledge Bases?
    • 132
    • Highly Influenced
    • Open Access
    XNLI: Evaluating Cross-lingual Sentence Representations
    • 187
    • Open Access

    References

    Publications referenced by this paper.
    SHOWING 1-10 OF 72 REFERENCES
    Universal Language Model Fine-tuning for Text Classification
    • 1,098
    • Open Access
    GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
    • 838
    • Highly Influential
    • Open Access
    Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank
    • 3,790
    • Highly Influential
    • Open Access
    Skip-Thought Vectors
    • 1,490
    • Open Access
    Natural Language Processing (Almost) from Scratch
    • 5,429
    • Open Access
    Semi-supervised sequence tagging with bidirectional language models
    • 316
    • Open Access
    A Simple but Tough-to-Beat Baseline for Sentence Embeddings
    • 641
    A unified architecture for natural language processing: deep neural networks with multitask learning
    • 4,011
    • Open Access