Corpus ID: 147704286

Unified Language Model Pre-training for Natural Language Understanding and Generation

@article{Dong2019UnifiedLM,
  title={Unified Language Model Pre-training for Natural Language Understanding and Generation},
  author={Li Dong and Nan Yang and Wenhui Wang and Furu Wei and Xiaodong Liu and Yu Wang and Jianfeng Gao and M. Zhou and H. Hon},
  journal={ArXiv},
  year={2019},
  volume={abs/1905.03197}
}
  • Li Dong, Nan Yang, +6 authors H. Hon
  • Published 2019
  • Computer Science
  • ArXiv
  • This paper presents a new Unified pre-trained Language Model (UniLM) that can be fine-tuned for both natural language understanding and generation tasks. [...] Key Method We can fine-tune UniLM as a unidirectional decoder, a bidirectional encoder, or a sequence-to-sequence model to support various downstream natural language understanding and generation tasks. UniLM compares favorably with BERT on the GLUE benchmark, and the SQuAD 2.0 and CoQA question answering tasks. Moreover, our model achieves new state-of…Expand Abstract
    308 Citations
    CopyBERT: A Unified Approach to Question Generation with Self-Attention
    • Highly Influenced
    • PDF
    SiBert: Enhanced Chinese Pre-trained Language Model with Sentence Insertion
    • Highly Influenced
    • PDF
    Optimus: Organizing Sentences via Pre-trained Modeling of a Latent Space
    • 15
    • PDF
    Transformer-based End-to-End Question Generation
    • 4
    • Highly Influenced
    • PDF
    Mixed-Lingual Pre-training for Cross-lingual Summarization
    • PDF

    References

    SHOWING 1-10 OF 61 REFERENCES
    Cross-Lingual Natural Language Generation via Pre-Training
    • 22
    • PDF
    Pre-trained Language Model Representations for Language Generation
    • 59
    • Highly Influential
    • PDF
    BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
    • 13,775
    • Highly Influential
    • PDF
    Language Models are Unsupervised Multitask Learners
    • 2,289
    • PDF
    Multi-Task Deep Neural Networks for Natural Language Understanding
    • 382
    • PDF
    MASS: Masked Sequence to Sequence Pre-training for Language Generation
    • 286
    • Highly Influential
    • PDF
    Improving Language Understanding by Generative Pre-Training
    • 1,711
    • Highly Influential
    • PDF
    GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
    • 1,118
    • PDF
    Text Summarization with Pretrained Encoders
    • 227
    • PDF
    Attention is All you Need
    • 15,021
    • Highly Influential
    • PDF