Corpus ID: 195069387

XLNet: Generalized Autoregressive Pretraining for Language Understanding

@inproceedings{Yang2019XLNetGA,
  title={XLNet: Generalized Autoregressive Pretraining for Language Understanding},
  author={Zhilin Yang and Zihang Dai and Yiming Yang and Jaime G. Carbonell and Ruslan Salakhutdinov and Quoc V. Le},
  booktitle={NeurIPS},
  year={2019}
}
  • Zhilin Yang, Zihang Dai, +3 authors Quoc V. Le
  • Published in NeurIPS 2019
  • Computer Science
  • With the capability of modeling bidirectional contexts, denoising autoencoding based pretraining like BERT achieves better performance than pretraining approaches based on autoregressive language modeling. [...] Key Method In light of these pros and cons, we propose XLNet, a generalized autoregressive pretraining method that (1) enables learning bidirectional contexts by maximizing the expected likelihood over all permutations of the factorization order and (2) overcomes the limitations of BERT thanks to its…Expand Abstract

    Citations

    Publications citing this paper.
    SHOWING 1-10 OF 1,032 CITATIONS

    MPNet: Masked and Permuted Pre-training for Language Understanding

    VIEW 23 EXCERPTS
    CITES METHODS, BACKGROUND & RESULTS
    HIGHLY INFLUENCED

    Optimus: Organizing Sentences via Pre-trained Modeling of a Latent Space

    VIEW 12 EXCERPTS
    CITES METHODS & BACKGROUND
    HIGHLY INFLUENCED

    MML: Maximal Multiverse Learning for Robust Fine-Tuning of Language Models

    VIEW 4 EXCERPTS
    CITES METHODS & BACKGROUND
    HIGHLY INFLUENCED

    UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training

    VIEW 8 EXCERPTS
    CITES METHODS & BACKGROUND
    HIGHLY INFLUENCED

    Abstractive Text Summarization based on Language Model Conditioning and Locality Modeling

    VIEW 4 EXCERPTS
    CITES METHODS & BACKGROUND
    HIGHLY INFLUENCED

    Natural Language Generation for Effective Knowledge Distillation

    VIEW 2 EXCERPTS
    CITES BACKGROUND

    Masked Language Model Scoring

    VIEW 2 EXCERPTS
    CITES BACKGROUND

    Incorporating BERT into Neural Machine Translation

    VIEW 3 EXCERPTS
    CITES METHODS & BACKGROUND

    FILTER CITATIONS BY YEAR

    2017
    2020

    CITATION STATISTICS

    • 215 Highly Influenced Citations

    • Averaged 344 Citations per year from 2018 through 2020

    • 135% Increase in citations per year in 2020 over 2019

    References

    Publications referenced by this paper.
    SHOWING 1-10 OF 49 REFERENCES

    Attention is All you Need

    VIEW 1 EXCERPT