Improving Language Understanding by Generative Pre-Training

@inproceedings{Radford2018ImprovingLU,
  title={Improving Language Understanding by Generative Pre-Training},
  author={Alec Radford},
  year={2018}
}
Natural language understanding comprises a wide range of diverse tasks such as textual entailment, question answering, semantic similarity assessment, and document classification. Although large unlabeled text corpora are abundant, labeled data for learning these specific tasks is scarce, making it challenging for discriminatively trained models to perform adequately. We demonstrate that large gains on these tasks can be realized by generative pre-training of a language model on a diverse… CONTINUE READING

Figures and Tables from this paper.

Citations

Publications citing this paper.
SHOWING 1-10 OF 268 CITATIONS

Cloze-driven Pretraining of Self-attention Networks

VIEW 17 EXCERPTS
CITES RESULTS, METHODS & BACKGROUND
HIGHLY INFLUENCED

Evidence Sentence Extraction for Machine Reading Comprehension

  • ArXiv
  • 2019
VIEW 12 EXCERPTS
CITES METHODS, BACKGROUND & RESULTS
HIGHLY INFLUENCED

Generating Long Sequences with Sparse Transformers

VIEW 5 EXCERPTS
CITES METHODS & BACKGROUND
HIGHLY INFLUENCED

Language Models with Transformers

VIEW 9 EXCERPTS
CITES METHODS & BACKGROUND
HIGHLY INFLUENCED

Learning to Rank for Plausible Plausibility

  • ACL
  • 2019
VIEW 7 EXCERPTS
HIGHLY INFLUENCED

Neural Machine Reading Comprehension: Methods and Trends

VIEW 9 EXCERPTS
CITES BACKGROUND & METHODS
HIGHLY INFLUENCED

Sequential transfer learning in NLP for text summarization

Pascal Fecht
  • 2019
VIEW 13 EXCERPTS
CITES METHODS & BACKGROUND
HIGHLY INFLUENCED

FILTER CITATIONS BY YEAR

2017
2019

CITATION STATISTICS

  • 80 Highly Influenced Citations

  • Averaged 89 Citations per year from 2017 through 2019

  • 332% Increase in citations per year in 2019 over 2018

References

Publications referenced by this paper.
SHOWING 1-10 OF 72 REFERENCES

Corpus of linguistic acceptability

A. Warstadt, A. Singh, S. R. Bowman
  • http://nyu-mll.github.io/cola
  • 2018
VIEW 4 EXCERPTS
HIGHLY INFLUENTIAL

Deep contextualized word representations

VIEW 9 EXCERPTS
HIGHLY INFLUENTIAL

Attention Is All You Need

VIEW 8 EXCERPTS
HIGHLY INFLUENTIAL

The fifth pascal recognizing textual entailment challenge

L. Bentivogli, P. Clark, I. Dagan, D. Giampiccolo
  • TAC
  • 2009
VIEW 3 EXCERPTS
HIGHLY INFLUENTIAL

Similar Papers