• Computer Science
  • Published in NAACL-HLT 2019

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

@inproceedings{Devlin2019BERTPO,
  title={BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding},
  author={Jacob Devlin and Ming-Wei Chang and Kenton Lee and Kristina Toutanova},
  booktitle={NAACL-HLT},
  year={2019}
}
We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers. Unlike recent language representation models, BERT is designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. As a result, the pre-trained BERT model can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such… CONTINUE READING

Citations

Publications citing this paper.
SHOWING 1-10 OF 2,068 CITATIONS

A MUTUAL INFORMATION MAXIMIZATION PERSPEC-

  • 2019
VIEW 6 EXCERPTS
CITES METHODS & BACKGROUND
HIGHLY INFLUENCED

A Mutual Information Maximization Perspective of Language Representation Learning

Lingpeng Kong, Cyprien de Masson d'Autume, +3 authors Dani Yogatama
  • ArXiv
  • 2019
VIEW 6 EXCERPTS
CITES METHODS & BACKGROUND
HIGHLY INFLUENCED

Incorporating Domain Knowledge into Medical NLI using Knowledge Graphs

  • IJCNLP 2019
  • 2019
VIEW 6 EXCERPTS
CITES BACKGROUND
HIGHLY INFLUENCED

75 Languages, 1 Model: Parsing Universal Dependencies Universally

  • IJCNLP 2019
  • 2019
VIEW 5 EXCERPTS
CITES METHODS & BACKGROUND
HIGHLY INFLUENCED

FILTER CITATIONS BY YEAR

2017
2019

CITATION STATISTICS

  • 804 Highly Influenced Citations

  • Averaged 687 Citations per year from 2017 through 2019

  • 2,543% Increase in citations per year in 2019 over 2018

References

Publications referenced by this paper.
SHOWING 1-10 OF 51 REFERENCES

Improving language understanding with unsupervised learning

Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever.
  • Technical report, OpenAI.
  • 2018
VIEW 10 EXCERPTS
HIGHLY INFLUENTIAL

Attention is All you Need

VIEW 4 EXCERPTS
HIGHLY INFLUENTIAL