Deep contextualized word representations

@article{Peters2018DeepCW,
  title={Deep contextualized word representations},
  author={Matthew E. Peters and Mark Neumann and Mohit Iyyer and Matthew Ph Gardner and Christopher Clark and Kenton Lee and Luke S. Zettlemoyer},
  journal={ArXiv},
  year={2018},
  volume={abs/1802.05365}
}
We introduce a new type of deep contextualized word representation that models both (1) complex characteristics of word use (e.g., syntax and semantics), and (2) how these uses vary across linguistic contexts (i.e., to model polysemy). Our word vectors are learned functions of the internal states of a deep bidirectional language model (biLM), which is pretrained on a large text corpus. We show that these representations can be easily added to existing models and significantly improve the state… CONTINUE READING

Similar Papers

Figures, Tables, Results, and Topics from this paper.

Key Quantitative Results

  • The addition of ELMo representations alone significantly improves the state of the art in every case, including up to 20% relative error reductions.
  • In every task considered, simply adding ELMo establishes a new state-of-the-art result, with relative error reductions ranging from 6 - 20% over strong base models.
  • After adding ELMo to the baseline model, test set F1 improved by 4.7% from 81.1% to 85.8%, a 24.9% relative error reduction over the baseline, and improving the overall single model state-of-the-art by 1.4%.

Citations

Publications citing this paper.
SHOWING 1-10 OF 968 CITATIONS

Constituency Parsing with a Self-Attentive Encoder

VIEW 5 EXCERPTS
CITES BACKGROUND & METHODS
HIGHLY INFLUENCED

A Lightweight Recurrent Network for Sequence Modeling

VIEW 5 EXCERPTS
CITES BACKGROUND & METHODS
HIGHLY INFLUENCED

ConSSED at SemEval-2019 Task 3: Configurable Semantic and Sentiment Emotion Detector

  • SemEval@NAACL-HLT
  • 2019
VIEW 5 EXCERPTS
CITES METHODS
HIGHLY INFLUENCED

LIAAD at SemDeep-5 Challenge: Word-in-Context (WiC)

VIEW 6 EXCERPTS
CITES BACKGROUND & METHODS
HIGHLY INFLUENCED

FILTER CITATIONS BY YEAR

2017
2019

CITATION STATISTICS

  • 285 Highly Influenced Citations

  • Averaged 322 Citations per year from 2017 through 2019

  • 94% Increase in citations per year in 2019 over 2018

References

Publications referenced by this paper.
SHOWING 1-10 OF 60 REFERENCES