Deep contextualized word representations
@inproceedings{Peters2018DeepCW, title={Deep contextualized word representations}, author={Matthew E. Peters and Mark Neumann and Mohit Iyyer and Matt Gardner and Christopher Clark and Kenton Lee and Luke Zettlemoyer}, booktitle={NAACL-HLT}, year={2018} }
We introduce a new type of deep contextualized word representation that models both (1) complex characteristics of word use (e.g., syntax and semantics), and (2) how these uses vary across linguistic contexts (i.e., to model polysemy. [...] Key Result We also present an analysis showing that exposing the deep internals of the pre-trained network is crucial, allowing downstream models to mix different types of semi-supervision signals.Expand Abstract
Supplemental Content
Figures, Tables, and Topics from this paper
Paper Mentions
News Article
News Article
News Article
Blog Post
5,046 Citations
Deep contextualized word embeddings from character language models for neural sequence labeling
- Computer Science
- 2019
- 1
- Highly Influenced
Dissecting Contextual Word Embeddings: Architecture and Representation
- Computer Science
- EMNLP
- 2018
- 175
- PDF
Improved Word Sense Disambiguation Using Pre-Trained Contextualized Word Representations
- Computer Science
- EMNLP/IJCNLP
- 2019
- 16
- Highly Influenced
- PDF
Retrofitting Contextualized Word Embeddings with Paraphrases
- Computer Science
- EMNLP/IJCNLP
- 2019
- 7
- Highly Influenced
- PDF
Quantifying the Contextualization of Word Representations with Semantic Class Probing
- Computer Science
- EMNLP 2020
- 2020
- 1
Quantifying the Contextualization of Word Representations with Semantic Class Probing
- Computer Science
- EMNLP
- 2020
- 2
- Highly Influenced
- PDF
Contextualized Word Representations for Self-Attention Network
- Computer Science
- 2018 13th International Conference on Computer Engineering and Systems (ICCES)
- 2018
Linguistic Knowledge and Transferability of Contextual Representations
- Computer Science
- NAACL-HLT
- 2019
- 227
- PDF
Context Analysis for Pre-trained Masked Language Models
- Computer Science
- EMNLP
- 2020
- Highly Influenced
- PDF
References
SHOWING 1-10 OF 65 REFERENCES
context2vec: Learning Generic Context Embedding with Bidirectional LSTM
- Computer Science
- CoNLL
- 2016
- 283
- Highly Influential
- PDF
Word Representations: A Simple and General Method for Semi-Supervised Learning
- Computer Science
- ACL
- 2010
- 1,999
- PDF
Finding Function in Form: Compositional Character Models for Open Vocabulary Word Representation
- Computer Science
- EMNLP
- 2015
- 514
- PDF
Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank
- Computer Science
- EMNLP
- 2013
- 4,110
- PDF
Enriching Word Vectors with Subword Information
- Computer Science
- Transactions of the Association for Computational Linguistics
- 2017
- 4,320
- PDF