Dissecting Contextual Word Embeddings: Architecture and Representation

@inproceedings{Peters2018DissectingCW,
  title={Dissecting Contextual Word Embeddings: Architecture and Representation},
  author={Matthew E. Peters and Mark Neumann and Luke S. Zettlemoyer and Wen-tau Yih},
  booktitle={EMNLP},
  year={2018}
}
Contextual word representations derived from pre-trained bidirectional language models (biLMs) have recently been shown to provide significant improvements to the state of the art for a wide range of NLP tasks. However, many questions remain as to how and why these models are so effective. In this paper, we present a detailed empirical study of how the choice of neural architecture (e.g. LSTM, CNN, or self attention) influences both end task accuracy and qualitative properties of the… CONTINUE READING

Figures, Tables, Results, and Topics from this paper.

Key Quantitative Results

  • All architectures improve significantly over the GloVe only baseline, with relative improvements of 13% – 25% for most tasks and architectures.

Citations

Publications citing this paper.
SHOWING 1-10 OF 39 CITATIONS

Casting Light on Invisible Cities: Computationally Engaging with Literary Criticism

  • NAACL-HLT
  • 2019
VIEW 11 EXCERPTS
CITES METHODS & BACKGROUND
HIGHLY INFLUENCED

LEARN FROM CONTEXT ? P ROBING FOR SENTENCE STRUCTURE IN CONTEXTUALIZED WORD REPRESENTATIONS

VIEW 13 EXCERPTS
CITES METHODS, BACKGROUND & RESULTS
HIGHLY INFLUENCED

Shallow Syntax in Deep Water

Swabha Swayamdipta, Matthew J. Peters, Brendan Roof, Chris Dyer, Noah A. Smith
  • ArXiv
  • 2019
VIEW 9 EXCERPTS
CITES METHODS, BACKGROUND & RESULTS
HIGHLY INFLUENCED

Syntactic Inductive Biases for Natural Language Processing

VIEW 6 EXCERPTS
CITES BACKGROUND & METHODS
HIGHLY INFLUENCED

What do you learn from context? Probing for sentence structure in contextualized word representations

  • ICLR
  • 2019
VIEW 13 EXCERPTS
CITES METHODS, BACKGROUND & RESULTS
HIGHLY INFLUENCED

A Survey of Reinforcement Learning Informed by Natural Language

VIEW 4 EXCERPTS
CITES BACKGROUND & METHODS
HIGHLY INFLUENCED

References

Publications referenced by this paper.