What do you learn from context? Probing for sentence structure in contextualized word representations

@article{Tenney2019WhatDY,
  title={What do you learn from context? Probing for sentence structure in contextualized word representations},
  author={Ian Tenney and Patrick Xia and Berlin Chen and Alex Wang and Adam Poliak and R. Thomas McCoy and Najoung Kim and Benjamin Van Durme and Samuel R. Bowman and Dipanjan Das and Ellie Pavlick},
  journal={ArXiv},
  year={2019},
  volume={abs/1905.06316}
}

• Ian Tenney, Patrick Xia, +8 authors Ellie Pavlick
• Published in ICLR 2019

Contextualized representation models such as ELMo (Peters et al., 2018a) and BERT (Devlin et al., 2018) have recently achieved state-of-the-art results on a diverse array of downstream NLP tasks. Building on recent token-level probing work, we introduce a novel edge probing task design and construct a broad suite of sub-sentence tasks derived from the traditional structured NLP pipeline. We probe word-level contextual representations from four recent models and investigate how they encode… CONTINUE READING

View PDF on ArXiv

Save to Library

Create Alert

Cite