Sharp Nearby, Fuzzy Far Away: How Neural Language Models Use Context

@inproceedings{Khandelwal2018SharpNF,
  title={Sharp Nearby, Fuzzy Far Away: How Neural Language Models Use Context},
  author={Urvashi Khandelwal and He He and Peng Qi and Daniel Jurafsky},
  booktitle={ACL},
  year={2018}
}
We know very little about how neural language models (LM) use prior linguistic context. In this paper, we investigate the role of context in an LSTM LM, through ablation studies. Specifically, we analyze the increase in perplexity when prior context words are shuffled, replaced, or dropped. On two standard datasets, Penn Treebank and WikiText-2, we find that the model is capable of using about 200 tokens of context on average, but sharply distinguishes nearby context (recent 50 tokens) from the… CONTINUE READING
Highly Cited
This paper has 18 citations. REVIEW CITATIONS

From This Paper

Topics from this paper.

Citations

Publications citing this paper.
Showing 1-10 of 14 extracted citations

Similar Papers

Loading similar papers…