LSTMs Exploit Linguistic Attributes of Data

@article{Liu2018LSTMsEL,
  title={LSTMs Exploit Linguistic Attributes of Data},
  author={Nelson F. Liu and Omer Levy and Roy Schwartz and Chenhao Tan and Noah A. Smith},
  journal={ArXiv},
  year={2018},
  volume={abs/1805.11653}
}
  • Nelson F. Liu, Omer Levy, +2 authors Noah A. Smith
  • Published 2018
  • Computer Science
  • ArXiv
  • While recurrent neural networks have found success in a variety of natural language processing applications, they are general models of sequential data. [...] Key Method Furthermore, we show that the LSTM learns to solve the memorization task by explicitly using a subset of its neurons to count timesteps in the input. We hypothesize that the patterns and structure in natural language data enable LSTMs to learn by providing approximate ways of reducing loss, but understanding the effect of different training…Expand Abstract
    12 Citations

    Figures and Topics from this paper.

    On Evaluating the Generalization of LSTM Models in Formal Languages
    On Evaluating the Generalization of LSTM Models in Formal Languages
    • 19
    • PDF
    Understanding Learning Dynamics Of Language Models with SVCCA
    • 35
    • PDF
    LSTMs Compose (and Learn) Bottom-Up
    • 1
    • Highly Influenced
    • PDF
    How LSTM Encodes Syntax: Exploring Context Vectors and Semi-Quantization on Natural Text
    State gradients for analyzing memory in LSTM language models
    • 1
    Word Interdependence Exposes How LSTMs Compose Representations
    • 3
    • Highly Influenced
    • PDF
    Analysis Methods in Neural Language Processing: A Survey
    • 128
    • PDF
    Language Models Learn POS First
    • 7
    • PDF
    Automatically Extracting Challenge Sets for Non local Phenomena in Neural Machine Translation

    References

    SHOWING 1-10 OF 25 REFERENCES
    Visualizing and Understanding Recurrent Networks
    • 744
    • PDF
    Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies
    • 401
    • PDF
    Visualizing and Understanding Neural Models in NLP
    • 388
    • PDF
    Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Tasks
    • 258
    • PDF
    Recurrent neural network based language model
    • 4,062
    • PDF
    On the Practical Computational Power of Finite Precision RNNs for Language Recognition
    • 101
    • PDF
    Extensions of recurrent neural network language model
    • 1,180
    • PDF
    What do Neural Machine Translation Models Learn about Morphology?
    • 190
    • PDF
    Finding Structure in Time
    • 8,544
    • PDF