Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies

@article{Linzen2016AssessingTA,
  title={Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies},
  author={Tal Linzen and Emmanuel Dupoux and Y. Goldberg},
  journal={Transactions of the Association for Computational Linguistics},
  year={2016},
  volume={4},
  pages={521-535}
}
  • Tal Linzen, Emmanuel Dupoux, Y. Goldberg
  • Published 2016
  • Computer Science
  • Transactions of the Association for Computational Linguistics
  • The success of long short-term memory (LSTM) neural networks in language processing is typically attributed to their ability to capture long-distance statistical regularities. [...] Key Method We probe the architecture’s grammatical competence both using training objectives with an explicit grammatical target (number prediction, grammaticality judgments) and using language models. In the strongly supervised settings, the LSTM achieved very high overall accuracy (less than 1% errors), but errors increased when…Expand Abstract
    401 Citations

    Topics from this paper.

    LSTMs Can Learn Syntax-Sensitive Dependencies Well, But Modeling Structure Makes Them Better
    • 73
    • Highly Influenced
    • PDF
    How much complexity does an RNN architecture need to learn syntax-sensitive dependencies?
    • 1
    • Highly Influenced
    • PDF
    Structural Supervision Improves Learning of Non-Local Grammatical Dependencies
    • 18
    • PDF
    Influence Paths for Characterizing Subject-Verb Number Agreement in LSTM Language Models
    • 1
    • Highly Influenced
    • PDF
    Evaluating the Ability of LSTMs to Learn Context-Free Grammars
    • 25
    • PDF
    Exploring the Syntactic Abilities of RNNs with Multi-task Learning
    • 22
    • PDF
    Attribution Analysis of Grammatical Dependencies in LSTMs
    • 2
    • Highly Influenced
    • PDF
    Can LSTM Learn to Capture Agreement? The Case of Basque
    • 15
    • PDF
    Scalable Syntax-Aware Language Models Using Knowledge Distillation
    • 12
    • Highly Influenced
    • PDF
    Structural Supervision Improves Few-Shot Learning and Syntactic Generalization in Neural Language Models
    • 1
    • Highly Influenced
    • PDF

    References

    SHOWING 1-10 OF 53 REFERENCES
    Visualizing and Understanding Neural Models in NLP
    • 388
    • PDF
    Language acquisition in the absence of explicit negative evidence: how important is starting small?
    • 243
    • PDF
    Visualizing and Understanding Recurrent Networks
    • 744
    • PDF
    The Acquisition of Anaphora by Simple Recurrent Networks
    • 16
    • PDF
    Statistical Representation of Grammaticality Judgements: the Limits of N-Gram Models
    • 24
    • PDF
    One billion word benchmark for measuring progress in statistical language modeling
    • 706
    • PDF
    Representation of Linguistic Form and Function in Recurrent Neural Networks
    • 101
    • PDF
    Grammar as a Foreign Language
    • 746
    • PDF
    LSTM recurrent networks learn simple context-free and context-sensitive languages
    • 456
    • PDF
    LSTM Neural Networks for Language Modeling
    • 1,149
    • PDF