LSTMs Exploit Linguistic Attributes of Data
@article{Liu2018LSTMsEL, title={LSTMs Exploit Linguistic Attributes of Data}, author={Nelson F. Liu and Omer Levy and Roy Schwartz and Chenhao Tan and Noah A. Smith}, journal={ArXiv}, year={2018}, volume={abs/1805.11653} }
While recurrent neural networks have found success in a variety of natural language processing applications, they are general models of sequential data. [...] Key Method Furthermore, we show that the LSTM learns to solve the memorization task by explicitly using a subset of its neurons to count timesteps in the input. We hypothesize that the patterns and structure in natural language data enable LSTMs to learn by providing approximate ways of reducing loss, but understanding the effect of different training…Expand Abstract
Supplemental Presentations
Paper Mentions
13 Citations
How LSTM Encodes Syntax: Exploring Context Vectors and Semi-Quantization on Natural Text
- Computer Science
- COLING
- 2020
- PDF
State gradients for analyzing memory in LSTM language models
- Computer Science
- Comput. Speech Lang.
- 2020
- 1
Word Interdependence Exposes How LSTMs Compose Representations
- Computer Science, Mathematics
- ArXiv
- 2020
- 3
- Highly Influenced
- PDF
Analysis Methods in Neural Language Processing: A Survey
- Computer Science
- Transactions of the Association for Computational Linguistics
- 2019
- 139
- PDF
Recoding latent sentence representations -- Dynamic gradient-based activation modification in RNNs
- Computer Science
- 2021
- Highly Influenced
- PDF
References
SHOWING 1-10 OF 25 REFERENCES
Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies
- Computer Science
- Transactions of the Association for Computational Linguistics
- 2016
- 427
- PDF
Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Tasks
- Computer Science
- ICLR
- 2017
- 273
- PDF
On the Practical Computational Power of Finite Precision RNNs for Language Recognition
- Computer Science, Mathematics
- ACL
- 2018
- 109
- PDF
Extensions of recurrent neural network language model
- Computer Science
- 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
- 2011
- 1,214
- PDF