Neural Tree Indexers for Text Understanding

@article{Munkhdalai2017NeuralTI,
  title={Neural Tree Indexers for Text Understanding},
  author={Tsendsuren Munkhdalai and Hong Yu},
  journal={Proceedings of the conference. Association for Computational Linguistics. Meeting},
  year={2017},
  volume={1},
  pages={
          11-21
        }
}
  • Tsendsuren Munkhdalai, Hong Yu
  • Published 15 July 2016
  • Computer Science
  • Proceedings of the conference. Association for Computational Linguistics. Meeting
Recurrent neural networks (RNNs) process input text sequentially and model the conditional transition between word tokens. In contrast, the advantages of recursive networks include that they explicitly model the compositionality and the recursive structure of natural language. However, the current recursive architecture is limited by its dependence on syntactic tree. In this paper, we introduce a robust syntactic parsing-independent tree structured model, Neural Tree Indexers (NTI) that… 

Figures and Tables from this paper

Unsupervised Learning of Task-Specific Tree Structures with Tree-LSTMs
TLDR
This paper proposes a novel tree-structured long short-term memory (Tree-LSTM) architecture that efficiently learns how to compose task-specific tree structures only from plain text data and evaluates the proposed model on natural language interface and sentiment analysis and shows that it outperforms or at least comparable to previous Tree-L STM-based works.
On Tree-Based Neural Sentence Modeling
TLDR
Replacing syntactic parsing trees with trivial trees in the encoders indicates that explicit syntax guidance may not be the main contributor to the superior performances of tree-based neural sentence modeling and shows that tree modeling gives better results when crucial words are closer to the final representation.
Improved Sentence Modeling using Suffix Bidirectional LSTM
TLDR
This work proposes a general and effective improvement to the BiLSTM model which encodes each suffix and prefix of a sequence of tokens in both forward and reverse directions, and introduces an alternate bias that favors long range dependencies.
Unsupervised Learning of Explainable Parse Trees for Improved Generalisation
TLDR
This work proposes an attention mechanism over Tree-LSTMs to learn more meaningful and explainable parse tree structures and shows that the discovered linguistic structures are more explainable, semantically meaningful, and grammatically correct than recent approaches.
Natural language inference using LSTM model with sentence fusion
TLDR
The results demonstrate that the LSTM with Sentence Fusion which reads premise and hypothesis to produce a final fusion representation from which a three-way classifier predicts label has a better performance than L STM RNN encoders and Lexicalized classifier.
Do latent tree learning models identify meaningful structure in sentences?
TLDR
This paper replicates two latent tree learning models in a shared codebase and finds that only one of these models outperforms conventional tree-structured models on sentence classification, and its parsing strategies are not especially consistent across random restarts.
Learning to Embed Sentences Using Attentive Recursive Trees
TLDR
An end-to-end reinforced training strategy for AR-Tree is proposed, which is demonstrated to consistently outperform, or be at least comparable to, the state-of-the-art sentence embedding methods on three sentence understanding tasks.
Syntax-Aware Attention for Natural Language Inference with Phrase-Level Matching
TLDR
This paper presents a syntax-aware attention model for NLI by which phrase-level matching between two sentences is allowed, and designs tree-structured semantic composition function that builds phrase representations according to syntactic trees.
Learning to Compose Task-Specific Tree Structures
TLDR
This paper proposes Gumbel Tree-LSTM, a novel tree-structured long short-term memory architecture that learns how to compose task-specific tree structures only from plain text data efficiently and finds that the model converges significantly faster than other models.
A Sequential Neural Encoder With Latent Structured Description for Modeling Sentences
TLDR
The experimental results demonstrate the effectiveness of the proposed SNELSD model on exploring task-dependent chunking patterns during the semantic modeling of sentences and the proposed method achieves better performance than conventional chain LSTMs and tree-structured LSTM on both tasks.
...
...

References

SHOWING 1-10 OF 52 REFERENCES
A Fast Unified Model for Parsing and Sentence Understanding
TLDR
The Stack-augmentedParser-Interpreter NeuralNetwork (SPINN) combines parsing and interpretation within a single tree-sequence hybrid model by integrating tree-structured sentence interpretation into the linear sequential structure of a shiftreduceparser.
Semantic Compositionality through Recursive Matrix-Vector Spaces
TLDR
A recursive neural network model that learns compositional vector representations for phrases and sentences of arbitrary syntactic type and length and can learn the meaning of operators in propositional logic and natural language is introduced.
Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks
TLDR
The Tree-LSTM is introduced, a generalization of LSTMs to tree-structured network topologies that outperform all existing systems and strong LSTM baselines on two tasks: predicting the semantic relatedness of two sentences and sentiment classification.
Tree-Structured Composition in Neural Networks without Tree-Structured Architectures
TLDR
It is found that an LSTM-based sequence model can indeed learn to exploit the underlying tree structure, and its performance consistently lags behind that of tree models, even on large training sets, suggesting that tree-structured models are more effective at exploiting recursive structure.
Long Short-Term Memory Over Recursive Structures
TLDR
This paper proposes to extend chain-structured long short-term memory to tree structures, in which a memory cell can reflect the history memories of multiple child cells or multiple descendant cells in a recursive process, and calls the model S-LSTM, which provides a principled way of considering long-distance interaction over hierarchies.
Long Short-Term Memory Over Tree Structures
TLDR
This paper proposes to extend chain-structured long short-term memory to tree structures, in which a memory cell can reflect the history memories of multiple child cells or multiple descendant cells in a recursive process, and calls the model S-LSTM, which provides a principled way of considering long-distance interaction over hierarchies.
Ask Me Anything: Dynamic Memory Networks for Natural Language Processing
TLDR
The dynamic memory network (DMN), a neural network architecture which processes input sequences and questions, forms episodic memories, and generates relevant answers, is introduced.
Learning Natural Language Inference with LSTM
TLDR
A special long short-term memory (LSTM) architecture for NLI that remembers important mismatches that are critical for predicting the contradiction or the neutral relationship label and achieves an accuracy of 86.1%, outperforming the state of the art.
Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank
TLDR
A Sentiment Treebank that includes fine grained sentiment labels for 215,154 phrases in the parse trees of 11,855 sentences and presents new challenges for sentiment compositionality, and introduces the Recursive Neural Tensor Network.
Grammar as a Foreign Language
TLDR
The domain agnostic attention-enhanced sequence-to-sequence model achieves state-of-the-art results on the most widely used syntactic constituency parsing dataset, when trained on a large synthetic corpus that was annotated using existing parsers.
...
...