• Corpus ID: 1185652

Modeling Compositionality with Multiplicative Recurrent Neural Networks

@article{Irsoy2015ModelingCW,
  title={Modeling Compositionality with Multiplicative Recurrent Neural Networks},
  author={Ozan Irsoy and Claire Cardie},
  journal={CoRR},
  year={2015},
  volume={abs/1412.6577}
}
We present the multiplicative recurrent neural network as a general model for compositional meaning in language, and evaluate it on the task of fine-grained sentiment analysis. We establish a connection to the previously investigated matrix-space models for compositionality, and show they are special cases of the multiplicative recurrent net. Our experiments show that these models perform comparably or better than Elman-type additive recurrent neural networks and outperform matrix-space models… 

Figures and Tables from this paper

Compositional matrix-space models of language: Definitions, properties, and learning methods
TLDR
An in-depth account of compositional matrix-space models (CMSMs), a type of generic models for natural language, wherein compositionality is realized via matrix multiplication, and their performance in compositionality prediction and sentiment analysis is evaluated.
Restricted Recurrent Neural Tensor Networks: Exploiting Word Frequency and Compositionality
TLDR
Perplexity evaluations show that for fixed hidden layer sizes, r-RNTNs improve language model performance over RNNs using only a small fraction of the parameters of unrestricted RNTNs.
Neural Semantic Encoders
TLDR
This paper demonstrated the effectiveness and the flexibility of NSE on five different natural language tasks: natural language inference, question answering, sentence classification, document sentiment analysis and machine translation where NSE achieved state-of-the-art performance when evaluated on publically available benchmarks.
Understanding Gating Operations in Recurrent Neural Networks through Opinion Expression Extraction
TLDR
Long Short-Term Memory recurrent neural networks are adopted to address the task of opinion expression extraction and explore the internal mechanisms of the model to provide a novel micro perspective to analyze the run-time processes and gain new insights into the advantages of LSTM selecting the source of information with its flexible connections and multiplier gating operations.
A Deep Recursive Neural Network Model for Fine-Grained Opinion Classification
TLDR
The results report that the proposed DRNN architecture achieves better prediction rate for fine-grained classification when compared with conventional shallow counterparts that employ similar parameters.
Ask Me Anything: Dynamic Memory Networks for Natural Language Processing
TLDR
The dynamic memory network (DMN), a neural network architecture which processes input sequences and questions, forms episodic memories, and generates relevant answers, is introduced.
Cross-domain & In-domain Sentiment Analysis with Memory-based Deep Neural Networks
TLDR
This work shows that MemDNN based classifiers improve the state-of-the-art on Amazon Reviews corpus with reference to document-level cross-domain sentiment classification, and DNC outperforms previous approaches in the analysis of a very large in-domain configuration in both binary and fine-grained document sentiment classification.
Neural Tree Indexers for Text Understanding
TLDR
A robust syntactic parsing-independent tree structured model, Neural Tree Indexers (NTI) is introduced that provides a middle ground between the sequential RNNs and the syntactic treebased recursive models, outperforming state-of-the-art recurrent and recursive neural networks.
Extended Dependency-Based Word Embeddings for Aspect Extraction
TLDR
This paper introduces outer product of dependency-based word vectors and specialized features as representation of words and shows that it is an effective way to achieve better extraction performance by improving word representations.
Neural word embeddings with multiplicative feature interactions for tensor-based compositions
TLDR
This work shows whether the word embeddings from extended CBOW models using multiplication or Tensor product between context words, reflecting the actual composition methods, can show better performance than those from the baseline CBOW model in actual tasks of compositions with multiplication or tensor-based methods.
...
1
2
3
...

References

SHOWING 1-10 OF 32 REFERENCES
Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank
TLDR
A Sentiment Treebank that includes fine grained sentiment labels for 215,154 phrases in the parse trees of 11,855 sentences and presents new challenges for sentiment compositionality, and introduces the Recursive Neural Tensor Network.
Compositional Matrix-Space Models for Sentiment Analysis
TLDR
This paper presents the first such algorithm for learning a matrix-space model for semantic composition, and its experimental results show statistically significant improvements in performance over a bag-of-words model.
Semantic Compositionality through Recursive Matrix-Vector Spaces
TLDR
A recursive neural network model that learns compositional vector representations for phrases and sentences of arbitrary syntactic type and length and can learn the meaning of operators in propositional logic and natural language is introduced.
Multi-Step Regression Learning for Compositional Distributional Semantics
TLDR
It is argued in the analysis that the nature of this learning method also renders it suitable for solving more subtle problems compositional distributional models might face, and is found to outperform existing leading methods.
Generating Text with Recurrent Neural Networks
TLDR
The power of RNNs trained with the new Hessian-Free optimizer by applying them to character-level language modeling tasks is demonstrated, and a new RNN variant that uses multiplicative connections which allow the current input character to determine the transition matrix from one hidden state vector to the next is introduced.
A unified architecture for natural language processing: deep neural networks with multitask learning
We describe a single convolutional neural network architecture that, given a sentence, outputs a host of language processing predictions: part-of-speech tags, chunks, named entity tags, semantic
Estimating Linear Models for Compositional Distributional Semantics
TLDR
This paper proposes a novel approach to estimate parameters for a class of compositional distributional models: the additive models, and demonstrates that this approach outperforms existing methods for determining a good model for compositional distributionsal semantics.
Three new graphical models for statistical language modelling
TLDR
It is shown how real-valued distributed representations for words can be learned at the same time as learning a large set of stochastic binary hidden features that are used to predict the distributed representation of the next word from previous distributed representations.
Distributed Representations of Words and Phrases and their Compositionality
TLDR
This paper presents a simple method for finding phrases in text, and shows that learning good vector representations for millions of phrases is possible and describes a simple alternative to the hierarchical softmax called negative sampling.
Compositional Matrix-Space Models of Language
TLDR
CMSMs, a novel type of generic compositional models for syntactic and semantic aspects of natural language, based on matrix multiplication is proposed and it is shown that it is able to cover and combine various common compositional NLP approaches.
...
1
2
3
4
...