Dependency-Based Word Embeddings

@inproceedings{Levy2014DependencyBasedWE,
  title={Dependency-Based Word Embeddings},
  author={Omer Levy and Yoav Goldberg},
  booktitle={ACL},
  year={2014}
}
While continuous word embeddings are gaining popularity, current models are based solely on linear contexts. In this work, we generalize the skip-gram model with negative sampling introduced by Mikolov et al. to include arbitrary contexts. In particular, we perform experiments with dependency-based contexts, and show that they produce markedly different embeddings. The dependencybased embeddings are less topical and exhibit more functional similarity than the original skip-gram embeddings. 
A Deeper Look into Dependency-Based Word Embeddings
TLDR
It is found that embeddings trained with Universal and Stanford dependency contexts excel at different tasks, and that enhanced dependencies often improve performance. Expand
Improving Distributional Similarity with Lessons Learned from Word Embeddings
TLDR
It is revealed that much of the performance gains of word embeddings are due to certain system design choices and hyperparameter optimizations, rather than the embedding algorithms themselves, and these modifications can be transferred to traditional distributional models, yielding similar gains. Expand
Supervised Phrase-boundary Embeddings
TLDR
This work proposes a new word embedding model, called SPhrase, that incorporates supervised phrase information that includes including this information within a context window produces superior embeddings for both intrinsic evaluation tasks and downstream extrinsic tasks. Expand
Training and Evaluating Improved Dependency-Based Word Embeddings
TLDR
This paper presents a novel multi-order dependency-based strategy to composite and represent the context under several essential constraints to improve the embeddings to be less sensitive to local context and more accurate in capturing semantic compositionality. Expand
Dual embeddings and metrics for word and relational similarity
TLDR
This work shows how the best aspects of both approaches to word embeddings using different metrics can be captured and shows state-of-the-art performance on standard word and relational similarity benchmarks. Expand
Learning word embeddings from dependency relations
TLDR
This work extends previous approach by learning distributed representations from dependency structure of a sentence which can capture long distance relations and proves that context can learn better semantics for words, which is proved on Semantic-Syntactic Word Relationship task. Expand
Syntree2Vec - An algorithm to augment syntactic hierarchy into word embeddings
TLDR
This work proposes a graph based embedding algorithm that improves the syntactic strength and gives robust performance on meagre data by infusing syntactic knowledge into the embeddings. Expand
On Learning Word Embeddings From Linguistically Augmented Text Corpora
TLDR
This paper proposes a method to generate effective word embeddings from limited data and shows that the proposed model outperforms existing works for the classical word similarity task and for a domain-specific application. Expand
Learning word embeddings via context grouping
TLDR
A grouping-based context predictive model is proposed by considering the interactions of context words, which generalizes the widely used CBOW model and Skip-Gram model and demonstrates better representations can be learned with suitable context groups. Expand
Context-Attentive Embeddings for Improved Sentence Representations
TLDR
A novel, straightforward yet highly effective method for combining multiple types of word embeddings in a single model, leading to state-of-the-art performance within the same model class on a variety of tasks. Expand
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 35 REFERENCES
Polyglot: Distributed Word Representations for Multilingual NLP
TLDR
This work quantitatively demonstrates the utility of word embeddings by using them as the sole features for training a part of speech tagger for a subset of these languages and investigates the semantic features captured through the proximity of word groupings. Expand
Word Representations: A Simple and General Method for Semi-Supervised Learning
TLDR
This work evaluates Brown clusters, Collobert and Weston (2008) embeddings, and HLBL (Mnih & Hinton, 2009) embeds of words on both NER and chunking, and finds that each of the three word representations improves the accuracy of these baselines. Expand
Class-Based n-gram Models of Natural Language
TLDR
This work addresses the problem of predicting a word from previous words in a sample of text and discusses n-gram models based on classes of words, finding that these models are able to extract classes that have the flavor of either syntactically based groupings or semanticallybased groupings, depending on the nature of the underlying statistics. Expand
Linguistic Regularities in Sparse and Explicit Word Representations
TLDR
It is demonstrated that analogy recovery is not restricted to neural word embeddings, and that a similar amount of relational similarities can be recovered from traditional distributional word representations. Expand
word2vec Explained: deriving Mikolov et al.'s negative-sampling word-embedding method
TLDR
This note is an attempt to explain equation (4) (negative sampling) in "Distributed Representations of Words and Phrases and their Compositionality" by Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado and Jeffrey Dean. Expand
Efficient Estimation of Word Representations in Vector Space
TLDR
Two novel model architectures for computing continuous vector representations of words from very large data sets are proposed and it is shown that these vectors provide state-of-the-art performance on the authors' test set for measuring syntactic and semantic word similarities. Expand
A Study on Similarity and Relatedness Using Distributional and WordNet-based Approaches
TLDR
This paper presents and compares WordNet-based and distributional similarity approaches, and pioneer cross-lingual similarity, showing that the methods are easily adapted for a cross-lingsual task with minor losses. Expand
Dependency-Based Construction of Semantic Space Models
TLDR
This article presents a novel framework for constructing semantic spaces that takes syntactic relations into account, and introduces a formalization for this class of models, which allows linguistic knowledge to guide the construction process. Expand
Linguistic Regularities in Continuous Space Word Representations
TLDR
The vector-space word representations that are implicitly learned by the input-layer weights are found to be surprisingly good at capturing syntactic and semantic regularities in language, and that each relationship is characterized by a relation-specific vector offset. Expand
Distributed Representations of Words and Phrases and their Compositionality
TLDR
This paper presents a simple method for finding phrases in text, and shows that learning good vector representations for millions of phrases is possible and describes a simple alternative to the hierarchical softmax called negative sampling. Expand
...
1
2
3
4
...