Analogy-based detection of morphological and semantic relations with word embeddings: what works and what doesn’t.

@inproceedings{Rogers2016AnalogybasedDO,
  title={Analogy-based detection of morphological and semantic relations with word embeddings: what works and what doesn’t.},
  author={Anna Rogers and Aleksandr Drozd and Satoshi Matsuoka},
  booktitle={NAACL},
  year={2016}
}
Following up on numerous reports of analogybased identification of “linguistic regularities” in word embeddings, this study applies the widely used vector offset method to 4 types of linguistic relations: inflectional and derivational morphology, and lexicographic and encyclopedic semantics. We present a balanced test set with 99,200 questions in 40 categories, and we systematically examine how accuracy for different categories is affected by window size and dimensionality of the SVD-based word… 

Figures and Tables from this paper

What’s in Your Embedding, And How It Predicts Task Performance
TLDR
This work presents a new approach based on scaled-up qualitative analysis of word vector neighborhoods that quantifies interpretable characteristics of a given model that enables multi-faceted evaluation, parameter search, and generally – a more principled, hypothesis-driven approach to development of distributional semantic representations.
Understanding the Source of Semantic Regularities in Word Embeddings
TLDR
This paper investigates the hypothesis that examples of a lexical relation in a corpus are fundamental to a neural word embedding’s ability to complete analogies involving the relation, and enhances the understanding of neuralword embeddings.
Inflecting Verbs with Word Embeddings: A Systematic Investigation of Morphological Information Captured by German Verb Embeddings
This study presents a generation task and classification task to systematically evaluate the information on morphological inflections for verbs captured in word embeddings for the morphologically
Fitting Semantic Relations to Word Embeddings
TLDR
It is shown that none of the tested classifiers can learn symmetric relations like synonymy and antonymy, since the source and target words of these relations are the same set, and with the asymmetric relations, both 3CosAvg and LRCos clearly outperform the baseline in all cases.
Derivational Morphological Relations in Word Embeddings
TLDR
The potential of word embeddings to identify properties of word derivations in the morphologically rich Czech language is explored and derivational relations between pairs of words are extracted from DeriNet, a Czech lexical network which organizes almost one million Czech lemmas into derivational trees.
Analogy-based Assessment of Domain-specific Word Embeddings
TLDR
These findings demonstrate that in comparison to analogy-based tests performed against general word embeddings, predictions by domain-specific word embedDings outperform in exactly those analogy categories that are both highly problematic and the location of domain knowledge.
Towards the Detection and Formal Representation of Semantic Shifts in Inflectional Morphology
TLDR
This study extracts word pairs of different grammatical number from WordNet that feature additional senses in the plural and evaluates their distribution in vector space, i.e., pre-trained word2vec and fastText embeddings and proposes an extension of OntoLex-Lemon to accommodate this phenomenon that it is called inflectional morpho-semantic variation.
Analogies minus analogy test: measuring regularities in word embeddings
TLDR
It is shown that, while the standard analogy test is flawed, several popular word embeddings do nevertheless encode linguistic regularities.
Assessing Lexical-Semantic Regularities in Portuguese Word Embeddings
TLDR
A new test, dubbed TALES, is created with an exclusive focus on Portuguese lexical-semantic relations, acquired from lexical resources, and suggests that word embeddings may be a useful source of information for enriching those resources, something the authors also discuss.
Production of Large Analogical Clusters from Smaller Example Seed Clusters Using Word Embeddings
TLDR
The analogical clusters produced by the method are shown to be of reasonably good quality, as shown by comparing human judgment against automatic NDCG@n scores.
...
...

References

SHOWING 1-10 OF 36 REFERENCES
Rehabilitation of Count-Based Models for Word Vector Representations
TLDR
A systematic study of the use of the Hellinger distance to extract semantic representations from the word co-occurrence statistics of large text corpora shows that this distance gives good performance on word similarity and analogy tasks, with a proper type and size of context, and a dimensionality reduction based on a stochastic low-rank approximation.
Multilingual Reliability and “Semantic” Structure of Continuous Word Spaces
TLDR
The results show that (i) morphological complexity causes a drop in accuracy, and (ii) continuous representations lack the ability to solve analogies of paradigmatic relations.
Extracting semantic representations from word co-occurrence statistics: stop-lists, stemming, and SVD
TLDR
This article investigates the use of three further factors—namely, the application of stop-lists, word stemming, and dimensionality reduction using singular value decomposition (SVD)—that have been used to provide improved performance elsewhere and introduces an additional semantic task and explores the advantages of using a much larger corpus.
Don’t count, predict! A systematic comparison of context-counting vs. context-predicting semantic vectors
TLDR
An extensive evaluation of context-predicting models with classic, count-vector-based distributional semantic approaches, on a wide range of lexical semantics tasks and across many parameter settings shows that the buzz around these models is fully justified.
Linguistic Regularities in Continuous Space Word Representations
TLDR
The vector-space word representations that are implicitly learned by the input-layer weights are found to be surprisingly good at capturing syntactic and semantic regularities in language, and that each relationship is characterized by a relation-specific vector offset.
Linguistic Regularities in Sparse and Explicit Word Representations
TLDR
It is demonstrated that analogy recovery is not restricted to neural word embeddings, and that a similar amount of relational similarities can be recovered from traditional distributional word representations.
Take and Took, Gaggle and Goose, Book and Read: Evaluating the Utility of Vector Differences for Lexical Relation Learning
TLDR
It is found that word embeddings capture a surprising amount of information, and that, under suitable supervised training, vector subtraction generalises well to a broad range of relations, including over unseen lexical items.
Improving Distributional Similarity with Lessons Learned from Word Embeddings
TLDR
It is revealed that much of the performance gains of word embeddings are due to certain system design choices and hyperparameter optimizations, rather than the embedding algorithms themselves, and these modifications can be transferred to traditional distributional models, yielding similar gains.
Learning Effective Word Embedding using Morphological Word Similarity
TLDR
A novel neural network architecture is introduced that leverages both contextual information and morphological word similarity to learn word embeddings and is able to refine the pre-defined morphological knowledge and obtain more accurate word similarity.
GloVe: Global Vectors for Word Representation
TLDR
A new global logbilinear regression model that combines the advantages of the two major model families in the literature: global matrix factorization and local context window methods and produces a vector space with meaningful substructure.
...
...