• Corpus ID: 7202537

Word Embeddings, Analogies, and Machine Learning: Beyond king - man + woman = queen

@inproceedings{Drozd2016WordEA,
  title={Word Embeddings, Analogies, and Machine Learning: Beyond king - man + woman = queen},
  author={Aleksandr Drozd and Anna Rogers and Satoshi Matsuoka},
  booktitle={COLING},
  year={2016}
}
Solving word analogies became one of the most popular benchmarks for word embeddings on the assumption that linear relations between word pairs (such as king:man :: woman:queen) are indicative of the quality of the embedding. We question this assumption by showing that the information not detected by linear offset may still be recoverable by a more sophisticated search method, and thus is actually encoded in the embedding. The general problem with linear offset is its sensitivity to the… 

Figures and Tables from this paper

Exploring Different Methods for Solving Analogies with Portuguese Word Embeddings
TLDR
Alternative analogy solving methods used to answer the Portuguese version of the Google Analogy Test, dubbed LX-4WAnalogies, which covers syntactic and semantic analogies of different kinds outperform 3CosAdd, and the best performance is consistently achieved with LRCos, in GloVe.
Rotate King to get Queen: Word Relationships as Orthogonal Transformations in Embedding Space
TLDR
These findings suggest that orthogonal and linear transformations can be as good a representation of word relationships as translation vectors astranslation vectors.
On the Utility of Word Embeddings for Enriching OpenWordNet-PT
TLDR
Results are promising for accelerating the identification of missing relation instances, as it is estimated that about 17% of the potential suggestions are good, a proportion that almost doubles if some are automatically invalidated.
Analogies minus analogy test: measuring regularities in word embeddings
TLDR
It is shown that, while the standard analogy test is flawed, several popular word embeddings do nevertheless encode linguistic regularities.
An Optimality Proof for the PairDiff operator for Representing Relations between Words
TLDR
This paper shows that for word embeddings where cross-dimensional correlations are zero, \PairDiff is the only bilinear operator that can minimise the $\ell_{2}$ loss between analogous word-pairs.
Understanding Linearity of Cross-Lingual Word Embedding Mappings
TLDR
This work presents a theoretical analysis that identifies the preservation of analogies encoded in monolingual word embeddings as a necessary andcient condition for the ground-truth CLWE mapping between those embedDings to be linear.
Why does PairDiff work? - A Mathematical Analysis of Bilinear Relational Compositional Operators for Analogy Detection
TLDR
A theoretical analysis of generalised bilinear operators that can be used to measure the l2 relational distance between two word-pairs shows that, if the word embed- dings are standardised and uncorrelated, such an operator will be independent of bil inear terms, and can be simplified to a linear form, where PairDiff is a special case.
Bringing Order to Neural Word Embeddings with Embeddings Augmented by Random Permutations (EARP)
TLDR
A new method for incorporating word order information into word vector embedding models by combining the benefits of permutation-based order encoding with the more recent method of skip-gram with negative sampling is presented.
Understanding the Source of Semantic Regularities in Word Embeddings
TLDR
This paper investigates the hypothesis that examples of a lexical relation in a corpus are fundamental to a neural word embedding’s ability to complete analogies involving the relation, and enhances the understanding of neuralword embeddings.
Assessing Lexical-Semantic Regularities in Portuguese Word Embeddings
TLDR
A new test, dubbed TALES, is created with an exclusive focus on Portuguese lexical-semantic relations, acquired from lexical resources, and suggests that word embeddings may be a useful source of information for enriching those resources, something the authors also discuss.
...
...

References

SHOWING 1-10 OF 33 REFERENCES
Take and Took, Gaggle and Goose, Book and Read: Evaluating the Utility of Vector Differences for Lexical Relation Learning
TLDR
It is found that word embeddings capture a surprising amount of information, and that, under suitable supervised training, vector subtraction generalises well to a broad range of relations, including over unseen lexical items.
Rehabilitation of Count-Based Models for Word Vector Representations
TLDR
A systematic study of the use of the Hellinger distance to extract semantic representations from the word co-occurrence statistics of large text corpora shows that this distance gives good performance on word similarity and analogy tasks, with a proper type and size of context, and a dimensionality reduction based on a stochastic low-rank approximation.
Linguistic Regularities in Sparse and Explicit Word Representations
TLDR
It is demonstrated that analogy recovery is not restricted to neural word embeddings, and that a similar amount of relational similarities can be recovered from traditional distributional word representations.
Linguistic Regularities in Continuous Space Word Representations
TLDR
The vector-space word representations that are implicitly learned by the input-layer weights are found to be surprisingly good at capturing syntactic and semantic regularities in language, and that each relationship is characterized by a relation-specific vector offset.
GloVe: Global Vectors for Word Representation
TLDR
A new global logbilinear regression model that combines the advantages of the two major model families in the literature: global matrix factorization and local context window methods and produces a vector space with meaningful substructure.
Learning Effective Word Embedding using Morphological Word Similarity
TLDR
A novel neural network architecture is introduced that leverages both contextual information and morphological word similarity to learn word embeddings and is able to refine the pre-defined morphological knowledge and obtain more accurate word similarity.
Improving Distributional Similarity with Lessons Learned from Word Embeddings
TLDR
It is revealed that much of the performance gains of word embeddings are due to certain system design choices and hyperparameter optimizations, rather than the embedding algorithms themselves, and these modifications can be transferred to traditional distributional models, yielding similar gains.
Analogy-based detection of morphological and semantic relations with word embeddings: what works and what doesn’t.
TLDR
This study applies the widely used vector offset method to 4 types of linguistic relations: inflectional and derivational morphology, and lexicographic and encyclopedic semantics, and systematically examines how accuracy for different categories is affected by window size and dimensionality of the SVD-based word embeddings.
A Uniform Approach to Analogies, Synonyms, Antonyms, and Associations
TLDR
A supervised corpus-based machine learning algorithm is introduced for classifying analogous word pairs and it is shown that it can solve multiple-choice SAT analogy questions, TOEFL synonyms questions, ESL synonym-antonym questions, and similar-associated-both questions from cognitive psychology.
Category Enhanced Word Embedding
TLDR
This paper incorporates category information of documents in the learning of word representations and proposes models that outperform several state-of-the-art models in word analogy and word similarity tasks and learns high-quality category embeddings that reflect topical meanings.
...
...