Evaluating Word Embeddings with Categorical Modularity

  title={Evaluating Word Embeddings with Categorical Modularity},
  author={S'ilvia Casacuberta and Karina Halevy and Dami{\'a}n E. Blasi},
We introduce categorical modularity, a novel low-resource intrinsic metric to evaluate word embedding quality. Categorical modularity is a graph modularity metric based on the k-nearest neighbor graph constructed with embedding vectors of words from a fixed set of semantic categories, in which the goal is to measure the proportion of words that have nearest neighbors within the same categories. We use a core set of 500 words belonging to 59 neurobiologically motivated semantic categories in 29… 

Figures and Tables from this paper


A Resource-Free Evaluation Metric for Cross-Lingual Word Embeddings Based on Graph Modularity
It is shown through experiments that modularity can serve as an intrinsic validation metric to improve unsupervised cross-lingual word embeddings, particularly on distant language pairs in low-resource settings.
Revisiting Correlations between Intrinsic and Extrinsic Evaluations of Word Embeddings
There are strong correlations between intrinsic and extrinsic evaluations, and the performance of different tasks can be affected by training corpora and context features to varying degrees.
GloVe: Global Vectors for Word Representation
A new global logbilinear regression model that combines the advantages of the two major model families in the literature: global matrix factorization and local context window methods and produces a vector space with meaningful substructure.
Evaluating word embeddings with fMRI and eye-tracking
It is argued below that the only meaningful evaluation procedure is comparison with measures of human word processing in the wild, and a practical procedure is presented, evaluating word embeddings as features in a multi-dimensional regression model predicting brain imaging or eyetracking word-level aggregate statistics.
Improving Distributional Similarity with Lessons Learned from Word Embeddings
It is revealed that much of the performance gains of word embeddings are due to certain system design choices and hyperparameter optimizations, rather than the embedding algorithms themselves, and these modifications can be transferred to traditional distributional models, yielding similar gains.
SemEval-2017 Task 2: Multilingual and Cross-lingual Semantic Word Similarity
Results show that systems that combine statistical knowledge from text corpora, in the form of word embeddings, and external knowledge from lexical resources are best performers in both subtasks.
Enriching Word Vectors with Subword Information
A new approach based on the skipgram model, where each word is represented as a bag of character n-grams, with words being represented as the sum of these representations, which achieves state-of-the-art performance on word similarity and analogy tasks.
A Survey of Word Embeddings Evaluation Methods
An extensive overview of the field of word embeddings evaluation is presented, highlighting main problems and proposing a typology of approaches to evaluation, summarizing 16 intrinsic methods and 12 extrinsic methods.
CogniVal: A Framework for Cognitive Word Embedding Evaluation
This paper presents the first multi-modal framework for evaluating English word representations based on cognitive lexical semantics, and finds strong correlations in the results between cognitive datasets, across recording modalities and to their performance on extrinsic NLP tasks.
subs2vec: Word embeddings from subtitles in 55 languages
A novel collection of word embeddings, numerical representations of lexical semantics, in 55 languages, trained on a large corpus of pseudo-conversational speech transcriptions from television shows and movies, and a novel evaluation method of particular relevance to psycholinguists.