AutoExtend: Extending Word Embeddings to Embeddings for Synsets and Lexemes

@inproceedings{Rothe2015AutoExtendEW,
  title={AutoExtend: Extending Word Embeddings to Embeddings for Synsets and Lexemes},
  author={Sascha Rothe and Hinrich Sch{\"u}tze},
  booktitle={ACL},
  year={2015}
}
We present \textit{AutoExtend}, a system to learn embeddings for synsets and lexemes. It is flexible in that it can take any word embeddings as input and does not need an additional training corpus. The synset/lexeme embeddings obtained live in the same vector space as the word embeddings. A sparse tensor formalization guarantees efficiency and parallelizability. We use WordNet as a lexical resource, but AutoExtend can be easily applied to other resources like Freebase. AutoExtend achieves… 

Figures and Tables from this paper

Visualising WordNet Embeddings: some preliminary results
TLDR
It is shown that the t-SNE co-ordinates can be used to reveal interesting semantic relations between word senses, and a new method is proposed that uses the simple x,y coordinates to compute semantic similarity.
Learning Sense-Specific Static Embeddings using Contextualised Word Embeddings as a Proxy
TLDR
Experimental results on multiple benchmarks for word sense disambiguation and sense discrimination tasks show that CDES can accurately learn sensespecific static embeddings reporting comparable performance to the current state-of-the-art sense embedDings.
Learning Sense-Specific Static Embeddings using Contextualised Word Embeddings as a Proxy
TLDR
Experimental results on multiple benchmarks for word sense disambiguation and sense discrimination tasks show that CDES can accurately learn sensespecific static embeddings reporting comparable performance to the current state-of-the-art sense embedDings.
Ontology-Aware Token Embeddings for Prepositional Phrase Attachment
TLDR
Using context-sensitive embeddings in a model for predicting prepositional phrase (PP) attachments improves the accuracy of the PP attachment model by 5.4% absolute points, which amounts to a 34.
Embedding Words and Senses Together via Joint Knowledge-Enhanced Training
TLDR
This work proposes a new model which learns word and sense embeddings jointly and exploits large corpora and knowledge from semantic networks in order to produce a unified vector space of word and senses.
context2vec: Learning Generic Context Embedding with Bidirectional LSTM
TLDR
This work presents a neural model for efficiently learning a generic context embedding function from large corpora, using bidirectional LSTM, and suggests they could be useful in a wide variety of NLP tasks.
HybridVec : Hybrid Distributional and Definitional Word Vectors
TLDR
Results suggest that a combination of distributional vectors (GloVE embeddings) and definitional word vectors produced from an autoencoder provide an improvement for Neural Machine Translation and warrants further testing.
Inducing Conceptual Embedding Spaces from Wikipedia
TLDR
This paper explores an automatic expansion of Google’s pretrained vectors using Wikipedia, adding millions of concepts and named entities in over 270 languages to reside in the same vector space, thus flexibly facilitating cross-lingual semantic applications.
CoKE : Word Sense Induction Using Contextualized Knowledge Embeddings
TLDR
This work introduces a new approach that can induce polysemy to any pre-defined embedding space by jointly grounding contextualized sense representations learned from sense-tagged corpora and word embeddings to a knowledge base without the need for re-training.
Inducing Conceptual Embedding Spaces from Wikipedia
TLDR
An automatic expansion of Google's pre-trained vectors using Wikipedia is explored, adding millions of concepts and named entities in over 270 languages to reside in the same vector space, thus flexibly facilitating cross-lingual semantic applications.
...
...

References

SHOWING 1-10 OF 49 REFERENCES
Learning Sense-specific Word Embeddings By Exploiting Bilingual Resources
TLDR
This paper addresses the problem of representing words with multiple and sense-specific embeddings, which are learned from bilingual parallel data by representing each word with a single embedding, which fails to capture polysemy.
Improving Word Representations via Global Context and Multiple Word Prototypes
TLDR
A new neural network architecture is presented which learns word embeddings that better capture the semantics of words by incorporating both local and global document context, and accounts for homonymy and polysemy by learning multiple embedDings per word.
Re-embedding words
TLDR
This work proposes a method that takes as input an existing embedding, some labeled data, and produces an embedding in the same space, but with a better predictive performance in the supervised task.
Efficient Non-parametric Estimation of Multiple Embeddings per Word in Vector Space
TLDR
An extension to the Skip-gram model that efficiently learns multiple embeddings per word type is presented, and its scalability is demonstrated by training with one machine on a corpus of nearly 1 billion tokens in less than 6 hours.
Knowledge-Powered Deep Learning for Word Embedding
TLDR
This study explores the capacity of leveraging morphological, syntactic, and semantic knowledge to achieve high-quality word embeddings, and explores these types of knowledge to define new basis for word representation, provide additional input information, and serve as auxiliary supervision in deep learning.
GloVe: Global Vectors for Word Representation
TLDR
A new global logbilinear regression model that combines the advantages of the two major model families in the literature: global matrix factorization and local context window methods and produces a vector space with meaningful substructure.
Improving Lexical Embeddings with Semantic Knowledge
TLDR
This work proposes a new learning objective that incorporates both a neural language model objective (Mikolov et al., 2013) and prior knowledge from semantic resources to learn improved lexical semantic embeddings.
Better Word Representations with Recursive Neural Networks for Morphology
TLDR
This paper combines recursive neural networks, where each morpheme is a basic unit, with neural language models to consider contextual information in learning morphologicallyaware word representations and proposes a novel model capable of building representations for morphologically complex words from their morphemes.
Learning Structured Embeddings of Knowledge Bases
TLDR
A learning process based on an innovative neural network architecture designed to embed any of these symbolic representations into a more flexible continuous vector space in which the original knowledge is kept and enhanced would allow data from any KB to be easily used in recent machine learning methods for prediction and information retrieval.
BabelNet: Building a Very Large Multilingual Semantic Network
TLDR
A very large, wide-coverage multilingual semantic network that integrates lexicographic and encyclopedic knowledge from WordNet and Wikipedia and Machine Translation is also applied to enrich the resource with lexical information for all languages.
...
...