Language Modelling Makes Sense: Propagating Representations through WordNet for Full-Coverage Word Sense Disambiguation

@article{Loureiro2019LanguageMM,
  title={Language Modelling Makes Sense: Propagating Representations through WordNet for Full-Coverage Word Sense Disambiguation},
  author={Daniel Loureiro and Al{\'i}pio M{\'a}rio Jorge},
  journal={ArXiv},
  year={2019},
  volume={abs/1906.10007}
}
Contextual embeddings represent a new generation of semantic representations learned from Neural Language Modelling (NLM) that addresses the issue of meaning conflation hampering traditional word embeddings. [] Key Method As a result, a simple Nearest Neighbors (k-NN) method using our representations is able to consistently surpass the performance of previous systems using powerful neural sequencing models. We also analyse the robustness of our approach when ignoring part-of-speech and lemma features…

Figures and Tables from this paper

LMMS Reloaded: Transformer-based Sense Embeddings for Disambiguation and Beyond
With More Contexts Comes Better Performance: Contextualized Sense Embeddings for All-Round Word Sense Disambiguation
TLDR
ARES representations enable a simple 1 Nearest-Neighbour algorithm to outperform state-of-the-art models, not only in the English Word Sense Disambiguation task, but also in the multilingual one, whilst training on sense-annotated data in English only.
Learning Sense-Specific Static Embeddings using Contextualised Word Embeddings as a Proxy
TLDR
Experimental results on multiple benchmarks for word sense disambiguation and sense discrimination tasks show that CDES can accurately learn sensespecific static embeddings reporting comparable performance to the current state-of-the-art sense embedDings.
Learning Sense-Specific Static Embeddings using Contextualised Word Embeddings as a Proxy
TLDR
Experimental results on multiple benchmarks for word sense disambiguation and sense discrimination tasks show that CDES can accurately learn sensespecific static embeddings reporting comparable performance to the current state-of-the-art sense embedDings.
Sparsity Makes Sense: Word Sense Disambiguation Using Sparse Contextualized Word Representations
TLDR
This paper introduces an information theory-inspired synset representation based on the co-occurrence of word senses and non-zero coordinates for word forms which allows to achieve an aggregated F-score of 78.8 over a combination of five standard word sense disambiguating benchmark datasets.
Language Models and Word Sense Disambiguation: An Overview and Analysis
TLDR
An in-depth quantitative and qualitative analysis of the celebrated BERT model with respect to lexical ambiguity reveals that in some cases language models come close to solving coarse-grained noun disambiguation under ideal conditions in terms of availability of training data and computing resources.
MuLaN: Multilingual Label propagatioN for Word Sense Disambiguation
TLDR
A semisupervised approach based upon a novel label propagation scheme, which, by jointly leveraging contextualized word embeddings and the multilingual information enclosed in a knowledge base, projects sense labels from a high-resource language, i.e., English, to lower-resourced ones.
Improved Word Sense Disambiguation with Enhanced Sense Representations
TLDR
This paper focuses on enhancing the sense representations via incorporating synonyms, example phrases or sentences showing usage of word senses, and sense gloss of hypernyms, and shows that incorporating such additional information boosts the performance on WSD.
Conception: Multilingually-Enhanced, Human-Readable Concept Vector Representations
TLDR
This paper proposes Conception, a novel technique for building language-independent vector representations of concepts which places multilinguality at its core while retaining explicit relationships between concepts, and results in high-coverage representations that outperform the state of the art in multilingual and cross-lingual Semantic Word Similarity and Word Sense Disambiguation.
...
...

References

SHOWING 1-10 OF 30 REFERENCES
Embeddings for Word Sense Disambiguation: An Evaluation Study
TLDR
This work proposes different methods through which word embeddings can be leveraged in a state-of-the-art supervised WSD system architecture, and performs a deep analysis of how different parameters affect performance.
Incorporating Glosses into Neural Word Sense Disambiguation
TLDR
GAS models the semantic relationship between the context and the gloss in an improved memory network framework, which breaks the barriers of the previous supervised methods and knowledge-based methods.
Leveraging Gloss Knowledge in Neural Word Sense Disambiguation by Hierarchical Co-Attention
TLDR
This paper introduces a co-attention mechanism to generate co-dependent representations for the context and gloss, and extends the attention mechanism in a hierarchical fashion in order to capture both word-level and sentence-level information.
An Enhanced Lesk Word Sense Disambiguation Algorithm through a Distributional Semantic Model
TLDR
A new Word Sense Disambiguation (WSD) algorithm which extends two well-known variations of the Lesk WSD method which relies on the use of a word similarity function defined on a distributional semantic space to compute the gloss-context overlap.
context2vec: Learning Generic Context Embedding with Bidirectional LSTM
TLDR
This work presents a neural model for efficiently learning a generic context embedding function from large corpora, using bidirectional LSTM, and suggests they could be useful in a wide variety of NLP tasks.
Deep Contextualized Word Representations
TLDR
A new type of deep contextualized word representation is introduced that models both complex characteristics of word use and how these uses vary across linguistic contexts, allowing downstream models to mix different types of semi-supervision signals.
A Unified Model for Word Sense Representation and Disambiguation
TLDR
A unified model for joint word sense representation and disambiguation, which will assign distinct representations for each word sense and improves the performance of contextual word similarity compared to existing WSR methods, outperforms state-of-the-art supervised methods on domainspecific WSD, and achieves competitive performance on coarse-grained all-words WSD.
Semi-supervised Word Sense Disambiguation with Neural Models
TLDR
This paper studies WSD with a sequence learning neural net, LSTM, to better capture the sequential and syntactic patterns of the text and demonstrates state-of-the-art results, especially on verbs.
Improving the Coverage and the Generalization Ability of Neural Word Sense Disambiguation through Hypernymy and Hyponymy Relationships
TLDR
This article proposes a new method that takes advantage of the knowledge present in WordNet, and especially the hypernymy and hyponymy relationships between synsets, in order to reduce the number of different sense tags that are necessary to disambiguate all words of the lexical database.
...
...