Word Sense Induction with Neural biLM and Symmetric Patterns

@inproceedings{Amrami2018WordSI,
  title={Word Sense Induction with Neural biLM and Symmetric Patterns},
  author={Asaf Amrami and Y. Goldberg},
  booktitle={EMNLP},
  year={2018}
}
An established method for Word Sense Induction (WSI) uses a language model to predict probable substitutes for target words, and induces senses by clustering these resulting substitute vectors. We replace the ngram-based language model (LM) with a recurrent one. Beyond being more accurate, the use of the recurrent LM allows us to effectively query it in a creative way, using what we call dynamic symmetric patterns. The combination of the RNN-LM and the dynamic symmetric patterns results in… Expand
Towards better substitution-based word sense induction
TLDR
This work extends the previous method to support a dynamic rather than a fixed number of clusters as supported by other prominent methods, and proposes a method for interpreting the resulting clusters by associating them with their most informative substitutes. Expand
Combining Neural Language Models for Word Sense Induction
TLDR
This work proposes methods of combining left and right contexts, resulting in better substitutes generated, and establishes new state-of-the-art level of WSI for the Russian language on two RUSSE 2018 datasets. Expand
Word Sense Disambiguation for 158 Languages using Word Embeddings Only
TLDR
This paper uses a method that takes as input a standard pre-trained word embedding model and induces a fully-fledged word sense inventory, which can be used for disambiguation in context to induce a collection of sense inventories for 158 languages. Expand
PolyLM: Learning about Polysemy through Language Modeling
TLDR
PolyLM is introduced, a method which formulates the task of learning sense embeddings as a language modeling problem, allowing contextualization techniques to be applied, and performs considerably better than previous sense embedding techniques, and matches the current state-of-the-art specialized WSI method despite having six times fewer parameters. Expand
A Comparative Study of Lexical Substitution Approaches based on Neural Language Models
TLDR
A large-scale comparative study of popular neural language and masked language models (LMs and MLMs), such as context2vec, ELMo, BERT, XLNet, applied to the task of lexical substitution shows that already competitive results achieved by SOTA LMs/MLMs can be further improved if information about the target word is injected properly. Expand
BOS at SemEval-2020 Task 1: Word Sense Induction via Lexical Substitution for Lexical Semantic Change Detection
TLDR
While WSI-based solution performs better in Sub task 1, which requires binary decisions, the second solution outperforms it in Subtask 2 and obtains the 3rd best result in this subtask. Expand
Always Keep your Target in Mind: Studying Semantics and Improving Performance of Neural Lexical Substitution
TLDR
A large-scale comparative study of lexical substitution methods employing both rather old and most recent language and masked language models (LMs and MLMs), such as context2vec, ELMo, BERT, RoBERTa, XLNet, shows that already competitive results achieved by SOTA LMs/MLMs can be further substantially improved if information about the target word is injected properly. Expand
Language Models and Word Sense Disambiguation: An Overview and Analysis
TLDR
An in-depth quantitative and qualitative analysis of the celebrated BERT model with respect to lexical ambiguity reveals that in some cases language models come close to solving coarse-grained noun disambiguation under ideal conditions in terms of availability of training data and computing resources. Expand
Density Measure in Context Clustering for Distributional Semantics of Word Sense Induction
TLDR
This paper shows that utilizing the silhouette score normally used as an internal evaluation metric to measure the clusters’ density in a parametric clustering algorithm, such as K-means, in the WSI task captures words’ senses better than the state-of-the-art models. Expand
ALaSca: an Automated approach for Large-Scale Lexical Substitution
The lexical substitution task aims at finding suitable replacements for words in context. It has proved to be useful in several areas, such as word sense induction and text simplification, as well asExpand
...
1
2
3
...

References

SHOWING 1-10 OF 18 REFERENCES
Semi-supervised Word Sense Disambiguation with Neural Models
TLDR
This paper studies WSD with a sequence learning neural net, LSTM, to better capture the sequential and syntactic patterns of the text and demonstrates state-of-the-art results, especially on verbs. Expand
A Sense-Topic Model for Word Sense Induction with Unsupervised Data Enrichment
TLDR
A sense-topic model is proposed for WSI, which treats sense and topic as two separate latent variables to be inferred jointly, and achieves significant improvements over the previous state-of-the-art, achieving the best results reported to date on the SemEval-2013 WSI task. Expand
AI-KU: Using Substitute Vectors and Co-Occurrence Modeling For Word Sense Induction and Disambiguation
TLDR
This work proposes a system that creates a substitute vector for each target word from the most likely substitutes suggested by a statistical language model and outperforms the other systems on graded word sense induction task in SemEval-2013. Expand
Neural Sequence Learning Models for Word Sense Disambiguation
TLDR
This work proposes and studies in depth a series of end-to-end neural architectures directly tailored to the task, from bidirectional Long Short-Term Memory to encoder-decoder models, and shows that sequence learning enables more versatile all-words models that consistently lead to state-of-the-art results, even against word experts with engineered features. Expand
Symmetric Pattern Based Word Embeddings for Improved Word Similarity Prediction
TLDR
A novel word level vector representation based on symmetric patterns (SPs) that performs exceptionally well on verbs, and a simple combination of the word similarity scores generated by the method and by word2vec results in a superior predictive power over that of each individual model. Expand
unimelb: Topic Modelling-based Word Sense Induction
TLDR
This paper describes a previously-proposed WSI methodology for the task, which is based on a Hierarchical Dirichlet Process (HDP), a nonparametric topic model, which requires no parameter tuning, uses the English ukWaC as an external resource, and achieves encouraging results over the shared task. Expand
Deep Contextualized Word Representations
TLDR
A new type of deep contextualized word representation is introduced that models both complex characteristics of word use and how these uses vary across linguistic contexts, allowing downstream models to mix different types of semi-supervision signals. Expand
context2vec: Learning Generic Context Embedding with Bidirectional LSTM
TLDR
This work presents a neural model for efficiently learning a generic context embedding function from large corpora, using bidirectional LSTM, and suggests they could be useful in a wide variety of NLP tasks. Expand
SemEval-2007 Task 02: Evaluating Word Sense Induction and Discrimination Systems
TLDR
This work reused the SemEval-2007 English lexical sample subtask of task 17, and set up both clustering-style unsupervised evaluation and a supervised evaluation (using the part of the dataset for mapping) to allow for comparison across sense-induction and discrimination systems. Expand
Structured Generative Models of Continuous Features for Word Sense Induction
TLDR
An EM algorithm is described to efficiently estimate model parameters and the Integrated Complete Likelihood criterion is used to automatically estimate the number of senses in a structured generative latent variable model that integrates information from multiple contextual representations for Word Sense Induction. Expand
...
1
2
...