• Corpus ID: 219176582

A Comparative Study of Lexical Substitution Approaches based on Neural Language Models

  title={A Comparative Study of Lexical Substitution Approaches based on Neural Language Models},
  author={Nikolay Arefyev and Boris Sheludko and A. V. Podolskiy and Alexander Panchenko},
Lexical substitution in context is an extremely powerful technology that can be used as a backbone of various NLP applications, such as word sense induction, lexical relation extraction, data augmentation, etc. In this paper, we present a large-scale comparative study of popular neural language and masked language models (LMs and MLMs), such as context2vec, ELMo, BERT, XLNet, applied to the task of lexical substitution. We show that already competitive results achieved by SOTA LMs/MLMs can be… 

Figures and Tables from this paper

CILex: An Investigation of Context Information for Lexical Substitution Methods

CILex is proposed, which uses contextual sentence embeddings along with methods that capture additional context information complimenting contextual word embedding for lexical substitution to ensure the semantic consistency of a substitute with the target word while maintaining the overall meaning of the sentence.

Swords: A Benchmark for Lexical Substitution with Improved Data Coverage and Quality

A context-free thesaurus is used to produce candidates and rely on human judgement to determine contextual appropriateness, and a new benchmark for lexical substitution is released, which has 3x as many substitutes per target word for the same level of quality.

LexSubCon: Integrating Knowledge from Lexical Resources into Contextual Embeddings for Lexical Substitution

This work introduces LexSubCon, an end-to-end lexical substitution framework based on contextual embedding models that can identify highly-accurate substitute candidates by combining contextual information with knowledge from structured lexical resources.

Combination of Contextualized and Non-Contextualized Layers for Lexical Substitution in French

An application of the state-of-the-art method based on BERT in French and a novel method using contextualized and non-contextualized layers to increase the suggestion of words having a lower probability in a given context but that are more semantically similar are proposed.

Representing word meaning in context via lexical substitutes

The experiments show the validity of LS for word meaning in context representation and justify the use of system-produced substitutes for WSI, and investigate to what extent the results translate to the fundamental semantic task of word sense induction.



Combining Lexical Substitutes in Neural Word Sense Induction

This work improves the approach to WSI proposed by Amrami and Goldberg (2018) based on clustering of lexical substitutes for an ambiguous word in a particular context obtained from neural language models by proposing methods for combining information from left and right context and similarity to the ambiguous word.

A Simple Word Embedding Model for Lexical Substitution

A simple model for lexical substitution, based on the popular skip-gram word embedding model, which is efficient, very simple to implement, and at the same time achieves state-ofthe-art results in an unsupervised setting.

BERT-based Lexical Substitution

This work proposes an end-to-end BERT-based lexical substitution approach which can propose and validate substitute candidates without using any annotated data or manually curated resources and achieves the state-of-the-art results in both LS07 and LS14 benchmarks.

Language Transfer Learning for Supervised Lexical Substitution

This work combines state-of-the-art unsupervised features obtained from syntactic word embeddings and distributional thesauri in a supervised delexicalized ranking system and shows that a supervised system can be trained effectively, even if training and evaluation data are from different languages.

Supervised All-Words Lexical Substitution using Delexicalized Features

A supervised lexical substitution system that does not use separate classifiers per word and is therefore applicable to any word in the vocabulary is proposed, which improves over the state of the art in the LexSub Best-Precision metric and the Generalized Average Precision measure.

Learning to Rank Lexical Substitutions

This paper customize and evaluate several learning-to-rank models to the lexical substitution task, including classification-based and regression-based approaches, and finds that the best models significantly advance the state-of-the-art.

Towards better substitution-based word sense induction

This work extends the previous method to support a dynamic rather than a fixed number of clusters as supported by other prominent methods, and proposes a method for interpreting the resulting clusters by associating them with their most informative substitutes.

A Comparison of Context-sensitive Models for Lexical Substitution

It is shown that powerful contextualized word representations, which give high performance in several semantics-related tasks, deal less well with the subtle in-context similarity relationships needed for substitution.

Deep Contextualized Word Representations

A new type of deep contextualized word representation is introduced that models both complex characteristics of word use and how these uses vary across linguistic contexts, allowing downstream models to mix different types of semi-supervision signals.

Word Sense Induction with Neural biLM and Symmetric Patterns

The combination of the RNN-LM and the dynamic symmetric patterns results in strong substitute vectors for WSI, allowing to surpass the current state-of-the-art on the SemEval 2013 WSI shared task by a large margin.