A Two-Stage Masked LM Method for Term Set Expansion

  title={A Two-Stage Masked LM Method for Term Set Expansion},
  author={Guy Kushilevitz and Shaul Markovitch and Yoav Goldberg},
We tackle the task of Term Set Expansion (TSE): given a small seed set of example terms from a semantic class, finding more members of that class. The task is of great practical utility, and also of theoretical utility as it requires generalization from few examples. Previous approaches to the TSE task can be characterized as either distributional or pattern-based. We harness the power of neural masked language models (MLM) and propose a novel TSE algorithm, which combines the pattern-based and… 

Tables from this paper

Automatic Context Pattern Generation for Entity Set Expansion
A context pattern generation module that utilizes autoregressive language models (e.g., GPT-2) to automatically generate high-quality context patterns for entities and proposes the GAPA, a novel ESE framework that leverages the aforementioned GenerAted PAtterns to expand target entities.
Contrastive Learning with Hard Negative Entities for Entity Set Expansion
The ProbExpan is proposed, a novel probabilistic ESE framework utilizing the entity representation obtained by the aforementioned language model to expand entities and outperforms previous state-of-the-art methods.
GausSetExpander: A Simple Approach for Entity Set Expansion
GausSetExpander, an unsupervised approach based on optimal transport techniques to re-frame the problem as choosing the entity that best completes the seed set as an elliptical distribution with a centroid which represents the mean and a spread that is represented by the scale parameter.
Wiki2row – the In’s and Out’s or row suggestion with a Large Scale Knowledge Base
The algorithm is extended to present the first approach to bias-aware row suggestion when table completion is not achievable, that is, when the authors cannot define a complete set of entities.
Rows from Many Sources: Enriching row completions from Wikidata with a pre-trained Language Model
This work presents state-of-the-art results for subject suggestion and gap filling measured on a standard benchmark (WikiTables), and synthesizes additional rows using free text generation via GPT-3.
SynSetExpan: An Iterative Framework for Joint Entity Set Expansion and Synonym Discovery
This work hypothesizes that these two tasks are tightly coupled because two synonymous entities tend to have similar likelihoods of belonging to various semantic classes, and designs SynSetExpan, a novel framework that enables two tasks to mutually enhance each other.
Biased TextRank: Unsupervised Graph-Based Content Extraction
This work presents two applications of Biased TextRank: focused summarization and explanation extraction, and shows that the algorithm leads to improved performance on two different datasets by significant ROUGE-N score margins.


Term Set Expansion based NLP Architect by Intel AI Lab
SetExpander enables users to easily select a seed set of terms, expand it, view the expanded set, validate it, re-expand the validated set and store it, thus simplifying the extraction of domain-specific fine-grained semantic classes.
Language Models as Knowledge Bases?
An in-depth analysis of the relational knowledge already present (without fine-tuning) in a wide range of state-of-the-art pretrained language models finds that BERT contains relational knowledge competitive with traditional NLP methods that have some access to oracle knowledge.
Employing Topic Models for Pattern-based Semantic Class Discovery
This paper studies the employment of topic models to automatically construct semantic classes, taking as the source data a collection of raw semantic classes (RASCs), which were extracted by applying predefined patterns to web pages.
Word Sense Induction with Neural biLM and Symmetric Patterns
The combination of the RNN-LM and the dynamic symmetric patterns results in strong substitute vectors for WSI, allowing to surpass the current state-of-the-art on the SemEval 2013 WSI shared task by a large margin.
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
A new language representation model, BERT, designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers, which can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks.
Language-Independent Set Expansion of Named Entities Using the Web
This paper proposes a novel method for expanding sets of named entities that can be applied to semi-structured documents written in any markup language and in any human language and shows that this system is superior to Google Sets in terms of mean average precision.
Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies
It is concluded that LSTMs can capture a non-trivial amount of grammatical structure given targeted supervision, but stronger architectures may be required to further reduce errors; furthermore, the language modeling signal is insufficient for capturing syntax-sensitive dependencies, and should be supplemented with more direct supervision if such dependencies need to be captured.
BERT Rediscovers the Classical NLP Pipeline
This work finds that the model represents the steps of the traditional NLP pipeline in an interpretable and localizable way, and that the regions responsible for each step appear in the expected sequence: POS tagging, parsing, NER, semantic roles, then coreference.
Automatic Acquisition of Hyponyms from Large Text Corpora
A set of lexico-syntactic patterns that are easily recognizable, that occur frequently and across text genre boundaries, and that indisputably indicate the lexical relation of interest are identified.
sense2vec - A Fast and Accurate Method for Word Sense Disambiguation In Neural Word Embeddings
A novel approach is presented which provides a fast and accurate way for a consuming NLP model to select a sense-disambiguated embedding, which can disambiguate both contrastive senses such as nominal and verbal senses as well as nuanced sensessuch as sarcasm.