COMPS: Conceptual Minimal Pair Sentences for testing Property Knowledge and Inheritance in Pre-trained Language Models

@article{Misra2022COMPSCM,
  title={COMPS: Conceptual Minimal Pair Sentences for testing Property Knowledge and Inheritance in Pre-trained Language Models},
  author={Kanishka Misra and Julia Taylor Rayz and Allyson Ettinger},
  journal={ArXiv},
  year={2022},
  volume={abs/2210.01963}
}
A characteristic feature of human semantic memory is its ability to not only store and retrieve the properties of concepts observed through experience, but to also facilitate the inheritance of properties ( can breathe ) from superordinate concepts ( ANIMAL ) to their subordinates ( DOG )—i.e. demonstrate property inheritance . In this paper, we present COMPS , a collection of minimal pair sentences that jointly tests pre-trained language models (PLMs) on their ability to attribute properties… 

Figures and Tables from this paper

Large Language Models Can Be Easily Distracted by Irrelevant Context

This work investigates the distractibility of large language models, i.e., how the model problem-solving accuracy can be affected by irrelevant context, and introduces Grade-School Math with Irrelevant Context (GSM-IC), an arithmetic reasoning dataset with irrelevant information in the problem description.

Dissociating language and thought in large language models: a cognitive perspective

Short abstract (100 words): Large language models (LLMs) have come closest among all models to date to mastering human language, yet opinions about their capabilities remain split. Here, we evaluate

Counteracts: Testing Stereotypical Representation in Pre-trained Language Models

The results indicate that pre-trained language models show a certain amount of robustness when using unrelated knowledge, and prefer shallow linguistic cues, such as word position and syntactic structure, to alter the internal stereotypical representation.

Language model acceptability judgements are not always robust to context

This paper investigates the stability of language models’ performance on targeted syntactic evaluations as they vary properties of the input context: the length of the context, the types of syntactic phenomena it contains, and whether or not there are violations of grammaticality.

Can language models handle recursively nested grammatical structures? A case study on comparing models and humans

How should we compare the capabilities of language models and humans? Here, I consider a case study: processing of recursively nested grammatical structures. Prior work has suggested that language

References

SHOWING 1-10 OF 71 REFERENCES

RoBERTa: A Robustly Optimized BERT Pretraining Approach

It is found that BERT was significantly undertrained, and can match or exceed the performance of every model published after it, and the best model achieves state-of-the-art results on GLUE, RACE and SQuAD.

Language Models are Unsupervised Multitask Learners

It is demonstrated that language models begin to learn these tasks without any explicit supervision when trained on a new dataset of millions of webpages called WebText, suggesting a promising path towards building language processing systems which learn to perform tasks from their naturally occurring demonstrations.

The Centre for Speech, Language and the Brain (CSLB) concept property norms

A new and large set of property norms that are designed to be a more flexible tool to meet the demands of many different disciplines interested in conceptual knowledge representation, from cognitive psychology to computational linguistics are introduced.

Probing Neural Language Models for Human Tacit Assumptions

This work constructs a diagnostic set of word prediction prompts to evaluate whether recent neural contextualized language models trained on large text corpora capture STAs, and finds models to be profoundly effective at retrieving concepts given associated properties.

Teaching Pre-Trained Models to Systematically Reason Over Implicit Knowledge

This work provides a first demonstration that LMs can be trained to reliably perform systematic reasoning combining both implicit, pre-trained knowledge and explicit natural language statements, and demonstrates that models learn to effectively perform inference which involves implicit taxonomic and world knowledge, chaining and counting.

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

A new language representation model, BERT, designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers, which can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks.

The Big Book of Concepts

Concepts embody our knowledge of the kinds of things there are in the world. Tying our past experiences to our present interactions with the environment, they enable us to recognize and understand

A Property Induction Framework for Neural Language Models

A framework that uses neural-network language models (LMs) to perform property induction—a task in which humans generalize novel property knowledge from one or more concepts to others, suggesting the presence of a taxonomic bias in their representations is presented.

Mapping Language Models to Grounded Conceptual Spaces

Meaning without reference in large language models

The widespread success of large language models (LLMs) has been met with skepticism that they possess anything like human concepts or meanings. Contrary to claims that LLMs possess no meaning
...