Eliciting Knowledge from Language Models Using Automatically Generated Prompts

@article{Shin2020ElicitingKF,
  title={Eliciting Knowledge from Language Models Using Automatically Generated Prompts},
  author={Taylor Shin and Yasaman Razeghi and Robert L Logan IV and Eric Wallace and Sameer Singh},
  journal={ArXiv},
  year={2020},
  volume={abs/2010.15980}
}
The remarkable success of pretrained language models has motivated the study of what kinds of knowledge these models learn during pretraining. Reformulating tasks as fill-in-the-blanks problems (e.g., cloze tests) is a natural approach for gauging such knowledge, however, its usage is limited by the manual effort and guesswork required to write suitable prompts. To address this, we develop AutoPrompt, an automated method to create prompts for a diverse set of tasks, based on a gradient-guided… Expand

Figures and Tables from this paper

Learning How to Ask: Querying LMs with Mixtures of Soft Prompts
TLDR
This work explores the idea of learning prompts by gradient descent—either fine-tuning prompts taken from previous work, or starting from random initialization, showing that the implicit factual knowledge in language models was previously underestimated. Expand
Prompt-Learning for Fine-Grained Entity Typing
TLDR
This work proposes a self-supervised strategy that carries out distribution-level optimization in prompt-learning to automatically summarize the information of entity types in fully supervised, few-shot and zero-shot scenarios. Expand
Knowledgeable Prompt-tuning: Incorporating Knowledge into Prompt Verbalizer for Text Classification
TLDR
This work focuses on incorporating external knowledge into the verbalizer, forming a knowledgeable prompt Tuning (KPT), to improve and stabilize prompttuning. Expand
Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm
TLDR
It is suggested that the function of few-shot examples in these cases is better described as locating an already learned task rather than meta-learning, which motivates rethinking the role of prompts in controlling and evaluating powerful language models. Expand
Knowledgeable or Educated Guess? Revisiting Language Models as Knowledge Bases
TLDR
It is found that previous decent performance mainly owes to the biased prompts which overfit dataset artifacts, and incorporating illustrative cases and external contexts improve knowledge prediction mainly due to entity type guidance and golden answer leakage. Expand
Reordering Examples Helps during Priming-based Few-Shot Learning
TLDR
This work introduces PERO (Prompting with Examples in the Right Order), where it is shown that PERO can learn to generalize efficiently using as few as 10 examples, in contrast to existing approaches. Expand
AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing
TLDR
This comprehensive survey paper explains various core concepts like pretraining, Pretraining methods, pretraining tasks, embeddings and downstream adaptation methods, presents a new taxonomy of T-PTLMs and gives brief overview of various benchmarks including both intrinsic and extrinsic. Expand
Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity
TLDR
This work uses the generative nature of the language models to construct an artificial development set and based on entropy statistics of the candidate permutations from this set the authors identify performant prompts and improves upon GPT-family models by on average 13% relative across eleven different established text classification tasks. Expand
Language Models are Few-shot Multilingual Learners
TLDR
It is shown that, given a few English examples as context, pre-trained language models can predict not only English test samples but also non-English ones, and they are competitive compared to the existing state-of-the-art cross-lingual models and translation models. Expand
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
TLDR
An analysis framework is proposed that links the pretraining and downstream tasks with an underlying latent variable generative model of text — the downstream classifier must recover a function of the posterior distribution over the latent variables. Expand
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 43 REFERENCES
Language Models as Knowledge Bases?
TLDR
An in-depth analysis of the relational knowledge already present (without fine-tuning) in a wide range of state-of-the-art pretrained language models finds that BERT contains relational knowledge competitive with traditional NLP methods that have some access to oracle knowledge. Expand
How Can We Know What Language Models Know?
TLDR
This paper proposes mining-based and paraphrasing-based methods to automatically generate high-quality and diverse prompts, as well as ensemble methods to combine answers from different prompts to provide a tighter lower bound on what LMs know. Expand
Language Models are Unsupervised Multitask Learners
TLDR
It is demonstrated that language models begin to learn these tasks without any explicit supervision when trained on a new dataset of millions of webpages called WebText, suggesting a promising path towards building language processing systems which learn to perform tasks from their naturally occurring demonstrations. Expand
A Simple Method for Commonsense Reasoning
TLDR
Key to this method is the use of language models, trained on a massive amount of unlabled data, to score multiple choice questions posed by commonsense reasoning tests, which outperform previous state-of-the-art methods by a large margin. Expand
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
TLDR
A benchmark of nine diverse NLU tasks, an auxiliary dataset for probing models for understanding of specific linguistic phenomena, and an online platform for evaluating and comparing models, which favors models that can represent linguistic knowledge in a way that facilitates sample-efficient learning and effective knowledge-transfer across tasks. Expand
Linguistic Knowledge and Transferability of Contextual Representations
TLDR
It is found that linear models trained on top of frozen contextual representations are competitive with state-of-the-art task-specific models in many cases, but fail on tasks requiring fine-grained linguistic knowledge. Expand
Unsupervised Question Answering by Cloze Translation
TLDR
It is found that modern QA models can learn to answer human questions surprisingly well using only synthetic training data, and is demonstrated that, without using the SQuAD training data at all, this approach achieves 56.4 F1 on SQuad v1. Expand
Inducing Relational Knowledge from BERT
TLDR
This work proposes a methodology for distilling relational knowledge from a pre-trained language model that fine-tune a language model to predict whether a given word pair is likely to be an instance of some relation, when given an instantiated template for that relation as input. Expand
Why Do Masked Neural Language Models Still Need Common Sense Knowledge?
TLDR
A test that measures what types of common sense knowledge do pretrained MNLMs understand is proposed and it is experimentally demonstrated that existing MNLM-based models can be elevated by combining knowledge from an external common sense repository. Expand
A large annotated corpus for learning natural language inference
TLDR
The Stanford Natural Language Inference corpus is introduced, a new, freely available collection of labeled sentence pairs, written by humans doing a novel grounded task based on image captioning, which allows a neural network-based model to perform competitively on natural language inference benchmarks for the first time. Expand
...
1
2
3
4
5
...