Corpus ID: 237532411

Reframing Instructional Prompts to GPTk's Language

  title={Reframing Instructional Prompts to GPTk's Language},
  author={Swaroop Mishra and Daniel Khashabi and Chitta Baral and Yejin Choi and Hannaneh Hajishirzi},
How can model designers turn task instructions into effective prompts for language models? Backed by extensive empirical analysis on GPT3, we observe important features for successful instructional prompts, and propose several reframing techniques for model designers to create such prompts. For example, a complex task can be decomposed into multiple simpler tasks (Fig.1a). We experiment over 12 NLP tasks across 6 diverse categories (question generation, classification, etc.). Our results show… Expand

Figures and Tables from this paper

On Transferability of Prompt Tuning for Natural Language Understanding
The findings show that improving PT with knowledge transfer is possible and promising, while prompts’ crosstask transferability is generally better than the cross-model transferability. Expand
Few-Shot Bot: Prompt-Based Learning for Dialogue Systems
An end-to-end chatbot named the Few-Shot Bot is created, which automatically selects the most appropriate conversational skill, queries different knowledge bases or the internet, and uses the retrieved knowledge to generate a human-like response, all using only few dialogue examples per skill. Expand
MetaICL: Learning to Learn In Context
We introduce MetaICL (Meta-training for InContext Learning), a new meta-training framework for few-shot learning where a pretrained language model is tuned to do in-context learning on a large set ofExpand


Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm
It is suggested that the function of few-shot examples in these cases is better described as locating an already learned task rather than meta-learning, which motivates rethinking the role of prompts in controlling and evaluating powerful language models. Expand
Natural Instructions: Benchmarking Generalization to New Tasks from Natural Language Instructions
This work uses the existing NLP datasets and the instructions used to crowdsource them to create NATURALINSTRUCTIONS, a dataset of instructions and task-specific input/output data that indicates that the existing models indeed benefit from instructions and hence, show improved generalization to new tasks. Expand
Cutting Down on Prompts and Parameters: Simple Few-Shot Learning with Language Models
This work shows that finetuning LMs in the few- shot setting can considerably reduce the need for prompt engineering, and recommends finetuned LMs for few-shot learning as it is more accurate, robust to different prompts, and can be made nearly as efficient as using frozen LMs. Expand
Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity
This work uses the generative nature of the language models to construct an artificial development set and based on entropy statistics of the candidate permutations from this set the authors identify performant prompts and improves upon GPT-family models by on average 13% relative across eleven different established text classification tasks. Expand
Reordering Examples Helps during Priming-based Few-Shot Learning
This work introduces PERO (Prompting with Examples in the Right Order), where it is shown that PERO can learn to generalize efficiently using as few as 10 examples, in contrast to existing approaches. Expand
The Power of Scale for Parameter-Efficient Prompt Tuning
This work explores “prompt tuning”, a simple yet effective mechanism for learning “soft prompts” to condition frozen language models to perform specific downstream tasks, and shows that conditioning a frozen model with soft prompts confers benefits in robustness to domain transfer, as compared to full model tuning. Expand
Eliciting Knowledge from Language Models Using Automatically Generated Prompts
The remarkable success of pretrained language models has motivated the study of what kinds of knowledge these models learn during pretraining. Reformulating tasks as fill-in-the-blanks problemsExpand
Language Models are Unsupervised Multitask Learners
It is demonstrated that language models begin to learn these tasks without any explicit supervision when trained on a new dataset of millions of webpages called WebText, suggesting a promising path towards building language processing systems which learn to perform tasks from their naturally occurring demonstrations. Expand
The Turking Test: Can Language Models Understand Instructions?
The Turking Test is presented, which examines a model's ability to follow natural language instructions of varying complexity, and reveals that a large pretrained language model performs poorly across all tasks. Expand
Looking Beyond the Surface: A Challenge Set for Reading Comprehension over Multiple Sentences
The dataset is the first to study multi-sentence inference at scale, with an open-ended set of question types that requires reasoning skills, and finds human solvers to achieve an F1-score of 88.1%. Expand