How Many Data Samples is an Additional Instruction Worth?

@article{Puri2022HowMD,
  title={How Many Data Samples is an Additional Instruction Worth?},
  author={Ravsehaj Singh Puri and Swaroop Mishra and Mihir Parmar and Chitta Baral},
  journal={ArXiv},
  year={2022},
  volume={abs/2203.09161}
}
Recently introduced instruction-paradigm empowers non-expert users to leverage NLP resources by defining a new task in natural language. Instruction-tuned models have significantly outperformed multitask learning models (without instruction); however they are far from state of the art task specific models. Conventional approaches to improve model performance via creating large datasets with lots of task instances or architectural/training changes in model may not be feasible for non-expert… 

HELP ME THINK: A Simple Prompting Strategy for Non-experts to Create Customized Content with Models

TLDR
This paper proposes a simple prompting strategy HELP ME THINK where GPT3 is encouraged to help non-expert users by asking a set of relevant questions and leveraging user answers to execute the task.

BioTABQA: Instruction Learning for Biomedical Table Question Answering

TLDR
Experimental results show that the instruction-tuned model outperforms single and multi task baselines on an average by ∼ 23% and ∼ 6% across various evaluation settings, and more importantly, instruction- Tuned models outperforms baselines by ∼ 5% on cross-tasks.

Is a Question Decomposition Unit All We Need?

TLDR
This work investigates if humans can decompose a hard question into a set of simpler questions that are relatively easier for models to solve and indicates that Human-in-the-loop Question Decomposition (HQD) can potentially provide an alternate path to building large LMs.

In-BoXBART: Get Instructions into Biomedical Multi-Task Learning

TLDR
This is the first attempt to propose a unified model in the biomedical domain and use instructions to achieve generalization across several biomedical tasks, and indicates that there is room for improvement across tasks in the BoX, implying the scope for future research direction.

References

SHOWING 1-10 OF 42 REFERENCES

Reframing Instructional Prompts to GPTk’s Language

TLDR
This work studies several classes of reframing techniques for manual reformulation of prompts into more effective ones, and hopes these empirically-driven techniques will pave the way towards more effective future prompting algorithms.

Learning from Task Descriptions

TLDR
This work introduces a framework for developing NLP systems that solve new tasks after reading their descriptions, synthesizing prior work in this area, and instantiates it with a new English language dataset, ZEST, structured for task-oriented evaluation on unseen tasks.

InstructionNER: A Multi-Task Instruction-Based Generative Framework for Few-shot NER

TLDR
This work proposes a multi-task instruction-based generative framework, named InstructionNER, for low-resource named entity recognition, which reformulate the NER task as a generation problem, which enriches source sentences with task-specific instructions and answer options, then inferences the entities and types in natural language.

Training language models to follow instructions with human feedback

TLDR
The results show that fine-tuning with human feedback is a promising direction for aligning language models with human intent and showing improvements in truthfulness and reductions in toxic output generation while having minimal performance regressions on public NLP datasets.

NL-Augmenter: A Framework for Task-Sensitive Natural Language Augmentation

TLDR
The efficacy of NL-Augmenter is demonstrated by using several of its tranformations to analyze the robustness of popular natural language models and the infrastructure, datacards and robutstness analysis results are shown.

MetaICL: Learning to Learn In Context

We introduce MetaICL (Meta-training for In-Context Learning), a new meta-training framework for few-shot learning where a pretrained language model is tuned to do in-context learning on a large set

Multitask Prompted Training Enables Zero-Shot Task Generalization

TLDR
A system for easily mapping any natural language tasks into a human-readable prompted form and fine-tune a pretrained encoder-decoder model on this multitask mixture covering a wide variety of tasks.

Finetuned Language Models Are Zero-Shot Learners

TLDR
It is shown that instruction tuning —finetuning language models on a collection of datasets described via instructions—substantially improves zero-shot performance on unseen tasks and outperforms few-shot GPT-3 by a large margin on ANLI, RTE, BoolQ, AI2-ARC, OpenbookQA, and StoryCloze.

A Recipe for Arbitrary Text Style Transfer with Large Language Models

TLDR
Augmented zero-shot learning is simple and demonstrates promising results not just on standard style transfer tasks such as sentiment, but also on arbitrary transformations such as ‘make this melodramatic’ or ‘insert a metaphor’.

Tailor: Generating and Perturbing Text with Semantic Controls

TLDR
This work presents Tailor, a semantically-controlled text generation system that builds on a pretrained seq2seq model and produces textual outputs conditioned on control codes derived from semantic representations, and shows that Tailor perturbations can improve model generalization through data augmentation.