Corpus ID: 233296635

Natural Instructions: Benchmarking Generalization to New Tasks from Natural Language Instructions

@article{Mishra2021NaturalIB,
  title={Natural Instructions: Benchmarking Generalization to New Tasks from Natural Language Instructions},
  author={Swaroop Mishra and Daniel Khashabi and Chitta Baral and Hanna Hajishirzi},
  journal={ArXiv},
  year={2021},
  volume={abs/2104.08773}
}
Can we enable NLP models to appropriately respond to instructional prompts and consequently generalize to new tasks? To study this question, we leverage the existing NLP datasets and the instructions that were used to crowdsource them to create NATURALINSTRUCTIONS, a dataset of instructions and task-specific input/output data. This dataset consists of 61 distinct language instructions and about 600k task instances, and is used to evaluate existing state-of-the-art languagemodels (LMs) in… Expand

References

SHOWING 1-10 OF 38 REFERENCES
The Turking Test: Can Language Models Understand Instructions?
Learning from Task Descriptions
What Makes Good In-Context Examples for GPT-3?
Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm
Natural language to SQL: Where are we today?
Zero-shot Learning by Generating Task-specific Adapters
ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks
...
1
2
3
4
...