CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP

@article{Ye2021CrossFitAF,
  title={CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP},
  author={Qinyuan Ye and Bill Yuchen Lin and Xiang Ren},
  journal={ArXiv},
  year={2021},
  volume={abs/2104.08835}
}
Humans can learn a new language task efficiently with only few examples, by leveraging their knowledge obtained when learning prior tasks. In this paper, we explore whether and how such cross-task generalization ability can be acquired, and further applied to build better few-shot learners across diverse NLP tasks. We introduce CrossFit, a problem setup for studying cross-task generalization ability, which standardizes seen/unseen task partitions, data access during different learning stages… 

KnowDA: All-in-One Knowledge Mixture Model for Data Augmentation in Few-Shot NLP

It is demonstrated that the proposed Knowledge Mixture enables pre-trained language models the capability of generating proper synthetic instances from scratch for complicated tasks (i.e., the data sample has long sequences or multiple sentences).

Zero- and Few-Shot NLP with Pretrained Language Models

This tutorial aims at bringing interested NLP researchers up to speed about the recent and ongoing techniques for zero- and few-shot learning with pretrained language models.

Exploring Universal Intrinsic Task Subspace via Prompt Tuning

Evidence is shown indicating that the adaptations of PLMs to various few-shot tasks can be reparameterized as optimizing only a few free parameters in a low-dimensional intrinsic task subspace, which may help to understand why PLMs could easily adapt to various NLP tasks with small-scale data.

CLUES: Few-Shot Learning Evaluation in NLU

CLUES is introduced, a benchmark for evaluating the few-shot learning capabilities of NLU models, and it is demonstrated that while recent models reach human performance when they have access to large amounts of labeled data, there is a huge gap in performance in the many-shot setting for most tasks.

Learning Instructions with Unlabeled Data for Zero-Shot Cross-Task Generalization

This work empirically explores the IT performance trends versus the number of labeled data, instructions, and training tasks, and comprehensively analyzes the key factors of UDIT to investigate how to better improve IT with unlabeled data.

CLUES: Few-Shot Learning Evaluation in Natural Language Understanding

It is demonstrated that while recent models reach human performance when they have access to large amounts of labeled data, there is a huge gap in performance in the few-shot setting for most tasks.

Few-shot Adaptation Works with UnpredicTable Data

This work automatically extracting 413,299 tasks from internet tables - orders of magnitude more than the next-largest public datasets - and finds that narrow subsets of the authors' dataset sometimes outperform more diverse datasets.

FLEX: Unifying Evaluation for Few-Shot NLP

The FLEX Principles are formulated, a set of requirements and best practices for unified, rigorous, valid, and cost-sensitive few-shot NLP evaluation that include Sample Size Design, a novel approach to benchmark design that optimizes statistical accuracy and precision while keeping evaluation costs manageable.

MetaICL: Learning to Learn In Context

We introduce MetaICL (Meta-training for In-Context Learning), a new meta-training framework for few-shot learning where a pretrained language model is tuned to do in-context learning on a large set

Generating Training Data with Language Models: Towards Zero-Shot Language Understanding

This paper presents a simple approach that uses both types of PLMs for fully zero-shot learning of NLU tasks without requiring any task-specific data: a unidirectional PLM generates class-conditioned texts guided by prompts, which are used as the training data for a bidirectionalPLM.
...

References

SHOWING 1-10 OF 176 REFERENCES

DReCa: A General Task Augmentation Strategy for Few-Shot Natural Language Inference

DReCA (Decomposing datasets into Reasoning Categories), a simple method for discovering and using latent reasoning categories in a dataset, to form additional high quality tasks to improve the accuracy of meta-learners.

Lifelong Learning of Few-shot Learners across NLP Tasks

This paper investigates the model’s ability of few-shot generalization to new tasks while retaining its performance on the previously learned tasks and proposes a continual meta-learning approach which learns to generate adapter weights from a few examples while regularizing changes of the weights to mitigate catastrophic forgetting.

Learning to Few-Shot Learn Across Diverse Natural Language Classification Tasks

LEOPARD is trained with the state-of-the-art transformer architecture and shows better generalization to tasks not seen at all during training, with as few as 4 examples per label, than self-supervised pre-training or multi-task training.

Zero-Shot Cross-Lingual Transfer with Meta Learning

This work considers the setting of training models on multiple different languages at the same time, when little or no data is available for languages other than English, and demonstrates the consistent effectiveness of meta-learning for a total of 15 languages.

Entailment as Few-Shot Learner

A new approach is proposed, named as EFL, that can turn small LMs into better few-shot learners, and improves the various existing SOTA few-shots learning methods by 12%, and yields competitive few- shot performance with 500 times larger models, such as GPT-3.

Language Models are Few-Shot Learners

GPT-3 achieves strong performance on many NLP datasets, including translation, question-answering, and cloze tasks, as well as several tasks that require on-the-fly reasoning or domain adaptation, such as unscrambling words, using a novel word in a sentence, or performing 3-digit arithmetic.

Exploring and Predicting Transferability across NLP Tasks

The results show that transfer learning is more beneficial than previously thought, especially when target task data is scarce, and can improve performance even when the source task is small or differs substantially from the target task.

Self-Supervised Meta-Learning for Few-Shot Natural Language Classification Tasks

This paper proposes a self-supervised approach to generate a large, rich, meta-learning task distribution from unlabeled text, and shows that this meta-training leads to better few-shot generalization than language-model pre-training followed by finetuning.

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

This systematic study compares pre-training objectives, architectures, unlabeled datasets, transfer approaches, and other factors on dozens of language understanding tasks and achieves state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more.

Making Pre-trained Language Models Better Few-shot Learners

The LM-BFF approach makes minimal assumptions on task resources and domain expertise, and hence constitutes a strong task-agnostic method for few-shot learning.
...