• Corpus ID: 244130199

Few-Shot Self-Rationalization with Natural Language Prompts

@article{Marasovi2021FewShotSW,
  title={Few-Shot Self-Rationalization with Natural Language Prompts},
  author={Ana Marasovi{\'c} and Iz Beltagy and Doug Downey and Matthew E. Peters},
  journal={ArXiv},
  year={2021},
  volume={abs/2111.08284}
}
Self-rationalization models that predict task labels and generate free-text elaborations for their predictions could enable more intuitive interaction with NLP systems. These models are, however, currently trained with a large amount of human-written free-text explanations for each task which hinders their broader usage. We propose to study a more real-istic setting of self-rationalization using few training examples. We present FEB—a stan-dardized collection of four existing English-language… 

Figures and Tables from this paper

Reframing Human-AI Collaboration for Generating Free-Text Explanations
TLDR
A pipeline that combines GPT-3 with a supervised that incorpo-rates binary acceptability judgments from humans in the loop is created and it is demonstrated that acceptability is partially correlated with various fine-grained attributes of explanations.
Few-Shot Out-of-Domain Transfer Learning of Natural Language Explanations
TLDR
This work introduces three vanilla approaches for few-shot transfer learning of NLEs for the case of few NLES but abundant labels, along with an adaptation of an existing vanilla fine-tuning approach, and establishes the best strategies out of the four identified training regimes.
Chain of Thought Prompting Elicits Reasoning in Large Language Models
TLDR
Experiments show that inducing a chain of thought via prompting can be enabled byently large language models to better perform reasoning tasks that otherwise have flat scaling curves.
Calibrating Trust of Multi-Hop Question Answering Systems with Decompositional Probes
TLDR
It is shown that decomposition is an effective form of probing QA systems as well as a promising approach to explanation generation and in-depth analyses show the need for improvements in decomposition systems.
The Unreliability of Explanations in Few-Shot In-Context Learning
TLDR
A framework for calibrating model predictions based on the reliability of explanations is presented and it is shown that explanations judged as good by humans—those that are logically consistent with the input and the prediction—usually indicate more accurate predictions.
On Advances in Text Generation from Images Beyond Captioning: A Case Study in Self-Rationalization
Integrating vision and language has gained no-table attention following the success of pretrained language models. Despite that, a frac-tion of emerging multimodal models is suitable for text
Can language models learn from explanations in context?
TLDR
Investigating whether explanations of few-shot examples can allow language models to adapt more efitively and showing that explanations of examples can improve performance shows that explanations can support the in-context learning abilities of large language models on challenging tasks.

References

SHOWING 1-10 OF 76 REFERENCES
Reframing Human-AI Collaboration for Generating Free-Text Explanations
TLDR
A pipeline that combines GPT-3 with a supervised that incorpo-rates binary acceptability judgments from humans in the loop is created and it is demonstrated that acceptability is partially correlated with various fine-grained attributes of explanations.
Towards Explainable NLP: A Generative Explanation Framework for Text Classification
TLDR
A novel generative explanation framework that learns to make classification decisions and generate fine-grained explanations at the same time and introduces the explainable factor and the minimum risk training approach that learn to generate more reasonable explanations.
Learning How to Ask: Querying LMs with Mixtures of Soft Prompts
TLDR
This work explores the idea of learning prompts by gradient descent—either fine-tuning prompts taken from previous work, or starting from random initialization, showing that the implicit factual knowledge in language models was previously underestimated.
Few-Shot Out-of-Domain Transfer Learning of Natural Language Explanations
TLDR
This work introduces three vanilla approaches for few-shot transfer learning of NLEs for the case of few NLES but abundant labels, along with an adaptation of an existing vanilla fine-tuning approach, and establishes the best strategies out of the four identified training regimes.
e-ViL: A Dataset and Benchmark for Natural Language Explanations in Vision-Language Tasks
TLDR
e-ViL is a benchmark for explainable vision-language tasks that establishes a unified evaluation framework and provides the first comprehensive comparison of existing approaches that generate NLEs for VL tasks.
e-SNLI: Natural Language Inference with Natural Language Explanations
TLDR
The Stanford Natural Language Inference dataset is extended with an additional layer of human-annotated natural language explanations of the entailment relations, which can be used for various goals, such as obtaining full sentence justifications of a model’s decisions, improving universal sentence representations and transferring to out-of-domain NLI datasets.
Explanations for CommonsenseQA: New Dataset and Models
TLDR
This work human-annotates a first-of-its-kind dataset of positive and negative properties, as well as free-flow explanations, for 11K QA pairs taken from the CQA dataset, and proposes a latent representation based property retrieval model aswell as a GPT-2 based property generation model with a novel two step fine-tuning procedure.
Explaining Question Answering Models through Text Generation
TLDR
A model for multi-choice question answering, where a LM-based generator generates a textual hypothesis that is later used by a classifier to answer the question, and produces hypotheses that elucidate the knowledge used by the LM for answering the question.
Training language models to follow instructions with human feedback
TLDR
The results show that fine-tuning with human feedback is a promising direction for aligning language models with human intent and showing improvements in truthfulness and reductions in toxic output generation while having minimal performance regressions on public NLP datasets.
Explain Yourself! Leveraging Language Models for Commonsense Reasoning
TLDR
This work collects human explanations for commonsense reasoning in the form of natural language sequences and highlighted annotations in a new dataset called Common Sense Explanations to train language models to automatically generate explanations that can be used during training and inference in a novel Commonsense Auto-Generated Explanation framework.
...
1
2
3
4
5
...