Generated Knowledge Prompting for Commonsense Reasoning

  title={Generated Knowledge Prompting for Commonsense Reasoning},
  author={Jiacheng Liu and Alisa Liu and Ximing Lu and Sean Welleck and Peter West and Ronan Le Bras and Yejin Choi and Hannaneh Hajishirzi},
It remains an open question whether incorporating external knowledge benefits commonsense reasoning while maintaining the flexibility of pretrained sequence models. To investigate this question, we develop generated knowledge prompting, which consists of generating knowledge from a language model, then providing the knowledge as additional input when answering a question. Our method does not require task-specific supervision for knowledge integration, or access to a structured knowledge base… 

Figures and Tables from this paper

Human Parity on CommonsenseQA: Augmenting Self-Attention with External Attention

It is found that the proposed external attention mechanism can significantly improve the performance of existing AI systems, allowing practitioners to easily customize foundation AI models to many diverse downstream applications.

Maieutic Prompting: Logically Consistent Reasoning with Recursive Explanations

This work de-velops M AIEUTIC PROMPTING, which infers a correct answer to a question even from the noisy and inconsistent generations of LM, and improves robustness in inference while providing interpretable rationales.

Elaboration-Generating Commonsense Question Answering at Scale

This work uses smaller language models to generate useful intermediate context, referred to here as elaborations, and alternates between updating two language models—an elaboration generator and an answer predictor—allowing each to influence the other.

ThinkSum: Probabilistic reasoning over sets using large language models

Large language models (LLMs) have a substantial capacity for high-level ana-logical reasoning: reproducing patterns in linear text that occur in their training data (zero-shot evaluation) or in the

Zemi: Learning Zero-Shot Semi-Parametric Language Models from Multiple Tasks

rate of 1e-4. We report results from the checkpoint that achieved the best overall performance across all tasks. All experiments are done on eight NVIDIA-V100 32GB GPUs.

Generate rather than Retrieve: Large Language Models are Strong Context Generators

The proposed method is evaluated on three different knowledge-intensive tasks and its effectiveness on both zero-shot and supervised settings is demonstrated.

What does a platypus look like? Generating customized prompts for zero-shot image classification

This work introduces a simple method to generate higher accuracy prompts, without relying on any explicit knowledge of the task domain and with far fewer hand-constructed sentences, and improves accuracy on a range of zero-shot image classification benchmarks, including over one percentage point gain on ImageNet.

Social Simulacra: Creating Populated Prototypes for Social Computing Systems

It is demonstrated that social simulacra shift the behaviors that they generate appropriately in response to design changes, and that they enable exploration of “what if?” scenarios where community members or moderators intervene.

Rationale-Augmented Ensembles in Language Models

It is demonstrated that rationale-augmented ensembles achieve more accurate and interpretable results than existing prompting approaches—including standard prompting without rationales and rationale-based chain-of-thought prompting—while simultaneously improving interpretability of model predictions through the associated rationales.

Can Foundation Models Help Us Achieve Perfect Secrecy?

FOCUS, a framework for personal tasks, is proposed and evaluated on popular privacy benchmarks, finding the approach, satisfying perfect secrecy, competes with strong collaborative learning baselines on 6 of 7 tasks.



Explain Yourself! Leveraging Language Models for Commonsense Reasoning

This work collects human explanations for commonsense reasoning in the form of natural language sequences and highlighted annotations in a new dataset called Common Sense Explanations to train language models to automatically generate explanations that can be used during training and inference in a novel Commonsense Auto-Generated Explanation framework.

Dynamic Neuro-Symbolic Knowledge Graph Construction for Zero-shot Commonsense Question Answering

This paper presents a novel approach that generates contextually-relevant symbolic knowledge structures on demand using generative neural commonsense knowledge models and achieves significant performance boosts over pretrained language models and vanilla knowledge models, all while providing interpretable reasoning paths for its predictions.

Towards Generalizable Neuro-Symbolic Systems for Commonsense Question Answering

This paper performs a survey of recent commonsense QA methods and provides a systematic analysis of popular knowledge resources and knowledge-integration methods, across benchmarks from multiple commonsense datasets, and shows that attention-based injection seems to be a preferable choice for knowledge integration.

Prompting Contrastive Explanations for Commonsense Reasoning Tasks

Inspired by the contrastive nature of human explanations, large pretrained language models are used to complete explanation prompts which contrast alternatives according to the key attribute(s) required to justify the correct answer.

Knowledge-driven Data Construction for Zero-shot Evaluation in Commonsense Question Answering

A novel neuro-symbolic framework for zero-shot question answering across commonsense tasks is proposed and it is shown that, while an individual knowledge graph is better suited for specific tasks, a global knowledge graph brings consistent gains across different tasks.

Improving Question Answering by Commonsense-Based Pre-Training

Results show that incorporating commonsense-based function improves the baseline on three question answering tasks that require commonsense reasoning and leverages useful evidence from an external commonsense knowledge base, which is missing in existing neural network models.

Designing Templates for Eliciting Commonsense Knowledge from Pretrained Sequence-to-Sequence Models

This work explores a template-based approach to extract implicit knowledge for commonsense reasoning on multiple-choice question answering tasks using the text-to-text transfer transformer (T5) model, and initiates further research to find generic natural language templates that can effectively leverage stored knowledge in pretrained models.

COMET: Commonsense Transformers for Automatic Knowledge Graph Construction

This investigation reveals promising results when implicit knowledge from deep pre-trained language models is transferred to generate explicit knowledge in commonsense knowledge graphs, and suggests that using generative commonsense models for automatic commonsense KB completion could soon be a plausible alternative to extractive methods.

A Simple Method for Commonsense Reasoning

Key to this method is the use of language models, trained on a massive amount of unlabled data, to score multiple choice questions posed by commonsense reasoning tests, which outperform previous state-of-the-art methods by a large margin.

How Additional Knowledge can Improve Natural Language Commonsense Question Answering

This work first categorizes external knowledge sources, and shows performance does improve on using such sources, then explores three different strategies for knowledge incorporation and four different models for question-answering using external commonsense knowledge.