Retrieval Enhanced Model for Commonsense Generation

  title={Retrieval Enhanced Model for Commonsense Generation},
  author={Han Wang and Yang Liu and Chenguang Zhu and Linjun Shou and Ming Gong and Yichong Xu and Michael Zeng},
Commonsense generation is a challenging task of generating a plausible sentence describing an everyday scenario using provided concepts. Its requirement of reasoning over commonsense knowledge and compositional generalization ability even puzzles strong pre-trained language generation models. We propose a novel framework using retrieval methods to enhance both the pre-training and fine-tuning for commonsense generation. We retrieve prototype sentence candidates by concept matching and use them… 

Figures and Tables from this paper

KGR^4: Retrieval, Retrospect, Refine and Rethink for Commonsense Generation
A novel Knowledge-enhanced Commonsense Generation framework, termed KGR, consisting of four stages: Retrieval, Retrospect, Refine, Rethink, which selects the output sentence from candidate sentences produced by generators with different hyper-parameters.
Contextualized Scene Imagination for Generative Commonsense Reasoning
An Imagine-and-Verbalize (I&V) method is proposed, which learns to imagine a relational scene knowledge graph (SKG) with relations between the input concepts, and leverage the SKG as a constraint when generating a plausible scene description.
SAPPHIRE: Approaches for Enhanced Concept-to-Text Generation
An in-depth qualitative analysis illustrates that SAPPHIRE effectively addresses many issues of the baseline model generations, including lack of commonsense, insufficient specificity, and poor fluency.
Knowledge-Augmented Methods for Natural Language Processing
Knowledge in natural language processing (NLP) has been a rising trend especially after the advent of large scale pre-trained models. NLP models with attention to knowledge can i) access unlimited
KommonGen: A Dataset for Korean Generative Commonsense Reasoning Evaluation
KommonGen: 한국어 생성 모델의 상식 추론 평가 데이터셋 서재형1◦, 박찬준, 문현석, 어수경, 강명훈, 이승훈, 임희석1,3∗* 고려대학교 컴퓨터학과, 서울시립대학교 도시사회학과, Human-inspired AI 연구소 {seojae777, bcj1210, glee889, djtnrud} {chaos038527,
NAREOR: The Narrative Reordering Problem
This paper proposes and investigates the task of Narrative Reordering (NAREOR), which involves rewriting a given story in a different narrative order while preserving its plot, and presents a dataset, NAREORC, with human rewritings of stories within ROCStories in non-linear orders, and conducts a detailed analysis of it.
Identifying relevant common sense information in knowledge graphs
Knowledge graphs are often used to store common sense information that is useful for various tasks. However, the extraction of contextually-relevant knowledge is an unsolved problem, and current
A Survey of Knowledge-Enhanced Text Generation
A comprehensive review of the research on knowledge-enhanced text generation over the past five years is presented, which includes two parts: (i) general methods and architectures for integrating knowledge into text generation; (ii) specific techniques and applications according to different forms of knowledge data.
Retrieve, Caption, Generate: Visual Grounding for Enhancing Commonsense in Text Generation Models
Comprehensive evaluation and analysis demonstrate that VisCTG noticeably improves model performance while successfully addressing several issues of the baseline generations, including poor commonsense, fluency, and specificity.


An Enhanced Knowledge Injection Model for Commonsense Generation
This work integrates two additional modules into the pretrained encoder-decoder model for prototype modeling to enhance the knowledge injection procedure and shows results that significantly improves the performance on all the metrics.
CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge
This work presents CommonsenseQA: a challenging new dataset for commonsense question answering, which extracts from ConceptNet multiple target concepts that have the same semantic relation to a single source concept.
A large annotated corpus for learning natural language inference
The Stanford Natural Language Inference corpus is introduced, a new, freely available collection of labeled sentence pairs, written by humans doing a novel grounded task based on image captioning, which allows a neural network-based model to perform competitively on natural language inference benchmarks for the first time.
Unified Language Model Pre-training for Natural Language Understanding and Generation
A new Unified pre-trained Language Model (UniLM) that can be fine-tuned for both natural language understanding and generation tasks that compares favorably with BERT on the GLUE benchmark, and the SQuAD 2.0 and CoQA question answering tasks.
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
A new language representation model, BERT, designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers, which can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks.
Language Models as Knowledge Bases?
An in-depth analysis of the relational knowledge already present (without fine-tuning) in a wide range of state-of-the-art pretrained language models finds that BERT contains relational knowledge competitive with traditional NLP methods that have some access to oracle knowledge.
Language Models are Unsupervised Multitask Learners
It is demonstrated that language models begin to learn these tasks without any explicit supervision when trained on a new dataset of millions of webpages called WebText, suggesting a promising path towards building language processing systems which learn to perform tasks from their naturally occurring demonstrations.
A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference
The Multi-Genre Natural Language Inference corpus is introduced, a dataset designed for use in the development and evaluation of machine learning models for sentence understanding and shows that it represents a substantially more difficult task than does the Stanford NLI corpus.
SPICE: Semantic Propositional Image Caption Evaluation
There is considerable interest in the task of automatically generating image captions. However, evaluation is challenging. Existing automatic evaluation metrics are primarily sensitive to n-gram
Transformers: State-of-the-Art Natural Language Processing
Transformers is an open-source library that consists of carefully engineered state-of-the art Transformer architectures under a unified API and a curated collection of pretrained models made by and available for the community.