Corpus ID: 53373094

Do Language Models Have Common Sense

@inproceedings{Trinh2018DoLM,
  title={Do Language Models Have Common Sense},
  author={Trieu H. Trinh and Quoc V. Le},
  year={2018}
}
It has been argued that current machine learning models do not have common sense, and therefore must be hard-coded with prior knowledge (Marcus, 2018). Here we show surprising evidence that language models can already learn to capture certain common sense knowledge. Our key observation is that a language model can compute the probability of any statement, and this probability can be used to evaluate the truthfulness of that statement. On the Winograd Schema Challenge (Levesque et al., 2011… Expand

Figures and Tables from this paper

Barack’s Wife Hillary: Using Knowledge Graphs for Fact-Aware Language Modeling
TLDR
This work introduces the knowledge graph language model (KGLM), a neural language model with mechanisms for selecting and copying facts from a knowledge graph that are relevant to the context that enable the model to render information it has never seen before, as well as generate out-of-vocabulary tokens. Expand
A Systematic Investigation of Commonsense Understanding in Large Language Models
TLDR
It is found that the impressive zeroshot performance of large language models is mostly due to existence of dataset bias in the authors' benchmarks, and that leveraging explicit commonsense knowledge does not yield substantial improvement. Expand
Commonsense Knowledge Mining from Pretrained Models
TLDR
This work develops a method for generating commonsense knowledge using a large, pre-trained bidirectional language model that can be used to rank a triple’s validity by the estimated pointwise mutual information between the two entities. Expand
TR at SemEval-2020 Task 4: Exploring the Limits of Language-model-based Common Sense Validation
TLDR
This paper examines the ability of large-scale pre-trained language models to distinguish commonsense from non-commonsense statements, and the utility of external resources that aim to supplement the world knowledge inherent in such language models, including commonsense knowledge graph embedding models, word concreteness ratings, and text-to-image generation models. Expand
Common Sense Bias in Semantic Role Labeling
TLDR
This work evaluates three SRL parsers on very simple transitive sentences with verbs usually associated with animate subjects and objects, such as, “Mary babysat Tom” and presents a challenge dataset for evaluating the extent to which parsers are sensitive to common sense bias. Expand
KagNet: Knowledge-Aware Graph Networks for Commonsense Reasoning
TLDR
This paper proposes a textual inference framework for answering commonsense questions, which effectively utilizes external, structured commonsense knowledge graphs to perform explainable inferences. Expand
Mix-review: Alleviate Forgetting in the Pretrain-Finetune Framework for Neural Language Generation Models
TLDR
This work focuses on the transformer encoder-decoder model for the open-domain dialogue response generation task, and finds that after standard fine-tuning, the model forgets important language generation skills acquired during large-scale pre-training. Expand
Analyzing the Forgetting Problem in Pretrain-Finetuning of Open-domain Dialogue Response Models
TLDR
This work focuses on the transformer encoder-decoder model for the open-domain dialogue response generation task, and finds that after standard finetuning, the model forgets some of the important language generation skills acquired during large-scale pretraining. Expand
IIE-NLP-NUT at SemEval-2020 Task 4: Guiding PLM with Prompt Template Reconstruction Strategy for ComVE
TLDR
This paper introduces the systems for the first two subtasks of SemEval Task4: Commonsense Validation and Explanation and proposes the input reconstruction strategy with prompt templates, which formalizes the subtasks into the multiple-choice question answering format and construct the input with the prompt templates. Expand
Conditional Response Augmentation for Dialogue Using Knowledge Distillation
TLDR
This paper proposes counterfactual augmentation, of considering whether unobserved utterances would “counterfactually” replace the labelled response, for the given context, and augment only if that is the case. Expand
...
1
2
...

References

SHOWING 1-10 OF 31 REFERENCES
A Corpus and Cloze Evaluation for Deeper Understanding of Commonsense Stories
TLDR
A new framework for evaluating story understanding and script learning: the `Story Cloze Test’, which requires a system to choose the correct ending to a four-sentence story, and a new corpus of 50k five- Sentence commonsense stories, ROCStories, to enable this evaluation. Expand
Improving Language Understanding by Generative Pre-Training
TLDR
The general task-agnostic model outperforms discriminatively trained models that use architectures specifically crafted for each task, significantly improving upon the state of the art in 9 out of the 12 tasks studied. Expand
Deep Contextualized Word Representations
TLDR
A new type of deep contextualized word representation is introduced that models both complex characteristics of word use and how these uses vary across linguistic contexts, allowing downstream models to mix different types of semi-supervision signals. Expand
Combing Context and Commonsense Knowledge Through Neural Networks for Solving Winograd Schema Problems
TLDR
A general framework to combine context and commonsense knowledge for solving the Winograd Schema (WS) and Pronoun Disambiguation Problems (PDP) and two methods to solve the WS and PDP problems are proposed. Expand
The LAMBADA dataset: Word prediction requiring a broad discourse context
TLDR
It is shown that LAMBADA exemplifies a wide range of linguistic phenomena, and that none of several state-of-the-art language models reaches accuracy above 1% on this novel benchmark. Expand
Broad Context Language Modeling as Reading Comprehension
TLDR
This work views LAMBADA as a reading comprehension problem and applies comprehension models based on neural networks, finding that neural network readers perform well in cases that involve selecting a name from the context based on dialogue or discourse cues but struggle when coreference resolution or external knowledge is needed. Expand
Unsupervised Pretraining for Sequence to Sequence Learning
TLDR
This work presents a general unsupervised learning method to improve the accuracy of sequence to sequence (seq2seq) models by pretraining the weights of the encoder and decoder with the pretrained weights of two language models and then fine-tuned with labeled data. Expand
Towards Addressing the Winograd Schema Challenge - Building and Using a Semantic Parser and a Knowledge Hunting Module
TLDR
This paper presents an approach that identifies the knowledge needed to answer a challenge question, hunts down that knowledge from text repositories, and then reasons with machines to come up with the answer. Expand
Commonsense Knowledge Base Completion
TLDR
This work develops neural network models for scoring tuples on arbitrary phrases and evaluates them by their ability to distinguish true held-out tuples from false ones and finds strong performance from a bilinear model using a simple additive architecture to model phrases. Expand
Commonsense mining as knowledge base completion? A study on the impact of novelty
TLDR
It is shown that a simple baseline method outperforms the previous state of the art on predicting more novel commonsense knowledge and novelty of predicted triples with respect to the training set as an important factor in interpreting results. Expand
...
1
2
3
4
...