Know What You Don’t Know: Unanswerable Questions for SQuAD

@inproceedings{Rajpurkar2018KnowWY,
  title={Know What You Don’t Know: Unanswerable Questions for SQuAD},
  author={Pranav Rajpurkar and Robin Jia and Percy Liang},
  booktitle={ACL},
  year={2018}
}
Extractive reading comprehension systems can often locate the correct answer to a question in a context document, but they also tend to make unreliable guesses on questions for which the correct answer is not stated in the context. [...] Key Method SQuAD 2.0 combines existing SQuAD data with over 50,000 unanswerable questions written adversarially by crowdworkers to look similar to answerable ones. To do well on SQuAD 2.0, systems must not only answer questions when possible, but also determine when no answer…Expand
Read + Verify: Machine Reading Comprehension with Unanswerable Questions
TLDR
This work proposes a novel read-then-verify system, which not only utilizes a neural reader to extract candidate answers and produce no-answer probabilities, but also leverages an answer verifier to decide whether the predicted answer is entailed by the input snippets. Expand
QANet for Question Answering on SQuAD 2 . 0
Question answering is a major challenge in Natural Language Processing. Extensive research covers setups where answers always exist. However, being able to differentiate answerable from unanswerableExpand
EQuANt (Enhanced Question Answer Network)
TLDR
This work presents Enhanced Question Answer Network (EQuANt), an MRC model which extends the successful QANet architecture of Yu et al. to cope with unanswerable questions and demonstrates the utility of multi-task learning in the MRC context. Expand
SQuAD2-CR: Semi-supervised Annotation for Cause and Rationales for Unanswerability in SQuAD 2.0
TLDR
SQuAD2-CR dataset is released, which contains annotations on unanswerable questions from the SQuAD 2.0 dataset, to enable an explanatory analysis of the model prediction and annotate explanation on why the most plausible answer span cannot be the answer and which part of the question causes unanswerability. Expand
Quizbowl: The Case for Incremental Question Answering
TLDR
This work makes two key contributions to machine learning research through Quizbowl: collecting and curating a large factoid QA dataset and an accompanying gameplay dataset, and developing a computational approach to playing Quiz Bowl that involves determining both what to answer and when to answer. Expand
BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions
TLDR
It is found that transferring from entailment data is more effective than transferring from paraphrase or extractive QA data, and that it, surprisingly, continues to be very beneficial even when starting from massive pre-trained language models such as BERT. Expand
Getting Closer to AI Complete Question Answering: A Set of Prerequisite Real Tasks
TLDR
QuAIL is presented, the first RC dataset to combine text-based, world knowledge and unanswerable questions, and to provide question type annotation that would enable diagnostics of the reasoning strategies by a given QA system. Expand
Relation Module for Non-answerable Prediction on Question Answering
TLDR
This paper aims to improve a MRC model's ability to determine whether a question has an answer in a given context (e.g. the recently proposed SQuAD 2.0 task) by developing a relation module that is adaptable to any MRC models. Expand
Answer Pointer Alternatives for Unanswerable Questions in SQuAD 2 . 0
Machine reading comprehension tasks where questions are answered by a segment of text in a passage are commonly approached by training a model to point to the start and end indices of the answer spanExpand
BiQuAD: Towards QA based on deeper text understanding
TLDR
This work introduces a new dataset called BiQuAD that requires deeper comprehension in order to answer questions in both extractive and deductive fashion and shows that state-of-the-art QA models do not perform well on the challenging long form contexts and reasoning requirements posed by the dataset. Expand
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 24 REFERENCES
SQuAD: 100,000+ Questions for Machine Comprehension of Text
TLDR
A strong logistic regression model is built, which achieves an F1 score of 51.0%, a significant improvement over a simple baseline (20%). Expand
TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension
TLDR
It is shown that, in comparison to other recently introduced large-scale datasets, TriviaQA has relatively complex, compositional questions, has considerable syntactic and lexical variability between questions and corresponding answer-evidence sentences, and requires more cross sentence reasoning to find answers. Expand
Adversarial Examples for Evaluating Reading Comprehension Systems
TLDR
This work proposes an adversarial evaluation scheme for the Stanford Question Answering Dataset that tests whether systems can answer questions about paragraphs that contain adversarially inserted sentences without changing the correct answer or misleading humans. Expand
MCTest: A Challenge Dataset for the Open-Domain Machine Comprehension of Text
TLDR
MCTest is presented, a freely available set of stories and associated questions intended for research on the machine comprehension of text that requires machines to answer multiple-choice reading comprehension questions about fictional stories, directly tackling the high-level goal of open-domain machine comprehension. Expand
NewsQA: A Machine Comprehension Dataset
TLDR
NewsQA, a challenging machine comprehension dataset of over 100,000 human-generated question-answer pairs, is presented and analysis confirms that NewsQA demands abilities beyond simple word matching and recognizing textual entailment. Expand
WikiQA: A Challenge Dataset for Open-Domain Question Answering
TLDR
The WIKIQA dataset is described, a new publicly available set of question and sentence pairs, collected and annotated for research on open-domain question answering, which is more than an order of magnitude larger than the previous dataset. Expand
Zero-Shot Relation Extraction via Reading Comprehension
TLDR
It is shown that relation extraction can be reduced to answering simple reading comprehension questions, by associating one or more natural-language questions with each relation slot, and that zero-shot generalization to unseen relation types is possible, at lower accuracy levels. Expand
MS MARCO: A Human Generated MAchine Reading COmprehension Dataset
TLDR
This new dataset is aimed to overcome a number of well-known weaknesses of previous publicly available datasets for the same task of reading comprehension and question answering, and is the most comprehensive real-world dataset of its kind in both quantity and quality. Expand
What is the Jeopardy Model? A Quasi-Synchronous Grammar for QA
TLDR
A probabilistic quasi-synchronous grammar, inspired by one proposed for machine translation, and parameterized by mixtures of a robust nonlexical syntax/alignment model with a(n optional) lexical-semantics-driven log-linear model is proposed. Expand
Making Neural QA as Simple as Possible but not Simpler
TLDR
This work proposes a simple heuristic that guides the development of neural baseline systems for the extractive QA task and finds that there are two ingredients necessary for building a high-performing neural QA system: the awareness of question words while processing the context and a composition function that goes beyond simple bag-of-words modeling, such as recurrent neural networks. Expand
...
1
2
3
...