DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs

  title={DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs},
  author={Dheeru Dua and Yizhong Wang and Pradeep Dasigi and Gabriel Stanovsky and Sameer Singh and Matt Gardner},
  • Dheeru Dua, Yizhong Wang, +3 authors Matt Gardner
  • Published in NAACL-HLT 2019
  • Computer Science
  • Reading comprehension has recently seen rapid progress, with systems matching humans on the most popular datasets for the task. However, a large body of work has highlighted the brittleness of these systems, showing that there is much work left to be done. We introduce a new reading comprehension benchmark, DROP, which requires Discrete Reasoning Over the content of Paragraphs. In this crowdsourced, adversarially-created, 55k-question benchmark, a system must resolve references in a question… CONTINUE READING
    141 Citations
    On Making Reading Comprehension More Comprehensive
    • 11
    • Highly Influenced
    • PDF
    Comprehensive Multi-Dataset Evaluation of Reading Comprehension
    • 2
    • PDF
    ReClor: A Reading Comprehension Dataset Requiring Logical Reasoning
    • 8
    • PDF
    Tag-based Multi-Span Extraction in Reading Comprehension
    • 5
    MOCHA: A Dataset for Training and Evaluating Generative Reading Comprehension Metrics
    Quoref: A Reading Comprehension Dataset with Questions Requiring Coreferential Reasoning
    • 36
    • PDF
    A Multi-Type Multi-Span Network for Reading Comprehension that Requires Discrete Reasoning
    • 18
    • Highly Influenced
    • PDF
    IIRC: A Dataset of Incomplete Information Reading Comprehension Questions
    R3: A Reading Comprehension Benchmark Requiring Reasoning Processes
    • 1
    • Highly Influenced
    • PDF


    Looking Beyond the Surface: A Challenge Set for Reading Comprehension over Multiple Sentences
    • 114
    • PDF
    The NarrativeQA Reading Comprehension Challenge
    • 205
    • PDF
    A Thorough Examination of the CNN/Daily Mail Reading Comprehension Task
    • 412
    • PDF
    TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension
    • 486
    • PDF
    Constructing Datasets for Multi-hop Reading Comprehension Across Documents
    • 209
    • PDF
    SQuAD: 100, 000+ Questions for Machine Comprehension of Text
    • 2,306
    • Highly Influential
    • PDF
    Know What You Don't Know: Unanswerable Questions for SQuAD
    • 656
    • PDF
    Bidirectional Attention Flow for Machine Comprehension
    • 1,191
    • Highly Influential
    • PDF