• DOI:10.1162/tacl_a_00331
Corpus ID: 209515389

What Does My QA Model Know? Devising Controlled Probes Using Expert Knowledge

@article{Richardson2019WhatDM,
  title={What Does My QA Model Know? Devising Controlled Probes Using Expert Knowledge},
  author={Kyle Richardson and Ashish Sabharwal},
  journal={Transactions of the Association for Computational Linguistics},
  year={2019},
  volume={8},
  pages={572-588}
}

Kyle Richardson, Ashish Sabharwal

Published 2019

Computer Science

Transactions of the Association for Computational Linguistics

Open-domain question answering (QA) involves many knowledge and reasoning challenges, but are successful QA models actually learning such knowledge when trained on benchmark QA tasks? We investigate this via several new diagnostic tasks probing whether multiple-choice QA models know definitions and taxonomic reasoning—two skills widespread in existing benchmarks and fundamental to more complex reasoning. We introduce a methodology for automatically building probe datasets from expert knowledge… CONTINUE READING

View on MIT Press

mitpressjournals.org

Save to Library

Create Alert

Cite

Launch Research Feed