What Does My QA Model Know? Devising Controlled Probes Using Expert Knowledge

@article{Richardson2019WhatDM,
  title={What Does My QA Model Know? Devising Controlled Probes Using Expert Knowledge},
  author={Kyle Richardson and Ashish Sabharwal},
  journal={Transactions of the Association for Computational Linguistics},
  year={2019},
  volume={8},
  pages={572-588}
}
  • Kyle Richardson, Ashish Sabharwal
  • Published 2019
  • Computer Science
  • Transactions of the Association for Computational Linguistics
  • Open-domain question answering (QA) involves many knowledge and reasoning challenges, but are successful QA models actually learning such knowledge when trained on benchmark QA tasks? We investigate this via several new diagnostic tasks probing whether multiple-choice QA models know definitions and taxonomic reasoning—two skills widespread in existing benchmarks and fundamental to more complex reasoning. We introduce a methodology for automatically building probe datasets from expert knowledge… CONTINUE READING
    14 Citations
    Explaining Question Answering Models through Text Generation
    • 8
    • Highly Influenced
    • PDF
    What do Models Learn from Question Answering Datasets?
    • 2
    • PDF
    Teaching Pre-Trained Models to Systematically Reason Over Implicit Knowledge
    • 5
    • PDF
    How Context Affects Language Models' Factual Predictions
    • 12
    • PDF
    Transformers as Soft Reasoners over Language
    • 22
    • PDF
    Semantics Altering Modifications for Evaluating Comprehension in Machine Reading
    • PDF
    Beyond Leaderboards: A survey of methods for revealing weaknesses in Natural Language Inference data and models
    • 3
    • Highly Influenced
    • PDF
    DQI: Measuring Data Quality in NLP
    • 5
    • PDF

    References

    SHOWING 1-10 OF 77 REFERENCES
    Language Models as Knowledge Bases?
    • 217
    • Highly Influential
    • PDF
    Investigating BERT's Knowledge of Language: Five Analysis Methods with NPIs
    • 37
    • Highly Influential
    • PDF
    Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
    • 164
    • PDF
    CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge
    • 163
    • PDF
    QASC: A Dataset for Question Answering via Sentence Composition
    • 35
    • PDF
    How Can We Know What Language Models Know?
    • 40
    • PDF
    Knowledge Questions from Knowledge Graphs
    • 21
    • PDF
    Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference
    • 213
    • Highly Influential
    • PDF