Evaluating Theory of Mind in Question Answering

@article{Nematzadeh2018EvaluatingTO,
  title={Evaluating Theory of Mind in Question Answering},
  author={A. Nematzadeh and Kaylee Burns and E. Grant and A. Gopnik and T. Griffiths},
  journal={ArXiv},
  year={2018},
  volume={abs/1808.09352}
}
  • A. Nematzadeh, Kaylee Burns, +2 authors T. Griffiths
  • Published 2018
  • Computer Science
  • ArXiv
  • We propose a new dataset for evaluating question answering models with respect to their capacity to reason about beliefs. [...] Key Result We find that all fail on our tasks, which require keeping track of inconsistent states of the world; moreover, the models' accuracy decreases notably when random sentences are introduced to the tasks at test.Expand Abstract
    11 Citations

    Figures, Tables, and Topics from this paper

    Explore Further: Topics Discussed in This Paper

    Revisiting the Evaluation of Theory of Mind through Question Answering
    • 1
    • PDF
    Exploiting Attention to Reveal Shortcomings in Memory Models
    • 3
    • PDF
    Characterizing Datasets for Social Visual Question Answering, and the New TinySocial Dataset
    • Zhanwen Chen, Shiyao Li, +7 authors M. Kunda
    • Computer Science
    • 2020 Joint IEEE 10th International Conference on Development and Learning and Epigenetic Robotics (ICDL-EpiRob)
    • 2020
    • PDF
    Experience Grounds Language
    • 32
    • PDF
    Analyzing and Interpreting Neural Networks for NLP: A Report on the First BlackboxNLP Workshop
    • 22
    • PDF
    Implementing Theory of Mind on a Robot Using Dynamic Epistemic Logic
    • 2
    • PDF
    Functional advantages of an adaptive Theory of Mind for robotics: a review of current architectures
    • F. Bianco, D. Ognibene
    • Computer Science
    • 2019 11th Computer Science and Electronic Engineering (CEEC)
    • 2019
    • 3
    • PDF

    References

    SHOWING 1-10 OF 18 REFERENCES
    How Can Memory-Augmented Neural Networks Pass a False-Belief Task?
    • 2
    • PDF
    Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks
    • 779
    • PDF
    Reasoning in Vector Space: An Exploratory Study of Question Answering
    • 31
    • PDF
    Memory Networks
    • 969
    • PDF
    Machine Theory of Mind
    • 109
    • PDF
    The Winograd Schema Challenge
    • 445
    • PDF
    A simple neural network module for relational reasoning
    • 860
    • Highly Influential
    • PDF
    Who is Mistaken?
    • 5
    • PDF
    It Takes Two to Tango: Towards Theory of AI's Mind
    • 31
    • PDF
    End-To-End Memory Networks
    • 1,621
    • Highly Influential
    • PDF