• DOI:10.48550/arXiv.2209.12617
• Corpus ID: 252531182

Evaluation of Question Answering Systems: Complexity of judging a natural language

@article{Farea2022EvaluationOQ,
  title={Evaluation of Question Answering Systems: Complexity of judging a natural language},
  author={Amer Ali Sallam Farea and Z. Yang and Kien Duong and Nadeesha Perera and Frank Emmert-Streib},
  journal={ArXiv},
  year={2022},
  volume={abs/2209.12617}
}

• A. Farea, Z. Yang, +2 authors F. Emmert-Streib
• Published 10 September 2022
• Computer Science
• ArXiv

Question answering (QA) systems are among the most important and rapidly developing research topics in natural language processing (NLP). A reason, therefore, is that a QA system allows humans to interact more naturally with a machine, e.g., via a virtual assistant or search engine. In the last decades, many QA systems have been proposed to address the requirements of different question-answering tasks. Furthermore, many error scores have been introduced, e.g., based on n-gram matching, word embeddings, or contextual embeddings to measure the performance of a QA system. This survey attempts to provide a systematic overview of the general framework of QA, QA paradigms, benchmark datasets, and assessment techniques for a quantitative evaluation of QA systems. The latter is particularly important because not only is the construction of a QA system complex but also its evaluation. We hypothesize that a reason, therefore, is that the quantitative formalization of human judgment is an open problem.