Reading comprehension tests for computer-based understanding evaluation

@article{Wellner2005ReadingCT,
  title={Reading comprehension tests for computer-based understanding evaluation},
  author={Ben Wellner and Lisa Ferro and Warren R. Greiff and Lynette Hirschman},
  journal={Natural Language Engineering},
  year={2005},
  volume={12},
  pages={305 - 334}
}
Reading comprehension (RC) tests involve reading a short passage of text and answering a series of questions pertaining to that text. We present a methodology for evaluation of the application of modern natural language technologies to the task of responding to RC tests. Our work is based on ABCs (Abduction Based Comprehension system), an automated system for taking tests requiring short answer phrases as responses. A central goal of ABCs is to serve as a testbed for understanding the role that… 
Evaluating the Natural Language Understanding of a Machine by Answering Multiple Choice Questions for a Comprehension Text using proposed LKD Model
  • Computer Science
  • 2019
TLDR
The proposed LKD is acting like a human brain for the machine for answering the questions inquired by MCQA system, which is like an inference engine which contains all possible inference for each sentence in the comprehension text.
Evaluating Machine Reading Systems through Comprehension Tests
This paper describes a methodology for testing and evaluating the performance of Machine Reading systems through Question Answering and Reading Comprehension Tests. The methodology is being used in
READING COMPREHENSION SYSTEM – A REVIEW
Reading Comprehension (RC) Systems are to understand a given text and return answers in response to questions about the text. Reading Comprehension can be viewed as single document question answering
Overview of QA4MRE at CLEF 2011: Question Answering for Machine Reading Evaluation
TLDR
The preparation of the data sets, the creation of the background collections to allow systems to acquire the required knowledge, the metric used for the evaluation of the systems' submissions, and the results of this first attempt of the QA4MRE challenge are described.
Conceptual Graph Matching Method for Reading Comprehension Tests
TLDR
A conceptual graph matching method towards RC tests to extract answer strings by first representing the text and questions as conceptual graphs, and then extracts subgraphs for every candidate answer concept from the text graph.
Towards Literate Artificial Intelligence
TLDR
A unified max-margin framework that learns to find hidden structures given a corpus of question-answer pairs, and uses what it learns to answer questions on novel texts to obtain state-of-the-art performance on two well-known natural language comprehension benchmarks.
An Automatically Generated Lexical Knowledge Base with Soft Definitions
TLDR
This thesis demonstrates a method to automatically extract a lexical knowledge base from dictionaries for the purpose of improving machine reading and proposes the use of a knowledge representation called Multiple Interpretation Graphs (MIGs), and a Lexical knowledge structure called auto-frames to support contextualization.
MCTest: A Challenge Dataset for the Open-Domain Machine Comprehension of Text
TLDR
MCTest is presented, a freely available set of stories and associated questions intended for research on the machine comprehension of text that requires machines to answer multiple-choice reading comprehension questions about fictional stories, directly tackling the high-level goal of open-domain machine comprehension.
ACL-05 Empirical Modeling of Semantic Equivalence and Entailment
TLDR
This workshop is intended to bring together people working on empirical, application-independent approaches to the practical problems of semantic inference to help foster discussion around common datasets and evaluation strategies that will help guide future work in this area.
Evaluating Semantic Evaluations: How RTE Measures Up
TLDR
It is found that although RTE does not correspond to a “real” or naturally occurring language processing task, it nonetheless provides clear and simple metrics, a tolerable cost of corpus development, good annotator reliability, and the possibility of finding noisy but plentiful training material.
...
1
2
3
...

References

SHOWING 1-10 OF 45 REFERENCES
Open-domain textual question answering techniques
TLDR
This paper discusses an approach that successfully enhanced an existing IS system with RC capabilities, which constitutes a possible foundation for more advanced forms of dialogue-based Q/A.
Deep Read: A Reading Comprehension System
TLDR
Initial work on Deep Read, an automated reading comprehension system that accepts arbitrary text input (a story) and answers questions about it is described, with a baseline system that retrieves the sentence containing the answer 30--40% of the time.
Analyses for elucidating current question answering technology
In this paper, we take a detailed look at the performance of components of an idealized question answering system on two different tasks: the TREC Question Answering task and a set of reading
A Rule-based Question Answering System for Reading Comprehension Tests
TLDR
A rule-based system that can read a short story and find the sentence in the story that best answers a given question, Quarc, uses heuristic rules that look for lexical and semantic clues in the question and the story.
Interpretation as Abduction
Overview of the TREC 2002 Question Answering Track
TLDR
This paper provides an overview of the TREC 2002 QA track, which defined how answer strings were judged, and established that different assessors have different ideas as to what constitutes a correct answer even for the limited type of questions used in the track.
Natural language question answering: the view from here
TLDR
The best systems are now able to answer more than two thirds of factual questions in this evaluation, with recent successes reported in a series of question-answering evaluations.
Semantic inference in natural language: validating a tractable approach
TLDR
This paper is concerned with an inferential approach to information extraction reporting on the results of an empirical study that was performed to validate the approach, and serves to validate experimentally a normal form hypothesis that guarantees tractability of inference in the RHO framework.
Semantic Inference in Natural Language: Validating a Tractable Approach
TLDR
The study brings together the RHO framework for tractable terminological knowledge representation and the Alembic message understanding system to validate experimentally a normal form hypothesis that guarantees tractability of inference in the R HO framework.
A Question Answering System Developed as a Project in a Natural Language Processing Course
TLDR
The Question Answering System constructed during a one semester graduate-level course on Natural Language Processing could improve the accuracy of question answering on the test set of the Remedia corpus over the reported levels by using a combination of syntactic and semantic features and machine learning techniques.
...
1
2
3
4
5
...