Crowdsourcing Multiple Choice Science Questions

@inproceedings{Welbl2017CrowdsourcingMC,
  title={Crowdsourcing Multiple Choice Science Questions},
  author={Johannes Welbl and Nelson F. Liu and Matt Gardner},
  booktitle={NUT@EMNLP},
  year={2017}
}
We present a novel method for obtaining high-quality, domain-targeted multiple choice questions from crowd workers. Generating these questions can be difficult without trading away originality, relevance or diversity in the answer options. Our method addresses these problems by leveraging a large corpus of domain-specific text and a small set of existing questions. It produces model suggestions for document selection and answer distractor choice which aid the human question generation process… Expand

Figures, Tables, and Topics from this paper

Quiz-Style Question Generation for News Stories
TLDR
This work proposes a series of novel techniques for applying large pre-trained Transformer encoder-decoder models, namely PEGASUS and T5, to the tasks of question-answer generation and distractor generation, and shows that these models outperform strong baselines using both automated metrics and human raters. Expand
Improving Question Answering with External Knowledge
TLDR
This work explores simple yet effective methods for exploiting two sources of externalknowledge for exploiting unstructured external knowledge for subject-area QA on multiple-choice question answering tasks in subject areas such as science. Expand
Answering Science Exam Questions Using Query Reformulation with Background Knowledge
TLDR
This paper presents a system that reformulates a given question into queries that are used to retrieve supporting text from a large corpus of science-related text and outperforms several strong baselines on the ARC dataset. Expand
Investigating Crowdsourcing to Generate Distractors for Multiple-Choice Assessments
TLDR
The results suggest that crowdsourcing can be a very useful tool in generating effective distractors (attractive to subjects who do not understand the targeted concept) and suggest that this method is faster, easier, and cheaper than is the traditional method of having one or more experts draft distractors. Expand
Ranking Distractors for Multiple Choice Questions
Automatically generating or ranking distractors for multiplechoice questions (MCQs) is still a challanging problem. In this work, we have focused towards automatic ranking of distractors for MCQs.Expand
Answering Science Exam Questions Using Query Rewriting with Background Knowledge
TLDR
A system that rewrites a given question into queries that are used to retrieve supporting text from a large corpus of science-related text is presented and is able to outperform several strong baselines on the ARC dataset. Expand
Generating Answer Candidates for Quizzes and Answer-Aware Question Generators
TLDR
This work proposes a model that can generate a specified number of answer candidates for a given passage of text, which can then be used by instructors to write questions manually or can be passed as an input to automatic answer-aware question generators. Expand
Distractor Generation for Multiple Choice Questions Using Learning to Rank
TLDR
This work investigates how machine learning models, specifically ranking models, can be used to select useful distractors for multiple choice questions and empirically studies feature-based and neural net based ranking models with experiments on the recently released SciQ dataset and the MCQL dataset. Expand
FriendsQA: Open-Domain Question Answering on TV Show Transcripts
TLDR
FriendsQA, a challenging question answering dataset that contains 1,222 dialogues and 10,610 open-domain questions, to tackle machine comprehension on everyday conversations, has a great potential of elevating QA research on multiparty dialogue to another level. Expand
Multiple Domain Question-Answer Generation
In this work we explore the usefulness and practicality of domain adaptation and multi-domain learning methods in question-answer generation. Unlike recent work in question-answer generation whichExpand
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 42 REFERENCES
SQuAD: 100,000+ Questions for Machine Comprehension of Text
TLDR
A strong logistic regression model is built, which achieves an F1 score of 51.0%, a significant improvement over a simple baseline (20%). Expand
Good Question! Statistical Ranking for Question Generation
TLDR
This work uses manually written rules to perform a sequence of general purpose syntactic transformations to turn declarative sentences into questions, which are ranked by a logistic regression model trained on a small, tailored dataset consisting of labeled output from the system. Expand
Who did What: A Large-Scale Person-Centered Cloze Dataset
TLDR
A new "Who-did-What" dataset of over 200,000 fill-in-the-gap (cloze) multiple choice reading comprehension problems constructed from the LDC English Gigaword newswire corpus is constructed and proposed as a challenge task for the community. Expand
Large-scale Simple Question Answering with Memory Networks
TLDR
This paper studies the impact of multitask and transfer learning for simple question answering; a setting for which the reasoning required to answer is quite easy, as long as one can retrieve the correct evidence given a question, which can be difficult in large-scale conditions. Expand
Text Understanding with the Attention Sum Reader Network
TLDR
A new, simple model is presented that uses attention to directly pick the answer from the context as opposed to computing the answer using a blended representation of words in the document as is usual in similar models, making the model particularly suitable for question-answering problems where the answer is a single word from the document. Expand
WikiQA: A Challenge Dataset for Open-Domain Question Answering
TLDR
The WIKIQA dataset is described, a new publicly available set of question and sentence pairs, collected and annotated for research on open-domain question answering, which is more than an order of magnitude larger than the previous dataset. Expand
Automatic Generation of Challenging Distractors Using Context-Sensitive Inference Rules
TLDR
This work proposes to employ context-sensitive lexical inference rules in order to generate distractors that are semantically similar to the gap target word in some sense, but not in the particular sense induced by the gap-fill context. Expand
MS MARCO: A Human Generated MAchine Reading COmprehension Dataset
TLDR
This new dataset is aimed to overcome a number of well-known weaknesses of previous publicly available datasets for the same task of reading comprehension and question answering, and is the most comprehensive real-world dataset of its kind in both quantity and quality. Expand
Answering Elementary Science Questions by Constructing Coherent Scenes using Background Knowledge
TLDR
This work shows that by using a simple “knowledge graph” representation of the question, it can leverage several large-scale linguistic resources to provide missing background knowledge, somewhat alleviating the knowledge bottleneck in previous approaches. Expand
A Selection Strategy to Improve Cloze Question Quality
We present a strategy to improve the quality of automatically generated cloze and open cloze questions which are used by the REAP tutoring system for assessment in the ill-defined domain of EnglishExpand
...
1
2
3
4
5
...