Validity and Mechanical Turk: An assessment of exclusion methods and interactive experiments
@article{Thomas2017ValidityAM, title={Validity and Mechanical Turk: An assessment of exclusion methods and interactive experiments}, author={Kyle A. Thomas and Scott Clifford}, journal={Comput. Hum. Behav.}, year={2017}, volume={77}, pages={184-197} }
Tables from this paper
295 Citations
Online panels in social science research: Expanding sampling methods beyond Mechanical Turk
- PsychologyBehavior research methods
- 2019
It is concluded that online research panels offer a unique opportunity for research, yet one with some important trade-offs, as compared with traditional student subject pools.
Amazon Mechanical Turk workers can provide consistent and economically meaningful data
- Economics
- 2018
We explore the consistency of the characteristics of individuals who participate in studies posted on Amazon Mechanical Turk (AMT). The primary individuals analyzed in this study are subjects who…
Exclusion Criteria in Experimental Philosophy
- PsychologyErkenntnis
- 2019
When experimental philosophers carry out studies on thought experiments, some participants are excluded based on certain exclusion criteria, mirroring standard social science vignette methodology.…
The micro-task market for lemons: data quality on Amazon's Mechanical Turk
- BiologyPolitical Science Research and Methods
- 2021
While there is no evidence of a “bot epidemic,” significant portions of the data on MTurk are of dubious quality and many respondents appear to respond humorously or insincerely, and these low-quality responses attenuate observed treatment effects.
The Experimenters' Dilemma: Inferential Preferences over Populations
- Economics
- 2021
We compare three populations commonly used in experiments by economists and other social scientists: undergraduate students at a physical location (lab), Amazon’s Mechanical Turk (MTurk), and…
MTurk, Prolific or Panels? Choosing the Right Audience for Online Research
- Business
- 2021
Researchers in various fields of behavioral research are increasingly using online audiences to conduct studies and surveys, but there is still considerable uncertainty about the quality of the…
Distinction without a difference? An assessment of MTurk Worker types
- Business
- 2020
Amazon’s Mechanical Turk (MTurk) platform is a popular tool for scholars seeking a reasonably representative population to recruit subjects for academic research that is cheaper than contract work…
Improving Data Quality Using Amazon Mechanical Turk Through Platform Setup
- BusinessCornell Hospitality Quarterly
- 2021
Focusing on task setup, selected platform-level strategies that have received relatively less attention in previous research are empirically tested to further enhance the contribution of the proposed best practices for MTurk usage.
Analyze the Attentive & Bypass Bias: Mock Vignette Checks in Survey Experiments
- Psychology
- 2020
Respondent inattentiveness threatens to undermine experimental studies. In response, researchers incorporate measures of attentiveness into their analyses, yet often in a way that risks introducing…
Noncompliant responding: Comparing exclusion criteria in MTurk personality research to improve data quality
- PsychologyPersonality and Individual Differences
- 2019
References
SHOWING 1-10 OF 110 REFERENCES
Data Collection in a Flat World: The Strengths and Weaknesses of Mechanical Turk Samples
- Psychology
- 2012
MTurk offers a highly valuable opportunity for data collection, and it is recommended that researchers using MTurk include screening questions that gauge attention and language comprehension, avoid questions with factual answers, and consider how individual differences in financial and social domains may influence results.
Evaluating Amazon's Mechanical Turk as a Tool for Experimental Behavioral Research
- PsychologyPloS one
- 2013
This paper replicates a diverse body of tasks from experimental psychology including the Stroop, Switching, Flanker, Simon, Posner Cuing, attentional blink, subliminal priming, and category learning tasks using participants recruited using AMT.
Amazon's Mechanical Turk
- PsychologyPerspectives on psychological science : a journal of the Association for Psychological Science
- 2011
Findings indicate that MTurk can be used to obtain high-quality data inexpensively and rapidly and the data obtained are at least as reliable as those obtained via traditional methods.
Are samples drawn from Mechanical Turk valid for research on political ideology?
- Sociology
- 2015
Amazon’s Mechanical Turk (MTurk) is an increasingly popular tool for the recruitment of research subjects. While there has been much focus on the demographic differences between MTurk samples and the…
Attentive Turkers: MTurk participants perform better on online attention checks than do subject pool participants
- PsychologyBehavior research methods
- 2016
In three online studies, participants from MTurk and collegiate populations participated in a task that included a measure of attentiveness to instructions (an instructional manipulation check: IMC), and MTurkers were more attentive to the instructions than were college students, even on novel IMCs.
Crowdsourcing performance evaluations of user interfaces
- Computer ScienceCHI
- 2013
MTurk may be a productive setting for conducting performance evaluations of user interfaces providing a complementary approach to existing methodologies, and three previously well-studied user interface designs are evaluated.
A reliability analysis of Mechanical Turk data
- PsychologyComput. Hum. Behav.
- 2015
Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk
- BusinessPolitical Analysis
- 2012
It is shown that respondents recruited in this manner are often more representative of the U.S. population than in-person convenience samples but less representative than subjects in Internet-based panels or national probability samples.
Generalizing from Survey Experiments Conducted on Mechanical Turk: A Replication Approach
- PsychologyPolitical Science Research and Methods
- 2018
To what extent do survey experimental treatment effect estimates generalize to other populations and contexts? Survey experiments conducted on convenience samples have often been criticized on the…
The pitfall of experimenting on the web: How unattended selective attrition leads to surprising (yet false) research conclusions.
- PsychologyJournal of personality and social psychology
- 2016
The authors find that experimental studies using online samples (e.g., MTurk) often violate the assumption of random assignment, because participant attrition-quitting a study before completing it…