Amazon's Mechanical Turk

@article{Buhrmester2011AmazonsMT,
  title={Amazon's Mechanical Turk},
  author={Michael D. Buhrmester and Tracy Nai Kwang and Samuel D. Gosling},
  journal={Perspectives on Psychological Science},
  year={2011},
  volume={6},
  pages={3 - 5}
}
Amazon’s Mechanical Turk (MTurk) is a relatively new website that contains the major elements required to conduct research: an integrated participant compensation system; a large participant pool; and a streamlined process of study design, participant recruitment, and data collection. In this article, we describe and evaluate the potential contributions of MTurk to psychology and other social sciences. Findings indicate that (a) MTurk participants are slightly more demographically diverse than… 

Tables from this paper

Data collection in a flat world: the strengths and weaknesses of mechanical turk samples

TLDR
MTurk offers a highly valuable opportunity for data collection, and it is recommended that researchers using MTurk include screening questions that gauge attention and language comprehension, avoid questions with factual answers, and consider how individual differences in financial and social domains may influence results.

Validity and Mechanical Turk: An assessment of exclusion methods and interactive experiments

Running Behavioral Operations Experiments Using Amazon's Mechanical Turk

TLDR
MTurk appears to be an important and relevant tool for researchers in behavioral operations, but researchers are cautioned to restrict use of this subject pool to experiments involving short-lived stimuli and behavioral manipulations.

Inside the Turk

Mechanical Turk (MTurk), an online labor market created by Amazon, has recently become popular among social scientists as a source of survey and experimental data. The workers who populate this

A reliability analysis of Mechanical Turk data

  • S. Rouse
  • Psychology
    Comput. Hum. Behav.
  • 2015

Amazon Mechanical Turk: A Web-Based Tool for Facilitating Experimental Research in ANLP

TLDR
This chapter describes the steps for acquiring IRB approval of MTurk experiments, designing experiments using the MTurK dashboard, and managing data, and discusses the potential benefits and limitations of using M Turk for ANLP experimentation.

TurkPrime.com: A versatile crowdsourcing data acquisition platform for the behavioral sciences

TLDR
How TurkPrime saves time and resources, improves data quality, and allows researchers to design and implement studies that were previously very difficult or impossible to carry out on MTurk is described.

The importance of assessing clinical phenomena in Mechanical Turk research.

TLDR
It is argued that MTurk participants differ from the general population in meaningful ways, and researchers should consider this when referring to this sample as truly representative.
...

References

SHOWING 1-10 OF 22 REFERENCES

Should we trust web-based studies? A comparative analysis of six preconceptions about internet questionnaires.

TLDR
Internet data collection methods, with a focus on self-report questionnaires from self-selected samples, are evaluated and compared with traditional paper-and-pencil methods and it is concluded that Internet methods can contribute to many areas of psychology.

Validating Internet research: A test of the psychometric equivalence of Internet and in-person samples

  • P. MeyersonW. Tryon
  • Psychology
    Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc
  • 2003
TLDR
The results suggest that data collection on the Web is (1) reliable, (2) valid, (3) reasonably representative, (4) cost effective, and (5) efficient.

The internet as psychological laboratory.

TLDR
Strengths and weaknesses of Web-based psychological research in general, and a sample of studies in particular, are reviewed with special attention to possible concerns about sampling and the use of deception.

College sophomores in the laboratory: Influences of a narrow data base on social psychology's view of human nature.

For the 2 decades prior to 1960, published research in social psychology was based on a wide variety of subjects and research sites. Content analyses show that since then such research has

Financial incentives and the "performance of crowds"

TLDR
It is found that increased financial incentives increase the quantity, but not the quality, of work performed by participants, where the difference appears to be due to an "anchoring" effect.

The Advent of Internet Surveys for Political Research: A Comparison of Telephone and Internet Samples

The Internet offers a number of advantages as a survey mode: low marginal cost per completed response, capabilities for providing respondents with large quantities of information, speed, and

The Big Five Trait taxonomy: History, measurement, and theoretical perspectives.

2 Taxonomy is always a contentious issue because the world does not come to us in neat little packages (S. Personality has been conceptualized from a variety of theoretical perspectives, and at

Stalking the perfect measure of implicit self-esteem: the blind men and the elephant revisited?

TLDR
Investigation of the reliability and validity of seven implicit self-esteem measures found some evidence that implicitSelfesteem measures are sensitive to context, but overall, the implicit measures were limited in their ability to predict criterion variables.