Do Mechanical Turks dream of square pie charts?

  title={Do Mechanical Turks dream of square pie charts?},
  author={Robert Kosara and Caroline Ziemkiewicz},
  booktitle={BELIV '10},
Online studies are an attractive alternative to the laborintensive lab study, and promise the possibility of reaching a larger variety and number of people than at a typical university. There are also a number of draw-backs, however, that have made these studies largely impractical so far. Amazon's Mechanical Turk is a web service that facilitates the assignment of small, web-based tasks to a large pool of anonymous workers. We used it to conduct several perception and cognition studies, one… 

Figures and Tables from this paper

Inside the Turk
Mechanical Turk (MTurk), an online labor market created by Amazon, has recently become popular among social scientists as a source of survey and experimental data. The workers who populate this
Investigating the Efficacy of Crowdsourcing on Evaluating Visual Decision Supporting System
This study replicated a controlled lab study of decision-making tasks with different sorting techniques using crowdsourcing, and found potential sources of problems that can be improved to make the crowdsourcing experiment more viable.
Repeated measures design in crowdsourcing-based experiments for visualization
The experience of using repeated measures in three empirical studies using MTurk suggests that crowdsourcing remains a valid experimental environment, provided that one can integrate the means to observe and alleviate the potential confounding effects of "unknown" or "uncontrolled" variables in the design of the experiment.
Using Online, Crowdsourcing Platforms for Data Collection in Personality Disorder Research: The Example of Amazon’s Mechanical Turk
It is concluded that platforms such as MTurk have much to offer PD researchers, especially for certain kinds of research (e.g., where large samples are required and there is a need for iterative sampling).
Gamification as a paradigm for the evaluation of visual analytics systems
This work proposes a theory, methodology and framework that can allow any visual analytics researcher to turn his/her evaluation task into an entertaining online game and first experiences with a prototype have shown that such an approach allows ten-thousands of evaluations to be done in a matter of days at no cost which is completely unthinkable with conventional methods.
Perception of Average Value in Multiclass Scatterplots
How the perception literature provides a set of expected constraints on the task is described, and predictions are evaluated with a large-scale perceptual study with crowd-sourced participants, which has concrete ramifications for the design of scatterplots.
The Weighted Average Illusion: Biases in Perceived Mean Position in Scatterplots
This work illustrates how ensemble processing mechanisms and mental shortcuts can significantly distort visual summaries of data, and can lead to misjudgments like the demonstrated weighted average illusion.
Who Are the Turkers? A Characterization of MTurk Workers Using the Personality Assessment Inventory
The present study addresses limitations and broadens the scope of existing research through the use of the Personality Assessment Inventory, a multiscale, self-report questionnaire which provides information regarding data validity and personality and psychopathology features standardized against a national U.S. census–matched normative sample.


Who are the Turkers? Worker Demographics in Amazon Mechanical Turk
Amazon Mechanical Turk (MTurk) is a crowdsourcing system in which tasks are distributed to a population of thousands of anonymous workers for completion. This system is becoming increasingly popular
Crowdsourcing user studies with Mechanical Turk
Although micro-task markets have great potential for rapidly collecting user measurements at low costs, it is found that special care is needed in formulating tasks in order to harness the capabilities of the approach.
The sheep market
The Sheep Market is a web-based artwork that appropriates Amazon's Mechanical Turk system to implicate thousands of workers in the creation of a massive database of drawings, calling attention to the massive and insignificant role each plays as part of a whole.
Crowdsourcing graphical perception: using mechanical turk to assess visualization design
The viability of Amazon's Mechanical Turk as a platform for graphical perception experiments is assessed and cost and performance data are reported and recommendations for the design of crowdsourced studies are distill.
Preconceptions and Individual Differences in Understanding Visual Metaphors
The results show that metaphor compatibility has a significant effect on accuracy, but that factors such as spatial ability and personality can lessen this effect, and a complex influence of self‐reported metaphor preference on performance.
Design Elements and the Perception of Information Structure
While there has been significant research on how low-level perceptual elements contribute to a user’s ability to compare or discern data points, less is known about how seemingly meaningless
Thoughts on User Studies: Why, How and When
The main goal is to encourage the use of studies in visualization, but it is recognized that other disciplines also offer important insights into visualization design, for example, the areas of visual design or the visual arts.
User Studies: Why, How, and When?
This article describes how the authors designed experiments to answer important questions from their own research and offers suggestions and lessons learned about experimental design.
The mini-IPIP scales: tiny-yet-effective measures of the Big Five factors of personality.
The Mini-IPIP scales showed a comparable pattern of convergent, discriminant, and criterion-related validity with other Big Five measures, indicating that it is a psychometrically acceptable and practically useful short measure of the Big Five factors of personality.
How well do line drawings depict shape?
The findings suggest that people interpret certain shapes almost as well from a line drawing as from a shaded image, that current computer graphics line drawing techniques can effectively depict shape and even match the effectiveness of artist's drawings, and that errors in depiction are often localized and can be traced to particular properties of the lines used.