Do Mechanical Turks dream of square pie charts?

@inproceedings{Kosara2010DoMT,
  title={Do Mechanical Turks dream of square pie charts?},
  author={Robert Kosara and Caroline Ziemkiewicz},
  booktitle={BELIV '10},
  year={2010}
}
Online studies are an attractive alternative to the laborintensive lab study, and promise the possibility of reaching a larger variety and number of people than at a typical university. There are also a number of draw-backs, however, that have made these studies largely impractical so far. Amazon's Mechanical Turk is a web service that facilitates the assignment of small, web-based tasks to a large pool of anonymous workers. We used it to conduct several perception and cognition studies, one… 
View on ACM
kosara.net
98 Citations
Highly Influential Citations
3
Background Citations
29
Methods Citations
13
Results Citations
12

Figures and Tables from this paper

98 Citations

Citation Type

Citation Type

Inside the Turk
Mechanical Turk (MTurk), an online labor market created by Amazon, has recently become popular among social scientists as a source of survey and experimental data. The workers who populate this…
Investigating the Efficacy of Crowdsourcing on Evaluating Visual Decision Supporting System
TLDR
This study replicated a controlled lab study of decision-making tasks with different sorting techniques using crowdsourcing, and found potential sources of problems that can be improved to make the crowdsourcing experiment more viable.
Repeated measures design in crowdsourcing-based experiments for visualization
TLDR
The experience of using repeated measures in three empirical studies using MTurk suggests that crowdsourcing remains a valid experimental environment, provided that one can integrate the means to observe and alleviate the potential confounding effects of "unknown" or "uncontrolled" variables in the design of the experiment.
Crowdsourcing Samples in Cognitive Science
Using Online, Crowdsourcing Platforms for Data Collection in Personality Disorder Research: The Example of Amazon’s Mechanical Turk
TLDR
It is concluded that platforms such as MTurk have much to offer PD researchers, especially for certain kinds of research (e.g., where large samples are required and there is a need for iterative sampling).
Gamification as a paradigm for the evaluation of visual analytics systems
TLDR
This work proposes a theory, methodology and framework that can allow any visual analytics researcher to turn his/her evaluation task into an entertaining online game and first experiences with a prototype have shown that such an approach allows ten-thousands of evaluations to be done in a matter of days at no cost which is completely unthinkable with conventional methods.
Exploring beyond simple demographic variables: Differences between traditional laboratory samples and crowdsourced online samples on the Big Five personality traits
Perception of Average Value in Multiclass Scatterplots
TLDR
How the perception literature provides a set of expected constraints on the task is described, and predictions are evaluated with a large-scale perceptual study with crowd-sourced participants, which has concrete ramifications for the design of scatterplots.
The Weighted Average Illusion: Biases in Perceived Mean Position in Scatterplots
TLDR
This work illustrates how ensemble processing mechanisms and mental shortcuts can significantly distort visual summaries of data, and can lead to misjudgments like the demonstrated weighted average illusion.
Who Are the Turkers? A Characterization of MTurk Workers Using the Personality Assessment Inventory
TLDR
The present study addresses limitations and broadens the scope of existing research through the use of the Personality Assessment Inventory, a multiscale, self-report questionnaire which provides information regarding data validity and personality and psychopathology features standardized against a national U.S. census–matched normative sample.
...
...

References

SHOWING 1-10 OF 16 REFERENCES
Who are the Turkers? Worker Demographics in Amazon Mechanical Turk
Amazon Mechanical Turk (MTurk) is a crowdsourcing system in which tasks are distributed to a population of thousands of anonymous workers for completion. This system is becoming increasingly popular…
Crowdsourcing user studies with Mechanical Turk
TLDR
Although micro-task markets have great potential for rapidly collecting user measurements at low costs, it is found that special care is needed in formulating tasks in order to harness the capabilities of the approach.
The sheep market
TLDR
The Sheep Market is a web-based artwork that appropriates Amazon's Mechanical Turk system to implicate thousands of workers in the creation of a massive database of drawings, calling attention to the massive and insignificant role each plays as part of a whole.
Crowdsourcing graphical perception: using mechanical turk to assess visualization design
TLDR
The viability of Amazon's Mechanical Turk as a platform for graphical perception experiments is assessed and cost and performance data are reported and recommendations for the design of crowdsourced studies are distill.
Preconceptions and Individual Differences in Understanding Visual Metaphors
TLDR
The results show that metaphor compatibility has a significant effect on accuracy, but that factors such as spatial ability and personality can lessen this effect, and a complex influence of self‐reported metaphor preference on performance.
Design Elements and the Perception of Information Structure
While there has been significant research on how low-level perceptual elements contribute to a user’s ability to compare or discern data points, less is known about how seemingly meaningless…
Thoughts on User Studies: Why, How and When
TLDR
The main goal is to encourage the use of studies in visualization, but it is recognized that other disciplines also offer important insights into visualization design, for example, the areas of visual design or the visual arts.
User Studies: Why, How, and When?
TLDR
This article describes how the authors designed experiments to answer important questions from their own research and offers suggestions and lessons learned about experimental design.
The mini-IPIP scales: tiny-yet-effective measures of the Big Five factors of personality.
TLDR
The Mini-IPIP scales showed a comparable pattern of convergent, discriminant, and criterion-related validity with other Big Five measures, indicating that it is a psychometrically acceptable and practically useful short measure of the Big Five factors of personality.
How well do line drawings depict shape?
TLDR
The findings suggest that people interpret certain shapes almost as well from a line drawing as from a shaded image, that current computer graphics line drawing techniques can effectively depict shape and even match the effectiveness of artist's drawings, and that errors in depiction are often localized and can be traced to particular properties of the lines used.
...
...