Do Mechanical Turks dream of square pie charts?

@inproceedings{Kosara2010DoMT,
  title={Do Mechanical Turks dream of square pie charts?},
  author={Robert Kosara and Caroline Ziemkiewicz},
  booktitle={BELIV '10},
  year={2010}
}
Online studies are an attractive alternative to the laborintensive lab study, and promise the possibility of reaching a larger variety and number of people than at a typical university. There are also a number of draw-backs, however, that have made these studies largely impractical so far. Amazon's Mechanical Turk is a web service that facilitates the assignment of small, web-based tasks to a large pool of anonymous workers. We used it to conduct several perception and cognition studies, one… Expand
View on ACM
kosara.net
97 Citations
Highly Influential Citations
2
Background Citations
25
Methods Citations
13
Results Citations
10

97 Citations

Citation Type

Citation Type

Inside the Turk
Mechanical Turk (MTurk), an online labor market created by Amazon, has recently become popular among social scientists as a source of survey and experimental data. The workers who populate this… Expand
Investigating the Efficacy of Crowdsourcing on Evaluating Visual Decision Supporting System
Crowdsourcing recently became a popular approach to substitute time consuming and expensive human subject studies, but its application is generally limited to simple and short-term experimental… Expand
Repeated measures design in crowdsourcing-based experiments for visualization
TLDR
The experience of using repeated measures in three empirical studies using MTurk suggests that crowdsourcing remains a valid experimental environment, provided that one can integrate the means to observe and alleviate the potential confounding effects of "unknown" or "uncontrolled" variables in the design of the experiment. Expand
Crowdsourcing Samples in Cognitive Science
Crowdsourcing data collection from research participants recruited from online labor markets is now common in cognitive science. We review who is in the crowd and who can be reached by the average… Expand
Using Online, Crowdsourcing Platforms for Data Collection in Personality Disorder Research: The Example of Amazon’s Mechanical Turk
TLDR
It is concluded that platforms such as MTurk have much to offer PD researchers, especially for certain kinds of research (e.g., where large samples are required and there is a need for iterative sampling). Expand
Gamification as a paradigm for the evaluation of visual analytics systems
TLDR
This work proposes a theory, methodology and framework that can allow any visual analytics researcher to turn his/her evaluation task into an entertaining online game and first experiences with a prototype have shown that such an approach allows ten-thousands of evaluations to be done in a matter of days at no cost which is completely unthinkable with conventional methods. Expand
Exploring beyond simple demographic variables: Differences between traditional laboratory samples and crowdsourced online samples on the Big Five personality traits
Abstract Amazon's Mechanical Turk (MTurk), a popular crowdsourcing website, is increasingly being utilized by researchers to obtain psychological data. This transition has prompted evaluation of… Expand
Perception of Average Value in Multiclass Scatterplots
TLDR
How the perception literature provides a set of expected constraints on the task is described, and predictions are evaluated with a large-scale perceptual study with crowd-sourced participants, which has concrete ramifications for the design of scatterplots. Expand
The Weighted Average Illusion: Biases in Perceived Mean Position in Scatterplots
  • Matt-Heun Hong, J. Witt, D. Szafir
  • Computer Science, Medicine
  • IEEE transactions on visualization and computer graphics
  • 2021
TLDR
This work illustrates how ensemble processing mechanisms and mental shortcuts can significantly distort visual summaries of data, and can lead to misjudgments like the demonstrated weighted average illusion. Expand
Who Are the Turkers? A Characterization of MTurk Workers Using the Personality Assessment Inventory
TLDR
The present study addresses limitations and broadens the scope of existing research through the use of the Personality Assessment Inventory, a multiscale, self-report questionnaire which provides information regarding data validity and personality and psychopathology features standardized against a national U.S. census–matched normative sample. Expand
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 16 REFERENCES
Who are the Turkers? Worker Demographics in Amazon Mechanical Turk
Amazon Mechanical Turk (MTurk) is a crowdsourcing system in which tasks are distributed to a population of thousands of anonymous workers for completion. This system is becoming increasingly popular… Expand
Crowdsourcing user studies with Mechanical Turk
TLDR
Although micro-task markets have great potential for rapidly collecting user measurements at low costs, it is found that special care is needed in formulating tasks in order to harness the capabilities of the approach. Expand
The sheep market
TLDR
The Sheep Market is a web-based artwork that appropriates Amazon's Mechanical Turk system to implicate thousands of workers in the creation of a massive database of drawings, calling attention to the massive and insignificant role each plays as part of a whole. Expand
Crowdsourcing graphical perception: using mechanical turk to assess visualization design
TLDR
The viability of Amazon's Mechanical Turk as a platform for graphical perception experiments is assessed and cost and performance data are reported and recommendations for the design of crowdsourced studies are distill. Expand
Preconceptions and Individual Differences in Understanding Visual Metaphors
TLDR
The results show that metaphor compatibility has a significant effect on accuracy, but that factors such as spatial ability and personality can lessen this effect, and a complex influence of self‐reported metaphor preference on performance. Expand
Design Elements and the Perception of Information Structure
While there has been significant research on how low-level perceptual elements contribute to a user’s ability to compare or discern data points, less is known about how seemingly meaningless… Expand
Thoughts on User Studies: Why, How and When
TLDR
The main goal is to encourage the use of studies in visualization, but it is recognized that other disciplines also offer important insights into visualization design, for example, the areas of visual design or the visual arts. Expand
User Studies: Why, How, and When?
TLDR
This article describes how the authors designed experiments to answer important questions from their own research and offers suggestions and lessons learned about experimental design. Expand
The mini-IPIP scales: tiny-yet-effective measures of the Big Five factors of personality.
TLDR
The Mini-IPIP scales showed a comparable pattern of convergent, discriminant, and criterion-related validity with other Big Five measures, indicating that it is a psychometrically acceptable and practically useful short measure of the Big Five factors of personality. Expand
Perceptual Organization in User-Generated Graph Layouts
  • F. V. Ham, B. Rogowitz
  • Computer Science, Medicine
  • IEEE Transactions on Visualization and Computer Graphics
  • 2008
TLDR
It is found that observers are able to recover cluster structure, that the distance between clusters is inversely related to the strength of the clustering, and that users exhibit the tendency to use edges to visually delineate perceptual groups. Expand
...
1
2
...