Do Mechanical Turks dream of square pie charts?
@inproceedings{Kosara2010DoMT, title={Do Mechanical Turks dream of square pie charts?}, author={Robert Kosara and Caroline Ziemkiewicz}, booktitle={BELIV '10}, year={2010} }
Online studies are an attractive alternative to the laborintensive lab study, and promise the possibility of reaching a larger variety and number of people than at a typical university. There are also a number of draw-backs, however, that have made these studies largely impractical so far.
Amazon's Mechanical Turk is a web service that facilitates the assignment of small, web-based tasks to a large pool of anonymous workers. We used it to conduct several perception and cognition studies, one…
98 Citations
Inside the Turk
- Sociology
- 2014
Mechanical Turk (MTurk), an online labor market created by Amazon, has recently become popular among social scientists as a source of survey and experimental data. The workers who populate this…
Investigating the Efficacy of Crowdsourcing on Evaluating Visual Decision Supporting System
- Computer Science
- 2011
This study replicated a controlled lab study of decision-making tasks with different sorting techniques using crowdsourcing, and found potential sources of problems that can be improved to make the crowdsourcing experiment more viable.
Repeated measures design in crowdsourcing-based experiments for visualization
- PsychologyBELIV
- 2014
The experience of using repeated measures in three empirical studies using MTurk suggests that crowdsourcing remains a valid experimental environment, provided that one can integrate the means to observe and alleviate the potential confounding effects of "unknown" or "uncontrolled" variables in the design of the experiment.
Using Online, Crowdsourcing Platforms for Data Collection in Personality Disorder Research: The Example of Amazon’s Mechanical Turk
- PsychologyPersonality disorders
- 2017
It is concluded that platforms such as MTurk have much to offer PD researchers, especially for certain kinds of research (e.g., where large samples are required and there is a need for iterative sampling).
Gamification as a paradigm for the evaluation of visual analytics systems
- Computer ScienceBELIV
- 2014
This work proposes a theory, methodology and framework that can allow any visual analytics researcher to turn his/her evaluation task into an entertaining online game and first experiences with a prototype have shown that such an approach allows ten-thousands of evaluations to be done in a matter of days at no cost which is completely unthinkable with conventional methods.
Exploring beyond simple demographic variables: Differences between traditional laboratory samples and crowdsourced online samples on the Big Five personality traits
- PsychologyPersonality and Individual Differences
- 2018
Perception of Average Value in Multiclass Scatterplots
- Computer ScienceIEEE Transactions on Visualization and Computer Graphics
- 2013
How the perception literature provides a set of expected constraints on the task is described, and predictions are evaluated with a large-scale perceptual study with crowd-sourced participants, which has concrete ramifications for the design of scatterplots.
The Weighted Average Illusion: Biases in Perceived Mean Position in Scatterplots
- Computer ScienceIEEE Transactions on Visualization and Computer Graphics
- 2022
This work illustrates how ensemble processing mechanisms and mental shortcuts can significantly distort visual summaries of data, and can lead to misjudgments like the demonstrated weighted average illusion.
Who Are the Turkers? A Characterization of MTurk Workers Using the Personality Assessment Inventory
- PsychologyAssessment
- 2019
The present study addresses limitations and broadens the scope of existing research through the use of the Personality Assessment Inventory, a multiscale, self-report questionnaire which provides information regarding data validity and personality and psychopathology features standardized against a national U.S. census–matched normative sample.
References
SHOWING 1-10 OF 16 REFERENCES
Who are the Turkers? Worker Demographics in Amazon Mechanical Turk
- Economics
- 2009
Amazon Mechanical Turk (MTurk) is a crowdsourcing system in which tasks are distributed to a population of thousands of anonymous workers for completion. This system is becoming increasingly popular…
Crowdsourcing user studies with Mechanical Turk
- Computer ScienceCHI
- 2008
Although micro-task markets have great potential for rapidly collecting user measurements at low costs, it is found that special care is needed in formulating tasks in order to harness the capabilities of the approach.
The sheep market
- ArtC&C '09
- 2009
The Sheep Market is a web-based artwork that appropriates Amazon's Mechanical Turk system to implicate thousands of workers in the creation of a massive database of drawings, calling attention to the massive and insignificant role each plays as part of a whole.
Crowdsourcing graphical perception: using mechanical turk to assess visualization design
- Computer ScienceCHI
- 2010
The viability of Amazon's Mechanical Turk as a platform for graphical perception experiments is assessed and cost and performance data are reported and recommendations for the design of crowdsourced studies are distill.
Preconceptions and Individual Differences in Understanding Visual Metaphors
- Psychology, ArtComput. Graph. Forum
- 2009
The results show that metaphor compatibility has a significant effect on accuracy, but that factors such as spatial ability and personality can lessen this effect, and a complex influence of self‐reported metaphor preference on performance.
Design Elements and the Perception of Information Structure
- Psychology
- 2009
While there has been significant research on how low-level perceptual elements contribute to a user’s ability to compare or discern data points, less is known about how seemingly meaningless…
Thoughts on User Studies: Why, How and When
- Art
- 1993
The main goal is to encourage the use of studies in visualization, but it is recognized that other disciplines also offer important insights into visualization design, for example, the areas of visual design or the visual arts.
User Studies: Why, How, and When?
- ArtIEEE Computer Graphics and Applications
- 2003
This article describes how the authors designed experiments to answer important questions from their own research and offers suggestions and lessons learned about experimental design.
The mini-IPIP scales: tiny-yet-effective measures of the Big Five factors of personality.
- PsychologyPsychological assessment
- 2006
The Mini-IPIP scales showed a comparable pattern of convergent, discriminant, and criterion-related validity with other Big Five measures, indicating that it is a psychometrically acceptable and practically useful short measure of the Big Five factors of personality.
How well do line drawings depict shape?
- ArtSIGGRAPH '09
- 2009
The findings suggest that people interpret certain shapes almost as well from a line drawing as from a shaded image, that current computer graphics line drawing techniques can effectively depict shape and even match the effectiveness of artist's drawings, and that errors in depiction are often localized and can be traced to particular properties of the lines used.