Understanding the Crowd: Ethical and Practical Matters in the Academic Use of Crowdsourcing

@inproceedings{Martin2015UnderstandingTC,
  title={Understanding the Crowd: Ethical and Practical Matters in the Academic Use of Crowdsourcing},
  author={David B. Martin and M. Sheelagh T. Carpendale and Neha Gupta and Tobias Hossfeld and Babak Naderi and Judith Redi and Ernestasia Siahaan and Ina Wechsung},
  booktitle={Crowdsourcing and Human-Centered Experiments},
  year={2015}
}
The driving force behind digital crowdsourcing are its workers: working, hidden behind the scenes, churning out data in experiments, participating in research studies, completing little tasks to accomplish HITs online. Understanding workers and crowdwork better is therefore key to develop a more effective and fair use of crowdsourcing for research. This chapter attempts to help develop an understanding of the various aspects of the crowd by drawing parallels between workers of different… 
Using Attention Testing to Select Crowdsourced Workers and Research Participants
TLDR
The web-delivered attention test attentiveWeb was developed and validated in two versions to come up with advance filters to identify workers who produce low-quality results and to gauge the attention of workers who pass the advance filter.
An investigation of factors affecting the visits of online crowdsourcing and labor platforms
TLDR
Investigation and analyze the visits of online labor platforms that offer crowdsourcing and crowdfunding services and shows that the sessions of an online labor marketplace website from mobile devices have an increasing trend to be positively correlated to the quality mechanisms a website deploys as well as on location-dependent factors.
Characterizing the Global Crowd Workforce: A Cross-Country Comparison of Crowdworker Demographics
TLDR
This study is the first large-scale country-level analysis of the characteristics of workers on the platform Figure Eight (formerly CrowdFlower), one of the two platforms dominating the micro-task market.
Monotasking or Multitasking: Designing for Crowdworkers' Preferences
TLDR
Practical design options that could enable workers, requesters and platform developers to make adjustments that would improve crowdworkers' experiences are enumerated.
CrowdWatcher: an open-source platform to catch the eye of the crowd
TLDR
The proposed open-source eye tracking platform CrowdWatcher enables researchers to measure gaze location and user engagement in a crowdsourcing context through traditional RGB webcams and provides complementary information to traditional self-reported data by taking gaze analysis into account.
Effects of personalization on participation in crowd-civic systems = 시민참여형 시스템에서 개인화가 참여에 미치는 영향
TLDR
This thesis designs a crowd-civic platform for collecting community issues with personalized motivationsupportive messages and conducts two studies: a pair-comparison experiment with 150 participants on Amazon’s Mechanical Turk and a live deployment study with 120 university members.
Crowdworkers’ Temporal Flexibility is Being Traded for the Convenience of Requesters Through 19 ‘Invisible Mechanisms’ Employed by Crowdworking Platforms: A Comparative Analysis Study of Nine Platforms
Crowdworking platforms are a prime example of a product that sells flexibility to its consumers. In this paper, we argue that crowdworking platforms sell temporal flexibility to requesters to the
Personalized Motivation-supportive Messages for Increasing Participation in Crowd-civic Systems
TLDR
This paper designed a crowd-civic platform for collecting community issues with personalized motivation-supportive messages and conducted two studies: a pair-comparison experiment with 150 participants on Amazon's Mechanical Turk and a live deployment study with 120 university members, indicating applicability of SDT's perspective in crowd- civic systems.
Information Visualization Evaluation Using Crowdsourcing
TLDR
A review of the use of crowdsourcing for evaluation in visualization research, which analyzed 190 crowdsourcing experiments reported in 82 papers that were published in major visualization conferences and journals between 2006 and 2017.
Explorer Information Visualization Evaluation Using Crowdsourcing
TLDR
A review of the use of crowdsourcing for evaluation in visualization research, which analyzed 190 crowdsourcing experiments reported in 82 papers that were published in major visualization conferences and journals between 2006 and 2017.
...
...

References

SHOWING 1-10 OF 48 REFERENCES
Working the Crowd: Employment and Labor Law in the Crowdsourcing Industry
This Article confronts the thorny questions that arise in attempting to apply traditional employment and labor law to “crowdsourcing,” an emerging online labor model unlike any that has existed to
An ethnographic study of crowdwork via Amazon Mechanical Turk in India
TLDR
This thesis aims to provide insights into the work of crowdwork, what entails ‘doing crowdwork’, from the perspective of the workers who partake in crowdsourced work through online platforms, with a particular focus on Amazon Mechanical Turk (AMT) as the principle site for work.
Experiences surveying the crowd: reflections on methods, participation, and reliability
TLDR
The results point to the value of participant engagement, identify potential changes in MTurk as a study venue, and highlight how communication among Turkers influences the data that researchers collect.
More than fun and money. Worker Motivation in Crowdsourcing - A Study on Mechanical Turk
TLDR
This work adapts different models from classic motivation theory, work motivation theory and Open Source Software Development to crowdsourcing markets and finds that the extrinsic motivational categories have a strong effect on the time spent on the platform.
Being a turker
TLDR
An ethnomethodological analysis of publicly available content on Turker Nation, a general forum for Amazon Mechanical Turk (AMT) users, provides novel depth and detail on how theTurker Nation members operate as economic actors, working out which Requesters and jobs are worthwhile to them.
Development and Validation of Extrinsic Motivation Scale for Crowdsourcing Micro-task Platforms
TLDR
A scale for measuring the extrinsic motivation of crowd workers is introduced and has been applied and validated in a crowdsourcing micro-task platform and theoretically follows the Self-Determination Theory of motivation.
The Dynamics of Micro-Task Crowdsourcing: The Case of Amazon MTurk
TLDR
This paper uses the main findings of the five year log analysis to propose features used in a predictive model aiming at determining the expected performance of any batch at a specific point in time, and shows that the number of tasks left in a batch and how recent the batch is are two key features of the prediction.
Anatomy of a Crowdsourcing Platform - Using the Example of Microworkers.com
TLDR
An inside view of the usage data from Micro workers is given and it is shown that there are significant differences to the well studied MTurk.
Beyond the Turk: An Empirical Comparison of Alternative Platforms for Crowdsourcing Online Behavioral Research
The success of Amazon Mechanical Turk (MTurk) as an online research platform has come at a price: MTurk exhibits slowing rates of population replenishment, and growing participants’ non-naivety.
We Are Dynamo: Overcoming Stalling and Friction in Collective Action for Crowd Workers
TLDR
Dynamo, a platform to support the Mechanical Turk community in forming publics around issues and then mobilizing, finds that collective action publics tread a precariously narrow path between the twin perils of stalling and friction, balancing with each step between losing momentum and flaring into acrimony.
...
...