The Challenge of Variable Effort Crowdsourcing and How Visible Gold Can Help

  title={The Challenge of Variable Effort Crowdsourcing and How Visible Gold Can Help},
  author={Danula Hettiachchi and Mike Schaekermann and Tristan McKinney and Matthew Lease},
  journal={Proceedings of the ACM on Human-Computer Interaction},
  pages={1 - 26}
We consider a class of variable effort human annotation tasks in which the number of labels required per item can greatly vary (e.g., finding all faces in an image, named entities in a text, bird calls in an audio recording, etc.). In such tasks, some items require far more effort than others to annotate. Furthermore, the per-item annotation effort is not known until after each item is annotated since determining the number of labels required is an implicit part of the annotation task itself… 
Investigating and Mitigating Biases in Crowdsourced Data
This workshop aims to foster discussion on ongoing research around biases in crowdsourced data and to identify future research directions to detect, quantify and mitigate biases before, during and after the labelling process such that both task requesters and crowd workers can benefit.
A Time-Series-Based New Behavior Trace Model for Crowd Workers That Ensures Quality Annotation
Two novel models based on workers’ behavior for task classification based on multiple time-series features with a machine learning classifier and a convolutional neural network classifier are proposed.
Can I only share my eyes? A Web Crowdsourcing based Face Partition Approach Towards Privacy-Aware Face Recognition
FaceCrowd is developed, a web crowdsourcing based face partition approach to improve the performance of current face recognition models by designing a novel crowdsourced partial face graph generated from privacy-preserved social media face images.


Crowd Worker Strategies in Relevance Judgment Tasks
This paper delves into the techniques and tools that highly experienced crowd workers use to be more efficient in completing crowdsourcing micro-tasks, and highlights the presence of frequently used shortcut patterns that can speed-up task completion, thus increasing the hourly wage of efficient workers.
Minimizing Efforts in Validating Crowd Answers
A probabilistic model is developed that allows the expert to guide the expert's work by collecting input on the most problematic cases, thereby achieving a set of high quality answers even if the expert does not validate the complete answer set.
How One Microtask Affects Another
A seemingly banal feature of nearly all crowdsourcing workflows that profoundly impacts workers' responses is identified, and it is shown that earlier tasks can have a strong influence on responses to later tasks, shifting the distribution of future responses by 30-50% (total variational distance).
Toward crowdsourcing micro-level behavior annotations: the challenges of interface, training, and generalization
OCTAB (Online Crowdsourcing Tool for Annotations of Behaviors), a web-based annotation tool that allows precise and convenient behavior annotations in videos, directly portable to popular crowdsourcing platforms, and a training module with specialized visualizations.
Revolt: Collaborative Crowdsourcing for Labeling Machine Learning Datasets
Revolt eliminates the burden of creating detailed label guidelines by harnessing crowd disagreements to identify ambiguous concepts and create rich structures (groups of semantically related items) for post-hoc label decisions.
Adversarial Attacks on Crowdsourcing Quality Control
It is shown that a mechanism based on gold questions, based on the use of a small set of tasks for which the requester knows the correct answer, is prone to an attack carried out by a group of colluding crowdworkers that is easy to implement and deploy.
Toward a Learning Science for Complex Crowdsourcing Tasks
This work explores how crowdworkers can be trained to tackle complex crowdsourcing tasks and shows that having workers validate the work of their peer workers can be even more effective than having them review expert examples if the authors only present solutions filtered by a threshold length.
Modeling Task Complexity in Crowdsourcing
A high-dimensional regression model is developed, to assess the influence of three classes of structural features on task complexity, and ultimately used to measure task complexity and show that features related to task complexity can improve the quality of task performance prediction, thus demonstrating the utility of complexity as a task modeling property.
Annotator Rationales for Labeling Tasks in Crowdsourcing
This work investigates asking judges to provide a specific form of rationale supporting each rating decision and suggests a win-win approach on an information retrieval task in which human judges rate the relevance of Web pages for different search topics.
CrowdDQS: Dynamic Question Selection in Crowdsourcing Systems
CrowdDQS is presented, a system that uses the most recent set of crowdsourced voting evidence to dynamically issue questions to workers on Amazon Mechanical Turk, and can accurately answer questions using up to 6x fewer votes than standard approaches.