Crowdsourcing Practice for Efficient Data Labeling: Aggregation, Incremental Relabeling, and Pricing

  title={Crowdsourcing Practice for Efficient Data Labeling: Aggregation, Incremental Relabeling, and Pricing},
  author={Alexey Drutsa and Dmitry Ustalov and Evfrosiniya Zerminova and Valentina Fedorova and Olga Megorskaya and Daria Baidakova},
  journal={Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data},
In this tutorial, we present a portion of unique industry experience in efficient data labeling via crowdsourcing shared by both leading researchers and engineers from Yandex. We will make an introduction to data labeling via public crowdsourcing marketplaces and will present the key components of efficient label collection. This will be followed by a practice session, where participants will choose one of the real label collection tasks, experiment with selecting settings for the labeling… 

Figures and Topics from this paper

Aggregation Techniques in Crowdsourcing: Multiple Choice Questions and Beyond
This tutorial aims to present common and recent label aggregation techniques for multiple-choice questions, multi-class labels, ratings, pairwise comparison, and image/text annotation.
CrowdSpeech and VoxDIY: Benchmark Datasets for Crowdsourced Audio Transcription
A principled pipeline for constructing datasets of crowdsourced audio transcriptions in any novel domain is designed and its applicability on an under-resourced language is shown by constructing VOXDIY — a counterpart of CROWDSPEECH for the Russian language.
Prediction of Hourly Earnings and Completion Time on a Crowdsourcing Platform
The solution to the problem of predicting user performance is found that demonstrates improvement of prediction quality by up to 25% for hourly earnings and up to $32%$ completion time w.r.t. a naive baseline which is based solely on historical performance of users on tasks.
Quality of Sentiment Analysis Tools: The Reasons of Inconsistency
A heuristic to assess the data quality of an augmented dataset and a new set of metrics to evaluate tool inconsistencies are introduced, indicating that tool inconsistencies is still an open problem, and they point towards promising research directions and accuracy improvements that can be obtained if such inconsistencies are resolved.


Practice of Efficient Data Collection via Crowdsourcing: Aggregation, Incremental Relabelling, and Pricing
This tutorial will make an introduction to data labelling via public crowdsourcing marketplaces and will present key components of efficient label collection, and invite beginners, advanced specialists, and researchers to learn how to efficiently collect labelled data.
Practice of Efficient Data Collection via Crowdsourcing at Large-Scale
An introduction to data labeling via public crowdsourcing marketplaces and key components of efficient label collection are presented and rich industrial experiences of applying these algorithms and constructing large-scale label collection pipelines are shared.
Exploiting Commonality and Interaction Effects in Crowdsourcing Tasks Using Latent Factor Models
Crowdsourcing services such as the Amazon Mechanical Turk [1] are increasingly being used to annotate large datasets for machine learning and data mining applications. The crowdsourced data labels
Pairwise ranking aggregation in a crowdsourced setting
This work proposes a new model to predict a gold-standard ranking that hinges on combining pairwise comparisons via crowdsourcing and formalizes this as an active learning strategy that incorporates an exploration-exploitation tradeoff and implements it using an efficient online Bayesian updating scheme.
Analysis of Minimax Error Rate for Crowdsourcing and Its Application to Worker Clustering Model
A minimax error rate is derived under more practical setting for a broader class of crowdsourcing models that includes the Dawid and Skene model as a special case and a worker clustering model is proposed, which is more practical than the DS model under real crowdsourcing settings.
Sprout: Crowd-Powered Task Design for Crowdsourcing
A novel meta-workflow is proposed that helps requesters optimize crowdsourcing task designs and Sprout, the open-source tool, which implements this workflow, improves task designs by eliciting points of confusion from crowd workers, enabling requesters to quickly understand these misconceptions and the overall space of questions.
Quality-Based Pricing for Crowdsourced Workers
The emergence of online paid crowdsourcing platforms, such as Amazon Mechanical Turk (AMT), presents us huge opportunities to distribute tasks to human workers around the world, on-demand and at
How Many Workers to Ask?: Adaptive Exploration for Collecting High Quality Labels
This paper conducts a data analysis on an industrial crowdsourcing platform, and uses the observations from this analysis to design new stopping rules that use the workers' quality scores in a non-trivial manner.
Regularized Minimax Conditional Entropy for Crowdsourcing
This paper proposes a minimax conditional entropy principle to infer ground truth from noisy crowdsourced labels, and derives a unique probabilistic labeling model jointly parameterized by worker ability and item difficulty.
Incremental Relabeling for Active Learning with Noisy Crowdsourced Annotations
  • Liyue Zhao, G. Sukthankar, R. Sukthankar
  • Computer Science
    2011 IEEE Third Int'l Conference on Privacy, Security, Risk and Trust and 2011 IEEE Third Int'l Conference on Social Computing
  • 2011
This work proposes an active learning method that is specifically designed to be robust to label noise and presents an application of the technique in the domain of activity recognition for eldercare and validate the proposed approach using both simulated and real-world experiments using Amazon Mechanical Turk.