• Corpus ID: 209140255

Practice of Efficient Data Collection via Crowdsourcing at Large-Scale

  title={Practice of Efficient Data Collection via Crowdsourcing at Large-Scale},
  author={Alexey Drutsa and Viktoriya Farafonova and Valentina Fedorova and Olga Megorskaya and Evfrosiniya Zerminova and Olga Zhilinskaya},
Modern machine learning algorithms need large datasets to be trained. Crowdsourcing has become a popular approach to label large datasets in a shorter time as well as at a lower cost comparing to that needed for a limited number of experts. However, as crowdsourcing performers are non-professional and vary in levels of expertise, such labels are much noisier than those obtained from experts. For this reason, in order to collect good quality data within a limited budget special techniques such… 
Crowdsourcing Practice for Efficient Data Labeling: Aggregation, Incremental Relabeling, and Pricing
This tutorial will make an introduction to data labeling via public crowdsourcing marketplaces and will present the key components of efficient label collection and the major theoretical results in efficient aggregation, incremental relabeling, and dynamic pricing.
Practice of Efficient Data Collection via Crowdsourcing: Aggregation, Incremental Relabelling, and Pricing
This tutorial will make an introduction to data labelling via public crowdsourcing marketplaces and will present key components of efficient label collection, and invite beginners, advanced specialists, and researchers to learn how to efficiently collect labelled data.
CrowdSpeech and VoxDIY: Benchmark Datasets for Crowdsourced Audio Transcription
A principled pipeline for constructing datasets of crowdsourced audio transcriptions in any novel domain is designed and its applicability on an under-resourced language is shown by constructing VOXDIY — a counterpart of CROWDSPEECH for the Russian language.
Prediction of Hourly Earnings and Completion Time on a Crowdsourcing Platform
The solution to the problem of predicting user performance is found that demonstrates improvement of prediction quality by up to 25% for hourly earnings and up to $32%$ completion time w.r.t. a naive baseline which is based solely on historical performance of users on tasks.
Docreader labeling system for line type classifier
We develop the document analysis system, which is able to extract text and text metadata (such as font size and style), and restore the document structure. Some parts of the pipeline are based on
A Quality Assessment Model for Blockchain-Based Crowdsourcing System
A trusted mechanism is introduced to construct a reputation model and a blockchain-based worker selection algorithm, based on which a new quality assessment algorithm is proposed, resulting in more accurate evaluation results.


Analysis of Minimax Error Rate for Crowdsourcing and Its Application to Worker Clustering Model
A minimax error rate is derived under more practical setting for a broader class of crowdsourcing models that includes the Dawid and Skene model as a special case and a worker clustering model is proposed, which is more practical than the DS model under real crowdsourcing settings.
Exploiting Commonality and Interaction Effects in Crowdsourcing Tasks Using Latent Factor Models
Crowdsourcing services such as the Amazon Mechanical Turk [1] are increasingly being used to annotate large datasets for machine learning and data mining applications. The crowdsourced data labels
How Many Workers to Ask?: Adaptive Exploration for Collecting High Quality Labels
This paper conducts a data analysis on an industrial crowdsourcing platform, and uses the observations from this analysis to design new stopping rules that use the workers' quality scores in a non-trivial manner.
Quality-Based Pricing for Crowdsourced Workers
The emergence of online paid crowdsourcing platforms, such as Amazon Mechanical Turk (AMT), presents us huge opportunities to distribute tasks to human workers around the world, on-demand and at
Approval Voting and Incentives in Crowdsourcing
This article introduces approval voting to utilize the expertise of workers who have partial knowledge of the true answer and coupling it with two strictly proper scoring rules, and establishes attractive properties of optimality and uniqueness of the scoring rules.
Pairwise ranking aggregation in a crowdsourced setting
This work proposes a new model to predict a gold-standard ranking that hinges on combining pairwise comparisons via crowdsourcing and formalizes this as an active learning strategy that incorporates an exploration-exploitation tradeoff and implements it using an efficient online Bayesian updating scheme.
Regularized Minimax Conditional Entropy for Crowdsourcing
This paper proposes a minimax conditional entropy principle to infer ground truth from noisy crowdsourced labels, and derives a unique probabilistic labeling model jointly parameterized by worker ability and item difficulty.
No Oops, You Won't Do It Again: Mechanisms for Self-correction in Crowdsourcing
This work proposes a two-stage setting for crowdsourcing where the worker first answers the questions, and is then allowed to change her answers after looking at a (noisy) reference answer, and develops mechanisms to incentivize workers to act appropriately.
Incremental Relabeling for Active Learning with Noisy Crowdsourced Annotations
  • Liyue Zhao, G. Sukthankar, R. Sukthankar
  • Computer Science
    2011 IEEE Third Int'l Conference on Privacy, Security, Risk and Trust and 2011 IEEE Third Int'l Conference on Social Computing
  • 2011
This work proposes an active learning method that is specifically designed to be robust to label noise and presents an application of the technique in the domain of activity recognition for eldercare and validate the proposed approach using both simulated and real-world experiments using Amazon Mechanical Turk.
Measuring Crowdsourcing Effort with Error-Time Curves
This work proposes a data-driven effort metric, ETA (error-time area), that can be used to determine a task's fair price and validate the ETA metric on ten common crowdsourcing tasks, finding that ETA closely tracks how workers would rank these tasks by effort.