Corpus ID: 237562915

A General-Purpose Crowdsourcing Computational Quality Control Toolkit for Python

  title={A General-Purpose Crowdsourcing Computational Quality Control Toolkit for Python},
  author={Dmitry Ustalov and Nikita Pavlichenko and Vladimir N. Losev and Evgeny Tulin and Iulian Giliazev},
Quality control is a crux of crowdsourcing. While most means for quality control are organizational and imply worker selection, golden tasks, and post-acceptance, computational quality control techniques allow parameterizing the whole crowdsourcing process of workers, tasks, and labels, inferring and revealing relationships between them. In this paper, we demonstrate Crowd-Kit, a general-purpose crowdsourcing computational quality control toolkit. It provides efficient implementations in Python… Expand

Tables from this paper

IMDB-WIKI-SbS: An Evaluation Dataset for Crowdsourced Pairwise Comparisons
The IMDB-WIKI-SbS dataset is presented, a new large-scale dataset for evaluating pairwise comparisons that contains 9,150 images appearing in 250,249 pairs annotated on a crowdsourcing platform and indicates its suitability for model evaluation. Expand


Truth Inference in Crowdsourcing: Is the Problem Solved?
It is believed that the truth inference problem is not fully solved, and the limitations of existing algorithms are identified and point out promising research directions. Expand
Microsoft COCO: Common Objects in Context
We present a new dataset with the goal of advancing the state-of-the-art in object recognition by placing the question of object recognition in the context of the broader question of sceneExpand
Calculating Worker Agreement with Aggregate (Wawa)
  • URL Bernstein, M. S.; Little, G.; Miller, R. C.; Hartmann, B.;
  • 2021
CrowdSpeech and Vox DIY: Benchmark Datasets for Crowdsourced Audio Transcription
Domain-specific data is the crux of the successful transfer of machine learning 1 systems from benchmarks to real life. Crowdsourcing has become one of the 2 standard tools for cheap andExpand
Adversarial Crowdsourcing Through Robust Rank-One Matrix Completion
This work proposes a new algorithm combining alternating minimization with extreme-value filtering and provide sufficient and necessary conditions to recover the original rank-one matrix when some of the revealed entries are corrupted with perturbations that are unknown and can be arbitrarily large. Expand
Crowdsourced Text Sequence Aggregation based on Hybrid Reliability and Representation
This work proposes a crowdsourced text sequence aggregation method which can capture the hybrid reliability information, i.e., the local question-wise reliability of text answers and global dataset- wise reliability of crowd workers and can effectively leverage the rich information of text sequences. Expand
A Dataset of Crowdsourced Word Sequences: Collections and Answer Aggregation for Ground Truth Creation
This work creates a dataset (CrowdWSA2019) which contains the translated sentences generated from multiple workers and provides three approaches as the baselines on the task of extractive word sequence aggregation which models the reliability of workers. Expand
Aggregation of pairwise comparisons with reduction of biases
A new aggregation model factorBT is introduced for pairwise comparisons, which accounts for certain factors of pairwise tasks that are known to be irrelevant to the result of comparisons but may affect workers' answers due to perceptual reasons. Expand
Uncertainty estimation in deep learning with application to spoken language assessment
Prior Networks combine the advantages of ensemble and single-model approaches to estimating uncertainty and are evaluated on a range classification datasets, where they are shown to outperform baseline approaches on the task of detecting out-of-distribution inputs. Expand
spark-crowd: A Spark Package for Learning from Crowdsourced Big Data
As the data sets increase in size, the process of manually labeling data becomes unfeasible by small groups of experts. Thus, it is common to rely on crowdsourcing platforms which provideExpand