• Publications
  • Influence
On the Relation Between Assessor's Agreement and Accuracy in Gamified Relevance Assessment
TLDR
Whether the agreement level can be used as a metric for estimating the quality of assessor's judgments is investigated, and recommendations for the design of judgments collection workflow are provided. Expand
Practice of Efficient Data Collection via Crowdsourcing: Aggregation, Incremental Relabelling, and Pricing
TLDR
This tutorial will make an introduction to data labelling via public crowdsourcing marketplaces and will present key components of efficient label collection, and invite beginners, advanced specialists, and researchers to learn how to efficiently collect labelled data. Expand
Practice of Efficient Data Collection via Crowdsourcing at Large-Scale
TLDR
An introduction to data labeling via public crowdsourcing marketplaces and key components of efficient label collection are presented and rich industrial experiences of applying these algorithms and constructing large-scale label collection pipelines are shared. Expand
Crowdsourcing Practice for Efficient Data Labeling: Aggregation, Incremental Relabeling, and Pricing
TLDR
This tutorial will make an introduction to data labeling via public crowdsourcing marketplaces and will present the key components of efficient label collection and the major theoretical results in efficient aggregation, incremental relabeling, and dynamic pricing. Expand
Crowdsourcing Natural Language Data at Scale: A Hands-On Tutorial
TLDR
This tutorial will make an introduction to data labeling via public crowdsourcing marketplaces and will present the key components of efficient label collection. Expand