Crowdsourcing with unsure option

  title={Crowdsourcing with unsure option},
  author={Yao-Xiang Ding and Zhi-Hua Zhou},
  journal={Machine Learning},
One of the fundamental issues in crowdsourcing is the trade-off between the number of workers needed for high-accuracy aggregation and the budget to pay. To save cost, it is important to ensure high quality of the crowd-sourced labels, hence the total cost on label collection will be reduced. Since the confidence of the workers often has a close relationship with their abilities, a possible way for quality control is to request the workers to return the labels only when they feel confident, by… CONTINUE READING