Crowdsourcing with unsure option

@article{Ding2017CrowdsourcingWU,
  title={Crowdsourcing with unsure option},
  author={Yao-Xiang Ding and Zhi-Hua Zhou},
  journal={Machine Learning},
  year={2017},
  volume={107},
  pages={749-766}
}
One of the fundamental issues in crowdsourcing is the trade-off between the number of workers needed for high-accuracy aggregation and the budget to pay. To save cost, it is important to ensure high quality of the crowd-sourced labels, hence the total cost on label collection will be reduced. Since the confidence of the workers often has a close relationship with their abilities, a possible way for quality control is to request the workers to return the labels only when they feel confident, by… CONTINUE READING