• DOI:10.48550/arXiv.2212.00007
• Corpus ID: 254125268

A Light-weight, Effective and Efficient Model for Label Aggregation in Crowdsourcing

@article{Yang2022ALE,
  title={A Light-weight, Effective and Efficient Model for Label Aggregation in Crowdsourcing},
  author={Yi Yang and Zhong-Qiu Zhao and Quan-wei Bai and Qing Liu and Weihua Li},
  journal={ArXiv},
  year={2022},
  volume={abs/2212.00007}
}

• Yi Yang, Zhong-Qiu Zhao, +2 authors Weihua Li
• Published 19 November 2022
• Computer Science
• ArXiv

Due to the noises in crowdsourced labels, label aggregation (LA) has emerged as a standard procedure to post-process crowdsourced labels. LA methods estimate true labels from crowdsourced labels by modeling worker qualities. Most existing LA methods are iterative in nature. They need to traverse all the crowdsourced labels multiple times in order to iteratively update true labels and worker qualities until convergence. Consequently, these methods have high space complexity O ( TM ) and time complexity O ( ITM ) , where T and M are the numbers of tasks and workers, respectively, and I is the number of iterations these algorithms take to converge. In this paper, we treat LA as a dynamic system and model it as a Dynamic Bayesian network. From the dynamic model we derive two light-weight algorithms, LA onepass and LA twopass , which can effectively and efficiently estimate worker qualities and true labels by traversing all the labels at most twice. As a result, the space and time complexities of the proposed algorithms are reduced to O ( M + T ) and O ( MT ) , respectively. Due to the dynamic nature and low complexities, the proposed algorithms can also estimate true labels online without re-visiting historical data. We theoretically prove the convergence of the proposed algorithms, and bound the error of estimated worker qualities. Experiments conducted on 20 real-world datasets demonstrate that the proposed algorithms can effectively and efficiently aggregate labels both offline and online even if they traverse all the labels at most twice.