Corpus ID: 207847375

Active Multi-Label Crowd Consensus

  title={Active Multi-Label Crowd Consensus},
  author={Jinzheng Tu and Guoxian Yu and C. Domeniconi and J. Wang and X. Zhang},
Crowdsourcing is an economic and efficient strategy aimed at collecting annotations of data through an online platform. Crowd workers with different expertise are paid for their service, and the task requester usually has a limited budget. How to collect reliable annotations for multi-label data and how to compute the consensus within budget is an interesting and challenging, but rarely studied, problem. In this paper, we propose a novel approach to accomplish Active Multi-label Crowd Consensus… Expand
1 Citations
Crowdsourcing with Multiple-Source Knowledge Transfer
The Crowdsourcing with Multiple-source Knowledge Transfer approach to transfer knowledge from multiple, similar, but different domains for a new task, and to reduce the negative impact of irrelevant sources is introduced. Expand


Multi-Label Inference for Crowdsourcing
This paper proposes a novel probabilistic method, which includes a multi-class multi-label dependency (MCMLD) model, and introduces a mixture of multiple independently multinoulli distributions to capture the correlation among the labels. Expand
Computing Crowd Consensus with Partial Agreement
A novel Bayesian nonparametric model is proposed to aggregate the partial-agreement answers in a generic way and consistently outperforms the state-of-the-art in terms of precision, recall, and robustness against faulty workers and data sparsity. Expand
Multi-Label Learning from Crowds
The proposed NAM/RAM (Neighborhood/Relevance Aware Multi-label crowdsourcing) modeling the crowds’ expertise and label correlations from different perspectives is extended to the active paradigm and proposed instance, label, and worker selection criteria such that the labeling cost is significantly saved compared to passive learning without labeling control. Expand
Groupsourcing: Team Competition Designs for Crowdsourcing
This paper introduces strategies for team based crowdsourcing, ranging from team formation processes where workers are randomly assigned to competing teams, over strategies involving self-organization where workers actively participate in team building, to combinations of team and individual competitions. Expand
Active Learning for Crowdsourcing Using Knowledge Transfer
A new probabilistic model is proposed that transfers knowledge from abundant unlabeled data in auxiliary domains to help estimate labelers' expertise and outperforms other state-of-the-art active learning methods. Expand
Domain-Weighted Majority Voting for Crowdsourcing
This paper proposes to learn the weights for weighted MV by exploiting the expertise of annotators, model the domain knowledge of different annotators with different distributions and treat the crowdsourcing problem as a domain adaptation problem. Expand
Multicategory Crowdsourcing Accounting for Variable Task Difficulty, Worker Skill, and Worker Intention
The proposed methods can defeat “tyranny of the masses”, i.e., they are especially advantageous when there is an (a priori unknown) minority of skilled workers amongst a large crowd of unskilled (and malicious) workers. Expand
Crowdsourced Label Aggregation Using Bilayer Collaborative Clustering
A novel bilayer collaborative clustering (BLCC) method for the label aggregation in crowdsourcing that first generates the conceptual-level features for the instances from their multiple noisy labels and infers the initially integrated labels by performing clustering on the conceptual -level features. Expand
Quality Control for Crowdsourced Multi-label Classification Using RAkEL
By incorporating an existing aggregation method for single-label classification tasks into RAkEL, this work proposes a novel quality control method for crowdsourced multi- label classification and demonstrates that it achieves better quality than the existing methods with real data especially when spammers are included in the worker pool. Expand
Multi-label Answer Aggregation Based on Joint Matrix Factorization
A Multi-Label answer aggregation approach based on Joint Matrix Factorization (ML-JMF), which can identify unreliable annotators, and is robust against their misleading answers through the assignment of small (zero) weights to their annotation. Expand