The face of quality in crowdsourcing relevance labels: demographics, personality and labeling accuracy

@inproceedings{Kazai2012TheFO,
  title={The face of quality in crowdsourcing relevance labels: demographics, personality and labeling accuracy},
  author={Gabriella Kazai and Jaap Kamps and Natasa Milic-Frayling},
  booktitle={CIKM},
  year={2012}
}
Information retrieval systems require human contributed relevance labels for their training and evaluation. Increasingly such labels are collected under the anonymous, uncontrolled conditions of crowdsourcing, leading to varied output quality. While a range of quality assurance and control techniques have now been developed to reduce noise during or after task completion, little is known about the workers themselves and possible relationships between workers' characteristics and the quality of… CONTINUE READING
Highly Cited
This paper has 76 citations. REVIEW CITATIONS

5 Figures & Tables

Topics

Statistics

01020201320142015201620172018
Citations per Year

77 Citations

Semantic Scholar estimates that this publication has 77 citations based on the available data.

See our FAQ for additional information.