Whose Vote Should Count More: Optimal Integration of Labels from Labelers of Unknown Expertise

  title={Whose Vote Should Count More: Optimal Integration of Labels from Labelers of Unknown Expertise},
  author={Jacob Whitehill and Paul Ruvolo and Tingfan Wu and Jacob Bergsma and Javier R. Movellan},
Modern machine learning-based approaches to computer vision require very large databases of hand labeled images. Some contemporary vision systems already require on the order of millions of images for training (e.g., Omron face detector [9]). New Internet-based services allow for a large number of labelers to collaborate around the world at very low cost. However, using these services brings interesting theoretical and practical challenges: (1) The labelers may have wide ranging levels of… CONTINUE READING
Highly Influential
This paper has highly influenced 89 other papers. REVIEW HIGHLY INFLUENTIAL CITATIONS
Highly Cited
This paper has 930 citations. REVIEW CITATIONS


Publications citing this paper.
Showing 1-10 of 566 extracted citations

931 Citations

Citations per Year
Semantic Scholar estimates that this publication has 931 citations based on the available data.

See our FAQ for additional information.


Publications referenced by this paper.
Showing 1-10 of 21 references

Maximum likelihood estimation of observer error-rates using the em algorithm

  • A. Dawid, A. Skene
  • Applied Statistics,
  • 1979
Highly Influential
3 Excerpts

An infomax model of eye movement

  • A. Dawid, A. Skene
  • Proceedings of the International Conference on…
  • 2008

Human - Based Character Recognition via Web Security Measures

  • C. McMillen, D. Abraham, M. Blum
  • Proceedings of the SIGCHI conference on Human…
  • 2004

Similar Papers

Loading similar papers…