Utility data annotation with Amazon Mechanical Turk

@article{Sorokin2008UtilityDA,
  title={Utility data annotation with Amazon Mechanical Turk},
  author={A. Sorokin and D. Forsyth},
  journal={2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops},
  year={2008},
  pages={1-8}
}
  • A. Sorokin, D. Forsyth
  • Published 2008
  • Computer Science
  • 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops
We show how to outsource data annotation to Amazon Mechanical Turk. Doing so has produced annotations in quite large numbers relatively cheaply. The quality is good, and can be checked and controlled. Annotations are produced quickly. We describe results for several different annotation problems. We describe some strategies for determining when the task is well specified and properly priced. 

Figures, Tables, and Topics from this paper

Collecting Image Annotations Using Amazon's Mechanical Turk
Online crowdsourcing: Rating annotators and obtaining cost-effective labels
  • P. Welinder, P. Perona
  • Computer Science
  • 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops
  • 2010
Data Quality from Crowdsourcing: A Study of Annotation Selection Criteria
How reliable are annotations via crowdsourcing: a study about inter-annotator agreement for multi-label image annotation
Quality Assessment for Crowdsourced Object Annotations
Efficiently Scaling up Crowdsourced Video Annotation
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 29 REFERENCES
LabelMe: A Database and Web-Based Tool for Image Annotation
Evaluation of Localized Semantics: Data, Methodology, and Experiments
Labeling images with a computer game
Building a Large Annotated Corpus of English: The Penn Treebank
Peekaboom: a game for locating objects in images
Caltech-256 Object Category Dataset
Learning to parse images of articulated bodies
A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics
...
1
2
3
...