Cheap and Fast - But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks

@inproceedings{Snow2008CheapAF,
  title={Cheap and Fast - But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks},
  author={R. Snow and Brendan T. O'Connor and Dan Jurafsky and A. Ng},
  booktitle={EMNLP},
  year={2008}
}
Human linguistic annotation is crucial for many natural language processing tasks but can be expensive and time-consuming. We explore the use of Amazon's Mechanical Turk system, a significantly cheaper and faster method for collecting annotations from a broad base of paid non-expert contributors over the Web. We investigate five tasks: affect recognition, word similarity, recognizing textual entailment, event temporal ordering, and word sense disambiguation. For all five, we show high agreement… Expand
Using the crowd for readability prediction
Amazon Mechanical Turk for Subjectivity Word Sense Disambiguation
Facilitating Corpus Annotation by Improving Annotation Aggregation
Anveshan: A Framework for Analysis of Multiple Annotators' Labeling Behavior
Collecting Image Annotations Using Amazon's Mechanical Turk
Scaling Semantic Frame Annotation
Fast, Cheap, and Creative: Evaluating Translation Quality Using Amazon's Mechanical Turk
Paragraph Acquisition and Selection for List Question Using Amazon's Mechanical Turk
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 44 REFERENCES
Scaling to Very Very Large Corpora for Natural Language Disambiguation
Creating a Research Collection of Question Answer Sentence Pairs with Amazon's Mechanical Turk
Automatic Extraction of Useful Facet Hierarchies from Text Databases
Building a Sense Tagged Corpus with Open Mind Word Expert
SWAT-MP: The SemEval-2007 Systems for Task 5 and Task 14
The Proposition Bank: An Annotated Corpus of Semantic Roles
Utility data annotation with Amazon Mechanical Turk
  • A. Sorokin, D. Forsyth
  • Computer Science
  • 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops
  • 2008
Verbosity: a game for collecting common-sense facts
...
1
2
3
4
5
...