Cheap and Fast - But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks

  title={Cheap and Fast - But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks},
  author={Rion Snow and Brendan T. O'Connor and Daniel Jurafsky and Andrew Y. Ng},
Human linguistic annotation is crucial for many natural language processing tasks but can be expensive and time-consuming. We explore the use of Amazon’s Mechanical Turk system, a significantly cheaper and faster method for collecting annotations from a broad base of paid non-expert contributors over the Web. We investigate five tasks: affect recognition, word similarity, recognizing textual entailment, event temporal ordering, and word sense disambiguation. For all five, we show high agreement… CONTINUE READING
Highly Influential
This paper has highly influenced 136 other papers. REVIEW HIGHLY INFLUENTIAL CITATIONS
Highly Cited
This paper has 2,118 citations. REVIEW CITATIONS


Publications citing this paper.
Showing 1-10 of 1,247 extracted citations

2,118 Citations

Citations per Year
Semantic Scholar estimates that this publication has 2,118 citations based on the available data.

See our FAQ for additional information.


Publications referenced by this paper.
Showing 1-10 of 32 references


  • James Pustejovsky, Patrick Hanks, +8 authors Marcia Lazo.
  • In
  • 2003
Highly Influential
7 Excerpts

Contextual Correlates of Semantic Similarity

  • George A. Miller, William G. Charles.
  • Language and Cognitive Processes, vol. 6, no. 1…
  • 1991
Highly Influential
4 Excerpts

Automatic Extraction of Useful Facet Terms from Text Documents

  • Wisam Dakka, Panagiotis G. Ipeirotis.
  • InProc. of ICDE-2008.
  • 2008

Evidence for Varying Search Results Summary Lengths

  • Michael Kaisser, Marti Hearst, John B. Lowe.
  • InProc. of ACL-2008.
  • 2008
1 Excerpt

Similar Papers

Loading similar papers…