Corpus ID: 227230470

Improving Human-Labeled Data through Dynamic Automatic Conflict Resolution

  title={Improving Human-Labeled Data through Dynamic Automatic Conflict Resolution},
  author={David Q. Sun and Hadas Kotek and C. Klein and Mayank Gupta and William Li and J. Williams},
  • David Q. Sun, Hadas Kotek, +3 authors J. Williams
  • Published in COLING 2020
  • Computer Science
  • This paper develops and implements a scalable methodology for (a) estimating the noisiness of labels produced by a typical crowdsourcing semantic annotation task, and (b) reducing the resulting error of the labeling process by as much as 20-30% in comparison to other common labeling strategies. Importantly, this new approach to the labeling process, which we name Dynamic Automatic Conflict Resolution (DACR), does not require a ground truth dataset and is instead based on inter-project… CONTINUE READING

    Figures and Tables from this paper


    Cheap and Fast - But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks
    • 1,989
    • PDF
    Empirical Methodology for Crowdsourcing Ground Truth
    • 9
    • Highly Influential
    • PDF
    The Multidimensional Wisdom of Crowds
    • 713
    • PDF
    Inter-annotator Agreement
    • 34
    • PDF
    Inconsistency Detection in Semantic Annotation
    • 18
    • PDF
    Repeated labeling using multiple noisy labelers
    • 144
    • PDF
    Learning part-of-speech taggers with inter-annotator agreement loss
    • 60
    • PDF
    Actively Estimating Crowd Annotation Consensus
    • 2
    • PDF
    The Benefits of a Model of Annotation
    • 123
    • PDF