Are We Modeling the Task or the Annotator? An Investigation of Annotator Bias in Natural Language Understanding Datasets

@article{Geva2019AreWM,
  title={Are We Modeling the Task or the Annotator? An Investigation of Annotator Bias in Natural Language Understanding Datasets},
  author={Mor Geva and Yoav Goldberg and Jonathan Berant},
  journal={ArXiv},
  year={2019},
  volume={abs/1908.07898}
}
Crowdsourcing has been the prevalent paradigm for creating natural language understanding datasets in recent years. A common crowdsourcing practice is to recruit a small number of high-quality workers, and have them massively generate examples. Having only a few workers generate the majority of examples raises concerns about data diversity, especially when workers freely generate sentences. In this paper, we perform a series of experiments showing these concerns are evident in three recent NLP… CONTINUE READING

Figures, Tables, and Topics from this paper.

Citations

Publications citing this paper.

An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction

Stefan Larson, Anish Mahendran, +8 authors Jason Mars
  • ArXiv
  • 2019
VIEW 1 EXCERPT

References

Publications referenced by this paper.
SHOWING 1-10 OF 28 REFERENCES

Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations

  • International Journal of Computer Vision
  • 2016
VIEW 1 EXCERPT
HIGHLY INFLUENTIAL

Comparing Bayesian Models of Annotation

  • Transactions of the Association for Computational Linguistics
  • 2018
VIEW 1 EXCERPT