Are We Modeling the Task or the Annotator? An Investigation of Annotator Bias in Natural Language Understanding Datasets

@inproceedings{Geva2019AreWM,
  title={Are We Modeling the Task or the Annotator? An Investigation of Annotator Bias in Natural Language Understanding Datasets},
  author={Mor Geva and Y. Goldberg and Jonathan Berant},
  booktitle={EMNLP/IJCNLP},
  year={2019}
}
  • Mor Geva, Y. Goldberg, Jonathan Berant
  • Published in EMNLP/IJCNLP 2019
  • Computer Science
  • Crowdsourcing has been the prevalent paradigm for creating natural language understanding datasets in recent years. [...] Key Result Our findings suggest that annotator bias should be monitored during dataset creation, and that test set annotators should be disjoint from training set annotators.Expand Abstract
    62 Citations

    Figures, Tables, and Topics from this paper.

    Evaluating NLP Models via Contrast Sets
    • 42
    • PDF
    Investigating Annotator Bias with a Graph-Based Approach
    • 1
    Evaluating Models' Local Decision Boundaries via Contrast Sets
    • 4
    • PDF
    Natural Language Inference with Mixed Effects
    Building Low-Resource NER Models Using Non-Speaker Annotation
    • 1
    • PDF
    OCNLI: Original Chinese Natural Language Inference
    Natural Perturbation for Robust Question Answering
    • 2

    References

    SHOWING 1-10 OF 29 REFERENCES
    Comparing Bayesian Models of Annotation
    • 17
    • PDF
    Crowdsourcing for NLP
    • 7
    • PDF
    A large annotated corpus for learning natural language inference
    • 1,557
    • PDF
    NewsQA: A Machine Comprehension Dataset
    • 333
    • PDF
    Corpus Annotation through Crowdsourcing: Towards Best Practice Guidelines
    • 112
    • PDF
    SQuAD: 100, 000+ Questions for Machine Comprehension of Text
    • 2,307
    • PDF
    CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge
    • 135
    • PDF