Weighted Krippendorff's alpha is a more reliable metrics for multi-coders ordinal annotations: experimental studies on emotion, opinion and coreference annotation

@inproceedings{Antoine2014WeightedKA,
  title={Weighted Krippendorff's alpha is a more reliable metrics for multi-coders ordinal annotations: experimental studies on emotion, opinion and coreference annotation},
  author={Jean-Yves Antoine and Jeanne Villaneau and Ana{\"i}s Lefeuvre},
  booktitle={EACL},
  year={2014}
}
The question of data reliability is of first importance to assess the quality of manually annotated corpora. Although Cohen’s κ is the prevailing reliability measure used in NLP, alternative statistics have been proposed. This paper presents an experimental study with four measures (Cohen’s κ, Scott’s π, binary and weighted Krippendorff ’ s α) on three tasks: emotion, opinion and coreference annotation. The reported studies investigate the factors of influence (annotator bias, category… CONTINUE READING

From This Paper

Figures, tables, and topics from this paper.

Citations

Publications citing this paper.
SHOWING 1-10 OF 23 CITATIONS

References

Publications referenced by this paper.
SHOWING 1-10 OF 37 REFERENCES

Influence of contextual information in emotion annotation for spoken dialogue

  • Zoraida Callejas, Ramon Lopez-Cozar
  • 2008
Highly Influential
3 Excerpts

Affective Interaction with a Companion Robot for vulnerable Children : a Linguistically based Model for Emotion Detection

  • Brian MacWhinney
  • 2011

Affective Interaction with a Companion Robot for vulnerable Children: a Linguistically based Model for Emotion

  • Sage Thousand Oaks, CA Marc Le Tallec, Jeanne Villaneau, Jean-Yves Antoine, Dominique Duhaut
  • 2011

Similar Papers

Loading similar papers…