A Comparison of Consensus , Consistency , and Measurement Approaches to Estimating Interrater Reliability

  title={A Comparison of Consensus , Consistency , and Measurement Approaches to Estimating Interrater Reliability},
  author={Steven E. Stemler},
This article argues that the general practice of describing interrater reliability as a single, unified concept is at best imprecise, and at worst potentially misleading. Rather than representing a single concept, different statistical methods for computing interrater reliability can be more accurately classified into one of three categories based upon the underlying goals of analysis. The three general categories introduced and described in this paper are: 1) consensus estimates, 2… CONTINUE READING
56 Citations
28 References
Similar Papers


Publications citing this paper.
Showing 1-10 of 56 extracted citations


Publications referenced by this paper.
Showing 1-10 of 28 references

FACETS: a computer program for many-facet Rasch measurement (Version 3.3.0)

  • J. M. Linacre
  • 1988
Highly Influential
6 Excerpts

Many-facet Rasch measurement

  • J. M. Linacre
  • 1994
Highly Influential
5 Excerpts

A comparison of consensus, consistency, and measurement approaches to estimating interrater reliability

  • Citation Stemler, E Steven
  • Practical Assessment, Research & Evaluation,
  • 2004

Applied multiple regression/correlation analysis for the behavioral sciences (Third ed.)

  • J. Cohen, P. Cohen, S. G. West, L. S. Aiken
  • 2003

Assessing the reliability of rating data

  • Barrett, March
  • Retrieved June
  • 2003

The restriction of variance hypothesis and interrater reliability and agreement: Are ratings from multiple sources really dissimilar

  • J. M. LeBreton, J. R. Burgess, R. B. Kaiser, E. Atchley, L. R. James
  • Organizational Research Methods,
  • 2003

Similar Papers

Loading similar papers…