Managing the 'Evaluator Effect' in User Testing

@inproceedings{Vermeeren2003ManagingT,
  title={Managing the 'Evaluator Effect' in User Testing},
  author={Arnold P. O. S. Vermeeren and Ilse van Kesteren and Mathilde M. Bekker},
  booktitle={INTERACT},
  year={2003}
}
If multiple evaluators analyse the outcomes of a single user test, the agreement between their lists of identified usability problems tends to be limited. This is called the ‘evaluator effect’. In the present paper, three user tests, taken from various domains, are reported and evaluator effects were measured. In all three studies, the evaluator effect proved to be less than in Jacobsen et al.'s (1998) study, but still present. Through detailed analysis of the data, it was possible to identify… CONTINUE READING