Comparative usability evaluation

@article{Molich2004ComparativeUE,
  title={Comparative usability evaluation},
  author={Rolf Molich and Meghan R. Ede and Klaus Kaasgaard and Barbara Karyukin},
  journal={Behaviour \& Information Technology},
  year={2004},
  volume={23},
  pages={65 - 74}
}
This paper reports on a study assessing the consistency of usability testing across organisations. [...] Key Result Even the tasks used by most or all teams produced very different results - around 70% of the findings for each of these tasks were unique. Our main conclusion is that our simple assumption that we are all doing the same and getting the same results in a usability test is plainly wrong.Expand
Comparative usability evaluation (CUE-4)
TLDR
The study shows that there was no practical difference between the results obtained from usability testing and expert reviews for the issues identified, and it was not possible to prove the existence of either missed problems or false alarms in expert reviews. Expand
Comparing Usability Problem Identification and Description by Practitioners and Students
TLDR
There was no difference in the number of problems reported by students and practitioners, but there was a difference in their ratings for following several of the guidelines, which provides a more complete assessment of usability reports. Expand
Usability Problem Reports for Comparative Studies: Consistency and Inspectability
TLDR
It was found that consistency of single analyst teams varied considerably and that a method like SlimDEVAN can help in making the analysis process and findings more inspectable. Expand
Heuristic evaluation: Comparing ways of finding and reporting usability problems
TLDR
An empirical study of a framework to evaluate the effectiveness of different types of support for structured usability problem reporting found that there were no significant differences between any of the four groups in effectiveness, efficiency and inter-evaluator reliability. Expand
Component-Specific Usability Testing
TLDR
A meta-analysis is carried out on the results of six experiments to support the claim that component-specific usability measures are on average statistically more powerful than overall usability measures when comparing different versions of a part of a system. Expand
Are We Testing Utility? Analysis of Usability Problem Types
TLDR
Usability problems and related redesign recommendations are the main outcome of usability tests although both are questioned in terms of impact in the design process, and early usability testing with a think-aloud protocol and an open task structure measure both utility and usability equally well. Expand
Artifacts, Tools and Generalizing Usability Test Results
TLDR
The effects of the reliability and validity problems on the application of usability testing and its role as one of the tools in the design process are discussed. Expand
Making a difference: a survey of the usability profession in Sweden
TLDR
The results indicate, among other things, that management support and project management support are essential for the usability worker, and they face problems such as, usability and user involvement having low priority in the projects. Expand
On the performance of novice evaluators in usability evaluations
The paper investigates the performance of novice evaluators in usability evaluations by presenting the results of a comparative usability evaluation that was conducted by nine novice evaluator teams.Expand
Making usability recommendations useful and usable
TLDR
The study finds that only 14 of the 84 studied comments addressing six usability problems contained recommendations that were both useful and usable, which means that half of the recommendations were not useful at all. Expand
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 24 REFERENCES
On the reliability of usability testing
TLDR
Six professional usability testing teams conducted a usability test on an early prototype of a dialog box and identified 36 usability problems, which showed more agreement among teams in this study compared to a previous study. Expand
Refining the Test Phase of Usability Evaluation: How Many Subjects Is Enough?
  • R. Virzi
  • Engineering, Computer Science
  • 1992
TLDR
Three experiments are reported in this paper that relate the proportion of usability problems identified in an evaluation to the number of subjects participating in that study, finding that 80% of the usability problems are detected with four or five subjects. Expand
The evaluator effect in usability tests
TLDR
In this study, four evaluators analyzed four videotaped usability test sessions and found that the evaluator effect had little effect on the reliability of usability tests. Expand
Comparative evaluation of usability tests
TLDR
Seven professional usability labs and one university student team have carried out independent, parallel usability tests of the same state-of-the-art, live, commercial web site, resulting in similarities and differences in process, results and reporting. Expand
The Evaluator Effect in Usability Studies: Problem Detection and Severity Judgments
TLDR
Both detection of usability problems and selection of the most severe problems are subject to considerable individual variability. Expand
The Evaluator Effect: A Chilling Fact About Usability Evaluation Methods
TLDR
It is certainly notable that a substantial evaluator effect persists for evaluators who apply the strict procedure of CW or observe users thinking out loud, and it is highly questionable to use a TA with 1evaluator as an authoritative statement about what problems an interface contains. Expand
Damaged Merchandise? A Review of Experiments That Compare Usability Evaluation Methods
TLDR
In this review, the design of 5 experiments that compared usability evaluation methods (UEMs) are examined, showing that small problems in the way these experiments were designed and conducted call into serious question what the authors thought they knew regarding the efficacy of various UEMs. Expand
A Practical Guide to Usability Testing
From the Publisher: In A Practical Guide to Usability Testing, the authors begin by defining usability, advocating and explaining the methods of usability engineering and reviewing many techniquesExpand
A mathematical model of the finding of usability problems
For 11 studies, we find that the detection of usability problems as a function of number of users tested or heuristic evaluators employed is well modeled as a Poisson process. The model can be usedExpand
SUS: A 'Quick and Dirty' Usability Scale
Usability does not exist in any absolute sense; it can only be defined with reference to particular contexts. This, in turn, means that there are no absolute measures of usability, since, if theExpand
...
1
2
3
...