A mathematical model of the finding of usability problems

@article{Nielsen1993AMM,
  title={A mathematical model of the finding of usability problems},
  author={Jakob Nielsen and Thomas K. Landauer},
  journal={Proceedings of the INTERACT '93 and CHI '93 Conference on Human Factors in Computing Systems},
  year={1993}
}
  • J. NielsenT. Landauer
  • Published 1 May 1993
  • Computer Science
  • Proceedings of the INTERACT '93 and CHI '93 Conference on Human Factors in Computing Systems
For 11 studies, we find that the detection of usability problems as a function of number of users tested or heuristic evaluators employed is well modeled as a Poisson process. [] Key Result For a “medium” example, we estimate that 16 evaluations would be worth their cost, with maximum benefit/cost ratio at four.
View on ACM
dl.acm.org
1,811 Citations
Highly Influential Citations
77
Background Citations
676
Methods Citations
268
Results Citations
31

Figures and Tables from this paper

1,811 Citations

Estimating sample size for usability testing

This study analyzed data collected from the user testing of a web application to verify the rule of thumb, commonly known as the “magic number 5”, and showed that the 5-user rule significantly underestimates the required sample size to achieve reasonable levels of problem detection.

Using a Dynamic Model to Simulate the Heuristic Evaluation of Usability

A dynamic simulation model is presented to analyze how different configurations of evaluator team have an effect upon the results of the heuristic evaluation method.

Number of Subjects in Web Usability Testing

This study provides evidence of the usefulness of SPRT in situations where determination of effectiveness is the goal, at a substantial reduction of number of users, and offers an alternative approach employing the Sequential Probability Ratio Test (SPRT) to determine product effectiveness.

Effect of Level of Problem Description on Problem Discovery Rates: Two Case Studies

This analysis investigated the effect of changing the level of description of usability problems on the estimate of the problem discovery rate (p), and described a method for using p to estimate the number of problems remaining available for discovery given the constraints associated with a particular participant population, application, and set of tasks.

Determining the Effectiveness of the Usability Problem Inspector: A Theory-Based Model and Tool for Finding Usability Problems

This work developed the user action framework and a corresponding evaluation tool, the usability problem inspector, to help organize usability concepts and issues into a knowledge base and showed that the UPI scored higher than heuristic evaluation in terms of thoroughness, validity, and effectiveness and was consistent with cognitive walkthrough for these same measures.

Effects of Importance and Detectability of Usability Problems on Sample Size Requirements

A case study is presented, which illustrates the utility of the proposed methodology and indicates that the current methods underestimate the number of participants in usability evaluation studies.

A Tale of Two Studies

Two (very different) approaches to designing an evaluation study for the same piece of software are described and both the approaches taken, the differing results found and the authors' comments on both of these are discussed.

Analysis of combinatorial user effect in international usability tests

Five aspects of user effect are explored, including optimality of sample size, evaluator effect, effect of heterogeneous subgroups, performance of task variants, and efficiency of problem discovery.

Extreme Discount Usability Engineering

This paper explores the circumstances under which extremely discounted usability engineering techniques produce results that are worthwhile to researchers and practitioners. We present a method for…

Quantifying the Usability Through a Variant of the Traditional Heuristic Evaluation Process

This case study presents the results of applying a variant to the traditional heuristic evaluation process to determine the usability degree of two web applications and serves as a framework for specialists in this field that are interested in retrieving quantitative data in addition to thetraditional usability issues that are identified.
...

References

SHOWING 1-10 OF 26 REFERENCES

Refining the Test Phase of Usability Evaluation: How Many Subjects Is Enough?

Three experiments are reported in this paper that relate the proportion of usability problems identified in an evaluation to the number of subjects participating in that study, finding that 80% of the usability problems are detected with four or five subjects.

Finding usability problems through heuristic evaluation

Usability specialists were better than non-specialists at performing heuristic evaluation, and “double experts” with specific expertise in the kind of interface being evaluated performed even better.…

A Cost-Effective Evaluation Method for Use by Designers

Comparison of empirical testing and walkthrough methods in user interface evaluation

The findings were replicated across the two systems and show that the empirical testing condition identified the largest number of problems, and identified a significant number of relatively severe problems that were missed by the walkthrough conditions.

User interface evaluation in the real world: a comparison of four techniques

A user interface for a software product was evaluated prior to its release by four groups, each applying a different technique: heuristic evaluation, software guidelines, cognitive walkthroughs, and usability testing.

Streamlining the Design Process: Running Fewer Subjects

An experiment conducted to determine the minimum number of subjects required for a usability test finds that with between 4 and 5 subjects, 80% of the usability problems are detected and that additional subjects are less and less likely to reveal new information.

Heuristic evaluation of user interfaces

Four experiments showed that individual evaluators were mostly quite bad at doing heuristic evaluations and that they only found between 20 and 51% of the usability problems in the interfaces they evaluated.

Estimating the number of subjects needed for a thinking aloud test

  • J. Nielsen
  • Computer Science
    Int. J. Hum. Comput. Stud.
  • 1994
Abstract Two studies of using the thinking aloud method for user interface testing showed that experimenters who were not usability specialists could use the method. However, they found only 28-30%…

Iterative user-interface design

A method for developing user interfaces by refining them iteratively over several versions, which not only eliminates problems of this nature, but also allows designers to take advantage of any insights into user needs that emerge from the tests.

Some Graphical Aids for Deciding When to Stop Testing Software

Several novel graphical procedures which are easy to implement are proposed, and these are illustrated for data from a large telecommunications software system.