Measuring the accuracy of diagnostic systems.

@article{Swets1988MeasuringTA,
  title={Measuring the accuracy of diagnostic systems.},
  author={John A. Swets},
  journal={Science},
  year={1988},
  volume={240 4857},
  pages={
          1285-93
        }
}
  • J. Swets
  • Published 3 June 1988
  • Computer Science
  • Science
Diagnostic systems of several kinds are used to distinguish between two classes of events, essentially "signals" and "noise". For them, analysis in terms of the "relative operating characteristic" of signal detection theory provides a precise and valid measure of diagnostic accuracy. It is the only measure available that is uninfluenced by decision biases and prior probabilities, and it places the performances of diverse systems on a common, easily interpreted scale. Representative values of… 
View on PubMed
wixtedlab.ucsd.edu
8,535 Citations
Highly Influential Citations
1,649
Background Citations
1,881
Methods Citations
2,203
Results Citations
101

8,535 Citations

ROC Analysis in the Evaluation of Intelligent Medical Systems

A new method for generating the probability density functions and confidence bounds for ROC points which is robust and accurate for any sample size is presented, particularly suited for evaluating the performance of systems where sample sizes are small.

Receiver-operating characteristic (ROC) plots: a fundamental evaluation tool in clinical medicine.

Receiver-operating characteristic (ROC) plots provide a pure index of accuracy by demonstrating the limits of a test's ability to discriminate between alternative states of health over the complete spectrum of operating conditions.

Classification of diagnostic tests.

  • E. Somoza
  • Computer Science
    International journal of bio-medical computing
  • 1994

Assessing Test Accuracy and its Clinical Consequences: A Primer for Receiver Operating Characteristic Curve Analysis

These are the basic features necessary for the acceptable performance of ROC curve analysis, which are not dealt with in any detail in current textbooks addressed to the needs of clinical chemists.

Evaluation of intelligent medical systems

This work establishes an important new method for generating pdf's, and provides an accurate and robust method of producing confidence intervals for ROC curves for the small sample sizes typical of intelligent medical systems.

Comparing and Optimizing Diagnostic Tests

  • E. SomozaD. Mossman
  • Medicine
    Medical decision making : an international journal of the Society for Medical Decision Making
  • 1992
Though it is not appropriate for all clinical situations, the ROC analysis method can be used for most diagnostic tests whenever information is desired for its own sake or when reducing uncertainty is the goal of testing.

Advances in statistical methodology for the evaluation of diagnostic and laboratory tests.

Two new simultaneous confidence regions associated with the ROC curve are presented, one based on Kolmogorov-Smirnov confidence bands for distribution functions and the other based on bootstrapping.

Receiver operating characteristic analysis for intelligent medical systems-a new approach for finding confidence intervals

A novel, robust and accurate method is proposed, derived from first principles, which calculates the probability density function for each point on a ROC curve for any given sample size, and provides an accurate and robust method of producing confidence intervals for ROC curves for the small sample sizes typical of intelligent medical systems.

Receiver operating characteristic studies and measurement errors.

Monte Carlo simulation indicates that, for several families of distributions, the bias-corrected estimators have smaller bias and comparable MSE to the usual estimator.

Relation of Observer Agreement to Accuracy According to a Two-receiver Signal Detection Model of Diagnosis

  • M. Quinn
  • Psychology
    Medical decision making : an international journal of the Society for Medical Decision Making
  • 1989
A model of test interpretation is defined which accounts for disagreement when observers use identical diagnostic strategies, and determines an optimal diagnostic strategy with advantages over maximum likelihood and Bayes diagnostic strategies.
...

References

SHOWING 1-10 OF 38 REFERENCES

Indices of discrimination or diagnostic accuracy: their ROCs and implied models.

  • J. Swets
  • Psychology
    Psychological bulletin
  • 1986
It is suggested that candidate indices be evaluated by calculating their relative operating characteristics (ROCs) so that the form of an index's ROC identifies the model of the discrimination process that is implied by the index, and that theoretical form can be compared with the forms of empirical ROCs.

Problems of spectrum and bias in evaluating the efficacy of diagnostic tests.

To determine why many diagnostic tests have proved to be valueless after optimistic introduction into medical practice, we reviewed a series of investigations and identified two major problems that…

ROC analysis applied to the evaluation of medical imaging techniques.

  • J. Swets
  • Medicine
    Investigative radiology
  • 1979
This paper presents a brief description of the ROC, and shows how it provides a measure of diagnostic accuracy that is free of judgmental bias.

ROC Methodology in Radiologic Imaging

  • C. Metz
  • Medicine
    Investigative radiology
  • 1986
This article develops ROC concepts in an intuitive way by identifying the fundamental issues that motivate ROC analysis and practical techniques for ROC data collection and data analysis.

On the scientific evaluation of diagnostic procedures.

Diagnosis is the cornerstone of modern clinical medicine. It requires accurate observation and rational deduction. In its most complete form it is the process of identifying a disease by…

An Evaluation of Polygraphers' Judgments: A Review From a Decision Theoretic Perspective

Hebrew University, Jerusalem, IsraelA critical discussion of the theory, practice, and evaluation process of the ControlQuestions Technique (CQT) is undertaken. Sources of possible contaminationin…

Form of empirical ROCs in discrimination and diagnostic tasks: implications for theory and measurement of performance.

  • J. Swets
  • Psychology
    Psychological bulletin
  • 1986
Measuring both accuracy and the decision criterion appropriately enables one to delineate which of these dependent variables is affected by which independent variables in psychological studies, and to assess the efficacy as well as the accuracy of diagnostic systems.

The Effect of Verification on the Assessment of Imaging Techniques

To assess relative accuracies of three different radiographic techniques, chest radiographs were taken of 66 patients with each of the three techniques, and the films were evaluated by six radiologists, finding that any one of the techniques could be shown to be better than the other, depending on how the truth was defined.

Construction of Receiver Operating Characteristic Curves when Disease Verification Is Subject to Selection Bias

  • R. GrayC. BeggR. Greenes
  • Mathematics
    Medical decision making : an international journal of the Society for Medical Decision Making
  • 1984
A method to provide an unbiased estimate of the ROC curve under these sample conditions is proposed, which requires information on the probability distribution of test results in the population from which the verified sample is drawn.

Assessment of diagnostic technologies.

A general protocol for rigorous evaluation of diagnostic systems in medicine was applied successfully in a comparative study of two radiologic techniques, and computed tomography was found to be substantially more accurate than radionuclide scanning.