Areas beneath the relative operating characteristics (ROC) and relative operating levels (ROL) curves: Statistical significance and interpretation

@article{Mason2002AreasBT,
  title={Areas beneath the relative operating characteristics (ROC) and relative operating levels (ROL) curves: Statistical significance and interpretation},
  author={Simon J. Mason and Nicholas E. Graham},
  journal={Quarterly Journal of the Royal Meteorological Society},
  year={2002},
  volume={128}
}
  • S. Mason, N. Graham
  • Published 1 July 2002
  • Environmental Science
  • Quarterly Journal of the Royal Meteorological Society
The areas beneath the relative (or receiver) operating characteristics (ROC) and relative operating levels (ROL) curves can be used as summary measures of forecast quality, but statistical significance tests for these areas are conducted infrequently in the atmospheric sciences. A development of signal‐detection theory, the ROC curve has been widely applied in the medical and psychology fields where significance tests and relationships to other common statistical methods have been established… 

Multiclass ROC Analysis

TLDR
An extension of the ROC curve to multiclass forecast problems is explored and several different approximations to the full extension are introduced using both artificial and actual forecast datasets.

An Extended Procedure for Implementing the Relative Operating Characteristic Graphical Method

Abstract The functional relationship between the relative operating characteristic (ROC) and the economic value (EV) graphical methods have been exploited to develop a hybrid procedure called the

Beyond accuracy: Measures for assessing machine learning models, pitfalls and guidelines

TLDR
Alternative performance measures are reviewed and recommended with respect to the goals of a specific machine learning model and their interpretation and practical aspects of model evaluation are focused on.

The Area Under the ROC Curve as a Measure of Clustering Quality

TLDR
This work elaborate on the use of AUC as an internal/relative measure of clustering quality, which is referred to as Area Under the Curve for Clustering (AUCC), and demonstrates that the AUCC of a given candidate clustering solution has an expected value under a null model of random clustering solutions, regardless of the size of the dataset.

Receiver operating characteristic - Wikipedia, the free encyclopedia

  • Computer Science
  • 2015
In statistics, a receiver operating characteristic (ROC), or ROC curve, is a graphical plot that illustrates the performance of a binary classifier system as its discrimination threshold is varied.

How Random Is a Classifier Given Its Area under Curve?

TLDR
This paper presents the exact probability distribution of the AUC of a truely random classifier, given a finite number of distinct genuine and imposter scores, and quantifies the "randomness" of the measured AUC.

Fuzzy ROC curves for unsupervised nonparametric ensemble techniques

TLDR
This paper explores a novel ensemble technique for unsupervised classification using nonparametric statistics, and the performance of different aggregation techniques for the combination of the multiple classification system decision values, or rankings, is illustrated.

Evaluating Probabilistic Forecasts with Bayesian Signal Detection Models

TLDR
This work proposes the use of signal detection theory (SDT) to evaluate the performance of both probabilistic forecasting systems and individual forecasters and shows how this approach allows ROC curves and AUCs to be applied to individuals within a group of forecasters, estimated as a function of time, and extended to measure differences in forecastability across different domains.

Objective evaluation metrics for automatic classification of EEG events

TLDR
The deficiencies of existing metrics for a seizure detection task are discussed and several new metrics are proposed that offer a more balanced view of performance, including a measure based on a concept borrowed from the spoken term detection literature, Actual Term-Weighted Value, and a new metric that accounts for the temporal alignment of the hypothesis to the reference annotation.
...

References

SHOWING 1-10 OF 78 REFERENCES

A New Approach for Testing the Significance of Differences Between ROC Curves Measured from Correlated Data

TLDR
Receiver Operating Characteristic analysis is now generally recognized as the most appropriate methodology for evaluating the diagnostic performance of medical imaging procedures and its theory and experimental methodology have been developed in considerable detail.

A method of comparing the areas under receiver operating characteristic curves derived from the same cases.

TLDR
This paper refines the statistical comparison of the areas under two ROC curves derived from the same set of patients by taking into account the correlation between the areas that is induced by the paired nature of the data.

Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach.

TLDR
A nonparametric approach to the analysis of areas under correlated ROC curves is presented, by using the theory on generalized U-statistics to generate an estimated covariance matrix.

An Evaluation of Methods for Estimating the Area Under the Receiver Operating Characteristic (ROC) Curve

  • R. CentorJ. Schwartz
  • Mathematics
    Medical decision making : an international journal of the Society for Medical Decision Making
  • 1985
TLDR
Generally, the nonparametric method yields lower area estimates than the maximum-likelihood-estimation technique, however, these differences generally were small, particularly with ROC curves derived from five or more cutoff points.

Comparing the Areas under More Than Two Independent ROC Curves

  • D. McClish
  • Mathematics
    Medical decision making : an international journal of the Society for Medical Decision Making
  • 1987
TLDR
The F test is recommended as the method of choice for comparing the areas, although for balanced designs the SR test, with its com putational simplicity, may be preferred.

The meaning and use of the area under a receiver operating characteristic (ROC) curve.

A representation and interpretation of the area under a receiver operating characteristic (ROC) curve obtained by the "rating" method, or by mathematical predictions based on patient characteristics,

Advances in statistical methodology for the evaluation of diagnostic and laboratory tests.

TLDR
Two new simultaneous confidence regions associated with the ROC curve are presented, one based on Kolmogorov-Smirnov confidence bands for distribution functions and the other based on bootstrapping.

The Area under the ROC Curve and Its Competitors

  • J. Hilden
  • Mathematics
    Medical decision making : an international journal of the Society for Medical Decision Making
  • 1991
TLDR
The area under the receiver operating characteristic (ROC) curve is shown here to be an inconsistent criterion: tests of indistinguishable clinical impacts may have different areas and a class of diagnosticity measures (DMs) of proven optimality is proposed instead.

Statistical Comparison of ROC Curves from Multiple Readers

TLDR
A new method of analysis is presented for the comparison of ROC curves from multiple readers that includes a non parametric estimation of the variances and covariances between the various areas under the curves.

Statistical significance tests for binormal ROC curves

...