The total operating characteristic to measure diagnostic ability for multiple thresholds

@article{Pontius2014TheTO,
  title={The total operating characteristic to measure diagnostic ability for multiple thresholds},
  author={Robert Gilmore Pontius and Kangping Si},
  journal={International Journal of Geographical Information Science},
  year={2014},
  volume={28},
  pages={570 - 583}
}
  • R. Pontius, Kangping Si
  • Published 1 March 2014
  • Business
  • International Journal of Geographical Information Science
The relative operating characteristic (ROC) is a popular statistical method to measure the association between observed and diagnosed presence of a characteristic. The diagnosis of presence or absence depends on whether the value of an index variable is above a threshold. ROC considers multiple possible thresholds. Each threshold generates a two-by-two contingency table, which contains four central entries: hits, misses, false alarms, and correct rejections. ROC reveals for each threshold only… 
Recommendations for using the relative operating characteristic (ROC)
The relative operating characteristic (ROC) is a widely-used method to measure diagnostic signals including predictions of land changes, species distributions, and ecological niches. The ROC measures
The Total Operating Characteristic from Stratified Random Sampling with an Application to Flood Mapping
TLDR
This manuscript derives a new methodology that uses stratified random sampling to generate the TOC and shows visually and quantitatively each index’s diagnostic ability relative to baselines.
Positive and negative predictive values by the TOC curve
Sensitivity and specificity are popular measures described by the receiver operating characteristic (ROC) curve. There are also two other measures such as the positive predictive value (PPV) and
The MCC-F1 curve: a performance evaluation technique for binary classification
TLDR
The MCC-F1 curve is proposed, which combines two informative single-threshold metrics, MCC and the F1 score, and provides a single value that integrates many aspects of classifier performance across the whole range of classification thresholds.
Techniques for the Validation of LUCC Modeling Outputs
TLDR
This chapter distinguishes between purely quantitative validation techniques and those that also consider the spatial allocation of simulated land use/cover changes (LUCC), and compares a group of the most common model software programs, in order to list their validation capabilities.
MODELING URBAN DYNAMICS USING RANDOM FOREST: IMPLEMENTING ROC AND TOC FOR MODEL EVALUATION
Abstract. The importance of spatial accuracy of land use/cover change maps necessitates the use of high performance models. To reach this goal, calibrating machine learning (ML) approaches to model
Landscape Applications of Machine Learning: Comparing Random Forests and Logistic Regression in Multi-Scale Optimized Predictive Modeling of American Marten Occurrence in Northern Idaho, USA
The American marten is a species that is dependent on old conifer forest at middle to high elevations and is highly sensitive to habitat loss and fragmentation. Our goal was to compoare logistic
A Comparative Study of Machine Learning Techniques to Simulate Land Use Changes
TLDR
This paper applies and compares three widely used data mining models: classification and regression tree (CART), multivariate adaptive regression spline (MARS) and random forest (RF), to simulate urban LUCs of Shirgah in Iran and verifies that having the highest accuracy in the testing run does not necessarily guarantee the highest accuracies in the validating run.
Comparing Quantity, Allocation and Configuration Accuracy of Multiple Land Change Models
TLDR
Compared the quantity, allocation, and configuration accuracy of four inductive pattern-based spatial allocation land change models (SLEUTH, GEOMOD, Land Change Modeler (LCM), and FUTURES), which suggest some models may better simulate quantity and allocation at the trade-off of configuration accuracy, and vice versa.
...
...

References

SHOWING 1-10 OF 76 REFERENCES
The meaning and use of the area under a receiver operating characteristic (ROC) curve.
A representation and interpretation of the area under a receiver operating characteristic (ROC) curve obtained by the "rating" method, or by mathematical predictions based on patient characteristics,
A receiver operating characteristic partial area index for highly sensitive diagnostic tests.
TLDR
A new ROC partial area index is developed, which measures clinical diagnostic performance more meaningfully in such situations, to summarize an ROC curve in only a high-sensitivity region.
The use of the area under the ROC curve in the evaluation of machine learning algorithms
A Simple Generalisation of the Area Under the ROC Curve for Multiple Class Classification Problems
TLDR
This work extends the definition of the area under the ROC curve to the case of more than two classes by averaging pairwise comparisons and proposes an alternative definition of proportion correct based on pairwise comparison of classes for a simple artificial case.
Use and Misuse of the Receiver Operating Characteristic Curve in Risk Prediction
The c statistic, or area under the receiver operating characteristic (ROC) curve, achieved popularity in diagnostic testing, in which the test characteristics of sensitivity and specificity are
Comparison of eight computer programs for receiver-operating characteristic analysis.
TLDR
Adequate ROC analysis and ROC plotting cannot be performed with a single program, and Analyse-It, AccuROC, and MedCalc can be recommended with certain limitations, but each program had different shortcomings.
A Suite of Tools for ROC Analysis of Spatial Models
TLDR
A suite of GIS tools designed to facilitate ROC curve analysis for GIS users by applying proper statistical tests and analysis procedures and interpret and discuss the ROC-related statistics resulting from various case studies.
The Relative Operating Characteristic in Psychology
TLDR
The ROC is an analytical technique that quite effectively isolates the effects of the observer's response bias, or decision criterion, in the study of discrimination behavior and enhances the understanding of the perceptual and cognitive phenomena that depend directly on these fundamental processes.
A review of methods for the assessment of prediction errors in conservation presence/absence models
TLDR
Thirteen recommendations are made to enable the objective selection of an error assessment technique for ecological presence/absence models and a new approach to estimating prediction error, which is based on the spatial characteristics of the errors, is proposed.
...
...