• Corpus ID: 235731509

Optimizing ROC Curves with a Sort-Based Surrogate Loss Function for Binary Classification and Changepoint Detection

  title={Optimizing ROC Curves with a Sort-Based Surrogate Loss Function for Binary Classification and Changepoint Detection},
  author={Jonathan A. Hillman and Toby Hocking},
Receiver Operating Characteristic (ROC) curves are plots of true positive rate versus false positive rate which are useful for evaluating binary classification models, but difficult to use for learning since the Area Under the Curve (AUC) is non-convex. ROC curves can also be used in other problems that have false positive and true positive rates such as changepoint detection. We show that in this more general context, the ROC curve can have loops, points with highly sub-optimal error rates… 

Figures from this paper


Optimizing Area Under Roc Curve with SVMs
A quadratic programming based algorithm for AUC maximization is proposed and it is shown that under certain conditions 2-norm soft margin Support Vector Ma- chines can also maximize AUC.
Measuring classifier performance: a coherent alternative to the area under the ROC curve
  • D. Hand
  • Computer Science
    Machine Learning
  • 2009
A simple valid alternative to the AUC is proposed, and the property of it being fundamentally incoherent in terms of misclassification costs is explored in detail.
A Structural SVM Based Approach for Optimizing Partial AUC
A structural SVM framework for directly optimizing the partial AUC between any two false positive rates and an efficient algorithm for solving this combinatorial optimization problem that has the same computational complexity as Joachims' algorithm for optimizing the usual AUC is developed.
Learning Decision Trees Using the Area Under the ROC Curve
This paper shows how a single decision tree can represent a set of classifiers by choosing different labellings of its leaves, or equivalently, an ordering on the leaves, and proposes a novel splitting criterion which chooses the split with the highest local AUC.
Optimising area under the ROC curve using gradient descent
This paper introduces RankOpt, a linear binary classifier which optimises the area under the ROC curve (the AUC). Unlike standard binary classifiers, RankOpt adopts the AUC statistic as its objective
AUC Optimization vs. Error Rate Minimization
The results show that the average AUC is monotonically increasing as a function of the classification accuracy, but that the standard deviation for uneven distributions and higher error rates is noticeable, so algorithms designed to minimize the error rate may not lead to the best possible AUC values.
Receiver operating characteristic curve generalization for non-monotone relationships
A receiver operating characteristic curve generalization, g, is proposed, useful in this context, where both the lower and the upper values of the marker are associated with higher probability of a positive result.
Optimizing Classifier Performance via an Approximation to the Wilcoxon-Mann-Whitney Statistic
This work proposes an objective function that is an approximation to the Wilcoxon-Mann-Whitney statistic, which is equivalent to the AUC, and applies it to real-world customer behavior prediction problems for a wireless service provider and a cable service provider, and achieves reliable improvements in the ROC curve.
Stochastic Online AUC Maximization
It is shown that AUC optimization can be equivalently formulated as a convex-concave saddle point problem and a stochastic online algorithm (SOLAM) is proposed which has time and space complexity of one datum.