Corpus ID: 70349898

Limitations of Pinned AUC for Measuring Unintended Bias

@article{Borkan2019LimitationsOP,
  title={Limitations of Pinned AUC for Measuring Unintended Bias},
  author={Daniel Borkan and Lucas Dixon and John Li and Jeffrey Scott Sorensen and Nithum Thain and L. Vasserman},
  journal={ArXiv},
  year={2019},
  volume={abs/1903.02088}
}
  • Daniel Borkan, Lucas Dixon, +3 authors L. Vasserman
  • Published 2019
  • Mathematics, Computer Science
  • ArXiv
  • This report examines the Pinned AUC metric introduced and highlights some of its limitations. Pinned AUC provides a threshold-agnostic measure of unintended bias in a classification model, inspired by the ROC-AUC metric. However, as we highlight in this report, there are ways that the metric can obscure different kinds of unintended biases when the underlying class distributions on which bias is being measured are not carefully controlled. 

    Tables and Topics from this paper.

    Debiasing Personal Identities in Toxicity Classification
    Unintended Bias in Misogyny Detection
    3

    References

    Publications referenced by this paper.
    SHOWING 1-4 OF 4 REFERENCES
    Nuanced Metrics for Measuring Unintended Bias with Real Data for Text Classification
    23
    Measuring and Mitigating Unintended Bias in Text Classification
    93
    Unintended bias and names of frequently targeted groups, 2018
    • 2018