Estimation of Fair Ranking Metrics with Incomplete Judgments

  title={Estimation of Fair Ranking Metrics with Incomplete Judgments},
  author={{\"O}mer Kirnap and Fernando Diaz and Asia J. Biega and Michael D. Ekstrand and Ben Carterette and Emine Yilmaz},
  journal={Proceedings of the Web Conference 2021},
There is increasing attention to evaluating the fairness of search system ranking decisions. These metrics often consider the membership of items to particular groups, often identified using protected attributes such as gender or ethnicity. To date, these metrics typically assume the availability and completeness of protected attribute labels of items. However, the protected attributes of individuals are rarely present, limiting the application of fair ranking metrics in large scale systems. In… Expand

Figures and Tables from this paper


Measuring Fairness in Ranked Outputs
Equity of Attention: Amortizing Individual Fairness in Rankings
Estimating average precision with incomplete and imperfect judgments
Exploring author gender in book rating and recommendation
A Practical Sampling Strategy for Efficient Retrieval Evaluation
Evaluating Stochastic Rankings with Expected Exposure