Comparing Fair Ranking Metrics
@article{Raj2020ComparingFR, title={Comparing Fair Ranking Metrics}, author={Amifa Raj and Connor Wood and Ananda Montoly and Michael D. Ekstrand}, journal={ArXiv}, year={2020}, volume={abs/2009.01311} }
Ranking is a fundamental aspect of recommender systems. However, ranked outputs can be susceptible to various biases; some of these may cause disadvantages to members of protected groups. Several metrics have been proposed to quantify the (un)fairness of rankings, but there has not been to date any direct comparison of these metrics. This complicates deciding what fairness metrics are applicable for specific scenarios, and assessing the extent to which metrics agree or disagree. In this paper…
10 Citations
FAIR: Fairness-Aware Information Retrieval Evaluation
- Computer ScienceJournal of the Association for Information Science and Technology
- 2022
By unifying standard IR metrics and fairness measures into an integrated metric, this metric offers a new perspective for evaluating fairness-aware ranking results and developed an effective ranking algorithm that jointly optimized user utility and fairness.
A Versatile Framework for Evaluating Ranked Lists in terms of Group Fairness and Relevance
- Computer ScienceArXiv
- 2022
A simple and versatile framework for evaluating ranked lists in terms of group fairness and relevance, where the groups can be either nominal or ordinal in nature, and can quantify intersectional group fairness based on multiple attribute sets is presented.
Overview of the TREC 2019 Fair Ranking Track
- Computer ScienceArXiv
- 2020
An overview of the TREC Fair Ranking track is presented, including the task definition, descriptions of the data and the annotation process, as well as a comparison of the performance of submitted systems.
When Fair Ranking Meets Uncertain Inference
- Computer ScienceSIGIR
- 2021
It is shown how demographic inferences drawn from real systems can lead to unfair rankings, and developers should not use inferred demographic data as input to fair ranking algorithms, unless the inferences are extremely accurate.
Toward Fair Recommendation in Two-sided Platforms
- Business, Computer ScienceACM Transactions on the Web
- 2022
This work proposes to provide fairness guarantees for both sides of a fair personalized recommendation problem to a constrained version of the problem of fairly allocating indivisible goods, and presents a modification of FairRec that improves the recommendation performance for the customers, while maintaining the same fairness guarantees.
Fairness and Discrimination in Information Access Systems
- Computer ScienceArXiv
- 2021
This monograph presents a taxonomy of the various dimensions of fair information access and survey the literature to date on this new and rapidly-growing topic.
Probabilistic Permutation Graph Search: Black-Box Optimization for Fairness in Ranking
- Computer ScienceArXiv
- 2022
In this paper, a novel way of representing permutation distributions, based on the notion of permutation graphs, is presented, called probabilistic permutation graph (PPG), which improves over PL for optimizing fairness metrics for queries with one session.
Exploring author gender in book rating and recommendation
- Computer ScienceUser Model. User Adapt. Interact.
- 2021
It is found that common collaborative filtering algorithms tend to propagate at least some of each user’s tendency to rate or read male or female authors into their resulting recommendations, although they differ in both the strength of this propagation and the variance in the gender balance of the recommendation lists they produce.
Addressing Bias and Fairness in Search Systems
- Computer ScienceSIGIR
- 2021
This tutorial will introduce the issues of biases in data, in algorithms, and overall in search processes and show how to think about and create systems that are fairer, with increasing diversity and transparency.
Pink for Princesses, Blue for Superheroes: The Need to Examine Gender Stereotypes in Kid's Products in Search and Recommendations
- PsychologyArXiv
- 2021
The need to investigate if and how gender stereotypes manifest in search and recommender systems is argued for and an agenda to support future research addressing the phenomena is outlined.
References
SHOWING 1-10 OF 31 REFERENCES
Fairness in Recommendation Ranking through Pairwise Comparisons
- Computer ScienceKDD
- 2019
This paper offers a set of novel metrics for evaluating algorithmic fairness concerns in recommender systems and shows how measuring fairness based on pairwise comparisons from randomized experiments provides a tractable means to reason about fairness in rankings fromRecommender systems.
Estimation of Fair Ranking Metrics with Incomplete Judgments
- Computer ScienceWWW
- 2021
This work proposes a robust and unbiased estimator which can operate even with very limited number of labeled items and provides a robust, reliable alternative to exhaustive or random data annotation.
Measuring Group Advantage: A Comparative Study of Fair Ranking Metrics
- Computer ScienceAIES
- 2021
It is proved that under reasonable assumptions, popular metrics in the literature exhibit the same behavior and that optimizing for one optimizes for all, and a practical statistical test is designed to identify whether observed data is likely to exhibit predictable group bias.
Measuring Fairness in Ranked Outputs
- Computer Science, EconomicsSSDBM
- 2017
A data generation procedure is developed that allows for systematically control the degree of unfairness in the output, and the proposed fairness measures for ranked outputs are applied to several real datasets, and results show potential for improving fairness of ranked outputs while maintaining accuracy.
Equity of Attention: Amortizing Individual Fairness in Rankings
- Computer ScienceSIGIR
- 2018
The challenge of achieving amortized individual fairness subject to constraints on ranking quality as an online optimization problem is formulated and solved as an integer linear program and it is demonstrated that the method can improve individual fairness while retaining high ranking quality.
FA*IR: A Fair Top-k Ranking Algorithm
- Computer ScienceCIKM
- 2017
This work defines and solves the Fair Top-k Ranking problem, and presents an efficient algorithm, which is the first algorithm grounded in statistical tests that can mitigate biases in the representation of an under-represented group along a ranked list.
Overview of the TREC 2019 Fair Ranking Track
- Computer ScienceArXiv
- 2020
An overview of the TREC Fair Ranking track is presented, including the task definition, descriptions of the data and the annotation process, as well as a comparison of the performance of submitted systems.
Fairness of Exposure in Rankings
- Computer ScienceKDD
- 2018
This work proposes a conceptual and computational framework that allows the formulation of fairness constraints on rankings in terms of exposure allocation, and develops efficient algorithms for finding rankings that maximize the utility for the user while provably satisfying a specifiable notion of fairness.
Quantifying the Impact of User Attentionon Fair Group Representation in Ranked Lists
- Computer ScienceWWW
- 2019
This work introduces a novel metric for auditing group fairness in ranked lists, and shows that determining fairness of a ranked output necessitates knowledge (or a model) of the end-users of the particular service.
When Fair Ranking Meets Uncertain Inference
- Computer ScienceSIGIR
- 2021
It is shown how demographic inferences drawn from real systems can lead to unfair rankings, and developers should not use inferred demographic data as input to fair ranking algorithms, unless the inferences are extremely accurate.