Evaluating Stochastic Rankings with Expected Exposure

@article{Diaz2020EvaluatingSR,
  title={Evaluating Stochastic Rankings with Expected Exposure},
  author={Fernando Diaz and Bhaskar Mitra and Michael D. Ekstrand and Asia J. Biega and Ben Carterette},
  journal={Proceedings of the 29th ACM International Conference on Information \& Knowledge Management},
  year={2020}
}
We introduce the concept of expected exposure as the average attention ranked items receive from users over repeated samples of the same query. Furthermore, we advocate for the adoption of the principle of equal expected exposure: given a fixed information need, no item should receive more or less expected exposure than any other item of the same relevance grade. We argue that this principle is desirable for many retrieval objectives and scenarios, including topical diversity and fair ranking… 

Figures and Tables from this paper

Fair Exposure of Documents in Information Retrieval: a Community Detection Approach
TLDR
The first steps toward a new framework for fair document exposure are presented, based on document linking and document community detection; communities are used to rank the documents to be retrieved according to an information need.
Counteracting Bias and Increasing Fairness in Search and Recommender Systems
TLDR
This tutorial will introduce the issues of biases in search and recommendation and show how to think about and create systems that are fairer, with increasing diversity and transparency.
Comparing Fair Ranking Metrics
TLDR
This work provides a direct comparative analysis identifying similarities and differences of fair ranking metrics selected for the work, and empirically compare them on the same experimental setup and data set.
Fairness and Discrimination in Information Access Systems
TLDR
This monograph presents a taxonomy of the various dimensions of fair information access and survey the literature to date on this new and rapidly-growing topic.
Joint Multisided Exposure Fairness for Recommendation
TLDR
This paper considers group attributes for both types of stakeholders to identify and mitigate fairness concerns that go beyond individual users and items towards more systemic biases in recommendation, as well as demonstrating how stochastic ranking policies can be optimized towards said fairness goals.
Probabilistic Permutation Graph Search: Black-Box Optimization for Fairness in Ranking
TLDR
A novel way of representing permutation distributions, based on the notion of permutation graphs, is presented, which improves over~\acPL for optimizing fairness metrics for queries with one session and is suitable for both deterministic and stochastic rankings.
A Versatile Framework for Evaluating Ranked Lists in terms of Group Fairness and Relevance
TLDR
A simple and versatile framework for evaluating ranked lists in terms of group fairness and relevance, where the groups can be either nominal or ordinal in nature, and can quantify intersectional group fairness based on multiple attribute sets is presented.
Introducing the Expohedron for Efficient Pareto-optimal Fairness-Utility Amortizations in Repeated Rankings
TLDR
This paper introduces a geometrical object, a polytope that is called expohedron, whose points represent all achievable exposures of items for a Position Based Model (PBM), and shows that it can be used to recover the whole Pareto frontier of the multi-objective fairness-utility optimization problem, using a simple geometric procedure with complexity.
Fairness of Exposure under Incomplete Exposure Estimation
  • Computer Science
  • 2022
TLDR
It is found that FELIX is able to significantly reduce the number of rankings with unknown exposure distribution without any drop in user utility of fairness, compared to previous fair ranking methods, both for full-length rankings and for top-k rankings.
Top-K Contextual Bandits with Equity of Exposure
TLDR
This work proposes a personalised exposure-aware arm selection algorithm that handles this relevance-fairness trade-off on a user-level, as recent work suggests that users’ openness to randomisation may vary greatly over the global populace.
...
...

References

SHOWING 1-10 OF 55 REFERENCES
Expected reciprocal rank for graded relevance
TLDR
This work presents a new editorial metric for graded relevance which overcomes this difficulty and implicitly discounts documents which are shown below very relevant documents and calls it Expected Reciprocal Rank (ERR).
Quantifying the Impact of User Attentionon Fair Group Representation in Ranked Lists
TLDR
This work introduces a novel metric for auditing group fairness in ranked lists, and shows that determining fairness of a ranked output necessitates knowledge (or a model) of the end-users of the particular service.
Evaluating diversified search results using per-intent graded relevance
TLDR
This work compares a wide range of traditional and diversified IR metrics after adding graded relevance assessments to the TREC 2009 Web track diversity task test collection, and shows that a family of metrics called D#-measures have several advantages over other metrics such as α-nDCG and Intent-Aware metrics.
Learning to Rank with Selection Bias in Personal Search
TLDR
It is empirically demonstrate that learning-to-rank that accounts for query-dependent selection bias yields significant improvements in search effectiveness through online experiments with one of the world's largest personal search engines.
Risky business: modeling and exploiting uncertainty in information retrieval
TLDR
A general framework for modeling uncertainty is presented and an asymmetric loss function with a single parameter that can model the level of risk the system is willing to accept is introduced, which can effectively adapt to users' different retrieval strategies.
Rank-biased precision for measurement of retrieval effectiveness
TLDR
A new effectiveness metric, rank-biased precision, is introduced that is derived from a simple model of user behavior, is robust if answer rankings are extended to greater depths, and allows accurate quantification of experimental uncertainty, even when only partial relevance judgments are available.
Ranking with Fairness Constraints
TLDR
This work studies the following variant of the traditional ranking problem when the objective satisfies properties that appear in common ranking metrics such as Discounted Cumulative Gain, Spearman's rho or Bradley-Terry.
Shuffling a Stacked Deck: The Case for Partially Randomized Ranking of Search Engine Results
TLDR
It is shown that a modest amount of randomness leads to improved search results, in the context of an economic objective function based on aggregate result quality amortized over time.
BPR: Bayesian Personalized Ranking from Implicit Feedback
TLDR
This paper presents a generic optimization criterion BPR-Opt for personalized ranking that is the maximum posterior estimator derived from a Bayesian analysis of the problem and provides a generic learning algorithm for optimizing models with respect to B PR-Opt.
A Stochastic Treatment of Learning to Rank Scoring Functions
TLDR
This work analytically studies the proposed sampling method and demonstrates when and why it leads to model robustness, and empirically shows that the application of the proposed method to a class of ranking loss functions leads to significant model quality improvements.
...
...