Evaluating Stochastic Rankings with Expected Exposure

  title={Evaluating Stochastic Rankings with Expected Exposure},
  author={Fernando Diaz and Bhaskar Mitra and Michael D. Ekstrand and Asia J. Biega and Ben Carterette},
  journal={Proceedings of the 29th ACM International Conference on Information \& Knowledge Management},
We introduce the concept of expected exposure as the average attention ranked items receive from users over repeated samples of the same query. Furthermore, we advocate for the adoption of the principle of equal expected exposure: given a fixed information need, no item should receive more or less expected exposure than any other item of the same relevance grade. We argue that this principle is desirable for many retrieval objectives and scenarios, including topical diversity and fair ranking… 

Figures and Tables from this paper

Fair Exposure of Documents in Information Retrieval: a Community Detection Approach

The first steps toward a new framework for fair document exposure are presented, based on document linking and document community detection; communities are used to rank the documents to be retrieved according to an information need.

Counteracting Bias and Increasing Fairness in Search and Recommender Systems

This tutorial will introduce the issues of biases in search and recommendation and show how to think about and create systems that are fairer, with increasing diversity and transparency.

Comparing Fair Ranking Metrics

This work provides a direct comparative analysis identifying similarities and differences of fair ranking metrics selected for the work, and empirically compare them on the same experimental setup and data set.

Fairness and Discrimination in Information Access Systems

This monograph presents a taxonomy of the various dimensions of fair information access and survey the literature to date on this new and rapidly-growing topic.

Random Isn't Always Fair: Candidate Set Imbalance and Exposure Inequality in Recommender Systems

It is shown that complete randomization at the second step can result in a higher degree of inequality relative to deterministic ordering of items by estimated relevance scores, and a simple post-processing algorithm is proposed in pursuit of reducing exposure inequality.

Fairness of Exposure in Light of Incomplete Exposure Estimation

This work defines a method called \method for finding stochastic policies that avoid showing rankings with unknown exposure distribution to the user without having to compromise user utility or item fairness and finds that this method can significantly reduce the number of rankings withunknown exposure distribution without a drop in users utility or fairness compared to existing fair ranking methods.

Joint Multisided Exposure Fairness for Recommendation

This paper considers group attributes for both types of stakeholders to identify and mitigate fairness concerns that go beyond individual users and items towards more systemic biases in recommendation, as well as demonstrating how stochastic ranking policies can be optimized towards said fairness goals.

Probabilistic Permutation Graph Search: Black-Box Optimization for Fairness in Ranking

A novel way of representing permutation distributions, based on the notion of permutation graphs, is presented, which improves over~\acPL for optimizing fairness metrics for queries with one session and is suitable for both deterministic and stochastic rankings.

A Versatile Framework for Evaluating Ranked Lists in terms of Group Fairness and Relevance

A simple and versatile framework for evaluating ranked lists in terms of group fairness and relevance, where the groups can be either nominal or ordinal in nature, and can quantify intersectional group fairness based on multiple attribute sets is presented.

Introducing the Expohedron for Efficient Pareto-optimal Fairness-Utility Amortizations in Repeated Rankings

This paper introduces a geometrical object, a polytope that is called expohedron, whose points represent all achievable exposures of items for a Position Based Model (PBM), and shows that it can be used to recover the whole Pareto frontier of the multi-objective fairness-utility optimization problem, using a simple geometric procedure with complexity.



Expected reciprocal rank for graded relevance

This work presents a new editorial metric for graded relevance which overcomes this difficulty and implicitly discounts documents which are shown below very relevant documents and calls it Expected Reciprocal Rank (ERR).

Quantifying the Impact of User Attentionon Fair Group Representation in Ranked Lists

This work introduces a novel metric for auditing group fairness in ranked lists, and shows that determining fairness of a ranked output necessitates knowledge (or a model) of the end-users of the particular service.

Evaluating diversified search results using per-intent graded relevance

This work compares a wide range of traditional and diversified IR metrics after adding graded relevance assessments to the TREC 2009 Web track diversity task test collection, and shows that a family of metrics called D#-measures have several advantages over other metrics such as α-nDCG and Intent-Aware metrics.

Learning to Rank with Selection Bias in Personal Search

It is empirically demonstrate that learning-to-rank that accounts for query-dependent selection bias yields significant improvements in search effectiveness through online experiments with one of the world's largest personal search engines.

Risky business: modeling and exploiting uncertainty in information retrieval

A general framework for modeling uncertainty is presented and an asymmetric loss function with a single parameter that can model the level of risk the system is willing to accept is introduced, which can effectively adapt to users' different retrieval strategies.

Rank-biased precision for measurement of retrieval effectiveness

A new effectiveness metric, rank-biased precision, is introduced that is derived from a simple model of user behavior, is robust if answer rankings are extended to greater depths, and allows accurate quantification of experimental uncertainty, even when only partial relevance judgments are available.

Ranking with Fairness Constraints

This work studies the following variant of the traditional ranking problem when the objective satisfies properties that appear in common ranking metrics such as Discounted Cumulative Gain, Spearman's rho or Bradley-Terry.

Shuffling a Stacked Deck: The Case for Partially Randomized Ranking of Search Engine Results

It is shown that a modest amount of randomness leads to improved search results, in the context of an economic objective function based on aggregate result quality amortized over time.

BPR: Bayesian Personalized Ranking from Implicit Feedback

This paper presents a generic optimization criterion BPR-Opt for personalized ranking that is the maximum posterior estimator derived from a Bayesian analysis of the problem and provides a generic learning algorithm for optimizing models with respect to B PR-Opt.

A Stochastic Treatment of Learning to Rank Scoring Functions

This work analytically studies the proposed sampling method and demonstrates when and why it leads to model robustness, and empirically shows that the application of the proposed method to a class of ranking loss functions leads to significant model quality improvements.