Recommendation Systems with Distribution-Free Reliability Guarantees

  title={Recommendation Systems with Distribution-Free Reliability Guarantees},
  author={Anastasios Nikolas Angelopoulos and Karl Krauth and Stephen Bates and Yixin Wang and Michael I. Jordan},
When building recommendation systems, we seek to output a helpful set of items to the user. Under the hood, a ranking model predicts which of two candidate items is better, and we must distill these pairwise comparisons into the user-facing output. However, a learned ranking model is never perfect, so taking its predictions at face value gives no guarantee that the user-facing output is reliable. Building from a pre-trained ranking model, we show how to return a set of items that is rigorously… 
1 Citations

Figures from this paper

Conformal Risk Control

The algorithm generalizes split conformal prediction together with its coverage guarantee and is able to bound the false negative rate, graph distance, and token-level F1-score.



Conformal matrix factorization based recommender system

Recommendations and user agency: the reachability of collaboratively-filtered information

This work considers directly the information availability problem through the lens of user recourse, using ideas of reachability, and proposes a computationally efficient audit for top-N linear recommender models.

On the Calibration and Uncertainty of Neural Learning to Rank Models for Conversational Search

Two techniques are used to model the uncertainty of neural rankers leading to the proposed stochastic rankers, which output a predictive distribution of relevance as opposed to point estimates, and uncertainty estimation is beneficial for both risk-aware neural ranking, and for predicting unanswerable conversational contexts.

Confidence-Aware Matrix Factorization for Recommender Systems

This paper proposes a Confidence-aware Matrix Factorization (CMF) framework to simultaneously optimize the accuracy of rating prediction and measure the prediction confidence in the model, and introduces variance parameters for both users and items in the matrix factorization process.

Content-based recommendations with Poisson factorization

We develop collaborative topic Poisson factorization (CTPF), a generative model of articles and reader preferences. CTPF can be used to build recommender systems by learning from reader histories and

Risky business: modeling and exploiting uncertainty in information retrieval

A general framework for modeling uncertainty is presented and an asymmetric loss function with a single parameter that can model the level of risk the system is willing to accept is introduced, which can effectively adapt to users' different retrieval strategies.

Fairness of Exposure in Rankings

This work proposes a conceptual and computational framework that allows the formulation of fairness constraints on rankings in terms of exposure allocation, and develops efficient algorithms for finding rankings that maximize the utility for the user while provably satisfying a specifiable notion of fairness.

Exploring the filter bubble: the effect of using recommender systems on content diversity

This paper examines the longitudinal impacts of a collaborative filtering-based recommender system on users and contributes a novel metric to measure content diversity based on information encoded in user-generated tags, and presents a new set of methods to examine the temporal effect of recommender systems on the user experience.

Related Pins at Pinterest: The Evolution of a Real-World Recommender System

A longitudinal study of the evolution of the Pinterest recommender system and its components from prototypes to present state, showing how organic growth led to a complex system and how it managed this complexity.

Conformal recommender system