Should I Follow the Crowd?: A Probabilistic Analysis of the Effectiveness of Popularity in Recommender Systems

@article{Caamares2018ShouldIF,
  title={Should I Follow the Crowd?: A Probabilistic Analysis of the Effectiveness of Popularity in Recommender Systems},
  author={Roc{\'i}o Ca{\~n}amares and Pablo Castells},
  journal={The 41st International ACM SIGIR Conference on Research \& Development in Information Retrieval},
  year={2018}
}
  • Rocío Cañamares, P. Castells
  • Published 27 June 2018
  • Computer Science
  • The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval
The use of IR methodology in the evaluation of recommender systems has become common practice in recent years. IR metrics have been found however to be strongly biased towards rewarding algorithms that recommend popular items "the same bias that state of the art recommendation algorithms display. Recent research has confirmed and measured such biases, and proposed methods to avoid them. The fundamental question remains open though whether popularity is really a bias we should avoid or not… 

Figures and Tables from this paper

Popularity Bias in False-positive Metrics for Recommender Systems Evaluation
TLDR
This analysis is the first to show that false-positive metrics tend to penalise popular items, the opposite behavior of true- positive metrics—causing a disagreement trend between both types of metrics in the presence of popularity biases.
Bias and Debias in Recommender System: A Survey and Future Directions
TLDR
This paper provides a taxonomy to position and organize the existing work on recommendation debiasing, and identifies some open challenges and envision some future directions on this important yet less investigated topic.
Causal Intervention for Leveraging Popularity Bias in Recommendation
TLDR
A new training and inference paradigm for recommendation named Popularity-bias Deconfounding and Adjusting (PDA) is proposed, which removes the confounding popularity bias in model training and adjusts the recommendation score with desired popularity bias via causal intervention.
Assessing ranking metrics in top-N recommendation
TLDR
A principled analysis of the robustness and the discriminative power of different ranking metrics for the offline evaluation of recommender systems is undertaken, drawing from previous studies in the information retrieval field.
Debiased offline evaluation of recommender systems: a weighted-sampling approach
TLDR
This paper empirically validate for the first time the effectiveness of SKEW and shows the approach to be a better estimator of the performance one would obtain on (unbiased) MAR test data.
Agreement and Disagreement between True and False-Positive Metrics in Recommender Systems Evaluation
TLDR
An explanation is found for the discrepancy be-tween the metrics in the effect of popularity biases, which impact false and true- positive metrics in very different ways: instead of rewarding the recommendation of popular items, as with true-positive, false-positive metrics penalize the popular.
Bias Issues and Solutions in Recommender System: Tutorial on the RecSys 2021
TLDR
This tutorial will introduce six types of biases in recommender system, along with their definitions and characteristics; review existing debiasing solutions,along with their strengths and weaknesses; and identify some open challenges and future directions.
A sampling approach to Debiasing the offline evaluation of recommender systems
TLDR
This paper proposes and formulates a novel sampling approach, which it is suggested is a better estimator of the performance that one would obtain on (unbiased) MAR test data, and compares its methods to SKEW and to two baselines which perform a random intervention on MNAR data.
Mitigating Confounding Bias in Recommendation via Information Bottleneck
TLDR
This paper describes the generation process of the biased and unbiased feedback in recommender systems via two respective causal diagrams, and proposes a novel method called debiased information bottleneck (DIB), which constrains the model to learn a biased embedding vector with independent biases in the training phase and uses only the unbiased component in the test phase to deliver more accurate recommendations.
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 41 REFERENCES
What recommenders recommend: an analysis of recommendation biases and possible countermeasures
TLDR
It is shown that popular recommendation techniques—despite often being similar when compared with the help of accuracy measures—can be quite different with respect to which items they recommend.
Statistical biases in Information Retrieval metrics for recommender systems
TLDR
This paper lays out an experimental configuration framework upon which to identify and analyse specific statistical biases arising in the adaptation of Information Retrieval metrics to recommendation tasks, namely sparsity and popularity biases.
On Unexpectedness in Recommender Systems
TLDR
A method to improve user satisfaction by generating unexpected recommendations based on the utility theory of economics is proposed, which outperforms baseline methods in terms of unexpectedness and other important metrics, such as coverage, aggregate diversity and dispersion, while avoiding any accuracy loss.
Exploring Social Network Effects on Popularity Biases in Recommender Systems
TLDR
Whether popularity should always be ex- pected to be an effective approach for recommendation, and what causes, factors and conditions determine and explain such effec- tiveness are explored.
Collaborative Filtering for Implicit Feedback Datasets
TLDR
This work identifies unique properties of implicit feedback datasets and proposes treating the data as indication of positive and negative preference associated with vastly varying confidence levels, which leads to a factor model which is especially tailored for implicit feedback recommenders.
Performance of recommender algorithms on top-n recommendation tasks
TLDR
An extensive evaluation of several state-of-the art recommender algorithms suggests that algorithms optimized for minimizing RMSE do not necessarily perform as expected in terms of top-N recommendation task, and new variants of two collaborative filtering algorithms are offered.
Item popularity and recommendation accuracy
TLDR
A new accuracy measure is defined that has the desirable property of providing nearly unbiased estimates concerning recommendation accuracy and also motivates a refinement for training collaborative-filtering approaches.
A Probabilistic Reformulation of Memory-Based Collaborative Filtering: Implications on Popularity Biases
TLDR
A probabilistic formulation giving rise to a formal version of heuristic k nearest-neighbor (kNN) collaborative filtering provides a principled explanation why kNN is an effective recommendation strategy, and identifies a key condition for this to be the case.
Evaluating Recommendation Systems
TLDR
This paper discusses how to compare recommenders based on a set of properties that are relevant for the application, and focuses on comparative studies, where a few algorithms are compared using some evaluation metric, rather than absolute benchmarking of algorithms.
Collaborative Filtering and the Missing at Random Assumption
TLDR
Experimental results are presented showing that incorporating an explicit model of the missing data mechanism can lead to significant improvements in prediction performance on the random sample of ratings.
...
1
2
3
4
5
...