Corpus ID: 2931043

Sturgeon and the Cool Kids: Problems with Random Decoys for Top-N Recommender Evaluation

@inproceedings{Ekstrand2017SturgeonAT,
  title={Sturgeon and the Cool Kids: Problems with Random Decoys for Top-N Recommender Evaluation},
  author={Michael D. Ekstrand and Vaibhav Mahant},
  booktitle={FLAIRS Conference},
  year={2017}
}
Top-N evaluation of recommender systems, typically carried out using metrics from information retrieval or machine learning, has several challenges. Two of these challenges are popularity bias, where the evaluation intrinsically favors algorithms that recommend popular items, and misclassified decoys, where items for which no user relevance is known are actually relevant to the user, but the evaluation is unaware and penalizes the recommender for suggesting them. One strategy for mitigating the… Expand
View Paper
md.ekstrandom.net
4 Citations
Background Citations
4

4 Citations

Citation Type

Citation Type

Monte Carlo Estimates of Evaluation Metric Error and Bias
Hands on Data and Algorithmic Bias in Recommender Systems
Estimating Error and Bias in Offline Evaluation Results
Transparent, Scrutable and Explainable User Models for Personalized Recommendation

References

SHOWING 1-10 OF 26 REFERENCES
Performance of recommender algorithms on top-n recommendation tasks
Precision-oriented evaluation of recommender systems: an algorithmic comparison
Performance prediction and evaluation in recommender systems: An information retrieval perspective
Improving recommendation lists through topic diversification
Evaluating collaborative filtering recommender systems
A Survey of Accuracy Evaluation Metrics of Recommendation Tasks
Factorization meets the neighborhood: a multifaceted collaborative filtering model
Comparative recommender system evaluation: benchmarking recommendation frameworks
Being accurate is not enough: how accuracy metrics have hurt recommender systems
Rethinking the recommender research ecosystem: reproducibility, openness, and LensKit
...
1
2
3
...