Corpus ID: 2931043

Sturgeon and the Cool Kids: Problems with Random Decoys for Top-N Recommender Evaluation

@inproceedings{Ekstrand2017SturgeonAT,
  title={Sturgeon and the Cool Kids: Problems with Random Decoys for Top-N Recommender Evaluation},
  author={Michael D. Ekstrand and Vaibhav Mahant},
  booktitle={FLAIRS Conference},
  year={2017}
}
Top-N evaluation of recommender systems, typically carried out using metrics from information retrieval or machine learning, has several challenges. Two of these challenges are popularity bias, where the evaluation intrinsically favors algorithms that recommend popular items, and misclassified decoys, where items for which no user relevance is known are actually relevant to the user, but the evaluation is unaware and penalizes the recommender for suggesting them. One strategy for mitigating the… Expand
4 Citations
Monte Carlo Estimates of Evaluation Metric Error and Bias
Hands on Data and Algorithmic Bias in Recommender Systems
Estimating Error and Bias in Offline Evaluation Results

References

SHOWING 1-10 OF 26 REFERENCES
Improving recommendation lists through topic diversification
Evaluating collaborative filtering recommender systems
A Survey of Accuracy Evaluation Metrics of Recommendation Tasks
Factorization meets the neighborhood: a multifaceted collaborative filtering model
Comparative recommender system evaluation: benchmarking recommendation frameworks
Rethinking the recommender research ecosystem: reproducibility, openness, and LensKit
...
1
2
3
...