• Corpus ID: 14046969

Study of a Bias in the Offline Evaluation of a Recommendation Algorithm

  title={Study of a Bias in the Offline Evaluation of a Recommendation Algorithm},
  author={Arnaud De Myttenaere and Boris Golden and B{\'e}n{\'e}dicte Le Grand and Fabrice Rossi},
Recommendation systems have been integrated into the majority of large online systems to filter and rank information according to user profiles. It thus influences the way users interact with the system and, as a consequence, bias the evaluation of the performance of a recommendation algorithm computed using historical data (via offline evaluation). This paper describes this bias and discuss the relevance of a weighted offline evaluation to reduce this bias for different classes of… 

Figures from this paper

An interest propagation based movie recommendation method for social tagging system
  • Haibo Liu, Shi Feng, Ge Yu
  • Computer Science
    2017 International Conference on Machine Learning and Cybernetics (ICMLC)
  • 2017
A hybrid method that combines the collaborative filtering with graph-based interest propagation is proposed for movie recommendation that can achieve better performance than several baselines for the movie recommendation problem.
Study of enterprise management training based on cluster computing
A sensitive-personal rank algorithm is proposed and the sensitivity of enterprise interest is combined to improve the correlation calculation method between fresh graduates and previous graduates, and the employment of similar former graduates is recommended to fresh graduates, providing reference and guidance for their employment.
Balanced News Using Constrained Bandit-based Personalization
We present a prototype for a news search engine that presents balanced viewpoints across liberal and conservative articles with the goal of depolarizing content and allowing users to escape their
Design of Intelligent English Writing Self-evaluation Auxiliary System
The random sampling based Bayesian classification and combinational algorithm is effective and feasible as an automatic scoring algorithm of writing self-evaluation auxiliary system and introduced the real-time multi-writing teaching mode.


Reducing Offline Evaluation Bias in Recommendation Systems
This paper analyses this evaluation bias and proposes a simple item weighting solution that reduces its impact and is evaluated on real world data extracted from Viadeo professional social network.
Evaluating collaborative filtering recommender systems
The key decisions in evaluating collaborative filtering recommender systems are reviewed: the user tasks being evaluated, the types of analysis and datasets being used, the ways in which prediction quality is measured, the evaluation of prediction attributes other than quality, and the user-based evaluation of the system as a whole.
Toward the next generation of recommender systems: a survey of the state-of-the-art and possible extensions
This paper presents an overview of the field of recommender systems and describes the current generation of recommendation methods that are usually classified into the following three main
User-centric evaluation of a K-furthest neighbor collaborative filtering recommender algorithm
This paper presents an inverted neighborhood model, k-Furthest Neighbors, to identify less ordinary neighborhoods for the purpose of creating more diverse recommendations and shows that even though the proposed furthest neighbor model is outperformed in the traditional evaluation setting, the perceived usefulness of the algorithm shows no significant difference in the results of the user study.
Evaluating Recommendation Systems
This paper discusses how to compare recommenders based on a set of properties that are relevant for the application, and focuses on comparative studies, where a few algorithms are compared using some evaluation metric, rather than absolute benchmarking of algorithms.
Being accurate is not enough: how accuracy metrics have hurt recommender systems
This paper proposes informal arguments that the recommender community should move beyond the conventional accuracy metrics and their associated experimental methodologies, and proposes new user-centric directions for evaluating recommender systems.
Unbiased offline evaluation of contextual-bandit-based news article recommendation algorithms
This paper introduces a replay methodology for contextual bandit algorithm evaluation that is completely data-driven and very easy to adapt to different applications and can provide provably unbiased evaluations.
Recommender Systems Handbook
This handbook illustrates how recommender systems can support the user in decision-making, planning and purchasing processes, and works for well known corporations such as Amazon, Google, Microsoft and AT&T.
A literature review and classification of recommender systems research