• Corpus ID: 5176749

Accurate is not always good: How Accuracy Metrics have hurt Recommender Systems

  title={Accurate is not always good: How Accuracy Metrics have hurt Recommender Systems},
  author={Sean M. McNee and John Riedl and Joseph A. Konstan},
Recommender systems have shown great potential to help users find interesting and relevant items from within a large information space. Most research up to this point has focused on improving the accuracy of recommender systems. We be lieve that not only has this narrow focus been misguided, but has even been detrimental to the field. The recommendations that are most accurate according to the standard metrics are sometimes not the recommendations that are most useful to users. In this paper… 
Evaluating the Relative Performance of Collaborative Filtering Recommender Systems
An evaluation framework based on a set of accuracy and beyond accuracy metrics, including a novel metric that captures the uniqueness of a recommendation list is presented, which finds that the matrix factorisation approach leads to more accurate and diverse recommendations, while being less biased toward popularity.
Beyond accuracy: evaluating recommender systems by coverage and serendipity
It is argued that the new ways of measuring coverage and serendipity reflect the quality impression perceived by the user in a better way than previous metrics thus leading to enhanced user satisfaction.
User-Centric Evaluation Framework for Multimedia Recommender Systems
A generic framework for evaluating the user experience using both subjective and objective measures of user experience is proposed, which will be tested and validated in the large-scale field trials of the FP7 MyMedia project.
Does Trust Matter for User Preferences? A Study on Epinions Ratings
This work investigates how useful it might be if explicit trust relationships are used to select the best neighbours (or predictors), for the provision of accurate recommendations and indicates that trust information can be helpful to provide a slight performance gain in a few cases.
Intelligent Techniques for Web Personalization & Recommender Systems
An improved general recommender system algorithm that autonomously selects the most accurate mediated or domain-specific advice giver for each user is contributed using data from two large recommender systems that potentially half of the users benefitted from the mediated advice givers while the other half required domain- specific weights.
Workshop on recommendation utility evaluation: beyond RMSE -- RUE 2012
The RUE 2012 workshop sought to identify and better understand the current gaps in recommender system evaluation methodologies, help lay directions for progress in addressing them, and contribute to the consolidation and convergence of experimental methods and practice.
Using Dynamically Promoted Experts for Music Recommendation
A recommender system that uses the concepts of Experts to find both novel and relevant recommendations that can provide a means to popularity bias while preserving the advantages of collaborative filtering is proposed.
User-perceived recommendation quality-factoring in the user interface
A general guideline to evaluate recommenders from perspectives other than accuracy is presented and a general experiment design is proposed to investigate the effects of quality factors on recommendations taking into account the system’s interface.
Algorithms in Recommender Systems
This presentation discussed methods and algorithms used in recommender systems, which traditionally is the most used approach, and talked about content-based and knowledge-basedRecommender systems and highlighted their merits.
Online Selection of Mediated and Domain-Specific Predictions for Improved Recommender Systems
An improved general recommender system algorithm that autonomously selects the most accurate mediated or domain-specific advice giver for each user is contributed.


Evaluating collaborative filtering recommender systems
The key decisions in evaluating collaborative filtering recommender systems are reviewed: the user tasks being evaluated, the types of analysis and datasets being used, the ways in which prediction quality is measured, the evaluation of prediction attributes other than quality, and the user-based evaluation of the system as a whole.
Getting to know you: learning new user preferences in recommender systems
Six techniques that collaborative filtering recommender systems can use to learn about new users are studied, showing that the choice of learning technique significantly affects the user experience, in both the user effort and the accuracy of the resulting predictions.
Improving recommendation lists through topic diversification
This work presents topic diversification, a novel method designed to balance and diversify personalized recommendation lists in order to reflect the user's complete spectrum of interests, and introduces the intra-list similarity metric to assess the topical diversity of recommendation lists.
Empirical Analysis of Predictive Algorithms for Collaborative Filtering
Several algorithms designed for collaborative filtering or recommender systems are described, including techniques based on correlation coefficients, vector-based similarity calculations, and statistical Bayesian methods, to compare the predictive accuracy of the various methods in a set of representative problem domains.
Learning Collaborative Information Filters
This work proposes a representation for collaborative filtering tasks that allows the application of virtually any machine learning algorithm, and identifies the shortcomings of current collaborative filtering techniques and proposes the use of learning algorithms paired with feature extraction techniques that specifically address the limitations of previous approaches.
Enhancing digital libraries with TechLens
This paper presents and experiments with hybrid recommender algorithms that combine collaborative filtering and content-based filtering to recommend research papers to users and shows that users value paper recommendations, that the hybrid algorithms can be successfully combined, and that these results can be applied to develop recommender systems for other types of digital libraries.
On the recommending of citations for research papers
This paper investigated six algorithms for selecting citations, evaluating them through offline experiments and an online experiment to gauge user opinion of the effectiveness of the algorithms and of the utility of such recommendations for common research tasks.
An algorithmic framework for performing collaborative filtering
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial
If TiVo Thinks You Are Gay, Here's How To Set It Straight --- Amazon.com Knows You, Too, Based on What You Buy; Why All the Cartoons? The Wall Street Journal
  • sect. A,
  • 2002
If TiVo Thinks You Are Gay, Here's How To Set It Straight ---Amazon.com Knows You, Too, Based on What You Buy; Why All the Cartoons?
  • The Wall Street Journal, sect. A
  • 2002