Evaluating collaborative filtering recommender systems
@article{Herlocker2004EvaluatingCF, title={Evaluating collaborative filtering recommender systems}, author={Jonathan L. Herlocker and Joseph A. Konstan and Loren G. Terveen and John Riedl}, journal={ACM Trans. Inf. Syst.}, year={2004}, volume={22}, pages={5-53} }
Recommender systems have been evaluated in many, often incomparable, ways. [] Key Result Metrics within each equivalency class were strongly correlated, while metrics from different equivalency classes were uncorrelated.
5,826 Citations
Evaluating the Relative Performance of Collaborative Filtering Recommender Systems
- Computer ScienceJ. Univers. Comput. Sci.
- 2015
An evaluation framework based on a set of accuracy and beyond accuracy metrics, including a novel metric that captures the uniqueness of a recommendation list is presented, which finds that the matrix factorisation approach leads to more accurate and diverse recommendations, while being less biased toward popularity.
Unifying inconsistent evaluation metrics in recommender systems
- Computer Science
- 2014
A novel and extensible framework for evaluation of recommender systems using maximum bounds of possible measures in different datasets is proposed and the results of applying this framework on a set of different recommender algorithms are provided.
The Comparability of Recommender System Evaluations and Characteristics of Docear ’ s Users
- Computer Science
- 2014
This paper shows that reporting demographic and usage-based data is crucial in order to create meaningful evaluations on Docear's recommender system and sets previous evaluations into context and helps others to compare their results with the authors'.
A Study of Evaluation Metrics for Recommender Algorithms
- Computer Science
- 2008
This paper compares recommender algorithms using two datasets; the standard MovieLens set and an e-commerce dataset that has implicit ratings based on browsing behaviour to introduce a measure that aids in the comparison and show how to compare results with baseline predictions based on random recommendation selections.
Prediction Accuracy Comparison Of Similarity Measures In Memory Based Collaborative Filtering Recommender Systems
- Computer Science
- 2013
Various similarity measures are explored and their effect on predictive accuracy when applied in building neighbourhood based collaborative filtering recommender systems is analyzed.
Performance Evaluation of Recommender Systems
- Computer Science
- 2017
Three classical methods offline analytics, user study, and online experiment to evaluate the performance of recommender systems are discussed and summarized to provide the designers with guidance for the comprehensive evaluation and selection of recommended algorithms.
A framework for collaborative filtering recommender systems
- Computer ScienceExpert Syst. Appl.
- 2011
Information Retrieval and User-Centric Recommender System Evaluation
- Computer ScienceUMAP Workshops
- 2013
This project aims to develop novel means of recommender systems evaluation which encompasses qualities identified through traditional evaluation metrics and user-centric factors, e.g. diversity, serendipity, novelty, etc., as well as bringing further insights in the topic by analyzing and translating the problem of evaluation from an Information Retrieval perspective.
Correlating perception-oriented aspects in user-centric recommender system evaluation
- Computer ScienceIIiX
- 2012
This work neglects the quality of the recommender system and focuses on the similarity of aspects related to users' perception of recommender systems, showing the correlation of concepts such as usefulness, ratings, obviousness, and serendipity from the users' perspectives.
Choice of metrics used in collaborative filtering and their impact on recommender systems
- Computer Science2008 2nd IEEE International Conference on Digital Ecosystems and Technologies
- 2008
These two memory-based metrics, Pearson correlation and cosine, are analyzed together with the less common mean squared difference to discover their advantages and drawbacks in very important aspects such as the impact when introducing different values of k-neighborhoods.
References
SHOWING 1-10 OF 68 REFERENCES
Methods and metrics for cold-start recommendations
- Computer ScienceSIGIR '02
- 2002
A method for recommending items that combines content and collaborative data under a single probabilistic framework is developed, and it is demonstrated empirically that the various components of the testing strategy combine to obtain deeper understanding of the performance characteristics of recommender systems.
The role of transparency in recommender systems
- Computer ScienceCHI Extended Abstracts
- 2002
Preliminary results indicate that users like and feel more confident about recommendations that they perceive as transparent, and the role of transprency (user understanding of why a particular recommendation was made) in Recommender Systems is examined.
Empirical Analysis of Predictive Algorithms for Collaborative Filtering
- Computer ScienceUAI
- 1998
Several algorithms designed for collaborative filtering or recommender systems are described, including techniques based on correlation coefficients, vector-based similarity calculations, and statistical Bayesian methods, to compare the predictive accuracy of the various methods in a set of representative problem domains.
Application of Dimensionality Reduction in Recommender System - A Case Study
- Computer Science
- 2000
This paper presents two different experiments where one technology called Singular Value Decomposition (SVD) is explored to reduce the dimensionality of recommender system databases and suggests that SVD has the potential to meet many of the challenges ofRecommender systems, under certain conditions.
Meta-recommendation systems: user-controlled integration of diverse recommendations
- Computer ScienceCIKM '02
- 2002
This paper addresses recommender systems and introduces a new class of recommender system called meta-recommenders, which provide users with personalized control over the generation of a single recommendation list formed from a combination of rich data using multiple information sources and recommendation techniques.
Item-based collaborative filtering recommendation algorithms
- Computer ScienceWWW '01
- 2001
This paper analyzes item-based collaborative ltering techniques and suggests that item- based algorithms provide dramatically better performance than user-based algorithms, while at the same time providing better quality than the best available userbased algorithms.
Using filtering agents to improve prediction quality in the GroupLens research collaborative filtering system
- Computer ScienceCSCW '98
- 1998
The filterbot model allows collaborative filtering systems to address sparsity by tapping the strength of content filtering techniques and is experimentally validated by showing that even simple filterbots such as spell checking can increase the utility for users of sparsely populated collaborative filtering system.
Getting to know you: learning new user preferences in recommender systems
- Computer ScienceIUI '02
- 2002
Six techniques that collaborative filtering recommender systems can use to learn about new users are studied, showing that the choice of learning technique significantly affects the user experience, in both the user effort and the accuracy of the resulting predictions.
Beyond Algorithms: An HCI Perspective on Recommender Systems
- Computer Science
- 2001
From a user’s perspective, an effective recommender system inspires trust in the system; has system logic that is at least somewhat transparent; points users towards new, not-yet-experienced items; provides details about recommended items, including pictures and community ratings; and finally, provides ways to refine recommendations by including or excluding particular genres.
An Empirical Analysis of Design Choices in Neighborhood-Based Collaborative Filtering Algorithms
- Computer ScienceInformation Retrieval
- 2004
An analysis framework is applied that divides the neighborhood-based prediction approach into three components and then examines variants of the key parameters in each component, and identifies the three components identified are similarity computation, neighbor selection, and rating combination.