Evaluating collaborative filtering recommender systems

@article{Herlocker2004EvaluatingCF,
  title={Evaluating collaborative filtering recommender systems},
  author={Jonathan L. Herlocker and J. Konstan and L. Terveen and J. Riedl},
  journal={ACM Trans. Inf. Syst.},
  year={2004},
  volume={22},
  pages={5-53}
}
Recommender systems have been evaluated in many, often incomparable, ways. [...] Key Result Metrics within each equivalency class were strongly correlated, while metrics from different equivalency classes were uncorrelated.Expand
View on ACM
grouplens.org
5,442 Citations
Highly Influential Citations
511
Background Citations
1,953
Methods Citations
1,188
Results Citations
42

Supplemental Presentations

Presentation Slides
Evaluating collaborative filtering recommender systems
Explore Further
Discover more papers related to the topics discussed in this paper

5,442 Citations

Citation Type

Citation Type

Evaluating the Relative Performance of Collaborative Filtering Recommender Systems
TLDR
An evaluation framework based on a set of accuracy and beyond accuracy metrics, including a novel metric that captures the uniqueness of a recommendation list is presented, which finds that the matrix factorisation approach leads to more accurate and diverse recommendations, while being less biased toward popularity. Expand
Unifying inconsistent evaluation metrics in recommender systems
TLDR
A novel and extensible framework for evaluation of recommender systems using maximum bounds of possible measures in different datasets is proposed and the results of applying this framework on a set of different recommender algorithms are provided. Expand
The Comparability of Recommender System Evaluations and Characteristics of Docear ’ s Users
Recommender systems are used in many fields, and many ideas have been proposed how to recommend useful items. In previous research, we showed that the effectiveness of recommendation approaches could… Expand
A Study of Evaluation Metrics for Recommender Algorithms
TLDR
This paper compares recommender algorithms using two datasets; the standard MovieLens set and an e-commerce dataset that has implicit ratings based on browsing behaviour to introduce a measure that aids in the comparison and show how to compare results with baseline predictions based on random recommendation selections. Expand
Prediction Accuracy Comparison Of Similarity Measures In Memory Based Collaborative Filtering Recommender Systems
TLDR
Various similarity measures are explored and their effect on predictive accuracy when applied in building neighbourhood based collaborative filtering recommender systems is analyzed. Expand
Performance Evaluation of Recommender Systems
TLDR
Three classical methods offline analytics, user study, and online experiment to evaluate the performance of recommender systems are discussed and summarized to provide the designers with guidance for the comprehensive evaluation and selection of recommended algorithms. Expand
A framework for collaborative filtering recommender systems
TLDR
A framework based on the above elements that enables the evaluation of the quality results of any collaborative filtering applied to the desired recommender systems, using four graphs: quality of the predictions, the recommendations, the novelty and the trust. Expand
Information Retrieval and User-Centric Recommender System Evaluation
TLDR
This project aims to develop novel means of recommender systems evaluation which encompasses qualities identified through traditional evaluation metrics and user-centric factors, e.g. diversity, serendipity, novelty, etc., as well as bringing further insights in the topic by analyzing and translating the problem of evaluation from an Information Retrieval perspective. Expand
Correlating perception-oriented aspects in user-centric recommender system evaluation
TLDR
This work neglects the quality of the recommender system and focuses on the similarity of aspects related to users' perception of recommender systems, showing the correlation of concepts such as usefulness, ratings, obviousness, and serendipity from the users' perspectives. Expand
Choice of metrics used in collaborative filtering and their impact on recommender systems
TLDR
These two memory-based metrics, Pearson correlation and cosine, are analyzed together with the less common mean squared difference to discover their advantages and drawbacks in very important aspects such as the impact when introducing different values of k-neighborhoods. Expand
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 85 REFERENCES
The role of transparency in recommender systems
TLDR
Preliminary results indicate that users like and feel more confident about recommendations that they perceive as transparent, and the role of transprency (user understanding of why a particular recommendation was made) in Recommender Systems is examined. Expand
Empirical Analysis of Predictive Algorithms for Collaborative Filtering
TLDR
Several algorithms designed for collaborative filtering or recommender systems are described, including techniques based on correlation coefficients, vector-based similarity calculations, and statistical Bayesian methods, to compare the predictive accuracy of the various methods in a set of representative problem domains. Expand
Application of Dimensionality Reduction in Recommender System - A Case Study
TLDR
This paper presents two different experiments where one technology called Singular Value Decomposition (SVD) is explored to reduce the dimensionality of recommender system databases and suggests that SVD has the potential to meet many of the challenges ofRecommender systems, under certain conditions. Expand
Meta-recommendation systems: user-controlled integration of diverse recommendations
TLDR
This paper addresses recommender systems and introduces a new class of recommender system called meta-recommenders, which provide users with personalized control over the generation of a single recommendation list formed from a combination of rich data using multiple information sources and recommendation techniques. Expand
Item-based collaborative filtering recommendation algorithms
TLDR
This paper analyzes item-based collaborative ltering techniques and suggests that item- based algorithms provide dramatically better performance than user-based algorithms, while at the same time providing better quality than the best available userbased algorithms. Expand
Using filtering agents to improve prediction quality in the GroupLens research collaborative filtering system
TLDR
The filterbot model allows collaborative filtering systems to address sparsity by tapping the strength of content filtering techniques and is experimentally validated by showing that even simple filterbots such as spell checking can increase the utility for users of sparsely populated collaborative filtering system. Expand
Getting to know you: learning new user preferences in recommender systems
TLDR
Six techniques that collaborative filtering recommender systems can use to learn about new users are studied, showing that the choice of learning technique significantly affects the user experience, in both the user effort and the accuracy of the resulting predictions. Expand
Beyond Algorithms: An HCI Perspective on Recommender Systems
TLDR
From a user’s perspective, an effective recommender system inspires trust in the system; has system logic that is at least somewhat transparent; points users towards new, not-yet-experienced items; provides details about recommended items, including pictures and community ratings; and finally, provides ways to refine recommendations by including or excluding particular genres. Expand
An Empirical Analysis of Design Choices in Neighborhood-Based Collaborative Filtering Algorithms
TLDR
An analysis framework is applied that divides the neighborhood-based prediction approach into three components and then examines variants of the key parameters in each component, and identifies the three components identified are similarity computation, neighbor selection, and rating combination. Expand
Evaluating expertise recommendations
TLDR
A systematic evaluation of the Expertise Recommender (ER), a system that recommends people who are likely to have expertise in a specific problem, suggests that the participants agree with the recommendations made by ER, and that ER significantly outperforms other expertise recommender systems when compared using similar metrics. Expand
...
1
2
3
4
5
...