Evaluating collaborative filtering recommender systems

@article{Herlocker2004EvaluatingCF,
  title={Evaluating collaborative filtering recommender systems},
  author={Jonathan L. Herlocker and Joseph A. Konstan and Loren G. Terveen and John Riedl},
  journal={ACM Trans. Inf. Syst.},
  year={2004},
  volume={22},
  pages={5-53}
}
Recommender systems have been evaluated in many, often incomparable, ways. [] Key Result Metrics within each equivalency class were strongly correlated, while metrics from different equivalency classes were uncorrelated.
View on ACM
grouplens.org
5,826 Citations
Highly Influential Citations
599
Background Citations
2,279
Methods Citations
1,419
Results Citations
48

Figures and Tables from this paper

5,826 Citations

Evaluating the Relative Performance of Collaborative Filtering Recommender Systems

An evaluation framework based on a set of accuracy and beyond accuracy metrics, including a novel metric that captures the uniqueness of a recommendation list is presented, which finds that the matrix factorisation approach leads to more accurate and diverse recommendations, while being less biased toward popularity.

Unifying inconsistent evaluation metrics in recommender systems

A novel and extensible framework for evaluation of recommender systems using maximum bounds of possible measures in different datasets is proposed and the results of applying this framework on a set of different recommender algorithms are provided.

The Comparability of Recommender System Evaluations and Characteristics of Docear ’ s Users

This paper shows that reporting demographic and usage-based data is crucial in order to create meaningful evaluations on Docear's recommender system and sets previous evaluations into context and helps others to compare their results with the authors'.

A Study of Evaluation Metrics for Recommender Algorithms

This paper compares recommender algorithms using two datasets; the standard MovieLens set and an e-commerce dataset that has implicit ratings based on browsing behaviour to introduce a measure that aids in the comparison and show how to compare results with baseline predictions based on random recommendation selections.

Prediction Accuracy Comparison Of Similarity Measures In Memory Based Collaborative Filtering Recommender Systems

Various similarity measures are explored and their effect on predictive accuracy when applied in building neighbourhood based collaborative filtering recommender systems is analyzed.

Performance Evaluation of Recommender Systems

Three classical methods offline analytics, user study, and online experiment to evaluate the performance of recommender systems are discussed and summarized to provide the designers with guidance for the comprehensive evaluation and selection of recommended algorithms.

A framework for collaborative filtering recommender systems

Information Retrieval and User-Centric Recommender System Evaluation

This project aims to develop novel means of recommender systems evaluation which encompasses qualities identified through traditional evaluation metrics and user-centric factors, e.g. diversity, serendipity, novelty, etc., as well as bringing further insights in the topic by analyzing and translating the problem of evaluation from an Information Retrieval perspective.

Correlating perception-oriented aspects in user-centric recommender system evaluation

This work neglects the quality of the recommender system and focuses on the similarity of aspects related to users' perception of recommender systems, showing the correlation of concepts such as usefulness, ratings, obviousness, and serendipity from the users' perspectives.

Choice of metrics used in collaborative filtering and their impact on recommender systems

These two memory-based metrics, Pearson correlation and cosine, are analyzed together with the less common mean squared difference to discover their advantages and drawbacks in very important aspects such as the impact when introducing different values of k-neighborhoods.
...

References

SHOWING 1-10 OF 68 REFERENCES

Methods and metrics for cold-start recommendations

A method for recommending items that combines content and collaborative data under a single probabilistic framework is developed, and it is demonstrated empirically that the various components of the testing strategy combine to obtain deeper understanding of the performance characteristics of recommender systems.

The role of transparency in recommender systems

Preliminary results indicate that users like and feel more confident about recommendations that they perceive as transparent, and the role of transprency (user understanding of why a particular recommendation was made) in Recommender Systems is examined.

Empirical Analysis of Predictive Algorithms for Collaborative Filtering

Several algorithms designed for collaborative filtering or recommender systems are described, including techniques based on correlation coefficients, vector-based similarity calculations, and statistical Bayesian methods, to compare the predictive accuracy of the various methods in a set of representative problem domains.

Application of Dimensionality Reduction in Recommender System - A Case Study

This paper presents two different experiments where one technology called Singular Value Decomposition (SVD) is explored to reduce the dimensionality of recommender system databases and suggests that SVD has the potential to meet many of the challenges ofRecommender systems, under certain conditions.

Meta-recommendation systems: user-controlled integration of diverse recommendations

This paper addresses recommender systems and introduces a new class of recommender system called meta-recommenders, which provide users with personalized control over the generation of a single recommendation list formed from a combination of rich data using multiple information sources and recommendation techniques.

Item-based collaborative filtering recommendation algorithms

This paper analyzes item-based collaborative ltering techniques and suggests that item- based algorithms provide dramatically better performance than user-based algorithms, while at the same time providing better quality than the best available userbased algorithms.

Using filtering agents to improve prediction quality in the GroupLens research collaborative filtering system

The filterbot model allows collaborative filtering systems to address sparsity by tapping the strength of content filtering techniques and is experimentally validated by showing that even simple filterbots such as spell checking can increase the utility for users of sparsely populated collaborative filtering system.

Getting to know you: learning new user preferences in recommender systems

Six techniques that collaborative filtering recommender systems can use to learn about new users are studied, showing that the choice of learning technique significantly affects the user experience, in both the user effort and the accuracy of the resulting predictions.

Beyond Algorithms: An HCI Perspective on Recommender Systems

From a user’s perspective, an effective recommender system inspires trust in the system; has system logic that is at least somewhat transparent; points users towards new, not-yet-experienced items; provides details about recommended items, including pictures and community ratings; and finally, provides ways to refine recommendations by including or excluding particular genres.

An Empirical Analysis of Design Choices in Neighborhood-Based Collaborative Filtering Algorithms

An analysis framework is applied that divides the neighborhood-based prediction approach into three components and then examines variants of the key parameters in each component, and identifies the three components identified are similarity computation, neighbor selection, and rating combination.
...