When recommenders fail: predicting recommender failure for algorithm selection and combination

@inproceedings{Ekstrand2012WhenRF,
  title={When recommenders fail: predicting recommender failure for algorithm selection and combination},
  author={Michael D. Ekstrand and John Riedl},
  booktitle={ACM Conference on Recommender Systems},
  year={2012}
}
Hybrid recommender systems --- systems using multiple algorithms together to improve recommendation quality --- have been well-known for many years and have shown good performance in recent demonstrations such as the NetFlix Prize. Modern hybridization techniques, such as feature-weighted linear stacking, take advantage of the hypothesis that the relative performance of recommenders varies by circumstance and attempt to optimize each item score to maximize the strengths of the component… 
View on ACM
files.grouplens.org
63 Citations
Highly Influential Citations
6
Background Citations
32
Methods Citations
16
Results Citations
1

Figures and Tables from this paper

63 Citations

On the Generalizability and Predictability of Recommender Systems

RecZilla is created, a meta-learning approach to recommender systems that uses a model to predict the best algorithm and hyperparameters for new, unseen datasets, and is able to substantially reduce the level of human involvement when faced with a new recommender system application.

Assessing and improving recommender systems to deal with user cold-start problem

This thesis proposes 4 approaches to deal with user cold-start problem using existing models available for analysis in the recommender systems and evaluated the proposed approaches in terms of prediction quality and ranking quality in real-world datasets under different recommendation domains.

Letting Users Choose Recommender Algorithms: An Experimental Study

This study gives users the ability to change the algorithm providing their movie recommendations and studied how they make use of this power, and examines log data from user interactions with this new feature to under-stand whether and how users switch among recommender algorithms, and select a final algorithm to use.

Rating-Based Collaborative Filtering: Algorithms and Evaluation

The concepts, algorithms, and means of evaluation that are at the core of collaborative filtering research and practice are reviewed, and two more recent directions in recommendation algorithms are presented: learning-to-rank and ensemble recommendation algorithms.

On hybrid modular recommendation systems for video streaming

The enabler, a hybrid recommendation system which employs various machine-learning algorithms for learning an efficient combination of several recommendation algorithms and selects the best blending for a given input, is proposed.

Recommender System: Personalizing User Experience or Scientifically Deceiving Users?

An overview of the recommender system is given, how various components of theRecommender system may be manipulated to allure innocent customers with false ratings are discussed, and the importance of engaging stakeholders to develop a robust recommendersystem is discussed.

MetaSelector: Meta-Learning for Recommendation with User-Level Adaptive Model Selection

This work proposes a meta-learning framework to facilitate user-level adaptive model selection in recommender systems and achieves improvements over single model baselines and sample-level model selector in terms of AUC and LogLoss.

Towards Recommender Engineering: tools and experiments for identifying recommender differences

The LensKit toolkit for conducting experiments on a wide variety of recommender algorithms and data sets under different experimental conditions, along with new developments in object-oriented software configuration to support this toolkit, and experiments on the configuration options of widely-used algorithms to provide guidance on tuning and configuring them are made.

Metalearning and Recommender Systems: A literature review and empirical study on the algorithm selection problem for Collaborative Filtering

MetaSelector: Meta-Learning for Recommendation with User-Level Adaptive Model Selection

This work proposes a meta-learning framework to facilitate user-level adaptive model selection in recommender systems and achieves improvements over single model baselines and sample-level model selector in terms of AUC and LogLoss.
...

References

SHOWING 1-10 OF 14 REFERENCES

A Survey of Accuracy Evaluation Metrics of Recommendation Tasks

This paper reviews the proper construction of offline experiments for deciding on the most appropriate algorithm, and discusses three important tasks of recommender systems, and classify a set of appropriate well known evaluation metrics for each task.

Hybrid Recommender Systems: Survey and Experiments

  • R. Burke
  • Computer Science
    User Modeling and User-Adapted Interaction
  • 2004
This paper surveys the landscape of actual and possible hybrid recommenders, and introduces a novel hybrid, EntreeC, a system that combines knowledge-based recommendation and collaborative filtering to recommend restaurants, and shows that semantic ratings obtained from the knowledge- based part of the system enhance the effectiveness of collaborative filtering.

Improving rating estimation in recommender systems using aggregation- and variance-based hierarchical models

It is experimentally shown that the optimal linear combination approach significantly dominates all other special cases, including the classical non-aggregated case and the previously studied aggregate methods, and therefore is the method of choice.

Item-based collaborative filtering recommendation algorithms

This paper analyzes item-based collaborative ltering techniques and suggests that item- based algorithms provide dramatically better performance than user-based algorithms, while at the same time providing better quality than the best available userbased algorithms.

Making recommendations better: an analytic model for human-recommender interaction

It is argued that recommenders need a deeper understanding of users and their information seeking tasks and recommender algorithms using a common language and an analytic process model.

An Empirical Analysis of Design Choices in Neighborhood-Based Collaborative Filtering Algorithms

An analysis framework is applied that divides the neighborhood-based prediction approach into three components and then examines variants of the key parameters in each component, and identifies the three components identified are similarity computation, neighbor selection, and rating combination.

Rethinking the recommender research ecosystem: reproducibility, openness, and LensKit

The utility of LensKit is demonstrated by replicating and extending a set of prior comparative studies of recommender algorithms, and a question recently raised by a leader in the recommender systems community on problems with error-based prediction evaluation is investigated.

Automatically building research reading lists

This work explores several methods for augmenting existing collaborative and content-based filtering algorithms with measures of the influence of a paper within the web of citations, including a novel method for using importance scores to influence collaborative filtering.

Feature-Weighted Linear Stacking

A linear technique, Feature-Weighted Linear Stacking (FWLS), that incorporates meta-features for improved accuracy while retaining the well-known virtues of linear regression regarding speed, stability, and interpretability is presented.

GroupLens: an open architecture for collaborative filtering of netnews

GroupLens is a system for collaborative filtering of netnews, to help people find articles they will like in the huge stream of available articles, and protect their privacy by entering ratings under pseudonyms, without reducing the effectiveness of the score prediction.