RecBench: Benchmarks for Evaluating Performance of Recommender System Architectures

  title={RecBench: Benchmarks for Evaluating Performance of Recommender System Architectures},
  author={Justin J. Levandoski and Michael D. Ekstrand and Michael Ludwig and Ahmed Eldawy and Mohamed F. Mokbel and John Riedl},
  journal={Proc. VLDB Endow.},
Traditionally, recommender systems have been “hand-built”, implemented as custom applications hard-wired to a particular recommendation task. Recently, the database community has begun exploring alternative DBMS-based recommender system architectures, whereby a database both stores the recommender system data (e.g., ratings data and the derived recommender models) and generates recommendations using SQL queries. In this paper, we present a comprehensive experimental comparison of both… 

Figures from this paper

BARS: Towards Open Benchmarking for Recommender Systems
This initiative project presents an initiative project aimed for open benchmarking for recommender systems, which sets up a standardized benchmarking pipeline for reproducible research, which integrates all the details about datasets, source code, hyper-parameter settings, running logs, and evaluation results.
Rethinking the recommender research ecosystem: reproducibility, openness, and LensKit
The utility of LensKit is demonstrated by replicating and extending a set of prior comparative studies of recommender algorithms, and a question recently raised by a leader in the recommender systems community on problems with error-based prediction evaluation is investigated.
An Overview of the CareDB Context and Preference-Aware Database System
The architecture of the CareDB system is described and the details for three of its novel query processing characteristics are described: (1) a generic and extensible preferenceaware query processing engine, (2) a framework to gracefully handle contextual attributes that are expensive to retrieve, and (3) aframework to efficiently process queries over uncertain contextual data.
Towards Scalable Personalization
This dissertation strengthens the notion of differential privacy in the context of recommenders by introducing distance-based differential privacy (D2P) which prevents curious users from even guessing any category (e.g., genre) in which a user might be interested in.
Capturing the Moment: Lightweight Similarity Computations
The main idea behind I-SIM is to disintegrate the similarity metric into mutually independent time-aware factors which can be updated incrementally, which enables lightweight similarity computations in an incremental and temporal manner.
Estimating importance of implicit factors in e-commerce recommender systems
The importance of different types of implicit user feedback for creating useful recommendations on an e-commerce website and some combinations of implicit factors and a test are discussed to see if they improve recommendation performance in comparison with the single factor ones.
Fast Online 'Next Best Offers' using Deep Learning
The design of iPrescribe is presented, a scalable low-latency architecture for recommending 'next-best-offers' in an online setting and its performance for implementations using different real-time streaming technology stacks is compared.
Un modelo híbrido de recomendación de etiquetas para sistemas de anotación social
This document proposes a different hybrid approach that simply solves the problem of recommendations based solely on the content of the resource, merging the list of recommendations with the most popular tags in the user’s tag history, thus allowing them to reuse terms assigned to others resources.
Performance assessment of an architecture with adaptative interfaces for people with special needs
An interoperable architecture is assessed, which enables interaction between people with some kind of special need and their environment and also tries to enhance the architecture design for improving system performance.


Evaluating collaborative filtering recommender systems
The key decisions in evaluating collaborative filtering recommender systems are reviewed: the user tasks being evaluated, the types of analysis and datasets being used, the ways in which prediction quality is measured, the evaluation of prediction attributes other than quality, and the user-based evaluation of the system as a whole.
Hybrid Recommender Systems: Survey and Experiments
  • R. Burke
  • Computer Science
    User Modeling and User-Adapted Interaction
  • 2004
This paper surveys the landscape of actual and possible hybrid recommenders, and introduces a novel hybrid, EntreeC, a system that combines knowledge-based recommendation and collaborative filtering to recommend restaurants, and shows that semantic ratings obtained from the knowledge- based part of the system enhance the effectiveness of collaborative filtering.
Item-based collaborative filtering recommendation algorithms
This paper analyzes item-based collaborative ltering techniques and suggests that item- based algorithms provide dramatically better performance than user-based algorithms, while at the same time providing better quality than the best available userbased algorithms. Recommendations: Item-to-Item Collaborative Filtering
This work compares three common approaches to solving the recommendation problem: traditional collaborative filtering, cluster models, and search-based methods, and their algorithm, which is called item-to-item collaborative filtering.
Toward the next generation of recommender systems: a survey of the state-of-the-art and possible extensions
This paper presents an overview of the field of recommender systems and describes the current generation of recommendation methods that are usually classified into the following three main
Empirical Analysis of Predictive Algorithms for Collaborative Filtering
Several algorithms designed for collaborative filtering or recommender systems are described, including techniques based on correlation coefficients, vector-based similarity calculations, and statistical Bayesian methods, to compare the predictive accuracy of the various methods in a set of representative problem domains.
Toward a personal recommender system
The need for personal recommender systems is reviewed and the deployment of personalRecommender systems using existing RSS and weblog technologies as the underlying communication infrastructure is proposed.
FlexRecs: expressing and combining flexible recommendations
A prototype flexible recommendation engine is described that realizes the proposed FlexRecs framework and its potential for capturing multiple, existing or novel, recommendations easily and having a flexible recommendation system that combines extensibility with reasonable performance is presented.
Google news personalization: scalable online collaborative filtering
This paper describes the approach to collaborative filtering for generating personalized recommendations for users of Google News using MinHash clustering, Probabilistic Latent Semantic Indexing, and covisitation counts, and combines recommendations from different algorithms using a linear model.
MovieLens unplugged: experiences with an occasionally connected recommender system
The results of a nine month field study show that although there are several challenges to overcome, mobile recommender systems have the potential to provide value to their users today.