Overview of NewsREEL'16: Multi-dimensional Evaluation of Real-Time Stream-Recommendation Algorithms

@inproceedings{Kille2016OverviewON,
  title={Overview of NewsREEL'16: Multi-dimensional Evaluation of Real-Time Stream-Recommendation Algorithms},
  author={Benjamin Kille and Andreas Lommatzsch and Gebrekirstos G. Gebremeskel and Frank Hopfgartner and Martha Larson and Jonas Seiler and Davide Malagoli and Andr{\'a}s Ser{\'e}ny and Torben Brodt and Arjen P. de Vries},
  booktitle={CLEF},
  year={2016}
}
Successful news recommendation requires facing the challenges of dynamic item sets, contextual item relevance, and of fulfilling non-functional requirements, such as response time. The CLEF NewsREEL challenge is a campaign-style evaluation lab allowing participants to tackle news recommendation and to optimize and evaluate their recommender algorithms both online and offline. In this paper, we summarize the objectives and challenges of NewsREEL 2016. We cover two contrasting perspectives on the… 

CLEF NewsREEL 2016: Comparing Multi-dimensional Offline and Online Evaluation of News Recommender Systems

TLDR
Results illustrate potentials for multi-dimensional evaluation of recommendation algorithms in a living lab and simulation based evaluation setting and overviews ideas on living lab evaluation that have been presented as part of a “New Ideas” track at the conference in Portugal.

Real-time News Recommendations using Apache Spark

TLDR
The evaluation of the implemented system shows that the approach is suitable for the news recommenation scenario and provides high-quality results while satisfying the tight time constraints.

Optimizing a Scalable News Recommender System

TLDR
The implemented recommender system is able to adapt to the continuous changes of the set of relevant news articles as well as it considers the different user preferences dependent from the hour the day, and can be easily extended to streams of much larger volume.

CLEF 2017 NewsREEL Overview: A Stream-Based Recommender Task for Evaluation and Education

TLDR
The experiences of NewsREEL participants as well as those of instructors teaching recommender systems to students are discussed, and the ability to support the education of future data scientists is showcased.

Clicks Pattern Analysis for Online News Recommendation Systems

TLDR
This working note chronologically describes the participants' participation in NewsREEL online task in the year 2016, and proposes that more attention should be paid on the time series patterns of clicks and impressions in this contest.

CLEF NewsREEL 2017 Overview : Contextual Bandit News Recommendation

TLDR
This work builds a delegation model based on the contextual bandit algorithm to investigate whether a bandit approach combined with context extracted from the user side, from the item side and from user-item interaction can help choose the appropriate recommender from a recommender algorithm pool for the incoming recommendation requests.

CLEF NewsREEL 2017: Contextual Bandit News Recommendation

TLDR
This work builds a delegation model based on the contextual bandit algorithm to investigate whether a bandit approach combined with context extracted from the user side, from the item side and from user-item interaction can help choose the appropriate recommender from a recommender algorithm pool for the incoming recommendation requests.

CLEF NewsREEL 2017 : Contextual Bandit News Recommendation

TLDR
This work builds a delegation model based on the contextual bandit algorithm to investigate whether a bandit approach combined with context extracted from the user side, from the item side and from user-item interaction can help choose the appropriate recommender from a recommender algorithm pool for the incoming recommendation requests.

A News Recommender Engine with a Killer Sequence

TLDR
This submission to the CLEF NewsREEL 2107 News Recommendation Evaluation Lab attempts to apply the concept of storytelling to the participating news domains by employing collaborative filtering to discover an optimal sequence of articles – a killer sequence.

References

SHOWING 1-10 OF 22 REFERENCES

Stream-Based Recommendations: Online and Offline Evaluation as a Service

TLDR
The evaluation framework makes possible the reproducible evaluation of recommender algorithms for stream data, taking into account recommender precision as well as the technical complexity of theRecommender algorithms.

Benchmarking News Recommendations: The CLEF NewsREEL Use Case

TLDR
The objectives and challenges of the NewsREel lab are discussed, last year's campaign is summarized and the main research challenges that can be addressed by participating in NewsREEL 2016 are outlined.

Shedding light on a living lab: the CLEF NEWSREEL open recommendation platform

TLDR
In the CLEF NEWSREEL lab, participants are invited to evaluate news recommendation techniques in real-time by providing news recommendations to actual users that visit commercial news portals to satisfy their information needs by using the Open Recommendation Platform.

Real-time top-n recommendation in social streams

TLDR
This paper focuses on analyzing social streams in real-time for personalized topic recommendation and discovery and presents Stream Ranking Matrix Factorization - RMFX, which uses a pairwise approach to matrix factorization in order to optimize the personalized ranking of topics.

Linked Open Data-Enabled Recommender Systems: ESWC 2014 Challenge on Book Recommendation

In this chapter we present a report of the ESWC 2014 Challenge on Linked Open Data-enabled Recommender Systems, which consisted of three tasks in the context of book recommendation: rating prediction

Workshop and challenge on news recommender systems

TLDR
This workshop brought together researchers and practitioners around the topics of designing and evaluating novel news recommender systems and offered a challenge allowing participants to evaluate their recommendation algorithms with actual user feedback.

Offline and online evaluation of news recommender systems at swissinfo.ch

TLDR
It is demonstrated that there is a major difference between offline and online accuracy evaluations, and in an offline setting, recommending most popular stories is the best strategy, while in a live environment this strategy is the poorest.

Real-time recommendations for user-item streams

TLDR
This paper analyzes the user-item interaction stream for several online news portals and develops several different algorithms optimized for providing recommendations based on streams fulfilling the requirements according to quality, robustness, scalability, and tight time-constraints.

Random Performance Differences Between Online Recommender System Algorithms

TLDR
The experiments aim to quantify the expected degree of variation in performance that cannot be attributed to differences between systems, and classify and discuss the non-algorithmic causes of performance differences observed.

Predicting Online Performance of News Recommender Systems Through Richer Evaluation Metrics

TLDR
This work investigates how metrics that can be measured offline can be used to predict the online performance of recommender systems, thus avoiding costly A-B testing and proposing a self-adjusting algorithm blend that optimizes a recommender's parameters over time.