Are we really making much progress? A worrying analysis of recent neural recommendation approaches

@article{Dacrema2019AreWR,
  title={Are we really making much progress? A worrying analysis of recent neural recommendation approaches},
  author={Maurizio Ferrari Dacrema and Paolo Cremonesi and D. Jannach},
  journal={Proceedings of the 13th ACM Conference on Recommender Systems},
  year={2019}
}
Deep learning techniques have become the method of choice for researchers working on algorithmic aspects of recommender systems. [] Key Result Overall, our work sheds light on a number of potential problems in today's machine learning scholarship and calls for improved scientific practices in this area.
View on ACM
re.public.polimi.it
342 Citations
Highly Influential Citations
26
Background Citations
169
Methods Citations
79
Results Citations
44

Figures and Tables from this paper

342 Citations

A Troubling Analysis of Reproducibility and Progress in Recommender Systems Research
TLDR
This analysis compares recent results in the area of neural recommendation approaches based on collaborative filtering against a consistent set of existing simple baselines to discuss common issues in today’s research practice, which have apparently led the field to a certain level of stagnation.
Empirical analysis of session-based recommendation algorithms
TLDR
Twelve algorithmic approaches to session-based recommendation are compared and it is found that the progress in terms of prediction accuracy that is achieved with neural methods is still limited and simple heuristic methods based on nearest-neighbors schemes are preferable over conceptually and computationally more complex methods.
Empirical analysis of session‑based recommendation algorithms A comparison of neural and non‑neural approaches
TLDR
Twelve algorithmic approaches to session-based recommendation are compared and it is found that the progress in terms of prediction accuracy that is achieved with neural methods is still limited and simple heuristic methods based on nearest-neighbors schemes are preferable over conceptually and computationally more complex methods.
Why Are Deep Learning Models Not Consistently Winning Recommender Systems Competitions Yet?: A Position Paper
TLDR
This paper investigates possible reasons for the almost consistent success of DL based models in recommendation-related machine learning competitions and considers multiple possible factors such as the characteristics and complexity of the problem settings, datasets, and DL methods.
Performance comparison of neural and non-neural approaches to session-based recommendation
TLDR
An extensive set of experiments were conducted, using a variety of datasets, in which it turned out that simple techniques in most cases outperform recent neural approaches and point to certain major limitations of today's research practice.
BETA-Rec: Build, Evaluate and Tune Automated Recommender Systems
TLDR
BETA-Rec, an open source project for Building, Evaluating and Tuning Automated Recommender Systems, aims to provide a practical data toolkit for building end-to-end recommendation systems in a standardized way and is designed to be both modular and extensible.
Methodological Issues in Recommender Systems Research (Extended Abstract)
TLDR
Analysis of research papers published recently at top-ranked conferences found only 7 were reproducible with reasonable effort, and 6 of them could often be outperformed by relatively simple heuristic methods, e.g., nearest neighbors.
Deep Learning for Recommender Systems: A Netflix Case Study
TLDR
The challenges encountered and lessons learned in using deep learning for recommender systems at Netflix are outlined and some of the take-aways that may generalize to other applications beyond Netflix are summarized.
On the Generalizability and Predictability of Recommender Systems
TLDR
RecZilla is created, a meta-learning approach to recommender systems that uses a model to predict the best algorithm and hyperparameters for new, unseen datasets, and is able to substantially reduce the level of human involvement when faced with a new recommender system application.
Application of WGAN-GP in recommendation and Questioning the relevance of GAN-based approaches
TLDR
A recommender system based on WGAN-GP is proposed which is founded on a previous model (CFGAN), and the empirical results show that it is competitive with state-of-the-art GAN approaches, but there is no evidence of advantage of using Wgan-GP instead of the original GAN, at least from the accuracy point of view.
...
...

References

SHOWING 1-10 OF 63 REFERENCES
Performance comparison of neural and non-neural approaches to session-based recommendation
TLDR
An extensive set of experiments were conducted, using a variety of datasets, in which it turned out that simple techniques in most cases outperform recent neural approaches and point to certain major limitations of today's research practice.
When Recurrent Neural Networks meet the Neighborhood for Session-Based Recommendation
TLDR
This work shows based on a comprehensive empirical evaluation that a heuristics-based nearest neighbor (kNN) scheme for sessions outperforms GRU4REC in the large majority of the tested configurations and datasets and ensures the scalability of the kNN method.
Session-based Recommendations with Recurrent Neural Networks
TLDR
It is argued that by modeling the whole session, more accurate recommendations can be provided by an RNN-based approach for session-based recommendations, and introduced several modifications to classic RNNs such as a ranking loss function that make it more viable for this specific problem.
Evaluation of session-based recommendation algorithms
TLDR
An in-depth performance comparison of a number of session-based recommendation algorithms based on recurrent neural networks, factorized Markov model approaches, as well as simpler methods based, e.g., on nearest neighbor schemes reveals that algorithms of this latter class often perform equally well or significantly better than today’s more complex approaches based on deep neural networks.
Sequence-Aware Recommender Systems
TLDR
A categorization of the corresponding recommendation tasks and goals is proposed, existing algorithmic solutions are summarized, methodological approaches when benchmarking what the authors call sequence-aware recommender systems are discussed, and open challenges in the area are outlined.
Multi-Pointer Co-Attention Networks for Recommendation
TLDR
A review- by-review pointer-based learning scheme that extracts important reviews from user and item reviews and subsequently matches them in a word-by-word fashion, which significantly outperforms existing state-of-the-art models.
Collaborative Denoising Auto-Encoders for Top-N Recommender Systems
TLDR
It is demonstrated that the proposed model is a generalization of several well-known collaborative filtering models but with more flexible components, and that CDAE consistently outperforms state-of-the-art top-N recommendation methods on a variety of common evaluation metrics.
Collaborative Deep Learning for Recommender Systems
TLDR
A hierarchical Bayesian model called collaborative deep learning (CDL), which jointly performs deep representation learning for the content information and collaborative filtering for the ratings (feedback) matrix is proposed, which can significantly advance the state of the art.
Towards reproducibility in recommender-systems research
TLDR
The recommender-system community needs to survey other research fields and learn from them, find a common understanding of reproducibility, identify and understand the determinants that affect reproduCibility, conduct more comprehensive experiments, and establish best-practice guidelines for recommender -systems research.
Neural Collaborative Filtering
TLDR
This work strives to develop techniques based on neural networks to tackle the key problem in recommendation --- collaborative filtering --- on the basis of implicit feedback, and presents a general framework named NCF, short for Neural network-based Collaborative Filtering.
...
...