Are we really making much progress? A worrying analysis of recent neural recommendation approaches

@article{Dacrema2019AreWR,
  title={Are we really making much progress? A worrying analysis of recent neural recommendation approaches},
  author={Maurizio Ferrari Dacrema and P. Cremonesi and D. Jannach},
  journal={Proceedings of the 13th ACM Conference on Recommender Systems},
  year={2019}
}
Deep learning techniques have become the method of choice for researchers working on algorithmic aspects of recommender systems. [...] Key Result Overall, our work sheds light on a number of potential problems in today's machine learning scholarship and calls for improved scientific practices in this area.Expand
View on ACM
re.public.polimi.it
231 Citations
Highly Influential Citations
15
Background Citations
97
Methods Citations
40
Results Citations
22

Supplemental Code

Github Repo
Via Papers with Code
MaurizioFD/RecSys2019_DeepLearning_Evaluation
This is the repository of our article published in RecSys 2019 "Are We Really Making Much Progress? A Worrying Analysis of Recent Neural Recommendation Approaches" and of several follow-up studies.
717 Stars • 198 Forks
Explore Further
Discover more papers related to the topics discussed in this paper

231 Citations

Citation Type

Citation Type

Why Are Deep Learning Models Not Consistently Winning Recommender Systems Competitions Yet?: A Position Paper
TLDR
This paper investigates possible reasons for the almost consistent success of DL based models in recommendation-related machine learning competitions and considers multiple possible factors such as the characteristics and complexity of the problem settings, datasets, and DL methods. Expand
Performance comparison of neural and non-neural approaches to session-based recommendation
TLDR
An extensive set of experiments were conducted, using a variety of datasets, in which it turned out that simple techniques in most cases outperform recent neural approaches and point to certain major limitations of today's research practice. Expand
Methodological Issues in Recommender Systems Research (Extended Abstract)
TLDR
Analysis of research papers published recently at top-ranked conferences found only 7 were reproducible with reasonable effort, and 6 of them could often be outperformed by relatively simple heuristic methods, e.g., nearest neighbors. Expand
Towards a Better Understanding of Linear Models for Recommendation
TLDR
Through the derivation and analysis of the closed-form solutions for two basic regression and matrix factorization approaches, it is found these two approaches are indeed inherently related but also diverge in how they "scale-down" the singular values of the original user-item interaction matrix. Expand
Efficient Neural Matrix Factorization without Sampling for Recommendation
TLDR
This work derives three new optimization methods through rigorous mathematical reasoning, which can efficiently learn model parameters from the whole data with a rather low time complexity, and presents a general framework named ENMF, short for Efficient Neural Matrix Factorization. Expand
A Survey on Neural Recommendation: From Collaborative Filtering to Content and Context Enriched Recommendation
TLDR
A systematic review on neural recommender models is conducted, aiming to summarize the field from the perspective of recommendation modeling, and discusses some promising directions in this field, including benchmarking recommender systems, graph reasoning based recommendation models, and explainable and fair recommendations for social good. Expand
Efficient Heterogeneous Collaborative Filtering without Negative Sampling for Recommendation
TLDR
This work proposes a novel non-sampling transfer learning solution, named Efficient Heterogeneous Collaborative Filtering (EHCF) for Top-N recommendation that can not only model fine-grained user-item relations, but also efficiently learn model parameters from the whole heterogeneous data with a rather low time complexity. Expand
Adversarial Machine Learning in Recommender Systems: State of the art and Challenges
TLDR
An exhaustive literature review of 60 articles published in major RS and ML journals and conferences is provided to present recent advances on AML-RS for the security of RS and to show another successful application of AML in generative adversarial networks (GANs), which use the core concept of learning in AML for generative applications. Expand
UserReg: A Simple but Strong Model for Rating Prediction
TLDR
A simple linear model based on Matrix Factorization (MF), called UserReg, is proposed, which regularizes users’ latent representations with explicit feedback information for rating prediction and can be used as a strong baseline for future CF research. Expand
On the instability of embeddings for recommender systems: the case of matrix factorization
TLDR
A generalization of MF is presented, called Nearest Neighbors Matrix Factorization (NNMF), which propagates the information about items and users to their neighbors, speeding up the training procedure and extending the amount of information that supports recommendations and representations. Expand
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 59 REFERENCES
Performance comparison of neural and non-neural approaches to session-based recommendation
TLDR
An extensive set of experiments were conducted, using a variety of datasets, in which it turned out that simple techniques in most cases outperform recent neural approaches and point to certain major limitations of today's research practice. Expand
When Recurrent Neural Networks meet the Neighborhood for Session-Based Recommendation
TLDR
This work shows based on a comprehensive empirical evaluation that a heuristics-based nearest neighbor (kNN) scheme for sessions outperforms GRU4REC in the large majority of the tested configurations and datasets and ensures the scalability of the kNN method. Expand
Session-based Recommendations with Recurrent Neural Networks
TLDR
It is argued that by modeling the whole session, more accurate recommendations can be provided by an RNN-based approach for session-based recommendations, and introduced several modifications to classic RNNs such as a ranking loss function that make it more viable for this specific problem. Expand
Evaluation of session-based recommendation algorithms
TLDR
An in-depth performance comparison of a number of session-based recommendation algorithms based on recurrent neural networks, factorized Markov model approaches, as well as simpler methods based, e.g., on nearest neighbor schemes reveals that algorithms of this latter class often perform equally well or significantly better than today’s more complex approaches based on deep neural networks. Expand
Sequence-Aware Recommender Systems
TLDR
A categorization of the corresponding recommendation tasks and goals is proposed, existing algorithmic solutions are summarized, methodological approaches when benchmarking what the authors call sequence-aware recommender systems are discussed, and open challenges in the area are outlined. Expand
Multi-Pointer Co-Attention Networks for Recommendation
TLDR
A review- by-review pointer-based learning scheme that extracts important reviews from user and item reviews and subsequently matches them in a word-by-word fashion, which significantly outperforms existing state-of-the-art models. Expand
Collaborative Denoising Auto-Encoders for Top-N Recommender Systems
TLDR
It is demonstrated that the proposed model is a generalization of several well-known collaborative filtering models but with more flexible components, and that CDAE consistently outperforms state-of-the-art top-N recommendation methods on a variety of common evaluation metrics. Expand
Collaborative Deep Learning for Recommender Systems
TLDR
A hierarchical Bayesian model called collaborative deep learning (CDL), which jointly performs deep representation learning for the content information and collaborative filtering for the ratings (feedback) matrix is proposed, which can significantly advance the state of the art. Expand
Towards reproducibility in recommender-systems research
TLDR
The recommender-system community needs to survey other research fields and learn from them, find a common understanding of reproducibility, identify and understand the determinants that affect reproduCibility, conduct more comprehensive experiments, and establish best-practice guidelines for recommender -systems research. Expand
Neural Collaborative Filtering
TLDR
This work strives to develop techniques based on neural networks to tackle the key problem in recommendation --- collaborative filtering --- on the basis of implicit feedback, and presents a general framework named NCF, short for Neural network-based Collaborative Filtering. Expand
...
1
2
3
4
5
...