Reproducibility and Artifact Consistency of the SIGIR 2022 Recommender Systems Papers Based on Message Passing

View PDF HTML (experimental)

Abstract:Graph-based techniques relying on neural networks and embeddings have gained attention as a way to develop Recommender Systems (RS) with several papers on the topic presented at SIGIR 2022 and 2023. Given the importance of ensuring that published research is methodologically sound and reproducible, in this paper we analyze 10 graph-based RS papers, most of which were published at SIGIR 2022, and assess their impact on subsequent work published in SIGIR 2023. Our analysis reveals several critical points that require attention: (i) the prevalence of bad practices, such as erroneous data splits or information leakage between training and testing data, which call into question the validity of the results; (ii) frequent inconsistencies between the provided artifacts (source code and data) and their descriptions in the paper, causing uncertainty about what is actually being evaluated; and (iii) the preference for new or complex baselines that are weaker compared to simpler ones, creating the impression of continuous improvement even when, particularly for the Amazon-Book dataset, the state-of-the-art has significantly worsened. Due to these issues, we are unable to confirm the claims made in most of the papers that we examined and attempted to reproduce.

Subjects:

Information Retrieval (cs.IR); Digital Libraries (cs.DL); Machine Learning (cs.LG); Neural and Evolutionary Computing (cs.NE)

Cite as: arXiv:2503.07823 [cs.IR]

(or arXiv:2503.07823v2 [cs.IR] for this version)

https://doi.org/10.48550/arXiv.2503.07823

arXiv-issued DOI via DataCite

Journal reference: ACM Transactions on Information Systems (2025)

Related DOI:

https://doi.org/10.1145/3772275

DOI(s) linking to related resources

Submission history

From: Michael Benigni [view email] [v1] Mon, 10 Mar 2025 20:09:04 UTC (1,124 KB) [v2] Fri, 27 Mar 2026 12:22:03 UTC (1,172 KB)