Diffusion Recommender Models and the Illusion of Progress: A Concerning Study of Reproducibility and a Conceptual Mismatch

View PDF HTML (experimental)

Abstract:Countless new machine learning models are published every year and are reported to significantly advance the state-of-the-art in top-n recommendation. However, earlier reproducibility studies indicate that progress in this area may be quite limited, due to widespread methodological issues, e.g., comparisons with untuned baseline models, creating an illusion of progress. In this work, we examine whether these problems persist in today's research by attempting to reproduce nine SIGIR 2023 and 2024 recommendation algorithms based on Denoising Diffusion Probabilistic Models, a recent but rapidly expanding research area. Only 25% of reported results are fully reproducible and, since the original papers relied on weak baselines, they do not establish the superiority of diffusion models over state-of-the-art methods. In our controlled evaluations, well-tuned simpler baselines consistently exceed the diffusion-based models' effectiveness reported in the original papers. Furthermore, we identify key mismatches between the characteristics of diffusion models and those of the traditional top-n recommendation task, raising doubts about their suitability for recommendation. Moreover, in the analyzed papers, the generative capabilities of these models are constrained to a minimum. Overall, our results call for greater scientific rigor and a disruptive change in the research and publication culture in this area.

Subjects:

Information Retrieval (cs.IR); Machine Learning (cs.LG); Neural and Evolutionary Computing (cs.NE)

Cite as: arXiv:2505.09364 [cs.IR]

(or arXiv:2505.09364v3 [cs.IR] for this version)

https://doi.org/10.48550/arXiv.2505.09364

arXiv-issued DOI via DataCite

Journal reference: ACM Transactions on Recommender Systems (2026)

Related DOI:

https://doi.org/10.1145/3795792

DOI(s) linking to related resources

Submission history

From: Michael Benigni [view email] [v1] Wed, 14 May 2025 13:13:53 UTC (9,333 KB) [v2] Thu, 15 May 2025 21:26:23 UTC (9,333 KB) [v3] Thu, 26 Mar 2026 20:09:03 UTC (146 KB)