SALMUBench: A Benchmark for Sensitive Association-Level Multimodal Unlearning

View PDF HTML (experimental)

Abstract:As multimodal models like CLIP become integral to downstream systems, the need to remove sensitive information is critical. However, machine unlearning for contrastively-trained encoders remains underexplored, and existing evaluations fail to diagnose fine-grained, association-level forgetting. We introduce SALMUBench (Sensitive Association-Level Multimodal Unlearning), a benchmark built upon a synthetic dataset of 60K persona-attribute associations and two foundational models: a Compromised model polluted with this data, and a Clean model without it. To isolate unlearning effects, both are trained from scratch on the same 400M-pair retain base, with the Compromised model additionally trained on the sensitive set. We propose a novel evaluation protocol with structured holdout sets (holdout identity, holdout association) to precisely measure unlearning efficacy and collateral damage. Our benchmark reveals that while utility-efficient deletion is feasible, current methods exhibit distinct failure modes: they either fail to forget effectively or over-generalize by erasing more than intended. SALMUBench sets a new standard for comprehensive unlearning evaluation, and we publicly release our dataset, models, evaluation scripts, and leaderboards to foster future research.

Comments: Accepted to CVPR 2026. Project page: this http URL

Subjects:

Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG)

Cite as: arXiv:2603.26316 [cs.CV]

(or arXiv:2603.26316v1 [cs.CV] for this version)

https://doi.org/10.48550/arXiv.2603.26316

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Cai Selvas-Sala [view email] [v1] Fri, 27 Mar 2026 11:33:59 UTC (7,695 KB)