MoD-DPO: Towards Mitigating Cross-modal Hallucinations in Omni LLMs using Modality Decoupled Preference Optimization

View PDF HTML (experimental)

Abstract:Omni-modal large language models (omni LLMs) have recently achieved strong performance across audiovisual understanding tasks, yet they remain highly susceptible to cross-modal hallucinations arising from spurious correlations and dominant language priors. In this work, we propose Modality-Decoupled Direct Preference Optimization (MoD-DPO), a simple and effective framework for improving modality grounding in omni LLMs. MoD-DPO introduces modality-aware regularization terms that explicitly enforce invariance to corruptions in irrelevant modalities and sensitivity to perturbations in relevant modalities, thereby reducing unintended cross-modal interactions. To further mitigate over-reliance on textual priors, we incorporate a language-prior debiasing penalty that discourages hallucination-prone text-only responses. Extensive experiments across multiple audiovisual hallucination benchmarks demonstrate that MoD-DPO consistently improves perception accuracy and hallucination resistance, outperforming previous preference optimization baselines under similar training budgets. Our findings underscore the importance of modality-faithful alignment and demonstrate a scalable path toward more reliable and resilient multimodal foundation models.

Comments: CVPR 2026. Project Page: this https URL

Subjects:

Computer Vision and Pattern Recognition (cs.CV); Computation and Language (cs.CL); Machine Learning (cs.LG)

Cite as: arXiv:2603.03192 [cs.CV]

(or arXiv:2603.03192v2 [cs.CV] for this version)

https://doi.org/10.48550/arXiv.2603.03192

arXiv-issued DOI via DataCite

Submission history

From: Ashutosh Chaubey [view email] [v1] Tue, 3 Mar 2026 17:50:24 UTC (3,570 KB) [v2] Fri, 27 Mar 2026 20:27:35 UTC (3,571 KB)