DiFlowDubber: Discrete Flow Matching for Automated Video Dubbing via Cross-Modal Alignment and Synchronization

View PDF HTML (experimental)

Abstract:Video dubbing has broad applications in filmmaking, multimedia creation, and assistive speech technology. Existing approaches either train directly on limited dubbing datasets or adopt a two-stage pipeline that adapts pre-trained text-to-speech (TTS) models, which often struggle to produce expressive prosody, rich acoustic characteristics, and precise synchronization. To address these issues, we propose DiFlowDubber with a novel two-stage training framework that effectively transfers knowledge from a pre-trained TTS model to video-driven dubbing, with a discrete flow matching generative backbone. Specifically, we design a FaPro module that captures global prosody and stylistic cues from facial expressions and leverages this information to guide the modeling of subsequent speech attributes. To ensure precise speech-lip synchronization, we introduce a Synchronizer module that bridges the modality gap among text, video, and speech, thereby improving cross-modal alignment and generating speech that is temporally synchronized with lip movements. Experiments on two primary benchmark datasets demonstrate that DiFlowDubber outperforms previous methods across multiple metrics.

Comments: Accepted at CVPR 2026 Findings

Subjects:

Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI); Multimedia (cs.MM); Sound (cs.SD)

Cite as: arXiv:2603.14267 [cs.CV]

(or arXiv:2603.14267v3 [cs.CV] for this version)

https://doi.org/10.48550/arXiv.2603.14267

arXiv-issued DOI via DataCite

Submission history

From: Ngoc Son Nguyen [view email] [v1] Sun, 15 Mar 2026 07:53:23 UTC (8,690 KB) [v2] Tue, 17 Mar 2026 05:01:44 UTC (8,684 KB) [v3] Fri, 27 Mar 2026 07:22:39 UTC (8,684 KB)