Hear What Matters! Text-conditioned Selective Video-to-Audio Generation

View PDF HTML (experimental)

Abstract:This work introduces a new task, text-conditioned selective video-to-audio (V2A) generation, which produces only the user-intended sound from a multi-object video. This capability is especially crucial in multimedia production, where audio tracks are handled individually for each sound source for precise editing, mixing, and creative control. We propose SELVA, a novel text-conditioned V2A model that treats the text prompt as an explicit selector to distinctly extract prompt-relevant sound-source visual features from the video encoder. To suppress text-irrelevant activations with efficient video encoder finetuning, the proposed supplementary tokens promote cross-attention to yield robust semantic and temporal grounding. SELVA further employs an autonomous video-mixing scheme in a self-supervised manner to overcome the lack of mono audio track supervision. We evaluate SELVA on VGG-MONOAUDIO, a curated benchmark of clean single-source videos for such a task. Extensive experiments and ablations consistently verify its effectiveness across audio quality, semantic alignment, and temporal synchronization.

Comments: accepted to CVPR 2026

Subjects:

Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG); Multimedia (cs.MM); Sound (cs.SD); Audio and Speech Processing (eess.AS)

Cite as: arXiv:2512.02650 [cs.CV]

(or arXiv:2512.02650v2 [cs.CV] for this version)

https://doi.org/10.48550/arXiv.2512.02650

arXiv-issued DOI via DataCite

Submission history

From: Junwon Lee [view email] [v1] Tue, 2 Dec 2025 11:12:16 UTC (36,037 KB) [v2] Fri, 27 Mar 2026 12:17:58 UTC (21,366 KB)