Towards Holistic Modeling for Video Frame Interpolation with Auto-regressive Diffusion Transformers

View PDF HTML (experimental)

Abstract:Existing video frame interpolation (VFI) methods often adopt a frame-centric approach, processing videos as independent short segments (e.g., triplets), which leads to temporal inconsistencies and motion artifacts. To overcome this, we propose a holistic, video-centric paradigm named Local Diffusion Forcing for Video Frame Interpolation (LDF-VFI). Our framework is built upon an auto-regressive diffusion transformer that models the entire video sequence to ensure long-range temporal coherence. To mitigate error accumulation inherent in auto-regressive generation, we introduce a novel skip-concatenate sampling strategy that effectively maintains temporal stability. Furthermore, LDF-VFI incorporates sparse, local attention and tiled VAE encoding, a combination that not only enables efficient processing of long sequences but also allows generalization to arbitrary spatial resolutions (e.g., 4K) at inference without retraining. An enhanced conditional VAE decoder, which leverages multi-scale features from the input video, further improves reconstruction fidelity. Empirically, LDF-VFI achieves state-of-the-art performance on challenging VFI benchmarks, demonstrating superior per-frame quality and temporal consistency, especially in scenes with large motion. The source code is available at this https URL.

Subjects:

Computer Vision and Pattern Recognition (cs.CV)

Cite as: arXiv:2601.14959 [cs.CV]

(or arXiv:2601.14959v2 [cs.CV] for this version)

https://doi.org/10.48550/arXiv.2601.14959

arXiv-issued DOI via DataCite

Submission history

From: Xinyu Peng [view email] [v1] Wed, 21 Jan 2026 12:58:52 UTC (1,633 KB) [v2] Mon, 30 Mar 2026 08:09:27 UTC (24,420 KB)