A.I.R.: Enabling Adaptive, Iterative, and Reasoning-based Frame Selection For Video Question Answering

View PDF HTML (experimental)

Abstract:Effectively applying Vision-Language Models (VLMs) to Video Question Answering (VideoQA) hinges on selecting a concise yet comprehensive set of frames, as processing entire videos is computationally infeasible. However, current frame selection methods face a critical trade-off: approaches relying on lightweight similarity models, such as CLIP, often fail to capture the nuances of complex queries, resulting in inaccurate similarity scores that cannot reflect the authentic query-frame relevance, which further undermines frame selection. Meanwhile, methods that leverage a VLM for deeper analysis achieve higher accuracy but incur prohibitive computational costs. To address these limitations, we propose A.I.R., a training-free approach for Adaptive, Iterative, and Reasoning-based frame selection. We leverage a powerful VLM to perform deep, semantic analysis on complex queries, and this analysis is deployed within a cost-effective iterative loop that processes only a small batch of the most high-potential frames at a time. Extensive experiments on various VideoQA benchmarks demonstrate that our approach outperforms existing frame selection methods, significantly boosts the performance of the foundation VLM, and achieves substantial gains in computational efficiency over other VLM-based techniques.

Comments: ICLR 2026 Paper

Subjects:

Computer Vision and Pattern Recognition (cs.CV)

Cite as: arXiv:2510.04428 [cs.CV]

(or arXiv:2510.04428v3 [cs.CV] for this version)

https://doi.org/10.48550/arXiv.2510.04428

arXiv-issued DOI via DataCite

Submission history

From: Yuanhao Zou [view email] [v1] Mon, 6 Oct 2025 01:51:13 UTC (11,986 KB) [v2] Thu, 26 Feb 2026 01:08:09 UTC (11,987 KB) [v3] Fri, 27 Mar 2026 02:48:21 UTC (11,987 KB)