VRR-QA: Visual Relational Reasoning in Videos Beyond Explicit Cues

View PDF HTML (experimental)

Abstract:Video Question Answering (VideoQA) has made significant strides by leveraging multimodal learning to align visual and textual modalities. However, current benchmarks overwhelmingly focus on questions answerable through explicit visual content - actions, objects, and events - directly observable within individual frames or short clips. To truly understand videos as humans do, models must go beyond what is directly shown, inferring hidden relationships and contextual cues that are only implied across frames. Current benchmarks fail to capture this essential aspect of video understanding. To address this gap, we introduce VRR-QA, a benchmark for Visual Relational Reasoning Beyond Explicit Cues. We curate our benchmark from creative and cinematic videos such as movies, that deliberately employ storytelling techniques which omit direct depictions of certain events or relations, requiring viewers to infer them. VRR-QA comprises 1K meticulously expert-annotated QA pairs drawn from 1K creative video clips covering 15 genres across 7 decades of content, from both live-action and animated titles. Our extensive evaluations on 14 leading VideoQA models reveals consistent and significant performance degradation, underscoring their reliance on surface-level visual cues and highlighting the difficulty of implicit reasoning. Even the best model substantially underperforms human baselines with only 64% accuracy. Performance variations across models further illustrate the complexity and diversity of the challenges presented by VRR-QA. By releasing both dataset and data collection framework, VRR-QA establishes a rigorous, diverse, and reproducible testbed for advancing VideoQA: this https URL.

Comments: Accepted at CVPR 2026

Subjects:

Computer Vision and Pattern Recognition (cs.CV)

Cite as: arXiv:2506.21742 [cs.CV]

(or arXiv:2506.21742v3 [cs.CV] for this version)

https://doi.org/10.48550/arXiv.2506.21742

arXiv-issued DOI via DataCite

Submission history

From: Sirnam Swetha [view email] [v1] Thu, 26 Jun 2025 19:53:54 UTC (13,561 KB) [v2] Sun, 5 Oct 2025 23:04:14 UTC (13,600 KB) [v3] Sun, 29 Mar 2026 20:30:11 UTC (15,181 KB)