VLM-3R: Vision-Language Models Augmented with Instruction-Aligned 3D Reconstruction
arXiv:2505.20279v3 Announce Type: replace-cross Abstract: The rapid advancement of Large Multimodal Models (LMMs) for 2D images and videos has motivated extending these models to understand 3D scenes, aiming for human-like visual-spatial intelligence. Nevertheless, achieving deep spatial understanding comparable to human capabilities poses significant challenges in model encoding and data acquisition. Existing methods frequently depend on external depth sensors for geometry capture or utilize off-the-shelf algorithms for pre-constructing 3D maps, thereby limiting their scalability, especially — Zhiwen Fan, Jian Zhang, Renjie Li, Junge Zhang, Runjin Chen, Hezhen Hu, Kevin Wang, Huaizhi Qu, Shijie Zhou, Dilin Wang, Zhicheng Yan, Hongyu Xu, Justin Theiss, Tianlong Chen, Jiachen Li, Zhengzhong Tu, Zhangyang Wang, Rakesh Ranjan
Authors:Zhiwen Fan, Jian Zhang, Renjie Li, Junge Zhang, Runjin Chen, Hezhen Hu, Kevin Wang, Huaizhi Qu, Shijie Zhou, Dilin Wang, Zhicheng Yan, Hongyu Xu, Justin Theiss, Tianlong Chen, Jiachen Li, Zhengzhong Tu, Zhangyang Wang, Rakesh Ranjan
View PDF HTML (experimental)
Abstract:The rapid advancement of Large Multimodal Models (LMMs) for 2D images and videos has motivated extending these models to understand 3D scenes, aiming for human-like visual-spatial intelligence. Nevertheless, achieving deep spatial understanding comparable to human capabilities poses significant challenges in model encoding and data acquisition. Existing methods frequently depend on external depth sensors for geometry capture or utilize off-the-shelf algorithms for pre-constructing 3D maps, thereby limiting their scalability, especially with prevalent monocular video inputs and for time-sensitive applications. In this work, we introduce VLM-3R, a unified framework for Vision-Language Models (VLMs) that incorporates 3D Reconstructive instruction tuning. VLM-3R processes monocular video frames by employing a geometry encoder to derive implicit 3D tokens that represent spatial understanding. Leveraging our Spatial-Visual-View Fusion and over 200K curated 3D reconstructive instruction tuning question-answer (QA) pairs, VLM-3R effectively aligns real-world spatial context with language instructions. This enables monocular 3D spatial assistance and embodied reasoning. To facilitate the evaluation of temporal reasoning, we introduce the Vision-Spatial-Temporal Intelligence benchmark, featuring over 138.6K QA pairs across five distinct tasks focused on evolving spatial relationships. Extensive experiments demonstrate that our model, VLM-3R, not only facilitates robust visual-spatial reasoning but also enables the understanding of temporal 3D context changes, excelling in both accuracy and scalability.
Comments: Project Page: this https URL
Subjects:
Computer Vision and Pattern Recognition (cs.CV); Computation and Language (cs.CL)
Cite as: arXiv:2505.20279 [cs.CV]
(or arXiv:2505.20279v3 [cs.CV] for this version)
https://doi.org/10.48550/arXiv.2505.20279
arXiv-issued DOI via DataCite
Submission history
From: Zhiwen Fan [view email] [v1] Mon, 26 May 2025 17:56:30 UTC (838 KB) [v2] Sun, 1 Jun 2025 21:20:16 UTC (838 KB) [v3] Sun, 29 Mar 2026 01:33:48 UTC (872 KB)
Sign in to highlight and annotate this article
Conversation starters
Daily AI Digest
Get the top 5 AI stories delivered to your inbox every morning.
More about
researchpaperarxiv
NevaMind AI: Advanced Memory for Proactive Agents
Unveiling memU: A Sophisticated Memory Solution for 24/7 Proactive AI Agents NevaMind AI is thrilled to introduce memU , an open-source project dedicated to providing advanced memory functionalities for AI agents operating around the clock. Designed with the demands of proactive systems in mind, such as the principles behind Moltbot (ClawDBot), memU aims to be a cornerstone for developing more intelligent and responsive AI. Why memU? In the rapidly evolving landscape of Artificial Intelligence, robust memory management is paramount for agents that need to perform complex tasks, maintain context over long interactions, and learn continuously. memU addresses this critical need by offering: 24/7 Proactive Operation : Ensures agents are always ready, minimizing latency and maximizing efficienc
AGI Won’t Automate Most Jobs—Economist Reveals Why They’re Not Worth It
Why AGI Won't Steal Your Job—And That Might Be Worse The fear that artificial general intelligence (AGI) will render most human labor obsolete has become a staple of modern discourse. But what if the real story is more nuanced—and more unsettling—than the dystopian narrative suggests? A new paper by one of the world's foremost economists of automation challenges the assumption that AGI will simply replace human workers en masse. Instead, it reveals a paradox: many jobs won't be automated not because they're irreplaceable, but because they're not worth the effort to automate. Key Takeaways: The traditional view of AGI as a universal job-killer is being questioned by leading economists. Many jobs may remain untouched by automation, not due to their complexity, but because they lack economic
Knowledge Map
Connected Articles — Knowledge Graph
This article is connected to other articles through shared AI topics and tags.
More in Research Papers
Academic Proof-of-Work in the Age of LLMs
Written quickly as part of the Inkhaven Residency . Related: Bureaucracy as active ingredient , pain as active ingredient A widely known secret in academia is that many of the formalities serve in large part proof of work . That is, the reason expensive procedures exist is that some way of filtering must exist, and the amount of effort invested can often be a good proxy for the quality of the work. Specifically, the pool of research is vast, and good research can often be hard to identify. Even engaging in research enough to understand its quality can be expensive. As a result, people look toward signs of visible, expensive effort in order to determine whether to engage in the research at all. Why do people insist only on reading research that’s published in well-formatted, well-written pa
Signals – finding the most informative agent traces without LLM judges (arxiv.org)
Hello Peeps Salman, Shuguang and Adil here from Katanemo Labs (a DigitalOcean company). Wanted to introduce our latest research on agentic systems called Signals. If you've been building agents, you've probably noticed that there are far too many agent traces/trajectories to review one by one, and using humans or extra LLM calls to inspect all of them gets expensive really fast. The paper proposes a lightweight way to compute structured “signals” from live agent interactions so you can surface the trajectories most worth looking at, without changing the agent’s online behavior. Computing Signals doesn't require a GPU. Signals are grouped into a simple taxonomy across interaction, execution, and environment patterns, including things like misalignment, stagnation, disengagement, failure, lo
[R] Looking for arXiv cs.LG endorser, inference monitoring using information geometry
Hi r/MachineLearning , I’m looking for an arXiv endorser in cs.LG for a paper on inference-time distribution shift detection for deployed LLMs. The core idea: instead of monitoring input embeddings (which is what existing tools do), we monitor the statistical manifold of the model’s output distributions using Fisher-Rao geodesic distance. We then run adaptive CUSUM (Page-Hinkley) on the resulting z-score stream to catch slow drift that per-request spike detection misses entirely. The methodology is grounded in published work on information geometry (Figshare, DOIs available). We’ve validated the signal on real OpenAI API logprobs, CUSUM caught gradual domain drift in 7 steps with zero false alarms during warmup, while spike detection missed it entirely. If anyone with cs.LG endorsement is
Discussion
Sign in to join the discussion
No comments yet — be the first to share your thoughts!