Out-of-Sight Embodied Agents: Multimodal Tracking, Sensor Fusion, and Trajectory Forecasting

View PDF HTML (experimental)

Abstract:Trajectory prediction is a fundamental problem in computer vision, vision-language-action models, world models, and autonomous systems, with broad impact on autonomous driving, robotics, and surveillance. However, most existing methods assume complete and clean observations, and therefore do not adequately handle out-of-sight agents or noisy sensing signals caused by limited camera coverage, occlusions, and the absence of ground-truth denoised trajectories. These challenges raise safety concerns and reduce robustness in real-world deployment. In this extended study, we introduce major improvements to Out-of-Sight Trajectory (OST), a task for predicting noise-free visual trajectories of out-of-sight objects from noisy sensor observations. Building on our prior work, we expand Out-of-Sight Trajectory Prediction (OOSTraj) from pedestrians to both pedestrians and vehicles, increasing its relevance to autonomous driving, robotics, and surveillance. Our improved Vision-Positioning Denoising Module exploits camera calibration to establish vision-position correspondence, mitigating the lack of direct visual cues and enabling effective unsupervised denoising of noisy sensor signals. Extensive experiments on the Vi-Fi and JRDB datasets show that our method achieves state-of-the-art results for both trajectory denoising and trajectory prediction, with clear gains over prior baselines. We also compare with classical denoising methods, including Kalman filtering, and adapt recent trajectory prediction models to this setting, establishing a stronger benchmark. To the best of our knowledge, this is the first work to use vision-positioning projection to denoise noisy sensor trajectories of out-of-sight agents, opening new directions for future research.

Comments: Published in IEEE Transactions on Pattern Analysis and Machine Intelligence (Early Access), pp. 1-14, March 23, 2026

Subjects:

Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG); Multiagent Systems (cs.MA); Multimedia (cs.MM); Robotics (cs.RO)

MSC classes: 68T45, 68U10, 68T07, 68T40, 93C85, 93E11, 62M20, 62M10, 68U05, 94A12

ACM classes: F.2.2; I.2.9; I.2.10; I.4.1; I.4.8; I.4.9; I.5.4; I.3.7

Cite as: arXiv:2509.15219 [cs.CV]

(or arXiv:2509.15219v2 [cs.CV] for this version)

https://doi.org/10.48550/arXiv.2509.15219

arXiv-issued DOI via DataCite

Journal reference: IEEE Transactions on Pattern Analysis and Machine Intelligence, 2026

Related DOI:

https://doi.org/10.1109/TPAMI.2026.3676710

DOI(s) linking to related resources

Submission history

From: Haichao Zhang [view email] [v1] Thu, 18 Sep 2025 17:59:16 UTC (3,235 KB) [v2] Thu, 26 Mar 2026 21:31:12 UTC (3,634 KB)