AnthroTAP: Learning Point Tracking with Real-World Motion

View PDF HTML (experimental)

Abstract:Point tracking models often struggle to generalize to real-world videos because large-scale training data is predominantly synthetic$\unicode{x2014}$the only source currently feasible to produce at scale. Collecting real-world annotations, however, is prohibitively expensive, as it requires tracking hundreds of points across frames. We introduce \textbf{AnthroTAP}, an automated pipeline that generates large-scale pseudo-labeled point tracking data from real human motion videos. Leveraging the structured complexity of human movement$\unicode{x2014}$non-rigid deformations, articulated motion, and frequent occlusions$\unicode{x2014}$AnthroTAP fits Skinned Multi-Person Linear (SMPL) models to detected humans, projects mesh vertices onto image planes, resolves occlusions via ray-casting, and filters unreliable tracks using optical flow consistency. A model trained on the AnthroTAP dataset achieves state-of-the-art performance on TAP-Vid, a challenging general-domain benchmark for tracking any point on diverse rigid and non-rigid objects (e.g., humans, animals, robots, and vehicles). Our approach outperforms recent self-training methods trained on vastly larger real datasets, while requiring only one day of training on 4 GPUs. AnthroTAP shows that structured human motion offers a scalable and effective source of real-world supervision for point tracking.

Comments: CVPR 2026. Project Page: this https URL

Subjects:

Computer Vision and Pattern Recognition (cs.CV)

Cite as: arXiv:2507.06233 [cs.CV]

(or arXiv:2507.06233v3 [cs.CV] for this version)

https://doi.org/10.48550/arXiv.2507.06233

arXiv-issued DOI via DataCite

Submission history

From: Seokju Cho [view email] [v1] Tue, 8 Jul 2025 17:59:58 UTC (11,138 KB) [v2] Sat, 13 Dec 2025 08:10:08 UTC (11,140 KB) [v3] Mon, 30 Mar 2026 05:17:49 UTC (11,648 KB)