TrackMAE: Video Representation Learning via Track Mask and Predict

View PDF HTML (experimental)

Abstract:Masked video modeling (MVM) has emerged as a simple and scalable self-supervised pretraining paradigm, but only encodes motion information implicitly, limiting the encoding of temporal dynamics in the learned representations. As a result, such models struggle on motion-centric tasks that require fine-grained motion awareness. To address this, we propose TrackMAE, a simple masked video modeling paradigm that explicitly uses motion information as a reconstruction signal. In TrackMAE, we use an off-the-shelf point tracker to sparsely track points in the input videos, generating motion trajectories. Furthermore, we exploit the extracted trajectories to improve random tube masking with a motion-aware masking strategy. We enhance video representations learned in both pixel and feature semantic reconstruction spaces by providing a complementary supervision signal in the form of motion targets. We evaluate on six datasets across diverse downstream settings and find that TrackMAE consistently outperforms state-of-the-art video self-supervised learning baselines, learning more discriminative and generalizable representations. Code available at this https URL

Comments: Accepted to CVPR 2026

Subjects:

Computer Vision and Pattern Recognition (cs.CV)

Cite as: arXiv:2603.27268 [cs.CV]

(or arXiv:2603.27268v1 [cs.CV] for this version)

https://doi.org/10.48550/arXiv.2603.27268

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Renaud Vandeghen [view email] [v1] Sat, 28 Mar 2026 13:35:23 UTC (4,130 KB)