ViPRA: Video Prediction for Robot Actions

View PDF

Abstract:Can we turn a video prediction model into a robot policy? Videos, including those of humans or teleoperated robots, capture rich physical interactions. However, most of them lack labeled actions, which limits their use in robot learning. We present Video Prediction for Robot Actions (ViPRA), a simple pretraining-finetuning framework that learns continuous robot control from these actionless videos. Instead of directly predicting actions, we train a video-language model to predict both future visual observations and motion-centric latent actions, which serve as intermediate representations of scene dynamics. We train these latent actions using perceptual losses and optical flow consistency to ensure they reflect physically grounded behavior. For downstream control, we introduce a chunked flow matching decoder that maps latent actions to robot-specific continuous action sequences, using only 100 to 200 teleoperated demonstrations. This approach avoids expensive action annotation, supports generalization across embodiments, and enables smooth, high-frequency continuous control upto 22 Hz via chunked action decoding. Unlike prior latent action works that treat pretraining as autoregressive policy learning, ViPRA explicitly models both what changes and how. Our method outperforms strong baselines, with a 16% gain on the SIMPLER benchmark and a 13% improvement across real world manipulation tasks. We have released models and code at this https URL

Comments: In ICLR 2026. Website: this https URL

Subjects:

Robotics (cs.RO); Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG)

Cite as: arXiv:2511.07732 [cs.RO]

(or arXiv:2511.07732v2 [cs.RO] for this version)

https://doi.org/10.48550/arXiv.2511.07732

arXiv-issued DOI via DataCite

Submission history

From: Sandeep Routray [view email] [v1] Tue, 11 Nov 2025 01:33:03 UTC (6,701 KB) [v2] Mon, 30 Mar 2026 17:59:36 UTC (6,793 KB)