Posterior Optimization with Clipped Objective for Bridging Efficiency and Stability in Generative Policy Learning

View PDF HTML (experimental)

Abstract:Expressive generative models have advanced robotic manipulation by capturing complex, multi-modal action distributions over temporally extended trajectories. However, fine-tuning these policies via RL remains challenging due to instability and sample inefficiency. We introduce Posterior Optimization with Clipped Objective (POCO), a principled RL framework that formulates policy improvement as a posterior inference problem tailored for temporal action chunks. Through an Expectation-Maximization procedure, POCO distills a reward-weighted implicit posterior into the policy without likelihood estimation. Furthermore, POCO adopts an offline-to-online paradigm that anchors online exploration to pre-trained priors, and its model-agnostic design scales to fine-tune large VLA models without architectural modifications. Evaluations across 7 simulation benchmarks and 4 contact-rich real-world tasks demonstrate that POCO prevents catastrophic policy collapse, outperforms SOTA baselines, and achieves a 96.7% success rate on real-world tasks. Videos are available at our project website this https URL.

Subjects:

Robotics (cs.RO)

Cite as: arXiv:2604.01860 [cs.RO]

(or arXiv:2604.01860v1 [cs.RO] for this version)

https://doi.org/10.48550/arXiv.2604.01860

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Yuhui Chen [view email] [v1] Thu, 2 Apr 2026 10:15:47 UTC (11,831 KB)