Rainbow-DemoRL: Combining Improvements in Demonstration-Augmented Reinforcement Learning

View PDF HTML (experimental)

Abstract:Several approaches have been proposed to improve the sample efficiency of online reinforcement learning (RL) by leveraging demonstrations collected offline. The offline data can be used directly as transitions to optimize RL objectives, or offline policy and value functions can first be learned from the data and then used for online finetuning or to provide reference actions. While each of these strategies has shown compelling results, it is unclear which method has the most impact on sample efficiency, whether these approaches can be combined, and if there are cumulative benefits. We classify existing demonstration-augmented RL approaches into three categories and perform an extensive empirical study of their strengths, weaknesses, and combinations to isolate the contribution of each strategy and determine effective hybrid combinations for sample-efficient online RL. Our analysis reveals that directly reusing offline data and initializing with behavior cloning consistently outperform more complex offline RL pretraining methods for improving online sample efficiency.

Comments: Accepted to ICRA 2026

Subjects:

Robotics (cs.RO); Machine Learning (cs.LG)

Cite as: arXiv:2603.27400 [cs.RO]

(or arXiv:2603.27400v1 [cs.RO] for this version)

https://doi.org/10.48550/arXiv.2603.27400

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Dwait Bhatt [view email] [v1] Sat, 28 Mar 2026 20:34:20 UTC (397 KB)