RetroAgent: From Solving to Evolving via Retrospective Dual Intrinsic Feedback

View PDF HTML (experimental)

Abstract:Standard reinforcement learning (RL) for large language model (LLM) agents typically optimizes extrinsic rewards, prioritizing isolated task completion over continual adaptation. Consequently, agents often converge to suboptimal policies due to limited exploration. Furthermore, accumulated experience remains implicitly trapped within model parameters, hindering its explicit reuse for guiding future decisions. Inspired by human retrospective self-improvement, we introduce RetroAgent, an online RL framework that trains agents to master complex interactive environments not only by solving tasks, but by evolving under the joint guidance of extrinsic task rewards and retrospective dual intrinsic feedback. Specifically, RetroAgent employs a hindsight self-reflection mechanism that generates two complementary signals: (1) intrinsic numerical feedback, which rewards promising exploration by tracking real-time incremental subtask progress relative to prior attempts; and (2) intrinsic language feedback, which enables explicit experience reuse by distilling reusable lessons into a memory buffer for subsequent decision-making. To effectively leverage these textual experiences, we propose Similarity & Utility-Aware Upper Confidence Bound (SimUtil-UCB), a retrieval strategy that balances relevance, historical utility, and exploration. Extensive experiments across four challenging agentic tasks show that RetroAgent achieves new state-of-the-art (SOTA) performance. Notably, it surpasses Group Relative Policy Optimization (GRPO) baselines by +18.3% on ALFWorld, +15.4% on WebShop, +27.1% on Sokoban, and +8.9% on MineSweeper, while exhibiting strong test-time adaptation and out-of-distribution generalization.

Comments: 48 pages, with fixed typos

Subjects:

Artificial Intelligence (cs.AI)

Cite as: arXiv:2603.08561 [cs.AI]

(or arXiv:2603.08561v5 [cs.AI] for this version)

https://doi.org/10.48550/arXiv.2603.08561

arXiv-issued DOI via DataCite

Submission history

From: Xiaoying Zhang [view email] [v1] Mon, 9 Mar 2026 16:23:33 UTC (3,850 KB) [v2] Wed, 11 Mar 2026 12:33:30 UTC (3,850 KB) [v3] Thu, 12 Mar 2026 11:31:58 UTC (3,851 KB) [v4] Thu, 26 Mar 2026 14:07:56 UTC (3,871 KB) [v5] Sat, 28 Mar 2026 15:39:47 UTC (3,872 KB)