Heddle: A Distributed Orchestration System for Agentic RL Rollout

View PDF HTML (experimental)

Abstract:Agentic Reinforcement Learning (RL) enables LLMs to solve complex tasks by alternating between a data-collection rollout phase and a policy training phase. During rollout, the agent generates trajectories, i.e., multi-step interactions between LLMs and external tools. Yet, frequent tool calls induce long-tailed trajectory generation that bottlenecks rollouts. This stems from step-centric designs that ignore trajectory context, triggering three system problems for long-tail trajectory generation: queueing delays, interference overhead, and inflated per-token time. We propose Heddle, a trajectory-centric system to optimize the when, where, and how of agentic rollout execution. Heddle integrates three core mechanisms: trajectory-level scheduling using runtime prediction and progressive priority to minimize cumulative queueing; trajectory-aware placement via presorted dynamic programming and opportunistic migration during idle tool call intervals to minimize interference; and trajectory-adaptive resource manager that dynamically tunes model parallelism to accelerate the per-token time of long-tail trajectories while maintaining high throughput for short trajectories. Evaluations across diverse agentic RL workloads demonstrate that Heddle effectively neutralizes the long-tail bottleneck, achieving up to 2.5$\times$ higher end-to-end rollout throughput compared to state-of-the-art baselines.

Subjects:

Machine Learning (cs.LG)

Cite as: arXiv:2603.28101 [cs.LG]

(or arXiv:2603.28101v1 [cs.LG] for this version)

https://doi.org/10.48550/arXiv.2603.28101

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Zili Zhang [view email] [v1] Mon, 30 Mar 2026 07:01:32 UTC (9,443 KB)