Stability and Sensitivity Analysis of Relative Temporal-Difference Learning: Extended Version

View PDF HTML (experimental)

Abstract:Relative temporal-difference (TD) learning was introduced to mitigate the slow convergence of TD methods when the discount factor approaches one by subtracting a baseline from the temporal-difference update. While this idea has been studied in the tabular setting, stability guarantees with function approximation remain poorly understood. This paper analyzes relative TD learning with linear function approximation. We establish stability conditions for the algorithm and show that the choice of baseline distribution plays a central role. In particular, when the baseline is chosen as the empirical distribution of the state-action process, the algorithm is stable for any non-negative baseline weight and any discount factor. We also provide a sensitivity analysis of the resulting parameter estimates, characterizing both asymptotic bias and covariance. The asymptotic covariance and asymptotic bias are shown to remain uniformly bounded as the discount factor approaches one.

Comments: Extended version for submission to the 2026 IEEE CDC

Subjects:

Machine Learning (cs.LG); Optimization and Control (math.OC)

MSC classes: 68T05, 93E35, 62L20, 93E20

Cite as: arXiv:2603.27874 [cs.LG]

(or arXiv:2603.27874v1 [cs.LG] for this version)

https://doi.org/10.48550/arXiv.2603.27874

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Sean Meyn [view email] [v1] Sun, 29 Mar 2026 21:19:19 UTC (820 KB)