Temporal Credit Is Free

View PDF HTML (experimental)

Abstract:Recurrent networks do not need Jacobian propagation to adapt online. The hidden state already carries temporal credit through the forward pass; immediate derivatives suffice if you stop corrupting them with stale trace memory and normalize gradient scales across parameter groups. An architectural rule predicts when normalization is needed: \b{eta}2 is required when gradients must pass through a nonlinear state update with no output bypass, and unnecessary otherwise. Across ten architectures, real primate neural data, and streaming ML benchmarks, immediate derivatives with RMSprop match or exceed full RTRL, scaling to n = 1024 at 1000x less memory.

Comments: 16 pages, 4 figures, 5 tables

Subjects:

Machine Learning (cs.LG)

Cite as: arXiv:2603.28750 [cs.LG]

(or arXiv:2603.28750v1 [cs.LG] for this version)

https://doi.org/10.48550/arXiv.2603.28750

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Aur Shalev Merin [view email] [v1] Mon, 30 Mar 2026 17:54:55 UTC (55 KB)