Subspace Optimization for Backpropagation-Free Continual Test-Time Adaptation

View PDF HTML (experimental)

Abstract:We introduce PACE, a backpropagation-free continual test-time adaptation system that directly optimizes the affine parameters of normalization layers. Existing derivative-free approaches struggle to balance runtime efficiency with learning capacity, as they either restrict updates to input prompts or require continuous, resource-intensive adaptation regardless of domain stability. To address these limitations, PACE leverages the Covariance Matrix Adaptation Evolution Strategy with the Fastfood projection to optimize high-dimensional affine parameters within a low-dimensional subspace, leading to superior adaptive performance. Furthermore, we enhance the runtime efficiency by incorporating an adaptation stopping criterion and a domain-specialized vector bank to eliminate redundant computation. Our framework achieves state-of-the-art accuracy across multiple benchmarks under continual distribution shifts, reducing runtime by over 50% compared to existing backpropagation-free methods.

Subjects:

Machine Learning (cs.LG)

Cite as: arXiv:2603.28678 [cs.LG]

(or arXiv:2603.28678v1 [cs.LG] for this version)

https://doi.org/10.48550/arXiv.2603.28678

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Damian Sójka [view email] [v1] Mon, 30 Mar 2026 16:58:13 UTC (483 KB)