LACE: Loss-Adaptive Capacity Expansion for Continual Learning

View PDF HTML (experimental)

Abstract:Fixed representational capacity is a fundamental constraint in continual learning: practitioners must guess an appropriate model width before training, without knowing how many distinct concepts the data contains. We propose LACE (Loss-Adaptive Capacity Expansion), a simple online mechanism that expands a model's representational capacity during training by monitoring its own loss signal. When sustained loss deviation exceeds a threshold - indicating that the current capacity is insufficient for newly encountered data - LACE adds new dimensions to the projection layer and trains them jointly with existing parameters. Across synthetic and real-data experiments, LACE triggers expansions exclusively at domain boundaries (100% boundary precision, zero false positives), matches the accuracy of a large fixed-capacity model while starting from a fraction of its dimensions, and produces adapter dimensions that are collectively critical to performance (3% accuracy drop when all adapters removed). We further demonstrate unsupervised domain separation in GPT-2 activations via layer-wise clustering, showing a U-shaped separability curve across layers that motivates adaptive capacity allocation in deep networks. LACE requires no labels, no replay buffers, and no external controllers, making it suitable for on-device continual learning under resource constraints.

Subjects:

Machine Learning (cs.LG)

Cite as: arXiv:2603.28611 [cs.LG]

(or arXiv:2603.28611v1 [cs.LG] for this version)

https://doi.org/10.48550/arXiv.2603.28611

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Shivnath Tathe Tathe [view email] [v1] Mon, 30 Mar 2026 15:58:33 UTC (1,792 KB)