Steering Sparse Autoencoder Latents to Control Dynamic Head Pruning in Vision Transformers (Student Abstract)

View PDF HTML (experimental)

Abstract:Dynamic head pruning in Vision Transformers (ViTs) improves efficiency by removing redundant attention heads, but existing pruning policies are often difficult to interpret and control. In this work, we propose a novel framework by integrating Sparse Autoencoders (SAEs) with dynamic pruning, leveraging their ability to disentangle dense embeddings into interpretable and controllable sparse latents. Specifically, we train an SAE on the final-layer residual embedding of the ViT and amplify the sparse latents with different strategies to alter pruning decisions. Among them, per-class steering reveals compact, class-specific head subsets that preserve accuracy. For example, bowl improves accuracy (76% to 82%) while reducing head usage (0.72 to 0.33) via heads h2 and h5. These results show that sparse latent features enable class-specific control of dynamic pruning, effectively bridging pruning efficiency and mechanistic interpretability in ViTs.

Comments: 3 pages, 5 figures. Accepted as AAAI 2026 Student Abstract. Includes additional appendix with extended analysis

Subjects:

Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)

Cite as: arXiv:2603.26743 [cs.CV]

(or arXiv:2603.26743v1 [cs.CV] for this version)

https://doi.org/10.48550/arXiv.2603.26743

arXiv-issued DOI via DataCite (pending registration)

Journal reference: Proceedings of the AAAI Conference on Artificial Intelligence (AAAI 2026), Vol. 40, No. 48, pp. 41263-41265

Related DOI:

https://doi.org/10.1609/aaai.v40i48.42236

DOI(s) linking to related resources

Submission history

From: Yousung Lee [view email] [v1] Mon, 23 Mar 2026 07:08:19 UTC (3,632 KB)