On the Role of Encoder Depth: Pruning Whisper and LoRA Fine-Tuning in SLAM-ASR

View PDF HTML (experimental)

Abstract:Automatic speech recognition (ASR) has advanced rapidly in recent years, driven by large-scale pretrained models and end-to-end architectures such as SLAM-ASR. A key component of SLAM-ASR systems is the Whisper speech encoder, which provides robust acoustic representations. While model pruning has been explored for the full Whisper encoder-decoder architecture, its impact within the SLAM-ASR setting remains under-investigated. In this work, we analyze the effects of layer pruning in the Whisper encoder when used as the acoustic backbone of SLAM-ASR. We further examine the extent to which LoRA-based fine-tuning can recover performance degradation caused by pruning. Experiments conducted across three Whisper variants (Small, Medium, Large-v2), three languages representing distinct resource levels (Danish, Dutch, English), and over 200 training runs demonstrate that pruning two encoder layers causes only 2-4% WER degradation, and that combining this pruning with LoRA adaptation consistently outperforms the unpruned baseline while reducing total parameters by 7-14%. Moreover, our error analysis reveals that LoRA primarily compensates through the language model's linguistic priors, reducing total word errors by 11-21% for Dutch and English, with substitutions and deletions showing the largest reductions. However, for low-resource Danish, the reduction is smaller (4-7%), and LoRA introduces increased insertion errors, indicating that compensation effectiveness depends on the LLM's pre-existing language proficiency and available training data.

Comments: Accepted at SPEAKABLE Workshop, LREC 2026

Subjects:

Computation and Language (cs.CL); Sound (cs.SD)

Cite as: arXiv:2603.27981 [cs.CL]

(or arXiv:2603.27981v1 [cs.CL] for this version)

https://doi.org/10.48550/arXiv.2603.27981

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Ravi Shekhar [view email] [v1] Mon, 30 Mar 2026 03:02:22 UTC (60 KB)