Diversity-Aware Reverse Kullback-Leibler Divergence for Large Language Model Distillation

View PDF HTML (experimental)

Abstract:Reverse Kullback-Leibler (RKL) divergence has recently emerged as the preferred objective for large language model (LLM) distillation, consistently outperforming forward KL (FKL), particularly in regimes with large vocabularies and significant teacher-student capacity mismatch, where RKL focuses learning on dominant modes rather than enforcing dense alignment. However, RKL introduces a structural limitation that drives the student toward overconfident predictions. We first provide an analysis of RKL by decomposing its gradients into target and non-target components, and show that non-target gradients consistently push the target logit upward even when the student already matches the teacher, thereby reducing output diversity. In addition, RKL provides weak supervision over non-target classes, leading to poor tail alignment. To address these issues, we propose Diversity-aware RKL (DRKL), which removes this gradient effect and strengthens non-target supervision while preserving the optimization benefits of RKL. Extensive experiments across datasets and model families demonstrate that DRKL consistently outperforms FKL, RKL, and other state-of-the-art distillation objectives, achieving better performance and a superior fidelity-diversity trade-off.

Subjects:

Machine Learning (cs.LG); Artificial Intelligence (cs.AI)

Cite as: arXiv:2604.00223 [cs.LG]

(or arXiv:2604.00223v1 [cs.LG] for this version)

https://doi.org/10.48550/arXiv.2604.00223

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Hoang-Chau Luong [view email] [v1] Tue, 31 Mar 2026 20:39:47 UTC (886 KB)