On-Policy Self-Distillation for Reasoning Compression

View PDF HTML (experimental)

Abstract:Reasoning models think out loud, but much of what they say is noise. We introduce OPSDC (On-Policy Self-Distillation for Reasoning Compression), a method that teaches models to reason more concisely by distilling their own concise behavior back into themselves. The entire approach reduces to one idea: condition the same model on a "be concise" instruction to obtain teacher logits, and minimize per-token reverse KL on the student's own rollouts. No ground-truth answers, no token budgets, no difficulty estimators. Just self-distillation. Yet this simplicity belies surprising sophistication: OPSDC automatically compresses easy problems aggressively while preserving the deliberation needed for hard ones. On Qwen3-8B and Qwen3-14B, we achieve 57-59% token reduction on MATH-500 while improving accuracy by 9-16 points absolute. On AIME 2024, the 14B model gains 10 points with 41% compression. The secret? Much of what reasoning models produce is not just redundant-it is actively harmful, compounding errors with every unnecessary token. Code is available at this https URL.

Subjects:

Machine Learning (cs.LG)

Cite as: arXiv:2603.05433 [cs.LG]

(or arXiv:2603.05433v4 [cs.LG] for this version)

https://doi.org/10.48550/arXiv.2603.05433

arXiv-issued DOI via DataCite

Submission history

From: Hejian Sang [view email] [v1] Thu, 5 Mar 2026 17:54:40 UTC (571 KB) [v2] Sun, 8 Mar 2026 06:29:26 UTC (570 KB) [v3] Tue, 17 Mar 2026 05:05:03 UTC (570 KB) [v4] Sat, 28 Mar 2026 03:56:28 UTC (591 KB)