Large Language Model as Token Compressor and Decompressor

View PDF HTML (experimental)

Abstract:In this paper, we establish the novel insight that an off-the-shelf LLM can function as an excellent token compressor and decompressor. To demonstrate, we design a self-expressive autoencoding learning framework fine-tunes a pretrained LLM to translate long texts into a compact internal language of discrete, variable-length latent codes, termed Z-tokens, and to reconstruct the original text exactly from them. The resulting representation is content-adaptive: semantically dense segments receive more Z-tokens, while redundant or predictable regions are aggressively compressed, via lightweight LoRA-based adapter heads. Empirically, our method achieves up to 18 times token reduction on Wikipedia, CNN/DailyMail, HotpotQA, and Qulac-style long-query datasets, while preserving reconstruction fidelity and downstream performance. This simple yet effective design supports applications including prompt compression and autoregressive generation directly in the Z-token space, offering a potential pathway toward token-efficient long-context reasoning.

Subjects:

Computation and Language (cs.CL)

Cite as: arXiv:2603.25340 [cs.CL]

(or arXiv:2603.25340v1 [cs.CL] for this version)

https://doi.org/10.48550/arXiv.2603.25340

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Wenbing Li None [view email] [v1] Thu, 26 Mar 2026 11:30:44 UTC (899 KB)