FastCache: Fast Caching for Diffusion Transformer Through Learnable Linear Approximation

View PDF HTML (experimental)

Abstract:Diffusion Transformers (DiT) are powerful generative models but remain computationally intensive due to their iterative structure and deep transformer stacks. To alleviate this inefficiency, we propose \textbf{FastCache}, a hidden-state-level caching and compression framework that accelerates DiT inference by exploiting redundancy within the model's internal representations. FastCache introduces a dual strategy: (1) a spatial-aware token selection mechanism that adaptively filters redundant tokens based on hidden-state saliency, and (2) a transformer-level cache that reuses latent activations across timesteps when changes fall below a predefined threshold. These modules work jointly to reduce unnecessary computation while preserving generation fidelity through learnable linear approximation. Theoretical analysis shows that FastCache maintains bounded approximation error under a hypothesis-testing-based decision rule. Empirical evaluations across multiple DiT variants demonstrate substantial reductions in latency and memory usage, achieving the best generation quality among existing cache methods, as measured by FID and t-FID. To further improve the speedup of FastCache, we also introduce a token merging module that merges redundant tokens based on k-NN density. Code is available at \href{this https URL}{this https URL}.

Subjects:

Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV); Multimedia (cs.MM); Performance (cs.PF)

Cite as: arXiv:2505.20353 [cs.LG]

(or arXiv:2505.20353v3 [cs.LG] for this version)

https://doi.org/10.48550/arXiv.2505.20353

arXiv-issued DOI via DataCite

Submission history

From: Dong Liu [view email] [v1] Mon, 26 May 2025 05:58:49 UTC (13,543 KB) [v2] Wed, 3 Sep 2025 06:56:21 UTC (9,229 KB) [v3] Fri, 27 Mar 2026 02:22:11 UTC (9,248 KB)