The Prism Hypothesis: Harmonizing Semantic and Pixel Representations via Unified Autoencoding

View PDF HTML (experimental)

Abstract:Deep representations across modalities are inherently intertwined. In this paper, we systematically analyze the spectral characteristics of various semantic and pixel encoders. Interestingly, our study uncovers a highly inspiring and rarely explored correspondence between an encoder's feature spectrum and its functional role: semantic encoders primarily capture low-frequency components that encode abstract meaning, whereas pixel encoders additionally retain high-frequency information that conveys fine-grained detail. This heuristic finding offers a unifying perspective that ties encoder behavior to its underlying spectral structure. We define it as the Prism Hypothesis, where each data modality can be viewed as a projection of the natural world onto a shared feature spectrum, just like the prism. Building on this insight, we propose Unified Autoencoding (UAE), a model that harmonizes semantic structure and pixel details via an innovative frequency-band modulator, enabling their seamless coexistence. Extensive experiments demonstrate that UAE effectively unifies semantic abstraction and pixel-level fidelity within a single latent space, achieving state-of-the-art performance. Moreover, we show that UAE can be directly applied to pixel-space modeling, significantly improving both FID and IS over the vanilla JIT baseline. Our code is avaliable at: this https URL.

Comments: Code link: this https URL

Subjects:

Computer Vision and Pattern Recognition (cs.CV)

Cite as: arXiv:2512.19693 [cs.CV]

(or arXiv:2512.19693v3 [cs.CV] for this version)

https://doi.org/10.48550/arXiv.2512.19693

arXiv-issued DOI via DataCite

Submission history

From: Weichen Fan [view email] [v1] Mon, 22 Dec 2025 18:59:57 UTC (2,410 KB) [v2] Fri, 23 Jan 2026 04:55:24 UTC (2,410 KB) [v3] Sun, 29 Mar 2026 13:58:03 UTC (2,274 KB)