PokeFusion Attention: A Lightweight Cross-Attention Mechanism for Style-Conditioned Image Generation

View PDF HTML (experimental)

Abstract:Style-conditioned text-to-image (T2I) generation with diffusion models requires both stable character structure and consistent, fine-grained style expression across diverse prompts. Existing approaches either rely on text-only prompting, which is often insufficient to specify visual style, or introduce reference-based adapters that depend on external images at inference time, increasing system complexity and limiting deployment flexibility. We propose PokeFusion Attention, a lightweight decoder-level cross-attention mechanism that models style as a learned distributional prior rather than instance-level conditioning. The method integrates textual semantics with learned style embeddings directly within the diffusion decoder, enabling effective stylized generation without requiring reference images at inference time. Only the cross-attention layers and a compact style projection module are trained, while the pretrained diffusion backbone remains frozen, resulting in a parameter-efficient and plug-and-play design. Experiments on a stylized character generation benchmark demonstrate that the proposed method improves style fidelity, semantic alignment, and structural consistency compared with representative adapter-based baselines, while maintaining low parameter overhead and simple inference.

Comments: 12 pages, 5 figures. Revised version with improved method description and corrected references

Subjects:

Computer Vision and Pattern Recognition (cs.CV)

Cite as: arXiv:2602.03220 [cs.CV]

(or arXiv:2602.03220v3 [cs.CV] for this version)

https://doi.org/10.48550/arXiv.2602.03220

arXiv-issued DOI via DataCite

Submission history

From: Jingbang Tang [view email] [v1] Tue, 3 Feb 2026 07:44:01 UTC (15,978 KB) [v2] Wed, 25 Mar 2026 21:42:41 UTC (1 KB) (withdrawn) [v3] Fri, 27 Mar 2026 05:14:13 UTC (15,649 KB)