ORION: ORthonormal Text Encoding for Universal VLM AdaptatION

View PDF HTML (experimental)

Abstract:Vision language models (VLMs) have demonstrated remarkable generalization across diverse tasks, yet their performance remains constrained by the quality and geometry of the textual prototypes used to represent classes. Standard zero shot classifiers, derived from frozen text encoders and handcrafted prompts, may yield correlated or weakly separated embeddings that limit task specific discriminability. We introduce ORION, a text encoder fine tuning framework that improves pretrained VLMs using only class names. Our method optimizes, via low rank adaptation, a novel loss integrating two terms, one promoting pairwise orthogonality between the textual representations of the classes of a given task and the other penalizing deviations from the initial class prototypes. Furthermore, we provide a probabilistic interpretation of our orthogonality penalty, connecting it to the general maximum likelihood estimation (MLE) principle via Huygens theorem. We report extensive experiments on 11 benchmarks and three large VLM backbones, showing that the refined textual embeddings yield powerful replacements for the standard CLIP prototypes. Added as plug and play module on top of various state of the art methods, and across different prediction settings (zero shot, few shot and test time adaptation), ORION improves the performance consistently and significantly.

Subjects:

Computer Vision and Pattern Recognition (cs.CV)

Cite as: arXiv:2602.19530 [cs.CV]

(or arXiv:2602.19530v2 [cs.CV] for this version)

https://doi.org/10.48550/arXiv.2602.19530

arXiv-issued DOI via DataCite

Journal reference: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2026

Submission history

From: Omprakash Chakraborty [view email] [v1] Mon, 23 Feb 2026 05:47:28 UTC (265 KB) [v2] Fri, 27 Mar 2026 15:33:37 UTC (265 KB)