CARPE: Context-Aware Image Representation Prioritization via Ensemble for Large Vision-Language Models

View PDF HTML (experimental)

Abstract:Large vision-language models (LVLMs) are typically trained using autoregressive language modeling objectives, which align visual representations with linguistic space. While effective for multimodal reasoning, this alignment can weaken vision-centric capabilities, causing LVLMs to underperform their base vision encoders on tasks such as image classification. To address this limitation, we propose Context-Aware Image Representation Prioritization via Ensemble (CARPE), a lightweight framework that integrates raw vision features with aligned LLM representations through vision-integration layers and a context-aware ensemble mechanism. This design enhances the model's ability to adaptively weight visual and textual modalities and enables the model to capture various aspects of image representations. Extensive experiments demonstrate that CARPE improves performance on both image classification and diverse vision-language benchmarks. Our results suggest that modality balancing plays a critical role in multimodal generalization by improving representation utilization within autoregressive LVLMs.

Subjects:

Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)

Cite as: arXiv:2601.13622 [cs.CV]

(or arXiv:2601.13622v3 [cs.CV] for this version)

https://doi.org/10.48550/arXiv.2601.13622

arXiv-issued DOI via DataCite

Submission history

From: Dong Hee Lee [view email] [v1] Tue, 20 Jan 2026 05:44:33 UTC (283 KB) [v2] Wed, 18 Mar 2026 04:41:36 UTC (278 KB) [v3] Thu, 26 Mar 2026 21:38:46 UTC (278 KB)