Can Generalist Vision Language Models (VLMs) Rival Specialist Medical VLMs? Benchmarking and Strategic Insights

View PDF HTML (experimental)

Abstract:Vision Language Models (VLMs) have shown promise in automating image diagnosis and interpretation in clinical settings. However, developing specialist medical VLMs requires substantial computational resources and carefully curated datasets, and it remains unclear under which conditions generalist and specialist medical VLMs each perform best. This study highlights the complementary strengths of specialist medical and generalist VLMs. Specialists remain valuable in modality-aligned use cases, but we find that efficiently fine-tuned generalist VLMs can achieve comparable or even superior performance in most tasks, particularly when transferring to unseen or rare OOD medical modalities. These results suggest that generalist VLMs, rather than being constrained by their lack of specialist medical pretraining, may offer a scalable and cost-effective pathway for advancing clinical AI development.

Comments: version 3

Subjects:

Image and Video Processing (eess.IV); Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV)

Cite as: arXiv:2506.17337 [eess.IV]

(or arXiv:2506.17337v4 [eess.IV] for this version)

https://doi.org/10.48550/arXiv.2506.17337

arXiv-issued DOI via DataCite

Submission history

From: Ruinan Jin [view email] [v1] Thu, 19 Jun 2025 07:59:00 UTC (1,075 KB) [v2] Tue, 16 Sep 2025 07:44:30 UTC (3,771 KB) [v3] Sat, 21 Feb 2026 02:56:44 UTC (3,872 KB) [v4] Sun, 29 Mar 2026 17:53:47 UTC (4,418 KB)