Revisiting 2D Foundation Models for Scalable 3D Medical Image Classification

Authors:Han Liu, Bogdan Georgescu, Yanbo Zhang, Youngjin Yoo, Michael Baumgartner, Riqiang Gao, Jianing Wang, Gengyan Zhao, Eli Gibson, Dorin Comaniciu, Sasa Grbic

View PDF HTML (experimental)

Abstract:3D medical image classification is essential for modern clinical workflows. Medical foundation models (FMs) have emerged as a promising approach for scaling to new tasks, yet current research suffers from three critical pitfalls: data-regime bias, suboptimal adaptation, and insufficient task coverage. In this paper, we address these pitfalls and introduce AnyMC3D, a scalable 3D classifier adapted from 2D FMs. Our method scales efficiently to new tasks by adding only lightweight plugins (about 1M parameters per task) on top of a single frozen backbone. This versatile framework also supports multi-view inputs, auxiliary pixel-level supervision, and interpretable heatmap generation. We establish a comprehensive benchmark of 12 tasks covering diverse pathologies, anatomies, and modalities, and systematically analyze state-of-the-art 3D classification techniques. Our analysis reveals key insights: (1) effective adaptation is essential to unlock FM potential, (2) general-purpose FMs can match medical-specific FMs if properly adapted, and (3) 2D-based methods surpass 3D architectures for 3D classification. For the first time, we demonstrate the feasibility of achieving state-of-the-art performance across diverse applications using a single scalable framework (including 1st place in the VLM3D challenge), eliminating the need for separate task-specific models.

Comments: 1st Place in VLM3D Challenge

Subjects:

Computer Vision and Pattern Recognition (cs.CV)

Cite as: arXiv:2512.12887 [cs.CV]

(or arXiv:2512.12887v2 [cs.CV] for this version)

https://doi.org/10.48550/arXiv.2512.12887

arXiv-issued DOI via DataCite

Submission history

From: Han Liu [view email] [v1] Mon, 15 Dec 2025 00:01:19 UTC (17,970 KB) [v2] Fri, 27 Mar 2026 04:52:26 UTC (4,677 KB)