Active In-Context Learning for Tabular Foundation Models

View PDF HTML (experimental)

Abstract:Active learning (AL) reduces labeling cost by querying informative samples, but in tabular settings its cold-start gains are often limited because uncertainty estimates are unreliable when models are trained on very few labels. Tabular foundation models such as TabPFN provide calibrated probabilistic predictions via in-context learning (ICL), i.e., without task-specific weight updates, enabling an AL regime in which the labeled context - rather than parameters - is iteratively optimized. We formalize Tabular Active In-Context Learning (Tab-AICL) and instantiate it with four acquisition rules: uncertainty (TabPFN-Margin), diversity (TabPFN-Coreset), an uncertainty-diversity hybrid (TabPFN-Hybrid), and a scalable two-stage method (TabPFN-Proxy-Hybrid) that shortlists candidates using a lightweight linear proxy before TabPFN-based selection. Across 20 classification benchmarks, Tab-AICL improves cold-start sample efficiency over retrained gradient-boosting baselines (CatBoost-Margin and XGBoost-Margin), measured by normalized AULC up to 100 labeled samples.

Comments: 8 pages, 4 figures, 6 tables

Subjects:

Machine Learning (cs.LG)

Cite as: arXiv:2603.27385 [cs.LG]

(or arXiv:2603.27385v1 [cs.LG] for this version)

https://doi.org/10.48550/arXiv.2603.27385

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Fabrizio Pittorino [view email] [v1] Sat, 28 Mar 2026 19:37:13 UTC (322 KB)