HEAD-QA v2: Expanding a Healthcare Benchmark for Reasoning

View PDF HTML (experimental)

Abstract:We introduce HEAD-QA v2, an expanded and updated version of a Spanish/English healthcare multiple-choice reasoning dataset originally released by Vilares and Gómez-Rodríguez (2019). The update responds to the growing need for high-quality datasets that capture the linguistic and conceptual complexity of healthcare reasoning. We extend the dataset to over 12,000 questions from ten years of Spanish professional exams, benchmark several open-source LLMs using prompting, RAG, and probability-based answer selection, and provide additional multilingual versions to support future work. Results indicate that performance is mainly driven by model scale and intrinsic reasoning ability, with complex inference strategies obtaining limited gains. Together, these results establish HEAD-QA v2 as a reliable resource for advancing research on biomedical reasoning and model improvement.

Comments: LREC 2026 camera-ready version

Subjects:

Computation and Language (cs.CL)

Cite as: arXiv:2511.15355 [cs.CL]

(or arXiv:2511.15355v2 [cs.CL] for this version)

https://doi.org/10.48550/arXiv.2511.15355

arXiv-issued DOI via DataCite

Submission history

From: David Vilares [view email] [v1] Wed, 19 Nov 2025 11:31:32 UTC (705 KB) [v2] Mon, 30 Mar 2026 08:04:09 UTC (705 KB)