HumMusQA: A Human-written Music Understanding QA Benchmark Dataset

View PDF HTML (experimental)

Abstract:The evaluation of music understanding in Large Audio-Language Models (LALMs) requires a rigorously defined benchmark that truly tests whether models can perceive and interpret music, a standard that current data methodologies frequently fail to meet. This paper introduces a meticulously structured approach to music evaluation, proposing a new dataset of 320 hand-written questions curated and validated by experts with musical training, arguing that such focused, manual curation is superior for probing complex audio comprehension. To demonstrate the use of the dataset, we benchmark six state-of-the-art LALMs and additionally test their robustness to uni-modal shortcuts.

Comments: Dataset available at this https URL

Subjects:

Computation and Language (cs.CL); Sound (cs.SD)

Cite as: arXiv:2603.27877 [cs.CL]

(or arXiv:2603.27877v1 [cs.CL] for this version)

https://doi.org/10.48550/arXiv.2603.27877

arXiv-issued DOI via DataCite (pending registration)

Journal reference: Proceedings of the 4th Workshop on NLP for Music and Audio (NLP4MusA 2026), pages 58-67, Rabat, Morocco. Association for Computational Linguistics

Related DOI:

https://doi.org/10.18653/v1/2026.nlp4musa-1.9

DOI(s) linking to related resources

Submission history

From: Benno Weck [view email] [v1] Sun, 29 Mar 2026 21:33:07 UTC (815 KB)