Unsupervised Evaluation of Deep Audio Embeddings for Music Structure Analysis

View PDF HTML (experimental)

Abstract:Music Structure Analysis (MSA) aims to uncover the high-level organization of musical pieces. State-of-the-art methods are often based on supervised deep learning, but these methods are bottlenecked by the need for heavily annotated data and inherent structural ambiguities. In this paper, we propose an unsupervised evaluation of nine open-source, generic pre-trained deep audio models, on MSA. For each model, we extract barwise embeddings and segment them using three unsupervised segmentation algorithms (Foote's checkerboard kernels, spectral clustering, and Correlation Block-Matching (CBM)), focusing exclusively on boundary retrieval. Our results demonstrate that modern, generic deep embeddings generally outperform traditional spectrogram-based baselines, but not systematically. Furthermore, our unsupervised boundary estimation methodology generally yields stronger performance than recent linear probing baselines. Among the evaluated techniques, the CBM algorithm consistently emerges as the most effective downstream segmentation method. Finally, we highlight the artificial inflation of standard evaluation metrics and advocate for the systematic adoption of trimming'', or even double trimming'' annotations to establish more rigorous MSA evaluation standards.

Comments: Submitted to the SMC 2026 conference. 2 figures and 2 tables in the main document, 7 figures in Appendix

Subjects:

Sound (cs.SD); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)

ACM classes: H.5.5

Cite as: arXiv:2603.27218 [cs.SD]

(or arXiv:2603.27218v1 [cs.SD] for this version)

https://doi.org/10.48550/arXiv.2603.27218

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Axel Marmoret [view email] [v1] Sat, 28 Mar 2026 10:18:54 UTC (200 KB)