ALBA: A European Portuguese Benchmark for Evaluating Language and Linguistic Dimensions in Generative LLMs

View PDF HTML (experimental)

Abstract:As Large Language Models (LLMs) expand across multilingual domains, evaluating their performance in under-represented languages becomes increasingly important. European Portuguese (pt-PT) is particularly affected, as existing training data and benchmarks are mainly in Brazilian Portuguese (pt-BR). To address this, we introduce ALBA, a linguistically grounded benchmark designed from the ground up to assess LLM proficiency in linguistic-related tasks in pt-PT across eight linguistic dimensions, including Language Variety, Culture-bound Semantics, Discourse Analysis, Word Plays, Syntax, Morphology, Lexicology, and Phonetics and Phonology. ALBA is manually constructed by language experts and paired with an LLM-as-a-judge framework for scalable evaluation of pt-PT generated language. Experiments on a diverse set of models reveal performance variability across linguistic dimensions, highlighting the need for comprehensive, variety-sensitive benchmarks that support further development of tools in pt-PT.

Comments: PROPOR 2026 - The 17th International Conference on Computational Processing of Portuguese

Subjects:

Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)

ACM classes: I.2.7

Cite as: arXiv:2603.26516 [cs.CL]

(or arXiv:2603.26516v1 [cs.CL] for this version)

https://doi.org/10.48550/arXiv.2603.26516

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Rafael Ferreira [view email] [v1] Fri, 27 Mar 2026 15:25:44 UTC (854 KB)