Open ASR Leaderboard: Towards Reproducible and Transparent Multilingual and Long-Form Speech Recognition Evaluation

View PDF HTML (experimental)

Abstract:We present the Open ASR Leaderboard, a reproducible benchmarking platform with community contributions from academia and industry. It compares 86 open-source and proprietary systems across 12 datasets, with English short- and long-form and multilingual short-form tracks. We standardize word error rate (WER) and inverse real-time factor (RTFx) evaluation for consistent accuracy-efficiency comparisons across model architectures and toolkits (e.g., ESPNet, NeMo, SpeechBrain, Transformers). We observe that Conformer-based encoders paired with transformer-based decoders achieve the best average WER, while connectionist temporal classification (CTC) and token-and-duration transducer (TDT) decoders offer superior RTFx, making them better suited for long-form and batched processing. All code and dataset loaders are open-sourced to support transparent, extensible evaluation. We present our evaluation methodology to facilitate community-driven benchmarking in ASR and other tasks.

Comments: Leaderboard: this https URL ; Code: this https URL

Subjects:

Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Sound (cs.SD); Audio and Speech Processing (eess.AS)

Cite as: arXiv:2510.06961 [cs.CL]

(or arXiv:2510.06961v4 [cs.CL] for this version)

https://doi.org/10.48550/arXiv.2510.06961

arXiv-issued DOI via DataCite

Submission history

From: Eric Bezzam [view email] [v1] Wed, 8 Oct 2025 12:44:51 UTC (25 KB) [v2] Thu, 9 Oct 2025 07:39:28 UTC (25 KB) [v3] Wed, 10 Dec 2025 17:30:55 UTC (23 KB) [v4] Mon, 30 Mar 2026 09:52:05 UTC (2,783 KB)