Automated Algorithm Design for Auto-Tuning Optimizers

View PDF HTML (experimental)

Abstract:Automatic performance tuning (auto-tuning) is essential for optimizing high-performance applications, where vast and irregular search spaces make manual exploration infeasible. While auto-tuners traditionally rely on classical approaches such as evolutionary, annealing, or surrogate-based optimizers, designing algorithms that efficiently find near-optimal configurations robustly across diverse tasks is challenging. We propose a new paradigm: using large language models (LLMs) to automatically generate optimization algorithms tailored to auto-tuning problems. We introduce a framework that prompts LLMs with problem descriptions and search space characteristics to synthesize, test, and iteratively refine specialized optimizers. These generated algorithms are evaluated on four real-world auto-tuning applications across six hardware platforms and compared against the state-of-the-art in two contemporary auto-tuning frameworks. The evaluation demonstrates that providing additional application- and search space-specific information in the generation stage results in an average performance improvement of 30.7% and 14.6%, respectively. In addition, our results show that LLM-generated optimizers can rival, and in various cases outperform, existing human-designed algorithms, with our best-performing generated optimization algorithms achieving an average 72.4% improvement over state-of-the-art optimizers for auto-tuning.

Subjects:

Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Neural and Evolutionary Computing (cs.NE)

Cite as: arXiv:2510.17899 [cs.LG]

(or arXiv:2510.17899v2 [cs.LG] for this version)

https://doi.org/10.48550/arXiv.2510.17899

arXiv-issued DOI via DataCite

Submission history

From: Floris-Jan Willemsen [view email] [v1] Sun, 19 Oct 2025 09:38:15 UTC (2,694 KB) [v2] Tue, 31 Mar 2026 10:10:10 UTC (2,670 KB)