Toward Evaluation Frameworks for Multi-Agent Scientific AI Systems

View PDF HTML (experimental)

Abstract:We analyze the challenges of benchmarking scientific (multi)-agentic systems, including the difficulty of distinguishing reasoning from retrieval, the risks of data/model contamination, the lack of reliable ground truth for novel research problems, the complications introduced by tool use, and the replication challenges due to the continuously changing/updating knowledge base. We discuss strategies for constructing contamination-resistant problems, generating scalable families of tasks, and the need for evaluating systems through multi-turn interactions that better reflect real scientific practice. As an early feasibility test, we demonstrate how to construct a dataset of novel research ideas to test the out-of-sample performance of our system. We also discuss the results of interviews with several researchers and engineers working in quantum science. Through those interviews, we examine how scientists expect to interact with AI systems and how these expectations should shape evaluation methods.

Comments: 13 pages, 3 figures

Subjects:

Computers and Society (cs.CY); Artificial Intelligence (cs.AI); Multiagent Systems (cs.MA); Quantum Physics (quant-ph)

Cite as: arXiv:2603.26718 [cs.CY]

(or arXiv:2603.26718v1 [cs.CY] for this version)

https://doi.org/10.48550/arXiv.2603.26718

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Marcin Abram [view email] [v1] Wed, 18 Mar 2026 16:05:52 UTC (401 KB)