CirrusBench: Evaluating LLM-based Agents Beyond Correctness in Real-World Cloud Service Environments

Authors:Yi Yu, Guangquan Hu, Chenghuang Shen, Xingyan Liu, Jing Gu, Hangyi Sun, Junzhuo Ma, Weiting Liu, Jianfeng Liu, Mingyue Pu, Yu Wang, Zhengdong Xiao, Rui Xie, Longjiu Luo, Qianrong Wang, Gurong Cui, Honglin Qiao, Wenlian Lu

View PDF HTML (experimental)

Abstract:The increasing agentic capabilities of Large Language Models (LLMs) have enabled their deployment in real-world applications, such as cloud services, where customer-assistant interactions exhibit high technical complexity and long-horizon dependencies, making robustness and resolution efficiency critical for customer satisfaction. However, existing benchmarks for LLM-based agents largely rely on synthetic environments that fail to capture the diversity and unpredictability of authentic customer inputs, often ignoring the resolution efficiency essential for real-world deployment. To bridge this gap, we introduce CirrusBench, a novel evaluation framework distinguished by its foundation in real-world data from authentic cloud service tickets. CirrusBench preserves the intricate multi-turn logical chains and realistic tool dependencies inherent to technical service environments. Moving beyond execution correctness, we introduce novel Customer-Centric metrics to define agent success, quantifying service quality through metrics such as the Normalized Efficiency Index and Multi-Turn Latency to explicitly measure resolution efficiency. Experiments utilizing our framework reveal that while state-of-the-art models demonstrate strong reasoning capabilities, they frequently struggle in complex, realistic multi-turn tasks and fail to meet the high-efficiency standards required for customer service, highlighting critical directions for the future development of LLM-based agents in practical technical service applications. CirrusBench evaluation framework is released at: this https URL

Comments: Submitted for SIGKDD 2026

Subjects:

Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Information Retrieval (cs.IR); Performance (cs.PF)

Cite as: arXiv:2603.28569 [cs.LG]

(or arXiv:2603.28569v1 [cs.LG] for this version)

https://doi.org/10.48550/arXiv.2603.28569

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Yu Yi [view email] [v1] Mon, 30 Mar 2026 15:26:00 UTC (1,216 KB)