What Is Your Agent's GPA? A Framework for Evaluating Agent Goal-Plan-Action Alignment

View PDF HTML (experimental)

Abstract:We introduce the Agent GPA (Goal-Plan-Action) framework, driven by the fundamental insight that critical agent failures emerge at the intersections of setting goals, devising plans, and executing actions. We operationalize the framework with a factorized suite of LLM judges designed to measure distinct elements of Goal-Plan-Act alignment. To make this methodology scalable and generalizable across diverse agent architectures and datasets, we use state-of-the-art automated prompt optimization techniques to systematically generate domain-specific evaluation criteria. We validate this approach across three benchmarks: a multi-agent research setting (TRAIL/GAIA), a single coding agent setting (TRAIL/SWE-bench), and a private, enterprise data-agent setting (Snowflake Intelligence). Extensive evaluation on TRAIL/GAIA demonstrates the core validity of the framework, which identifies a broad range of agent failures (95% of human-annotated errors), localizes errors to enable targeted debugging (86% of human-annotated errors), and exhibits strong agreement with human evaluators. Crucially, by applying our automated methodology to both public datasets, we demonstrate that our GPA judges generally achieve the highest error coverage (ranging from 76% to 86%) in comparison to manual prompting approaches. We also leverage an evolutionary coding agent to improve judge consistency by up to 38% through iterative refinement of evaluation rubrics. Overall, Agent GPA provides a rigorous and generalizable paradigm for targeted agent evaluation.

Subjects:

Artificial Intelligence (cs.AI); Multiagent Systems (cs.MA)

Cite as: arXiv:2510.08847 [cs.AI]

(or arXiv:2510.08847v2 [cs.AI] for this version)

https://doi.org/10.48550/arXiv.2510.08847

arXiv-issued DOI via DataCite

Submission history

From: Allison Jia [view email] [v1] Thu, 9 Oct 2025 22:40:19 UTC (424 KB) [v2] Fri, 27 Mar 2026 23:39:02 UTC (1,273 KB)