ProbGuard: Probabilistic Runtime Monitoring for LLM Agent Safety

View PDF HTML (experimental)

Abstract:Large Language Model (LLM) agents increasingly operate across domains such as robotics, virtual assistants, and web automation. However, their stochastic decision-making introduces safety risks that are difficult to anticipate during execution. Existing runtime monitoring frameworks, such as AgentSpec, primarily rely on reactive safety rules that detect violations only when unsafe behavior is imminent or has already occurred, limiting their ability to handle long-horizon dependencies. We present ProbGuard, a proactive runtime monitoring framework for LLM agents that anticipates safety violations through probabilistic risk prediction. ProbGuard abstracts agent executions into symbolic states and learns a Discrete-Time Markov Chain (DTMC) from execution traces to model behavioral dynamics. At runtime, the monitor estimates the probability that future executions will reach unsafe states and triggers interventions when this risk exceeds a user-defined threshold. To improve robustness, ProbGuard incorporates semantic validity constraints in the abstraction and provides PAC-style guarantees on the learned model under standard assumptions. We evaluate ProbGuard in two safety-critical domains: autonomous driving and embodied household agents. Across evaluated scenarios, ProbGuard consistently predicts traffic law violations and collisions in advance, with warnings up to 38.66 seconds ahead of occurrence. In embodied agent tasks, ProbGuard reduces unsafe behavior by up to 65.37% while preserving up to 80.4% task completion. ProbGuard is implemented as an extensible open-source runtime monitor integrated with the LangChain agent framework and introduces minimal runtime overhead.

Subjects:

Artificial Intelligence (cs.AI); Software Engineering (cs.SE)

Cite as: arXiv:2508.00500 [cs.AI]

(or arXiv:2508.00500v3 [cs.AI] for this version)

https://doi.org/10.48550/arXiv.2508.00500

arXiv-issued DOI via DataCite

Submission history

From: Christopher M. Poskitt [view email] [v1] Fri, 1 Aug 2025 10:24:47 UTC (837 KB) [v2] Tue, 6 Jan 2026 03:51:07 UTC (498 KB) [v3] Fri, 27 Mar 2026 08:07:20 UTC (767 KB)