Process-Centric Analysis of Agentic Software Systems

Agentic systems are modern software systems: they consist of orchestrated modules, expose interfaces, and are deployed in software pipelines. Unlike conventional programs, their execution trajectories are inherently stochastic and adaptive to the problems they solve. Evaluation of such systems is often outcome-centric, focusing only on whether the final task succeeds or fails. This narrow view overlooks how agents reason, plan, act, and revise their strategies during execution.

We introduce Graphectory, a graph-based representation that systematically encodes the temporal and semantic relations in agent trajectories, enabling process-centric metrics and analyses of workflow quality. Using Graphectory, we analyze 4,000 trajectories from two leading software engineering agents, SWE-agent and OpenHands, paired with four backbone large language models on SWE-bench Verified.

Our fully automated analyses reveal that stronger models and richer prompts lead to more complex yet more coherent workflows, while unsuccessful trajectories tend to exhibit chaotic, repetitive, or backtracking behaviors. We further develop a real-time monitoring framework that constructs and analyzes Graphectory and Langutory online to detect process-level issues, notify the agent with diagnostic feedback, and roll back execution when appropriate. Experiments show that online monitoring with intervention improves resolution rates by 6.9%–23.5% on problematic instances while significantly shortening trajectories with near-zero overhead.