Title: Networked Agent Memory and Causality Representation: Experiences towards Interpretable Cloud-Scale Root-Causing
Authors: Yanyu Ren (Tsinghua University), Xianshang Lin (Alibaba Cloud), Chenxu Wang (Nanjing University), Li Chen, Shuai Wang, Kaihui Gao (Zhongguancun Laboratory), Dan Li, Yunfeng Bai (Tsinghua University), Chen Tian (Nanjing University), Xinlei Zhang, Yunguang Li, Tao Lin, Ennan Zhai (Alibaba Cloud)
Scribe: Mengrui Zhang (Xiamen University)
Introduction
Provider A operates a global infrastructure spanning more than 30 regions, roughly 100 availability zones, and about 100,000 servers and network devices. A single incident can raise more than 2,000 alerts within minutes. A local trigger, such as a faulty board or a configuration error, can propagate through links, routing paths, and protocol dependencies before appearing as service-level degradation. Unrelated alerts from the same subnetwork may be merged into the incident, leaving operators with a noisy dashboard and no direct access to devices because of privacy and safety constraints.
Existing approaches expose three gaps. Rules and fixed workflows are brittle when an unseen failure mode appears. A monolithic LLM agent cannot retain a large alert storm, while naive partitioning loses cross-device context. Finally, a root-cause label or free-form summary does not show the causal path that operators need to verify. XiHe is a production multi-agent framework designed around adaptability to unknown failures, scalability with incident size, and verifiable interpretability. It also targets stable reasoning, cost-efficient operation, and a 90-second diagnosis deadline within a five-minute operational SLA.
Key idea and contribution
Guided multi-agent reasoning with Xilets. XiHe decomposes an incident into a decentralized swarm of Xilets, each scoped to a manageable fault domain or alert signature. XiProxy normalizes heterogeneous alerts and logs into a common format, then creates device-bound Xilets for localized devices and links, as well as signature-bound Xilets for high-volume event types such as change logs. Device scopes are chosen with memory-aware horizontal fusion of similar devices and vertical clustering of adjacent devices referenced by one alert. An incident instantiates 6.04 Xilets on average, 13.87 for incidents involving at least 125 devices, and at most 27 Xilets.
Operational Knowledge Base (OKB) as grounded guidance. The OKB contains alert-to-anomaly candidate mappings and an anomaly causal schema. Built from more than ten years of operational experience, it covers 19 anomaly types and 124 alert signatures; each signature maps to at most four candidates, with an average of 1.25. Operators also codified 26 direct causal relations, and the schema supports up to three propagation generations with up to two missing intermediate anomalies. This abstraction keeps the knowledge tractable while constraining agents to physically plausible hypotheses. Xilets observe a two-minute sliding window, use OKB candidates to form hypotheses, and call MCP tools to check topology and historical evidence before proposing an anomaly or causal relation.
Networked Agent Memory (NAM). Distributed Xilets create partial observability, so NAM lets agents exchange structured evidence without centralizing the whole incident. Its communication primitives are Request (REQ), Confirmation (CONF), and Broadcast Findings (BCST). A Xilet asks neighboring or signature-bound agents for evidence, receives concise anomaly identifiers and descriptions, or broadcasts high-confidence findings to avoid redundant searches. Communication follows physical adjacency and uses structured artifacts rather than raw token sequences; assistance threads handle requests concurrently with each Xilet’s main reasoning loop.
Networked Causality Representation (NCR). NCR consolidates distributed reasoning into a causal graph. Nodes represent inferred anomalies with fault domains and types, raw alerts attach as observable symptoms, and directed edges encode propagation relations proposed by Xilets. Before an edge is accepted, it is checked against OKB rules and evidence such as topology. The schema can restore silent intermediate anomalies, allowing a device that emitted no alert to become the root when downstream evidence supports it. XiHe measures graph quality with Structural Concentration (SC), Topological Connectivity (TC), and Knowledge Compliance (KC), and streams intermediate graphs to the operator dashboard.
Stability, cost, and latency controls. XiHe runs multiple independent Xilet swarms and uses metric-weighted consensus voting to reduce stochastic variation. Three swarms achieve 94.6% accuracy, close to four swarms, so the production system uses three. Early stopping terminates execution when the causal structure converges or no Xilet has further actions. Xilets execute concurrently, and NCR post-processes the graph before publishing the result. LLM serving, the operational knowledge, and the complete pipeline run inside a secure production boundary.
Evaluation
Deployment and operator feedback. XiHe has run for 12 months in the global production cloud, assisting with more than 3,000 incidents and used routinely by over 20 on-call operators across 30+ regions, around 100 availability zones, and O(10^5) devices. Its interactive NCR graph is integrated into the incident dashboard. Strict operator labeling over 470 responses found accuracy above 86% for propagation path, incident type, key-alert attribution, and root-cause attribution. After two months of OKB and prompt refinement, satisfaction exceeded 95% and the paper reports a 95.1% satisfaction rate overall. An A/B test against existing dashboards, which already include rule-based and agent tools, reduced end-to-end operator root-causing time by 25.8%; the paper reports 12.2% reduction for known issues and 27.6% for unknown incidents.
Accuracy across incident types. On 624 compatible incidents, a diagnosis is correct only when both the fault domain and anomaly type match the senior-operator annotation. XiHe reaches 94.6% overall accuracy and reduces the error rate by 54.9% relative to OTA, the strongest baseline covering all incident types. Accuracy reaches 98.5% for change incidents, while XiHe exceeds baselines by 14.3% on link failures and 8.7% on traffic failures. The gain comes from preserving low-severity but causally important logs, separating noise from propagation chains, and reasoning over anomaly abstractions.
Interpretability. Against Cloud Atlas+ (CA+), the baseline with structured causal output, XiHe improves SC by 59.4%, TC by 9.2%, and KC by 13.5%. Anomaly nodes make propagation chains more concentrated, concurrent topology-aware NAM adds cross-scope causal relations, and NCR enforces knowledge-grounded structure. These metrics matter because operators can inspect the graph against raw alerts and topology instead of accepting an opaque root-cause label.
Scalability. On 1,067 incidents selected by alert volume and device count, XiHe’s accuracy decreases only from 93% to 89% when alert volume grows by 125x, while OTA and CA+ fall from 85% and 83% to 53% and 77%. At 625x alert volume, XiHe maintains about 90% accuracy, while both baselines fail. At this 625x scale, each Xilet processes fewer than 3,000 prompt tokens on average; OTA would require about 630,000 tokens and exceed a 128k context limit. When device count exceeds 125, XiHe remains above 90% accuracy, demonstrating that alert volume is the primary scalability constraint.
Stability, cost, and completion latency. A single swarm fails on 14.0% of incidents because of LLM stochasticity. Three swarms achieve 94.6% accuracy with an average tri-swarm cost of $0.13 per incident, reported as 28% cheaper than BiAn. The majority of causal relations appear within 30–60 seconds. Early stopping fires for 48.8% of incidents and reduces average completion time from 95.2 to 73.4 seconds without reducing accuracy or interpretability; only 0.9% of incidents benefit from running beyond 90 seconds.
Ablation and production lessons. Relative to the one-shot OTA baseline, multi-round tool interaction improves accuracy by 2.3%, adding NAM reduces incorrect diagnoses by 36.2%, and adding NCR reduces remaining errors by 21.1%. The deployment also showed that retrieval based on semantic similarity is unreliable for physical networks because similar alert text does not imply physical reachability. Abstracting alert signatures into 19 anomaly types reduces 8,256 alert relation pairs to 171 anomaly-level relations. Just-in-time knowledge injection avoids hallucinated alerts caused by oversized prompts, while streaming NCR lets operators act on partial graphs within about 60 seconds.
These results show why XiHe is useful in production: it combines broad failure-mode coverage, bounded per-agent memory, and a causal explanation that operators can audit. Its main failure mode is a missing root-cause alert, especially for indirect link failures or silent devices; in such cases it narrows the fault boundary and feeds the miss back as a monitoring blind spot rather than inventing unsupported evidence. XiHe diagnoses and explains causes, while corrective mitigation remains a human-operated step.
Q&A
Q1: Why did the system not use a graph neural network (GNN) for temporal and topological root-causing?
A1: We did not use a GNN for two reasons. First, we want to use the generality of an LLM to reason about unknown or unseen failure modes, whereas a GNN would rely more heavily on historical cases. Second, embedding incidents involving hundreds of devices into fixed-length vectors creates scaling and optimization difficulties that are outside this paper’s focus.
Q2: How does XiHe achieve short latency, and what engineering choices guide the LLM?
A2: We give each Xilet multiple threads so it can handle hypotheses and NAM assistance requests concurrently. We also limit context with a two-minute sliding window, run many Xilets in parallel, and apply early stopping after graph convergence. Together with memory-aware scoping, these mechanisms allow us to meet the 90-second target; the detailed serving and prompt mechanisms are described in the paper.
Q3: How is the operational knowledge base maintained when new alert patterns and failure modes appear?
A3: Our monitoring system produces the alert signatures, and we track changes when new signatures are introduced. We add anomaly types or refine the OKB when we manually investigate an unseen failure. During the first two months of deployment, we used operator feedback to refine the OKB and prompts; later maintenance remained lightweight. The 19-type anomaly abstraction and 171 relation pairs keep this curation tractable.
Q4: Does XiHe require network-specific trained models, such as time-series or topology models, or is a general LLM sufficient?
A4: We did not fine-tune a network-specific model in this work. Our Xilets use the publicly available Qwen3-MAX-Instruct backbone with tool calls.
Personal thoughts
XiHe’s strongest design choice is treating interpretability as an operational requirement rather than an after-the-fact explanation layer. Xilets divide the alert storm according to physical scope, NAM reconnects the distributed evidence, and NCR exposes the propagation chain that an operator must verify. The deployment numbers are persuasive because they measure not only accuracy, but also operator time, satisfaction, cost, latency, and behavior under 625x alert growth.