Title: AIDA: Accelerating Root Cause Analysis for Multi-Vendor Device Failures with LLM-Powered Reasoning
Authors: Haoran Xu (Sun Yat-sen University and Alibaba Cloud); Xuan Zeng, Xumiao Zhang (Alibaba Cloud); Xiaoxi Zhang (Sun Yat-sen University); Yang Lv, Peng Zhang (Alibaba Cloud); Deke Guo (Sun Yat-sen University); Ennan Zhai (Alibaba Cloud)
Reviewer: Xing Fang (SNG fifth-year PhD student)
Introduction
This paper, a collaboration between Sun Yat-sen University and Alibaba Cloud, presents AIDA, a system for root-cause analysis of failures in multi-vendor network devices. Traditional monitoring systems can usually identify which device has failed, but they are far less effective at explaining why it failed. AIDA extracts expert reasoning from historical diagnostic email threads between operators and device vendors, organizes it into a continuously updated knowledge graph, and uses retrieval-augmented, multi-step reasoning to generate RCA reports with log evidence and confidence scores. The system has been deployed in Alibaba Cloud’s production network for more than a year.
Failure handling in a large cloud network typically has two stages. The first uses monitoring systems to locate the faulty device and restores service by isolating it or rerouting traffic. The second performs device-level root-cause analysis to identify the particular software bug, hardware component, or error type and prevent recurrence. Rebooting or bypassing a device may restore service temporarily, but it does not remove the underlying fault, making RCA essential to long-term network reliability.
Alibaba Cloud’s production network spans dozens of datacenters and geographic regions and contains hundreds of thousands of devices from multiple hardware and software vendors. Many failure modes are known primarily to the vendors themselves. Operators therefore exchange several rounds of email with vendor experts: they first describe the symptom, then collect command output and logs on request, wait for analysis, gather more evidence, and eventually receive an RCA report. The paper defines the interval from the first log collection to the final report as Time to RCA (TRCA). Data from six vendors shows that the first reply is usually prompt, but a complete RCA often takes tens or hundreds of hours because of the repeated diagnostic exchanges that follow.
Figure 1: Traditional device RCA requires several rounds of log exchange between operators and vendor experts, and diagnosis can last for tens of hours.
Existing automation does not solve the problem directly. Rule-based and conventional machine-learning methods generally require prior knowledge of device internals and struggle with changing hardware and software versions. Failures also have a pronounced long tail: 27% of failures in the paper’s dataset belong to rare categories that occur fewer than ten times per year. A general-purpose LLM can read email and logs, but it may miss critical details, hallucinate evidence, or produce conclusions that operators cannot verify.
Core Ideas and Contributions
AIDA’s first challenge is recovering genuine expert reasoning from complex email threads. Historical email mixes natural language, vendor terminology, configuration snippets, command output, and large volumes of logs, while a single conclusion may emerge only after several rounds of interaction. The system must extract not only the final cause, but also the causal sequence from failure symptom to diagnostic action, log evidence, and root-cause conclusion. Because a general model lacks specialized device-diagnosis knowledge, the authors use a small amount of expert annotation and reinforcement learning to train a dedicated reasoning-chain extractor.
The second challenge is duplication and noise in historical data. Common failures appear repeatedly in the email archive, so naïve RAG may retrieve several nearly identical cases, consume the context window, and hide rare but relevant incidents. A generic knowledge graph can also treat mere co-occurrence in one email as causality, mixing useful diagnostic chains with locations, administrator names, and ordinary status facts. The key insight is that device RCA needs neither a larger text archive nor a generic entity graph, but a structured reasoning graph that preserves diagnostic order and causal meaning.
Figure 2: Duplicate cases reduce the effective recall of naïve RAG, while a generic knowledge graph may mix causal relationships with irrelevant facts.
The third challenge appears during online reasoning. A single device can produce tens of thousands of log lines, and complete reasoning chains share many repetitive steps. Sending every case and log to an LLM at once would exceed context limits and encourage reasoning drift and error propagation. AIDA therefore decomposes diagnosis into verifiable steps: retrieve candidate reasoning chains from the problem description, extract evidence using the log template attached to each node, validate candidate causes one by one, and only then aggregate confidence and generate a report.
AIDA has two phases: knowledge fusion and online diagnosis. Offline, it identifies problem descriptions and final reports in historical RCA email, trains a reasoning-chain extractor, merges equivalent nodes across cases into a knowledge graph, and creates semantic templates for filtering runtime logs. Online, it retrieves reasoning chains for a new failure, filters relevant logs, and uses multi-step verification and confidence ranking to determine the most likely root cause.
Figure 3: AIDA combines reasoning-chain extractor training, RCA knowledge-graph construction, and knowledge-guided online diagnosis.
Email preprocessing and reasoning-chain extraction. The system begins with 21,482 historical emails. The team manually classified 5,411 emails from 981 failures in about ten person-hours, then trained an SVM to distinguish problem descriptions and RCA reports with 95% classification accuracy. Two experts subsequently annotated 50 reasoning chains in 2.5 person-hours. These examples were used for supervised fine-tuning, followed by GRPO reinforcement learning. The reward function evaluates both content completeness and logical coherence. Log statements used as evidence must match the original email exactly, preventing the extractor from inventing logs while building the knowledge base.
Knowledge-graph construction. Each reasoning chain contains a problem-description node, several evidence nodes, and a root-cause summary. AIDA first partitions cases by vendor, device model, and related properties to prevent erroneous merging across incompatible devices. It then combines text similarity, SimRank structural similarity, and adaptive weighting to merge semantically equivalent evidence and cause nodes. Highly specific evidence closely associated with a root cause emphasizes graph structure, whereas common states such as “resources normal” rely more heavily on text. AIDA also generates semantics-aware regular-expression templates from evidence nodes and periodically removes knowledge associated with retired devices while adding new cases, allowing the graph to evolve with the production network.
Knowledge-guided multi-step diagnosis. For a new failure, AIDA retrieves candidate chains based on the problem description, vendor, operating system, and hardware type. It performs log filtering in two rounds. The first applies multiple templates to extract potentially relevant fragments; the second combines more specific templates to narrow the evidence to the target module. The LLM then checks, node by node, whether the logs support each symptom and whether the device modules and causal relationships are consistent across the chain. Only validated chains enter confidence ranking. If evidence conflicts or the knowledge graph lacks a matching cause, AIDA explicitly returns “insufficient information” and escalates the case to a human instead of forcing an answer.
Figure 4: AIDA verifies candidate reasoning chains step by step, rejects nodes without evidence, and then ranks valid root causes and generates the report.
Experimental Evaluation
AIDA has processed 846 device failures during more than a year of production deployment. After full rollout, it handled an average of 101 cases per month. Its mean precision across the deployment was 95.4%, while accuracy increased from 70.2% in the initial phase to 85.4% after full deployment as knowledge coverage improved. For cases not directly represented in the training data or knowledge graph, AIDA still achieved 75.3% precision and abstained when evidence was insufficient.
Deployment dramatically reduced RCA time. Median TRCA fell from 72.6 hours in the manual workflow to 1.6 minutes. For complex long-tail cases, the 90th-percentile TRCA fell from 329.9 hours to 19.4 hours. These difficult cases may still require human intervention, but AIDA automatically completes many investigations that once took hours or days. Its own core analysis averages 37.4 seconds, with a 90th percentile of 54.4 seconds; log filtering, stepwise reasoning, and chain verification and summarization do not create a new performance bottleneck.
Figure 5: AIDA’s phased rollout, the distribution of manual and automated RCA time, and internal system latency.
The offline evaluation uses 180 challenging production failures. Across six backbone LLMs of different sizes, AIDA averages 82.8% accuracy and 94.8% precision, outperforming the strongest baseline by 8.0 and 9.8 percentage points on average. Structured knowledge matters more than simply scaling the model: AIDA with an 8B model reaches 83.1% accuracy, exceeding the 76.0% of LightRAG with a 72B model.
Ablation results further confirm the value of each component. Structure-aware node merging raises RCA precision from 93.9% to 96.6%, while removing multi-step reasoning lowers precision by 3.9 percentage points. Knowledge fusion reduces graph size from 2,367 nodes to 882, a 62.7% decrease. Two-stage log filtering cuts the average input per case from 16,729.4 lines to 367.7 lines, leaving only about 32.2 lines for each LLM call while reducing evidence-log recall by just 3.4 percentage points.
Q&A
Q1: LLM reasoning is evolving rapidly. How can AIDA continue to evolve as the underlying models improve?
A1: AIDA’s main design centers on constructing and organizing the RCA knowledge base while keeping it decoupled from online LLM reasoning. Online diagnosis does not depend on one end-to-end fine-tuned model, so a stronger reasoning model can replace the underlying LLM without redesigning the entire RCA workflow.
Personal thoughts
I appreciate AIDA’s decision to abstain when evidence is insufficient. In production-network RCA, a polished but incorrect conclusion can directly mislead repair and change operations, so high precision is often more valuable than covering every failure. Treating “unable to determine” as a valid outcome also matches how operators actually use an assisted-diagnosis tool.
AIDA’s effectiveness, however, depends heavily on years of diagnostic email and accumulated vendor knowledge. A network with few historical incidents, incomplete records, or rapidly changing devices may not obtain the same coverage. In practice, continually maintaining high-quality failure records, evidence, and version information may be a greater deployment cost than selecting a larger model.