Title: Evolution of AliYANG: Model-driven and LLM-assisted Network Configuration Management
Authors: Mohan Yu (Jilin University and Alibaba Cloud), Xumiao Zhang, Zhe An (Alibaba Cloud), En Wang (Jilin University), Bin Zheng (Alibaba Cloud)
Scribe: Mengrui Zhang (Xiamen University)
Introduction
Configuration management in a cloud network spans an entire lifecycle: architects define configuration specifications, controllers generate and deploy full configurations during construction, operators apply incremental changes as the network evolves, and auditing systems continuously detect drift and risk. Every stage must remain correct, repeatable, and scalable across multiple vendors. Before AliYANG, these tasks were implemented with separate templates and scripts. The production system discussed in the presentation contained roughly 42,000 templates and 5.5 million lines of specialized code. This complexity grows along three dimensions: management interfaces such as CLI, NETCONF, and gNMI; lifecycle stages such as generation, change, and auditing; and vendor-specific syntax and semantics.
YANG provides structured schemas and validation within a vendor, yet standard YANG does not capture CLI text or semantic comparison rules needed for auditing. Vendor YANG models also represent equivalent features with different names, schema locations, constraints, and scopes. OpenConfig covers only a subset of native device functionality and can lose vendor-specific semantics. AliYANG addresses these gaps through an extended modeling framework, a vendor-independent core model, the NetCMDB production system, and offline LLM-assisted model evolution.
Key idea and contribution
-
Extend YANG to encode CLI behavior and semantic equality. AliYANG introduces 12 CLI-aware extensions that describe keywords, command views, field placement, ordering, optional parameters, and deletion commands. It also provides eight comparison extensions for order-insensitive collections, case-insensitive values, device-generated fields, and composite keys. CLI syntax becomes model data that can drive generation, parsing, and semantic diff from one specification.
-
Separate vendor semantics from shared cross-vendor meaning. An augmented vendor model starts from the vendor’s native YANG when one is available and adds the CLI semantics required in production. When a vendor provides no YANG model, engineers construct one from device documentation or existing tools. A Core YANG model then merges semantically equivalent elements across vendors. Its design follows a commonality-first principle: shared behavior enters the core, while genuine vendor-specific behavior remains available through structured augmentations. The core covers configuration for routing, interfaces, ACLs, QoS, monitoring, and system services; operational state such as counters and forwarding tables remains outside its scope.
-
Make models executable through NetCMDB. NetCMDB compiles vendor and core YANG models into strongly typed configuration objects and builder interfaces in a host programming language. The Core YANG model is the controller’s primary abstraction, while structured augmentations retain access to vendor-specific functions when needed. The system translates between core and vendor objects, serializes them into CLI text, NETCONF XML, or JSON/Protobuf for gNMI, parses deployed configurations, and compares objects using model-defined semantics. Full generation constructs and serializes objects; incremental change computes semantic differences; auditing parses and compares intended and deployed objects; repair renders a difference as vendor-specific commands. Versioned YANG Suites select the applicable model subset for each hardware model and OS version. A dependency graph and topological ordering preserve command dependencies in generated configurations.
Evaluation
- Production scale and operational impact. NetCMDB has run in production for more than three years on a 20-server cluster. It manages on the order of 100,000 devices, with approximately 10,000 configuration lines per device and more than one billion lines in total. Model-based parsing enables static validation before configurations reach physical devices or simulators. Indexed queries over structured configuration data reduce network-wide questions, such as finding every device that references a route policy, from more than 20 hours of device-by-device auditing to seconds. Audits can also generate correctly ordered, vendor-specific incremental repair commands. With LLM assistance, onboarding a new vendor fell from two to three person-months of full-time work to about one month at roughly 50% engineer utilization, a four-to-sixfold reduction in engineering effort.
- Vendor-model augmentation. Patch-based generation outperforms direct regeneration of an entire YANG model across every reported similarity metric: BLEU improves by 10.39%, ROUGE-1 F1 by 5.53%, ROUGE-2 F1 by 7.48%, and ROUGE-L F1 by 8.77%. Direct generation introduces 22 hallucinated nodes and omits three nodes, while patch-based generation reports zero in both categories. Iterative validation generally improves quality, although additional rounds can regress for some models. Qwen-Plus with three rounds lies on the reported Pareto frontier, scoring 0.67 in 99.65 seconds and nearly matching GPT-5.2 in less than half the processing time.
- Core-model generation. The evaluation separately measures syntactic validity and semantic quality. Models with similar syntax scores can differ substantially in functional coverage, type correctness, ranges, enumerations, and structural clarity. Qwen3-Max achieves the highest semantic score, Qwen-Plus has the lowest processing time, and GPT-5.2 has the highest latency without achieving the best semantic result. The experiment shows that domain-specific semantic evaluation is necessary even when generated YANG passes a syntax checker.
- Translation-code generation. Generated vendor-to-core and core-to-vendor code is evaluated at two levels: compilation and compilation plus round-trip tests on real configuration instances. A substantial fraction of generated code compiles while remaining semantically incorrect and failing the round-trip tests. Performance also varies across vendors; GLM-4.7 generally has a higher compile-and-test success rate than Qwen3-Coder-Plus. The result explains why compilation alone is insufficient and why the translation workflow retains deterministic semantic tests.
Q&A
Q1: Round-trip translation may return the original representation even when the model never captured the operator’s true intent. How does the system detect this problem, and how should that end-to-end result be interpreted?
A1: I agree that round-trip success is a necessary consistency check and does not constitute a formal proof of operator intent. We validate generated models syntactically, compile the translation code, and run schema-aware round-trip tests over representative real configurations. The final round trip checks semantic equivalence after vendor-to-core and core-to-vendor translation, but it cannot prove that the original model fully expresses operator intent. We could use formal network-verification tools such as Batfish as a complementary approach.
Q2: What drives the LLM refinement iterations, and does the system use methods such as LLM-as-a-judge to improve accuracy over time?
A2: Our current refinement loop mirrors an engineer’s debugging process. We return syntax-validation errors, compiler diagnostics, and round-trip test failures to the LLM together with the task context, then ask it to revise the patch, model draft, or translation code. In our evaluation, we use LLM-as-a-judge to score vendor-model coverage and Core YANG semantic quality, but we do not use that judge as the production repair signal. We believe more advanced judging and feedback techniques could improve the loop and reduce convergence time.
Q3: How does the system handle configuration differences across devices and software versions, and how are LLM-generated models used in production?
A3: We use LLMs to assist with vendor-model augmentation, Core YANG construction, and translation-code generation; we do not ask them to generate or push live device configurations. We maintain versioned vendor YANG models in NetCMDB, and identify the applicable subset for each device by its hardware model and OS version as a YANG Suite. Our controller uses the Core YANG abstraction and the selected vendor models to generate device-specific configurations through NetCMDB. Our engineers review, refine, and test all LLM-generated intermediate artifacts before integration. This process reduces the engineering effort for onboarding a new vendor by four to six times.
Personal thoughts
The most compelling aspect of AliYANG is its allocation of responsibility. Configuration semantics live in long-lived models, execution and validation live in deterministic software, and LLMs help engineers produce repetitive model and translation artifacts. This boundary makes the use of LLMs credible in a high-risk operational setting: every generated artifact remains inspectable, compilable, and testable before it can influence production. The commonality-first core model is also a pragmatic response to heterogeneous devices because it preserves native capabilities while still giving controllers a stable cross-vendor interface.