Title: UBEP: Re-architecting Expert Parallelism Communication Library for Production Superpods
Authors: Yipeng Liu, Chang Liu, Si Shen, Jiaqi Zheng, Yuyang Yang, Guanhua Li, Yuquan Zhang, Yimeng Xu, Guihai Chen (State Key Laboratory for Novel Software Technology, Nanjing University); Mingfan Li, Zhongzhe Hu, Zhiyuan Huang, Qihang Duan, Junsong Wang, Wenkai Ling, Baochuan Yang, Xianzhi Yu, Han Bao, Yijie Chen (Huawei Technologies Co., Ltd.)
Introduction
Mixture-of-Experts (MoE) models have become increasingly important for scaling large language models efficiently, but their sparse and dynamic token routing introduces expensive All-to-All communication among experts. Meanwhile, modern AI infrastructure is evolving toward high-bandwidth, low-latency superpods such as NVIDIA NVL72/576 and Huawei CloudMatrix384 (CM384). Although these systems provide much higher interconnect bandwidth and unified global address spaces, the paper shows that the bottleneck is shifting from raw network bandwidth to software overhead. Existing expert-parallel communication libraries largely rely on Bulk Synchronous Parallel (BSP) execution with global barriers, expensive synchronization, and topology-agnostic scheduling. These mechanisms serialize otherwise independent operations and create stragglers, preventing MoE workloads from fully utilizing the capabilities of modern multi-tier superpods.
Key idea and contribution:
The authors build UBEP (Unified-Bus Expert Parallelism), a production-ready expert-parallel communication library designed specifically for modern superpods. The key idea is to replace coarse-grained BSP-style execution with fine-grained, dependency-driven execution. UBEP introduces Kernel Decomposition to divide the traditional monolithic All-to-All operation into independent tasks and assign different AIV cores to token transmission and metadata processing. This allows operations such as Token-Sending and CalCumSum to execute concurrently instead of being separated by global barriers.
UBEP further introduces two mechanisms to address synchronization and topology-related bottlenecks. Hierarchical Token-level Scheduling jointly considers token load and physical network distance, balancing one-hop and two-hop traffic across AIVs to reduce stragglers. Data-as-Flag exploits the CM384 fabric’s 512-byte atomic load/store capability to integrate synchronization information directly with data transmission, reducing or eliminating explicit synchronization barriers. Together, these techniques allow UBEP to exploit the low-latency, high-bandwidth characteristics of modern superpods more effectively.
Evaluation
The authors evaluate UBEP on a production Huawei CM384 superpod using up to 256 NPU dies and several MoE models ranging from Qwen3-30B to DeepSeek-V3.2. Compared with CANN EP on the same hardware, UBEP improves effective bandwidth by 35.3%–40.8%, while Kernel Decomposition and Data-as-Flag reduce dispatch latency by 34.7%–52.4%. Topology-aware scheduling also reduces the maximum AIV latency from 62.2 μs to 43.5 μs in the evaluated 64-NPU setting. At the application level, UBEP reduces P99 end-to-end TPOT by up to 11.1% across Qwen3, GLM, and DeepSeek models. This result is significant because it demonstrates that, as interconnects become faster, improving the software execution and synchronization model can provide substantial performance gains even without increasing the underlying network bandwidth.
Q1: Since UBEP’s design and evaluation are mainly based on the Huawei CloudMatrix384 architecture, can it also work on other similar architectures, such as NVIDIA GPUs?
A1: Yes, I think so. Our current implementation is mainly adapted to architectures like CloudMatrix384, but GPUs have similar synchronization problems. Recent NVIDIA work has also explored mechanisms similar to Data-as-Flag. On GPUs, we can map our AIV-level design to warps or thread blocks, so we believe this is a general approach that can be applied to other platforms as well.
Q2: Have you already evaluated UBEP on NVIDIA GPUs or other GPU platforms?
A2: Not yet. We have not presented experimental results for UBEP on GPUs, but this is something we would like to explore in future work.
Personal thoughts
I think the most interesting takeaway from this paper is its observation that the bottleneck changes as hardware evolves. When network bandwidth was relatively limited, communication time dominated and software synchronization overhead was less visible. On modern superpods, however, the network becomes fast enough that global barriers, kernel scheduling, and synchronization themselves become major bottlenecks. I particularly like the combination of Kernel Decomposition and Data-as-Flag because it does not simply try to make data transfer faster; instead, it changes how communication tasks are organized and synchronized to better match the capabilities of the underlying hardware.
At the same time, the portability of UBEP remains an open question. Some of its implementation relies on CM384-specific features, especially 512-byte atomic memory operations, and the current evaluation is limited to the Huawei platform. It would therefore be interesting to see how much of the performance improvement can be reproduced on NVIDIA NVLink-based superpods or other scale-up fabrics. Another interesting direction is to extend the same dependency-driven execution model beyond MoE All-to-All communication. Since the paper shows that communication accounts for only part of end-to-end TPOT, applying similar fine-grained techniques to computation, framework scheduling, and other collective operations may lead to larger end-to-end improvements.