Title: Balancing and Beyond: Communication-Centric Optimizations in Expert Parallelism
Authors: Jiamin Cao, Qingxu Li, Yaozhong Liu, Jiaqi Gao, Yan Zhang, Shangfeng Shi, Zian Chen, Yizhi Wang, Jun Zhang, Kunling He, Ennan Zhai, Jianbo Dong, Binzhang Fu, Dennis Cai (Alibaba Cloud)
Introduction
Mixture-of-Experts (MoE) models enable large-scale LLM inference by activating only a small subset of experts for each token, while Expert Parallelism (EP) distributes experts across GPUs. In production, however, EP suffers from two major bottlenecks: expert workload imbalance and inefficient communication. Hot experts can receive far more tokens than others, creating compute and communication stragglers, while existing EPLB methods rely on historical statistics and do not react quickly to workload changes. Even after balancing, fixed communication strategies such as DeepEP may create NIC hotspots, redundant inter-host transfers, and inefficient overlap for small or medium batches. The paper therefore studies how to jointly optimize load balancing and communication for production MoE inference.
Key idea and contribution:
The authors build EPIC, which follows the principle of “first establish balance, then optimize communication.” For load balancing, EPIC combines communication-aware periodic expert placement with real-time intra-host expert migration. It considers both GPU compute load and GPU-to-NIC topology, and dynamically swaps experts between overloaded and underloaded GPUs when the expected benefit exceeds the migration cost. After balancing, EPIC improves communication through multiple transport modes, topology-aware PXN relay, and de-redundancy to reduce inter-host traffic. It also proposes Per-Expert Overlap (PEO), which pipelines dispatch, expert computation, and combine along the expert dimension instead of splitting the batch as TBO does. This preserves GEMM efficiency while still overlapping communication and computation. EPIC also includes memory optimizations and fault-recovery mechanisms for production deployment.
Evaluation
EPIC is evaluated on NVIDIA H20 and H800 clusters using DeepSeek-R1 and Qwen3-Coder with production workloads. Its balancing mechanisms reduce the imbalance ratio by about 40% and improve TPOT by 10–18%. Communication optimizations reduce communication time by up to 40% and TPOT by up to 21%, while PEO improves TPOT by up to 17.6% for small and medium batches. In production, EPIC improves TPS by up to 27% and has been deployed across O(10K) GPUs.This result is significant because the optimizations provide measurable end-to-end benefits in real large-scale LLM serving, rather than only improving isolated communication benchmarks.
Personal thoughts
I like that this paper approaches EP optimization from an end-to-end production perspective. Its key insight is that the bottleneck changes over time: workload imbalance must be addressed first, and communication becomes the main problem only after the system is balanced. The design of PEO is also interesting because it improves overlap without sacrificing GEMM efficiency. One limitation is that real-time migration is restricted to intra-host transfers, while policy selection still depends on offline profiling for different models, hardware, and batch sizes. A useful future direction would be more adaptive online optimization that jointly decides expert placement, communication routing, and overlap strategies under changing workloads and heterogeneous hardware.