Title: DistDPU: A Disaggregated DPU Architecture for High-Performance and Cost-Efficient AI Clouds
Authors: Hao Mei (Fudan University); Lizhou Gao (Tencent); Yuanyi Zhu (Fudan University); Liang Wang, Peng Yang, Chao Pei (Tencent); Chuhao Chen, Zijian Li (Fudan University); Jian Zhao, Dongbo Gu, Hongchen Ren, Jiyuan Chen, Junpeng Zhang, Yunpeng Guan, Jianye Yuan, Jian Wang (Tencent); Yibo Huang (University of Michigan); Yang Xu (Fudan University)
Scribe: Yuntao Zhao (Xiamen University)
Introduction
Large-scale model training and inference are placing increasingly demanding bandwidth requirements on server networks, and per-server network capacity in some AI clusters has already reached the Tbps level. Existing cloud platforms typically scale aggregate bandwidth by stacking multiple complete DPUs within a server. However, this approach also duplicates processors, memory, and management components on each DPU, causing control-plane resources to increase proportionally with bandwidth.
Based on observations from real-world AI cloud deployments, the paper finds that such a scaling approach does not align well with the characteristics of AI workloads. AI communication is dominated by a relatively small number of long-lived, high-throughput “elephant flows.” As a result, dataplane forwarding capacity needs to grow rapidly, while control-plane tasks such as connection maintenance and policy management do not increase at the same rate. To address this divergence between control-plane and dataplane demands, the authors propose DistDPU, a disaggregated DPU architecture that divides a conventional DPU into an Orchestration Module (OM) for centralized management and an Execution Module (EM) for high-speed execution, allowing the two types of resources to scale independently.
Key Idea and Contribution
The core of DistDPU is a “Brain–Muscle” architecture: the Orchestration Module (OM) acts as the “Brain,” responsible for complex global control and management, while the Execution Module (EM) serves as the “Muscle,” handling high-speed execution directly related to bandwidth. In this way, DistDPU separates complex management functions from high-speed dataplane processing, avoiding the need to duplicate complete control-plane resources when additional network bandwidth is required.
(1) Decoupling the Orchestration Module (OM) and Execution Module (EM)
DistDPU reorganizes the functions of a conventional DPU according to whether they need to scale with network bandwidth. The OM centralizes policy decisions, configuration generation, global coordination, key management, storage management, and interaction with the cloud control plane. The EM retains only the key dataplane execution functions that need to scale with bandwidth, including device virtualization, Ethernet/RDMA processing, and network virtualization forwarding. Within these functional domains, policy and orchestration remain in the OM, while per-packet processing and low-latency execution are placed in the EM.
(2) Datapath design for AI networking
To prevent the disaggregated architecture from affecting high-speed communication, DistDPU implements Ethernet/RDMA engines and network virtualization functions in the EM. SmartEP separates device configuration processing from high-speed data access, allowing device presentation to be adjusted flexibly while keeping critical data transfers on the hardware fast path. In addition, the paper introduces Direct Offset Translation (DOT) to reduce address translation overhead for GPUDirect RDMA in virtualized environments.
(3) Unified inter-module management
After disaggregation, the OM must be able to securely and transparently manage multiple physically distributed EMs. To achieve this, DistDPU introduces the M2M Fabric, which supports configuration distribution, status queries, and monitoring data transfer between the OM and multiple EMs. The M2M Fabric hides the underlying physical connectivity from the OM, allowing multiple EMs to appear as a unified DPU from the management perspective.
Evaluation
The paper first uses production deployment results to validate the mismatch between control-plane and dataplane resource demands. DistDPU has been deployed at scale in production and serves more than 10,000 GPUs. Over a 24-hour observation period, the average CPU utilization of the OM remains below 15%, even when dataplane bandwidth reaches the Tbps level. This indicates that control-plane computation does not increase proportionally with network bandwidth.
(1) Cost and Power Consumption
The paper normalizes the cost and power consumption of DistDPU to 1. In the evaluation, the per-node hardware cost and power consumption of Multi-BF3 are 4.91× and 2.77× those of DistDPU, respectively. To exclude the impact of pricing differences between vendors, the authors also compare against Multi-DPU using their own hardware. The hardware cost and power consumption of Multi-DPU are 2.45× and 3.46× those of DistDPU, respectively, showing that the disaggregated architecture can reduce the additional overhead caused by duplicated control-plane resources.
(2) Network Performance
In end-to-end RDMA experiments, DistDPU achieves peak bandwidth comparable to Multi-BF3 and Multi-CX7, with a performance gap of only 0.4%–2.8%. This shows that separating the OM from the EM does not introduce significant overhead to high-speed data transmission.
(3) Large Model Training and Inference
For Mixtral 8×7B training, DistDPU improves throughput by 0.43%–1.26% compared with Multi-BF3 and Multi-CX7. For Llama2-7B, the performance difference among the three systems is only −0.03%–0.1%. For DeepSeek-R1 inference, the difference in Time to First Token (TTFT) between DistDPU and the baselines is 0.2%–1%, while the difference in Time Per Output Token (TPOT) is below 0.2%. These results show that DistDPU can maintain end-to-end performance close to existing high-performance NIC solutions in practical AI workloads.
In addition, the control communication overhead of the M2M Fabric is low. For a standard 4B message, the PCIe write and read latencies are 2.2 μs and 4.9 μs, respectively. Although the bandwidth required for configuration distribution and monitoring data collection increases with the number of EMs, it remains well below the capacity provided by the underlying link.
Q&A
Q1: Compared with the original multi-DPU design, does this disaggregated architecture introduce a new single point of failure? What happens if the entire OM or the full-feature NIC hardware fails?
A1: Actually, we have already considered this issue in the design. There is a background monitoring process running on the OM that continuously monitors the status of the modules. If it finds that a module is not responsive, the system will reinitialize the corresponding process. So this kind of failure handling has already been considered in the design.
Personal thoughts
The most noteworthy aspect of DistDPU is that it does not simply attempt to improve DPU processing performance. Instead, it redesigns the internal organization of the DPU according to the actual resource demands of AI workloads. By centralizing control functions in the OM and placing bandwidth-related processing capabilities in the EMs, the system can scale network capacity without repeatedly provisioning large amounts of control-plane resources, which closely matches the AI cloud workload characteristics observed in the paper. However, the benefits of this design rely on the fact that control-plane load in AI workloads grows much more slowly than dataplane bandwidth demand. For general-purpose cloud workloads with more connections and more frequent control events, whether the same level of resource savings can be achieved still requires further validation. In addition, the current design mainly uses a single OM to manage multiple EMs, while more complex multi-OM configurations are left for future work.