Title: HyNA: Taming Tail Latency in MoE Training with Hybrid Switch Silicon
Authors: Yang Liu, Tianxiang Liu, Haipeng Yao (Beijing University of Posts and Telecommunications)
Introduction
As large language models (LLMs) continue to scale toward trillion-parameter models, especially with the adoption of Mixture-of-Experts (MoE) architectures, distributed training faces a growing communication bottleneck rather than a pure computation limitation. The increasing gap between GPU computing capability and network bandwidth creates a “Communication Wall,” where communication delays prevent expensive accelerators from being fully utilized. This problem becomes more severe in MoE training because its sparse and bursty traffic patterns introduce frequent collisions and unpredictable tail latency. Existing solutions such as Parameter Server (PS) architectures suffer from incast congestion, CPU/NIC/PCIe overhead, and limited scalability, while In-Network Aggregation (INA) approaches still rely on external fallback mechanisms when encountering hash collisions or numerical exceptions. These fallback paths introduce microsecond-level latency spikes, causing synchronization delays in Bulk Synchronous Parallel (BSP) training and limiting the effectiveness of network acceleration
Key idea and contribution:
The authors propose HyNA (Hybrid Network Aggregator), a serverless in-network aggregation system designed to eliminate the tail latency problem in MoE training communication. The key idea is to combine a high-speed programmable switch pipeline with embedded RISC-V cores in a heterogeneous switch architecture. HyNA uses the RMT (Reconfigurable Match Tables) pipeline as a fast path to process more than 99.9% of normal gradient aggregation operations at wire speed, while using the embedded RISC-V cores as an exception path for complex cases such as hash collisions and numerical overflows. Instead of sending these exceptions back to external servers, HyNA introduces On-Chip Closure, keeping all processing inside the switch ASIC and converting unpredictable network latency into deterministic on-chip latency.
Besides the hardware architecture, HyNA also designs a hybrid precision protocol to balance performance and accuracy. Workers initially use INT32 quantization for efficient aggregation in the switch pipeline, while overflow cases are automatically redirected to RISC-V cores for FP32 computation to preserve bit-level accuracy. The authors implement a 100 Gbps FPGA prototype and perform a 7nm ASIC synthesis analysis, demonstrating that integrating 32 RISC-V cores introduces only limited hardware overhead while enabling efficient large-scale MoE training acceleration.
Evaluation
The authors evaluate HyNA using a hardware-in-the-loop platform based on a Xilinx Alveo U280 FPGA, replaying realistic gradient traces from Llama-3, Qwen, and DeepSeek-V2 workloads. The results show that HyNA achieves 84.5 Gbps aggregation throughput, improving performance by 7.35× compared with BytePS and about 1.4× compared with ATP. In MoE training scenarios, HyNA reduces gradient synchronization time by up to 1.6× by eliminating fallback-induced stragglers, while maintaining model accuracy equivalent to FP32-based approaches. The ASIC analysis further shows that HyNA requires only about 2.9% additional chip area and 9.4 W power overhead in a 12.8 Tbps switch design.
This result is significant because it demonstrates that carefully designed heterogeneous switch hardware can overcome the tail latency problem in AI networking without sacrificing numerical correctness or requiring expensive external aggregation servers.
Personal thoughts
HyNA’s main contribution is that it focuses on a subtle but important problem in in-network aggregation: tail latency caused by rare exceptional events. Many previous designs optimize average throughput, but BSP training performance is often determined by the slowest worker. HyNA’s observation that a small number of collisions and overflows can dominate overall training time is insightful. The combination of RMT fast path and RISC-V exception path provides a practical balance between hardware efficiency and programmability, avoiding the limitations of both pure switch-based and host-based designs.
However, HyNA also leaves several open questions. The evaluation is mainly based on FPGA prototypes and ASIC synthesis rather than deployment on a real large-scale MoE training cluster. In addition, the system mainly focuses on the gradient synchronization phase of MoE training, while the All-to-All expert routing communication remains outside its scope. Future work could explore whether similar heterogeneous network hardware designs can accelerate other AI communication patterns and whether such architectures can be widely adopted in production-scale datacenters.