Title: Turbo: Efficiently Serving Long-Context Large Language Models with In-Network Aggregation
Authors: Ying Wan, Yuchen Xu, Chuwen Zhang, Yingsheng Huang, Yong Feng, Wenquan Xu, Jialin Li, Mingwei Xu, Wenfei Wu, Congcong Miao (Southeast University; State Key Laboratory of Internet Architecture; Peking University; Tsinghua Unigroup; Tsinghua University; National University of Singapore; Beijing Key Laboratory of Software and Hardware Cooperative Artificial Intelligence Systems)
Introduction
As large language models (LLMs) evolve to support increasingly long contexts, the KV cache grows linearly with context length and becomes a critical memory bottleneck during inference. For example, serving GPT-3 175B with a 100K-token context requires about 500 GB of KV cache, far exceeding the memory capacity of a single GPU. Sequence Parallelism (SP) addresses this problem by distributing KV cache blocks across multiple GPUs, but existing approaches introduce new communication and computation bottlenecks. Pass-KV transfers large KV blocks and suffers from high network traffic and serial computation, while Pass-Q is more suitable for decoding but requires the master GPU to broadcast queries, collect partial attention results, and perform aggregation. As the number of workers increases, this centralized design creates GPU computation overhead, additional memory usage, redundant network traffic, and severe incast congestion. Therefore, efficiently supporting long-context inference requires not only distributing KV cache storage but also eliminating the communication and aggregation bottleneck at the master GPU.
Key idea and contribution:
The authors propose Turbo, an in-network aggregation system for accelerating long-context LLM inference. Turbo builds on the Pass-Q approach but moves query broadcast and attention aggregation from the master GPU into programmable network switches. Instead of sending the same query separately to every worker, the master sends only one copy and the switches multicast it to workers. Each worker computes partial attention using its locally stored KV cache, and the switches progressively aggregate these partial results before returning only the final attention result to the master. This reduces the master-side communication complexity from O(nd) to O(d), while also reducing GPU computation, memory overhead, redundant traffic, and incast congestion.
Implementing attention aggregation on commodity programmable switches is challenging because RMT switches have limited memory, integer-only ALUs, and strict pipeline-stage constraints. Turbo addresses these limitations with three key techniques. First, online table-based aggregation transforms the original global scaling-reduce operation into incremental pairwise aggregation and approximates complex nonlinear operations such as exponentiation and division using compact lookup tables. Second, rolling forward state update propagates aggregation states through the switch pipeline to overcome restrictions on cross-stage state updates without sacrificing line-rate processing. Third, Turbo constructs a load-aware aggregation tree by formulating tree construction as a Maximum Flow problem, distributing aggregation workloads across switches to prevent overloaded switches from becoming stragglers. The authors further implement Turbo on Tofino2 switches and FPGA hardware and integrate it with LLM serving frameworks, demonstrating the practicality of using the network itself as an accelerator for LLM inference.
Evaluation
The authors evaluate Turbo using a real testbed with eight RTX 3090 GPUs and a Tofino2 programmable switch, together with large-scale NS-3 simulations, accuracy experiments, and system overhead analysis. On the real hardware testbed, Turbo reduces end-to-end decoding latency by up to 37% as KV cache size increases. Large-scale simulations further show that Turbo can reduce master-side network traffic by one to five orders of magnitude and substantially improve inference latency and scalability under high concurrency. Turbo also maintains negligible accuracy loss: experiments with 50K–250K-token contexts report KL divergence below 0.01, while its 11-bit lookup tables consume about 43.8% of switch SRAM and can be configured in less than 40 μs. These results demonstrate that Turbo can achieve significant performance improvements while remaining feasible on existing programmable switch hardware.
This result is significant because it shows that programmable networks can do more than simply transport LLM inference traffic: they can directly execute parts of complex attention aggregation, removing communication and computation bottlenecks from GPUs and enabling long-context inference to scale more efficiently.
Q: Is programmability necessary for Turbo, and can its design be applied to other hardware architectures such as ASICs?
A: Turbo’s fundamental idea does not depend on a specific programmable switch architecture. The key concept of offloading query broadcast and attention aggregation from GPUs to the network can also be applied to other hardware platforms. However, Turbo’s current implementation is specifically optimized for RMT-based programmable switches such as Tofino, so mechanisms such as rolling forward state update are designed around their pipeline and state-access restrictions. If Turbo is implemented on ASICs, SmartNICs, FPGAs, or other network hardware, the overall in-network aggregation idea can remain the same, but the detailed implementation would need to be redesigned according to the capabilities and constraints of the target hardware.
Personal thoughts
I think the most interesting aspect of Turbo is that it goes beyond simply using the network to accelerate data movement. Attention aggregation involves nonlinear operations such as exponentiation and division, which are fundamentally different from the simple sum, min, or max operations commonly supported by previous in-network computing systems. Turbo shows that through algorithm-hardware co-design, complex attention aggregation can be transformed into online pairwise operations and lookup-table-based computations that fit within the constraints of programmable switches. In particular, the combination of online aggregation and rolling forward state updates is an interesting example of redesigning an algorithm around the capabilities of network hardware rather than treating the switch as only a communication device.
However, there are still several open questions. Turbo’s real hardware evaluation uses only eight GPUs, so its performance and robustness in production-scale clusters with hundreds or thousands of GPUs remain to be demonstrated. Its lookup-table design also introduces a trade-off between numerical precision and limited switch SRAM resources, while dynamic workloads and failures could make aggregation-tree management more complicated at larger scales. It would be interesting to explore whether similar ideas could be implemented using SmartNICs, FPGAs, or specialized ASICs with richer computation capabilities. More broadly, Turbo suggests an interesting direction in which the network becomes an active accelerator for AI workloads, and future work could investigate what other LLM inference operations can be efficiently offloaded into the network.