Title: DualPath: Accelerating Agentic LLM Inference by Harvesting Disaggregated KV-Cache Storage I/O
Authors: Yongtong Wu (Peking University, DeepSeek-AI), Shaoyuan Chen (Tsinghua University, DeepSeek-AI), Rilin Huang (Peking University), Yixuan Tan (DeepSeek-AI), Yinmin Zhong (Peking University, DeepSeek-AI), Mingxing Zhang (Tsinghua University), Xin Jin (Peking University), Panpan Huang (DeepSeek-AI)
Introduction
This paper studies the KV-Cache storage I/O bottleneck in Agentic LLM inference. In multi-turn, long-context tasks, KV-Cache hit rates are high, and a large amount of cached data needs to be loaded from external storage. This saturates the storage network bandwidth of Prefill nodes, while the bandwidth of Decode nodes remains underutilized. Existing methods mainly reduce the amount of KV-Cache data or optimize storage methods, but they cannot address the I/O bandwidth imbalance across different inference nodes. Therefore, this paper focuses on how to utilize idle bandwidth to improve KV-Cache loading efficiency.
Key idea and contribution:
This paper proposes DualPath, whose core idea is to introduce a new KV-Cache loading path. In addition to the traditional “storage-to-Prefill” path, KV-Cache can first be loaded to Decode nodes and then transferred to Prefill nodes through a high-speed RDMA network, thereby utilizing the idle storage bandwidth of Decode nodes.
DualPath mainly consists of dual-path KV-Cache loading, a traffic management mechanism, and an adaptive request scheduler. The system can dynamically select the KV-Cache loading path, balance GPU and network resources, and prevent KV-Cache transfers from interfering with normal model communication.
Evaluation
The paper evaluates DualPath on three models and real-world Agentic workloads. The results show that DualPath improves throughput by up to 1.87× in offline inference and by an average of 1.96× in online serving, while satisfying service-level objectives (SLOs). This result is significant because it demonstrates that Agentic LLM inference efficiency can be substantially improved simply by making better use of existing network resources, without relying on additional expensive hardware bandwidth.
Personal thoughts
I think one valuable aspect of this paper is that it does not simply improve hardware performance. Instead, it identifies and utilizes the previously idle I/O resources of Decode nodes and addresses system resource imbalance by redesigning the data transfer path. However, the approach relies on RDMA, network traffic isolation, and complex scheduling mechanisms, which impose certain requirements on the network architecture of real-world clusters. Future work could further investigate how to make DualPath adaptable to a wider range of hardware and network environments.