Title: KVServe: Service-Aware KV Cache Compression for Communication-Efficient Disaggregated LLM Serving
Authors: Zedong Liu, Xinyang Ma, Dejun Luo (University of Chinese Academy of Sciences), Hairui Zhao (Institute of Computing Technology, Chinese Academy of Sciences), Bing Lu (Institute of Computing Technology, Chinese Academy of Sciences), Wenjing Huang (University of Chinese Academy of Sciences), Yida Gu, Xingchen Liu, Zheng Wei (Institute of Computing Technology, Chinese Academy of Sciences), Jinyang Liu (University of California, Riverside), Dingwen Tao, Guangming Tan (Institute of Computing Technology, Chinese Academy of Sciences)
Introduction
This paper studies KV Cache communication and compression optimization in disaggregated large language model (LLM) inference. As context lengths and system scales increase, cross-node KV Cache transfer is gradually becoming a major bottleneck in end-to-end latency. Although existing KV compression methods can reduce the amount of transferred data, most of them use fixed configurations and cannot easily adapt to dynamically changing workloads, network bandwidth, SLOs, and quality requirements. Compression overhead may even increase latency in some cases. Therefore, an adaptive approach that can dynamically select compression strategies according to the actual service state is needed.
Key idea and contribution
KVServe mainly consists of three components: a modular compression strategy pool, a Bayesian analysis engine, and a service-aware online controller. The Bayesian analysis engine efficiently searches a large number of candidate strategies and constructs a Pareto-optimal candidate set. The online controller then selects an appropriate strategy according to the real-time service state, thereby reducing KV communication latency while satisfying service quality and service-level objective (SLO) requirements.
Evaluation
The paper integrates KVServe into vLLM and evaluates it across various datasets, models, GPUs, and network configurations. The results show that, compared with baselines and existing KV compression methods, KVServe achieves up to a 9.13× speedup in Prefill/Decode disaggregation scenarios and reduces time to first token by up to 32.8× in KV disaggregation scenarios. This result is significant because it demonstrates that adaptive KV compression can substantially reduce communication overhead and inference latency across different real-world serving environments, thereby improving the efficiency and scalability of large-model serving systems.
Q: You mentioned using a bandit algorithm to correct runtime drift. However, as I understand it, runtime drift mainly comes from bandwidth. For a given infrastructure, shouldn’t the bandwidth configuration be fixed? If so, why not incorporate the bandwidth configuration into the offline profiling stage from the beginning?
A: Our offline profiling search engine combines the speed, compression ratio, and accuracy of different components to form three-dimensional profiling results. During the online decision-making stage, the appropriate profile is selected based on the current bandwidth because bandwidth may change during inference.
In addition, users and their requirements may also change during runtime inference. Therefore, our online decision-making mechanism performs adaptive selection to satisfy user requirements and runtime inference requirements. In other words, bandwidth changes at runtime, so it is not a fixed configuration.
Personal thoughts
This paper does not simply propose a new KV compression algorithm. Instead, it treats compression strategy selection as a system problem that changes dynamically with the service state and simultaneously considers bandwidth, latency, and model quality, which is more consistent with real-world deployment scenarios. However, KVServe requires offline analysis and candidate strategy search in advance, which still introduces some computational cost, and new models or workloads may require re-analysis. Future research could further investigate how to reduce the cost of offline analysis and whether the system can adapt more quickly to previously unseen workloads and network environments, thereby enabling more general online adaptive KV compression.