Title: Efficient Remote KV Cache Reuse with GPU-native Video Codec
Authors: Liang Mi (Nanjing University and Institute for AI Industry Research (AIR), Tsinghua University), Weijun Wang (Institute for AI Industry Research (AIR), Tsinghua University), Jinghan Chen (Nanjing University), Ting Cao (Institute for AI Industry Research (AIR), Tsinghua University), Haipeng Dai (Nanjing University), Yunxin Liu (Institute for AI Industry Research (AIR), Tsinghua University)
Introduction
This paper studies the problem of inefficient remote KV Cache reuse in bandwidth-constrained environments. Remote KV Cache reuse can avoid redundant computation, but it requires transferring a large amount of KV Cache from remote storage. When network bandwidth is low, the transfer overhead increases significantly. Although existing compression methods can reduce the amount of transferred data, decompression usually relies on CUDA cores, which can compete with LLM inference for GPU resources, or requires additional expensive hardware such as SmartNICs. Therefore, their practical benefits are limited.
Key idea and contribution:
This paper proposes KVCodec, which uses the otherwise idle dedicated video codec hardware in GPUs to compress and decompress KV Cache, thereby avoiding the use of general-purpose GPU computing resources needed by LLM inference.
KVCodec mainly includes two key designs. The first is a tensor layout suitable for video encoding, which improves the lossless video compression ratio by reorganizing KV Cache data. The second is an efficient remote KV fetching mechanism, which reduces the impact of resource contention and network fluctuations through KV-fetch-aware scheduling, dynamic resolution, and pipelined transmission and decoding.
Evaluation
The paper evaluates KVCodec on models of different scales, including LWM-7B, Yi-34B, and Llama-70B, across multiple GPUs and network environments ranging from 1 to 40 Gbps. The results show that, compared with existing methods, KVCodec reduces time to first token (TTFT) by up to 3.51× while maintaining model accuracy. At the same time, its compression ratio is approximately 2.17× higher than that of CacheGen. This result is significant because it demonstrates that existing but typically idle video codec hardware in GPUs can be directly utilized to substantially improve the efficiency of remote KV Cache reuse without requiring additional expensive hardware.
Q: Your work studies remote KV Cache reuse, but compression is performed offline. Is the system still applicable to Prefill-Decode disaggregation scenarios, where KV Cache is essentially generated online?
A: Due to hardware limitations, only some GPUs are equipped with dedicated hardware encoders. Currently, online compression is still a bottleneck. In future work, we plan to explore pipelined layer-by-layer GPU encoding to reduce encoding latency in online scenarios.
Q: Newer GPUs, such as Blackwell and later models, already include dedicated compression and decompression engines that can be used by the NV compiler. Given that dedicated engines already exist, where do you see the future value of this work? Do you still think video encoding codecs will be helpful?
A: We can discuss this question further offline. You can also directly contact the co-authors.
Personal thoughts
I think one interesting aspect of this paper is that it identifies long-underutilized video codec resources in GPUs and applies them to LLM system optimization, which is a clever approach to hardware resource reuse. At the same time, it does not only optimize the compression algorithm, but also considers the complete system pipeline, including data layout, scheduling, and network transmission. Future work could further investigate how to improve the utilization of video codec hardware and how to extend this approach to more GPU architectures and online KV Cache transfer scenarios.