Title: ZipCCL: Efficient Lossless Data Compression of Communication Collectives for Accelerating LLM Training
Authors: Wenxiang Lin (Harbin Institute of Technology); Xinglin Pan and Ruibo Fan (The Hong Kong University of Science and Technology); Shaohuai Shi (Harbin Institute of Technology); Xiaowen Chu (The Hong Kong University of Science and Technology)
Scribe: Ziyi Wang (Xiamen University)
Introduction:
Communication is a critical bottleneck in large-scale distributed LLM training. Existing solutions rely on scheduling, topology-aware collectives, or lossy compression, but information loss may affect convergence and final accuracy. Lossless compression preserves bit-exact correctness, yet remains underexplored in training communication.
The main challenge is that compression and decompression overhead can outweigh the savings from reduced traffic. General-purpose GPU compressors such as DietGPU and nvCOMP are not communication-aware and introduce additional data movement and kernel overhead, potentially making compressed communication slower than uncompressed communication.
ZipCCL observes that LLM communication tensors, including activations, gradients, and parameters, typically follow near-Gaussian or log-normal distributions, making their exponents highly compressible. Figure 1 shows that lossless compression reduces Qwen3-A22B All-to-All traffic by approximately 1.33× on average. However, general-purpose compressors such as DietGPU and nvCOMP can still be slower than uncompressed NCCL because of their high processing overhead—a phenomenon termed the performance cliff.
Key idea and contribution:
ZipCCL is a lossless compressed communication library of collectives for LLM training and can serve as a drop-in replacement for NCCL. Its central idea is to exploit the near-Gaussian distribution of LLM training tensors to reduce communication volume while co-designing communication and compression so that compression overhead does not offset the communication benefit. The authors introduce three main techniques.
First, theoretically grounded exponent coding. ZipCCL determines the frequent exponent range directly from the tensor standard deviation, avoiding online histogram construction and sorting. For BF16 data, frequent exponents are encoded from 8 bits to 3 bits, while infrequent exponents retain their complete information separately, thereby guaranteeing lossless reconstruction. Ideally, this method reduces each value from 16 bits to approximately 11 bits, decreasing data volume by about 31%.
Second, GPU-optimized compression and decompression kernels. ZipCCL uses a communication-aware data layout, optimizes global-memory alignment and shared-memory access, and pipelines data movement with compression computation to reduce compression and decompression overhead.
Third, adaptive communication strategies. To address expert load imbalance in MoE training, ZipCCL divides All-to-All data into a static part and a dynamic part, allowing workers that finish computation earlier to begin communication in advance. For Reduce-Scatter, the system dynamically switches between the compressed path and the native communication path according to hardware performance and compression benefit.
As shown in Figure 5, ZipCCL integrates data compression, compressed-size synchronization, collective communication, and decompression into a unified workflow. Design-1 in Figure 5(b) first compresses the data and exchanges size information before transmitting the compressed data. Design-2 in Figure 5(c) divides the data into static and dynamic parts, transmits the static data first, and then exchanges size information and transmits the dynamic data. This allows workers that become ready earlier to communicate first, mitigating waiting time caused by imbalanced MoE expert computation.
Evaluation:
The authors primarily evaluate ZipCCL on a cluster containing up to 64 NVIDIA RTX A6000 GPUs with 200 Gb/s inter-node network bandwidth, and additionally validate it on a 16-GPU H800 cluster with 400 Gb/s network bandwidth. ZipCCL is integrated into Megatron-LM and TorchTitan. Configurations based on DeepSeek-V3 and Qwen3-MoE are used to evaluate All-to-All in MoE training, while Llama3-8B is used to evaluate All-Gather and Reduce-Scatter in FSDP training. The baselines include standard NCCL, DietGPU, and nvCOMP-ANS.
End-to-end performance. ZipCCL achieves approximately 1.16x training speedup on both DeepSeek-V3 and Qwen3-MoE and up to 1.18x speedup on Llama3-8B. Considering communication time alone, ZipCCL accelerates All-to-All by 1.25x and 1.24x on DeepSeek-V3 and Qwen3-MoE, respectively, and accelerates All-Gather by 1.35x on Llama3-8B. As the number of GPUs increases from 16 to 64, the end-to-end speedup on DeepSeek-V3 increases from 1.06x to 1.16x, indicating that lossless compression becomes more beneficial as communication accounts for a larger fraction of execution time. Compared with DietGPU, ZipCCL improves end-to-end throughput by approximately 1.26-1.83x, validating the importance of communication-aware GPU kernels.
Ablation study. Under MoE expert load imbalance, a basic compressed All-to-All design may lose all performance benefit and can even reduce training speed. Relative to this basic design, the static-dynamic data separation in ZipCCL improves end-to-end performance by 1.25-1.37x and communication performance by 1.40-1.55x. The adaptive switcher for Reduce-Scatter consistently matches or outperforms fixed schemes across seven hardware configurations, with an average improvement of approximately 1.39x.
Applicability analysis. The H800 experiments further show that ZipCCL is not beneficial on every link. On high-bandwidth intra-node NVLink, compression and decompression overhead can exceed the saved transmission time. On the more bandwidth-constrained inter-node RDMA path, ZipCCL substantially reduces total communication time. On 16 H800 GPUs, ZipCCL achieves approximately 1.13x end-to-end training speedup in both Megatron-LM and TorchTitan. Because the entire compression process is strictly lossless, ZipCCL does not alter model values or affect training convergence or final model quality.
By jointly designing for LLM data distributions, the GPU memory system, and collective communication workflows, lossless compression can be deployed on the critical path of training communication to deliver practical end-to-end speedups without risking model accuracy.
No Q&A
Personal thoughts:
I believe the central value of this paper lies in how it reframes the problem: the focus is not merely on making lossless compression itself run faster, but on enabling it to deliver genuine end‑to‑end benefits in collective communication. Rather than confining the work to codec optimization, the authors adopt a co‑design approach that accounts for LLM tensor distributions, GPU memory systems, and collective communication workflows, truly integrating lossless compression—previously considered too expensive—into the critical path of training communication. This system‑oriented rethinking is instructive.