Title: DynamiQ: Accelerating Gradient Synchronization using Compressed Multi-hop All-reduce
Authors: Wenchen Han (University College London); Shay Vargaftik (VMware Research by Broadcom); Michael Mitzenmacher (Harvard University); Ran Ben Basat (University College London and Broadcom)
Scribe: Ziyi Wang (Xiamen University)
Introduction
Distributed Data Parallel (DDP) is a widely used parallelization strategy for large-model training and fine-tuning. Each worker independently computes its local gradients and then synchronizes them through a multi-hop all-reduce topology, such as a ring or butterfly. As model sizes and cluster scales continue to grow, gradient aggregation has gradually become a major communication bottleneck. Gradient compression offers an intuitive optimization approach by reducing the amount of gradient data transmitted and thereby shortening communication time.
Gradient compression is a direct way to alleviate this bottleneck, but representative existing methods, such as THC, Omni-Reduce, and the recently proposed MXFP microscaling formats, were largely designed for parameter-server architectures and do not effectively accommodate the characteristics of multi-hop all-reduce. The fundamental difference is that gradients are accumulated hop-by-hop into partial sums during multi-hop aggregation. If an intermediate worker recompresses a partial sum after every hop, quantization error accumulates across hops and can impair model convergence and final accuracy. Conversely, increasing the representation bitwidth of partial sums to preserve accuracy weakens the communication benefit of compression. This dilemma calls for a compression mechanism designed specifically for multi-hop all-reduce.
Key idea and contribution
DynamiQ addresses this problem with a gradient quantization framework co-designed with multi-hop all-reduce. Its central idea is variable bitwidth allocation: coordinates with larger magnitudes receive more bits, while smaller ones receive fewer, optimizing the accuracy-bandwidth trade-off under a fixed communication budget. To avoid per-coordinate metadata overhead and alignment issues, DynamiQ uses a hierarchical organization: consecutive coordinates form groups, multiple groups form super-groups. All entries in a super-group share a bitwidth; all entries in a group share a scale factor.
The overall workflow consists of two phases. Figure 2 overviews this two-phase workflow.
In the statistics collection phase, each worker computes the mean and squared ℓ₂ norm of each super-group and synchronizes these statistics through a lightweight all-reduce, which transfers less than 1% of the original gradient traffic. Each worker then derives the same bitwidth allocation and reorders the super-groups by bitwidth, placing data with the same bitwidth contiguously for efficient GPU processing.
In the main all-reduce phase, each intermediate worker decompresses the incoming partial sum, accumulates its local gradient, recompresses the result, and forwards it to the next hop. This hop-by-hop recompression adapts the quantization to the evolving partial sums, thereby limiting error accumulation. After the all-reduce completes, each worker restores the original super-group order and adds back the previously subtracted mean.
In addition to variable bitwidth allocation, DynamiQ incorporates three auxiliary techniques to further reduce compression error:
- Non-uniform quantization places more quantization levels near zero to match the prevalence of small-magnitude values in actual gradient distributions.
- Hierarchical quantization applies a second level of quantization to group-level scale factors, reducing metadata overhead while preserving unbiasedness.
- Correlated rounding uses shared randomness across workers to encourage rounding errors to cancel during aggregation, reducing variance by up to 2x.
At the system level, the authors use a fused CUDA kernel that combines decompression, accumulation, and recompression into a single kernel. This avoids materializing intermediate tensors and eliminates additional global-memory accesses, keeping the computational overhead extremely low.
Evaluation
The authors implement DynamiQ on top of PyTorch DDP and NCCL P2P and evaluate it on several LLM fine-tuning tasks, including BERT-large, Gemma 1B, and LLaMA 1B, using both ring and butterfly all-reduce topologies.
The baselines include uncompressed BF16 as well as Omni-Reduce, THC, MXFP4, MXFP6, and MXFP8.
The primary metric is time-to-accuracy (TTA), rather than compression ratio or per-iteration communication throughput alone. The reason is that even when each communication round is fast, compression error may require additional iterations for convergence, in which case the total training time may not decrease.
In the ring all-reduce experiments shown in Figure 4, DynamiQ achieves better TTA across all tasks. For example, on Gemma 1B, DynamiQ reaches the target perplexity approximately 18% and 28% faster than MXFP8 and MXFP6, respectively. Some baselines converge even more slowly than BF16 or fail to reach the target accuracy because of their larger compression error.
Overall, DynamiQ improves TTA by up to 34.2% over the best existing compression method and is the only evaluated method that consistently reaches approximately 99.9% of the BF16 baseline accuracy while significantly accelerating training. Its compression error, measured by vNMSE, is approximately 2.5-3x lower than that of MXFP8 and substantially lower than those of MXFP4, THC, and Omni-Reduce.
Q: I have a question about how you account for or abstract hardware differences when constructing the optimal algorithm—for example, differences in interconnects or between scale-up and scale-out domains.
A: From a network-topology perspective, there is essentially no difference between scale-up and scale-out. You just need to encode components such as NVSwitches and network switches. The real differences are at the hardware-functionality level. For example, if we need to support multicast, switch-based computation, or in-memory computation, our usage variables can capture these differences, because the corresponding constraints are defined only for specific vertices, that is, specific hardware components.
Personal thoughts
The paper’s key insight is that gradient compression should be designed together with the communication architecture. In multi-hop all-reduce, repeated accumulation and recompression can propagate quantization errors, so simply changing the topology is insufficient.
DynamiQ addresses this by co-designing quantization, communication topology, and GPU implementation. Its use of time-to-accuracy is also valuable because it captures both communication efficiency and convergence quality.