Harvest: Adaptive Photonic Switching Schedules for Collective Communication in Scale-up Domains
Authors: Mahir Rahman, Samuel Joseph, Nihar Kodkani (Purdue University); Behnaz Arzani (Microsoft Research); Vamsi Addanki (Purdue University)
Introduction
Collective communication increasingly limits the performance of scale-up AI systems. Silicon photonics offers high bandwidth, longer reach, and better energy efficiency than electrical interconnects, but its circuit-switched nature creates a fundamental trade-off. A static topology avoids reconfiguration overhead but suffers from multi-hop forwarding, congestion, and propagation delay; reconfiguring before every collective step can establish direct optical paths but repeatedly pays the switching cost. Existing approaches often choose one extreme or optimize an aggregate traffic matrix, which ignores the ordered steps and data dependencies of collective algorithms. Harvest therefore asks when and how a photonic interconnect should be reconfigured during a known collective to minimize its total completion time.
Key idea and contribution
Harvest represents a collective as an ordered sequence of communication matrices and message sizes. It extends the classic $\alpha$-$\beta$ model to explicitly include startup latency, propagation delay, congestion through maximum concurrent flow, and photonic reconfiguration delay. Its key policy is selective reconfiguration: the topology changes only when the resulting reduction in congestion and path length outweighs the switching cost.
For arbitrary collectives, Harvest partitions the communication steps into contiguous intervals. A Mixed-Integer Second-Order Cone Program finds the best single topology for each interval, while dynamic programming chooses the interval boundaries and number of reconfigurations that minimize total completion time. Schedules can be synthesized offline and cached. For Recursive Doubling AllReduce, the authors prove that the topology matching the first step of an interval remains connected and is optimal for all steps in that interval. This reduces topology search to $O(1)$ and yields an optimizer-free schedule synthesis algorithm with $O((\log n)^4)$ complexity.
Evaluation
The evaluation combines Astra-Sim packet-level simulation, Gurobi-based numerical optimization, and hardware emulation using eight GPUs, eight BlueField-3 NICs, and 100Gbps optical transceivers. It covers 8–64-GPU networks, 2–6 ports per GPU, reconfiguration delays from 10ns to 10ms, and multiple AllReduce, All-to-All, and Broadcast algorithms. For 1–256KB messages and sub-microsecond reconfiguration, Harvest accelerates Recursive Doubling, Swing, and All-to-All by up to 6.4×, 4.7×, and 20× over static topologies. With 100μs reconfiguration and 1–256KB messages, it improves over reconfigure-every-step BvN schedules by 7.3×, 10×, and 5.3× on average, respectively. In the intermediate regime, selective reconfiguration provides roughly 2× speedup over the better of the static and BvN baselines. Hardware emulation shows similar trends and up to 3× speedup over a static ring. Recursive Doubling schedule generation takes under 20μs for 64 GPUs and under 35μs on average for 1,024 nodes. This result is significant because it demonstrates that the practical value of photonic reconfigurability comes from adapting the switching frequency to both the collective and the hardware delay, rather than always keeping the topology fixed or always reconfiguring it.
Q&A
Q1: Harvest assumes that the demand matrix for each step is available, but dynamically collecting such information can be difficult in practice. How is the demand obtained?
A1: The speaker explained that Harvest targets collectives whose algorithms are already known. For structured operations such as AllReduce, the communication partners, step order, and message sizes can usually be derived from the collective algorithm and model configuration rather than inferred from live network traffic. The resulting schedule can be precomputed and cached. For algorithms such as Recursive Doubling, where the topology search space reduces to $O(1)$, a schedule may also be generated quickly during training.
Q2: The evaluation considers reconfiguration delays up to about 10ms, while synchronizing some optical transceivers may take hundreds of milliseconds. Can Harvest handle such devices?
A2: The speaker said that reconfiguration delay is an input parameter. Operators can supply a larger measured value, and Harvest will synthesize a schedule under that constraint. As the cost increases, the framework selects fewer reconfigurations and may eventually fall back to a static topology. Thus, the formulation is not inherently limited to 10ms, although the paper does not directly validate real devices with hundreds-of-milliseconds switching delays.
Q3: What are the algorithmic complexity and supported network scale, and why does Recursive Doubling no longer require the general optimizer?
A3: For Recursive Doubling, the demand matrix at every step is known, and the paper proves that the topology matching the first step of an interval is the optimal candidate for that interval. Topology search therefore becomes $O(1)$, leaving only the dynamic program that decides when to reconfigure. The general optimizer is evaluated on 8–64 GPUs, while the specialized algorithm is evaluated up to 1,024 nodes with average synthesis time below 35μs. The 1,024-node result relies on Recursive Doubling’s geometric structure and should not be interpreted as the scalability of the general MISOCP solver.
Q4: Has Harvest been evaluated with Mixture-of-Experts models, whose expert routing and All-to-All demand may change on every pass?
A4: The speaker said that the general framework can accept an All-to-All demand and synthesize a schedule, but acknowledged that MoE traffic changes dynamically. Whether reconfiguration is worthwhile depends on the current traffic pattern and whether the congestion reduction exceeds the switching cost. No concrete MoE experiment or complete online scheduling mechanism was presented in the answer, so demand prediction, synthesis latency, and end-to-end gains under dynamic expert routing remain open questions.
Personal thoughts
The paper’s strongest contribution is its reframing of photonic switching: the key question is not whether to reconfigure, but when to do so. Harvest places collective dependencies, congestion, propagation delay, and hardware switching cost in one optimization framework, providing a principled choice between static and per-step reconfiguration.
The main limitation is that the evaluation relies on simulation and hardware emulation rather than an end-to-end reconfigurable photonic system, while the general MISOCP formulation is primarily demonstrated at up to 64 GPUs. Important next steps include online scheduling for dynamic MoE traffic and joint synthesis of the collective algorithm, routing, and photonic topology.
Paper: https://doi.org/10.1145/3789240.3829166
Open-access version: [2602.09188] Harvest: Adaptive Photonic Switching Schedules for Collective Communication in Scale-up Domains