Title: Opus: Photonic Rail-Optimized Fabric in ML Datacenters
Authors: Eric Ding (Cornell University); Barry Lyu (University of Michigan); Bhaskar Kataria, Rachee Singh (Cornell University)
Reviewer: Liu Yipeng (first-year master’s student, SNG)
Introduction
Large-scale machine learning training usually adopts a rail-optimized fabric, where GPUs with the same rank in different scale-up domains communicate through separate rails for data parallelism (DP) and pipeline parallelism (PP). Existing solutions rely on high-radix electrical switches to provide all-to-all connectivity, and the switch chips, queuing, packet parsing, and electro-optical conversion add substantial cost and power consumption. Optical circuit switches (OCSs) can provide low-power, low-latency end-to-end optical paths, but at any moment they mainly support one-to-one connections and are therefore difficult to use directly for mixed-parallel communication that includes DP, PP, and TP.
Opus keeps the rail communication semantics and GPU mapping, replaces only the electrical switches in scale-out, and reconfigures optical paths at the boundaries between parallel phases of training iterations. Communication phases of different parallel dimensions have clear dependencies and non-overlapping windows, so the same physical ports can be reused as different rail topologies in different phases.
Core Idea and Contributions
The core idea of this paper is to reuse optical paths by exploiting the time gaps between different parallel dimensions during training iterations. An optical switch can only establish a limited number of one-to-one connections at a time, but DP, PP, and EP communication do not happen simultaneously. The same set of GPU ports can serve DP in one phase and switch to PP in the next. Therefore, the system does not need a separate static rail for each parallel dimension; instead, it temporarily changes the optical paths according to the current communication phase.
The contribution of the paper is an application-layer control plane: a runtime shim observes communication operations, an orchestrator for each rail generates the required topology, and a controller coordinates reconfiguration and concurrent access. The system also prepares the circuit for the next phase before the current phase ends, hiding part of the OCS latency inside communication gaps. The prototype makes no changes to RoCEv2, NCCL, or the Mellanox driver, and the design is validated through a physical OCS testbed, supercomputer simulation, and large-scale simulation.
Experimental Evaluation
Opus is evaluated at three scales: a physical testbed with a Polatis OCS, a simulation on the Perlmutter supercomputer with up to 64 GPUs, and large-scale simulations in Astra-Sim with up to 2,048 GPUs.
On the real testbed, four servers and two optical rails run Llama-3 training. The OCS optical link switches in about 25 ms; the experiment finds that the end-to-end delay is mainly caused by the NIC firmware reporting link-up, which takes about 3 seconds after disabling auto-negotiation. This bottleneck is not a fundamental limitation of Opus or the OCS. In the Perlmutter simulation, with 50 ms OCS reconfiguration latency, Opus’s training step time is only 1.05× to 1.08× that of the native electrical network; with preconfiguration enabled, it drops to 1.01× to 1.02×. With zero reconfiguration latency, the extra control-plane overhead is about 6.13%, and preconfiguration reduces it to 0.79%. Large-scale simulations show that under production-relevant OCS latencies of up to 100 ms, Opus increases iteration time by less than 6.7%, while reducing network infrastructure cost by up to 4.27× and network power consumption by more than 23× compared with an electrical rail-optimized fabric. Sensitivity analysis across models, parallelism levels, link bandwidth, and reconfiguration latency shows that the wider the idle window between communication phases, the better preconfiguration hides the latency; workloads with only one scale-out parallel dimension do not need frequent reconfiguration.
Q&A
Q1: In your experiments, one OCS is responsible for two rails, so you can obviously coordinate the two rails because they are switched at the same time by the same switch. What are your ideas for doing that coordination when the rails use different switches? In other words, what does the control plane from the GPUs to the switches look like, and how do you coordinate switching across switches according to the data flow?
A1: Our testbed cannot show the synchronization issue across different switches. For a larger-scale system, we think that because the switches are located on different rails and different communication groups are mapped to their own rails, synchronization across switches may not directly affect the communication of any individual group. Each group is allocated to a corresponding rail. However, synchronization across different rails will still affect overall coordination, especially the switching between different communication groups. This is something we have not studied in depth yet, and it needs further investigation, especially regarding its impact on application performance.
Q2: In your simulations, you showed that as the job size grows, the communication time goes down. I understand that under the current parameters, about 95% of the communication windows are still long enough. But to me, the obvious question is whether there is a regime where it is not obvious whether you should wait for reconfiguration or instead take the performance hit of running the collective on a suboptimal network. Have you thought about how to determine that threshold and how to decide when to skip reconfiguration and keep using the old network?
A2: Our experiments show that at larger scales, the communication windows are usually longer than 1 second. The reconfiguration times used in our simulations are around 5 ms to 100 ms. We believe future technologies may reduce reconfiguration time further, to around 1 ms or even below 1 ms, which should help address this issue. As for the decision threshold you mentioned, we have not yet implemented a full adaptive policy in the control plane. Intuitively, the system needs to compare “the time spent waiting for reconfiguration” with “the extra time caused by using the old topology to finish the current communication,” and only reconfigure when the former is worthwhile.
Personal View
Opus turns the physical constraint of optical switching into a scheduling opportunity for training phases. The key is not to chase static all-to-all connectivity, but to reuse a limited set of connections by exploiting the time structure of communication phases. The paper is also refreshingly honest about the current NIC firmware’s link-up delay, which matters a lot for real deployment. Future work still needs to handle dynamic job arrivals, failure switching, multi-tenant isolation, and imperfect overlap between training communication phases. If OCSs and NICs can provide fast, incremental reconfiguration interfaces, Opus’s benefits will be much easier to realize in production clusters.