Title: Connex: Endpoint Mobility Primitives for Dynamic LLM Serving
Authors: Yanying Lin (University of Chinese Academy of Sciences, Penn), Vincent Liu, Tao Luo (University of Pennsylvania), ChengZhong Xu (University of Macau), Kejiang Ye (Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences)
Introduction
This paper studies GPU communication issues in dynamic and elastic LLM serving. Modern LLM inference systems dynamically add, remove, or migrate workers in response to changes in workload, failures, or resources. However, existing GPU communication frameworks such as NCCL are mainly designed for fixed topologies. Once the set of nodes changes, communication groups must be rebuilt and globally synchronized, which may cause several seconds of service interruption and significant P99 latency spikes. In addition, existing systems lack mechanisms for seamlessly migrating data that is already in transit and are also prone to network congestion caused by recovery and migration traffic. Therefore, they have difficulty meeting the low-latency and high-availability requirements of elastic LLM serving.
Key idea and contribution:
This paper proposes Connex, a communication system for elastic LLM inference that treats endpoint mobility as a fundamental capability of the communication layer. It allows workers to maintain continuous communication while joining, leaving, or migrating, thereby avoiding global pauses and reconfiguration.
Connex mainly achieves this through three mechanisms: versioned routing to quickly update invalid paths, a handoff protocol to ensure that data remains ordered and is not duplicated during migration, and a credit-based backpressure mechanism to control traffic and isolate critical inference traffic from background migration traffic.
Evaluation
The paper evaluates Connex on a 5-node GPU cluster using both synthetic scenarios and dynamic node-change scenarios that are close to production environments. The results show that, compared with baselines such as NCCL, Connex reduces P99 latency spikes by up to 85%, while the handoff time for large-model migration remains below 200 ms. Under moderate load, it maintains a 100% effective request completion rate during node changes. In addition, the local handoff latency of Connex is approximately 11 ms, which is significantly lower than the approximately 554 ms required to rebuild an NCCL communication group. This result is significant because it demonstrates that LLM services can continue operating stably during dynamic scaling or node failures without pausing the entire system for communication reconfiguration.
Q: If there is no mobility at all, does introducing this mechanism add extra overhead to the system?
A: There is a small amount of overhead. What we do is try to keep the related work off the common path as much as possible. However, we are not saying that there is absolutely no overhead. We embed some metadata in the packets so that, when the topology changes, we can still identify where the data belongs. For very large data transfers, this type of overhead is technically very small. Therefore, when there is no mobility, it does not affect the existing quality of service.
Q: What characteristics of large language model (LM) serving does your design exploit to improve system efficiency, and why? This question is not about endpoint mobility in general, but specifically about LM serving.
A: In this case, LM serving has several very important characteristics. First, from the demand side, we do not want latency to be too high. This is a type of workload that is highly latency-sensitive. It is different from training, where an interruption in the middle may not have as much impact. For online serving, however, we do not want latency to increase significantly. This is one of the most important factors in this problem, although the concept itself still has some generality. In addition, given how large-scale LM serving systems are currently operated, they usually contain multiple pipeline stages, which creates more opportunities or possibilities for topology changes. This is very important. The greater likelihood of topology changes is also the motivation for why we need to consider mobility.
Q: As you mentioned, at a much larger scale, for example 10,000 nodes, how can this mobility contract be maintained? In addition, how does it compose with collective communication operations that already exist in the cluster?
A: In this evaluation, we conducted experiments on a 5-GPU cluster. For very large clusters, I cannot be completely certain about how the system would behave in practice. However, I expect this approach to be useful because, in this work, we avoid reinitialization by making only local modifications and by recovering or reusing connections in the existing topology whenever possible. From this perspective, I think it should scale well and provide substantial benefits. In addition, because we separate data transmission from the control mechanisms related to mobility, this should allow clusters at the scale of 10,000 nodes to operate smoothly.
Personal thoughts
I think one valuable aspect of this paper is that it does not simply pursue higher communication bandwidth, but instead focuses on communication continuity and reliability in dynamic environments, which is very important for future large-scale and elastic LLM serving. In particular, handling endpoint migration, traffic control, and routing updates uniformly at the communication layer can reduce the complexity of upper-layer inference systems. However, the current Connex prototype is mainly validated on relatively small-scale clusters, while scalability is still left by the authors as future work. Therefore, it would be worthwhile to further study its scalability on clusters with hundreds or thousands of GPUs, as well as whether it can maintain stable performance under more frequent and complex node failure scenarios.