TurboBus: Pooling PCIe Bandwidth for LLM Workloads via Scale-Up Fabrics
Authors: Xinyu Yang, Kaiqiang Xu, Kai Chen (Hong Kong University of Science and Technology)
Reviewer: Haodong Chen (Second-year Ph.D. Student, SNG)
Introduction
GPU memory offloading allows LLM systems to place model weights, KV caches, gradients, or optimizer states in host memory, but it shifts the performance bottleneck to CPU–GPU transfers. In the workloads studied in the paper, PCIe transfers account for up to about 90% of inference time in FlexGen and 24% of iteration time in ZeRO-Offload. Surprisingly, these workloads are bottlenecked by an individual GPU’s PCIe link while more than 60% of the server’s aggregate PCIe bandwidth remains idle. Their bursty, phase-driven transfers leave different links idle at different times, both across ranks of one job and across co-located jobs, yet static per-GPU link ownership prevents a busy GPU from borrowing this capacity. Existing relay-based systems are mostly limited to intra-job, host-to-device model loading, use store-and-forward execution, or rely on page-fault-driven migration; they do not provide a general solution for bidirectional, cross-job GPU memory offloading with isolation and concurrency control.
Key idea and contribution
TurboBus pools PCIe bandwidth through the server’s scale-up fabric. A transfer may use both the source GPU’s direct PCIe path and relay paths that first send data over NVLink/NVSwitch to neighboring GPUs and then use their idle PCIe links. A privileged per-node daemon has visibility into all GPUs and performs relaying on behalf of unprivileged applications, preserving process isolation. Host memory is shared through memory mapping and device memory through CUDA IPC, so the data plane remains zero-copy. A lightweight client library exposes tensor allocation and transfer APIs and requires only localized application changes.
The data plane splits large transfers into fixed-size blocks and streams them through relay GPUs with double buffering. This overlaps NVLink and PCIe, bounds relay memory at 64 MB on the V100 testbed, and enables block-level allocation that fills otherwise idle paths with blocks from other requests. The scheduler runs H2D and D2H transfers concurrently by exploiting NVLink’s bandwidth headroom over PCIe and synchronizes their launches to avoid CUDA runtime interference. It also chunks large transfers to bound head-of-line blocking for small requests and uses token-gated admission to prevent multiple large transfers from slowing one another. The authors implement a roughly 3,900-line prototype and identify a broader systems principle: resources that are individually bottlenecked but collectively underutilized can be pooled by borrowing paths over a faster fabric.
Evaluation
The evaluation uses a four-GPU DGX V100 as the primary testbed and validates relay behavior on RTX 3090 and eight-GPU H800 servers. Microbenchmarks show that profiled block sizes bring two-link relay throughput within roughly 5%–6% of the ideal, block-level allocation improves throughput by 20%, and chunking plus token control reduces completion time by 15%–27%. For on-demand model loading, TurboBus reduces median TTFT by 17%–25% in a multi-tenant serverless emulation and reduces TTFT by up to 40% in isolation, within 5% of the analytical optimum; however, contention can increase P99 TTFT by up to 10% relative to PipeSwitch+DeepPlan. For FlexGen with 90% of the KV cache offloaded, TurboBus improves throughput by up to 1.6× while imposing less than 1% overhead on the co-located workload. It also accelerates four-GPU ZeRO-Offload training by 4%–7%. This result is significant because TurboBus converts otherwise stranded, already-deployed PCIe capacity into application-level gains without requiring new PCIe hardware, while revealing that scale-up fabrics can serve as a general intra-node I/O pooling layer rather than only a GPU collective-communication network.
Q&A
Q1: If several GPUs simultaneously select the same idle GPU as a relay, could its NVLink ingress and PCIe egress become a new congestion hotspot? How does TurboBus coordinate such bandwidth borrowing within a node?
A1: The speaker agreed that this is an important issue and explained that it also involves higher-level job placement and cluster scheduling. TurboBus’s benefits depend on the resource profile of the co-located workload: a compute-bound job uses less PCIe bandwidth and is therefore a better bandwidth donor. A follow-up direction is to co-locate jobs with complementary resource demands so that pooling gains are maximized without creating hotspots. The speaker also mentioned the group’s FuseLink work on related cross-node bandwidth pooling, but did not claim that the current TurboBus design fully resolves simultaneous contention for the same relay path.
Q2: TurboBus uses blocks of roughly 16 MB for pipelined transfers. Under a higher tensor-parallelism degree, where NVLink may already be heavily utilized, does TurboBus remain effective, and what is its impact on P99 latency?
A2: The speaker explicitly said that this experiment had not been performed and therefore did not provide a conclusion. This means the reported results should not be directly extrapolated to highly tensor-parallel workloads with saturated NVLink. Competition between relay traffic and tensor-parallel communication, particularly its effect on tail latency, remains to be evaluated.
Q3: Is TurboBus applicable to disaggregated architectures, for example over RDMA?
A3: The speaker said that disaggregated architectures were not a focus of this work and were not directly evaluated. Extending the idea to large-scale disaggregated inference would require an additional scheduling layer that jointly places GPU-demanding and bandwidth-demanding workloads so that they can borrow complementary idle resources. The answer therefore framed disaggregation as a possible extension rather than a capability already demonstrated by the paper.
Personal Thoughts
The paper’s key insight is to reinterpret limited per-GPU PCIe bandwidth as a bandwidth-fragmentation problem caused by static device ownership. TurboBus uses NVLink/NVSwitch to borrow idle PCIe paths from other GPUs, providing practical application-level gains without requiring new hardware.
Its effectiveness, however, depends on the availability of idle PCIe capacity and sufficient GPU-interconnect bandwidth. Relay traffic may contend with tensor-parallel or collective communication when NVLink is heavily utilized, while cross-NUMA transfers may shift the bottleneck to UPI or host memory. Promising directions include topology-aware path selection, joint communication scheduling, and stronger tail-latency and tenant-isolation guarantees.