Title: CSIG: Congestion Signaling for Datacenter Transports
Authors: Abhiram Ravi, Nandita Dukkipati, Weiwu Pang, Neal Cardwell, Brad Karp, MJ Akhbari, Weida Huang, Konstantinos Prasopoulos, KK Yap, Arjun Singh, and Amin Vahdat (Google LLC)
Scribe: Jinghui Jiang (Xiamen University)
Introduction
Modern datacenter workloads, especially distributed AI/ML services, generate synchronized microbursts and rapidly alternate between congestion and idle capacity. End-host transports need timely and precise network feedback to respond to these changes, but the signals available in deployed networks are limited. ECN provides only a binary indication and must be averaged over many packets, while RTT and end-to-end delay combine congestion from multiple hops and cannot identify the true bottleneck. More expressive in-band telemetry systems such as P4-INT, IFA, and iOAM can report per-hop state, but their variable-length headers complicate line-rate parsing, consume bandwidth, interact poorly with MTUs and NIC offloads, and become entangled with tunneling and encryption. The paper asks whether a datacenter network can expose fine-grained congestion information on every packet while remaining simple enough to deploy across existing switches, NICs, and transport stacks.
Key idea and contribution
CSIG’s central insight is that transport control loops usually do not need a complete trace of every hop. They mainly need the value and location of the single bottleneck that limits the flow. CSIG therefore carries a fixed-size summary of a path extremum rather than appending telemetry at each hop. A sender places a CSIG tag at the end of the Layer 2 header and selects a signal type. As the packet traverses the network, each switch compares its local measurement with the value already in the tag and replaces the value and locator only when its condition is more severe. For example, switches compute the minimum relative available bandwidth, min(ABW/C), or the maximum per-hop delay, max(Delay). The receiver extracts the resulting bottleneck signal and reflects it to the sender in a transport-specific Layer 4 header.
The Layer 2 placement is a deliberate deployability choice. The tag remains at a predictable offset, is visible to switches without parsing inner transport headers, and is compatible with common Layer 3 and higher-layer tunnels as well as end-to-end encryption. CSIG defines a 4-byte Compact format and an 8-byte Wide format. The Compact tag contains a 3-bit signal type, a 5-bit signal value, and a 7-bit locator; the Wide tag provides a 20-bit value and a 16-bit locator. A packet carries one signal type, but a transport can alternate signal requests across packets when it needs several metrics. Since the header size does not grow with path length, the Compact format adds less than 0.2% bandwidth overhead for 4 KB or 9 KB MTUs and can be enabled on every data packet.
The paper presents both hardware-native and software-assisted switch implementations. New ASICs can compute per-packet signals such as available bandwidth, normalized queue depth, and per-hop delay directly in the data plane, with measurement intervals as short as approximately 1 microsecond. For brownfield deployment, the authors build a user-space High-precision Switch Telemetry (HST) agent. It bypasses slow vendor SDK paths, reads hardware counters through a dedicated CPU-to-ASIC channel, computes a bucketized signal, and writes the current value into switch tables. The ASIC then performs compare-and-replace operations at line rate. This software-assisted design works on commodity ASICs dating back to 2010, although its updates are coarser, typically around 100 microseconds, and it usually supports only the Compact tag. The authors validate CSIG across five switch-silicon generations, four NIC generations, and five transport stacks, including Linux TCP, PonyExpress, Falcon, RoCE, and a virtualization stack.
One major use of CSIG is Fast Ramp-Up (FRU), a congestion-control primitive that uses ABW/C to claim idle capacity more quickly than additive increase. If a bottleneck is currently utilized by a fraction u, a sender can estimate a safe multiplicative growth factor of 1/u. FRU combines this factor with the amount of data in flight to estimate the flow’s available capacity and move its congestion window toward that target over the next RTT. A queue-delay threshold acts as a circuit breaker: if queues have already begun to build, FRU stops using potentially stale available-bandwidth feedback. The authors integrate FRU into Swift and describe variants for CUBIC and NSCC. They also use the bottleneck locator to reduce the multipathing degree when congestion is on the last hop, and use absolute available bandwidth to make Protective Load Balancing choose between candidate paths more intelligently.
CSIG additionally connects network telemetry to application context. Because the signal travels with application packets, it can be attached directly to RPC traces instead of being correlated afterward with separate switch measurements. The implementation exposes CSIG through RPC tracing, host-based SDN traffic matrices, and fabric telemetry. Operators can therefore determine not only that an RPC was slow, but also whether its bandwidth was constrained and whether the bottleneck was at a ToR uplink, the last hop, or a higher network tier.
Evaluation
The evaluation combines a deployment across hundreds of production clusters in Google’s Jupiter network, controlled testbeds, and simulations. In production, adding FRU to Swift reduces median RPC latency for distributed LLM serving by approximately 20% and reduces median unclaimed bandwidth by 60%. It also reduces median KV-cache transfer time by 5.8%, or 9.3% under peak query load. The improvement is concentrated in clusters with substantial unused bandwidth and is negligible when little spare capacity exists, which is consistent with FRU’s purpose. The 99th-percentile RPC latency does not improve noticeably because the deepest tail occurs when available bandwidth is already zero. FRU increases p99.9 RTT by less than 10%, reflecting a tradeoff between faster capacity use and slightly more queueing.
In a controlled incast experiment, CSIG-guided dynamic multipathing reduces p99 latency from 222 ms to 89 ms, a 2.5x improvement, by recognizing a last-hop bottleneck and draining unnecessary subflows. CSIG-informed PLB lowers rerouting from 45 to 12 times per second and increases throughput from 31.4 to 36.5 Gbps under heavy background load. Simulations show that replacing NSCC’s cumulative path-delay signal with bottleneck-only max(Delay) improves the victim-to-non-victim throughput ratio from 4.5:1 to 1.4:1; for collective-like traffic, it improves victim throughput by about 65% and reduces flow completion time by about 39%. Across the tested NICs and switches, the 4-byte and 8-byte tags impose less than 0.1% goodput overhead and less than 0.01% impact on packet rate or latency.
The observability case studies show a second kind of value. CSIG reveals that a production multipathing rollout moved congestion away from ToR uplinks, distinguishes the network behavior of storage reads and writes using more than 100 million RPC samples, and identifies a link flap associated with a stalled ML training step using microsecond-scale traces. This result is significant because CSIG is not only a new congestion signal: it provides a common, deployable interface through which transports and applications can react to the same real-time bottleneck information, without requiring full per-hop telemetry or a wholesale hardware replacement.
Q
Q: By reporting only one path extremum, does CSIG discard information that would be useful to congestion control or debugging?
A
A: Yes, CSIG intentionally trades complete path visibility for low overhead and deployability. It cannot reconstruct the state of every hop from one packet, and a Compact packet carries only one signal type. However, the paper argues that many transport algorithms ultimately react to the worst hop, so the bottleneck value and locator are sufficient for control decisions such as congestion-window growth, path selection, and multipathing degree. A sender can alternate signal types across packets when a control loop needs multiple measurements, as demonstrated for HPCC. For debugging that requires a full path history, CSIG would complement rather than replace detailed switch telemetry; its main benefit is that it immediately identifies the bottleneck experienced by a particular application flow.
Personal thoughts
I like the paper’s decision to treat deployability as part of the protocol design rather than as an implementation detail. The key simplification, reporting only the bottleneck instead of every hop, removes several problems at once: header growth, MTU changes, variable parsing offsets, and much of the interaction with tunnels and encryption. The software-assisted implementation is particularly convincing because it shows how useful feedback can be added to old fixed-function switches, while the production rollout discussion describes practical safeguards such as strip-and-forward boundaries, transport negotiation, automatic fallback, and state verification. These details make the work feel like an operational system rather than only a packet format.
The limitations are equally important. Software-assisted CSIG uses bucketized values and updates around every 100 microseconds, which can be substantially longer than a datacenter RTT. Its reuse of VLAN resources also prevents deployment in some bare-metal environments that require VLAN-based tenant isolation. The production FRU results mainly improve the median rather than the deep tail, and the evaluation relies on comparisons around a staged rollout rather than a fully randomized experiment. In addition, the strongest hardware-native features, including per-packet delay and higher-resolution Wide tags, are not evaluated at the same production scale as the brownfield implementation.
A useful direction for future work is to study how senders should choose signal types and measurement windows under heterogeneous RTTs and rapidly changing workloads. It would also be valuable to compare bottleneck-only signaling with full INT under failures or multi-bottleneck paths, where information discarded by the extremum may matter for diagnosis. More broadly, CSIG suggests that the same L2 substrate could carry signals beyond congestion, such as link health or reliability, but expanding the protocol will require preserving the simplicity that makes the current design deployable.