Title: Understanding Host Network Stack Latency
Authors: Tianyu Zuo (University of Virginia); Jaehyun Hwang (Sungkyunkwan University); Ao Tang, Rachit Agarwal (Cornell University); Qizhe Cai (University of Virginia)
Introduction
This paper revisits a fundamental assumption in datacenter networking: that millisecond-scale tail latencies in Linux network stacks are inherent to the protocol processing path. While prior work has advocated for userspace stacks or specialized hardware as solutions, the authors ask a more basic question — where does the time actually go? Using fine-grained, end-to-end latency decomposition on a 400 Gbps testbed, they discover a counterintuitive answer: Linux itself achieves microsecond-scale tail latencies under isolated conditions. The real culprits are CPU resource management, the interaction between the scheduler and NIC interrupts, and mis-accounted softIRQ time. This matters because it suggests we can fix the problem within Linux rather than abandoning it.
Key idea and contribution:
The authors decompose a single TCP RPC into 12 components using kernel-level timestamps. Their central insight is that high tail latency arises from three distinct sources, each requiring a different fix:
-
Mis-accounted softIRQ time (
ACCa): Linux CFS charges softIRQ processing time to the interrupted thread, not the thread that owns the packet. This creates a positive feedback loop where threads handling more interrupts appear to have consumed more CPU time and are scheduled less often. The authors implement accurate IRQ accounting that corrects this at every scheduling event, reducing P99.9 latency by ~2.74× at the 48-thread拐点. -
Wrong fairness metric (
PCSched): Even with correct accounting, CPU runtime is a poor fairness unit for network workloads because cache and memory effects cause identical code paths to consume different CPU cycles for the same network progress. The authors prototypePCSched, which schedules threads by request/response count rather than runtime, further reducing tail latency to ~276 μs (~4.1× improvement over baseline). -
Interrupt modulation mismatch (
AutoDIM): Dynamic Interrupt Moderation (DIM) struggles with bursty, unpredictable traffic. The authors show that a simple heuristic based on in-flight packets can reduce P99.9 to ~215 μs, eliminating the anomaly where tail latency first rises then falls with load.
Together, these improvements achieve up to 5.3× tail latency reduction with minimal throughput impact.
Evaluation
The evaluation uses a 400 Gbps direct‑connect testbed with dual-socket Intel Xeon Gold 6530 CPUs and ConnectX‑7 NICs, running a custom Netperf ping‑pong workload with 64B RPCs. Under isolated single‑thread conditions, baseline Linux achieves P99.9 ~19 μs — within striking distance of userspace TAS (~9 μs). When scaling to 48 threads sharing a core, baseline P99.9 spikes to ~1139 μs, with rx_sched dominating. ACCa reduces this to ~415 μs, PCSched to ~276 μs, and AutoDIM to ~215 μs. In Redis + YCSB with 4KB RPCs, ACCa improves tail latency by ~2× at the拐点, while AutoDIM adds another ~1.42× reduction on 95/5 workloads. This result is significant because it demonstrates that Linux can achieve near‑userspace performance without abandoning the kernel — but only if we fix the right bottlenecks. The findings generalize across RPC sizes, incast/outcast patterns, and even the newer EEVDF scheduler.
Q1: Is ACCa being upstreamed? What was the developer response?
A: The team has fixed the inaccurate time accounting and plans to upstream the patch, as it could benefit existing Linux applications. However, a deeper takeaway is that CPU runtime as a fairness metric is not always ideal — and may not be the right abstraction even after accounting fixes. No evidence of formal review or mainline merge was provided in the talk.
Q2: How do you separate instrumentation cost from real scheduling delay?
A: The authors performed A/B comparisons with and without instrumentation; throughput and latency were not significantly affected. They use sampling, timestamping, and per‑CPU ftrace buffers to minimize overhead. Small mismatches between component sums and end‑to‑end measurements are acknowledged as inevitable trace overhead.
Q3: Since Linux has moved to EEVDF, does this paper’s focus on CFS limit its relevance?
A: EEVDF still relies on per‑thread virtual runtime, lag, and virtual deadlines — so accurate accounting remains critical. The authors use Linux 5.10+CFS as a controlled baseline and supplement with EEVDF tests on Linux 6.12. They find that lag evaporation in EEVDF can accidentally hide accounting errors by compressing runtime gaps during sleep/wake cycles, but this is a separate unfairness, not a fix. ACCa still provides up to ~1.15× benefit under EEVDF.
Q4: Have you tested threaded NAPI, which moves NAPI processing from softirq to kernel threads?
A: Not yet, but the authors consider it a valuable follow‑up direction.
Personal thoughts
What I appreciate most about this paper is its diagnostic rigor. Instead of treating the Linux network stack as a black box, the authors decompose latency into measurable components, identify three distinct root causes, and validate each with targeted fixes. The 12‑stage breakdown, cache‑stall analysis, EEVDF comparison, and Redis validation make the conclusions compelling — not just another tuning heuristic. The willingness to push patches upstream and engage with the community also speaks to the work’s practical relevance.
That said, several caveats warrant caution. The 5.3× improvement is a peak from combining all fixes under specific conditions; the 18.6× claim in multi‑core experiments is also拐点‑specific and not generalizable. PCSched and AutoDIM remain proof‑of‑concept prototypes, not production‑ready solutions. The evaluation relies on a 400 Gbps direct‑connect setup with a single NIC model — we don’t know how these results hold on 25 Gbps or with older hardware. Future work should systematically evaluate threaded NAPI, quantify measurement overhead more rigorously, and test whether ACCa-style fixes remain effective across diverse NICs and traffic mixes. For practitioners, the safest takeaway is pragmatic: keep active connections per core low, use in‑flight requests to amortize overhead, and treat CPU scheduling and NIC interrupt policy as a unified end‑to‑end problem rather than separate knobs.