Title: Anytest: Localizing the Root Cause of Hardware Transport Performance Anomalies
Authors: Zhaochen Zhang (Nanjing University), Jiaqi Gao, Sheng Cheng, Peiwen Yu, Feiyang Xue, Chang Liu (Alibaba Cloud), Boliang Liu (Fudan University), Kun Liu, Yubin Li (Alibaba Cloud), Rui Li, Li Wang, Peirui Cao, Qingkai Meng, Guihai Chen (Nanjing University), Gang Lu, Bo Shi, Shuguang Cheng, Shu Ma, Yongqing Xi, Binzhang Fu, Dennis Cai (Alibaba Cloud), Chen Tian (Nanjing University)
Scribe: Mengrui Zhang (Xiamen University)
Introduction
RoCEv2 fabrics support distributed storage, HPC, and LLM workloads, but large deployments increasingly experience transport-layer network performance anomalies (NPAs), including throughput collapse, persistent unfairness, and latency inflation without link failures. RDMA moves transport logic from inspectable software into proprietary RNIC and switch hardware. Operators can often detect a symptom such as packet loss, but counters aggregate behavior at roughly 10 ms to 1 s while congestion-control state machines react at sub-microsecond timescales. The same symptom can also arise from sender, receiver, or switch behavior in a tightly coupled feedback loop.
Production diagnosis therefore follows a trace-reproduce-localize workflow. Application behavior is first captured with an RDMA call-tracing library, and the anomaly is reproduced with a traffic generator such as Synapse. Reproduction still does not identify whether the cause is an aggressive congestion controller, an unexpected CNP interval, switch marking, queueing, or scheduling. Existing counter inference is in situ but coarse, hardware testers are precise but usually offline, and test-suite generators often observe only flow-level outcomes. Anytest is an in-situ black-box testing tool for commodity RoCEv2 RNICs and Ethernet switches that combines role isolation, controlled packet-level event injection, microsecond measurements, and production deployment without recabling or hardware modification.
Key idea and contribution
Surround the device under test with white-box DPDK endpoints. Anytest places a small number of hosts in the affected production cluster and keeps the target COTS NIC or switch in its real environment. DPDK emulators replace the missing communication peers with protocol-correct software endpoints. They inject controlled transport events, record packet-level responses, and expose transient behavior that coarse counters erase. An orchestrator configures the experiment, deploys binaries, coordinates execution, collects traces, and produces a human-readable report.
Isolate one RoCEv2 transport role at a time. Anytest uses three explicit roles: the Reaction Point (RP) for sender-side congestion reaction, the Notification Point (NP) for receiver-side notification generation, and the Congestion Point (CP) for switch-side congestion signaling. In an RP test, a target sender NIC runs real RDMA traffic while DPDK emulates NP and CP and injects controlled CNP patterns. In an NP test, DPDK sends packets with controlled ECN marks to a target receiver NIC and records ACK/NACK/CNP outputs. In a CP test, DPDK RP endpoints send traffic through a target switch, while a DPDK NP records packet delay and ECN-marking ratio. This decomposition attributes a deviation to one role rather than to an inseparable sender-switch-receiver interaction.
Inject precise events and measure at microsecond resolution. Anytest controls packet sequences, congestion feedback, ECN marks, loss, reordering, and arbitrary inter-packet gaps. Its measurement pipeline records throughput, packet delay, ECN marking, and control-packet sequences with microsecond-level timing. Hardware timestamps and lock-free SPSC rings keep logging and aggregation away from the critical datapath. Tests suspend service traffic only on selected hosts involved in the experiment; they do not require stopping all cluster traffic, packet taps, special testers, or recabling.
Reach line rate with protocol-correct DPDK engineering. Anytest uses pre-initialized packet-template pools to avoid per-packet allocation and payload initialization, an O(1) ICRC update that changes only the runtime PSN field, and a two-thread prepare/send pipeline that preserves packet ordering while using batched transmission. The optimized ICRC path sustains 200 Gbps with 1024-byte MTU and 800 Gbps with 4096-byte MTU in single-core measurements. Padding-based interval control provides microsecond-scale packet timing without reducing the datapath to per-packet send calls.
Turn localization into a regression workflow. After an NPA is detected and reproduced, operators select a symptom track and run role- and function-specific regression tests against the affected hardware. Parameter sweeps identify a reliable trigger and narrow the cause to a responsible role, function class, and trigger specification. If the existing suite cannot explain the anomaly, engineers design a new stimulus and later fold the resulting test into the offline regression suite, so future occurrences can follow an automated path.
Evaluation
Production localization workflow. Anytest has been deployed for approximately one year and evaluated across 58 production NPAs. It reduces mean localization effort to 3.1 person-hours per case. The regression suite resolves 67% of incidents with zero manual effort; approximately 33% require engineers to design new stimuli, select sweep dimensions, or interpret previously unseen behavior. Once a new case is understood, its test is added to the suite.
Representative root causes. Anytest localizes congestion collapse caused by receiver-side CNP behavior, incast unfairness caused by unexpected source-QP aggregation under a PER_QP timer, and interval-sensitive DCQCN behavior. It also identifies firmware-dependent congestion-control divergence, sharp latency inflation under ETS scheduling, and unfair bandwidth allocation in a VOQ-based switch. In the VOQ case, identical offered rates still produced a 25/25/50 Gbps split, ruling out ECN and CNP effects and attributing the anomaly to switch scheduling; the issue was fixed by a switch configuration update. In the ETS case, background traffic produced approximately 30 microseconds mean latency inflation and approximately 70 microseconds p99/p99.9 inflation at the worst measured point.
Operational significance. The results show that packet timing and role-isolated stimuli can turn opaque hardware behavior into repeatable evidence in the live production topology, firmware, and configuration where the NPA occurs. Anytest does not inspect internal firmware or ASIC state directly, and it cannot localize anomalies such as PFC that are outside its on-wire observability. Its tests can also perturb shared resources, so they are scoped to selected hosts and ports, bounded in intensity and duration, and repeated to distinguish systematic behavior from noise.
This result is significant because it changes NPA localization from correlating coarse counters to running controlled experiments that attribute a symptom to a transport role and a concrete trigger. The resulting regression suite also gates future firmware, driver, and configuration changes.
Personal thoughts
Anytest’s strongest idea is methodological: when transport logic is hidden inside hardware, operators can still make it diagnosable by controlling every external role and measuring the wire-level response. The RP/NP/CP decomposition makes ambiguous symptoms testable, while the line-rate DPDK implementation keeps the tool close to real production behavior instead of turning diagnosis into an offline approximation.