Title: Rules Offload Engine (ROE): Accelerating Host SDN Policy Evaluation
Authors: Anshuman Verma, Tian Tan, Ahmed Abdelsalam, Milan Dasgupta, Jonathan Hunter, Zach Libby, Narayanan Ravichandran, Harish Srinivasan, Lok Chand Koppaka, Matt Reat, Nadeen Gebara, Vishal Gondaliya, Ezz Hamed, Rahul Garlapati, Abdullah Mughrabi, Dev Desai, Alexander Malysh, Shwetha Bhat, Rohan Kandi, Megan Sng, Tushar Garg, Muluken Hailesellasie, Andrew Putnam, Derek Chiou, Osman Ertugay, Alireza Dabagh, Vivek Bhanu, Daniel Firestone (Microsoft)
Reviewer: Ying Yao (SNG first-year PhD student)
Introduction
Public clouds rely on Software-Defined Networking (SDN) to provide functions such as access control, address translation, tunneling, load balancing, and network isolation for a vast number of virtual machines. Taking Azure as an example, each virtual network interface is associated with a set of layered SDN policies; data packets must sequentially match rules across these layers and trigger corresponding actions.
Microsoft’s existing Virtual Filtering Platform (VFP) evaluates these policies for new connections and consolidates the results into a Unified Flow Entry (UFE). AccelNet, running on an FPGA SmartNIC, caches the UFE in a Generic Flow Table (GFT), allowing subsequent packets to bypass the VFP and undergo high-speed processing in hardware.
The challenge lies in the first packet of a new connection. Since no corresponding UFE exists in the GFT at this stage, AccelNet must treat it as an exception packet and pass it to the VFP. Software policy evaluation involves irregular memory access, numerous conditional branches, and multi-layer rule traversal, compounded by I/O and queuing latencies. When cloud applications frequently establish short-lived connections, the processing of these initial packets becomes a primary bottleneck for the connection establishment rate (Connections Per Second, or CPS).
This paper introduces the Rules Offload Engine (ROE), an accelerator positioned between AccelNet and the VFP specifically for SDN rule evaluation. The ROE processes the majority of exception packets—those that result in GFT misses—directly on the FPGA, while only a small fraction of complex or rare cases fall back to the VFP; this effectively offloads policy evaluation for common new connections to the SmartNIC.
Core Ideas and Contributions
ROE employs a co-design approach for its instruction set, compiler, processing cores, and system interfaces, tailored to the characteristics of SDN policy evaluation. The VFP provides high-level policies, which the compiler translates into specialized programs; upon the arrival of a new connection, ROE executes the program to generate a Unified Flow Entry (UFE) and writes it to AccelNet, enabling subsequent packets to utilize the existing hardware fast path.
(1) Network-Rule-Oriented Specialized Instruction Set
The ROE Instruction Set Architecture (RISA) provides instructions such as MATCH, ALU, CTRL, HASH, and μENGINE. The MATCH instruction performs comparison, masking, shifting, and complex conditional updates in a single operation, thereby reducing the number of instructions and branches required for rule matching; the μENGINE instruction is used to invoke specialized hardware modules for tasks such as UFE creation and packet transmission.
RISA integrates policy data and control operations into read-only instructions, primarily utilizing register and immediate addressing. Packet header fields and modification results are stored in dedicated registers, and rules are embedded directly into the instruction stream by the compiler; consequently, general-purpose data caches and load/store units are eliminated, reducing both the impact of cache misses and FPGA area overhead.
(2) RO-Bridge and RISA-Compiler
RO-Bridge serves as the software control layer connecting the VFP, AccelNet, and ROE. It retrieves policies from the VFP, invokes the RISA-Compiler to generate programs, and maintains consistency regarding port lifecycles, hardware resources, and policy versions. The compiler also reorganizes IP Tries, interval trees, and hash tables to align with ROE’s cache structure, thereby minimizing the overhead of rule traversal.
The compiler speculatively generates reverse flows: while processing a forward flow, it pre-constructs the reverse UFE that is likely to be used. If the prediction is correct, the first reverse packet avoids a full policy evaluation, potentially doubling throughput under typical workloads; if the prediction is incorrect, the speculative entry is discarded. For policy updates, the system generates the program in a new code page and completes the switchover after draining in-flight packets associated with the old version and flushing relevant caches, ensuring uninterrupted packet processing. (3) ROE-Core Designed for High-Concurrency Connections
ROE comprises six cores; each employs an 8-stage in-order pipeline and supports eight fine-grained interleaved threads. Consecutive instructions within the same thread are spaced eight cycles apart to minimize stalls caused by data dependencies. Under peak CPS loads, core utilization can approach 97%.
ROE-Core implements only instruction caches: each core has a private 32 KB L1 cache, while the six cores share a 256 KB L2 cache; packet headers and intermediate states are stored in per-thread registers. Complex yet infrequent tasks are offloaded to shared micro-engines. For instance, the Unified Flow Create Engine handles UFE generation and installation, while the Flow Management Engine manages connection timeouts, teardown, and state reclamation. A Bypass and Re-ordering Queue ensures that, following parallel processing by multiple threads, packets are returned to AccelNet in their original arrival order.
(4) Fallback Mechanism Compatible with Production Systems
ROE does not attempt to handle every scenario. Scenarios such as ARP requests, resource exhaustion, or UFE installation failures trigger a fallback to the software path via VFP-deferral. Once a connection is handed over to the VFP, the system ensures that subsequent packets remain under VFP management and clears any intermediate state previously created by ROE, thereby preventing a single bidirectional connection from being managed separately by both hardware and software.
The system also supports UFE re-evaluation following policy updates, QoS, connection state tracking, monitoring, and state preservation and restoration during upgrades. With approximately 8 million UFEs present in the GFT, both preservation and restoration can be completed within 600 ms, enabling ROE to be integrated as an incremental component into the existing Azure network stack.
Experimental Evaluation
The authors deployed virtual machines on two production nodes to compare ROE+AccelNet, AccelNet-only configurations, and the Sirius appliance (based on dedicated ASICs). The experiments covered scenarios involving standard virtual networks (VNET) and Network Virtual Appliances (NVAs) that require middlebox processing, primarily measuring Connections Per Second (CPS) and TCP SYN-to-SYN-ACK round-trip latency.
(1) Connection Establishment Performance
Compared to AccelNet alone, ROE+AccelNet achieved CPS improvements of up to 11x in NVA scenarios and 12.3x in VNET scenarios. Connection establishment latency dropped by up to an order of magnitude, with the system reaching a peak of approximately 400,000 CPS. While Sirius achieved higher throughput in some tests, it requires dedicated ASICs and distinct policy interfaces; in contrast, ROE resides directly on the server node, allowing it to leverage the VFP policy abstraction already widely deployed in Azure.
(2) Resource and Policy Scalability
A single ROE-Core utilizes approximately 4.8K LUTs, 6.4K registers, and 19 BRAMs; the complete ROE processing subsystem consumes only about 5% of the target FPGA’s total resources, leaving ample space for other functions such as AccelNet.
As the number of IP prefixes increases from 16 to 20,000, the size of the static RISA program grows rapidly, yet the number of instructions actually executed per packet remains largely stable. A typical Azure VNET program comprises approximately 92,000 static instructions, whereas an average of only 2,500 are executed per packet, demonstrating that scaling up the policy size does not result in a proportional increase in dynamic execution overhead.
Cache experiments also revealed that simply increasing L2 capacity yields negligible improvements in miss rates within multi-VM environments, primarily due to inter-tenant cache conflicts. Conversely, partitioning the cache, reducing L1 cache line sizes, and employing hash indexing for L2 proved more effective at lowering miss rates.
(3) Real-world Deployment
ROE has been deployed across thousands of nodes as part of the second-generation Azure Boost. At the time of writing, it served approximately 18% of the tenant user base and handled production traffic across multiple regions, with each node managing up to approximately 8 million active flows. The system has been in operation for approximately one year, undergoing multiple rounds of hardware, software, and functional upgrades.
Q&A
Q1: Most RISA operations are read-only, yet the system supports statistics and telemetry. How does the ROE write statistical data, and how do other software components read it?
A1: The system utilizes a dedicated counter acceleration module to maintain telemetry data and periodically writes it to shared memory via DMA. Other software components can read this data and upload it to a database for diagnostics and debugging; consequently, there is no need to implement a general-purpose data-writing path within the ROE-Core.
Q2: Where exactly is the ROE located within the system, and what is its relationship with AccelNet?
A2: Both the ROE and AccelNet reside on the same FPGA within the SmartNIC; the ROE can be viewed as an auxiliary processing path alongside AccelNet. When AccelNet encounters an exception packet it cannot handle, it offloads the request to the ROE via a queue. The ROE performs policy evaluation, generates a UFE (Unified Forwarding Entry), and returns both the UFE and the packet to AccelNet, which then executes the final action.
Q3: Why was the decision made to implement the ROE on existing FPGA SmartNICs rather than adopting a new DPU architecture?
A3: Azure has already deployed FPGAs on a massive scale. By simply updating the FPGA image and associated software, existing nodes can achieve performance improvements through the ROE. While a DPU could potentially implement similar functionality, it could not be directly integrated into the existing fleet; therefore, the team chose a solution that facilitates incremental deployment and allows for the reuse of the existing AccelNet infrastructure.
Personal thoughts
The most noteworthy aspect of ROE is its accurate identification of the “first-packet gap” within the hardware fast path: while established connections can be forwarded at high speed using cached entries, every new connection must still undergo a complex policy evaluation process. Rather than cramming all logic into a fixed pipeline or simply adopting general-purpose embedded cores, the paper strikes a balance between policy expressiveness and hardware efficiency through the use of specialized instructions, just-in-time (JIT) compilation, fine-grained multithreading, and shared micro-engines.
On the other hand, ROE is deeply integrated with Azure’s existing VFP, AccelNet, and FPGA SmartNIC technologies. Replicating this solution on platforms lacking similar infrastructure entails significant system integration costs; furthermore, if complex or anomalous traffic accounts for a large proportion of the workload, falling back to the VFP software path could once again become a performance bottleneck. Real-world deployment experience also demonstrates that capabilities such as explicit graceful degradation, mixed-version compatibility, telemetry, and rollback mechanisms should be treated as core system features from the initial design phase. For hardware accelerators deployed in cloud data centers, these capabilities may be just as critical as peak performance.