Re-architecting End-host Networking with CXL: Coherence, Memory, and Offloading

---

Review Form

Reviewer: The Guardian (Adversarial Skeptic)

Summary

The paper presents CXL-NIC, a Network Interface Controller architecture built on Compute Express Link (CXL) to address the performance limitations of traditional PCIe-based NICs. The authors propose two designs: a Type-1 CXL-NIC using CXL.cache to replace DMA/MMIO operations, and a Type-2 CXL-NIC that adds coherent on-device memory via CXL.mem. The central thesis is that by leveraging CXL's coherence and memory semantics, significant reductions in packet and application processing latency can be achieved. The designs are prototyped on an FPGA and evaluated against a commercial PCIe-based SmartNIC.

While the premise of using CXL to improve NIC performance is sound, the work is undermined by a significant methodological flaw: the experimental evaluation compares an FPGA-based CXL prototype against a commercial, ASIC-based PCIe SmartNIC. This comparison introduces numerous confounding variables, making it impossible to attribute the observed performance differences solely to the CXL interconnect. Consequently, the paper's primary claims of latency reduction are not adequately substantiated by the provided evidence.

Strengths

1. Detailed Protocol-Level Optimizations: The paper demonstrates a strong command of the CXL.cache protocol. Section 4.3 provides a granular analysis of how different request types (e.g., CS-read for prefetching, CO-read for polling, NC-write for packet transfers) can be strategically employed to optimize different stages of the networking datapath. This exploration is valuable for the community.
2. Insightful Analysis of On-Device Memory: The evaluation of packet buffer placement for the Type-2 device (Section 7.2, Figure 13) yields an important, if negative, result: naively placing packet buffers in NIC memory degrades performance due to remote access latency and coherence overhead. This is a crucial finding that cautions against simplistic architectural assumptions.
3. Coherent Problem Formulation: The paper correctly identifies the fundamental bottlenecks in the PCIe-based CPU-NIC datapath (Section 3.1) and logically proposes CXL as a potential solution. The motivation is clear and well-grounded in the limitations of existing interconnects.

Weaknesses

1. Fundamentally Unsound Experimental Comparison: The paper's primary claims rest on a comparison between two vastly different hardware platforms: an Intel Agilex-7 FPGA running at 400 MHz and an NVIDIA BlueField-3 (BF-3) ASIC SmartNIC with ARM cores running at 2.5GHz. This is not a valid apples-to-apples comparison. The observed latency differences could stem from a multitude of factors unrelated to the CXL protocol itself, including:
   • The internal architecture of the NICs (FPGA logic vs. hardened ASIC blocks).
   • The performance of the on-board memory controllers and subsystems.
   • The compute capabilities used for evaluation (FPGA logic vs. ARM cores for the KVS workload).
     The authors explicitly state they could not create a PCIe baseline on the same FPGA (Section 6, page 9), which confirms this is a critical, unaddressed confounder that invalidates the main quantitative claims.
2. Conflation of Protocol Benefits and Implementation Limitations: The throughput evaluation (Section 7.2, Figures 11 and 12) is bound by the 400 MHz clock frequency of the FPGA and the single CXL request per cycle limit of the IP. The results demonstrate the efficiency of their design relative to their implementation's theoretical peak, not the absolute throughput capability of a CXL-based NIC architecture. The claims should be heavily qualified to reflect that these are artifacts of a slow prototype, not fundamental characteristics of the CXL approach.
3. Obscured Absolute Performance: The majority of the key results are presented as normalized figures relative to the BF-3 baseline (e.g., Figures 10, 13, 14, 15). This prevents a clear assessment of the absolute performance of the CXL-NIC. A 49% reduction in tail latency (Section 7.1, page 10) is meaningless without knowing the absolute baseline latency. The proposed CXL-NIC could still be substantially slower than the commercial ASIC in absolute terms.
4. Weak Comparison to CC-NIC: The comparison against the state-of-the-art CC-NIC (Figure 14) is based on an emulation of the UPI protocol using CXL.cache primitives (CS-read and CO-write). This is a questionable methodology. UPI and CXL have different underlying coherence semantics, link-layer protocols, and performance characteristics. The authors provide no evidence that this emulation is a faithful or accurate representation of a real UPI-based NIC, rendering the 37% claimed improvement unreliable.
5. Uncontrolled Variables in Application Study: The KVS application evaluation (Section 5.4 and Figure 15) compares a handler implemented in FPGA logic on the CXL-NIC against a software handler running on the BF-3's ARM cores. The 39% tail latency reduction cannot be uniquely attributed to CXL. It is highly likely that a hardware-accelerated FSM on an FPGA is simply faster at this specific task than general-purpose ARM cores running software, irrespective of the host interconnect. The experiment fails to isolate the variable of interest.

Questions to Address In Rebuttal

1. The central weakness of this paper is the comparison between a 400 MHz FPGA prototype and a commercial ASIC SmartNIC. How can the authors justify that the observed performance differences (e.g., in Figures 10 and 15) are due to the CXL vs. PCIe interconnect, and not the vast architectural, clock speed, and compute substrate differences between the two devices?
2. Please provide absolute latency numbers (in microseconds or nanoseconds) for the key results presented in Figures 10, 14, and 15. This is necessary to evaluate whether the proposed CXL-NIC is performant in an absolute sense, not just relative to a potentially mismatched baseline.
3. Please justify the methodology for emulating a UPI-based CC-NIC using CXL.cache primitives (Section 7.2, page 12). How does this emulation account for the architectural and protocol-level differences between UPI and CXL, and why should this be considered a valid point of comparison?
4. In Section 7.2, the paper presents a "latency-optimal configuration." How was the vast design space of data placement and request type combinations (per Figure 8) explored to rigorously justify this claim of optimality? What evidence supports that this specific configuration is globally optimal, rather than just the best among a small, tested subset?
5. For the KVS evaluation (Figure 15), how have the authors isolated the performance impact of the CXL interconnect from the performance impact of implementing the KVS handler in FPGA hardware logic versus software on ARM cores? Without this isolation, the claim that CXL is the source of the benefit is unsupported.

---

Review Form:

Reviewer: The Synthesizer (Contextual Analyst)

Summary

This paper presents a comprehensive re-architecting of the end-host network interface controller (NIC) using the emerging Compute Express Link (CXL) standard. The authors identify the long-standing performance bottlenecks of the traditional PCIe interconnect—namely high latency for small packet operations due to DMA/MMIO overhead and the lack of hardware cache coherence between the CPU and the NIC.

The core contribution is the design and evaluation of "CXL-NIC," a novel NIC architecture that systematically replaces legacy PCIe mechanisms with CXL's more efficient, cache-coherent protocols. The work is presented in two stages: first, a Type-1 CXL-NIC that leverages CXL.cache to create a low-latency, coherent datapath for control and data between the host CPU and the NIC. Second, this design is extended to a Type-2 CXL-NIC, which introduces coherent on-NIC memory via CXL.mem, enabling flexible data placement and new opportunities for near-data processing. The authors demonstrate the power of this new architecture with a compelling networking-application co-acceleration case study of a Key-Value Store (KVS). An FPGA-based prototype shows significant latency reductions compared to both a commodity PCIe SmartNIC and a prior academic coherent NIC design.

Strengths

1. Excellent Problem-Solution Fit and Timeliness: The work tackles a classic and persistent problem in systems architecture: the host I/O bottleneck. The application of CXL to this domain is not just novel but exceptionally well-suited. While much of the early CXL discourse has focused on memory expansion (CXL.mem), this paper provides one of the first in-depth, systems-level explorations of CXL.cache for a "killer application." It moves the conversation from characterizing a new interconnect to architecting a new class of device with it.

2. Systematic and Principled Design Exploration: The paper's strength lies in its methodical approach. The progression from a Type-1 device (focusing on coherence and datapath) to a Type-2 device (adding memory and offload) is logical and allows for a clear separation of concerns. The detailed analysis in Section 4.3 on leveraging specific CXL cache line state control requests (e.g., CS-read for prefetching, CO-read for polling, NC-write for packet data) is particularly insightful. It demonstrates a deep understanding of the protocol and moves beyond a simple "replace DMA with CXL" narrative to a nuanced, optimized design.

3. Strong Grounding in the Literature and Context: This work is well-positioned within the broader landscape of high-performance networking and interconnect research. It implicitly builds on the legacy of projects that sought tighter CPU-I/O integration (e.g., DDIO, on-chip NICs) but demonstrates a more practical path forward using an open standard. The direct comparison to CC-NIC (Section 7.2, page 12, Figure 14), which used a proprietary interconnect (UPI), is crucial. It effectively argues that CXL provides a standardized way to achieve the benefits of cache coherence that were previously confined to specialized, proprietary systems.

4. Compelling Application-Level Demonstration: The KVS co-acceleration use case (Section 5.4, page 9) is a powerful demonstration of the architecture's potential. By hosting "hot" data in the NIC's coherent memory and using CXL.cache to handle misses by fetching from host memory, the authors showcase a seamless, hardware-managed tiered memory system spanning the host and the device. This is a glimpse into the future of tightly integrated heterogeneous computing and elevates the paper beyond just being a networking study.

Weaknesses

As a contextual analyst, I view these less as flaws and more as areas ripe for future discussion and exploration.

1. The Inevitable FPGA vs. ASIC Question: The evaluation is commendably performed on real hardware, which is a significant strength. However, the performance is necessarily limited by the FPGA's clock frequency and the maturity of the CXL IP. While the relative gains are clear, the absolute performance numbers may not fully represent the potential of an ASIC implementation. The discussion could benefit from some thoughtful speculation on how these architectural benefits would scale at higher line rates (400G+) and with the lower latencies of an ASIC design.

2. Software and Programmability Implications: The paper proposes a "CXL-NIC DPDK" framework, which mirrors the familiar DPDK model. This is a practical choice for evaluation. However, the paradigm shift from explicit DMA management to an implicitly coherent, NUMA-like memory model is profound. Does this genuinely simplify the programming model for application developers in the long run? A deeper discussion on the software abstractions needed to manage this new hardware—particularly around NUMA-awareness, data placement policies, and debugging—would add significant value.

3. Limited Scope of Application Co-Design: The KVS example is excellent but stands alone. The true power of this architecture lies in its generality. A brief discussion on other application domains that could be similarly transformed (e.g., distributed databases, HPC communication libraries like MPI, AI/ML inference serving) would help to broaden the paper's perceived impact.

Questions to Address In Rebuttal

1. The discussion in Section 8 regarding the lack of atomic operations support in CXL 1.1 is critical for multi-queue, multi-threaded scenarios. Could the authors elaborate on the practical scalability limitations this imposes on the current prototype? Are there software-based workarounds (e.g., delegating synchronization points to a single hardware engine) that could mitigate this until CXL 2.0 atomics are widely available?

2. The intelligent use of the NC-P (non-cacheable push) operation (Section 5.3, page 8) to inject data into the host LLC is a fascinating parallel to Intel's DDIO. The proposed "adaptive push-write gating" and "post-push write-back" mechanisms seem to address the classic "LLC pollution" problem. Could you comment on the complexity of tuning these mechanisms? How sensitive are they to workload characteristics, and do you envision this being managed by the driver or a higher-level runtime system?

3. Your latency-optimal configuration (Section 7.2, page 11) involves a hybrid memory layout (some structures on the NIC, some on the host). This co-optimization is a key result. How did you arrive at this specific configuration? Does this suggest that a "one-size-fits-all" data placement strategy is suboptimal, and that future systems will require runtime profiling and dynamic data migration between host and NIC memory to achieve the best performance?

Review Form

Reviewer: The Innovator (Novelty Specialist)

Summary

This paper presents CXL-NIC, a Network Interface Controller architecture built on the Compute Express Link (CXL) standard. The authors aim to overcome the well-documented limitations of PCIe for low-latency networking by leveraging CXL's coherence and memory semantics. The work proposes a Type-1 CXL-NIC using CXL.cache to replace inefficient DMA/MMIO operations, and extends this to a Type-2 design that utilizes CXL.mem for coherent on-NIC memory, enabling flexible data placement and application offloading. The core claims of novelty appear to be: (1) a set of datapath optimizations using specific CXL.cache line state controls (e.g., CO-read for polling, CS-read for prefetching), and (2) an intelligent mechanism for pushing data into the host LLC using NC-P with feedback control.

While the paper presents a timely and well-executed systems study on real hardware, its claims of conceptual novelty are overstated. Many of the core architectural ideas are adaptations of patterns previously established in the literature on proprietary coherent interconnects. The primary contribution of this work is not the invention of new coherent networking patterns, but rather the mapping of these known patterns onto the CXL standard's specific primitives and the first comprehensive performance evaluation of such a system on a real CXL-enabled FPGA platform.

Strengths

1. First-of-a-Kind Implementation: This work appears to be one of the first, if not the first, to design, implement, and evaluate a full-featured NIC on a real CXL hardware platform (Intel Agilex-7). Moving from simulation to a real-world prototype is a significant and valuable contribution to the community.

2. Thorough Exploration of CXL Primitives: The paper does an excellent job dissecting the CXL.cache protocol and mapping its specific request types (NC-write, CS-read, CO-read, NC-P) to different networking operations (packet transfer, descriptor prefetch, tail pointer polling). This detailed analysis in Sections 4.3 and 5.3 is valuable for future work in this area.

3. Demonstration of CXL's Potential: The experimental results effectively demonstrate the performance benefits of using a coherent interconnect like CXL over traditional PCIe, providing concrete data to support the ongoing industry shift.

Weaknesses

The central weakness of this paper is the limited conceptual novelty of its core architectural ideas. A search of prior art reveals significant conceptual overlap, primarily with work on NICs attached via other coherent fabrics.

1. Coherent Polling is Not New: The "event-driven Tx datapath" described in Section 4.3, where the NIC uses a CO-read request to poll a tail pointer in host memory, is the central optimization for the Tx path. This mechanism is functionally identical to the "inline signaling" technique proposed and implemented in CC-NIC [48]. In CC-NIC, the NIC also polls a descriptor location in host memory using a coherent read, leveraging the UPI protocol to remain quiet until the host CPU writes to that location. The authors of this paper even state they "adopt the inline signaling technique from CC-NIC [48]". This is therefore an adaptation of a known technique to a new protocol, not a novel architectural concept.

2. Data Pushing to LLC is Conceptually Similar to DDIO: The "Intelligent Usage of NC-P" (Section 5.3) to push data directly into the host LLC is presented as a key CXL-specific optimization. However, this is conceptually the same goal as Intel's Direct Data I/O (DDIO). The "leaky DMA" problem mentioned is a well-known issue with DDIO. The proposed solutions—"Adaptive push-write gating" and "Post-push write-back"—are simple, reactive control heuristics (a configurable flag and a delayed write). While these mechanisms are new in their specific implementation, they represent incremental control knobs on an existing concept rather than a fundamentally new architectural approach to I/O data delivery.

3. Coherent On-NIC Memory is an Established Concept: The idea of using coherent, device-attached memory for networking structures (Section 5) has been explored before. CC-NIC [48] already proposed a "cache-coherent interface to the NIC" with the possibility of writer-homed buffers. Furthermore, platforms like Enzian [7] and work like Dagger [26] have extensively explored tightly coupling FPGAs to CPUs via proprietary coherent links for accelerating RPCs and other network functions. The novel element here is the use of the CXL.mem standard, but the architectural pattern of offloading state to coherent device memory is not new.

The delta between this work and CC-NIC [48] is primarily the interconnect (open-standard CXL vs. proprietary UPI) and the specific primitives used. For instance, CXL's NC-write avoids the need for CLFLUSH that CC-NIC required. This is an important finding about the CXL protocol's advantages, but it is not a novel architectural invention by the authors.

Questions to Address In Rebuttal

1. Please articulate the fundamental conceptual novelty of your proposed datapath optimizations over those presented in CC-NIC [48]. Beyond the fact that CXL provides different primitives than UPI (e.g., NC-write), what new architectural principle or communication pattern is being introduced here for the first time?

2. The proposed NC-P control mechanisms in Section 5.3 are presented as a key contribution. Can you argue why these simple, heuristic-based controls should be considered a significant research contribution, as opposed to a straightforward engineering solution to the known problems of DDIO-like mechanisms?

3. Given the significant conceptual overlap with prior work, would it be more accurate to frame the primary contribution of this paper as the first end-to-end design and characterization of a CXL-based NIC on real hardware, providing a roadmap for mapping known coherent communication patterns to the CXL standard?