ccAI: A Compatible and Confidential System for AI Computing

---

Review Form

Reviewer: The Guardian (Adverserial Skeptic)

Summary

The authors present ccAI, a hardware-software co-design intended to provide confidential computing for AI workloads on heterogeneous systems with legacy xPUs. The system's architecture is anchored on a hardware module, the PCIe Security Controller (PCIe-SC), which intercepts and secures PCIe traffic between a Trusted VM (TVM) and an xPU. This is complemented by a software component, the "Adaptor," which operates within the TVM to manage security operations without modifying the user application or the xPU's native driver stack. The core proposition is that by operating at the PCIe packet level, ccAI can offer a compatible, transparent, and secure solution for a wide range of xPUs that lack native confidential computing features.

Strengths

1. Problem Motivation: The paper correctly identifies a critical and practical gap in the current ecosystem: the vast majority of deployed xPUs lack the confidential computing capabilities of cutting-edge hardware like the NVIDIA H100, yet they process sensitive AI workloads. Addressing this is a worthwhile endeavor.
2. Architectural Approach: The central idea of leveraging the PCIe interconnect as a universal enforcement point is logical. Since PCIe is the de facto standard, this approach has the potential to be more broadly applicable than solutions tied to specific xPU architectures or TEE designs.
3. Prototyping Effort: The authors have clearly invested significant engineering effort in building a functional prototype. Implementing the PCIe-SC on an FPGA and integrating it with five distinct, real-world xPUs from multiple vendors (NVIDIA, Tenstorrent, Enflame) is a non-trivial accomplishment and lends a degree of credibility to the feasibility of the design.

Weaknesses

My primary concerns with this submission relate to the strength of its claims regarding compatibility, security rigor, and the representativeness of the performance evaluation.

1. Overstated Compatibility and Transparency: The claim of "no xPU SW changes" (Table 2, page 10) is a significant overstatement. The paper details the introduction of a new kernel module, ccAI_adaptor, within the TVM (Section 7.1, page 8). This module is not part of the original xPU software stack and creates a new, non-trivial dependency. It interacts with the PCIe-SC, allocates memory, and handles encryption. Any update to the native xPU driver that alters its memory access patterns, DMA semantics, or MMIO interactions could break the assumptions made by the Adaptor, requiring a corresponding update. This is not seamless transparency; it is shifting the modification burden from the driver itself to an adjacent, tightly-coupled kernel module. The compatibility is therefore fragile.

2. Insufficient Scrutiny of the TCB and Security Guarantees: The security of the entire system hinges on the correctness of the PCIe-SC and its configuration.

   • Hardware Complexity: The PCIe-SC implementation consumes 218.6K ALUTs and 195.7K logic registers (Table 3, page 11). This is a substantial and complex piece of hardware, effectively a sophisticated NIC/firewall on the PCIe bus. A hardware design of this complexity is a major attack surface in itself. The paper provides no evidence of formal verification or rigorous testing to ensure the hardware is free of critical bugs that could be exploited to bypass security policies.
   • Packet Filter Brittleness: The security enforcement relies on a set of L1/L2 table rules (Figure 5, page 6). How are these rules generated? The paper glosses over this critical process. For any new xPU or even a new driver version, this rule set must be perfectly defined to distinguish between benign and malicious traffic. An incorrect or incomplete rule set is tantamount to an open door. This suggests a high operational burden and a high risk of misconfiguration, which undermines the practical security guarantees.
   • Unaddressed Threat Vectors: The paper dismisses side-channel attacks as orthogonal (Section 2.2, page 4). However, the PCIe-SC itself introduces new potential side channels. For instance, packet processing time within the PCIe-SC could vary depending on the security action taken (e.g., pass-through vs. decryption). An attacker monitoring PCIe bus traffic timing could potentially infer information about the operations being performed. This is not an orthogonal issue; it is a new vulnerability created by the proposed architecture.

3. Unconvincing Performance Evaluation: The reported low overhead figures (0.05% - 5.67%) appear to be the result of testing under favorable, compute-bound conditions, which do not sufficiently stress the ccAI architecture.

   • Non-Representative Workloads: The majority of the LLM evaluations, particularly in Figure 9 (page 11), are performed with a batch size of 1. While useful for latency measurements, this configuration is heavily compute-bound and minimizes the relative impact of I/O overhead. In real-world cloud inference scenarios, throughput is maximized by using larger batch sizes, which dramatically increases the volume of data traversing the PCIe bus. The evaluation fails to demonstrate how ccAI performs under such I/O-intensive, high-throughput conditions where the per-packet overhead of the PCIe-SC would be most pronounced.
   • Insufficient Stress Testing: The stress test in Section 8.6 (page 12) artificially limits PCIe bandwidth but does not present a workload that saturates the link. The crucial test is one where the application is bottlenecked by PCIe bandwidth in the baseline configuration. Only then can the true overhead of ccAI's cryptographic and filtering operations be accurately measured. The current test does not provide this insight.

Questions to Address In Rebuttal

1. Please reconcile the claim of "no software changes" with the necessity of the ccAI_adaptor kernel module. How do the authors guarantee that the Adaptor will remain compatible across future, potentially significant updates to the proprietary xPU drivers it must coexist with?

2. Given the substantial complexity of the PCIe-SC hardware (Table 3), what specific steps were taken to verify its functional correctness and security properties? Was any form of formal methods or exhaustive simulation employed to prove it is not a source of vulnerabilities itself?

3. The paper's performance claims hinge on benchmarks that are not I/O-bound (e.g., batch size 1). Please provide performance evaluation data for a high-throughput scenario where the workload is designed to saturate the PCIe bus (e.g., large-batch inference on a model with significant weight loading, or a data-intensive training workload).

4. Please elaborate on the generation and management process for the Packet Filter rules (Figure 5). Is this a manual, per-device, per-driver process? If so, how does this scale and how do you prevent human error from introducing critical security flaws? What is the performance penalty for rule lookup as the rule set grows in complexity?

---

Review Form:

Reviewer: The Synthesizer (Contextual Analyst)

Summary

This paper presents ccAI, a novel system designed to provide confidential computing for a wide range of AI accelerators (xPUs). The authors identify a critical gap in the current landscape: while cutting-edge hardware like the NVIDIA H100 offers built-in confidentiality, the vast majority of deployed accelerators lack such features, and existing academic solutions often suffer from poor compatibility or require significant application/driver modifications.

The core contribution of ccAI is its architectural choice to enforce security at the PCIe interconnect level. This is achieved through a hardware-software co-design comprising two key components: 1) a PCIe Security Controller (PCIe-SC), a hardware module that sits between the host and the xPU to intercept, filter, and process all PCIe packets, and 2) a software "Adaptor" running within a Trusted VM (TVM) on the host, which transparently manages the secure workflow without altering the user application or the native xPU drivers. By treating the PCIe packet stream as a universal interface, ccAI aims to deliver a compatible, transparent, and secure solution for heterogeneous AI computing environments.

Strengths

1. A Pragmatic and Powerful Architectural Choice: The single most important idea in this paper is the decision to place the security boundary on the PCIe bus. This is a brilliant move that reframes the problem. Instead of trying to secure the infinitely complex internals of various xPUs or modify proprietary driver stacks, the authors treat the accelerator as a black box and secure its sole communication channel with the host. This abstraction is the key to achieving the paper's primary goal of compatibility. It elegantly sidesteps the vendor-specific details that have plagued many previous approaches.

2. Excellent Contextualization and Problem Framing: The authors demonstrate a clear and panoramic understanding of the confidential computing landscape. Figure 1 (Section 3, page 3) is particularly effective, providing a concise taxonomy of existing approaches (TEE-based, HW-based, TDISP, etc.) and clearly positioning ccAI as a distinct and complementary solution. The paper correctly identifies that while standards like TDISP are the long-term future, there is a pressing, immediate need for a solution that can secure the massive installed base of legacy hardware. ccAI is presented not just as a research project, but as a practical answer to a real-world market and operational gap.

3. Strong System-Oriented Design: The work goes beyond a simple conceptual model. The authors have considered the full lifecycle of a secure workload, including a secure boot process for the PCIe-SC, remote attestation protocols (Section 6, page 7), and key management. The design of the Packet Filter and Packet Handlers (Section 4, page 5) shows a thoughtful approach to balancing security and performance by categorizing packet types and applying tailored protection policies. This level of detail suggests a mature and well-considered system design.

4. Comprehensive and Convincing Evaluation: The experimental validation is a significant strength. By implementing a prototype and testing it across five distinct xPUs from three different vendors (NVIDIA GPUs, a Tenstorrent NPU, and an Enflame GPU, as detailed in Section 7, page 8), the authors provide strong evidence for their central claim of compatibility. The low performance overheads reported across a wide range of Large Language Models (LLMs) further bolster the argument for the system's practicality.

Weaknesses

While the core idea is strong, the paper could be improved by addressing the practical implications of its proposed hardware.

1. The Practicality of the PCIe-SC: The entire system hinges on the existence and deployment of the PCIe-SC hardware module. While an FPGA prototype is excellent for academic validation, the path to real-world deployment is a significant hurdle. Is this envisioned as a standalone PCIe card that an xPU plugs into? An integrated component on future motherboards? An offering from a third-party hardware security company? The paper lacks a discussion of the form factor, cost, and supply chain implications, which are crucial for assessing its potential for widespread adoption in cloud data centers.

2. Scalability and Performance Ceiling: The evaluation demonstrates low overhead on current hardware. However, the bandwidth requirements of next-generation accelerators are growing exponentially. The prototype is based on an Intel Agilex 7 FPGA. A critical question is whether this architecture can scale to saturate the PCIe Gen5/Gen6 links of future flagship GPUs without becoming the primary performance bottleneck. While an ASIC implementation would be faster, some analysis of the architectural throughput limits would strengthen the paper's claims of future viability.

3. The Trusted Computing Base (TCB) of the Intermediary: By introducing the PCIe-SC, the authors have created a new, critical piece of hardware that becomes part of the TCB. The security analysis in Section 8.2 (page 10) is good, but it could further explore the threat model against the PCIe-SC itself. What is the mechanism for securely updating its firmware? How is its physical integrity maintained beyond the proposed chassis sealing? While the TCB is small compared to a full driver stack, its criticality warrants a more in-depth discussion.

Questions to Address In Rebuttal

1. Could the authors elaborate on the envisioned deployment model for the PCIe-SC in a typical cloud environment? What would be the most likely path to market for such a device, and what are the primary barriers to its adoption by cloud providers?

2. The paper compares ccAI's performance overhead to the NVIDIA H100's confidential mode. Could you discuss the architectural trade-offs? While ccAI is more compatible, does the H100's tight integration of security features provide fundamental performance or security advantages that an external PCIe device can never fully match?

3. How do the authors see ccAI co-existing with the emerging TDISP standard in the long term? Is ccAI primarily a bridging solution for the next 5-10 years until TDISP is ubiquitous, or does the packet-level inspection and policy enforcement of the PCIe-SC offer complementary security guarantees that would remain valuable even in a TDISP-enabled ecosystem?

---

Review Form

Reviewer: The Innovator (Novelty Specialist)

Summary

The paper proposes ccAI, a system for retrofitting confidential computing capabilities onto legacy, general-purpose xPUs (GPUs, NPUs, etc.) within a heterogeneous cloud environment. The authors' central claim of novelty rests on their specific architectural approach, which aims to provide strong security guarantees with high compatibility and user transparency, a combination they argue is lacking in prior art.

The system is composed of two primary components: (1) a hardware module, the PCIe Security Controller (PCIe-SC), which is physically interposed between the host's PCIe bus and the target xPU, and (2) a software component, the Adaptor, which resides in a Trusted VM (TVM) and coordinates with the PCIe-SC. The core mechanism involves the PCIe-SC intercepting all PCIe traffic to and from the xPU at the packet level. It uses a set of filtering rules and handlers to enforce security policies, such as encrypting/decrypting data payloads for DMA and validating MMIO operations, while remaining transparent to the unmodified xPU driver and user application.

The core novelty is not in the individual ideas of hardware-based security or using TEEs, but in the specific architectural synthesis: using an external, packet-level PCIe security module to create a confidentiality boundary for unmodified, legacy hardware and software stacks. This positions ccAI as a solution for the vast ecosystem of existing accelerators that lack the built-in features of NVIDIA's H100 or do not comply with emerging standards like TDISP.

Strengths

1. Novel Architectural Niche: The primary strength of this work is its novel architectural arrangement. While prior art has explored confidential accelerators, they typically fall into three categories that ccAI cleverly sidesteps:

   • Device-Integrated Hardware (e.g., NVIDIA H100 [50]): These solutions are vendor-specific, proprietary, and only available on the latest, most expensive hardware. ccAI's approach of externalizing the security module makes it, in principle, vendor-agnostic and applicable to legacy devices.
   • TEE-based Software Modifications (e.g., Cronus [40], CAGE [85]): These systems often require significant and complex modifications to the xPU driver stack to partition it and run critical parts within a TEE. ccAI's proposed hardware/software co-design aims for a much smaller software footprint (the "Adaptor") and claims to leave the complex driver stack untouched, a significant delta in terms of transparency and compatibility.
   • Forthcoming Standards (e.g., TDISP): These require compliance from the CPU, platform, and the xPU device itself. The novelty of ccAI is that it provides a concrete solution for the present-day reality where such compliant hardware is not widely deployed.

2. Packet-Level Abstraction: Anchoring the security mechanism at the level of the PCIe packet (as detailed in Section 4, page 5) is a powerful and novel choice for this specific problem domain. PCIe is the lingua franca of host-device communication. By operating at this level, ccAI establishes a uniform enforcement point that is conceptually independent of the specific xPU architecture (e.g., GPU vs. NPU), thus providing a more generalizable solution than prior works that are often tailored to a specific accelerator's command submission workflow.

Weaknesses

1. The "In-line Security Appliance" Precedent: While the application to confidential xPU computing is novel, the fundamental concept of an in-line hardware appliance on a communication bus (be it PCIe, Ethernet, etc.) that filters, modifies, and secures traffic is not entirely new. The paper would be strengthened by more clearly positioning its novelty against this broader class of "bump-in-the-wire" security devices. The novelty is in the details of the packet handlers and the co-design with the TVM-side Adaptor, not the general idea of intercepting traffic.

2. Generalization Claim vs. Inherent Specificity: The paper's core claim is broad compatibility. However, the true novelty of a general solution is tested by its ability to handle diversity without becoming a collection of special cases. The "Packet Filter" (Section 4.1, page 6) relies on rules based on address spaces and requester IDs. Different xPUs and their drivers use MMIO and DMA regions in vastly different, sometimes idiosyncratic, ways. The paper does not provide sufficient evidence that a simple, generalizable rule set can be defined for truly disparate devices (e.g., an NVIDIA GPU vs. a Tenstorrent NPU) without requiring extensive, device-specific reverse engineering and tuning. The delta between ccAI and a device-specific solution may be smaller in practice than claimed.

3. Handling of Proprietary Sidebands and Packets: The architectural novelty assumes all critical communication happens over standard, well-documented PCIe packets. However, many complex devices use vendor-defined message types or sideband communication channels that may not be visible or interpretable as standard DMA/MMIO. The brief mention of "Customized packets" in the Discussion (Section 9, page 12) acknowledges this but doesn't fully address the threat to the novelty of a universal solution. If the PCIe-SC cannot parse or secure these proprietary flows, the security guarantees are incomplete, reducing the significance of the advancement.

Questions to Address In Rebuttal

1. The novelty of the "Adaptor" component hinges on it being a lightweight, minimally intrusive module. For a completely new xPU not evaluated in the paper (e.g., an AMD GPU or a Google TPU), what would be the precise engineering effort required to develop the necessary Adaptor hooks and Packet Filter rules? Please provide a concrete, step-by-step process. This will help clarify whether the solution is genuinely general or just a framework for creating bespoke drivers.

2. The packet filtering mechanism (Figure 5, page 6) is key to the design. Can the authors provide a concrete example of a non-trivial security policy that differentiates between two distinct operations on the same GPU? For example, how would the L1/L2 table rules distinguish between a legitimate kernel launch command write and a malicious MMIO write attempting to access a configuration register that could compromise isolation?

3. The proposed PCIe-SC is a novel hardware component. Its viability depends on its ability to scale with technology. The prototype is evaluated on a PCIe 4.0 system. What are the architectural bottlenecks in the PCIe-SC design (e.g., table lookup logic, cryptographic engine throughput) that would need to be overcome to support the line rates and lower latencies of future PCIe 6.0 or 7.0 interfaces? Is the proposed architecture fundamentally scalable?