Debunking the CUDA Myth Towards GPU-based AI Systems: Evaluation of the Performance and Programmability of Intel's Gaudi NPU for AI Model Serving

Review Form

Reviewer: The Guardian (Adversarial Skeptic)

Summary

This paper presents a performance and programmability evaluation of the Intel Gaudi-2 NPU, positioning it as an alternative to the NVIDIA A100 GPU for AI model serving. The authors develop a suite of microbenchmarks to characterize primitive operations (compute, memory, communication) and evaluate two end-to-end workloads: recommendation models (DLRM) and large language models (LLM). The central thesis is that Gaudi-2 is a competitive alternative and that NVIDIA's dominance relies more on its software ecosystem than the CUDA programming model itself, a claim encapsulated by the provocative title, "Debunking the CUDA Myth."

While the paper provides a useful and detailed characterization of the Gaudi-2 architecture, its primary conclusions are undermined by significant methodological choices, internal inconsistencies, and unsupported claims. The work mistakes demonstrating Gaudi-2's competence in specific, favorable scenarios (e.g., large GEMMs) for a genuine challenge to the incumbent's deeply integrated hardware-software solution, especially in areas requiring fine-grained control and programmability.

Strengths

1. Comprehensive Microbenchmarking: The paper's most valuable contribution is the systematic, multi-faceted microbenchmark analysis in Section 3. The evaluation across GEMM, non-GEMM, memory access patterns, and collective communication provides a clear-eyed view of the Gaudi-2 processor's architectural characteristics.

2. Analysis of MME Reconfigurability: The reverse-engineering of the MME's configurable systolic array (Section 3.2, Figure 7) is insightful. The authors demonstrate convincingly how this feature improves compute utilization on irregularly-shaped GEMMs compared to a fixed-geometry systolic array. This is a solid piece of architectural analysis.

3. Highlighting System-Level Bottlenecks: The communication analysis (Section 3.4, Figure 10) correctly identifies the system-level interconnect (P2P RoCE vs. NVSwitch) as a key performance differentiator, appropriately separating the chip's capabilities from the server system's design.

Weaknesses

1. Misleading Title and Framing: The title "Debunking the CUDA Myth" establishes a premise the paper fails to support. The authors' own findings, particularly the struggles with programmability in the vLLM case study (Section 4.2), demonstrate the exact opposite: the tight coupling of the CUDA programming model with underlying hardware features like Tensor Cores (via WMMA APIs) constitutes a formidable advantage that the Gaudi software stack cannot replicate. The "myth" appears to be very much a reality.

2. Outdated and Convenient Comparison Point: The selection of the NVIDIA A100, a previous-generation accelerator, is a critical flaw. While justified on the basis of a shared process node (7nm), it is not the relevant competitor in the market at the time of publication. A comparison against the H100 would provide a far more honest assessment of Gaudi-2's competitiveness and would almost certainly show a much larger performance deficit, potentially invalidating the paper's core thesis. This choice appears engineered to make the Gaudi-2's performance seem more competitive than it is.

3. Major Contradiction in vLLM Performance Analysis: The vLLM case study (Section 4.2) contains a significant internal contradiction. The authors report that their optimized PagedAttention kernel on Gaudi-2 achieves only 45% of the A100's performance (Figure 17(c)). This is a catastrophic performance gap for the most critical component of transformer inference. Yet, they proceed to claim that the end-to-end performance is "similar" and "competitive" (Figure 17(d, e)), attributing this to Amdahl's Law and superior MLP performance. This is an extraordinary claim that lacks the necessary evidence. A detailed latency breakdown of the entire end-to-end inference process is required to substantiate how a >2x slowdown in the dominant kernel does not translate to a significant end-to-end slowdown. The current presentation is speculative and unconvincing.

4. Downplaying Critical Hardware and Programmability Limitations:

   • The paper identifies Gaudi-2's 256-byte minimum memory access granularity as a weakness but understates its impact. In the DLRM case study (Section 4.1), this limitation results in performance of only 47% of A100 for small embedding vectors. This is not a minor issue; it is a fundamental architectural mismatch for an entire class of important sparse workloads.
   • The authors admit that the core MME compute units are not directly programmable via TPC-C, contrasting sharply with CUDA's direct access to Tensor Cores. They frame their reliance on the "black-box" Gaudi graph compiler as a high-level programming paradigm, but for performance engineers, this is a severe limitation that prevents the implementation of novel, state-of-the-art kernels (e.g., a from-scratch FlashAttention).

5. Unsupported Attribution of Performance Gaps: In Key Takeaway #6 (Page 12), the authors claim the DLRM performance gap for small vectors "primarily stems from A100's superior hardware architecture rather than from the differences in the programming models." This is an unsubstantiated assertion. They provide no experiment or analysis to decouple these two factors. An equally plausible explanation is that the CUDA/FBGEMM software stack is simply better at scheduling memory operations and hiding latency for this access pattern—a direct function of the programming model and its compiler.

Questions to Address In Rebuttal

1. Please justify the "Debunking the CUDA Myth" framing. Given that your vLLM case study highlights a critical lack of low-level programmability for the MME and a resultant 55% performance deficit in the core PagedAttention kernel, in what precise way has the "myth" of CUDA's performant ecosystem been debunked?

2. Provide a clear and compelling rationale for benchmarking against the A100 instead of its contemporary, the H100. How would your central conclusions about Gaudi-2's competitiveness change if compared against the current state-of-the-art?

3. To resolve the contradiction in Section 4.2, provide a detailed profiling breakdown (e.g., a flame graph or latency-per-layer table) of an end-to-end LLM inference request on both Gaudi-2 and A100. This data must explicitly show the time spent in the attention kernels versus the MLP layers to validate your claim that the >2x slower attention performance is masked by faster MLP execution.

4. Regarding the conclusion in Key Takeaway #6, what specific analysis allows you to definitively attribute the poor performance on small embedding vectors to hardware alone, excluding the role of the TPC-C programming model, its compiler, or the runtime's ability to manage fine-grained memory requests?

Excellent. I will now embody "The Synthesizer" and provide a peer review for the paper.

---

Review Form

Reviewer: The Synthesizer (Contextual Analyst)

Summary

This paper presents a timely and comprehensive characterization of Intel's Gaudi-2 NPU, positioning it as a potential challenger to NVIDIA's long-standing dominance in the AI accelerator market. The authors draw a compelling parallel to the 2010 Intel paper that questioned the GPU's supposed 100x advantage over CPUs, framing this work as a spiritual successor that interrogates the "CUDA myth" in the modern AI era.

The work's core contribution is its multi-faceted evaluation methodology, which goes far beyond simple performance benchmarks. It synthesizes three critical dimensions:

1. Micro-architectural Performance: A deep dive into the raw capabilities of Gaudi-2's compute (MME, TPC), memory, and communication subsystems in comparison to the NVIDIA A100.
2. End-to-End Application Performance: An analysis of Gaudi-2's performance and energy efficiency on highly relevant, large-scale AI workloads—namely, Recommendation Systems (RecSys) and Large Language Models (LLMs).
3. Programmability and Software Optimization: Two insightful case studies on porting and optimizing state-of-the-art algorithms (FBGEMM for RecSys, vLLM for LLMs) to Gaudi, exploring the capabilities and limitations of its software stack at both the low-level (TPC-C) and high-level (PyTorch).

The authors conclude that while Gaudi-2's hardware is competitive and even superior in some aspects (e.g., GEMM-heavy LLMs), its ultimate success hinges on bridging the software and ecosystem gap. They argue that NVIDIA's true "moat" is less about CUDA as a language and more about the rich, mature software ecosystem built upon it, and that with sufficient investment in high-level framework integration, competitors like Intel can become viable alternatives.

Strengths

The primary strength of this paper is its holistic and contextual approach. It successfully elevates a hardware characterization study into a broader commentary on the dynamics of the AI systems landscape.

1. Excellent Framing and Historical Context: The explicit connection to the 2010 "Debunking the GPU Myth" paper (mentioned in the Introduction, page 2) is brilliant. It provides an immediate and powerful narrative frame, positioning Intel's current role as the "underdog" and lending historical weight to their investigation of the hardware-software power balance.

2. Comprehensive, Multi-level Analysis: The paper avoids the common pitfall of focusing only on peak FLOPS or a single workload. By connecting microbenchmark results (Section 3.2-3.4) to end-to-end application behavior (Section 3.5), the authors provide a much more nuanced and credible picture. For example, they identify Gaudi-2's weakness in small, random memory accesses (Key takeaway #3, page 9) and then demonstrate its real-world impact on embedding-heavy RecSys models (Figure 11, page 10).

3. Novel Focus on Programmability: The case studies in Section 4 are the most significant contribution of this work. While performance numbers are valuable, understanding the effort and methodology required to achieve that performance on a new architecture is arguably more important for the community. The vLLM case study (Section 4.2, page 12), in particular, is a masterful demonstration of the challenges of working with a less mature, more abstracted software stack. It highlights the critical role of the graph compiler and the limitations imposed by the lack of direct MME control, a crucial insight for both systems programmers and future hardware designers.

4. Connecting to the Broader Academic Landscape: This work sits at the intersection of several key research areas: computer architecture (DSA vs. GPU), systems software (vLLM, FBGEMM), and compilers (the role of MLIR and graph compilation). By evaluating a real, commercially significant system, it provides a grounded case study that is relevant to researchers across these domains. It serves as an excellent reference point for anyone looking to understand the practical challenges of moving high-performance computing beyond the NVIDIA ecosystem.

Weaknesses

The weaknesses of the paper are minor and largely inherent to the nature of such a rapidly evolving field, rather than fundamental flaws in the methodology or conclusions.

1. A Snapshot in a Moving Stream: The analysis is based on a specific version of the Intel Gaudi Software stack (v1.18.0). Given the immaturity of the ecosystem compared to CUDA, it is highly likely that many of the observed software limitations (e.g., the "black box" nature of the graph compiler, suboptimal library kernels) will be addressed in future releases. The paper acknowledges this implicitly, but it could be more explicit about how its conclusions might be altered by a more mature software stack.

2. Limited Scope of Competitors: The comparison is exclusively between Gaudi-2 and the A100. While this is a logical and well-justified choice (as explained on page 2), the AI accelerator landscape is diversifying with strong offerings from AMD (e.g., MI300) and other cloud providers' internal silicon. Acknowledging this broader context more directly in the discussion would strengthen the paper's panoramic view. The "Future Work" section (page 13) does a good job of this, but a brief mention earlier would be helpful.

3. The "Myth" Remains Partially Intact: The paper's title makes a bold claim. While it successfully argues that Gaudi's hardware is not the bottleneck, its own findings (especially in Section 4) show that the inability to easily program that hardware at a low level (a key feature of CUDA) remains a significant hurdle. One could argue the "CUDA myth" is less about the language itself and more about the philosophy of direct, flexible hardware control it represents, which Gaudi's stack currently abstracts away. The paper's conclusion touches on this, but the argument could be sharpened.

Questions to Address In Rebuttal

1. The vLLM optimization case study (Section 4.2) is fascinating. It shows how performance was recovered by restructuring the problem at the PyTorch level to better suit the Gaudi graph compiler. This seems to place a significant burden on the application developer to understand the compiler's preferences. How does this compare to emerging programming models like OpenAI's Triton, which also uses a high-level language (Python) but aims to give programmers more explicit control over performance-critical transformations? Could you speculate on whether a Triton-like interface for Gaudi would be a more effective path forward than improving the "black box" heuristics of the current graph compiler?

2. Your analysis shows that the Gaudi-2 hardware is highly competitive, yet the final performance is often dictated by the software stack. Given that software is a rapidly moving target, which of your key takeaways do you believe are most fundamental to the Gaudi architectural philosophy and are likely to remain true for future generations (e.g., Gaudi-3 and beyond), and which are more likely to be rendered obsolete by near-term software improvements?

3. Regarding the title, your work convincingly demonstrates that high-level frameworks can abstract away the need for a CUDA-like language for many AI practitioners. However, it also shows that for developers of cutting-edge libraries (like vLLM), the lack of low-level control is a major impediment. Does this suggest that the "CUDA myth" is not debunked, but rather bifurcated? That is, it's a "myth" for application users, but a "reality" for systems programmers and library developers who need to extract every last drop of performance from the hardware.

Here is a peer review of the paper from the perspective of "The Innovator."

---

Review Form

Reviewer: The Innovator (Novelty Specialist)

Summary

This paper presents a comprehensive performance and programmability evaluation of the Intel Gaudi-2 NPU, positioning it as a competitor to the NVIDIA A100 GPU for AI model serving. The authors conduct this evaluation across three primary axes: (1) low-level microbenchmarks for compute, memory, and communication primitives; (2) end-to-end performance on representative AI workloads (RecSys and LLMs); and (3) two case studies on programmability, demonstrating low-level (TPC-C) and high-level (PyTorch) optimization strategies. The central thesis is that the Gaudi NPU architecture is competitive and that the dominance of NVIDIA's ecosystem is more attributable to its software maturity than an insurmountable hardware or programming model advantage.

My evaluation focuses exclusively on the novelty of the contributions. While the paper is well-executed, its primary contribution is one of characterization and insight, rather than the creation of a novel artifact or technique. The work's novelty rests on its claim to be the first academic study to perform this specific comparison with this level of depth, particularly regarding programmability.

Strengths

The primary strength of this paper, from a novelty perspective, lies in its synthesis and depth. While individual components of the evaluation methodology are not new, their application to the Gaudi-2 architecture in direct, rigorous comparison to the A100 provides novel insights that are not present in prior literature.

1. Novel Insights into Programmability (Section 4, page 11): The most novel aspect of this work is the detailed exploration of Gaudi's programmability. Prior works, such as Emani et al. [14] and Zhang et al. [83], focused on running LLMs and reporting performance. This paper goes a crucial step further by attempting to optimize key kernels from the ground up. The case study on implementing a BatchedTable operator in TPC-C for DLRM (Section 4.1) and optimizing PagedAttention at the PyTorch level for vLLM (Section 4.2) provides a genuinely new perspective on the practical challenges and limitations of the Gaudi software stack, such as the black-box nature of the MME. This moves beyond performance measurement to an analysis of the development ecosystem itself.

2. Quantification of Architectural Trade-offs: The paper uncovers and quantifies specific architectural behaviors that were previously only qualitatively described in technical reports or were unknown. For example, the reverse-engineering and analysis of the MME's reconfigurability (Figure 7, page 6) is a novel contribution that explains Gaudi-2's unexpectedly high utilization on irregular GEMM shapes. Similarly, quantifying the sharp performance degradation for random memory accesses smaller than its 256-byte minimum granularity (Key takeaway #3, page 9) is a concrete, new data point for the community.

3. Explicit Differentiation from Prior Art: The authors clearly position their work against existing studies [14, 83] in Section 6 (page 14), correctly identifying the gaps they fill: a broader workload analysis (including RecSys), a focus on energy efficiency, and a deep dive into programmability. This demonstrates a clear understanding of the existing landscape and carves out a well-defined novel contribution.

Weaknesses

The weaknesses of this paper are intrinsically linked to its nature as a characterization study. The core ideas behind the methods used are not, in themselves, novel.

1. Methodological Non-Novelty: The use of microbenchmarks to probe hardware capabilities is a standard and well-established technique in computer architecture (e.g., [36, 50]). Similarly, evaluating end-to-end application performance is the standard for system papers. The novelty here is entirely in the subject of the study (Gaudi-2) rather than the method of study.

2. Overlap with "Folklore" and Technical Documentation: Some of the findings, while rigorously confirmed here for the first time in an academic venue, may exist as "folklore" or can be inferred from Intel's own documentation. For instance, the recommendation to unroll TPC loops to hide pipeline latency (Section 2.2, page 5) is a known best practice from Intel's developer guides [27]. The paper's contribution is to measure the precise impact of this practice (Figure 8b, page 8), which is valuable but represents a confirmation rather than a discovery.

3. Incremental Nature of Insights: While the paper presents several new insights, one could argue that they are incremental additions to the body of knowledge rather than a paradigm shift. The core finding—that a specialized accelerator is competitive with a GPU on some workloads but faces software maturity challenges—is a recurring theme in the history of domain-specific architectures. The value is in the specific details for Gaudi, but the high-level narrative is not new.

Questions to Address In Rebuttal

1. On the Significance of MME Insights: Your analysis of the MME's dynamic reconfigurability (Section 3.2, page 6) is presented as a key finding. However, given that the MME is not directly programmable by users and is managed by a proprietary graph compiler, what is the actionable, novel takeaway for the broader research community beyond "Intel's compiler is effective"? How does this insight inform the design of future open hardware or compiler stacks?

2. Novelty of Programmability Challenges: The vLLM case study (Section 4.2, page 12) highlights the difficulty of optimizing attention due to the lack of low-level MME control, forcing optimizations to the PyTorch level. Is this insight novel in the sense that it reveals a fundamental, permanent design choice in the Gaudi programming model, or is it simply a snapshot of the current SDK's immaturity? The novelty of this finding is significantly diminished if a future SDK release simply exposes the required low-level controls.

3. Delta Over Prior Art: Beyond being broader and including energy analysis, what is the single most significant architectural insight presented in this paper that fundamentally changes the understanding of Gaudi NPUs compared to the picture painted by prior work like Emani et al. [14] and Zhang et al. [83]? Please be specific.

4. Assessing the "CUDA Myth": The title invokes the 2010 ISCA paper "Debunking the 100X GPU vs. CPU Myth". That paper showed that with proper optimization, the performance gap between CPUs and GPUs was much smaller than claimed. Your paper finds that Gaudi-2's performance is competitive, but its programmability is a significant hurdle requiring substantial optimization effort (e.g., achieving only 45% of A100's performance on PagedAttention even after optimization). Does your work truly "debunk" the CUDA myth, or does it in fact reinforce it by demonstrating how critical a mature software ecosystem and direct hardware control (as provided by CUDA for Tensor Cores) are to achieving peak performance?