Elk: Exploring the Efficiency of Inter-core Connected AI Chips with Deep Learning Compiler Techniques

Review Form:

Reviewer: The Guardian (Adverserial Skeptic)

Summary

The authors present ELK, a compiler framework designed to optimize the performance of deep learning models on Inter-core Connected AI (ICCA) chips equipped with off-chip High Bandwidth Memory (HBM). The paper identifies a key performance challenge as the three-way contention between on-chip compute, inter-core communication, and off-chip I/O. ELK's proposed solution involves an inductive operator scheduling policy, a cost-aware memory allocation algorithm, and a preload order permutation strategy to navigate these tradeoffs. The evaluation, conducted on an emulation framework and a simulator, claims to achieve up to 94.84% of an ideal roofline performance.

While the problem is well-defined and relevant, the work suffers from significant internal contradictions regarding its optimality claims and relies on an evaluation methodology with questionable fidelity, particularly in its baselines and HBM emulation.

Strengths

1. Problem Formulation: The paper does a commendable job of identifying and articulating the fundamental performance tradeoffs on ICCA chips with HBM. The characterization of contentions in on-chip memory space, interconnect bandwidth, and memory access (Section 2.3, page 3) provides a clear motivation and foundation for the work.
2. Holistic Approach: The framework attempts to co-optimize multiple interdependent factors (tile sizes, preload depth, data placement, preload order), which is the correct direction for tackling this complex problem. Structuring these factors as a searchable compiler space is a logical approach.
3. Design Space Exploration: The sensitivity analysis presented in Section 6.4, which explores the impact of varying network topologies and bandwidths, provides useful, quantified insights for architects of future ICCA systems.

Weaknesses

1. Contradictory Claims of Optimality: The paper makes strong but unsubstantiated claims of optimality. Section 4.2 (page 6) states that the proposed scheduling algorithm "provably finds the end-to-end plan with the shortest total time". However, this claim is directly contradicted by the core memory allocation algorithm described in Section 4.3 (page 7), which employs a greedy heuristic to select from Pareto-optimal plans ("selects the most 'cost-effective' operator"). Furthermore, the preload order permutation in Section 4.4 (page 8) relies on heuristics to prune the search space, such as only reordering "HBM-heavy" operators. An algorithm that relies on greedy heuristics in its critical sub-problems is not globally optimal. The claims of optimality are misleading and must be rectified.
2. Flawed "Ideal" Baseline: The "Ideal" baseline used for comparison is defined as a theoretical machine with separate, contention-free interconnects for preload and execution, and no on-chip memory capacity constraints (Section 6.1, page 10). This is a physically unrealizable architecture. Evaluating against this unrealistic ideal serves to inflate the reported performance figures (e.g., "achieves 94.84% of the ideal performance"). The purpose of a scheduler on a real machine is to manage contention, not to perform on a magical machine where none exists. A rigorous evaluation requires a comparison against a much tighter, more realistic upper bound, such as a solution from a constraint solver for a small model or an oracle with perfect future knowledge on the emulated hardware.
3. Low-Fidelity HBM Emulation: The experimental methodology for emulating HBM access is a significant concern. The authors state they use a single core on their IPU-POD4 to act as an "HBM controller" that broadcasts data to all other cores (Section 5, page 9). This one-to-many broadcast pattern is not representative of a real system with multiple HBM controllers connected to the interconnect fabric, as depicted in Figure 1. A real system would generate complex many-to-many traffic patterns, leading to different and likely more severe contention scenarios. This simplification calls into question the validity of the interconnect utilization and contention results, which are central to the paper's claims.
4. Insufficient Justification of Novelty: The core algorithmic components are not sufficiently differentiated from standard practices. The intra-operator tradeoff analysis involves generating a Pareto-optimal curve of plans, and the inter-operator tradeoff is resolved with a greedy heuristic. This general approach is common in resource allocation problems. The paper fails to adequately argue for the specific novelty of its cost-aware allocation algorithm beyond its application to this specific hardware context.

Questions to Address In Rebuttal

1. Please clarify the central claim of your paper. Is the ELK framework optimal or heuristic? You must reconcile the claim of a "provably" optimal scheduling algorithm in Section 4.2 with the explicit use of greedy heuristics for memory allocation in Section 4.3 and for search space pruning in Section 4.4.
2. Please justify the selection of your "Ideal" baseline. Given that it models a physically impossible machine, how can it serve as a meaningful benchmark for a scheduler whose primary role is to manage contention? Could you provide results against a more grounded upper bound, even for a smaller problem, to demonstrate the true quality of your scheduler?
3. Can you provide a more detailed defense of your HBM emulation methodology? Specifically, how does a single-source broadcast traffic pattern accurately model the interconnect contention generated by multiple, distributed HBM controllers? What is the potential impact on your results if a more realistic many-to-many traffic pattern was modeled instead?
4. The decision to only reorder "HBM-heavy" operators is a critical heuristic for tractability. What analysis was performed to validate this heuristic? Please provide data or a strong argument to show that the performance potential from reordering operators with smaller memory footprints is negligible.

---

Review Form

Reviewer: The Synthesizer (Contextual Analyst)

Summary

This paper introduces ELK, a compiler framework designed to optimize the performance of Deep Learning (DL) models on a class of hardware I will refer to as Inter-Core Connected AI (ICCA) chips. The central thesis is that the performance of these architectures, which feature both large, distributed on-chip SRAM and high-bandwidth off-chip memory (HBM), is governed by a fundamental three-way trade-off between on-chip computation, inter-core communication, and off-chip I/O.

The authors' core contribution is to formalize this trade-off space and develop a compiler that systematically explores it. ELK employs several novel techniques, including a two-level inductive scheduling policy to manage the overlap between execution and preloading, a cost-aware memory allocation algorithm to optimally partition on-chip SRAM, and a preload order permutation strategy to minimize memory pressure and interconnect contention. The work is evaluated through a robust framework that includes an emulator built on real Graphcore IPU hardware and a flexible simulator for architectural design space exploration. The results demonstrate that ELK can achieve performance close to 95% of a theoretical roofline, significantly outperforming more straightforward compilation strategies.

Strengths

1. Excellent Problem Formulation: The paper does a superb job of identifying and articulating the core challenge. The "fundamental tussle" among compute, communication, and I/O is not new in systems research, but its specific manifestation on ICCA chips is a timely and important problem. Figure 2 provides a wonderfully clear and concise conceptual diagram of the resource contentions that motivate the entire work. This clear framing elevates the paper, making its contributions easy to understand and appreciate.

2. Systematic and Principled Approach: ELK is not merely a collection of ad-hoc heuristics. The authors have constructed a principled framework that maps the high-level performance factors to concrete, tunable compiler parameters (as shown in Figure 4). The multi-stage optimization process, from inductive scheduling (§4.2) to cost-aware allocation (§4.3) and finally preload reordering (§4.4), represents a comprehensive and well-reasoned strategy for navigating the complex optimization space.

3. Significant Contribution to Architectural Exploration: Perhaps the most impactful aspect of this work is its utility beyond just being a compiler. The framework presented, particularly the simulator, serves as a powerful tool for architectural design space exploration (§6.4). By allowing researchers to quantify the impact of scaling HBM bandwidth, interconnect topology (mesh vs. all-to-all), and core count, ELK provides a methodology for co-designing future AI hardware and the software that will run on it. This connects the domains of compilers and computer architecture in a very meaningful way.

4. Novel and Insightful Techniques: The concept of preload order permutation (§4.4) is particularly insightful. The realization that the order of data preloading—not just the amount—can be manipulated to manage the lifetime of large tensors in on-chip memory (Figure 13) is a subtle but powerful optimization. It shows a deep understanding of the system's temporal behavior.

5. Strong and Convincing Evaluation: The decision to build both an emulator on real hardware and a configurable simulator provides the best of both worlds: the credibility of real-system measurements and the flexibility for what-if analysis. The performance breakdown in Figure 18(a) is especially compelling, as it visually demonstrates how ELK achieves its speedup by converting idle "preload" and "interconnect" time into productive "overlapped" time.

Weaknesses

While the work is strong, its placement in the broader landscape could be strengthened, and some practical limitations could be addressed more directly.

1. Portability of the Cost Model: The framework's effectiveness hinges on the accuracy of its performance cost models. While the authors demonstrate impressive accuracy for the IPU architecture (Figure 12), the effort required to port this model to a fundamentally different ICCA chip (e.g., SambaNova's RDU with its spatial dataflow model, or Cerebras's WSE) is non-trivial. The paper could benefit from a discussion on what architectural features are most critical to model and the potential challenges in adapting ELK to hardware with different programming or execution paradigms.

2. Limited Discussion on Dynamic Workloads: The optimization process described is entirely static and performed ahead-of-time (AOT). This is well-suited for models like BERT or GPT where the computation graph is fixed. However, an increasingly important class of models, such as Mixture-of-Experts (MoE), involves input-dependent dataflow. The authors briefly mention MoE in the discussion (§7), but a deeper analysis of how ELK's static assumptions would break down and how its principles might be adapted for just-in-time (JIT) compilation or runtime scheduling would broaden the work's context.

3. Connection to Classical HPC and Distributed Systems: The problem of overlapping I/O, communication, and computation is a cornerstone of high-performance computing (HPC). While the domain is different (intra-chip vs. inter-node), many of the underlying principles are analogous. The paper could be strengthened by explicitly connecting its techniques to concepts from the broader parallel computing literature, such as communication-avoiding algorithms, asynchronous I/O scheduling, or task-graph-based execution models (e.g., Legion, Dask). Doing so would help contextualize ELK's contributions for a wider audience.

Questions to Address In Rebuttal

1. Regarding the cost model's portability: Could the authors elaborate on the process of creating a cost model for a new ICCA architecture? What are the key architectural parameters that must be captured, and how much of the existing modeling infrastructure could be reused versus needing a complete rewrite?

2. Regarding dynamic workloads like Mixture-of-Experts (MoE): As discussed in Section 7, MoE requires loading different "expert" weights at runtime. How would the inductive scheduling approach handle this uncertainty? Would a hybrid online/offline strategy be necessary, where ELK pre-optimizes plans for each expert, and a runtime system selects them?

3. Could the authors comment on the relationship between their work and classical communication-avoiding algorithms in HPC? ELK's strategy of broadcasting shared data during preload to reduce inter-core traffic during execution (§3.3) seems spiritually similar. Are there deeper parallels or lessons from that domain that could inform future iterations of ELK?

---

Review Form

Reviewer: The Innovator (Novelty Specialist)

Summary

This paper presents ELK, a compiler framework designed to optimize the performance of deep learning models on Inter-core Connected AI (ICCA) chips that are augmented with off-chip HBM. The authors identify a core performance challenge as a "fundamental tussle" among three factors: per-core compute, inter-core communication, and off-chip I/O. The central claim of the paper is that ELK is the first framework to jointly and holistically optimize these three competing factors. The proposed novel contributions are a set of compiler techniques to navigate this trade-off space: (1) a two-level inductive operator scheduling policy to decide on the number of operators to preload, (2) a cost-aware memory allocation algorithm to partition on-chip SRAM between executing and preloading operators, and (3) a preload order permutation scheme to decouple preload order from execution order, aiming to reduce resource contention. The work is evaluated using an emulator built on real hardware and a custom simulator.

My analysis concludes that while the high-level concepts leveraged by the authors (e.g., overlapping operations, scheduling heuristics, Pareto-optimality) are well-established in the broader field of compilers and high-performance computing, their specific synthesis, formulation, and application to the unique resource constraints of ICCA chips represent a novel and meaningful contribution.

Strengths

1. Novel Problem Formulation: The primary strength of this work is its clear identification and formalization of the coupled resource contention problem on ICCA chips with HBM (as illustrated in Figure 2, page 2). Prior work has typically focused on subsets of this problem (e.g., compute/communication in T10 [34], or compute/I/O in traditional GPU compilers [74]). The articulation of the three-way trade-off and its mapping to concrete compiler parameters is a novel conceptual contribution that frames the problem space effectively.

2. Novel Combination and Application of Techniques: The paper introduces several techniques that, while inspired by existing principles, are applied in a novel context.

   • The Preload Order Permutation (Section 4.4, page 8) is the most distinct idea. Decoupling the preloading sequence from the execution sequence for entire DL operators to specifically manage on-chip memory lifetimes and interconnect traffic "rush hours" is a new approach in this domain. Standard prefetching schemes operate at a much finer granularity.
   • The Cost-Aware Memory Allocation (Section 4.3, page 6) synthesizes two known ideas—Pareto-optimal plan generation and greedy selection—into a novel heuristic for inter-operator resource partitioning. The idea of selecting plans from multiple operators' Pareto curves simultaneously to satisfy a global memory budget is a non-trivial and new formulation.

3. Systematic Exploration Framework: The ELK framework itself represents a novel system for systematically navigating the identified trade-off space. The way it structures the problem into a series of nested search and optimization stages (Figure 9, page 5) is a new and coherent methodology for this specific hardware architecture.

Weaknesses

My critique is centered on the degree of fundamental novelty of the underlying algorithmic building blocks. The paper could be strengthened by more explicitly positioning its work against broader prior art in scheduling and compiler optimization, beyond the immediate scope of DL compilers.

1. Component Techniques Are Adaptations of Prior Concepts:

   • Inductive Operator Scheduling (Section 4.2, page 6): The core idea of solving a scheduling problem by working backward from the final state and making locally optimal choices is a classic dynamic programming pattern. While the authors' formalization in Theorem 4.2 is useful, the algorithmic pattern itself is not fundamentally new. The novelty is in its application, not its invention.
   • Use of Pareto-Optimal Curves (Section 4.3, page 7): The use of Pareto frontiers to represent time-space trade-offs is a standard technique in compiler auto-tuning (e.g., Ansor [70]) and hardware design space exploration. The paper claims novelty in its use for inter-operator trade-offs, which is fair, but the foundational tool is not new.

2. Complexity vs. Benefit: The proposed solution is highly complex, involving a multi-stage search that includes exhaustive search over preload numbers, Pareto curve generation for every operator, a greedy heuristic for memory allocation, and a pruned search over permutations. The performance gains (e.g., 1.37x over the Static baseline) are significant but not orders of magnitude. A critical analysis is needed to ascertain if a simpler set of heuristics could have achieved a substantial fraction of these gains. The paper does not explore this, making it difficult to assess if the full novel complexity is justified. The novelty comes at the cost of a sophisticated and potentially fragile compilation pipeline.

Questions to Address In Rebuttal

1. The concept of generating multiple implementation plans for an operation and selecting one based on system-wide constraints is central to auto-tuning compilers like Ansor [70] and Halide. How does ELK's cost-aware memory allocation (Section 4.3) differ fundamentally from the search and cost-modeling strategies in these frameworks, beyond the specific context of ICCA preloading? Please clarify the key "delta" in the allocation algorithm itself.

2. The "Preload Order Permutation" (Section 4.4) can be viewed as a form of coarse-grained, software-managed, out-of-order preloading. Could the authors contrast their approach with established concepts in software prefetching and out-of-order execution, highlighting precisely what is novel about their method at the algorithmic level, rather than just the application target?

3. The entire framework relies on a series of heuristics (greedy allocation, permutation pruning based on HBM-heavy operators). How sensitive are the final results to these heuristics? For example, what is the performance impact if the cost model for inter-core transfer (Figure 12, page 7) has a 10-20% error rate, and how might that affect the greedy choices made during memory allocation? This would help clarify whether the novel framework is robust or brittle.