Leveraging Chiplet-Locality for Efficient Memory Mapping in Multi-Chip Module GPUs

---

Review Form

Reviewer: The Guardian (Adverserial Skeptic)

Summary

The authors propose CLAP (Chiplet-Locality Aware Page Placement), a memory management mechanism for multi-chip module (MCM) GPUs that aims to select an optimal effective page size for different data structures. The work is predicated on a property the authors term "chiplet-locality," which they define as the tendency for contiguous virtual pages to be accessed by the same chiplet. CLAP works by first profiling a fraction of memory allocations using small pages to identify this locality. It then uses a tree-based analysis to determine the largest granularity of contiguous pages that maintain locality and maps the remainder of the data structure using this effective size. These physically contiguous regions are then intended to be covered by a single, coalesced TLB entry, thereby achieving the translation efficiency of large pages while preserving the placement granularity of small pages.

Strengths

1. Problem Motivation: The paper does an adequate job of motivating the core problem. The trade-off between address translation overhead (favoring large pages) and data placement locality (favoring small pages) in an MCM context is a genuine challenge. The preliminary data in Figure 1 and Figure 2 (page 2) effectively illustrates that a one-size-fits-all approach to paging is suboptimal.

2. Core Concept: The high-level idea of creating physically contiguous page-like regions to match an application's access granularity is a logical approach to bypassing the need for extensive hardware support for numerous, fixed intermediate page sizes.

3. Evaluation Scope: The authors compare their proposal against a reasonable set of baselines, including static small and large pages, an idealized C-NUMA implementation, and other prior work in MCM GPU optimization.

Weaknesses

My primary concerns with this submission center on the foundational premise of "chiplet-locality," the introduction of several unscrutinized "magic numbers" in the methodology, and the potential oversimplification of hardware overheads.

1. The Foundational Premise of "Chiplet-Locality" is Circular: The paper's central claim rests on the existence of "chiplet-locality" as an intrinsic workload property. However, the evidence presented seems to be an artifact of the experimental setup. In Section 3.1 (page 4), the authors state their baseline uses a First-Touch (FT) policy, which inherently places data pages on the chiplet of the thread that first requests it. The analysis in Figure 10 (page 6), which shows very high chiplet-locality, is performed after this locality-aware policy has already been applied. Therefore, the paper is not measuring an intrinsic property of the application, but rather the effectiveness of the baseline FT policy. The claim that this is a fundamental characteristic of GPU workloads is not rigorously substantiated and appears to be a self-fulfilling prophecy.

2. Arbitrary Methodological Thresholds: The CLAP mechanism is governed by several key parameters that lack rigorous justification.

   • The Partial Memory Mapping (PMM) threshold is set to 20% (Section 4.2, page 8). The authors claim this is a "conservative choice empirically derived," but provide no sensitivity analysis to support this. For applications with distinct execution phases, the access patterns in the first 20% of page faults may be entirely unrepresentative of the subsequent 80%.
   • The Opportunistic Large Paging (OLP) mechanism is disabled if more than 5% of VA blocks release their reservations. This is another "magic number" presented without analysis. A robust system should not depend on such finely-tuned, unexplained constants.

3. Understated Hardware Complexity and Assumptions:

   • The Remote Tracker (RT) mechanism requires commandeering an "unused bit of the last-level page table entry (PTE)" to store an allocation ID (Section 4.3, page 8). While the authors claim modern PTEs have reserved bits (page 9), these are often targets for other system software or future hardware features. Assuming exclusive access to these bits for a single optimization is a significant architectural imposition.
   • The claimed area overhead for the TLB coalescing logic (0.0003% of the die area, Section 4.6, page 11) seems exceptionally low. While the logic itself may be simple, its integration into the critical path of the TLB/MMU, including control logic and potential timing implications, is non-trivial. This figure is presented without sufficient breakdown to be credible.

4. Evaluation Concerns: The "Ideal C-NUMA" baseline assumes zero latency for page migrations and related operations (Section 5, page 12). While this establishes an upper bound, it also creates a strawman. The primary advantage of a proactive scheme like CLAP should be its ability to avoid the high, non-zero costs of a reactive scheme. By idealizing the baseline, the paper obscures the true magnitude of this benefit and presents a comparison that is arguably flattering to the proposed work.

Questions to Address In Rebuttal

The authors must address the following points to substantiate their claims:

1. On "Chiplet-Locality": Please clarify the methodology used to generate the data in Figure 10. Can you demonstrate that "chiplet-locality" is an intrinsic property independent of the initial page placement policy? Specifically, what would the results of that analysis be if the initial PMM phase used a chiplet-agnostic policy, such as round-robin or random page placement?

2. On Methodological Robustness: Can you provide a detailed sensitivity analysis for the PMM threshold (e.g., varying from 5% to 50%) and the OLP disable threshold? How does the system's performance change, and how do you justify that your chosen values are optimal or robust across diverse workloads, particularly those with dynamic behavior?

3. On Application Dynamics: The CLAP mechanism is fundamentally proactive. How does it handle workloads where data access patterns change significantly after the initial PMM phase is complete (e.g., in different kernel invocations)? The "CLAP+migration" extension presented in Figure 20 (page 14) suggests this is a known limitation. Does this imply that for dynamic applications, the core benefit of CLAP is voided, requiring a full fallback to a reactive migration scheme?

4. On Hardware Feasibility: Please provide a more thorough justification for the claimed hardware overheads. Regarding the RT, what are the known or anticipated competing uses for the reserved PTE bits you intend to use? For the TLB coalescing logic, can you provide a more detailed breakdown of the components included in the 0.0024mm² area estimate and discuss its impact on TLB access latency?

---

Review Form

Reviewer: The Synthesizer (Contextual Analyst)

Summary

This paper addresses a fundamental tension in the memory systems of modern Multi-Chip Module (MCM) GPUs: the trade-off between address translation efficiency and memory access locality. The authors correctly identify that large pages, while beneficial for reducing TLB misses, can lead to poor data placement and increased remote memory traffic across chiplets. Conversely, small pages allow for fine-grained, locality-aware placement but suffer from higher translation overhead.

The core contribution is the identification and exploitation of a workload property the authors term "chiplet-locality"—the tendency for groups of virtually contiguous pages to be accessed predominantly by a single chiplet. Building on this insight, the paper proposes CLAP (Chiplet-Locality Aware Page Placement), a mechanism that proactively profiles data structures to determine their characteristic chiplet-locality granularity. CLAP then maps these page groups to physically contiguous frames on the appropriate chiplet, effectively creating "large page-like regions." These regions reap the address translation benefits of large pages (via a proposed TLB coalescing mechanism) without sacrificing the fine-grained placement necessary for high memory locality.

Strengths

The true strength of this paper lies in its elegant synthesis of ideas from operating systems, computer architecture, and parallel programming to solve a timely and important problem.

1. Excellent Problem Formulation and Contextualization: The paper does a superb job of positioning its work. It correctly identifies the rise of MCM GPUs as a pivotal shift in high-performance computing and clearly articulates the resulting NUMA-like challenges. The introductory analysis in Section 1 (page 1) and the motivational study in Section 3 (page 4) are compelling, effectively demonstrating that a one-size-fits-all paging policy is suboptimal. The paper situates itself perfectly at the intersection of physical data placement strategies (e.g., [13], [47]) and address translation optimizations (e.g., [32], [87]), arguing convincingly that these two aspects must be considered jointly.

2. The "Chiplet-Locality" Insight: While the underlying principle of spatial locality is well-known, its formalization as "chiplet-locality" in the context of MCM GPUs is a valuable conceptual contribution. It connects the structured parallelism of the GPU programming model (e.g., threadblocks) to the physical hierarchy of the hardware (chiplets). By observing that this locality has a consistent, per-data-structure granularity, the authors uncover a predictable behavior that is ripe for optimization. The quantification of this property in Figure 10 (page 6) provides a solid empirical foundation for the entire approach.

3. A Proactive, Low-Overhead Design: The proposed CLAP mechanism is a clever alternative to reactive, migration-based schemes like C-NUMA [28, 34], which are often ill-suited to the GPU's execution model and incur high overheads (e.g., TLB shootdowns). By using a brief, low-overhead profiling phase (PMM) at the beginning of an allocation's lifecycle, CLAP makes a one-time, predictive decision. This proactive approach avoids the complexities and performance penalties of continuous runtime monitoring and data migration, making it a much more natural fit for GPU systems.

4. Bridging the Gap Between Page Sizes: The solution of creating physically contiguous regions of small pages is elegant. It circumvents the need for complex hardware support for a multitude of arbitrary page sizes. Instead, it leverages the existing 64KB page infrastructure and relies on a well-defined TLB coalescing mechanism [86] to achieve the performance benefits of larger, intermediate page sizes. This makes the proposal practical and more easily adoptable.

Weaknesses

The weaknesses are not in the core idea, which is sound, but in the assumptions about workload behavior and the full implications of the proposed hardware.

1. Static Workload Assumption: The core design of CLAP seems best suited for applications where memory access patterns are established early and remain stable. The initial profiling during the PMM phase determines the mapping for the lifetime of the data structure. While the authors present an extension using page migration for dynamic scenarios (Section 5.2, page 14), this feels like an add-on rather than a fundamental part of the design. The effectiveness of CLAP could diminish in workloads with significant phase changes or highly dynamic memory allocation patterns where the initial profile quickly becomes stale.

2. Complexity of the Remote Tracker (RT): The paper presents the RT as a simple, low-area hardware addition. However, any modification to the GMMU and the page walk pipeline warrants scrutiny. The claim that the RT is "not on the critical path of memory accesses" (page 9) is asserted but could benefit from a more detailed analysis. For latency-critical applications, even minor delays or resource contention within the GMMU could have a performance impact.

3. Interaction with System-Level Schedulers: The concept of chiplet-locality relies on a relatively stable mapping of threadblocks to chiplets, as provided by the First-Touch or Static-Analysis policies. The paper does not explore how CLAP would interact with more dynamic, system-level schedulers that might perform load balancing by migrating threadblocks (or entire kernel grids) between chiplets. In such a scenario, the chiplet predicted to be the primary accessor during the PMM phase may no longer be correct later in the execution.

Questions to Address In Rebuttal

1. On Dynamic Behavior: The CLAP+migration experiment is promising. Could the authors elaborate on the criteria and overhead for triggering a re-evaluation of a data structure's mapping? For instance, would the Remote Tracker need to be extended to continuously monitor for pattern shifts post-PMM, and how would the system decide that the cost of migration is worth the benefit?

2. On the Remote Tracker's Criticality: Could the authors provide a more detailed breakdown of the interaction between a page walk and the RT? Is the RT lookup and update fully pipelined and/or asynchronous with the page walk's primary function of fetching a PTE, ensuring zero impact on translation latency?

3. On Broader Applicability: How does the efficacy of CLAP depend on the application having sufficient parallelism to saturate all chiplets? In cases where a workload only utilizes a subset of chiplets, would CLAP's analysis still hold, or would the concept of a "preferred" chiplet become less meaningful? Furthermore, how would CLAP handle data structures that are intentionally shared and frequently accessed by all chiplets (beyond the matrix-B GEMM example, which has a predictable broadcast pattern)?

Review Form

Reviewer: The Innovator (Novelty Specialist)

---

Summary

The paper introduces CLAP (Chiplet-Locality Aware Page Placement), a hardware/software co-design to manage memory pages in Multi-Chip Module (MCM) GPUs. The central problem addressed is the well-known trade-off between large pages (which reduce TLB overhead but can cause remote accesses and poor locality) and small pages (which improve locality at the cost of higher TLB pressure).

The authors' proposed solution is to determine a "suitable page size" on a per-data-structure basis. This is achieved by:

1. Profiling a small fraction (20%) of a data structure's pages using a first-touch policy with small base pages (64KB).
2. Using a new hardware unit, the Remote Tracker (RT), to monitor the locality of these initial placements.
3. Employing a driver-level tree-based analysis (MMA) to identify the largest granularity of virtually contiguous pages that are consistently accessed by the same chiplet—a property the authors term "chiplet-locality."
4. Mapping the remainder of the data structure by creating physically contiguous regions of this "suitable size" from base pages.
5. Leveraging a TLB coalescing mechanism to treat these physically contiguous base pages as a single, larger effective page in the TLB.

The core novel claim appears to be the synthesis of these components into a proactive, predictive system that structures physical memory to create "synthetic" large pages tailored to the observed access patterns of GPU applications.

Strengths

The primary strength of this work lies in its cohesive, full-stack approach to a fundamental problem. While the constituent parts of the solution are not entirely new in isolation, their combination and application to the MCM-GPU domain are well-conceived.

The most compelling aspect is the idea of proactively creating physical contiguity based on a profile. Instead of relying on reactive migration (like C-NUMA) or hoping for incidental contiguity (as in traditional memory allocators), CLAP deliberately engineers the memory layout to maximize the effectiveness of a TLB coalescing mechanism. This proactive stance, guided by the "chiplet-locality" heuristic, is an elegant way to get the benefits of large pages without paying their full locality penalty.

The concept of "chiplet-locality" itself, while an intuitive extension of spatial locality in parallel workloads, is framed and quantified in a useful manner, providing a clear target for the optimization.

Weaknesses

My main concern with this paper is the degree of novelty of its core technical components. When deconstructed, the mechanism appears to be a clever recombination of pre-existing concepts.

1. Dynamic Page Granularity: The idea of dynamically adjusting page sizes based on access patterns is not new. The most direct prior art is C-NUMA [28, 34], which dynamically promotes and demotes pages between base and huge sizes in response to traffic. While the authors correctly differentiate CLAP as proactive and migration-averse, the fundamental goal of matching page granularity to access locality is the same. The paper needs to more strongly argue why its proactive, profile-then-map approach is a significant conceptual leap forward rather than an implementation choice.

2. TLB Coalescing: The hardware support for merging TLB entries for contiguous memory regions is a well-established technique. The paper's mechanism, described in Section 4.6, is functionally very similar to prior work like CoLT [86]. The authors' contribution here seems to be the implementation and integration, but not the invention of the core concept.

3. Profiling for Placement: Using a sampling/profiling phase to guide data placement is a standard technique in systems research. For example, GRIT [104] (cited by the authors) also profiles page accesses to guide migration decisions in multi-GPU systems. The use of a small hardware tracker is a common way to reduce software profiling overhead.

The novelty, therefore, rests entirely on the synthesis. The paper would be stronger if it explicitly framed its contribution as such: a novel co-design that synergistically combines known techniques (profiling, dynamic sizing, TLB coalescing) in a way that is uniquely suited for the predictable, parallel access patterns of MCM-GPUs. As written, it sometimes reads as if these individual mechanisms are novel contributions in their own right.

Questions to Address In Rebuttal

1. The authors define "chiplet-locality" as the core property they leverage. How is this phenomenon fundamentally different from the well-understood concept of spatial locality exhibited by blocks of threads in a GPU programming model? Please justify why coining this new term and building a mechanism to measure it represents a novel insight, rather than an application of known locality principles to a new hardware topology.

2. The proactive "profile-then-map" strategy is positioned as superior to C-NUMA's reactive migration-based approach. However, this assumes that access patterns are static after the initial profiling phase. Could the authors comment on workloads where the access pattern evolves over time? In such cases, would CLAP's static decision (made after the 20% PMM phase) become suboptimal, and would a reactive approach like C-NUMA prove more robust?

3. The proposed solution constructs physically contiguous pages to enable TLB coalescing. This requirement for physical contiguity could increase memory fragmentation, especially for data structures with fine-grained or irregular chiplet-locality. How does CLAP compare to a system that uses a more flexible, scatter-gather style of TLB entry (e.g., using segment registers or block-based PTEs) that does not require physical contiguity? Is the added complexity of managing physical contiguity justified over alternative hardware designs that achieve similar TLB reach?

4. The complexity vs. benefit trade-off needs further justification. The performance benefits of CLAP are primarily realized by reducing TLB misses. If we consider an alternative path, such as significantly increasing the size and sophistication of the page walk caches or using speculative page walkers [85], could similar performance gains be achieved without the added complexity in the memory manager and the requirement for physical contiguity? Please defend the novelty of your approach in the context of these alternative solutions to the same root problem.