Drishti: Do Not Forget Slicing While Designing Last-Level Cache Replacement Policies for Many-Core Systems

---

Review Form

Reviewer: The Guardian (Adversarial Skeptic)

Summary

This paper posits that state-of-the-art LLC replacement policies, such as Hawkeye and Mockingjay, are sub-optimal in many-core systems with sliced LLCs. The authors identify two primary deficiencies: 1) "myopic" reuse predictions resulting from access streams being scattered across slices, and 2) "underutilized" sampled sets where randomly selected sets for monitoring receive too few misses to provide useful training data. To address this, the paper proposes "Drishti," a set of two enhancements: a per-core, globally-accessible reuse predictor to create a non-myopic view, and a dynamic sampled cache (DSC) that intelligently selects high-miss-rate sets for monitoring. The authors claim that these enhancements significantly improve the performance of existing policies, with Mockingjay's geomean speedup over LRU on a 32-core system increasing from 6.7% to 13.2%.

Strengths

1. The paper correctly identifies the "myopic" prediction problem in sliced LLC architectures as a potential performance limiter. The scattering of a single PC's access stream across multiple per-slice predictors is a valid architectural concern.
2. The motivation for the proposed enhancements is supported by some initial data analysis (e.g., Figures 2, 3, and 5), which illustrates the access scattering and skewed miss distribution that form the basis of the authors' arguments.
3. The authors provide an ablation study (Figure 17) that attempts to isolate the performance contributions of each proposed mechanism (the global predictor and the dynamic sampled cache).

Weaknesses

1. Critical Dependency on Unrealistic Hardware: The central and most significant weakness is the proposal's complete reliance on a dedicated, low-latency, three-cycle interconnect (NOCSTAR). This is not a simple policy tweak but a fundamental alteration of the on-chip network fabric. The authors' own data in Figure 11a demonstrates that without this idealized network, their proposal results in a significant performance slowdown (up to 9% on average for 32 cores). This makes NOCSTAR a hard requirement, not an optimization. The practicality, area, power, and complexity costs of adding a second, dedicated network are non-trivial and are not sufficiently justified against the performance gains. The proposal's viability hinges entirely on this assumption.

2. Selective Scope and Inconsistent Baseline Comparison: The authors explicitly state in the introduction (Section 1, Page 1) that they "do not consider machine learning (ML) [55] and reinforcement learning (RL) [38]-based LLC replacement policies." This is a critical omission, as these policies represent the current state-of-the-art frontier. This selective exclusion creates a simplified problem space where Drishti's enhancements may appear more effective than they would against stronger, more adaptive baselines. This exclusion is then directly contradicted in Section 6 (Page 12), where the authors claim applicability to and provide results for CHROME [38] (RL-based) and Glider [55] (deep learning-based). This inconsistency suggests either a flawed initial premise or a post-hoc attempt to broaden the paper's applicability without a rigorous, head-to-head comparison in the main evaluation.

3. Fragile Justification for Dynamic Sampled Cache (DSC): The justification for the DSC relies heavily on workloads with highly skewed miss distributions (e.g., mcf in Figure 5a). The authors concede that for workloads with uniform distributions (lbm in Figure 5c), this mechanism is ineffective and must be disabled via a detection heuristic (Section 4.2, Page 7). This admission reveals the DSC is not a universally applicable improvement but a mode-based optimization that adds the complexity of phase/behavior detection logic. It is unclear how this mechanism performs on the wide spectrum of workloads between these two extremes. The claim of a net hardware saving (Table 3) is also tenuous; while the sampled cache structure is smaller, the proposal adds k-bit saturating counters for every LLC set, selection logic, and the aforementioned NOCSTAR interconnect. The overall system complexity is demonstrably higher.

4. Insufficient Detail in Motivational Analysis: In the motivational analysis (Section 3.1, Figure 3), the methodology for simulating the "global view" predictor is not sufficiently detailed. It appears to be an idealized oracle with perfect and instantaneous access to all sampled set information across all slices. This sets an unachievably high bar and may inflate the perceived potential of a global predictor, making the authors' practical implementation seem closer to the ideal than it actually is. The performance gap between this oracle and the authors' NOCSTAR-based implementation is not quantified.

Questions to Address In Rebuttal

1. Please provide a detailed analysis of the performance of Drishti using the existing on-chip mesh interconnect, assuming realistic contention and latency derived from the increased predictor traffic. Given that your own data (Figure 11) shows a performance loss without NOCSTAR, how can this proposal be considered a practical enhancement to replacement policies rather than a coupled co-design of a policy and a new interconnect?

2. Please justify the exclusion of ML/RL-based policies like CHROME or Glider from the main evaluation (Section 5), especially given that you test against them in Section 6. To properly situate Drishti's contribution, a full comparison against these true state-of-the-art policies is necessary in the main results tables and figures.

3. How does the Dynamic Sampled Cache (DSC) perform on workloads where the miss distribution across sets is relatively flat but not perfectly uniform (i.e., not as skewed as mcf, but not as flat as lbm)? What is the performance and hardware overhead of the workload detection logic that is required to enable/disable the DSC?

4. Clarify the precise simulation setup for the idealized "global view" predictor used in Figure 3. Is this a contention-free model with zero-cycle access latency? Please provide data comparing this ideal oracle to your implemented per-core global predictor with the three-cycle NOCSTAR latency to properly frame the implementation's efficiency.

---

Review Form:

Reviewer: The Synthesizer (Contextual Analyst)

Summary

This paper, "Drishti," addresses a timely and important disconnect between the design of state-of-the-art Last-Level Cache (LLC) replacement policies and their deployment in modern many-core systems. The authors correctly observe that while advanced policies like Hawkeye and Mockingjay show significant promise, their evaluation has largely been confined to monolithic LLC models. In contrast, commercial processors utilize sliced, non-uniform cache architectures (NUCA).

The core contribution is not a new replacement policy, but rather a set of two well-motivated architectural enhancements that make existing policies "slice-aware." The authors first identify the problem of "myopic predictions," where per-slice predictors have an incomplete view of a program's global reuse behavior. Their solution is a clever compromise: a local (per-slice) sampled cache that feeds into a per-core, yet globally-aware, reuse predictor. Second, they identify that randomly selected sampled sets are often "underutilized," receiving too few misses to effectively train the predictor. They propose a dynamic sampled cache that intelligently selects sets with high miss rates (MPKA) to maximize learning efficiency. The paper demonstrates that these enhancements, when applied to Hawkeye and Mockingjay, can substantially boost their effectiveness, for instance, nearly doubling the performance gains of Mockingjay on a 32-core system.

Strengths

1. Excellent Problem Formulation: The paper's greatest strength is its premise. It tackles a practical and increasingly relevant issue that is often overlooked in academic studies. By grounding their work in the reality of commercial sliced LLCs (as mentioned in Section 1, page 1), the authors immediately establish the significance of their investigation. This is a crucial step in bridging the gap between theoretical microarchitectural improvements and real-world processor design.

2. Clear, Data-Driven Motivation: The motivation presented in Section 3 (page 3) is compelling. Figure 2, which quantifies how many Program Counters (PCs) are confined to a single slice, provides a powerful and intuitive argument for why per-slice predictors are inherently myopic. Furthermore, the analysis of underutilized sampled sets (Section 3.2, page 4), culminating in the simple yet powerful experiment in Table 1, perfectly justifies the need for a more intelligent set selection mechanism.

3. Elegant and Generalizable Solutions: The proposed enhancements are not overly complex and demonstrate a deep understanding of the design trade-offs. The "per-core yet global" predictor is a pragmatic solution that balances the need for a global view against the communication overhead of a fully centralized structure. The dynamic sampled cache uses a well-understood technique (saturating counters) to solve the problem in a simple and effective manner. Most importantly, as shown in Section 6 (page 12), these ideas are not limited to Hawkeye and Mockingjay. The authors demonstrate their applicability to a wider class of prediction-based policies (including SHiP++, CHROME, and Glider), elevating the work from a specific optimization to a more general architectural principle.

4. Contextualization within the Field: This work fits beautifully into the ongoing evolution of cache management. It follows the trajectory from simple heuristics (LRU), to predictive policies (RRIP, SHiP), to emulating optimality with large-scale tracking (Hawkeye, Mockingjay). Drishti represents the next logical step: adapting these sophisticated predictors to the physical and distributed reality of modern hardware. It addresses the "systems" aspect of a microarchitectural problem.

Weaknesses

1. Framing as an "Enhancement": While technically accurate, framing the work solely as an enhancement to existing policies may undersell its conceptual contribution. The core ideas—decoupling the scope of sampling from the scope of prediction and making the sampling process adaptive—are fundamental principles that could inform the design of future replacement policies from the ground up.

2. Reliance on a Dedicated Interconnect: The proposal for a dedicated interconnect (NOCSTAR, Section 4.1.4, page 6) to handle predictor traffic is a practical solution but also introduces non-trivial design complexity and area/power overhead, however small. While the authors justify its low cost, it represents an additional system component that must be validated. An exploration of alternative approaches, such as leveraging quality-of-service (QoS) mechanisms on the main network-on-chip (NoC), would have strengthened this aspect of the proposal.

3. Trace-Driven Simulation Limitations: The use of a trace-driven simulator (ChampSim) is standard practice and perfectly acceptable for this type of study. However, it inherently cannot capture complex feedback loops where changes in cache behavior might alter the application's execution path, memory access timing, or prefetching behavior. This is a minor limitation but worth noting, as the true system-level impact could differ slightly in an execution-driven environment.

Questions to Address In Rebuttal

1. The utility of each enhancement is analyzed in Figure 17 (page 10). It appears the move to a global predictor provides the largest performance jump, with the dynamic sampled cache (DSC) providing a further, significant boost. Could the authors confirm this interpretation and comment on the synergy between the two proposals? Are the gains largely additive, or is there a super-linear effect where the DSC is particularly effective because it is training a more powerful global predictor?

2. Regarding the NOCSTAR interconnect, could the authors elaborate on why a dedicated network is preferable to using a high-priority virtual channel on the existing NoC? While the paper argues for low latency, could a QoS approach provide "good enough" latency without the cost of dedicated wiring and arbiters?

3. The paper effectively shows scalability up to 128 cores. As we look toward future systems with hundreds of cores, does the "per-core yet global" predictor model begin to face new scalability challenges? For example, does the storage for the per-core predictors become prohibitive, or does the aggregate traffic to these distributed predictors start to congest the dedicated interconnect?

---

Review Form

Reviewer: The Innovator (Novelty Specialist)

Summary

The paper identifies a valid and often overlooked problem: state-of-the-art Last-Level Cache (LLC) replacement policies, such as Hawkeye and Mockingjay, which were designed for monolithic caches, suffer from degraded performance on the sliced LLC architectures common in modern many-core processors. The authors attribute this degradation to two primary factors: "myopic predictions" from per-slice predictors that lack a global view of an application's reuse behavior, and "underutilized sampled sets" where randomly chosen sets for monitoring see too few misses to provide a useful training signal.

To address this, the paper proposes "Drishti," a set of two hardware enhancements. The first is a "per-core yet global" reuse predictor architecture, which aims to provide a global view of a core's access patterns without the bottleneck of a single centralized predictor. This is paired with a local, per-slice sampled cache. The second enhancement is a "dynamic sampled cache," which eschews random set selection in favor of a mechanism that periodically identifies and samples LLC sets with the highest miss rates (MPKA), thereby focusing the monitoring effort where it is most impactful.

Strengths

1. Problem Identification: The paper correctly identifies a significant gap between academic research on LLC replacement policies and the reality of commercial hardware. The analysis in Section 3.1 (page 3), particularly Figures 2 and 3, provides a clear and compelling demonstration of the "myopic" prediction problem in sliced LLCs. This is a valuable insight.

2. System Integration: The authors propose a complete system solution. The two enhancements are designed to work in concert, addressing two distinct but related weaknesses of existing policies in a sliced environment. The consideration of interconnect traffic and the proposal to use a dedicated interconnect (NOCSTAR) shows a thoroughness in engineering the solution.

Weaknesses

My evaluation is centered on the fundamental novelty of the proposed ideas. While the engineering and integration are competent, the core concepts appear to be reformulations or applications of pre-existing principles.

1. "Per-Core yet Global" Predictor Lacks Conceptual Novelty: The core idea here is to overcome the limitations of distributed, uncoordinated decision-making by introducing a shared, global perspective. This is a foundational concept in distributed systems and parallel computing, not a new one. The paper itself acknowledges that a fully centralized predictor is a "trivial solution" (Abstract, page 1). The proposed "per-core yet global" architecture is an engineering trade-off to manage the scalability of this known solution. It is essentially a form of state replication (one predictor instance per core) where each replica is updated by a single logical stream (the core's accesses). While this specific arrangement might be new in the context of LLC predictors, the architectural pattern itself is not novel. The problem of balancing global state with distributed access is well-trodden ground. The novelty lies in the application, not the invention.

2. "Dynamic Sampled Cache" is an Application of a Known Principle: The insight that random sampling can be inefficient and that monitoring should focus on "hotspots" or high-activity regions is not new. The principle of dynamically identifying high-pressure resources to guide policy is seen in other areas of cache management. For instance:

   • Utility-Based Cache Partitioning (UCP): This class of techniques monitors the marginal utility (e.g., miss rate reduction) of allocating cache resources to different applications. This inherently involves identifying which applications/cores are creating the most memory pressure.
   • Adaptive Set Management: Prior work like The V-Way Cache [47] and The Set Balancing Cache [50] dynamically adjust cache associativity or line placement based on monitoring pressure within individual sets. The underlying principle is the same: identify high-contention sets and act upon them.

   Drishti's contribution is to apply this principle of "focus on the hotspots" to the specific problem of selecting which sets to sample for training a reuse predictor. The mechanism proposed—using per-set saturating counters to track MPKA—is a straightforward heuristic implementation of this principle. The idea is an incremental refinement of sampling strategy, not a fundamentally new concept.

3. Overall Contribution is Systematization, Not Invention: The primary contribution of this paper is the careful identification of a real-world system issue and the competent engineering of a solution by combining and adapting existing architectural principles. It is a work of system integration. While valuable, it does not present a novel algorithmic or architectural paradigm for cache replacement. The "delta" over prior art is in the specific application and combination of ideas, which, while effective, is not large from a conceptual standpoint.

Questions to Address In Rebuttal

1. The concept of creating a global view to overcome myopic local decisions is a classic problem. Please articulate the fundamental architectural novelty of the "per-core yet global" predictor beyond it being a point in the design space between fully distributed and fully centralized state. What makes this specific arrangement conceptually distinct from other forms of replicated, coherent state in parallel architectures?

2. Please contrast the core principle of your dynamic sampled cache with prior work in adaptive cache management (e.g., adaptive associativity, cache partitioning) that also dynamically identify high-pressure sets/regions to guide policy decisions. Is the novelty in the principle itself, or solely in its application to selecting predictor training sets?

3. The proposed hardware changes (a dedicated NOCSTAR interconnect, per-set MPKA counters) are significant. Given that the underlying concepts are adaptations of existing principles, could a substantial fraction of the performance gain be achieved through a less complex approach, perhaps by leveraging existing coherence traffic or performance counters to approximate a global view and identify high-miss sets? Please justify why this level of hardware complexity is necessary for the claimed advance.