Stratum: System-Hardware Co-Design with Tiered Monolithic 3D-Stackable DRAM for Efficient MoE Serving

---

Review Form:

Reviewer: The Guardian (Adversarial Skeptic)

Summary

The authors propose Stratum, a system-hardware co-design for Mixture-of-Experts (MoE) model inference. The system is predicated on a future memory technology, Monolithic 3D-Stackable DRAM (Mono3D DRAM), integrated with a Near-Memory Processing (NMP) logic die via hybrid bonding. The core contribution is a set of co-optimizations across the stack. At the hardware level, the authors propose an "in-memory tiering" mechanism to exploit simulated latency variations across the vertical layers of the Mono3D DRAM. At the system level, they introduce a topic-aware scheduler that uses a lightweight classifier to predict query topics, mapping frequently used "hot" experts to the faster memory tiers. The authors claim significant improvements in throughput (up to 8.29×) and energy efficiency (up to 7.66×) over conventional GPU-HBM baselines.

Strengths

1. Comprehensive Scope: The paper attempts a full cross-stack analysis, from device-level simulation of Mono3D DRAM (Sec 6.1.1) to system-level serving policies (Sec 5). This holistic approach is commendable, as optimizations at one level often have unexamined consequences at others.
2. Clear Problem Formulation: The work correctly identifies that MoE models, despite their computational sparsity, are fundamentally bottlenecked by the memory capacity and bandwidth required to store and access massive expert parameters. The motivation for exploring beyond-HBM memory architectures is well-founded.
3. Detailed NMP Microarchitecture: The proposed NMP architecture in Figure 7 (page 6) is described with a reasonable level of detail, including PE, PU, and chip-level organization. This provides a concrete basis for the performance and area modeling, moving beyond high-level conceptual claims.

Weaknesses

The paper’s ambitious claims rest on a chain of optimistic assumptions and insufficiently validated components. The entire structure is fragile, and a weakness in any single link calls the overall conclusion into question.

1. Foundational Reliance on Simulated, Forward-Looking Technology: The entire premise of the paper hinges on the specific performance characteristics of 1024-layer Mono3D DRAM, which does not exist commercially. The crucial latency variation between tiers (a 1.6x difference from fastest to slowest, Sec 6.2.1, pg 11), which is the sole motivation for the tiering mechanism, is derived from Coventor and NeuroSim simulations (Table 1, Figure 14). This is not measured data. If the real-world manufacturing process yields a device with less latency variation, or if thermal crosstalk between layers negates these differences under load, the primary benefit of Stratum's data placement strategy is severely diminished or eliminated. The paper presents these simulated results as fact without a sensitivity analysis.

2. The Fragility of the Topic-Aware Scheduling: The system's performance is critically dependent on a "lightweight topic classifier" (Sec 5.1). The authors report 85.0% accuracy on the Chatbot Arena dataset (pg 9). This implies a 15% misclassification rate. A single misclassification would presumably place a "hot" expert on a slow tier, resulting in worst-case memory latency for that token's computation. The evaluation does not quantify the performance degradation under misclassification. This is a critical omission. A 15% chance of hitting a major performance penalty is unacceptable in a production serving system. The system's performance in the face of this realistic failure mode is not analyzed.

3. Unsubstantiated and Potentially Misleading Performance Claims: The headline claim of "up to 8.29×" improvement is a classic red flag. As seen in Figure 16 (pg 13), this peak number occurs for the smallest model (OLMOE) at a specific sequence length. The gains for the larger and more complex Llama-4-Scout model are a much more modest ~4.5×. More importantly, the fairness of the GPU baseline comparison is questionable. The paper states the baseline is vLLM on H100 GPUs, but provides no details on the configuration. Was the baseline fully optimized? Was it truly memory-bound, or was it bottlenecked elsewhere? Without a detailed roofline analysis or performance counter data from the GPU, it is impossible to verify that the baseline is not a strawman. The system is designed to excel at memory-bound tasks; the authors must first rigorously prove that the baseline workloads are indeed memory-bound.

4. Practical Constraints are Glossed Over:

   • Expert Swapping Cost: Table 4 (pg 12) claims a sub-1% time overhead for expert swapping. This analysis appears to assume ideal conditions. The cost of moving gigabytes of parameter data between DRAM tiers, even with the proposed row-swap buffer, is non-trivial. The paper does not analyze the impact of memory bank conflicts or contention on the ring network during these swaps, especially under heavy load.
   • Thermal and Power Assumptions: The thermal analysis (Sec 6.2.2, pg 11) concludes a 45W power budget for the logic die is feasible with "high-end liquid cooling solutions." This is a best-case scenario. The paper does not model the performance impact of thermal throttling if this ideal cooling is not achieved. The power breakdown in Figure 15 (pg 11) is based on synthesis and simulation, which can often underestimate real-world dynamic power consumption.
   • Generality: The entire optimization relies on requests having clear, classifiable topics with predictable expert affinities. The system's performance on workloads without this property (e.g., general conversation, creative writing, multi-topic queries) is unaddressed. This severely limits the claimed applicability of the approach.

Questions to Address In Rebuttal

1. Please provide a sensitivity analysis showing how Stratum's throughput advantage changes as the simulated latency variation across Mono3D DRAM tiers is reduced. For example, what is the performance gain if the fastest-to-slowest tier latency ratio is only 1.2x, rather than the assumed 1.6x?
2. The topic classifier has a non-zero error rate. Please provide an ablation study that quantifies the impact of topic misclassification (e.g., at 5%, 15%, and 25% error rates) on the overall system throughput and latency distribution.
3. Please provide evidence that the GPU baselines were not configured as a strawman. Specifically, provide profiler data (e.g., from NSight) for the H100 running vLLM to demonstrate that the workload is fundamentally memory-bandwidth-bound and that the GPU's compute resources are not being underutilized for other reasons.
4. The expert swapping cost analysis in Table 4 seems to assume an idle system. How does this overhead change when swapping occurs concurrently with active inference requests that are contending for the same memory banks and on-chip network resources?
5. How does the system perform on a mixed workload of queries where a significant fraction (e.g., 50%) has no strong topic affinity and thus activates experts in a pseudo-random or uniform pattern? This would test the system's performance when its primary optimization heuristic fails.

---

Review Form

Reviewer: The Synthesizer (Contextual Analyst)

Summary

This paper presents Stratum, a comprehensive system-hardware co-design for accelerating the serving of Mixture-of-Experts (MoE) Large Language Models. The work's central and most compelling idea is the synergistic mapping between an emerging hardware technology and an emergent property of large AI models. Specifically, it leverages Monolithic 3D-Stackable DRAM (Mono3D DRAM), a technology characterized by non-uniform access latencies across its vertically stacked layers. The authors astutely observe that this physical heterogeneity can be exploited by mapping frequently accessed "hot" experts—identified via a topic-based prediction system—to the faster, upper memory tiers, while relegating "cold" experts to the slower, deeper tiers.

The proposed system integrates this tiered Mono3D DRAM with a near-memory processor (NMP) on a logic die, connected via high-density hybrid bonding. The co-design extends across the stack, encompassing a hardware architecture for the NMP (Section 3.2, page 5), operator mapping strategies for MoE and attention computations (Section 4, page 6), and a system-level scheduler that classifies incoming queries by topic to inform expert placement (Section 5, page 8). The cross-layer evaluation demonstrates significant improvements in throughput and energy efficiency over conventional GPU-HBM baselines.

Strengths

1. Novel and Elegant Co-Design Synergy: The core contribution is not merely the application of a new memory technology, but the profound insight that connects the physical properties of that technology to the behavioral properties of the target application. The paper brilliantly turns a potential hardware drawback—the variable access latency of deep Mono3D DRAM stacks (visualized in Figure 2, page 3)—into a key architectural feature. This is elegantly matched with the observation of topic-specific expert affinity in MoE models (profiled in Figure 4, page 4), creating a powerful, cross-stack optimization principle. This represents a mature form of co-design that goes beyond simple acceleration.

2. Forward-Looking and Relevant Problem Domain: The paper tackles two critical, forward-looking problems simultaneously: (1) the memory wall in serving extremely large models like MoEs, and (2) the architectural implications of next-generation 3D memory integration. By moving beyond the well-trodden ground of HBM-based PIM/NMP, the authors provide a valuable architectural blueprint for a technology (Mono3D DRAM) that is a strong candidate for future high-performance systems. This positions the work not as an incremental improvement, but as a pioneering exploration of a future design space.

3. Comprehensive, Multi-Layered Approach: The strength of the work lies in its completeness. The authors have considered the problem from the device level (DRAM timing parameters in Table 1, page 10), through circuit and architecture (NMP design in Figure 7, page 6), and up to the system software level (topic-aware scheduling in Figure 6, page 5). This end-to-end perspective lends significant credibility to the claimed performance benefits, as it accounts for constraints and overheads at each layer of the system.

4. Contextualization within the Field: The paper does a good job of situating itself relative to prior work in PIM/NMP for transformers (e.g., AttAcc, Duplex) and highlighting its key differentiators, primarily the shift to Mono3D DRAM and the exploitation of its unique properties (Section 7, page 13). It builds upon the established trend of moving compute closer to memory while introducing a novel axis of optimization (latency tiering).

Weaknesses

1. Contingency on an Emerging Technology: The work's greatest strength is also its primary weakness. The entire premise and the impressive results are predicated on the maturation and adoption of Monolithic 3D DRAM as described. While this is a hallmark of forward-looking architectural research, the practical impact is contingent on manufacturing trends and the resolution of potential yield and thermal challenges associated with such dense 3D integration.

2. Sensitivity to Model Behavior: The co-design is exquisitely tuned to the phenomenon of topic-expert specialization. This raises questions about its robustness. If future MoE training methodologies were to change—for instance, by explicitly encouraging more uniform expert usage to improve generalization—the core premise of Stratum's data placement strategy would be undermined. The system's performance is tightly coupled to a specific, albeit currently observed, emergent behavior of MoE models.

3. Potential Overheads in Dynamic Scenarios: The paper demonstrates that the overhead of swapping experts between tiers is negligible for a given batch transition (Table 4, page 12). However, in a real-world serving scenario with a highly diverse and rapidly changing mix of query topics, the frequency of these swaps could increase. There is a potential risk of "memory thrashing" at the tier level if the topic distribution of incoming requests is chaotic, which could degrade performance in ways not fully captured by the current evaluation.

Questions to Address In Rebuttal

1. Robustness to Model Drift: The core optimization relies on strong topic-expert affinity. How does the performance advantage of Stratum degrade as this affinity weakens? For example, what happens if the hot/cold expert distinction becomes less pronounced, with usage probabilities being more evenly distributed?

2. Impact of Prediction Inaccuracy: The system's effectiveness is front-loaded by a lightweight topic classifier. The evaluation in Section 5.1 (page 9) shows high accuracy, but not 100%. What is the performance penalty of a misclassification? For instance, if a "math" query is misclassified as "legal," the system would presumably preload the wrong experts into the fast tiers, leading to slower execution. Can the authors quantify this impact?

3. Generalizability of the Architecture: The Stratum NMP and tiered memory system is highly optimized for the sparse, dynamic nature of MoE models. Does this specialized architecture offer significant benefits for other classes of models? For example, could the tiered memory system be repurposed to accelerate traditional dense transformers by placing attention KV caches in faster tiers, or is its utility fundamentally tied to the expert-based structure of MoEs?

4. Scaling and Physical Constraints: The paper assumes up to 1024 vertically stacked layers. As the stack depth increases, the latency disparity between the top and bottom tiers also grows (as shown in Figure 14, page 11). Is there a point of diminishing returns where the slowest tier becomes so latent that it's impractical, or where thermal density becomes an insurmountable challenge for the NMP logic die?

---

Review Form

Reviewer: The Innovator (Novelty Specialist)

Summary

The paper proposes Stratum, a system-hardware co-design for serving Mixture-of-Experts (MoE) Large Language Models. The system is built upon an emerging memory technology, Monolithic 3D-Stackable DRAM (Mono3D DRAM), integrated with a Near-Memory Processor (NMP) on a logic die via hybrid bonding.

The primary novel claim, as I deconstruct it, is twofold:

1. At the hardware level: The architectural exploitation of an inherent physical property of Mono3D DRAM—namely, the layer-dependent access latency variation (Figure 2, page 3)—to create fine-grained, physical, intra-chip memory tiers.
2. At the system level: A co-design that maps a known software behavior of MoE models—topic-based expert affinity ("hot" vs. "cold" experts)—directly onto these novel physical memory tiers to optimize data access.

The work claims to be the first to propose such a co-design leveraging this specific memory technology for MoE serving.

Strengths

The core strength of this paper lies in its identification and architectural exploitation of a device-level characteristic.

1. Novel Architectural Insight: The central idea of turning a physical-layer non-ideality (latency variation across vertically stacked wordlines) into a system-level feature (memory tiering) is genuinely novel and insightful. Instead of designing for the worst-case latency as is conventional, the authors embrace the heterogeneity. This demonstrates a deep, cross-layer thinking that is rare and commendable. This is detailed in Section 2.1 (page 3) and visualized in Figure 14 (page 11).

2. Synergistic Co-Design: The novelty is further strengthened by the tight coupling between the hardware insight and the application domain. The concept of expert affinity in MoE models is known (e.g., [87], [33]), but prior work has not had a hardware substrate that so elegantly maps to this logical concept. The mapping of hot/cold experts to fast/slow physical tiers (Section 5.2, page 9) is a powerful and novel synthesis of existing ideas from different domains.

3. Significant Delta from HBM-based PIM/NMP: The paper correctly differentiates itself from prior art in NMP for LLMs (e.g., Duplex [89], AttAcc [67]), which are based on HBM. The architectural shift to Mono3D DRAM with its dense hybrid bonding fundamentally changes the design constraints (higher internal bandwidth, no TSV bottleneck), justifying a new NMP design. The novelty here is in the adaptation to and exploitation of this new memory paradigm.

Weaknesses

My critique is centered on the precise boundaries of the novelty and the potential transience of the underlying physical premise.

1. Constituent Ideas Are Not Novel: While the synthesis is novel, the paper could be more precise about the prior art for its constituent components. The observation of expert affinity is not new ([87]), memory tiering as a concept is decades old, and near-memory processing for transformers is an established research direction. The paper's novelty rests entirely on the combination and the physical mapping. The authors should frame their contribution more sharply as a novel architectural mapping rather than implying the invention of these base concepts.

2. NMP Architecture is Evolutionary, Not Revolutionary: The proposed Stratum NMP architecture (Figure 7, page 6) is a well-engineered solution but does not appear to introduce fundamentally new processing concepts. It combines tensor cores, a ring interconnect, and SIMD special function units, which are all well-understood building blocks in accelerator design. The "delta" compared to prior NMP designs like Duplex [89] seems to be primarily in the interconnect topology (ring vs. global buffer/crossbar) and the direct integration with Mono3D banks. This is a significant engineering adaptation, but its conceptual novelty as a processor architecture is limited.

3. Contingency on a Technological "Flaw": The entire premise of in-memory tiering hinges on the significant latency variation across Mono3D DRAM layers. This variation stems from the staircase structure for wordline contacts (Figure 2, page 3). It is conceivable that device and circuit designers will view this as a defect to be engineered away in future generations of the technology, striving for uniform access times. If they succeed, the core hardware motivation for this work vanishes. The novelty is thus tied to a potentially transient property of an emerging technology.

Questions to Address In Rebuttal

The authors should use the rebuttal to clarify the following points regarding the novelty and significance of their contribution.

1. The concept of topic-based expert affinity and classifying experts as "hot" or "cold" has been explored for optimizing MoE serving on existing hardware (e.g., [87]). Please confirm that your primary novel contribution is not the identification of this affinity, but rather the creation of a new hardware architecture (tiered Mono3D) that provides a physical substrate for this logical classification, and the co-design that maps between them.

2. The NMP architecture in Figure 7 integrates tensor cores, a ring network, and SIMD function units. Beyond the adaptation to leverage the high internal bandwidth of Mono3D DRAM, what are the fundamentally new architectural concepts in the processing unit or interconnect design itself when compared to the principles used in prior NMP systems like Duplex [89]?

3. The proposed tiering mechanism is predicated on the access latency heterogeneity in Mono3D DRAM. How fundamental is this property? Is it not a target for elimination by device-level engineers in future iterations of the technology? How would the value proposition of Stratum change if next-generation Mono3D DRAM achieved, for instance, less than a 20% latency variation between the fastest and slowest layers?

4. The system introduces significant complexity, including a topic classifier, a dynamic SLO-aware scheduler, and a memory mapper for expert swapping. Could a simpler baseline, such as caching the weights of the most globally popular experts (independent of topic) in a large SRAM on the logic die, achieve a substantial fraction of the performance benefit without the overhead of dynamic topic classification and physical data migration between tiers? A comparison against such a baseline would help quantify the benefit of the novel tiering mechanism itself.