DECA: A Near-Core LLM Decompression Accelerator Grounded on a 3D Roofline Model

---

Review Form

Reviewer: The Guardian (Adversarial Skeptic)

Summary

The authors identify that LLM inference on modern CPUs is bottlenecked by the software-based decompression of quantized and sparsified weight matrices, which precedes their processing by hardware GeMM engines like Intel's TMUL. To address this, they first propose a 3D performance model, "Roof-Surface," to characterize the interaction between memory bandwidth, vector decompression units, and matrix execution units. Based on insights from this model, they propose DECA, a near-core hardware accelerator to offload decompression. Finally, they introduce a new ISA extension, TEPL, to enable efficient, low-latency, out-of-order invocation of DECA. Their simulation results claim up to a 4x speedup on compressed GeMM kernels and a 1.6x-1.9x reduction in next-token latency for large LLMs compared to an optimized software baseline.

Strengths

1. The paper addresses a relevant and timely problem: the performance bottleneck of weight decompression for LLM inference on CPUs.
2. The use of a state-of-the-art software baseline (Intel's libxsmm kernels) provides a strong point of comparison for the proposed hardware solution.
3. The fundamental idea of decomposing the performance problem into three interacting domains (memory, vector, matrix) is a logical approach to bottleneck analysis.

Weaknesses

My primary concerns with this submission revolve around the foundational model's validity, the justification for costly ISA extensions, and the fidelity of the evaluation methodology.

1. The Roof-Surface Model is an Oversimplification that Ignores First-Order Effects: The central thesis rests on the Roof-Surface model providing "clear insights." However, the model is an idealized representation of pipelined throughputs that systematically ignores crucial, real-world system effects. In Table 2 (page 5), the authors' own data reveals significant discrepancies between the model's predictions (R-S) and measured reality (Real). For instance, for BF16_50%, the model predicts 11.8 TFLOPS while the real performance is 9.2 TFLOPS—a 28% overestimation. The authors dismiss this as "non-plotted factors such as memory latency or cache latency." These are not minor factors; they are fundamental to system performance. A model that cannot account for over 25% of the performance limitation is not a sound foundation upon which to base microarchitectural design decisions. The claims of the model's explanatory power are therefore overstated.

2. Insufficient Justification for a New ISA Extension: The introduction of the TEPL instruction is a significant architectural modification that sets a very high bar for justification. The authors motivate TEPL by comparing it to a store-based invocation method that requires explicit memory fences (Figure 8, page 7), which is known to be inefficient for fine-grained core-accelerator communication. This comparison appears to be against a strawman baseline. The paper fails to explore or evaluate more sophisticated software-only or architectural mechanisms that could mitigate this latency without requiring bespoke ISA changes, such as optimized polling, doorbell registers with better scheduler integration, or advanced prefetching schemes. The conclusion that TEPL is necessary is therefore premature and insufficiently supported.

3. Ambiguous Simulation Fidelity: The evaluation is conducted using an "in-house simulator based on Sniper" (Section 7.1, page 9). Sniper is an interval-based simulator, which raises questions about its cycle-level accuracy for modeling the complex out-of-order execution and pipeline interactions that are central to the benefits claimed for TEPL. The paper provides no details on how the simulator models the TEPL Queue, speculative instruction issue, or the squash signaling between the core and DECA. Without rigorous validation or more detailed explanation of the simulation infrastructure, the results concerning TEPL's ability to "hide communication latency" remain unsubstantiated.

4. Comparison Against Unrealistic Alternatives: In Figure 14 (page 11), the authors compare DECA against "More AVX Units" and "Wider AVX Units." This comparison is not based on a realistic microarchitectural model, but on an abstract thought experiment of "removing the dynamic instructions from 3 out of 4 iterations." This completely sidesteps the immense microarchitectural challenges (e.g., L1 cache port scaling, register file bandwidth, instruction issue width) that make such a design infeasible, a point the authors themselves concede in Section 4. This comparison is therefore misleading, as it pits a detailed simulation of the proposed accelerator against a non-physical, idealized abstraction of an alternative.

Questions to Address In Rebuttal

1. Regarding the Roof-Surface model: How can the model be considered a reliable guide for microarchitecture design when it fails to account for system effects (latency, caching) that result in a ~20-30% performance deviation, as shown in your own results (Table 2)? Please justify why these first-order effects were excluded from your "3D performance model."

2. Regarding the TEPL ISA extension: Before proposing a new instruction, what alternative software-only or existing architectural mechanisms for low-latency core-accelerator communication were evaluated and why were they deemed insufficient? Please provide data to show that a fenced, store-based invocation is the strongest possible baseline short of new ISA.

3. Regarding simulation: Can you provide specific details on how your Sniper-based simulator models the out-of-order execution, speculative issue, and retirement of TEPL instructions? Specifically, how are structural hazards on the DECA loaders and dependencies with the core's ROB modeled to ensure the performance claims are not an artifact of an overly optimistic simulation environment?

4. Regarding the DSE in Section 8.2: The selection of the "best" {W,L} pair for DECA appears to be optimized to solve the problem as defined by your own idealized Roof-Surface model. Given the model's discussed inaccuracies, how can you be certain that this configuration is truly optimal for a real system where other factors (e.g., latency) are at play?

5. The abstract claims a next-token generation time reduction of "1.6x-1.9x." However, in Table 6 (page 12), the Llama2-70B result for N=16 and Q8_30% shows a speedup of 116.6 / 75.7 ≈ 1.54x, which is outside the claimed range. Please clarify the precise set of results used to establish this range.

---

Review Form

Reviewer: The Synthesizer (Contextual Analyst)

Summary

This paper presents a holistic, model-driven approach to solving the weight decompression bottleneck in Large Language Model (LLM) inference on CPUs. The authors identify that with high-bandwidth memory (HBM) and powerful in-core matrix engines (like Intel's AMX), the software-based decompression of compressed (quantized and sparsified) weights becomes the new performance limiter.

The core contribution is threefold:

1. A Novel Performance Model: The "Roof-Surface," a 3D extension of the classic Roofline model, which elegantly visualizes the performance bounds imposed by the three key interacting resources: memory bandwidth, vector unit throughput, and matrix unit throughput.
2. A Near-Core Accelerator: DECA, a specialized hardware unit designed to offload dequantization and de-sparsification from the CPU's vector units. The design of DECA is explicitly motivated and guided by insights from the Roof-Surface model.
3. An ISA Extension: The Tile External Preprocess & Load (TEPL) instruction, which enables efficient, out-of-order, low-latency communication between the CPU core and the DECA accelerator, effectively hiding communication overheads by overlapping them with computation.

The evaluation, conducted on a simulated 56-core server, demonstrates that this co-designed system can accelerate compressed matrix multiplication kernels by up to 4x over optimized software and reduce end-to-end next-token latency for large LLMs like Llama2-70B by 1.6-1.9x.

Strengths

The primary strength of this paper lies in its principled, top-to-bottom systems thinking. It is a superb example of how insightful performance modeling can directly inform and justify hardware and ISA design.

1. The Roof-Surface Model: This is the paper's most significant and enduring contribution. The classic 2D Roofline model is a cornerstone of performance analysis, but it falls short when a third resource becomes a first-order performance limiter. The authors' extension to a 3D "surface" (Section 4, page 5) is both intuitive and powerful. It provides a clear, quantitative language to explain why the software-only approach fails on HBM-equipped systems (it becomes "VEC-bound") and precisely what level of acceleration is needed to overcome this bottleneck without over-provisioning hardware. This model has the potential for broad applicability beyond this specific problem.

2. Holistic Co-Design: The paper does not simply propose an accelerator in a vacuum. Instead, it presents a complete solution. The Roof-Surface model identifies the problem, the DECA accelerator provides the hardware muscle, and the TEPL instruction provides the necessary low-level communication primitive to make the hardware effective. This tight integration of modeling, microarchitecture, and ISA is the hallmark of a high-quality computer architecture paper. The motivation for TEPL, contrasting the inefficient store-based invocation (Figure 8, page 7) with the streamlined out-of-order TEPL approach (Figure 9, page 7), is particularly compelling.

3. Timeliness and Practical Impact: The work addresses a critical, real-world bottleneck for a major workload. As CPUs continue to integrate HBM and more powerful matrix engines to stay competitive with GPUs for AI inference, the "software glue" problem will only become more acute. This paper provides a clear, well-reasoned blueprint for how CPU architects can solve this problem, potentially making high-performance, low-latency LLM inference more accessible and cost-effective on general-purpose servers. The comparison in Table 7 (page 12), which shows the proposed system closing a significant portion of the performance gap with a contemporary GPU, underscores this practical relevance.

4. Excellent Contextualization: The authors do a commendable job of situating their work within the broader landscape of in-core and near-core acceleration. The taxonomy presented in Table 8 (page 13) is particularly useful, clearly differentiating DECA from prior work by its unique combination of support for flexible compression, cooperation with matrix units (rather than replacing them), and fine-grained, low-overhead interleaving with the core.

Weaknesses

The weaknesses of the paper are minor and mostly relate to the boundaries of its exploration rather than fundamental flaws in the core idea.

1. Limited Scope of Evaluated Compression Schemes: The evaluation focuses on established but relatively simple schemes (unstructured sparsity, BF8, MXFP4). The field of model compression is evolving rapidly, with more complex methods like product quantization, non-uniform quantization, and structured sparsity patterns gaining traction. While the LUT-based design of DECA suggests flexibility, it is not immediately clear how it would handle schemes that require more complex arithmetic than a simple lookup and scaling operation.

2. Deeper Architectural Implications of TEPL: The TEPL instruction is well-motivated, but its introduction has ripple effects on the core's front-end, scheduler, and ROB. While the paper describes the high-level mechanism (TEPL Queue, speculative issue), a more in-depth discussion of the complexity and area/power cost of these changes to a modern out-of-order core would strengthen the proposal.

3. Unexplored Generality of the Roof-Surface Model: The authors correctly claim in Section 9.2 (page 13) that the Roof-Surface model can be generalized. This is one of the most exciting aspects of the work. However, this claim would be significantly bolstered by briefly applying the model to another, different problem domain to demonstrate its broader utility beyond LLM decompression (e.g., a bioinformatics pipeline, a graphics rendering stage, or a data analytics query).

Questions to Address In Rebuttal

1. The LUT-based dequantization in DECA is flexible for existing formats. How would the architecture adapt to future, more complex schemes that may not be easily captured by a lookup table, such as those requiring on-the-fly arithmetic to reconstruct weights? Does the DECA pipeline have extensibility points for such cases?

2. The paper's strongest contribution is arguably the Roof-Surface model. Can the authors provide another concrete example of a workload with a three-way (or more) dependency chain where their generalized modeling approach from Section 9.2 would yield non-obvious insights that a traditional Roofline analysis would miss?

3. Regarding the TEPL ISA extension: What alternative, less invasive core-accelerator communication mechanisms were considered? For instance, could a system of memory-mapped FIFOs combined with intelligent core-side prefetching and synchronization instructions achieve a similar level of latency hiding without the complexity of a new, blocking, out-of-order instruction? What was the deciding factor that made TEPL the superior choice?

Review Form

Reviewer: The Innovator (Novelty Specialist)

Summary

This paper presents three distinct but interconnected contributions aimed at accelerating compressed Large Language Model (LLM) inference on CPUs equipped with in-core matrix engines. The core problem—that software-based decompression of quantized/sparsified weights becomes a bottleneck—is well-established. The authors' proposed solution consists of:

1. A novel performance analysis framework called the Roof-Surface model, a 3D extension of the traditional 2D roofline model. This model aims to correctly identify bottlenecks in a pipelined system involving memory, vector units, and matrix units.
2. A near-core decompression accelerator named DECA, which offloads de-quantization and de-sparsification from the CPU's vector units to feed the in-core matrix engine.
3. A new ISA extension, Tile External Preprocess and Load (TEPL), designed for efficient, out-of-order invocation of the DECA accelerator, hiding communication latency.

My analysis concludes that while the constituent hardware components of the accelerator are based on known techniques, the specific combination of the performance model, the cooperative accelerator architecture, and the tight ISA integration represents a significant and novel system-level contribution. The novelty lies not in a single isolated idea, but in the synergistic design where each component is justified by and enhances the others.

Strengths

The primary strength of this paper is the introduction of multiple layers of novelty that build upon each other cohesively.

1. Novelty of the Performance Model: The Roof-Surface model (Section 4, page 5) is a genuinely novel extension of performance modeling for a specific, important class of problems. While multi-dimensional performance models are not unheard of, the specific formalization of a three-stage, pipelined dependency (Memory -> Vector -> Matrix) with corresponding arithmetic intensities (AIXM and AIXV) is new. It correctly identifies the vector-decompression bottleneck where the traditional 2D roofline model fails (as shown in Figure 3b, page 4). This model is not just a theoretical exercise; the authors effectively use it to motivate the need for an accelerator and to perform a quantitative design space exploration for it (Section 8.2, page 11).

2. Novelty in Architectural Division of Labor: The concept of a near-core accelerator is not new. However, DECA's novelty lies in its specific role as a cooperative pre-processing unit for an existing, powerful in-core engine (the TMUL). Much prior work on in/near-core accelerators for sparse workloads (e.g., VEGETA [39], RASA [40]) proposes replacing or heavily augmenting the core's compute units to handle compressed formats directly. In contrast, DECA decouples the decompression task from the GeMM task, offloading the former to a specialized unit while leaving the latter to the highly-optimized TMUL. This specific architectural partitioning is a novel approach to the problem.

3. Novelty of the ISA Integration: The TEPL instruction (Section 5.3, page 7) is a sophisticated and novel solution to the command-and-control problem for tightly-coupled accelerators. The naive approach using memory-mapped stores and loads is correctly identified as inefficient due to serialization and exposed latency. TEPL's design as a single instruction that both triggers the accelerator and receives the result, while being fully integrated into the core's out-of-order engine (using renaming and speculative execution), is a clean and powerful abstraction. This goes significantly beyond the prior art of loosely-coupled accelerators or those requiring no core modifications (e.g., SPADE [19], as noted in Table 8, page 13). The integration is the key innovation here.

Weaknesses

From a novelty perspective, the weaknesses are minor and relate to the granularity of the claims.

1. Constituent Components are Not Fundamentally New: The microarchitecture of the DECA pipeline itself (Section 6, page 8) is a synthesis of well-understood components. It uses Look-Up Tables (LUTs) for dequantization, POPCNT and Parallel Prefix Sum circuits for bitmask processing, and a crossbar for expansion. None of these blocks are, in isolation, novel inventions. The paper's novelty claim rests on their specific arrangement and dimensioning, which is guided by the Roof-Surface model. This is a system-level novelty, not a circuit-level one.

2. Limited Exploration of Non-ISA Alternatives: The paper makes a strong case for TEPL by demonstrating the flaws in a simple store-based invocation mechanism. However, the design space of command-and-control without new instructions is broader. For example, a system with dedicated command queues managed by the accelerator, which the core polls, could potentially offer better performance than the baseline store-with-fence approach. While I suspect the authors' TEPL solution is superior, a more thorough dismissal of advanced non-ISA-modifying alternatives would strengthen the novelty claim of requiring an ISA extension.

Questions to Address In Rebuttal

1. The paper's related work mentions NVIDIA's TMA [52] as a mechanism for supplying data to Tensor Cores and suggests augmenting it with DECA-like features could be an interesting direction (Section 9.1, page 13). Could the authors more precisely articulate the novel "delta" between the proposed TEPL instruction and the existing TMA mechanism? TMA is also, in essence, an accelerator for managing tile movement. How is TEPL's tight integration with the CPU's speculative, out-of-order core fundamentally different from the integration of TMA within an SM?

2. The generality of the Roof-Surface model is briefly discussed (Section 9.2, page 13) by proposing a generalized equation (3). Beyond this abstract formulation, could the authors provide one concrete example of another existing, real-world architecture or application (outside of CPU LLM decompression) where this 3-domain (or n-domain) pipelined model would provide insights that a traditional 2D roofline model would miss?

3. Regarding the TEPL design, was an alternative that relied on a dedicated, hardware-managed command FIFO between the core and DECA considered? Such a design might avoid the full complexity of a new, renamed instruction class while still allowing for out-of-order dispatch and avoiding memory fences. What would be the performance and complexity trade-offs of such a design compared to TEPL?