Kelle: Co-design KV Caching and eDRAM for Efficient LLM Serving in Edge Computing

---

Review Form

Reviewer: The Guardian (Adversarial Skeptic)

Summary

The authors propose "Kelle," a hardware-software co-design for LLM inference on edge devices that uses eDRAM as the primary storage for the KV cache. The core contributions are twofold: 1) an algorithm named AERP that combines attention-based token eviction with a recomputation policy to manage the KV cache size, and 2) a two-dimensional adaptive refresh policy (2DRP) for the eDRAM that modulates refresh rates based on token importance and bit significance, thereby intentionally introducing errors to save power. These are implemented in a custom accelerator. The authors claim significant speedup (3.9×) and energy savings (4.5×) over an SRAM-based baseline. While the problem is relevant, the work rests on a foundation of questionable experimental design and insufficiently substantiated claims, particularly concerning the main performance comparison.

Strengths

1. Problem Formulation: The paper correctly identifies the KV cache as a primary memory bottleneck for LLM inference on resource-constrained edge devices, and the exploration of eDRAM as an alternative to SRAM is a logical direction.
2. Ablation Study Structure: The authors have made an effort to isolate the performance contributions of their various techniques. The comparisons between Original+SRAM, Original+eDRAM, AEP+SRAM, and AERP+SRAM (Section 8.1.2, Figure 13) provide some insight into the relative benefits of each component of their design, assuming the baselines are fair.
3. Comprehensive Algorithm Design: The proposed solution is multifaceted, addressing the eDRAM challenge from multiple angles: cache content management (AERP), refresh power (2DRP), and data lifetime (Kelle Scheduler).

Weaknesses

My primary role is to ensure the rigor of published work. This paper, in its current form, exhibits several critical weaknesses that undermine the validity of its conclusions.

1. Fundamentally Confounded Main Comparison: The central claim of 3.9× speedup and 4.5× energy savings is derived from a comparison between Kelle+eDRAM and an Original+SRAM baseline. As stated in Section 8.1.1, the SRAM baseline is area-matched, resulting in a smaller 24×24 systolic array compared to Kelle's 32×32 array. This is an unacceptable confounder. The reported gains are not solely from the memory subsystem innovation but are significantly influenced by the fact that the Kelle system has nearly 80% more compute resources (32²/24² ≈ 1.78). This renders the headline performance numbers highly misleading. A valid comparison would require matching the compute capabilities.

2. Insufficient Justification for Heuristics: The proposed AERP algorithm relies on a seemingly arbitrary threshold. In Section 4.1.2, the decision to recompute KV vectors for a token if it is retained in "at least 50% of the heads" lacks any theoretical or empirical justification. The paper presents no sensitivity analysis for this crucial parameter. How does performance change at 40% or 60%? Without this, the chosen value appears to be a "magic number" that may have been tuned for the specific experiments shown.

3. Superficial Evaluation of Error Injection: The 2DRP policy is predicated on the idea that LLMs are tolerant to a certain level of data corruption in the KV cache. The authors' evaluation of this tolerance is dangerously shallow. They rely primarily on perplexity (PPL) (Figure 8), which is a coarse, statistical measure of fluency. The claim in Section 7.1 that an average retention failure rate of 2e-3 has a negligible impact is not sufficiently proven. While Table 5 shows "comparable" scores on TruthfulQA, a minor drop in accuracy on a multiple-choice benchmark does not adequately characterize the risk of catastrophic failures, such as factual hallucination or safety violations, which could be triggered by specific bit-flips in critical tokens. A system that knowingly corrupts data requires a far more rigorous and targeted evaluation of its failure modes.

4. "Straw Man" Baseline Scheduler: The baseline computation pattern shown in Figure 12a appears intentionally inefficient, with long, serialized data dependencies that inflate the data lifetime in eDRAM. Any reasonably optimized system would attempt to co-schedule dependent operations to improve data locality and reduce lifetime. The gains attributed to the "Kelle Scheduler" may be significantly overstated by comparing against this naive baseline.

5. Understated System Complexity: The paper proposes a complex memory subsystem (Section 5.1, Figure 10) with data split into four groups (MSB/LSB for HST/LST tokens) across 32 banks, managed by custom eviction and refresh controllers. While the area of the systolic evictor is mentioned (Section 8.1.4), the overhead of the intricate control logic, potential timing challenges, and addressing complexity for this fine-grained management is not adequately discussed or quantified.

Questions to Address In Rebuttal

The authors must address the following points directly to salvage the credibility of this work.

1. Provide a new end-to-end performance comparison (speedup and energy) against an SRAM-based baseline that is compute-matched (i.e., also uses a 32×32 systolic array). This is the only way to isolate the true contribution of the Kelle memory system. The area and power of this new, larger SRAM baseline must be reported.

2. Present a sensitivity analysis for the 50% popularity threshold in the recomputation policy (Section 4.1.2). How do system performance, accuracy, and storage savings vary as this threshold is changed? Justify your final choice of 50%.

3. The evaluation of the 2DRP's error injection is insufficient. Please provide a more rigorous analysis of its impact on model factuality and safety. This should go beyond PPL and standard zero-shot tasks and utilize benchmarks specifically designed to detect factual inconsistency (e.g., FactScore) or adversarial safety risks.

4. Justify the selection of the "Baseline" scheduler in Figure 12. Is this representative of typical, optimized LLM inference schedulers, or is it a worst-case scenario designed to amplify the benefits of the Kelle scheduler? Please compare against a more aggressive, locality-aware baseline scheduler.

5. In Section 8.4.1, you claim Kelle can support long contexts (up to 60K tokens) by offloading to DRAM, stating that the paging process is simplified. However, prefetching this volume of KV data from DRAM for every token generation step would incur substantial latency and energy overhead. Please provide a quantitative analysis of this "paging" overhead and demonstrate that it does not negate the on-chip benefits for such long sequences.

---

Review Form

Reviewer: The Synthesizer (Contextual Analyst)

Summary

This paper, "Kelle," presents a hardware-software co-design for efficient Large Language Model (LLM) inference on edge devices, targeting the significant bottleneck of the Key-Value (KV) cache. The core contribution is the principled replacement of traditional on-chip SRAM with embedded DRAM (eDRAM) as the primary storage for the KV cache. Recognizing that eDRAM's high density comes at the cost of power-intensive periodic refreshes, the authors propose a suite of tightly-coupled optimizations to mitigate this overhead. These include: 1) an Attention-based Eviction and Recomputation Policy (AERP) to manage the limited cache size and reduce the lifetime of stored data, and 2) a Two-Dimensional Adaptive Refresh Policy (2DRP) that cleverly exploits the inherent error tolerance of LLMs by reducing refresh rates for less significant bits and less important tokens. These policies are instantiated in a complete accelerator design featuring a custom memory subsystem and a novel "systolic evictor" to implement the AERP with minimal latency. The work demonstrates significant improvements in speed and energy efficiency, positioning eDRAM as a viable and powerful component for future edge AI systems.

Strengths

1. Excellent Problem-Solution Fit and Timeliness: The paper identifies one of the most pressing problems in deploying modern AI: the memory capacity wall for LLM inference. The proposal to use eDRAM is a fantastic synthesis of an existing, mature technology with a new and critical problem domain. While eDRAM has been explored for CPU caches and CNN accelerators, its application to the LLM KV cache is both novel and exceptionally well-motivated. The authors correctly identify that the KV cache is fundamentally a capacity and bandwidth problem, which aligns perfectly with eDRAM's primary strengths over SRAM.

2. Insightful and Elegant Co-design Principle: The true strength of this work lies not just in using eDRAM, but in the deep, synergistic co-design. The central insight, empirically validated in Figure 8 (page 5), is that LLMs exhibit graceful degradation to certain types of data corruption. The 2DRP policy (Section 4.2, page 5) is a direct and clever exploitation of this property, mapping a characteristic of the software model (variable importance of data) directly onto a physical knob in the hardware (refresh rate). This moves beyond a simple component swap into a truly holistic system design, which is commendable.

3. Comprehensive System-Level Approach: The authors present a complete system, from algorithm to architecture. They do not stop at the conceptual level but detail the hardware necessary to make their vision a reality, including the Kelle accelerator architecture (Figure 9, page 6), the memory subsystem layout (Figure 10, page 7), and the novel systolic evictor (Section 5.3, page 7). This end-to-end thinking significantly increases the credibility and impact of the work, demonstrating a clear path from idea to implementation.

4. Thorough and Illuminating Evaluation: The experimental evaluation is extensive, covering multiple models, datasets, and a strong set of ablation studies (Section 8.3, page 11). The comparison against four distinct baselines in Section 8.1.1 (page 10) is particularly effective, as it allows the reader to disentangle the benefits derived from simply using eDRAM versus those from the AERP and 2DRP algorithms. This methodical breakdown provides clear evidence for the efficacy of each proposed contribution.

Weaknesses

1. Limited Contextualization Against Other Memory Technologies: The paper does an excellent job of framing the SRAM vs. eDRAM trade-off. However, the broader field of computer architecture is actively exploring a rich landscape of emerging memory technologies (e.g., MRAM, RRAM, FeFETs) for on-chip memory. These technologies could potentially offer the density benefits of eDRAM without the refresh overhead, albeit with their own challenges (e.g., write latency, endurance). A discussion on where Kelle's principles fit within this wider context would strengthen the paper's long-term relevance. For instance, could the AERP policy be beneficial for NVMs to manage write endurance? This is a missed opportunity to position the work more broadly.

2. Understated Implementation Complexity: The proposed co-design, particularly the 2DRP policy, introduces non-trivial control logic. The memory controller must now track importance scores and manage multiple, fine-grained refresh domains dynamically. While the paper quantifies the overhead of the systolic evictor (Section 8.1.4, page 11), a more detailed analysis of the control overhead for the memory system itself would be beneficial. The elegance of the idea may hide a significant complexity cost that should be acknowledged and justified more explicitly.

3. The Fundamental Assumption of Error Tolerance: The 2DRP policy hinges on the assumption that LLMs are resilient to bit-flips in certain data. While the authors show this holds for their 16-bit evaluation, this property may become less robust as the field pushes towards aggressive, low-bit quantization (e.g., 4-bit, 3-bit, or even binary formats). In such schemes, every single bit carries substantially more information, and a single bit-flip could be catastrophic. The paper would be strengthened by a discussion of how its core principles might adapt or break under these more aggressive quantization scenarios.

Questions to Address In Rebuttal

1. Could the authors elaborate on how their co-design principles, particularly AERP and the concept of mapping data importance to hardware properties, might compare to or be adapted for other emerging on-chip memory technologies like MRAM or RRAM, which offer density without refresh but introduce challenges like write endurance and latency?

2. The 2DRP policy requires a sophisticated eDRAM controller capable of managing dynamic, fine-grained refresh domains. Could the authors provide a more detailed estimate of the area and power overhead of this control logic relative to a standard eDRAM controller? Is there a risk that this control complexity negates some of the energy savings from reduced refresh operations?

3. The viability of 2DRP rests on the error tolerance of the LLM. How do the authors expect this approach to perform with models that are aggressively quantized to 4-bit or lower representations? Does the reduced information redundancy in low-bit formats present a fundamental challenge to a strategy that relies on tolerating data corruption?

---

Review Form

Reviewer: The Innovator (Novelty Specialist)

Summary

The paper proposes "Kelle," a hardware-software co-design for efficient LLM inference on edge devices. The central idea is to use embedded DRAM (eDRAM) as the primary on-chip storage for the KV cache to leverage its high density and low leakage power compared to SRAM. To mitigate the primary drawback of eDRAM—the high cost of periodic refresh operations—the authors introduce a suite of techniques: (1) an Attention-based Eviction and Recomputation Policy (AERP) to manage the cache size and data lifetime, (2) a Two-Dimensional Adaptive Refresh Policy (2DRP) to reduce refresh frequency by exploiting data criticality, and (3) a custom hardware accelerator featuring a novel "Systolic Evictor" to implement these policies efficiently. The authors claim this co-design results in significant speedup and energy savings.

My review will assess the novelty of these constituent ideas by situating them within the landscape of prior art.

Strengths

The primary strength of this work lies in the synthesis of multiple techniques into a cohesive system targeted at a very specific and timely problem. While many of the individual concepts have precedent in other domains, their application and integration for managing an LLM KV cache in eDRAM is a novel endeavor.

The most genuinely novel contribution at the microarchitectural level appears to be the Systolic Evictor (Section 5.3, page 7). The concept of a computational unit that operates in a systolic, on-the-fly manner, tightly coupled with the main systolic array (RSA), to identify the eviction candidate without stalling the pipeline is a clever and, to my knowledge, new design. It solves a specific performance problem created by the proposed eviction policy in an elegant way.

Weaknesses

My main concern is that the paper presents several core algorithmic and policy-level ideas as fundamentally new, when they are in fact adaptations or direct parallels of well-established concepts from prior work. The "delta," or the degree of novelty, is smaller than implied.

1. Use of eDRAM for On-Chip Acceleration: The foundational premise of using eDRAM as a dense, low-power alternative to SRAM for on-chip neural network accelerators is not new. This path has been well-trodden, particularly in the domain of CNNs. For instance, DaDianNao [15] explored eDRAM for machine learning supercomputers, and RANA [76] specifically proposed refresh-optimized eDRAM for efficient neural acceleration. The novelty here is the application to LLMs, not the core architectural choice.

2. Attention-based Eviction Policy: The AERP eviction policy, which prunes tokens based on their summed attention scores (Equation 3, page 4), is conceptually almost identical to the "heavy-hitter" identification method proposed in H2O [98]. H2O also identifies important tokens by their high accumulated attention scores. The paper needs to clearly articulate the fundamental algorithmic innovation that differentiates its eviction policy from H2O, beyond the novelty of its hardware implementation (the Systolic Evictor). As it stands, the policy itself appears derivative.

3. Two-Dimensional Adaptive Refresh Policy (2DRP): The 2DRP is presented as a key innovation, but it is a synthesis of two known principles.

   • Importance-Aware Refresh: The idea of reducing refresh rates for less critical data is not new. RANA [76] did precisely this by linking refresh frequency to the impact of bit errors on CNN accuracy. Kelle's first dimension—refreshing High Score Tokens (HST) more frequently than Low Score Tokens (LST)—is a direct analogue of this principle applied to the LLM domain.
   • Bit-level Differential Refresh: The second dimension—refreshing Most Significant Bits (MSBs) more frequently than Least Significant Bits (LSBs)—is a classic technique in approximate and fault-tolerant memory design. The principle that errors in LSBs are less impactful than errors in MSBs is fundamental.
   • The novelty of 2DRP is therefore in the combination of these two axes for the specific use case of an LLM KV cache. It is a good piece of engineering, but not a fundamentally new concept in memory management.

4. KV Vector Recomputation: The trade-off of computation for memory is a cornerstone of computer science. While applying it to the KV cache is logical, the policy of recomputing based on token "popularity" (Section 4.1.2, page 5) is a heuristic. The novelty is limited to the specific formulation of this heuristic.

In summary, the paper constructs a powerful system, but it does so primarily by adapting and combining existing ideas. The work would be stronger if it more accurately positioned its contributions as a novel synthesis and application of these principles, rather than implying they are new from first principles.

Questions to Address In Rebuttal

1. Please explicitly detail the fundamental algorithmic difference between your attention-based eviction policy in AERP and the heavy-hitter identification method in H2O [98]. Why should your policy be considered a novel contribution distinct from this prior work?

2. Could the authors reframe the contribution of 2DRP? Given prior art like RANA [76] on importance-aware refresh for NNs and established work on bit-level differential reliability, please clarify if the novelty lies in the discovery of these principles or in their specific synthesis and application to the LLM KV cache.

3. The recomputation policy relies on a heuristic threshold where input vectors (x_n) are stored if the token is popular in "> 50% of the heads". How was this threshold determined? Please provide an analysis of the system's sensitivity to this specific value, as the robustness of a heuristic is key to evaluating its contribution.