Oaken: Fast and Efficient LLM Serving with Online-Offline Hybrid KV Cache Quantization

Review Form: The Guardian

---

Summary

The authors present Oaken, a solution for accelerating LLM inference serving by quantizing the Key-Value (KV) cache. The core contribution is a hybrid online-offline quantization algorithm. This approach involves profiling KV cache distributions offline to establish static outlier thresholds. These thresholds are then used online to partition KV values into three groups (inner, middle, outer). The authors propose a "group-shift" technique to narrow the dynamic range of outlier groups for better low-bit quantization and a "fused dense-and-sparse" encoding scheme to reduce the storage overhead of sparse outliers. The algorithm is implemented with custom hardware modules intended for integration into LLM accelerators. The evaluation, performed using a custom simulator, claims significant throughput improvements over baselines like NVIDIA A100 with vLLM and other quantization methods, with minimal accuracy degradation.

---

Strengths

1. Algorithmic Novelty: The concept of combining offline profiling with online quantization to reduce runtime overhead is sound. The proposed group-shift and fused dense-sparse encoding techniques are clever algorithmic contributions aimed at maximizing the compression ratio while managing quantization loss.
2. Problem Relevance: The paper correctly identifies the KV cache as a primary bottleneck in batched LLM inference, a critical and well-recognized problem in the field. Addressing both memory capacity and bandwidth constraints is the correct focus.
3. Comprehensive Hardware Design: The authors go beyond a purely algorithmic proposal by detailing a hardware implementation (Section 5), including quantization/dequantization engines and a specialized MMU. This demonstrates a thorough, system-level consideration of the problem.

---

Weaknesses

My review identifies several critical weaknesses that challenge the validity and robustness of the paper's central claims.

1. Unsupported Foundational Assumption: The entire premise of the "offline" profiling strategy rests on the assumption that KV cache value distributions are largely independent of the input data distribution (Observation 2, Section 4.1, page 6). The authors provide evidence for this using four NLP datasets (Wikitext2, PIQA, Hellaswag, Winogrande) which are stylistically similar. This is insufficient proof. The methodology is critically flawed if these static thresholds do not generalize to out-of-distribution prompts, such as those involving code generation, structured data, or multilingual text. The paper presents no evidence to support this crucial generalization, rendering the "offline" component potentially brittle.

2. Misleading and Inconsistent Accuracy Claims: The abstract claims an impressively "minimal accuracy loss of only 0.54% on average, compared to state-of-the-art KV cache quantization techniques." However, the data in Table 2 (page 11) does not substantiate this claim clearly and contradicts it in some cases.

   • Comparison to FP16 Baseline: The actual accuracy loss compared to the FP16 Original baseline is often significantly higher. For example, on Llama2-7B with Hellaswag, the accuracy drops from 75.98% to 73.72%, a loss of 2.26%. On Winogrande, the drop is from 69.13% to 67.64%, a loss of 1.49%. It is unclear how the 0.54% figure was calculated; it seems to selectively ignore the worst-case results.
   • Comparison to SOTA: The claim of superiority over other quantization methods is also inconsistent. For Llama2-7B on Wikitext2, Oaken's perplexity (5.53) is worse than KVQuant's (5.49). On PIQA, Oaken's accuracy (78.29%) is lower than KVQuant's (78.35%). While Oaken may perform better on other metrics, the claim of consistent superiority is an overstatement not fully supported by the authors' own data.

3. Use of Unjustified Global Hyperparameters: The authors fix the quantization group ratios to 4% outer, 90% middle, and 6% inner for all experiments across all models and datasets (Section 6.1, page 10). The justification provided in Figure 12(a) (page 12) is based solely on a single model (Llama2-7B) on a single dataset (Wikitext2). This is a classic case of over-fitting a hyperparameter to a narrow experimental setup and then generalizing without evidence. There is no reason to believe this ratio is optimal, or even good, for other models with different architectures (e.g., Mixtral-8x7B) or sizes. This undermines the robustness of the reported results.

4. Inequitable Baseline Comparisons: The performance evaluation methodology appears biased. In Section 6.1, the authors state they "disable" weight and activation quantization features for baselines like QServe and Tender for a "fair comparison." This is not a fair comparison. These are integrated systems, and their performance may rely on the interplay of all quantization features. By selectively disabling core functionalities, the authors are not comparing Oaken to the true state-of-the-art but to a crippled version of it. A proper evaluation must compare against these systems in their fully-enabled, intended configurations.

5. Conflation of Algorithmic and System-Level Gains: The paper presents Oaken-LPDDR as the top-performing configuration for large-scale serving. However, a significant portion of its ability to handle larger batches comes from the system-level choice of high-capacity LPDDR memory, not solely from the quantization algorithm. The paper needs to more rigorously disentangle the performance gains directly attributable to the compression from the Oaken algorithm versus the gains from the underlying memory hardware choice. The algorithm enables the use of LPDDR, but it does not, by itself, grant the capacity.

---

Questions to Address In Rebuttal

1. Please provide a precise, step-by-step calculation demonstrating how the "0.54% average accuracy loss" figure was derived from the results in Table 2. Explicitly state which baseline was used for this comparison and justify this choice.
2. The core assumption of data-independent KV cache distribution is foundational. Please provide experimental evidence showing that the offline-profiled thresholds are robust for out-of-distribution tasks, such as code generation or mathematical reasoning.
3. Please justify the use of a single, global 4%/90%/6% group ratio across all models. Provide a sensitivity analysis for this hyperparameter on at least one other model, such as Mixtral-8x7B or OPT-30B, to demonstrate that the chosen configuration is not simply cherry-picked.
4. Can the authors justify their decision to disable core features of the baseline systems (QServe, Tender)? Please either provide results comparing Oaken to these systems in their fully-optimized, published configurations or provide a stronger argument for why the current comparison is fair and representative.
5. Regarding the results in Figure 13 (page 12), how much of Oaken-LPDDR's ability to handle longer sequences (16K+) is due to the capacity of LPDDR versus the compression from the Oaken algorithm? Can you quantify the contribution of each?

Review Form: The Synthesizer

---

Summary

This paper introduces Oaken, a hardware-software co-designed solution to address the critical key-value (KV) cache bottleneck in large language model (LLM) inference serving. The core contribution is a novel online-offline hybrid quantization algorithm. This approach performs a one-time, offline profiling of a given model to determine robust thresholds for identifying outlier values in the KV cache. These pre-computed thresholds are then used during live inference to perform a lightweight, dynamic online quantization, separating values into inlier and outlier groups without the need for expensive online sorting or analysis.

To translate this algorithmic gain into performance, Oaken proposes custom hardware modules—including quantization/dequantization engines and a specialized memory management unit (MMU)—designed to be integrated into an LLM accelerator. The authors implement and evaluate this system, demonstrating significant throughput improvements (up to 1.58× over state-of-the-art methods on an A100 GPU) with minimal accuracy degradation. The work situates itself as a practical and effective solution to the conflicting demands for high memory bandwidth and high memory capacity in modern LLM serving.

---

Strengths

1. Elegant Core Idea: The central concept of an online-offline hybrid approach is a particularly insightful contribution to the field of KV cache quantization. The current landscape is largely divided between:

   • Fully Online Methods (e.g., KVQuant [22], KIVI [43]), which offer high accuracy by dynamically identifying outliers for every request but pay a steep performance penalty for sorting and mixed-precision handling.
   • Static/Reordering Methods (e.g., QServe [41], Atom [86]), which have lower runtime overhead but may sacrifice accuracy by relying on less adaptive, coarse-grained heuristics like channel reordering.

   Oaken carves out a clever and pragmatic middle ground. The insight that a model's activation distributions have stable, intrinsic properties that can be captured once offline (as shown in Section 4.1, page 6) is powerful. This allows the system to reap the benefits of outlier-aware quantization without bearing the crippling cost of dynamic discovery, effectively finding a sweet spot on the accuracy-performance curve.

2. Holistic System Co-Design: A major strength of this paper is its commitment to a full-stack solution. The authors do not merely propose an algorithm and speculate on its performance benefits. Instead, they meticulously design the hardware necessary to make the algorithm effective. The design of the custom quantization engine, the fused dense-and-sparse encoding scheme (Section 4.5, page 8), and the dual-table MMU for handling the resulting data structures (Section 5.2, page 9) demonstrates a deep understanding of the practical challenges. This co-design makes the impressive performance results far more credible than those in algorithm-only papers, as it directly accounts for implementation overheads.

3. Excellent Contextualization and Problem Framing: The paper does an outstanding job of positioning its work within the broader research landscape. The introductory discussion and, in particular, Figure 1 (page 2), provide a clear and compelling map of the solution space for LLM inference. By plotting existing works on a "bandwidth-capacity trade-off" axis, the authors immediately establish the context and significance of their contribution. This framing helps the reader understand that Oaken is not just another quantization paper, but a systemic attempt to push the Pareto frontier of serving efficiency. The analysis in Section 3, which motivates the work by examining the limitations of HBM vs. LPDDR memory, further grounds the research in tangible, real-world system design constraints.

---

Weaknesses

While the paper is strong, its central premise rests on an assumption whose boundaries could be explored more deeply.

1. Robustness of Offline Profiling: The entire approach is predicated on "Observation 2" (Section 4.1, page 6)—that the range of KV cache values is consistent across different input datasets. The authors validate this on several standard academic benchmarks. However, in real-world deployment, LLM services encounter a vast and unpredictable range of inputs, including adversarial prompts, out-of-distribution topics (e.g., code generation, non-English languages), and different fine-tuning domains. The paper would be strengthened by a discussion on the sensitivity of the offline-generated thresholds. How gracefully does the system degrade if it encounters inputs that generate activation distributions that deviate significantly from the profiling set? A sensitivity analysis could better define the operational envelope of Oaken.

2. Disentangling Algorithmic vs. Hardware Gains: The co-design is a strength, but it also makes it slightly difficult to isolate the benefit of the Oaken algorithm itself from the custom accelerator it runs on. The baselines are either GPU-based systems (vLLM, QServe) or another accelerator (Tender). While the latency breakdown in Figure 12b (page 12) is useful, a direct throughput comparison of an "Oaken-on-GPU" software kernel against other GPU baselines in Figure 11 would be highly illuminating. This would allow the community to understand how much of Oaken's advantage stems from its more efficient algorithm versus the inherent benefits of a specialized ASIC implementation, providing a clearer picture of its applicability to existing commodity hardware.

---

Questions to Address In Rebuttal

1. Could the authors comment on the robustness of the offline profiling step? Have they investigated how sensitive the chosen thresholds are to more dramatic domain shifts in input prompts (e.g., from prose to source code, or across different languages)? What would be the recommended procedure for a production system—is one-time profiling on a general-purpose dataset sufficient, or would you recommend re-profiling for specific, fine-tuned models or domains?

2. The hardware-software co-design is a key feature. To help the community better appreciate the algorithmic contribution, would it be possible to provide throughput data for a CUDA-kernel implementation of the Oaken quantization algorithm running on the A100 GPU? Placing this as a baseline in Figure 11 would clarify how much of the performance gain over methods like KVQuant comes from the superior algorithm (avoiding online sorting) versus the custom hardware.

3. The paper proposes Oaken as a set of modules that can be integrated with "any LLM accelerators." Could the authors briefly elaborate on the practical considerations for such an integration? For example, what are the primary interface requirements between the Oaken DMA unit and a host accelerator's processing cores and memory subsystem?

Reviewer: The Innovator (Novelty Specialist)

Summary

The paper presents Oaken, an algorithm-hardware co-design for accelerating batched LLM inference by quantizing the Key-Value (KV) cache. The authors identify the high cost of online outlier detection in existing mixed-precision KV cache quantization schemes as a key performance bottleneck.

The central claim to novelty lies in a hybrid online-offline approach. The authors propose to:

1. Offline Profile for Thresholds: Determine static thresholds that partition KV cache values into three groups (inner, middle, outer) based on pre-computed statistics from sample prompts.
2. Online Scale Calculation: During inference, dynamically calculate the quantization scale (min/max) for each of the three groups online, but only for the newly generated token.
3. Group-Shift Quantization: For the outer and middle groups (outliers), shift the values by the offline-determined thresholds before quantization to narrow their dynamic range.
4. Fused Dense-and-Sparse Encoding: Propose a hardware-aware memory layout where a portion of the sparse outlier data is stored in the zeroed-out entries of the dense inlier matrix to reduce metadata overhead.

These algorithmic choices are coupled with custom hardware units (quantization/dequantization engines and an MMU) designed to accelerate this specific workflow. The authors claim this co-design achieves superior throughput compared to existing GPU-based solutions with minimal accuracy loss.

Strengths

The primary strength of this work is the clever synthesis of existing quantization concepts into a novel, high-performance system designed to solve a well-defined problem. The novelty is not in a single disruptive idea, but in the specific combination and refinement of several techniques.

1. The Online-Offline Hybrid Scheme: The core idea of using offline profiling to determine static group thresholds while calculating quantization scales online is a novel and pragmatic solution to the performance-accuracy trade-off. It correctly identifies that online sorting (e.g., topK in KVQuant [22]) is prohibitively expensive, while a purely static, offline approach may lack adaptivity. This hybrid method appears to be a new point in the design space.

2. Group-Shift Quantization: The technique of subtracting offline-derived thresholds from outlier values before quantization (Section 4.4, page 7) is a non-trivial algorithmic contribution. It directly addresses the challenge of quantizing wide-range values to a low bitwidth without resorting to higher-precision formats (e.g., FP16), thereby reducing the overhead associated with mixed-precision data handling.

3. Fused Dense-and-Sparse Encoding: This memory layout optimization (Section 4.5, page 8) is a clever, hardware-aware contribution. While storing outliers sparsely is not new (e.g., SqueezeLLM [30]), repurposing the zeroed entries in the dense matrix to store part of the outlier data is a specific and novel technique for reducing the significant metadata overhead of sparse formats.

Weaknesses

From a novelty perspective, the primary weakness is that the constituent components of the proposed solution are variations on well-established themes in quantization and systems design. The work is more of an expert-level integration and optimization than a fundamental conceptual breakthrough.

1. Incremental Algorithmic Concepts:

   • The concept of identifying and separately handling outliers is the foundation of most modern high-accuracy quantization schemes (e.g., LLM.int8() [15], SmoothQuant [71], Olive [19]).
   • The extension from a binary inlier/outlier split to a ternary inner/middle/outer grouping (Section 4.3, page 6) is an incremental, rather than fundamental, advancement. The motivation to preserve small-magnitude values is well-documented in prior art [2, 13, 27, 34], which the authors cite.
   • The "group-shift" technique, while effective, is conceptually analogous to affine quantization, where a zero-point is used to shift the data range. Here, the shift value is cleverly tied to the outlier threshold, but the underlying principle of data shifting pre-quantization is not new.

2. Coupled Hardware Novelty: The hardware modules described in Section 5 (page 8) are themselves direct, albeit efficient, implementations of the proposed algorithm. Their novelty is therefore coupled to, and not independent of, the algorithmic novelty. The idea of building custom hardware for quantization/dequantization is not new; the contribution is that this hardware is purpose-built for the specific Oaken algorithm. As such, the hardware's novelty cannot be evaluated in isolation.

Questions to Address In Rebuttal

1. Delineation from Prior Hybrid Approaches: Could the authors more precisely delineate the novelty of their online-offline hybrid scheme against prior works that use offline calibration data to simplify online computation? For instance, SmoothQuant [71] uses an offline calibration set to determine scaling factors that are applied statically during inference. While the mechanism is different, the principle of using offline analysis to avoid expensive online calculations is shared. Please clarify what makes the "offline thresholds, online scaling" combination fundamentally new.

2. Necessity of the Three-Group Design: The paper justifies the three-group (inner/middle/outer) design to handle both large- and small-magnitude outliers. However, Table 3 (page 12) explores variations with four and five groups but does not include a direct comparison to a simpler two-group (inlier/outlier) scheme that uses the same group-shift and fused encoding techniques. Is the complexity of the third group (and its associated thresholds) essential for the reported performance, or would a two-group system achieve comparable results with less overhead? This is critical to justifying the novel complexity introduced.

3. Generality of Fused Encoding: The fused dense-and-sparse encoding is presented as a key optimization for reducing the bitwidth of an outlier entry from 23 to 8 bits. How sensitive is this technique to the specific bitwidths chosen (4-bit inlier, 5-bit outlier)? Does the benefit and the 8-bit alignment hold if, for example, 3-bit or 2-bit quantization were used for inliers, which is a direction of active research? Please comment on the generality of this novel encoding scheme beyond the specific configuration presented.