AxCore: A Quantization-Aware Approximate GEMM Unit for LLM Inference

---

Review Form

Reviewer: The Guardian (Adversarial Skeptic)

Summary

The paper presents AxCore, a general matrix-matrix multiplication (GEMM) unit designed for Large Language Model (LLM) inference. The core contribution is the replacement of conventional floating-point multipliers with a multiplier-free design based on Floating-Point Multiplication Approximation (FPMA), which utilizes integer addition. This approach is integrated into a mixed-precision systolic array that directly operates on 4-bit quantized floating-point (FP4) weights and 16-bit floating-point (FP16) activations. The authors propose several supplementary techniques to maintain accuracy, including a method for handling subnormal numbers (SNC), a constant-based error compensation scheme, and an adaptive format-aware quantization algorithm. The paper claims significant improvements in compute density and competitive or superior model accuracy compared to both conventional FP units and state-of-the-art INT4-based accelerators.

Strengths

1. Direct Targeting of a Key Bottleneck: The work correctly identifies the FP multiplier as a primary contributor to area and power costs in GEMM units and proposes a direct architectural alternative. The ambition to create a multiplier-free design is commendable.
2. System-Level Approach: The authors do not merely propose an approximate arithmetic trick but consider its integration into a full system, including a systolic array architecture (Section 5, page 7) and dataflow optimizations like Correction Advancing and Normalization Postponing (Section 5.3, page 8).
3. Inclusion of an Ablation Study: The authors provide a breakdown of accuracy improvements from their various techniques in Table 2 (page 11), which is a necessary step to justify the inclusion of each component (SNC, error compensation, etc.).

Weaknesses

My primary concerns with this paper stem from questionable methodological choices in the evaluation that appear to conflate distinct contributions, leading to claims that are not rigorously supported by the evidence provided.

1. Confounded Accuracy Evaluation: The central claim that AxCore achieves "comparable or better perplexity" (Abstract, page 1) is not soundly demonstrated. The paper compares AxCore—which includes both an approximate compute core and a novel format-aware quantization scheme—against baselines using standard quantization methods (e.g., GPTQ for FIGNA, as stated in Section 6.5.1, page 11). The observed accuracy improvements (e.g., PPL of 9.78 for AxCore vs. 9.82 for FPC and 9.95 for FIGNA on OPT-30B, Figure 1, page 2) are more likely attributable to the sophisticated quantization scheme rather than the approximate nature of the compute unit itself. An approximate method should not, by definition, be more accurate than an exact one given the same inputs. This is an apples-to-oranges comparison that conflates the benefits of a quantization algorithm with the performance of the underlying hardware. The paper lacks a crucial baseline: an exact FP4xFP16 multiplier using the authors' proposed format-aware quantization. Without this, the isolated impact of the mpFPMA approximation on accuracy remains unknown and unsubstantiated.

2. Oversimplified and Poorly Justified Error Compensation: The proposed "Mean-Based Constant Compensation" (Section 4.3.2, page 6) is a significant point of weakness. Equation (11) defines the correction constant C₁ by averaging the approximation error across all possible mantissa combinations. This implicitly assumes a uniform distribution of mantissa values. This assumption is highly unlikely to hold for the weights and, particularly, the activations within a neural network, which are known to have highly structured, non-uniform distributions. The paper provides no analysis to show that this single, pre-computed constant is robust across different models, layers, or even different input prompts that would induce varying activation distributions. This appears to be a crude heuristic, and its effectiveness is not convincingly demonstrated beyond the global perplexity numbers.

3. Unaddressed Implications of Stochastic Rounding: The handling of subnormal numbers relies on a "random selection policy" (Section 4.2.2, page 5) to mitigate rounding bias. This is a form of stochastic rounding. The implementation details in Section 5.2.2 (page 8) suggest this is not truly random but is derived from an activation bit. Regardless of the source, this introduces non-determinism into the computation, meaning identical inputs can produce different outputs on subsequent runs. This is a critical flaw for many deployment scenarios, especially those requiring verifiability, debugging, and regulatory compliance. The authors fail to discuss the implications of this non-determinism or quantify its impact on numerical stability.

4. Inadequate Comparison of Number Formats: The paper claims superiority over INT4-based designs like FIGNA. However, this comparison is fraught. FP4 and INT4 are fundamentally different 4-bit representations. While the authors claim FP4 offers "higher accuracy potential" (Section 2.3, page 3), this inherent advantage of the number format is used to claim superiority for their architecture. A rigorous comparison would require demonstrating AxCore's benefits over an architecture using an exact FP4 multiplier, or by implementing the proposed approximation techniques within an integer-based framework to provide a more direct comparison to INT4 designs.

Questions to Address In Rebuttal

The authors must address the following points to substantiate their claims:

1. On Confounded Evaluation: Can you provide perplexity results for a baseline system that uses your proposed adaptive format-aware quantization but with a conventional, exact FP4xFP16 multiplier? This is the only way to decouple the effects of your quantization algorithm from your approximate compute core and fairly assess the accuracy degradation caused by mpFPMA.

2. On Error Compensation: Please provide an analysis of the sensitivity of model accuracy to the constant compensation value C₁. How does the actual distribution of mantissas in representative LLMs (e.g., OPT, LLaMA) compare to the uniform distribution assumed in your derivation of C₁, and what is the resulting error if the distributions diverge significantly?

3. On Non-Determinism: Please clarify the exact mechanism for implementing the "random selection policy" for subnormal rounding. Acknowledge and discuss the implications of the resulting non-determinism for bit-for-bit result reproducibility in a production environment. If the method is pseudo-random, what is its period and have you analyzed potential correlations?

4. On Baseline Fairness: Given that your primary architectural claim rests on the efficiency of the mpFPMA unit, why was the decision made to present the main accuracy results (Table 2) against baselines using different number formats (INT4) and different quantization algorithms (GPTQ)? How can you claim the architecture itself is superior when so many other variables differ?

---

Review Form

Reviewer: The Synthesizer (Contextual Analyst)

Summary

The paper presents AxCore, a novel hardware accelerator architecture for Large Language Model (LLM) inference. The core contribution lies in the synergistic combination of two distinct but complementary research thrusts: weight-only quantization and approximate computing via Floating-Point Multiplication Approximation (FPMA). By fusing these concepts, the authors propose a multiplier-less mixed-precision GEMM unit where expensive floating-point multipliers are entirely replaced by simple integer adders within a systolic array.

To make this fusion practical and accurate, the paper introduces several key innovations:

1. A specialized Processing Element (PE) that performs mixed-precision FPMA directly on compressed weights (e.g., FP4) and high-precision activations (e.g., FP16).
2. A lightweight but critical accuracy preservation strategy that includes a novel hardware unit for Subnormal Number Conversion (SNC), constant-based error compensation, and an adaptive, block-wise selection of FP4 formats.
3. A set of clever systolic array optimizations, such as "Correction Advancing" and "Normalization Postponing," that reduce hardware redundancy and complexity.

The authors demonstrate through extensive evaluation that AxCore achieves significant improvements in compute density (up to 12.5x over conventional FP units) and energy efficiency, while maintaining model accuracy on par with, or even exceeding, state-of-the-art INT4-based accelerator designs.

Strengths

This is a well-executed and insightful piece of work that makes a valuable contribution to the field of hardware acceleration for AI. Its primary strengths are:

1. Elegant Conceptual Fusion: The most significant strength of this paper is its successful synthesis of ideas from approximate computing and model quantization. Rather than simply applying FPMA to a quantized model, the authors have deeply considered the second-order effects and co-designed the hardware and software components. This creates a cohesive system where the approximation, quantization format, and hardware microarchitecture are mutually supportive. This work serves as an excellent case study in bridging the gap between computer arithmetic theory and practical accelerator design for modern workloads.

2. Addressing a Critical, Understated Problem: The paper's focus on handling subnormal numbers in low-bit floating-point formats (Section 4.2, page 5) is particularly commendable. As the field aggressively pushes towards 4-bit and smaller data types, the limited exponent range makes subnormal values far more common than in traditional FP32/FP16 arithmetic. The authors correctly identify that this breaks the mathematical assumptions of FPMA and would otherwise lead to significant accuracy degradation (as shown in Figure 4, page 4). Their proposed Subnormal Number Conversion (SNC) is a pragmatic and well-justified solution to a real, emerging problem that many other works in this space overlook.

3. Strong Co-design Philosophy: The work is a prime example of effective software-hardware co-design. The offline, adaptive format-aware quantization strategy (Section 4.4, page 6) is not just a software trick; it is enabled by a hardware design that can concurrently support multiple FP4 formats without significant overhead. Similarly, the mean-based error compensation is an offline analysis that translates into a simple, low-cost online hardware correction. This holistic approach leads to a far more optimized result than if the software and hardware were designed in isolation.

4. Thorough and Convincing Evaluation: The evaluation is comprehensive and well-structured. The ablation study presented in Table 2 (page 11) is particularly powerful, as it clearly dissects the contribution of each accuracy-preserving technique (SNC, error compensation, etc.). This provides convincing evidence that the final excellent results are not accidental, but a direct consequence of their design choices. The comparison against strong and recent baselines, including the INT4-based FIGNA and LUT-based FIGLUT, grounds the performance claims in the context of the current state-of-the-art.

Weaknesses

The paper is strong, and its weaknesses are more related to missed opportunities for broader contextualization and exploration rather than fundamental flaws.

1. Limited Exploration of the Design Space: The paper is heavily focused on a W4A16 mixed-precision scenario. While this is a highly relevant and popular configuration for LLM inference, the underlying principles seem more general. The discussion could be strengthened by exploring how the AxCore philosophy might extend to other data formats, such as FP8 (as used in NVIDIA's Hopper) or even non-standard formats like block floating point. This would help contextualize where the FPMA-based approach is most effective and where its limitations might lie.

2. Positioning Relative to Logarithmic Computing: The FPMA technique is explicitly based on Mitchell's approximation, which is a cornerstone of Logarithmic Number Systems (LNS). The paper could benefit from briefly positioning itself within this broader historical context of computer arithmetic. AxCore can be seen as a highly specialized, lightweight, and workload-aware application of LNS principles, avoiding the overhead of a general-purpose LNS datapath (e.g., lookup tables for log/antilog conversion) by keeping activations in the linear domain. Acknowledging this connection would not diminish the novelty but rather highlight how the authors have cleverly adapted and simplified a classic idea for the modern LLM domain.

3. Potential Brittleness of Constant-Based Compensation: The mean-based error compensation (Section 4.3.2, page 6) is an elegant, low-cost solution. However, by using a single pre-computed constant, there is a small risk that its effectiveness is dependent on the data distribution of the calibration set. While the results show this works well, a brief discussion of the sensitivity of this method to out-of-distribution shifts in activation statistics would add nuance.

Questions to Address In Rebuttal

1. Regarding the Subnormal Number Conversion (SNC) unit: The use of stochastic rounding for ambiguous cases (Section 4.2.2, page 5) is an interesting choice to mitigate bias. Could the authors comment on the hardware cost and complexity of the random bit generation required for this? Was a simpler deterministic rounding scheme (e.g., round-to-nearest-even) evaluated, and what was its comparative impact on model accuracy?

2. Regarding the adaptive format-aware quantization: The selection is made from a set of three representative FP4 formats (E3M0, E2M1, E1M2). What is the sensitivity of the final accuracy to this specific set of choices? Is there a point of diminishing returns, or could further gains be realized by considering a wider array of custom FP4 formats?

3. The core idea is potent for weight-only quantization. Could the authors speculate on the applicability of the AxCore architecture to future scenarios that might also quantize activations (e.g., W4A8)? What would be the primary new challenges in extending the mpFPMA concept to handle two low-bit, approximate inputs, particularly concerning alignment and dynamic range?