MHE-TPE: Multi-Operand High-Radix Encoder for Mixed-Precision Fixed-Point Tensor Processing Engines

---

Review Form

Reviewer: The Guardian (Adversarial Skeptic)

Summary

The authors present MHE-TPE, a tensor processing engine architecture for mixed-precision fixed-point GEMM. The work identifies two valid issues in existing spatial accelerators: redundant partial product (PP) reductions and imbalanced compute density scaling with lower precision. To address this, the paper proposes a multi-operand high-radix encoder (MHE) to halve the number of PPs in vector-inner products and a spatiotemporal mapping strategy to achieve balanced throughput scaling. While the core concept of joint operand encoding is intriguing, the paper's claims of superiority are predicated on a flawed comparative analysis and a concerning lack of discussion regarding significant architectural overheads and practical limitations. The evaluation, while using standard tools, fails to establish the true cost-benefit of the proposed complexity.

Strengths

1. The paper correctly identifies and clearly articulates two critical and persistent challenges in the design of mixed-precision spatial accelerators: redundancy in PP reduction (Section 2.1) and the failure of real-world hardware to achieve theoretical throughput gains at lower precisions (Section 2.2).
2. The core architectural idea of jointly encoding two multiplicands to operate on two multipliers (the MHE concept, Section 3.1) is a novel approach to PP reduction at the vector level, moving beyond the single-multiplier scope of traditional Booth encoding.
3. The experimental methodology is based on a full RTL-to-GDS flow using industry-standard tools on a modern-ish process node (UMC 22nm), lending a degree of credibility to the reported area, power, and timing figures for the implemented components.

Weaknesses

1. Fundamentally Flawed Baseline Comparison: The central claim of superior efficiency rests on the comparisons in Table 8. The authors compare their flexible, mixed-precision MHE-TPE against a "Systolic Array (TPU-like)" baseline. However, the numbers provided for this baseline are for dedicated, single-precision hardware. A reconfigurable architecture will necessarily incur overhead for its flexibility that a dedicated design does not. This comparison is misleading and inflates the perceived benefits of MHE-TPE. A rigorous evaluation would require constructing and synthesizing a comparable reconfigurable baseline (e.g., using tiled low-precision multipliers with a reconfigurable reduction network) on the same UMC 22nm process. Without this, the claims of superior TOPS/W and TOPS/mm² are unsubstantiated.

2. Understated Architectural Overheads and Complexity: The paper glosses over several sources of significant overhead:

   • Dual-Clock Domain: The use of a 4x fast clock for MHE/MHD and a slow clock for the compressor trees (Section 3.2.3, Page 6) is a complex design choice. The costs associated with robust cross-clock domain (CDC) synchronization logic (arbiters, synchronizer flops, gray code counters, etc.) in terms of area, power, and potential timing verification challenges are non-trivial. These costs do not appear to be explicitly broken out in the area/power analysis in Table 5.
   • VPP Pre-computation Latency: The VPP LUT generation requires 6 cycles of pre-computation (Section 3.2.2, Page 6). The paper frames this within a WS dataflow, assuming high data reuse. However, this preprocessing overhead could severely degrade performance for GEMM workloads with a small K-dimension, where the reuse of Matrix B is limited. The paper fails to provide any sensitivity analysis of performance with respect to the K-dimension.
   • Control Fabric: The spatiotemporal mapping methodology requires a sophisticated control fabric to manage the temporal iteration over Matrix A bit-slices and the spatial allocation of Matrix B bit-slices across different TPE Tiles. The area and power of this control logic are not detailed.

3. Inflexible Architectural Constraints: The design appears to be rigidly tied to a 4-bit spatial slicing of Matrix B. As stated in Section 4.7 (Page 13), the VPP LUT bit-width is constrained to 6 bits to support this 4-bit slicing. This imposes a fundamental architectural limitation. The entire premise of spatial scaling for precision relies on this partitioning. The paper does not discuss the ramifications of this constraint or how the architecture would adapt if, for example, 8-bit or 6-bit native slices were more efficient for a given workload.

4. Proposed Solutions for Low Utilization Are Not Implemented: In the analysis of CNN workloads (Section 4.6.2, Page 12), the authors note that utilization drops to 60.88% in deeper layers. They propose a "transposed dataflow layout" as a "viable optimization strategy." This is a purely hypothetical solution. There is no evidence presented that the MHE-TPE hardware, with its specific systolic-like dataflows, can actually support such a transposition efficiently, nor is the area/power overhead of the necessary routing/multiplexing logic for such a dataflow accounted for. Proposing a software fix for a hardware limitation without showing hardware support is a critical weakness.

Questions to Address In Rebuttal

1. Can the authors provide a direct, apples-to-apples comparison against a baseline mixed-precision accelerator (e.g., a tiled INT4 multiplier array with a reconfigurable reduction tree) designed and synthesized using the same UMC 22nm process and toolchain?
2. Please provide a detailed breakdown of the area and power overhead specifically for the cross-clock domain synchronization logic required by your dual-clock design. How much does this contribute to the total TPE Tile area and power?
3. How does the overall performance (effective TOPS) of the MHE-TPE array degrade as the matrix dimension K decreases (e.g., K = 64, 32, 16)? At what point does the 6-cycle VPP LUT pre-computation overhead negate the benefits of PP reduction?
4. Regarding the "transposed dataflow layout" proposed to fix low utilization in CNNs: Does the hardware described in Figure 8 contain the necessary interconnects (e.g., crossbars, routing muxes) to implement this dataflow? If so, what is their area and power cost? If not, the claim of this being a viable solution for your architecture is unsubstantiated.
5. The architecture is built around a 4-bit spatial slice for Matrix B. What would be the architectural implications and required redesign effort to support a native 8-bit spatial slice? Does this hardcoded 4-bit granularity represent a fundamental performance limiter for future workloads?

---

Review Form

Reviewer: The Synthesizer (Contextual Analyst)

Summary

This paper introduces MHE-TPE, a novel architecture for mixed-precision fixed-point tensor processing. The authors identify two fundamental limitations in conventional spatial architectures like systolic arrays: 1) redundant computation in the reduction of partial products (PPs) across both spatial and temporal domains, and 2) imbalanced computational density, where throughput gains from lower precision fail to reach theoretical limits (e.g., INT4 delivering only 2x, not 4x, the throughput of INT8).

The core contribution is a paradigm shift from optimizing single scalar multiplications to optimizing vector inner products directly. This is achieved through a Multi-operand High-Radix Encoder (MHE) that processes pairs of multiplicands (Am,2k, Am,2k+1) to generate selection signals for a pre-computed Vector Partial Product (VPP) Lookup Table. This VPP LUT stores linear combinations of corresponding multiplier pairs (B2k,n, B2k+1,n), effectively halving the number of terms that must be summed in the subsequent reduction tree. To address the mixed-precision scaling problem, the authors propose an elegant spatiotemporal mapping strategy: higher precision in the multiplicand matrix (A) is handled by iterating temporally over bit-slices, while higher precision in the multiplier matrix (B) is handled by distributing bit-slices spatially across adjacent processing tiles. This decoupling allows for balanced, near-theoretical throughput scaling across a wide range of precisions (INT2 to INT32). The authors provide a detailed microarchitectural design and extensive experimental validation, including synthesis results and performance on modern DNN workloads like Llama3 and ResNet50.

Strengths

The primary strength of this work lies in its conceptual elegance and its direct, effective solution to well-known, practical problems in accelerator design.

1. A Novel Architectural Primitive: The concept of a multi-operand encoder for vector inner products is a significant step beyond traditional Booth encoding. Instead of viewing GEMM as a grid of independent MAC operations, the authors reconsider the dot product as the fundamental unit to be optimized. By co-encoding pairs of operands, they fuse two multiplications and an addition into a single lookup and scaling operation before the main reduction network. This is a powerful and generalizable idea that reduces hardware complexity in the most critical component—the reduction tree.

2. Solves the Mixed-Precision Scaling Problem: The spatiotemporal mapping scheme is a standout contribution. The challenge of achieving balanced performance scaling with mixed precision is a major issue in both commercial and academic accelerators, which often rely on composing smaller multipliers, leading to overhead and inefficiency. The authors' method of mapping Matrix A's precision to the time domain and Matrix B's precision to the spatial domain is a clean, scalable solution. It ensures that hardware resources are used efficiently, delivering the 4x density for INT4 vs. INT8 that theory promises but practice rarely delivers. This is a significant practical achievement.

3. Broad Applicability and Generality: Unlike many contemporary works that rely on specific data properties like bit-level sparsity (e.g., Stripes, Laconic) or activation distributions (e.g., LUT-based approaches like LUTein), the MHE-TPE approach is fundamentally mathematical. It exploits the algebraic structure of the dot product itself. This makes the architecture broadly applicable to any dense or sparse GEMM computation without depending on favorable data statistics, enhancing its value as a general-purpose building block.

4. Thorough and Convincing Evaluation: The experimental methodology is comprehensive. The authors analyze the architecture from the component level (Table 4) to the full array (Table 6), exploring a wide design space (Table 5). The analysis under different process corners and conditions (Table 7, Figure 11) and, most importantly, the evaluation on relevant, modern workloads like Llama3 and ResNet50 (Figures 13 & 14) provide strong evidence for the practicality and effectiveness of their design. This level of detail builds significant confidence in the reported results.

Weaknesses

The paper's core ideas are strong, and the weaknesses are relatively minor, mostly relating to the exploration of design-space boundaries and the presentation of complex ideas.

1. Implicit Design Constraints of the VPP LUT: The VPP LUT is central to the design. The paper explains that it stores 8 pre-computed values derived from a 4-bit slice of two B operands. While the spatial mapping of 4-bit slices is well-justified for scaling Matrix B's precision, the paper could benefit from a deeper discussion on why a 4-bit slice is the optimal design point for the LUT itself. What are the area/power/timing implications of designing a VPP LUT based on, for example, 2-bit or 8-bit slices of B? This would help clarify the trade-offs that led to the current design.

2. Clarity of the Array-Level Dataflow: While the component-level diagrams (Figures 6 & 7) are clear, the paper could improve the intuitive leap to the full array-level spatiotemporal mapping (Figure 10). A more explicit, step-by-step example walking through a simple mixed-precision GEMM (e.g., INT8 x INT4) showing how the data for Matrix A iterates temporally and how the high/low nibbles of Matrix B are placed in different tiles would significantly aid reader comprehension.

3. Overhead of Inter-Tile Communication and Control: The design relies on a Local Reduce Module (LRM) for cross-tile accumulation and systolic broadcast of selection signals. While the components are evaluated, the complexity and overhead of the control logic and routing required to manage this dynamic, precision-dependent dataflow are not explicitly broken out. In a physical implementation, this control fabric can contribute non-trivially to area and power.

Questions to Address In Rebuttal

1. Could the authors elaborate on the design trade-offs of the VPP LUT's input bit-width? Is the choice of processing 4-bit slices of Matrix B fundamental to the architecture's efficiency, or could it be adapted for wider slices (e.g., 8-bit)? If so, how would this impact the LUT's complexity and the overall TPE design?

2. The spatiotemporal mapping is a powerful concept. Could the authors comment on the complexity of the control logic required to manage this mapping? Specifically, how does the LRM handle the synchronization and bit-shifting required for different precision combinations (e.g., INT2xA x INT32xB vs. INT32xA x INT2xB)?

3. To provide a more direct comparison, could the authors provide an estimate of the area and power savings of their MHE-TPE tile (e.g., for an M=32, K=32 INT8 computation) compared to a hypothetical baseline tile implemented in the same UMC 22nm process, but constructed by composing four INT4 standard Booth multipliers with a conventional reduction tree, as critiqued in Section 2.2? This would help to precisely quantify the benefits of the proposed approach against its most direct alternative.

---

Review Form

Reviewer: The Innovator (Novelty Specialist)

Summary

The authors propose a novel microarchitecture, the Multi-operand High-Radix Encoder Tensor Processing Engine (MHE-TPE), designed to mitigate two key inefficiencies in mixed-precision spatial GEMM accelerators: redundant partial product (PP) reduction and imbalanced computational density scaling. The central claim to novelty lies in the "Multi-operand High-radix Encoder" (MHE). This mechanism performs a joint encoding of two multiplicand operands to generate a single selection signal. This signal is then used to retrieve a "Vector Partial Product" (VPP) from a small, pre-computed lookup table (LUT) containing linear combinations of the corresponding two multiplier operands.

By structuring the computation around 2-element vector inner products (M(A_2k)B_2k + M(A_{2k+1})B_{2k+1}), this approach effectively halves the number of PPs that need to be reduced in the subsequent hardware stages. The paper builds a complete architectural framework around this core idea, featuring a three-stage pipeline (encoding, VPP generation, reduction) and a spatiotemporal mapping strategy that assigns multiplicand (Matrix A) precision to the temporal domain and multiplier (Matrix B) precision to the spatial domain.

Strengths

The primary strength of this work is that its core contribution—the multi-operand encoder (MHE)—appears to be genuinely novel. My analysis of prior art confirms the following:

1. Novel Extension of Booth Encoding: While high-radix encoding (e.g., Radix-4, Radix-8 Booth) is a well-established technique for reducing PPs from a single multiplicand, the proposed MHE extends this concept into a vector dimension. It is, to my knowledge, the first architecture to propose a joint, simultaneous encoding of multiple multiplicand operands to reduce the number of PPs for a vector inner product. This is not merely a higher-radix encoder but a "higher-dimension" encoder.

2. Unique Application of LUTs: The use of LUTs in accelerators is not new (e.g., LUT-TensorCore [32], LUTein [16]). However, the implementation here is distinct. Prior work typically uses single-operand lookups to store pre-computed results (e.g., activation values or partial MBE terms). In contrast, the VPP LUT in this work (Figure 5, page 4) stores pre-computed linear combinations of two distinct multiplier operands (B_2k, B_{2k+1}). The lookup is indexed by a signal derived from two distinct multiplicand operands. This joint operand treatment at the hardware encoding level is a significant conceptual departure from existing LUT-based designs.

3. Coherent Architectural Framework: The proposed three-stage computational paradigm and the spatiotemporal mapping strategy (Figure 10, page 8) are logical and novel consequences of the core MHE encoding scheme. By decoupling the reduction dimension (K) from the physical compressor tree size, the architecture achieves a more balanced and predictable scaling of throughput with precision, which is a non-trivial architectural innovation.

Weaknesses

My critique is focused on the contextualization of the novelty and its inherent trade-offs, rather than the novelty itself.

1. Insufficient Differentiation from Fused Vector Operations: While the hardware implementation is novel, the functional goal—replacing two multiplications and an addition with a single, more complex operation—is conceptually related to fused vector operations or custom instructions in DSPs. The paper would be stronger if it more explicitly differentiated the MHE concept from this broader domain, highlighting why a hardware-level joint encoding approach is fundamentally different and more efficient than a higher-level instruction fusion.

2. Framing of the Problem: The motivation presented in Section 2 (page 3) identifies redundancy in standard MBE-based multipliers. This redundancy, arising from the limited codomain of the MBE function ({-2, -1, 0, 1, 2}), is a known phenomenon. The novelty is not in observing this redundancy, but in the specific mechanism proposed to exploit it at the vector level. The paper could sharpen its contribution by stating this more directly, framing the work as a novel exploitation of a known property, rather than the discovery of the property itself.

3. Implicit Complexity Trade-offs: The MHE introduces new hardware components: the VPP Select Encoder, the VPP LUT, and the pre-computation adder (Section 3.1, page 4). This represents a non-trivial area, power, and latency cost paid upfront to simplify the downstream reduction tree. The dual-clock scheme (Section 3.2.3, page 6) further adds design complexity. While the results demonstrate a net benefit for the chosen configurations, a more detailed analysis of the break-even point (e.g., for what reduction dimension K does this approach become superior to a standard design?) would better contextualize the practical domain of this novel contribution.

Questions to Address In Rebuttal

1. The proposed MHE is based on a 2-element vector inner product. What are the theoretical and practical barriers to extending this to a 3- or 4-element vector inner product? Would the VPP LUT size (currently 8 entries) and the complexity of the VPP Select Encoder grow exponentially, rendering the novelty of this approach fundamentally limited to the 2-element case?

2. The architecture's mixed-precision capability for Matrix B relies on spatial mapping of 4-bit slices across different TPE Tiles (Section 3.3.2, page 7). This fixes the internal VPP LUT data path to a width compatible with 4-bit inputs (+2 expansion bits). Does this design choice create a structural inflexibility compared to conventional bit-serial or bit-sliced architectures that can be reconfigured for, say, 1-bit or 2-bit operations on Matrix B without stranding hardware resources? How does this impact the generality of the proposed "unified hardware"?

3. The VPP LUT must be populated during a "preprocessing phase" before computation can begin. For workloads with very low data reuse (i.e., small M dimension in a GEMM), could the latency and energy overhead of this preprocessing phase negate the benefits gained from the reduced PP count? Please provide an estimate of the M dimension at which the MHE-TPE begins to outperform a conventional systolic array of the same area.