PolymorPIC: Embedding Polymorphic Processing-in-Cache in RISC-V based Processor for Full-stack Efficient AI Inference

---

Review Form

Reviewer: The Guardian (Adversarial Skeptic)

Summary

The paper presents PolymorPIC, a Processing-in-Cache (PIC) architecture integrated into the Last-Level Cache (LLC) of a RISC-V System-on-Chip (SoC). The stated goal is to accelerate neural network inference, specifically bit-serial matrix multiplication (BSMM), while addressing system-level challenges such as coherence, programmability, and area efficiency, which the authors claim are overlooked by prior work. The proposed solution involves reconfigurable Homogeneous Memory Arrays (HMAs) that can function as cache, compute units, or buffers. The authors introduce a coherence protocol (DCF and PAI) to manage the transition between cache and PIC modes, claiming it is "processor-safe." The system is implemented on an FPGA with a Linux OS, and evaluation results are presented based on ASIC synthesis, claiming significant improvements in area and energy efficiency over baseline processors and competing accelerators like Gemmini.

While the ambition to create a full-stack, processor-safe PIC system is commendable, the manuscript suffers from significant methodological weaknesses in its evaluation and makes several unsubstantiated or overstated claims that undermine its conclusions. The core contributions, particularly regarding coherence and performance, require much stronger validation.

Strengths

1. System-Level Integration: The effort to integrate a PIC architecture into a full SoC stack, including a RISC-V core (BOOM), standard bus protocols (TileLink), and verification with a running operating system (Linux), is a notable strength. This moves beyond circuit-level proofs-of-concept common in this domain.
2. Use of Standard-Cell SRAM: The decision to leverage standard SRAM arrays generated by a memory compiler (as mentioned in Section 3.2 and 7.1.4) is a practical approach that avoids the area and compatibility issues associated with the custom bit-cells used in prior works like Neural Cache [25].
3. Focus on Coherence: The paper correctly identifies cache coherence as a critical system-level barrier for PIC adoption. The explicit attempt to design mechanisms for allocation, isolation, and release (Section 5) is a necessary and important research direction.

Weaknesses

1. Misleading Performance Metrics and Unfair Baselines: The headline performance claims are built on questionable comparisons.

   • The claim of a 1543.8x energy efficiency improvement (Abstract) is made against a general-purpose OoO core (BOOM). This is a classic accelerator-vs-CPU comparison that, while technically correct, is largely meaningless. Any dedicated hardware will show orders-of-magnitude improvement over a general-purpose core for its target workload. This number is inflammatory and does not provide a useful comparison against state-of-the-art accelerators.
   • The comparison against Gemmini appears to be deliberately handicapped. Section 7.1.3 states the total buffer capacity for the NPU was constrained to match the cache size of PolymorPIC. Gemmini's systolic array architecture is highly sensitive to buffer size for hiding memory latency; forcing a 1MB buffer configuration for an 8x8 or 16x16 array may not be optimal and could cripple the baseline's performance. More critically, the "area efficiency" metric (GOPS/mm²) in Figure 15 is highly suspect. PolymorPIC's absolute throughput is clearly lower than Gemmini16 (Figure 16). The only way PolymorPIC can claim higher "area efficiency" is if the normalization area includes the entire processor, memory, and peripherals, thereby diluting the large area of the Gemmini NPU across the entire SoC. Efficiency should be judged on the area of the added accelerator components, not the whole chip. This is an apples-to-oranges comparison.

2. Insufficiently Validated Coherence Mechanism: The proposed "processor-safe" coherence strategy is not rigorously proven.

   • The Direct Cache Flush (DCF) mechanism (Section 5.1, Figure 8) requires the Switch Controller to iterate through every set in the LLC to query the directory for a specific way. For a 1MB, 16-way cache with 1024 sets (as per their BOOM-S configuration), this is 1024 queries to the directory per way being switched. While better than the naive approach, the absolute latency of this operation is not quantified, nor is its impact on the memory subsystem's availability for other processor requests.
   • The PIC Array Isolation (PAI) mechanism (Section 5.2) relies on a single PIC_mode flag per way to make it "invisible" to the MESI replacement policy. This seems overly simplistic. It does not address more complex coherence scenarios, such as snoops from other cores in a multi-core system, or handling of speculative memory accesses by an OoO core that might target an address mapped to an isolated way. The claim of being "processor-safe" is not substantiated beyond a trivial case.

3. "Full-Stack Verification" is Superficial: The claim of being "successfully end-to-end verified on a validation platform with an operating system running" is an overstatement based on the evidence provided.

   • The evaluation in Section 7.4 involves running a single SPEC2017 benchmark in parallel with the BSMM computation. This is a controlled "hero run" scenario. It does not stress the system with realistic OS-level complexities like high interrupt frequency, context switching during PIC operations, or contention on the memory bus from multiple I/O devices. The interaction between the OS scheduler and the PIC resource management is not explored. Therefore, the robustness of the system in a real-world, dynamic environment remains unproven.

4. Novelty of Scheduling is Unclear: Section 6.3 discusses scheduling using standard Input/Weight Stationary (I/WS) and Output Stationary (OS) dataflows. While the analysis in Figure 17 is useful, these dataflows are not novel; they are standard practice in the design of NN accelerators. The paper fails to articulate a specific, novel scheduling algorithm or contribution beyond applying existing concepts to their HMA architecture. The software stack's role in making these scheduling decisions is not detailed sufficiently.

Questions to Address In Rebuttal

1. Regarding Performance Claims:

   • Please clarify the exact methodology for calculating "area efficiency" in Figure 15. Is the normalization area (the denominator in GOPS/mm²) the area of the accelerator-specific add-ons only, or the total SoC area? If the latter, provide a strong justification for why this is a fair metric for comparing a tightly integrated PIC unit with a loosely coupled co-processor like Gemmini.
   • Provide evidence that the buffer size and configuration used for the Gemmini baseline are representative of an optimized system, rather than a configuration constrained to match PolymorPIC's LLC size.
   • Please re-frame the 1543.8x energy efficiency claim in the context of accelerator-vs-accelerator comparisons, as the current framing against a CPU baseline is not insightful.

2. Regarding the Coherence Mechanism:

   • What is the absolute cycle latency of the DCF operation for allocating a single way (e.g., in the 1MB LLC configuration), and how does this scale with the number of sets and ways?
   • Elaborate on the PAI mechanism's handling of complex coherence scenarios. For example, in a multi-core BOOM configuration, if Core 1 has isolated Way 5 for PIC, what happens when Core 2 issues a read request that would map to Way 5 and misses in all other ways?
   • How does the system handle an interrupt or a high-priority preemption request that occurs mid-way through a DCF or PIC execution phase? Is the state machine gracefully pausable and resumable?

3. Regarding System Verification:

   • Beyond the single parallel SPEC run shown in Figure 22, what further stress tests were conducted to validate the "processor-safe" claim under more dynamic OS conditions (e.g., heavy I/O, frequent context switching, virtual memory pressure leading to page swaps)?

4. Regarding Scheduling:

   • Please clarify the precise division of labor between the hardware PIC scheduler and the software stack (Table 1). Who makes the high-level decision between I/WS and OS dataflows for a given layer, and based on what cost model? What novel scheduling algorithm, if any, is being proposed?

---

Review Form

Reviewer: The Synthesizer (Contextual Analyst)

Summary

This paper introduces PolymorPIC, a full-stack architecture that integrates polymorphic processing-in-cache (PIC) capabilities into the Last-Level Cache (LLC) of a RISC-V System-on-Chip (SoC). The core contribution is not merely the act of computation within a cache, but the holistic, system-level design that makes this feasible, programmable, and safe to operate alongside a host processor running a full operating system.

The authors achieve this through three key thrusts:

1. An architecture based on reconfigurable Homogeneous Memory Arrays (HMAs) that can dynamically function as standard cache, data buffers, or bit-serial compute units, thereby eliminating the large, dedicated buffers typical of standalone Neural Processing Units (NPUs).
2. A practical cache coherence strategy that allows for the safe partitioning and isolation of cache ways for computation without halting the host processor, a concept they term "processor-safe PIC."
3. A complete end-to-end implementation and verification on an FPGA, including custom ISA extensions, a C-level software stack, and an evaluation that demonstrates significant gains in area and energy efficiency for AI inference workloads.

In essence, this work presents a compelling blueprint for transforming the LLC from a passive memory component into an active, efficient, and reusable compute resource for edge AI.

Strengths

1. Excellent System-Level Synthesis: The primary strength of this paper is its successful synthesis of ideas from multiple domains—processing-in-memory, bit-serial accelerators, and cache management—into a single, coherent, and demonstrably working system. Many prior works have focused on novel PIC circuits or high-level concepts, but this paper provides the crucial "glue" by addressing system-level challenges like coherence, programmability, and OS compatibility. The full-stack approach, from hardware RTL to a C-function library (Section 4.2), is a significant achievement and makes the contribution far more tangible and impactful.

2. The "Processor-Safe" Coherence Model: The proposed coherence strategy (detailed in Section 3.3 and Section 5) is a cornerstone of this work's practical significance. By enabling parts of the cache to be used for acceleration while the rest remains available to the host CPU, PolymorPIC overcomes a major limitation of earlier designs that often required commandeering the entire cache, stalling the processor. This allows for genuine parallelism between general-purpose tasks and AI acceleration (as shown in Figure 22, page 13), which is critical for responsive AIoT devices.

3. Elegant Architectural Unification (HMAs): The concept of Homogeneous Memory Arrays (HMAs) is an elegant solution to the area-inefficiency of specialized accelerators. As shown conceptually in Figure 4 (page 4), unifying the functions of compute engine, local/global buffer, and standard cache into a single, reconfigurable SRAM structure is the key enabler for the impressive area efficiency reported. This hardware is not idle during non-AI workloads; it simply serves its primary function as an LLC, making it a highly cost-effective design point.

4. Comprehensive and Rigorous Evaluation: The evaluation in Section 7 is thorough and well-contextualized. The authors compare PolymorPIC against a wide spectrum of relevant designs, including CPU-only, vector processors (Hwacha), mainstream NPUs (Gemmini), standalone bit-serial accelerators (ANT, BBS), and other PIC designs (MAICC, Duality Cache). This broad comparison effectively situates their work in the current landscape and provides convincing evidence for their claims of superior area and energy efficiency for edge-class SoCs.

Weaknesses

While this is a strong paper, there are areas where the positioning and future implications could be discussed more deeply.

1. Trade-off Between Efficiency and Peak Throughput: The results, particularly in Figure 16 (page 11), show that while PolymorPIC is highly efficient, its absolute throughput is lower than some dedicated, highly-optimized NPU designs like BBS. This is an expected and perfectly acceptable trade-off—the core value proposition is efficiency, not raw performance leadership. However, the paper could benefit from a more explicit discussion of this trade-off and the specific application domains where PolymorPIC's design point (maximum efficiency for a given area/power budget) is more valuable than maximum possible throughput.

2. Scalability to Multi-Core Systems: The paper presents a compelling solution for a single-core SoC. The coherence model works cleanly by partitioning ways in the L2 cache, which is private or last-level for that core. The challenges would multiply in a multi-core system sharing a last-level PIC-enabled cache (e.g., L3). Issues of inter-core synchronization, managing PIC resource allocation between cores, and maintaining coherence for shared data would become significantly more complex. While beyond the scope of this paper's implementation, a brief discussion of these future challenges would strengthen its contextualization.

Questions to Address In Rebuttal

1. Positioning vs. Throughput-Optimized NPUs: The paper convincingly demonstrates superior area and energy efficiency. Could you please explicitly frame the application space for PolymorPIC in contrast to throughput-focused designs like BBS? Is the target primarily area- and power-constrained edge devices where "good enough" performance with maximum efficiency is the goal, rather than applications requiring the highest possible inference rate?

2. Path to Multi-Core Scalability: The "processor-safe" coherence mechanism is a key strength for the presented single-core system. Could you elaborate on the primary challenges and your envisioned architectural solutions for extending this model to a multi-core SoC where multiple cores share the PolymorPIC-enabled LLC? For instance, how would requests for PIC allocation from different cores be arbitrated, and how would you manage snoop traffic related to the PIC-isolated ways?

3. Developer Experience and Programmability: The software stack (Section 4.2.2, page 6) is a vital part of the full-stack claim. From a programmer's perspective, how much manual optimization is required to efficiently map a new NN model onto PolymorPIC? Specifically, how does the effort of managing data tiling, choosing a dataflow (I/WS vs. OS), and configuring the Mat-CUs/Mat-SBs compare to using a more conventional NPU with a mature compiler toolchain that automates much of this process?

---

Review Form

Reviewer: The Innovator (Novelty Specialist)

Summary

The paper presents "PolymorPIC," a polymorphic processing-in-cache (PIC) architecture integrated into the last-level cache (LLC) of a RISC-V System-on-Chip (SoC). The stated goal is to accelerate neural network inference, specifically bit-serial matrix multiplication (BSMM), with high area and energy efficiency. The core architectural proposal is a "Homogeneous Memory Array" (HMA) concept, where standard SRAM arrays can be dynamically reconfigured at runtime to function as a normal cache, a computational unit (Mat-CU), or a data buffer (Mat-SB). To enable this dynamic switching in a multi-tasking environment, the authors propose a specific coherence strategy involving a "Direct Cache Flush" (DCF) for rapid allocation and "PIC Array Isolation" (PAI) for processor-safe execution. The authors claim this is the first full-stack, end-to-end verified PIC-enabled SoC that can run a full operating system.

Strengths

The primary novelty of this work lies not in a single groundbreaking idea, but in the specific synthesis of several known concepts into a coherent, practical, and fully-realized system.

1. Architectural Novelty in Practicality: While PIC is a well-explored field, many seminal works rely on custom SRAM bit-cells (e.g., Neural Cache [25], Duality Cache [29]) which suffer from poor area density and portability. The core novel contribution of PolymorPIC is the architectural design that enables BSMM acceleration using standard, compiler-generated SRAM arrays (Section 3.2, page 4). This HMA concept, which repurposes entire digital sub-arrays for compute or buffering, represents a significant step towards making PIC architectures manufacturable and integrable with standard design flows. The "delta" over prior art here is the move from bit-cell-level modification to array-level functional polymorphism.

2. System-Level Coherence as a Novel Contribution: Most prior PIC literature focuses on the accelerator microarchitecture and often hand-waves the complexities of system integration. This paper’s explicit focus on a "processor-safe" coherence mechanism is a novel contribution in the context of PIC design. While the constituent ideas (direct flushing via way/set ID, using a directory bit for way isolation) are not fundamentally new concepts in cache design, their specific application and combination to solve the rapid, safe mode-switching problem for PIC is new. The PAI mechanism (Section 5.2, page 7), in particular, provides an elegant, low-overhead solution to a critical system-level problem that has been a barrier to the practical deployment of PIC.

3. Demonstration of a Full-Stack System: The claim of being the "first full-stack PIC-enabled SoC, whose functionality is successfully end-to-end verified on a validation platform with an operating system running" (Section 2, page 2) is a substantial novelty claim. Many academic accelerators are evaluated in simulation or as standalone hardware kernels. Demonstrating a PIC architecture that coexists with an operating system on a RISC-V core, handles virtual memory, and can run SPEC benchmarks in parallel with PIC computations (Section 7.4, page 13) elevates this work from a pure architectural proposal to a demonstrated system. This end-to-end integration is a significant and novel engineering achievement in this domain.

Weaknesses

The paper's claims of novelty could be tempered by a more explicit acknowledgment of the conceptual heritage of its components.

1. Incremental Novelty of Individual Components: The paper presents its components as largely new inventions, whereas they are more accurately described as novel applications or optimizations of existing concepts.

   • Bit-Serial MM: This is a well-established technique for efficient NN acceleration, as acknowledged by the authors' citation of BISMO [63] and others. The novelty is purely in its implementation venue (the LLC).
   • PIC Array Isolation (PAI): This mechanism is functionally analogous to cache way-partitioning or way-locking, techniques that have existed for years to provide quality-of-service or security isolation in caches. PAI uses a directory bit to make a way "invisible" to the replacement policy; this is conceptually identical to how partitioning is often implemented. The novelty is the purpose (enabling safe in-cache computation) rather than the mechanism.
   • Direct Cache Flush (DCF): This is a logical optimization of a standard cache flush. Instead of iterating through memory addresses to find cache entries, it directly targets cache indices (wayID, setID). This is a straightforward microarchitectural enhancement, not a fundamental new coherence protocol. Its novelty is limited.

2. Overstated Terminology: The term "Homogeneous Memory Array" (HMA) is presented as a new architectural primitive. However, it could be argued that this is a new name for a reconfigurable SRAM macro-cell that has been augmented with minimal compute logic. The novelty lies in the specific configuration and control logic, not necessarily in the invention of a new class of memory array. The paper would be stronger if it positioned HMA as a specific, novel implementation of a reconfigurable memory architecture rather than a new fundamental concept.

3. Complexity vs. Benefit Justification: The paper demonstrates significant efficiency gains. However, the added complexity to the LLC—including MAC units, adders, registers, and control FSMs within each Mat (Figure 9, page 7)—is not trivial. This fundamentally alters the design and validation of what is typically a simple memory structure. While the results are compelling against an NPU like Gemmini, the comparison is with a specific type of NPU. The novelty of the trade-off (complex cache vs. dedicated accelerator) is interesting but may not be universally superior, and the paper should be careful not to overgeneralize its benefits. The true innovation is proving this trade-off point is viable, but the complexity cost should be more critically analyzed.

Questions to Address In Rebuttal

1. Please explicitly differentiate the proposed "PIC Array Isolation" (PAI) mechanism from prior art in cache way-partitioning and way-locking (e.g., Intel CAT, or academic proposals for security/QoS). Is the implementation fundamentally different, or is the novelty purely in its application to enable processor-safe PIC?

2. The "Homogeneous Memory Array" (HMA) is a central concept. Can the authors clarify how HMA differs conceptually from a standard SRAM array augmented with peripheral compute logic and a reconfigurable datapath? Is the homogeneity claim based on the reuse of the memory bit-cells themselves, or the uniform structure of the augmented Mats?

3. The work claims to be the "first full-stack" PIC-enabled SoC. To substantiate this novelty claim, could the authors detail one or two specific system-level challenges (e.g., related to the OS scheduler, memory management unit interactions beyond page table walking, or interrupt handling) that they had to solve for this integration, which were not addressed by prior PIC simulation-based studies?