Characterizing and Optimizing Realistic Workloads on a Commercial Compute-in-SRAM Device

Review Form

Reviewer: The Guardian (Adversarial Skeptic)

Summary

The authors present a performance characterization of a commercial compute-in-SRAM device, the GSI APU. They propose an analytical performance model and three optimization strategies (communication-aware reduction mapping, coalesced DMA, broadcast-friendly layouts) aimed at mitigating data movement bottlenecks. These are evaluated on the Phoenix benchmark suite and a Retrieval-Augmented Generation (RAG) workload. The central claims are that these optimizations yield significant speedups (e.g., up to 6.6x in RAG retrieval over CPU) and that the system can match the RAG performance of an NVIDIA A6000 GPU while being substantially more energy efficient. However, the work's conclusions rest on a precarious foundation of a partially simulated system, questionable baseline comparisons, and optimizations that may not generalize beyond the target device's idiosyncratic architecture.

Strengths

1. Use of Commercial Hardware: The study is grounded in a real, commercial compute-in-SRAM device. This is a commendable departure from the purely simulation-based or small-prototype studies that are common in this field, providing valuable data on the characteristics and challenges of a production system.
2. Relevant Workload: The choice of Retrieval-Augmented Generation (RAG) is timely and complex. It stresses memory bandwidth, compute, and data layout, making it a suitable test case for evaluating the claimed benefits of compute-in-SRAM architectures.
3. Systematic Optimization Breakdown: The paper does a clear job of isolating and evaluating its proposed optimizations. Figure 12 provides a lucid breakdown of how each optimization contributes to the final performance of the binary matrix multiplication kernel, which helps in understanding their individual effects.

Weaknesses

1. The RAG Evaluation is Fundamentally Flawed by a Hybrid Real-Simulated Methodology: The paper's headline claim regarding RAG performance matching an A6000 GPU is severely undermined by the experimental setup. The authors concede in Section 5.3.1 (Page 11) that the GSI APU's actual DDR bandwidth (23.8 GB/s) would be a "bottleneck." To circumvent this, they model the off-chip memory using a simulated HBM2e system. This is a critical methodological flaw. The paper is no longer "Characterizing... a Commercial... Device" but rather characterizing a hypothetical system that does not exist. The performance and energy results for the most significant workload (RAG) are therefore speculative. The interaction latency and energy between the real APU's memory controller and the simulated HBM are not detailed, leaving the accuracy of this hybrid model in serious doubt.

2. Insufficient Detail and Rigor in Baseline Comparisons: The claims of superiority depend entirely on the fairness of the baselines, which are not sufficiently established.

   • GPU Baseline: The paper claims to match the performance of an NVIDIA A6000 GPU on RAG. However, there is a stark lack of detail on the GPU implementation. Was an optimized library like FAISS-GPU used? If so, which index was employed? A brute-force inner product search on a high-end GPU can be heavily optimized with CUDA. Without a detailed description of the GPU software configuration and optimization level, the claim of "matching performance" is unsubstantiated. It is possible the presented APU system is being compared against a suboptimal GPU implementation.
   • CPU Baseline: While the use of FAISS with AVX512 and OpenMP (Section 5.3.2, Page 11) is a respectable starting point, the term "optimized CPU baseline" is used without detailing what specific optimizations were performed beyond using the library as-is.

3. Overstated Generality of Optimizations: The paper presents its three optimizations as general principles for compute-in-SRAM. However, their efficacy appears deeply tied to the unique and arguably peculiar architecture of the GSI APU.

   • The core "communication-aware reduction mapping" optimization hinges on the fact that intra-VR reductions are significantly more expensive than inter-VR reductions on this specific device (Section 2.1.2, Page 4). This is a microarchitectural artifact of the GSI APU's ultra-long vector design, not a fundamental property of all C-SRAM systems.
   • Similarly, coalescing DMA via "subgroup copy" (Section 4.3, Page 8) relies on a specific hardware feature.
   • Consequently, the paper demonstrates clever engineering for one specific device but fails to provide convincing evidence that these are broadly applicable architectural principles. The conclusions are over-generalized from a single, atypical data point.

4. The Analytical Framework Lacks True Predictive Power: The proposed analytical framework (Section 3, Page 5) appears to be more of an empirical curve-fitting exercise than a first-principles model.

   • Equation 1, which models the latency of subgroup reductions, is a cubic polynomial whose coefficients are themselves logarithmic functions of group size. The justification for this specific functional form is superficial ("multi-level shifting, alignment, and accumulation"). This is an observation, not an explanation. An insightful model would derive this complexity from architectural primitives.
   • The model is validated in Table 7 by showing it can reproduce the performance of the same device from which its parameters were measured. This demonstrates descriptive accuracy but provides no evidence for its claimed utility in "architectural design space exploration" (Section 3.1, Page 5), which requires predictive power for architectures with different parameters.

Questions to Address In Rebuttal

1. Please provide a rigorous justification for using a simulated HBM memory system for the RAG evaluation. How can the paper's central claims about RAG performance and energy on a "commercial device" be considered valid when the most critical system component for this memory-bound problem is hypothetical? Provide details on how the simulation was integrated with the real hardware to ensure model fidelity.

2. Provide explicit details of the GPU software stack used for the RAG baseline. Specify the exact library (e.g., FAISS-GPU), version, index type (e.g., IndexFlatIP), and any custom CUDA kernel development or tuning performed. This is essential to validate the claim of matching GPU performance.

3. Elaborate on how the proposed optimizations, particularly the reliance on the cost disparity between intra-VR and inter-VR operations, can be generalized to other compute-in-SRAM architectures (e.g., bit-serial, associative, or different vector lengths) that do not share the GSI APU's specific microarchitecture.

4. Provide a more fundamental, first-principles derivation for the cubic complexity of the reduction cost model in Equation 1. What specific sequence of micro-operations leads to this complexity, and why should we expect this to hold for other designs?

---

Review Form

Reviewer: The Synthesizer (Contextual Analyst)

Summary

This paper presents a comprehensive characterization and optimization study of a commercial Compute-in-SRAM (CiSRAM) accelerator, the GSI APU, on realistic, data-intensive workloads. The authors' core contribution is bridging the significant gap between the theoretical promise of CiSRAM architectures, which have largely been studied via simulation, and their practical viability. They achieve this through a three-pronged approach: (1) developing an analytical performance model to expose architectural trade-offs, (2) proposing a set of architecture-aware optimizations focused on data layout and movement, and (3) demonstrating the system's effectiveness on a modern, high-impact workload—Retrieval-Augmented Generation (RAG) for large language models. The key result is that their optimized CiSRAM system can match the retrieval latency of a high-end NVIDIA A6000 GPU while consuming over 50x less energy, providing a compelling real-world data point for the future of memory-centric computing.

Strengths

This is an excellent and timely paper that the community should pay close attention to. Its primary strengths are:

1. Grounded in Reality with a Commercial Device: The most significant strength of this work is its use of a real, commercial CiSRAM chip. For years, the architecture community has seen promising simulation-based results for compute-in-memory (e.g., CAPE [11], Compute Caches [1]). This paper provides a much-needed anchor to reality, revealing the practical challenges (like the asymmetry between inter- and intra-vector operations, as discussed in Section 2.1) and immense potential of these architectures. This is a crucial step in maturing the field from academic exploration to practical system design.

2. High-Impact and Well-Chosen Workload: The choice to focus on Retrieval-Augmented Generation (RAG) is superb. RAG is a critical component of modern AI systems, and its core, the nearest-neighbor search, is fundamentally a data-bound problem—a perfect match for a memory-centric accelerator. By connecting CiSRAM to the LLM ecosystem, the authors make their work immediately relevant to one of the most active areas of research and industry today. The end-to-end evaluation in Section 5.3 (page 11) is particularly compelling.

3. Systematic and Principled Optimization Strategy: The authors don't simply present a heroic hand-tuned result. They build a case for their optimizations systematically. The analytical framework (Section 3, page 5) provides a clear model for reasoning about performance, and the three proposed optimizations (Communication-Aware Reduction Mapping, DMA Coalescing, and Broadcast-Friendly Layouts in Section 4, page 7) directly address the key bottlenecks identified in the architecture. The breakdown in Figure 12 (page 10) clearly illustrates the contribution of each optimization, which is excellent scientific practice.

4. Exceptional Energy Efficiency Results: The headline result—matching an A6000 GPU's performance on RAG retrieval with 54.4×–117.9× lower energy consumption (Section 5.3.5, page 12)—is staggering. This isn't an incremental improvement; it's a step-function change in efficiency. This single result provides a powerful argument for pursuing specialized CiSRAM hardware for data-intensive search and comparison tasks, especially in power-constrained environments like the edge.

5. Strong Potential for Broader Impact: This paper serves as a foundational case study for a whole class of emerging architectures. The lessons learned about data layout, communication patterns, and the programming model are likely to be applicable to future CiSRAM and PIM designs. It essentially provides a "playbook" for how to extract performance from these unconventional systems.

Weaknesses

While this is a strong paper, there are areas where its context and implications could be explored further. My points are not meant to detract from the quality of the work but rather to frame its limitations and suggest avenues for future inquiry.

1. Generalizability of the Optimizations: The proposed optimizations are highly effective but also highly tailored to the specific microarchitecture of the GSI APU—namely, its extremely long vector registers (32K elements) and the significant performance delta between inter- and intra-VR operations. It is not immediately clear how these specific techniques would translate to other CiSRAM designs that might feature different vector lengths, memory bank organizations, or reduction network capabilities. The contribution could be strengthened by a discussion on the principles that would generalize versus the implementation details that are device-specific.

2. Reliance on a Simulated Memory System for RAG: The authors are transparent about modeling the off-chip memory system (HBM2e) with Ramulator for the RAG experiments (Section 5.3.1, page 11). While this is a reasonable and necessary choice to avoid having the low-end DDR on the evaluation board become an unfair bottleneck, it does mean the end-to-end results are a hybrid of real measurement and simulation. This is a minor weakness, but it's important to acknowledge that the system-level performance is projected, not fully measured.

3. Programmability Remains a Major Hurdle: The paper demonstrates what is possible with careful, expert-driven optimization. However, it implicitly highlights the immense programmability challenge of such architectures. The required transformations (e.g., redesigning data layouts for broadcasting, as shown in Figure 11 on page 10) are non-trivial and seem unlikely to be discovered by a conventional compiler. The paper would benefit from a discussion of the path from this manual effort to a more accessible programming model.

Questions to Address In Rebuttal

I am strongly in favor of accepting this paper. The following questions are intended to help the authors strengthen the final version and to clarify the broader context of their work.

1. On Generalizability: Your analytical framework and optimizations are deeply tied to the GSI APU's architecture. Could you elaborate on which parts of your framework you believe are fundamental to most CiSRAM vector architectures, and which are specific to the GSI device? For instance, if a future CiSRAM device had hardware support for efficient intra-vector reductions, how would your optimization strategy change?

2. On the Path to Automation: The optimizations presented required significant manual effort and deep architectural knowledge. What do you see as the key challenges and opportunities in building a compiler or library ecosystem that could automate these data layout and loop mapping transformations for CiSRAM targets, making them accessible to non-expert programmers?

3. On Future Workloads: Based on your deep characterization of this device, what other application domains beyond RAG and the Phoenix suite do you believe are the most promising "killer apps" for this style of architecture? Specifically, what workload characteristics (e.g., data types, memory access patterns, compute kernels) make an application a good or bad fit for this platform?

Review Form

Reviewer: The Innovator (Novelty Specialist)

Summary

This paper presents a performance characterization and optimization study of the GSI APU, a commercial Compute-in-SRAM (CiS) device. The authors make three primary claims of novelty: (1) a comprehensive evaluation of this device on realistic, large-scale workloads (Phoenix benchmarks, Retrieval-Augmented Generation); (2) an analytical performance framework for this class of architecture; and (3) a set of three optimizations—communication-aware reduction mapping, coalesced DMA, and broadcast-friendly data layouts—that significantly improve performance.

My analysis concludes that the primary novel contribution of this work lies in the first claim: the rigorous, end-to-end experimental characterization of a commercial CiS accelerator on complex, modern workloads. This provides valuable, and to my knowledge, first-of-its-kind data on the practical viability of such architectures. However, the secondary claims regarding the novelty of the analytical framework and, most notably, the proposed optimizations are significantly overstated. These optimizations are direct applications of well-established, decades-old principles from the field of parallel computing, particularly from the GPU domain. The contribution is in the application and tuning of these principles to a new hardware target, not in the invention of the principles themselves.

Strengths

The core strength and genuine novelty of this paper is its experimental contribution. While prior work has microbenchmarked the GSI APU on smaller kernels ([18], [19], [33]), this paper is the first I am aware of to conduct a thorough, end-to-end evaluation on workloads as complex and data-intensive as RAG over 200GB corpora. This is not a simulation or a small prototype evaluation; it is an empirical study on commercial hardware. The findings, such as matching an NVIDIA A6000 GPU in RAG latency while using over 50x less energy (Section 5.3.5, page 12), provide a critical data point for the architecture community on the potential of CiS. This characterization is a valuable and novel contribution.

Weaknesses

My primary concern is the lack of novelty in the "proposed optimizations" detailed in Section 4 (page 7). The paper presents these as new contributions, but they are functionally and conceptually analogous to standard optimization techniques for parallel architectures.

1. Communication-Aware Reduction Mapping: The core idea presented in Section 4.2 is to map a reduction from an expensive communication domain (intra-VR spatial reduction) to a cheaper one (inter-VR temporal reduction via element-wise operations). This is a fundamental principle of parallel algorithm design. For any architecture with a non-uniform memory/communication hierarchy, programmers and compilers seek to map computation to minimize costly data movement. This is conceptually identical to optimizing reductions on a GPU by favoring warp-level or shared-memory-based reductions over more expensive global atomic operations. The principle is not new; its application to the APU's specific VR structure is an implementation detail.

2. Coalesced DMA: The technique described in Section 4.3 and depicted in Figure 10 is a direct parallel to "coalesced memory access," a foundational optimization for GPUs since their earliest programming models. The goal of combining multiple small, disparate memory accesses into a single, large, contiguous transaction to maximize memory bus utilization is textbook parallel programming. The paper even uses the standard term "coalescing." While the use of the APU's subgroup copy primitive is specific to this hardware, the underlying optimization strategy is not novel.

3. Broadcast-Friendly Data Layout: The transformation described in Section 4.4, which reorganizes data to make elements accessed together contiguous in memory (Figure 11), is a classic data layout optimization. This is analogous to Array-of-Structs (AoS) vs. Struct-of-Arrays (SoA) transformations used to optimize for SIMD/SIMT execution. The goal is to align the data structure in memory with the hardware's natural access granularity. Again, this is a well-known principle, not a new one.

The analytical framework (Section 3) is a useful engineering contribution for modeling this specific device. However, the methodology—profiling latencies of primitive operations (Tables 4 and 5) and composing them into a performance model—is a standard approach for building bottom-up performance estimators. It does not represent a novel modeling paradigm.

Questions to Address In Rebuttal

1. The paper presents three core optimizations as novel contributions. Please explicitly differentiate these from their well-established analogues in the parallel computing literature, particularly GPU optimizations (e.g., hierarchical reduction strategies, coalesced memory access, and AoS/SoA data layout transformations). What, precisely, is the novel conceptual leap beyond applying known principles to a new microarchitecture?

2. In Section 4.2, the concept of mapping a spatial reduction to a temporal one is described. This sounds like a form of loop transformation to change the order of operations and improve data locality. Could the authors frame this contribution in the context of established compiler transformation theory and clarify what makes their mapping scheme fundamentally new?

3. Regarding the analytical framework, is the claimed novelty in the methodology of building the model, or in the specific model parameters derived for the GSI APU? If the latter, the contribution should be framed as a specific device model rather than a general framework.