Chasoň: Supporting Cross HBM Channel Data Migration to Enable Efficient Sparse Algebraic Acceleration

---

Review Form:

Reviewer: The Guardian (Adversarial Skeptic)

Summary

The paper proposes Chasoň, an HBM-based streaming accelerator for sparse matrix-vector multiplication (SpMV). The central contribution is a novel scheduling scheme, Cross-HBM Channel out-of-order Scheduling (CrHCS), designed to mitigate the processing element (PE) underutilization found in prior work like Serpens. CrHCS enables the migration of non-zero data elements across HBM channels to fill pipeline stalls that would otherwise occur. The authors implement Chasoň on an AMD Alveo U55C FPGA and evaluate it against Serpens, high-end GPUs, and a CPU, claiming significant improvements in performance and energy efficiency stemming from improved PE utilization.

Strengths

1. The paper correctly identifies a significant and well-known limitation in state-of-the-art HBM-based sparse accelerators: PE stalls due to workload imbalance across rows mapped to a single HBM channel. The motivation presented in Section 2 is clear and logically sound.
2. The proposed solution, migrating non-zeroes between channels, is a conceptually logical extension to the existing intra-channel scheduling schemes.
3. The experimental evaluation is broad, utilizing a large set of 800 matrices and comparing against multiple relevant platforms (FPGA, GPU, CPU).

Weaknesses

My primary concerns with this work center on the validity of its core assumptions, the fairness of its experimental comparison, and the analysis of its performance claims.

1. Unsubstantiated Core Scheduling Assumption: The entire efficacy of CrHCS rests on a critically optimistic and unproven assumption. In Section 3.3, the authors state, "Our experiments have shown that CrHCS never fails to find a RAW dependency-free value to migrate." This is an exceptionally strong claim presented without any supporting data, theoretical proof, or statistical analysis. The motivational example in Figure 2c conveniently shows a perfect pipeline with 0% underutilization, which is predicated on the constant availability of suitable non-zero candidates from the neighboring channel. This appears to be a best-case scenario presented as a general capability. The paper fails to characterize the conditions under which a suitable candidate would not be available, which would break the core premise of the work.

2. Flawed and Unfair Baseline Comparison: The comparison to the primary baseline, Serpens, is questionable.

   • The authors report their implementation of Serpens achieves only 223MHz, while Chasoň achieves 301MHz on the same U55C platform (Section 5.2). A ~35% frequency advantage for Chasoň is a major confounding variable. The authors attribute this to "reduced logic congestion," but this is a vague justification. A more rigorous analysis is required to prove that their port of Serpens is faithful and optimized, and not simply a poor implementation used to inflate Chasoň's relative performance.
   • The resource comparison in Table 1 shows that Chasoň consumes significantly more resources than Serpens (+58% LUT, +66% FF, +57% DSP). The authors dismiss the need for an iso-area comparison by asserting that Serpens could not benefit from more PEs (Section 4.5). This is an unsubstantiated claim and sidesteps a fundamental requirement for fair hardware accelerator comparisons. The authors are comparing a larger, more complex, and higher-frequency design to a smaller, simpler, and slower one. The performance gains are therefore not solely attributable to the CrHCS scheduling algorithm.

3. Misleading Attribution of Performance Gains: The paper attributes its speedup over Serpens to "data transfer reduction" (Figure 15, Section 6.2). This is a confusing and misleading framing. Both Chasoň and Serpens must process the exact same number of non-zero elements. The issue with Serpens is not that it "transfers zeros," but that its pipeline stalls for lack of available work. Chasoň avoids these stalls by transferring useful non-zero data from another channel. The gain comes from increased temporal utilization of the datapath (i.e., stall reduction), not from reducing the total volume of data transferred from HBM. This mischaracterization obscures the true mechanism of improvement and suggests a misunderstanding of the performance dynamics.

4. Introduction of New, Unanalyzed Bottlenecks: The architectural components required to support CrHCS (Router, Reduction Unit, Re-order Unit) introduce non-trivial latency and complexity. The authors' own analysis provides evidence of this. On page 12, they note that the C5 matrix, despite having a larger "data transfer reduction" factor, achieves lower speedup than the MY matrix because its larger column dimension creates contention and latency in the ScUGs and Reduction Unit. This is a critical finding: the overhead of the proposed architecture can itself become the primary performance bottleneck, potentially negating the benefits of CrHCS. This trade-off is not systematically explored or quantified.

Questions to Address In Rebuttal

1. Regarding the core assumption of CrHCS: Please provide statistical data from your 800-matrix evaluation that quantifies the probability of finding a RAW-dependency-free non-zero element in the neighboring channel for any given stall cycle. How does this probability change with matrix structure and sparsity? On what basis can the authors claim this "never fails"?

2. Regarding the baseline comparison: Please provide a detailed justification for the 35% frequency advantage over your implementation of Serpens. Furthermore, please provide a detailed argument for why a rigorous iso-resource comparison is not necessary, or alternatively, provide such a comparison. For instance, what would the performance of Serpens be if it were scaled to consume the same resources as Chasoň?

3. Regarding performance attribution: Please clarify the "data transfer reduction" claim. Is it not more accurate to state that the improvement comes from reducing pipeline stall cycles? Please re-frame the argument in terms of pipeline occupancy or throughput.

4. Regarding new bottlenecks: The paper admits that the reduction and re-ordering logic can become a bottleneck (C5 vs. MY matrix example). Please provide a sensitivity analysis that characterizes how the latency of these new architectural units impacts the overall speedup as a function of matrix properties (e.g., number of columns, non-zero distribution). At what point do the overheads of Chasoň's architecture outweigh the benefits of stall reduction?

---

Review Form

Reviewer: The Synthesizer (Contextual Analyst)

Summary

This paper introduces Chasoň, an HBM-based streaming accelerator for Sparse Matrix-Vector Multiplication (SpMV). The central contribution is a novel out-of-order scheduling scheme, "Cross-HBM Channel OoO Scheduling" (CrHCS), which enables dynamic data migration across HBM channels. The authors identify a key limitation in state-of-the-art accelerators like Serpens: while they parallelize work across HBM channels, their scheduling is confined within a channel ("intra-channel"). This leads to significant PE underutilization when the matrix rows assigned to a particular channel are sparse.

Chasoň's CrHCS addresses this by allowing a PE group associated with one channel to "borrow" non-zero elements from a neighboring channel's data stream, effectively filling its own pipeline stalls. The paper presents the necessary architectural support for this scheme, including data tagging, a routing mechanism within the PEs, and dedicated on-chip memory structures to manage results from both "private" and "shared" (migrated) data. The authors implement Chasoň on an AMD Alveo U55C FPGA and demonstrate significant improvements in PE utilization, leading to substantial performance and energy efficiency gains over Serpens, as well as modern GPU and CPU platforms.

Strengths

This is a strong paper with a clear, compelling, and well-executed core idea.

1. Excellent Problem Formulation and Motivation: The paper does an outstanding job of contextualizing its contribution. It clearly explains the evolution from row-based to PE-aware scheduling (Section 2.2, pages 2-3) and uses a simple, effective diagram (Figure 2, page 3) to illustrate the limitations of the current state-of-the-art. The motivation is powerfully reinforced by Figure 3 (page 4), which quantifies the PE underutilization problem across a large dataset, showing that ~70% underutilization is common. This makes the need for a better solution undeniable.

2. A Novel and Elegant Core Concept: The idea of inter-channel data migration is a genuinely new perspective in the design space of HBM-based sparse accelerators. The prevailing wisdom has been to treat the HBM channels as independent data streams feeding isolated processing element groups (PEGs). Chasoň cleverly breaks this isolation to solve the fundamental problem of load imbalance. This is a microarchitectural solution to a classic parallel computing challenge, and it feels both innovative and intuitive. It's a natural evolution of out-of-order principles applied at a coarser, inter-channel granularity.

3. Thorough and Credible Implementation: This is not merely a conceptual or simulation-based study. The authors provide a detailed architectural design (Section 4, pages 5-7) and a full implementation on a modern FPGA platform. The design considers the practical details required to make CrHCS work correctly, such as the pvt and PE_src flags for data provenance and the Rearrange Unit to ensure final results are correctly ordered. Achieving a 301MHz clock frequency on the Alveo U55C with this level of complexity lends significant credibility to the work.

4. Strong Connection to the Broader Landscape: The work is well-positioned within the literature. It correctly identifies Serpens [71], Sextans [72], and other HBM-based accelerators as its direct lineage. By building upon and significantly improving a known, open-source architecture (Serpens), the authors make their contribution easy to understand and appreciate. The principle of dynamically sharing work to smooth out irregularities connects to broader themes in parallel processing, systolic array design (e.g., [35]), and even processing-in-memory concepts where data locality and movement are paramount.

Weaknesses

The weaknesses are minor and relate more to exploring the full design space of the proposed idea rather than fundamental flaws in the presented work.

1. Limited Exploration of Migration Policies: The implementation restricts data migration to only the immediate next channel (as mentioned in Section 6.1, page 10). While justified by on-chip resource constraints, this feels somewhat arbitrary from a conceptual standpoint. A richer discussion on the design space of migration policies would strengthen the paper. For instance, what are the trade-offs of bidirectional migration (borrowing from channel i-1 and channel i+1), or a wider but shallower "neighborhood" of channels? This seems like a critical design parameter that is fixed here without much exploration.

2. Interplay with Static Data Partitioning: The effectiveness of the dynamic migration scheme is likely coupled with the initial static assignment of matrix rows to HBM channels. The paper does not discuss this relationship. For example, if a "poor" static partitioning creates a pattern where a sparse channel is always followed by another sparse channel, the "next-channel" policy would be ineffective. A brief analysis of this interplay would add depth.

3. Analysis of Latency Overhead: While the paper successfully argues that reducing stalls and data transfers yields a large net performance gain, the latency overhead of the new architectural units (Router, Reduction Unit, Reorder Unit) is not explicitly quantified. The authors state these units are "fundamental" (Section 6.2.2, page 12), but a more direct analysis—perhaps in terms of additional pipeline stages or critical path impact—would provide a more complete picture of the architectural trade-offs.

Questions to Address In Rebuttal

1. The current CrHCS implementation limits data migration to the immediate "next" channel in a unidirectional manner. Could the authors elaborate on the design trade-offs here? Would a bidirectional (i.e., borrowing from both previous and next channels) or a wider neighborhood migration scheme be feasible, and what would be the expected impact on resource usage (especially URAMs) and performance?

2. How sensitive is the proposed scheme to the initial static mapping of matrix rows to HBM channels? For example, if the workload distribution across channels is highly correlated (e.g., alternating blocks of dense and sparse regions), could the fixed "next channel" migration policy become a bottleneck?

3. The paper's core idea seems highly generalizable. Beyond the successful application to SpMV, have the authors considered its potential for other sparse kernels that are often accelerated on HBM platforms, such as SpMM (Sparse-Matrix Dense-Matrix Multiplication) or SpGEMM (Sparse-Matrix Sparse-Matrix Multiplication), where load imbalance across parallel units can be even more severe? (The authors briefly touch on SpMM in Section 7.1, but I would be interested to hear more on the potential challenges).

---

Review Form

Reviewer: The Innovator (Novelty Specialist)

Summary

The paper introduces "Chasoň," an HBM-based streaming accelerator for sparse matrix-vector multiplication (SpMV). The central claim to novelty lies in its non-zero scheduling scheme, "Cross-HBM Channel out-of-order Scheduling" (CrHCS). This scheme is designed to address the problem of processing element (PE) underutilization, which the authors identify as a key weakness in prior art like Serpens [71]. Where Serpens and similar works use intra-channel out-of-order scheduling, CrHCS extends this to inter-channel scheduling. This is achieved by allowing non-zero data elements scheduled for one HBM channel to be migrated to an adjacent channel to fill stalls in its data stream. To support this, the authors propose an architecture with specific hardware for routing, re-ordering, and reducing partial sums originating from these migrated, or "shared," data elements.

Strengths

1. Clear Articulation of the Novel "Delta": The paper does an excellent job of isolating its contribution relative to the immediate state-of-the-art. It correctly identifies that HBM-based accelerators like Serpens [71] and Sextans [72] treat HBM channels as independent data streams feeding dedicated PE groups. The core novel idea of Chasoň is to break this strict channel-to-PEG isolation at the data scheduling level. The shift from an intra-channel to an inter-channel scheduling scope is a specific and well-defined advancement.

2. Novel Application of a Known Concept: The underlying principle of CrHCS is a form of work stealing or dynamic load balancing—transferring work from a source with available tasks to an idle resource. While work stealing is a classic concept in parallel computing, its application in this specific context is novel. The authors have adapted this general principle to the unique constraints of a streaming HBM-FPGA architecture, where data is pre-scheduled into fixed-width streams. This is not a trivial application and represents a new approach for this class of accelerators.

3. Enabling Architectural and Data Structure Novelty: To realize the CrHCS scheme, the authors introduce necessary and novel modifications. The inclusion of the pvt (private) and PE_src (source PE) flags in the 64-bit data element (Section 3.2, page 4) is a simple but clever mechanism to handle the bookkeeping of migrated data without requiring complex control logic. The architectural additions—specifically the Router within the PE and the system-level Reduction and Re-order Units (Section 4, pages 5-7)—are a direct and logical consequence of the scheduling scheme and are new in their composition and purpose for this problem.

Weaknesses

1. Conceptual vs. Foundational Novelty: While the application of inter-channel data migration is novel for HBM-based sparse accelerators, the paper could be stronger by explicitly positioning CrHCS within the broader literature of work stealing and dynamic load balancing. The current framing presents the idea as wholly new, whereas its foundational concept is well-established. Acknowledging this and then detailing why their implementation is non-trivial and unique to the HBM architecture would more accurately place the contribution.

2. Incremental Nature of Architectural Components: The hardware units proposed to support CrHCS (Router, Reduction Unit, Re-order Unit) are, in isolation, standard digital design building blocks. The Router is a multiplexer-based structure, the Reduction Unit is an adder tree, and the Re-order Unit is a form of sorter or stream aligner. The novelty is not in the components themselves, but entirely in their synthesis to support the new scheduling algorithm. The paper is clear about their function, but it's important to recognize that the architectural novelty is compositional, not elemental.

3. Limited Scope of the Novel Mechanism: The implementation of CrHCS is constrained to migrating data only from the immediate next channel (e.g., channel i+1 to i). The authors justify this by citing on-chip memory limitations on the target FPGA (Section 6.1, page 10). While this is a practical engineering decision, it significantly limits the generality and novelty of the proposed data migration scheme. A truly novel interconnection scheme might support more flexible migration patterns (e.g., any-to-any, or from the two nearest neighbors). The current implementation is a 1D, unidirectional work-stealing approach, which is the simplest possible form of inter-resource cooperation.

Questions to Address In Rebuttal

1. On the Generality of CrHCS: The decision to migrate data only from the immediate next channel appears pragmatic but limits the solution's generality. Could the authors elaborate on the architectural complexity (e.g., interconnect, URAM scaling for partial sums, and Re-order Unit complexity) of a more flexible "any-to-any" or "bi-directional nearest-neighbor" channel migration scheme? What is the anticipated crossover point where the hardware overhead and control complexity would outweigh the benefits of filling more stalls?

2. Comparison to Classic Work Stealing: Please contrast CrHCS with classic work-stealing algorithms (e.g., Cilk). What specific constraints of the HBM/FPGA streaming architecture (e.g., fixed AXI widths, offline scheduling, lack of a shared task queue) prevent a more traditional implementation, and how does CrHCS's offline, stream-injection approach uniquely address them?

3. Overhead of the Novel Scheduler: The paper focuses on the hardware implementation, but CrHCS requires a more sophisticated offline scheduler compared to the simpler round-robin assignment in Serpens. What is the computational overhead of this new scheduling step? While this cost is likely amortized over many runs, a quantification would help assess if the novelty in scheduling introduces a new, non-trivial bottleneck in the overall workflow before execution can even begin.