Coruscant: Co-Designing GPU Kernel and Sparse Tensor Core to Advocate Unstructured Sparsity in Efficient LLM Inference

---

Review Form

Reviewer: The Guardian (Adversarial Skeptic)

Summary

The authors present Coruscant, a co-designed software kernel and hardware architecture extension aimed at accelerating LLM inference by exploiting unstructured sparsity in the 30-70% range. The work is composed of three main contributions: (1) a bitmap-based sparse format designed to improve compression ratios over existing formats like CSR in this moderate sparsity regime; (2) a corresponding GPU SpMM kernel that uses this format to reduce data movement; and (3) a proposed hardware modification to the Tensor Core, the "Coruscant Sparse Tensor Core," which integrates a "Bitmap Decoder" to operate directly on the compressed format. The authors claim significant speedups over cuBLAS and Flash-LLM, and further gains with their proposed hardware.

Strengths

1. Problem Motivation: The paper correctly identifies a critical gap in the literature. While many works focus on extreme sparsity (>90%) or rigidly structured sparsity (2:4), the moderate, unstructured sparsity regime (30-70%) common in accuracy-preserving LLM pruning is underserved by existing hardware and software. Figure 1 provides a standard but clear motivation for focusing on the memory-bound nature of the decode stage.

2. Identification of Format Weakness: The analysis in Section 2.1 and Figure 3 effectively demonstrates that conventional index-based sparse formats (CSR, COO) are inefficient and can even lead to memory expansion at the target sparsity levels. This provides a solid rationale for exploring alternative representations like bitmaps.

Weaknesses

My analysis reveals several significant methodological and analytical flaws that call the paper's central claims into question.

1. Highly Questionable Hardware Simulation Methodology: The evaluation of the proposed Coruscant Sparse Tensor Core (STC) is fundamentally weak. In Section 4, the authors state they "simulate the behavior...by removing bitmap decoding and shared memory access writes...and retaining the standard HMMA instructions." This is not a simulation; it is a simplistic analytical projection. This approach completely ignores potential microarchitectural side effects, such as pipeline stalls caused by the new decoder logic, increased register file pressure, or contention on operand buses. It assumes the proposed "Bitmap Decoder" is a zero-cost, zero-latency oracle. Consequently, all performance results for the Coruscant STC (e.g., in Figures 11b, 15, 19) are speculative at best and likely overly optimistic.

2. Inadequate Comparison to Hardware-Accelerated Baselines: The comparison against NVIDIA's 2:4 sparsity in Section 5.3.4 and Figure 21 is telling. The authors' software kernel is slower than the 2:4 kernel, and even their proposed hardware (Coruscant STC) only "nearly matches" the 2:4 kernel's latency at 50% sparsity. They attribute the baseline's advantage to higher warp occupancy, but this is not an excuse—it is a critical performance factor. Their proposed dataflow and kernel implementation appear to be less efficient at utilizing the GPU's resources than the industry standard. The paper fails to make a convincing case for unstructured sparsity when it cannot clearly outperform the existing, hardware-accelerated structured sparsity solution at its home turf of 50% sparsity.

3. Discrepancy Between Theoretical and Actual Compression: There is a clear disconnect between the ideal compression ratios presented in Figure 3 and the actual memory footprint reduction achieved by the kernel, as discussed in Section 5.2.2. The authors admit this is due to padding added "to fully utilize the GPU memory bandwidth." This overhead is non-trivial and weakens the core premise that their format leads to superior memory efficiency in practice. The paper lacks a rigorous quantification of this padding overhead across different matrix sizes and sparsities, making the real-world benefit of their format ambiguous.

4. Superficial Hardware Overhead Analysis: The area and power analysis in Section 5.3.2 and Table 2 is not credible. The authors report synthesizing the Bitmap Decoder in a 45nm process and then scaling the results to 7nm using generic equations from Stillmaker et al. [48]. This methodology is widely understood to be inaccurate for modern nodes, as it fails to account for the dominance of wire delay, leakage power, and complex physical design rules. A proper hardware analysis would require synthesis in a modern PDK or a detailed, bottom-up analysis of the circuit. The presented numbers appear to be a back-of-the-envelope calculation designed to minimize the perceived cost of their hardware addition.

5. Weak Link Between Performance Claims and Accuracy Motivation: The paper begins by arguing that unstructured sparsity is superior for maintaining model accuracy (Table 1, Figure 2). However, the core end-to-end evaluation in Section 5.1 and Figure 15 only reports performance metrics (tokens/sec). It is never explicitly shown that the models pruned to 30/50/70% for this performance evaluation actually maintain their accuracy. A fair comparison would require evaluating performance against a dense baseline of equivalent accuracy, which may involve techniques other than pruning. The paper implicitly compares a lower-accuracy pruned model to a full-accuracy dense model, which inflates the perceived efficiency gains.

Questions to Address In Rebuttal

The authors must provide clear and convincing responses to the following points:

1. On Hardware Methodology: Justify the use of an analytical model for the Coruscant STC performance evaluation. Provide evidence, such as from a detailed microarchitectural simulator (e.g., GPGPU-Sim), that your model accurately captures pipeline dynamics and that the Bitmap Decoder does not introduce new performance bottlenecks. Without this, the STC results are unsubstantiated.

2. On Baseline Performance: Explain, with detailed profiling data (e.g., from Nsight Compute), the precise reasons for the Coruscant kernel’s lower performance and resource utilization compared to the 2:4 cuSPARSELt kernel at 50% sparsity. Why should the community adopt a more complex unstructured approach if it fails to outperform the simpler structured alternative in a direct comparison?

3. On Compression Overhead: Provide a new table or figure that directly compares the "ideal" compression ratio of your format (from Figure 3) against the "actual" memory footprint of the kernel (including all padding and metadata overheads) for every data point presented in the evaluation (all sparsities, matrix sizes, and batch sizes).

4. On Accuracy in Evaluation: For the end-to-end results in Figure 15, what are the measured perplexity scores for the pruned Llama 2 models at 30%, 50%, and 70% sparsity? How do these scores compare to the dense baseline? The claimed speedups are meaningless without the context of the accuracy trade-off.

---

Review Form

Reviewer: The Synthesizer (Contextual Analyst)

Summary

This paper presents Coruscant, a compelling co-design of a GPU kernel and a sparse tensor core architecture aimed at making unstructured sparsity practical for Large Language Model (LLM) inference. The authors identify a critical gap in the current landscape: state-of-the-art pruning techniques naturally produce unstructured sparsity in the 30%-70% range, yet modern hardware, notably NVIDIA's 2:4 semi-structured sparsity, cannot efficiently exploit it, forcing a trade-off between model accuracy and hardware performance.

Coruscant's core contribution is a full-stack solution to this problem. It introduces:

1. A memory-efficient, bitmap-based sparse format designed for this moderate sparsity regime.
2. A software-only GPU kernel that leverages this format on existing commercial GPUs to reduce memory footprint and outperform dense (cuBLAS) and other sparse (Flash-LLM) kernels.
3. A minimal, pragmatic extension to the GPU's Sparse Tensor Core (STC) that directly decodes the bitmap format in hardware, eliminating the software decompression overhead for even greater performance gains.

The work is well-motivated by the memory-bound nature of the LLM decode phase and provides a clear pathway for the machine learning and computer architecture communities to reconcile the benefits of unstructured pruning with the demands of efficient hardware execution.

Strengths

The primary strength of this paper is its insightful positioning and holistic approach to a significant, real-world problem.

1. Excellent Problem Formulation: The authors have correctly identified a crucial point of friction between two advancing fields. On one hand, LLM pruning research (e.g., SparseGPT, Wanda) is pushing towards more flexible, unstructured methods to preserve accuracy. On the other, the hardware community has converged on rigid, structured solutions like 2:4 sparsity for efficiency. Coruscant builds a much-needed bridge between these two worlds. The analysis in Section 2.1 and Figures 2 & 3 effectively establishes the need for a solution in the 30-70% sparsity range, a zone where prior formats are shown to be inefficient.

2. Pragmatic Full-Stack Co-Design: This is not just a theoretical hardware proposal. The authors provide an immediately useful software kernel for today's GPUs and a well-defined, minimally invasive hardware modification for tomorrow's. This two-pronged approach dramatically increases the work's potential impact. It allows the community to adopt the ideas in software now while providing a clear, low-overhead blueprint for hardware vendors. The proposed "Bitmap Decoder" is a small, targeted addition, not a complete redesign, which makes it far more plausible for adoption.

3. Strong Contextualization and Evaluation: The paper does an admirable job of placing itself within the vast literature of sparse computation and LLM optimization. The comparisons are not just against the standard dense baseline (cuBLAS) but also against relevant, cutting-edge sparse kernels like Flash-LLM, SparTA, and Sputnik (Figure 16). Furthermore, the architectural comparison against prior academic STC designs like DSTC and RM-STC (Section 5.3.3) is particularly insightful, correctly arguing that Coruscant's simpler, memory-focused design is better suited for the "skinny matrix" SpMM workloads of LLM inference, as opposed to the compute-bound workloads those prior works targeted.

4. Connecting Pruning to System-Level Benefits: The work successfully translates the benefits of its approach from kernel-level speedups to tangible system-level advantages. The end-to-end evaluation (Section 5.1, Figure 15) shows not only increased token throughput but also the ability to run larger batch sizes by freeing up VRAM, directly addressing the "Out of Memory" errors that plague LLM serving. This demonstrates a deep understanding of the end-user's problem.

Weaknesses

The weaknesses of the paper are minor and primarily relate to missed opportunities to further broaden its context and impact.

1. Limited Discussion on Interaction with Other Compression Techniques: The paper is laser-focused on sparsity. However, in practice, sparsity is almost always combined with quantization. There is no discussion of how the Coruscant format would interact with weight quantization schemes (e.g., 8-bit or 4-bit integers). Would the bitmap overhead become more significant relative to the compressed data? Does the STC design need modification to handle quantized data types? Acknowledging this interaction is crucial for a complete system-level solution.

2. Focus on Weight Sparsity Only: The work is entirely centered on static, unstructured weight sparsity. Another emerging area of interest is activation sparsity, which is dynamic and data-dependent. While this is clearly outside the paper's primary scope, the core idea of a hardware-accelerated bitmap decoder could potentially be relevant there as well. A brief mention of this as a future direction would strengthen the paper's long-term vision.

3. The "Why Now?" Argument Could Be Sharpened: The motivation is good, but it could be even more powerful. The paper explains what the problem is, but it could spend more time on why solving it is becoming existential for the future of LLMs. As models grow and context windows expand, the memory footprint of weights (even without the KV cache) becomes a fundamental limiter. Framing Coruscant not just as an optimization but as an enabling technology for future, more powerful models on commodity hardware would elevate its perceived importance.

Questions to Address In Rebuttal

1. Synergy with Quantization: Could the authors comment on the synergy or potential conflicts between their bitmap-based format and popular quantization schemes (e.g., INT8, FP8, or INT4)? How would the memory footprint and hardware design change if Coruscant were to support pruned and quantized models?

2. The Next Bottleneck: Coruscant convincingly argues for a solution to the weight memory bandwidth bottleneck in sparse LLM inference. With this addressed by the proposed STC, what do the authors foresee as the next major bottleneck? Does this approach simply shift the performance problem to another part of the system, perhaps the instruction frontend or the register file bandwidth, especially given the more complex decoding logic?

3. Generality Beyond LLMs: While the work is excellently motivated by LLM inference, the proposed format and hardware seem more general. Have the authors considered its applicability to other domains that feature moderate, unstructured sparsity, such as graph neural networks (GNNs) or certain scientific simulations? How would the "skinny matrix" assumption hold up in those domains, and would the design still be effective?

Review Form

Reviewer: The Innovator (Novelty Specialist)

Summary

The paper presents "Coruscant," a co-designed system comprising a bitmap-based sparse format, a corresponding GPU SpMM kernel, and a minimal extension to the GPU Tensor Core, all aimed at accelerating LLM inference with unstructured weight sparsity. The authors identify a critical gap in existing solutions, which are either inefficient for moderate sparsity levels (30-70%) or enforce restrictive semi-structured patterns like 2:4 sparsity. The core proposal is to use a simple bitmap to represent sparsity within tiles, which improves compression for this specific sparsity range, and then to design a software kernel and a hardware "Bitmap Decoder" to process this format efficiently for the memory-bound, skinny matrix multiplications characteristic of LLM decode steps.

Strengths

1. Specialization as a Novel Contribution: The primary strength and novel aspect of this work is not the invention of a new primitive, but the highly effective specialization of existing concepts for a specific, important workload. The authors correctly identify that prior sparse tensor core designs (e.g., DSTC, RM-STC) were targeted at general-purpose, compute-bound problems, leading to complex and area-intensive hardware. Coruscant’s novelty lies in its insight that the LLM inference problem (sparse-weight, dense-activation SpMM) allows for a significant simplification:

   • It only requires single-operand sparsity, eliminating the need for complex scatter-gather and merge logic for the output matrix that plagues dual-operand sparse designs.
   • It targets a memory-bound regime, correctly prioritizing compression efficiency (via larger tile sizes) over maximizing computation skipping. This is a crucial and well-justified trade-off.

2. Architectural Elegance and Simplicity: The proposed hardware modification, the "Bitmap Decoder" (Figure 14, page 7), is minimalistic. It reuses the existing accumulation and data paths of a dense tensor core, adding only the logic to decode the bitmap and select the appropriate non-zero values from registers. This stands in stark contrast to the significant architectural rework proposed in prior art for unstructured sparsity. The claimed low area overhead (Table 2, page 10) makes this a practical and economically viable proposal, which is a form of novelty in itself.

3. Kernel-Level Innovation: The co-designed GPU kernel contains a subtle but important novel element: the use of column-wise tiling to mitigate shared memory bank conflicts during the software-based decompression stage (Figure 9, page 6). This is a non-obvious implementation detail that directly addresses a key performance bottleneck for this class of algorithm and demonstrates a deep understanding of the GPU execution model.

Weaknesses

1. Fundamental Primitives are Not Novel: The primary weakness from a novelty perspective is that the core building blocks are well-established.

   • Bitmap-based Sparse Formats: Representing sparse data with bitmaps is a classic technique. It is not a novel concept.
   • Hardware Decoders for Bitmaps: The idea of a hardware unit that processes a bitmap to gate or select data is also not fundamentally new.
   • Sparse Tensor Cores for Unstructured Sparsity: The most direct prior art, DSTC [55] and RM-STC [21], which the authors thankfully cite and compare against in Section 5.3.3, have already proposed the use of bitmap-based formats for unstructured sparsity in tensor cores.

   Therefore, the claim to novelty must rest entirely on the specific architectural choices and co-design for the LLM workload, not on the invention of the underlying mechanisms. The paper's framing could be clearer on this point; it sometimes implies the format itself is the primary innovation, when in fact the true innovation is the simplified architecture it enables for this specific problem.

2. Limited Scope of Novelty: The contribution is sharply focused on skinny SpMM. While this is the paper's stated goal, it means the proposed architecture is less general than prior work. The novelty is derived from removing generality, which is a valid but narrow path for contribution. The paper would be strengthened by more explicitly defining the boundaries where its approach is superior and where more general (but complex) approaches like DSTC would be preferable.

Questions to Address In Rebuttal

1. Given that DSTC [55] and RM-STC [21] have already proposed bitmap-based sparse tensor cores for unstructured sparsity, please articulate precisely what the core architectural novelty of the Coruscant STC is, beyond a change in tile size and specialization for single-sided sparsity. The rebuttal should focus on why this specialization constitutes a significant inventive step over these prior works, rather than an incremental engineering optimization.

2. The column-wise tiling strategy (Figure 9, page 6) to avoid shared memory bank conflicts is a key part of the software kernel's performance. Is this technique novel in the context of GPU-based sparse matrix decompression, or are there precedents for this specific approach in prior literature on SpMM or other sparse kernels?

3. The paper's core trade-off is prioritizing memory compression over computation skipping, which is well-suited for the target memory-bound workload. Can you characterize the crossover point (e.g., in terms of batch size 'N' or a machine's operational intensity) where the computation-skipping benefits of a more complex architecture like DSTC would begin to outweigh the compression benefits of Coruscant? A clear analysis of this trade-off boundary would better situate the novelty of your contribution.