S-DMA: Sparse Diffusion Models Acceleration via Spatiality-Aware Prediction and Dimension-Adaptive Dataflow

---

Review Form

Reviewer: The Guardian (Adversarial Skeptic)

Summary

The authors present S-DMA, a software-hardware co-design framework intended to accelerate sparse Diffusion Models (DMs). The core contributions are threefold: 1) A "Spatiality-Aware Similarity" (SpASim) method that reduces the complexity of sparsity prediction from O(N²) to O(N) by assuming local similarity; 2) A "NAND-based Similarity" computation that approximates cosine similarity using bitwise operations on sign or most significant bits (MSBs) to reduce hardware overhead; and 3) A "Dimension-Adaptive Dataflow" designed to unify sparse convolution and GEMM operations for processing on a dedicated PE array. The authors claim significant speedup and energy efficiency improvements over a baseline GPU and state-of-the-art (SOTA) DM accelerators.

Strengths

1. Problem Identification: The paper correctly identifies a critical bottleneck in prior work: the computational overhead of the sparsity prediction step itself can negate the benefits of sparse computation (Challenge 1 and 2, Figure 2, page 3). Focusing on reducing this overhead is a valid and important research direction.

2. Comprehensive Co-design: The proposed solution is a full-stack effort, spanning from algorithmic heuristics (SpASim, NAND-similarity) to microarchitectural implementations (SP²U, reduction network). This holistic approach is commendable.

3. Operator Unification: The dimension-adaptive dataflow (Section 3.3, page 6; Figure 8, page 7) is a technically sound approach to homogenize the dataflow for different sparse operator types (convolution and GEMM), which is a non-trivial challenge in hardware design.

Weaknesses

My primary concerns with this manuscript revolve around the fragility of its core assumptions, the justification for its approximation methods, and the soundness of its experimental comparisons.

1. Unjustified Locality Heuristic (SpASim): The central premise of SpASim—that semantically similar tokens are spatially proximal—is a strong heuristic that lacks robust validation.

   • The evaluation is performed on standard datasets (COCO, GSO) which are dominated by images with clear subjects and backgrounds, naturally favoring such a locality assumption. The work presents no evidence of how SpASim performs on adversarial inputs designed to violate this assumption (e.g., complex textures, abstract patterns, or images with fine-grained, distributed details).
   • The "adaptive" selection of the window size K (Algorithm 1, page 6) is performed offline. This is a misnomer; the system is not adaptive at runtime. A single, pre-calibrated K value is used for all inference tasks, which is brittle and may perform poorly on out-of-distribution inputs.

2. Extreme and Under-analyzed Similarity Approximation: The NAND-based similarity is a radical simplification of cosine similarity.

   • For SeS, using only the sign bit discards all magnitude information. For SpS, the paper claims to use MSBs because the distribution is non-negative (Section 3.2, page 6), but provides no analysis on how many bits are used or a sensitivity analysis of quality vs. the number of bits. Is a single MSB truly sufficient to capture similarity in a "long-tail positive distribution"? This seems highly improbable and is not substantiated.
   • The hardware savings reported in Figure 7 (page 6) are against an XNOR-based design, not the full MAC and normalization pipeline of cosine similarity. This presents the savings in an overly favorable light.

3. Unsupported "Zero-Latency" Claims: The paper repeatedly makes claims of "no additional inference latency" or "fully overlapped" operation for its auxiliary hardware components.

   • For the SP²U's sorting mechanism (Section 4.2, page 7), the claim that sorting is "fully overlapped" and introduces "no additional inference latency" is unsubstantiated. A formal analysis of potential pipeline hazards or stalls is required. It is difficult to believe there are no conditions under which the main PE array would have to wait for the SP²U.
   • Similarly, the Sparsity-Aware Reduction Network (Section 4.4, page 8) claims its accumulation can be "fully overlapped with PE computation, introducing no additional inference delay." Re-accumulating partial results from different PE lines based on dynamic sparsity masks is a complex routing and synchronization problem. Without cycle-level simulation data or a detailed pipeline diagram, this claim is not credible.

4. Fundamentally Flawed Baseline Comparisons: The experimental evaluation, particularly against SOTA accelerators, is misleading.

   • As acknowledged in Section 5.3 (page 10), the chosen baselines (Cambricon-D, Ditto) primarily exploit inter-step temporal sparsity. S-DMA exploits intra-step spatial/semantic sparsity. These are orthogonal, not competing, optimization strategies. Claiming a 7.05x speedup over them is an apples-to-oranges comparison and does not represent a legitimate scientific advance over their techniques. A proper comparison would be against an accelerator designed for intra-step sparsity or a system that integrates both approaches.
   • The reported speedups over the NVIDIA A100 GPU (up to 51.11x, Figure 12, page 10) are suspiciously high. This typically indicates that the GPU baseline is not sufficiently optimized. The authors provide no details on the implementation of "GPU+Sparsity." Was this implemented using highly-optimized CUDA kernels and libraries like cuSPARSE, or a naive high-level implementation? Without these details, the GPU baseline appears to be a strawman.

Questions to Address In Rebuttal

The authors must address the following points to establish the validity of their work:

1. On SpASim's Robustness: Provide evidence that the SpASim method is robust. This should include evaluation on a dataset specifically curated to challenge the spatial locality assumption. Furthermore, justify the use of an offline-tuned, fixed K and discuss the performance degradation when an input image violates the characteristics of the tuning set.

2. On NAND-based Similarity: Please provide a detailed sensitivity analysis for the NAND-based similarity metric. Specifically for SpS, clarify precisely how many MSBs are used and show how generation quality (e.g., FID, CLIP) degrades as the number of bits is reduced. Justify why this coarse approximation does not lead to catastrophic failures in semantic understanding.

3. On Architectural Latency Claims: Substantiate the claims of "no additional latency" for the SP²U sorter and the reduction network. Provide pipeline diagrams and/or cycle-level performance data demonstrating the absence of stalls across a range of sparsity patterns, including highly irregular ones.

4. On Experimental Baselines: Justify the direct comparison of S-DMA to accelerators (Cambricon-D, Ditto) that optimize for a completely different and orthogonal type of sparsity. Acknowledge that these are not competing approaches and re-frame the results accordingly. Provide comprehensive details on the implementation and optimization level of the GPU baselines to prove they are not strawmen.

---

Review Form

Reviewer: The Synthesizer (Contextual Analyst)

Summary

This paper presents S-DMA, a comprehensive software-hardware co-design framework for accelerating sparse diffusion models (DMs). The authors correctly identify that while sparsity offers a promising path to reduce the immense computational cost of DMs, existing methods are critically hampered by two second-order effects: 1) the significant computational overhead of predicting where sparsity exists, and 2) the hardware inefficiency of processing the diverse and irregular sparsity patterns that emerge across different operators, namely convolutions (CONV) and general matrix multiplications (GEMM).

To address this, S-DMA proposes a holistic solution. On the software side, it introduces a "Spatiality-Aware Similarity" (SpASim) method that leverages the inherent local correlations in image data to reduce the complexity of sparsity prediction from O(N²) to O(N). It further proposes a hardware-friendly, NAND-based similarity computation to replace expensive multiply-accumulate operations. On the hardware side, the work designs a dedicated accelerator featuring a novel "Dimension-Adaptive Dataflow." This key architectural contribution unifies the execution of sparse CONV and sparse GEMM operations into a single, efficient GEMM-based pipeline, overcoming a major challenge in accelerating hybrid models. The architecture is supported by a lightweight sparsity prediction unit (SP2U) and a sparsity-aware reduction network. The authors demonstrate significant speedup and energy efficiency gains over both high-end GPUs and other state-of-the-art DM accelerators.

Strengths

The true strength of this paper lies in its holistic and deeply integrated approach to a complex problem. It moves beyond simply applying known sparsity techniques and instead re-evaluates the entire sparse inference pipeline from first principles.

1. Excellent Problem Formulation: The authors' core insight is that the cost of finding sparsity can negate its benefits. By framing the problem around the three challenges in Figure 2 (page 3)—prediction complexity, prediction overhead, and low PE utilization—they provide a clear and compelling motivation for their work. This demonstrates a mature understanding of the practical barriers to deploying sparse acceleration.

2. Elegant Algorithmic and Architectural Synergy: The proposed solutions are not independent optimizations but a tightly coupled set of ideas. The SpASim algorithm is motivated by the spatial locality of the target domain (images), and the NAND-based similarity is a direct consequence of designing an algorithm with hardware implementation costs in mind. The "Dimension-Adaptive Dataflow" is the centerpiece of this synergy, providing a hardware substrate that can efficiently execute the sparse workloads created by the software-side prediction. This transformation of sparse convolution into a structured sparse GEMM (Section 3.3, page 6) is a particularly clever contribution that avoids the well-known overheads of traditional im2col-based approaches.

3. Strong Contextualization within the Field: The paper does an excellent job of placing itself within the broader landscape of AI acceleration. It correctly identifies the limitations of prior work, such as the unsuitability of sign-based similarity from ViT accelerators (like AdapTiV) for the non-negative attention maps in DMs (Section 2.2, page 4). It also distinguishes itself from other DM accelerators (like Cambricon-D and Ditto) by tackling a different and complementary form of sparsity (semantic/spatial vs. temporal/value). This demonstrates a nuanced understanding of the research frontier.

4. Significant Potential Impact: Diffusion models are a dominant workload in generative AI, and their computational demands are a major bottleneck. S-DMA provides a compelling blueprint for future specialized hardware. By making sparsity practical and efficient, this work could significantly reduce the latency and energy cost of DM inference, enabling their deployment in a wider range of applications, from on-device editing to real-time content generation. The core ideas—especially the unified dataflow—could also prove influential for accelerating other hybrid CNN-Transformer architectures.

Weaknesses

The weaknesses of the paper are primarily related to its scope and the exploration of its boundaries. The core contribution is sound, but its context could be further enriched.

1. Limited Discussion on Generalizability: The framework is highly optimized for the U-Net-based architectures common in DMs. While this specialization is a strength, the paper would benefit from a discussion on how the core concepts might generalize. For instance, could the dimension-adaptive dataflow be applied to other models that mix attention and convolution, such as vision transformers with convolutional stems or mobile-friendly hybrid networks? A brief exploration of this could broaden the paper's perceived impact.

2. Sensitivity of the SpASim Method: The performance of the SpASim method relies on the window size K, which is determined offline (Algorithm 1, page 6). The evaluation in Section 5.2 (page 9) shows this works well for the tested benchmarks, but a deeper analysis of its robustness would be welcome. How sensitive is the performance to this hyperparameter? For example, would a model trained on a different data distribution or a highly unusual user prompt require re-profiling to find a new optimal K?

3. Lack of Comparison with Structured Sparsity: The work focuses on exploiting dynamic, fine-grained sparsity. It would be valuable to briefly contrast this with structured sparsity approaches (e.g., block or vector sparsity). Structured methods are often considered easier to support in hardware and could present a different trade-off between compression rate and hardware complexity. Including this discussion would provide a more complete picture of the design space.

Questions to Address in Rebuttal

1. The core architectural contribution is the dimension-adaptive dataflow that unifies sparse CONV and GEMM. Could the authors comment on the applicability of this technique to other hybrid CNN-Transformer architectures outside the diffusion model space, such as those used in object detection or semantic segmentation?

2. The selection of the local window hyperparameter K is performed offline. How robust is a pre-selected K value to variations in input prompts or generation tasks? For instance, does an image edit focusing on a tiny detail versus a large global change affect the optimal K, and if so, how does S-DMA handle such dynamic variation?

3. The paper's evaluation focuses on its sparsity-centric approach in isolation. How do the authors envision S-DMA synergizing with orthogonal acceleration techniques for DMs, such as quantization, knowledge distillation, or the differential/temporal computing exploited by competitors like Cambricon-D? Are the performance gains expected to be additive?

4. The NAND-based similarity is a creative and effective hardware-aware approximation. Have the authors considered if this low-cost hardware primitive could be adapted for other similarity-based tasks in machine learning beyond sparsity prediction, such as in retrieval or clustering algorithms?

Review Form

Reviewer: The Innovator (Novelty Specialist)

Summary

The paper introduces S-DMA, a software-hardware co-design framework to accelerate Diffusion Models (DMs) by exploiting semantic and spatial sparsity. The authors propose three core contributions: (1) a "Spatiality-Aware Similarity" (SpASim) algorithm that reduces the complexity of sparsity prediction from O(N²) to O(N) by leveraging local similarity; (2) a "NAND-based Similarity" computation method that replaces expensive multipliers with bitwise logic for both symmetric (SeS) and non-negative (SpS) activation distributions; and (3) a "Dimension-Adaptive Dataflow" and corresponding hardware that unifies sparse convolution and sparse GEMM operations into a dense GEMM format.

While the paper presents a well-integrated and high-performing system, my analysis concludes that the foundational ideas behind each of the core contributions are not new. Instead, they represent effective adaptations and combinations of well-established principles from the fields of efficient Transformers and sparse hardware acceleration, applied specifically to the domain of Diffusion Models. The novelty is therefore in the application and system-level integration, not in the core concepts themselves.

Strengths

1. System-Level Co-Design: The work is a comprehensive example of software-hardware co-design, connecting algorithmic optimizations directly to bespoke hardware units (SP²U, reduction network).
2. Problem Formulation: The authors correctly identify and articulate the key challenges (Section 1, page 2, Figure 2) in accelerating sparse DMs: the overhead of sparsity prediction and the difficulty of handling heterogeneous sparse operators.
3. Holistic Sparsity Support: The framework's ability to handle both semantic sparsity (token merging) and spatial sparsity (image editing) within a unified architecture is a notable engineering achievement.

Weaknesses

The primary weakness of this paper, from the perspective of novelty, is that its core contributions are derivations of prior art.

1. Spatiality-Aware Similarity (SpASim) is an application of Local Attention: The central idea of SpASim (Section 3.1, page 5) is to reduce a quadratic O(N²) similarity computation to linear O(N) by restricting comparisons to local windows. This concept is the cornerstone of numerous efficient Transformer models developed over the past several years to overcome the exact same bottleneck. Architectures like the Swin Transformer (windowed attention) or methods like Longformer (sliding window attention) are built on this exact principle of exploiting locality to make attention tractable. While the authors apply this to the sparsity prediction step for DMs, the algorithmic principle of trading global comparison for local comparison to achieve linear complexity is not a novel contribution.

2. NAND-based Similarity is an incremental extension of Bitwise Similarity Proxies: The proposal to use cheap bitwise operations as a proxy for expensive cosine similarity (Section 3.2, page 6) is not new. The authors themselves reference AdapTiV [48], which uses XNOR-based sign similarity for this purpose in Vision Transformers. The authors' claim to novelty rests on adapting this for DMs, where SpS activations are non-negative, by using the Most Significant Bit (MSB) instead of the sign bit. Using MSBs as a low-cost proxy for magnitude is a standard technique in approximate computing. The move from XNOR to NAND gates (Figure 7, page 6) is a minor circuit-level optimization. Therefore, this contribution is a small, albeit clever, delta over existing work, adapting a known technique to a slightly different data distribution.

3. Dimension-Adaptive Dataflow is a form of Sparse Data Compaction: The proposed dataflow (Section 3.3, page 6, Figure 8) aims to unify sparse convolution and GEMM by transforming sparse convolution into a dense GEMM operation. This is achieved by gathering active tokens/channels and permuting them into a dense block. The concept of converting sparse operations into dense ones by gathering non-zero elements to feed a dense systolic array or PE array is a foundational technique in sparse accelerator design. While this method avoids the memory overhead of the classic im2col transformation for sparse inputs, the "gather-compute-scatter" pattern is not a new architectural paradigm. The novelty lies in the specific permutation strategy for DMs, not in the fundamental approach of data compaction for efficient hardware utilization.

Questions to Address In Rebuttal

1. Regarding SpASim: The authors must explicitly differentiate their contribution from the large body of existing work on local and windowed attention in the Transformer literature. Beyond applying a known technique to a new problem (sparsity prediction in DMs), what is the fundamental algorithmic novelty?

2. Regarding NAND-based Similarity: Can the authors argue that the extension from sign-based similarity (as in AdapTiV [48]) to a hybrid sign/MSB-based approach is a non-obvious conceptual leap? Given that using MSBs as magnitude comparators is a common heuristic, the rebuttal should clarify why this specific adaptation constitutes a significant novel contribution.

3. Regarding the Dimension-Adaptive Dataflow: Please contrast the proposed dimension permutation technique with other structured sparsity or data compaction schemes in the hardware accelerator literature. How does this approach differ fundamentally from prior "gather-compute-scatter" architectures designed to handle sparse activations? The defense of novelty should focus on the architectural concept, not just its specific tuning for DM workloads.