Using Analytical Performance/Power Model and Fine-Grained DVFS to Enhance AI Accelerator Energy Efficiency

Paper Title: Using Analytical Performance/Power Model and Fine-Grained DVFS to Enhance AI Accelerator Energy Efficiency
Reviewer Persona: The Guardian (Adversarial Skeptic)

---

Summary

The authors present an approach for improving the energy efficiency of a modern AI accelerator (the Ascend NPU) by leveraging its fine-grained DVFS capabilities. The core of their contribution is a pair of analytical models: a performance model that predicts operator execution time as a function of frequency, and a power model that incorporates temperature effects. The performance model is derived from a white-box timeline analysis of different operator execution scenarios. These models are then used within a genetic algorithm-based search to generate DVFS schedules. The authors report a 13.44% reduction in AICore power for a 1.76% performance loss on workloads like GPT-3 training.

Strengths

1. Real-System Demonstration: The work is evaluated on a modern, proprietary AI accelerator with millisecond-level DVFS capabilities. This provides a valuable data point, as much of the prior work in this area relies on simulation or older GPU hardware with slower DVFS mechanisms.
2. Inclusion of Temperature in Power Model: The authors correctly identify temperature as a factor in static power and incorporate a temperature-dependent term into their power model (Section 5). This is a step towards greater physical realism compared to many existing models that ignore this effect.
3. Attempt at White-Box Analysis: The detailed timeline analysis in Section 4.2, which categorizes operator execution into four distinct scenarios, is an ambitious attempt to provide a first-principles understanding of performance scaling.

Weaknesses

My primary concerns with this paper lie in the disconnect between the theoretical analysis and the practical implementation, the robustness of the models, and the overstatement of the work's generalizability.

1. A Disconnect in Performance Modeling: The paper dedicates significant effort (Section 4.2, page 4-5) to deriving that an operator's cycle count is a "convex piecewise linear function" of frequency. This is presented as a key insight. However, in Section 4.3 (page 6), this entire derivation is abandoned due to "practical challenges." The authors then resort to fitting a simple, non-piecewise function (T(f) = (af+c)/f). This feels like a methodological bait-and-switch. The elaborate timeline analysis serves as little more than a weak justification for choosing a convex fitting function, but it does not inform the final model's structure. The core analytical contribution is therefore not actually used in the implementation.

2. Questionable Model Validation: The validation of the performance model is suspect. The authors explicitly exclude all operators with execution times below 20 microseconds (Section 7.2, page 10). They state this accounts for 58.3% of all operators by count. While they claim this is only 0.9% of the total execution time, this exclusion is a form of data filtering that can artificially inflate the reported accuracy. The cumulative effect of errors on numerous small operators is not analyzed. An average error of 1.96% on a pre-filtered dataset is not as impressive as it appears.

3. Fragile Power Model: The power model's accuracy is presented with an average error of 4.62%, but the distribution of this error is highly problematic. The authors' own data in Table 2 (page 10) shows that nearly 20% of predictions have an error greater than 10%. Such a heavy tail of high-error predictions can easily lead a DVFS strategy to make significantly suboptimal decisions, yet the impact of this error distribution on the final outcome is never discussed.

4. Dilution of "Fine-Grained" Control: The paper's premise is operator-level DVFS. However, the preprocessing methodology described in Section 6.2 and illustrated in Figure 13 (page 9) groups operators into larger "Low Frequency Candidate" (LFC) and "High Frequency Candidate" (HFC) stages. The DVFS decisions appear to be made at the granularity of these stages, not individual operators. This contradicts the central claim of operator-level control and significantly reduces the search space, potentially missing finer-grained optimization opportunities.

5. Unsubstantiated Claims of Generalizability: In Section 8.3, the authors claim the performance model can be applied to other hardware like GPUs and TPUs because they share an "abstract" memory hierarchy. This is a gross oversimplification. The proposed Ld/St/Core model completely ignores fundamental architectural features of GPUs, such as warp-based execution, complex multi-level schedulers, and massive thread-level parallelism, which are the dominant factors in their performance scaling. The claim of generalizability is asserted without any supporting evidence.

Questions to Address In Rebuttal

1. Please justify the methodological leap in Section 4.3. If the piecewise linear model derived from your core analysis is intractable, what specific value does the detailed derivation in Section 4.2 provide beyond a generic observation of convexity?

2. Can you provide an analysis of your performance model's accuracy without excluding the 58.3% of operators shorter than 20µs? How does this exclusion affect the model's ability to predict performance for workloads dominated by many small kernels?

3. Given that 19.4% of your power model's predictions have >10% error (Table 2), how can you be confident that your genetic algorithm is not converging on a suboptimal DVFS schedule that is merely an artifact of significant modeling errors for certain operators?

4. Please clarify the true granularity of your DVFS policy. Are frequency decisions made for each individual operator, or at the boundaries of the preprocessed LFC/HFC stages shown in Figure 13? If the latter, please revise the claims of "operator-level" optimization.

5. Beyond stating that memory hierarchies are abstractly similar, what concrete evidence supports the claim that your performance model, which lacks any concept of warp scheduling or thread-level parallelism, can be generalized to modern GPU architectures?

Paper Title: Using Analytical Performance/Power Model and Fine-Grained DVFS to Enhance AI Accelerator Energy Efficiency
Reviewer Persona: The Synthesizer (Contextual Analyst)

---

Summary

This paper presents a comprehensive, end-to-end methodology for applying fine-grained, operator-level Dynamic Voltage and Frequency Scaling (DVFS) to enhance the energy efficiency of the Huawei Ascend NPU. The work is enabled by a recent hardware capability on this platform that allows for millisecond-level frequency changes, a significant reduction compared to the coarser-grained control available on many contemporary accelerators.

The authors' methodology consists of three main components:

1. An analytical, white-box performance model derived from a detailed timeline analysis of operator execution. The central insight is that an operator's cycle count can be modeled as a convex, piecewise linear function of frequency.
2. A physically-grounded power model that, notably, incorporates a temperature-dependent term to account for leakage current, enhancing its accuracy.
3. A DVFS strategy generator that uses operator classification and a genetic algorithm to navigate the vast search space of operator-level frequency settings, balancing performance loss against energy savings.

Evaluated on real hardware with modern workloads like GPT-3 training, the proposed system achieves a 13.44% power reduction in the NPU's computing core (AICore) and a 4.95% reduction at the full chip level, with a tightly constrained performance degradation of only 1.76%. This work serves as a valuable case study and a practical blueprint for exploiting emerging fine-grained power management features in AI accelerators.

---

Strengths

This is an excellent systems paper that successfully connects low-level hardware characteristics to high-level workload optimization. Its primary strengths are:

1. Timeliness and Seizing a New Opportunity: The core contribution is built upon a crucial, recent evolution in hardware capabilities: millisecond-level DVFS. While the idea of fine-grained DVFS has been explored in simulation, this paper is one of the first to demonstrate its practical application and benefits on real, production-grade hardware. It effectively provides a roadmap for the community on how to leverage these new features as they become more common.

2. Principled, Insightful Performance Modeling: The paper's most significant intellectual contribution is the performance model detailed in Section 4 (pages 3-6). Rather than treating the performance-frequency relationship as a black box to be fitted by a generic function, the authors conduct a rigorous timeline analysis of different operator execution scenarios (e.g., PingPong-free vs. PingPong, independent vs. dependent memory operations). From this analysis, they derive the fundamental insight that the cycle count behaves as a convex piecewise linear function. This white-box approach is not only more robust but also provides valuable intuition about the underlying system bottlenecks.

3. Grounded in Reality: The work is thoroughly evaluated on a modern, commercially relevant AI accelerator (Ascend NPU) with complex, real-world applications (GPT-3, BERT). This is not a simulation study. The authors tackle the full complexity of the software stack (PyTorch, CANN) and system measurement, making their reported energy savings both credible and impactful. Providing deep insights into a non-NVIDIA high-performance architecture is, in itself, a valuable service to the academic community.

4. Holistic, End-to-End System: The authors present a complete solution, from low-level characterization and modeling to a high-level search-based policy generator. This end-to-end perspective demonstrates a mature engineering effort and provides a more convincing argument than a paper focusing on just one piece of the puzzle.

---

Weaknesses

The paper is strong, and its weaknesses are more about clarifying the boundaries of the contribution rather than fundamental flaws.

1. Unclear Generalizability of the Performance Model: While the core idea that performance is limited by either computation or memory bandwidth is universal, the specific timeline analyses in Section 4.2 (pages 4-5) appear tightly coupled to the Ascend NPU's specific architecture and execution model. The four scenarios presented are insightful but may not map directly to other architectures with different memory systems or scheduling logic (e.g., out-of-order execution, different memory prefetching mechanisms). The paper would be strengthened by more clearly delineating the fundamental principles from the platform-specific details.

2. Modest Impact of the Temperature-Aware Power Model: The inclusion of temperature in the power model (Section 5, page 6) is a nice nod to physical reality. However, the authors' own analysis shows it provides a relatively small improvement in accuracy (error reduces from 4.97% to 4.62%, as mentioned in the ablation on page 10) and models a component that accounts for a minority of the total power (page 11). While intellectually sound, its practical contribution to the final result seems minor, and its prominence in the abstract might slightly overstate its importance relative to the much more impactful performance model.

3. Inherent Hardware Limitations: The work is constrained by the DVFS capabilities of the underlying hardware, which only allows control over the AICore. As the authors correctly note in Section 8.2 (page 12), uncore components like HBM and interconnects constitute a major portion of the chip's power budget (averaging 80%). This fundamentally caps the total system-level energy savings achievable. While this is not a flaw of the authors' method, the paper should frame its results with this context in mind—they have likely pushed the core-only DVFS approach close to its practical limit.

---

Questions to Address In Rebuttal

I would appreciate the authors' perspective on the following points to help solidify the paper's contribution:

1. On the Portability of the Performance Model: Your performance model's derivation in Section 4 is a key strength. Could you elaborate on which parts of this analysis (e.g., the classification into the four scenarios) are fundamental to any accelerator with a standard memory hierarchy (L1/L2/HBM), versus which are specific to the Ascend NPU's pipeline, DMA engine, and scheduling model? This would help readers understand how to adapt this excellent work to other platforms.

2. On the Practical Overhead: The end-to-end process requires profiling runs and a non-trivial genetic algorithm search to generate a policy. For a new, unseen deep learning model, what is the approximate time and computational overhead required to generate a new DVFS policy? How does this one-time cost compare to the energy savings over a typical, long-running training job?

3. On the Model Inference Scenario: In Section 8.4 (page 13), you astutely observe that inference is often host-bound, creating idle periods that are ripe for DVFS exploitation. This seems to be a fundamentally different optimization target than the training scenario (i.e., exploiting slack time vs. actively trading performance for power on a busy device). Is your detailed performance modeling still necessary for this scenario, or would a simpler reactive policy (e.g., "frequency down when idle") achieve most of the benefits?

Reviewer: The Innovator (Novelty Specialist)

---

Summary

The authors present an end-to-end methodology for applying fine-grained, operator-level Dynamic Voltage and Frequency Scaling (DVFS) to a Huawei Ascend NPU to improve energy efficiency. The core of their approach rests on three claims of novelty: (1) a white-box, analytical performance model that concludes an operator's cycle count is a convex piecewise linear function of frequency; (2) a power model that explicitly incorporates a temperature-dependent term for leakage current; and (3) the application of these models within a Genetic Algorithm (GA) framework to generate DVFS schedules at a millisecond granularity, a capability enabled by the specific hardware platform.

My review focuses exclusively on the novelty of these contributions in the context of prior art.

---

Strengths

The primary novel contribution of this work is the analytical derivation of the performance model.

1. Novel Analytical Insight into Performance: The most significant contribution is the detailed timeline analysis presented in Section 4 (pages 4-6). While prior works have created analytical performance models for accelerators (e.g., CRISP [28]), the specific breakdown into four scenarios (PingPong vs. non-PingPong, dependent vs. independent Ld/St) to formally derive that the cycle count is a convex, piecewise linear function of frequency is a novel theoretical insight. This provides a strong justification for their choice of fitting functions, moving beyond the purely empirical or black-box modeling approaches seen in much of the prior DVFS literature [3, 8, 43]. This framework provides the "why" behind the observed performance-frequency relationship.

2. First Experimental Demonstration of Operator-Level DVFS on a Commercial AI Accelerator: To my knowledge, this is the first work to experimentally demonstrate and evaluate a complete system for operator-level DVFS on a real, commercially available AI accelerator. Previous studies on GPUs [32, 38, 46] have been limited to coarser granularities (sub-phases, kernels, or entire applications) due to hardware latency (~15ms on NVIDIA V100, as noted by the authors in Section 1, page 2). This paper leverages a specific hardware feature (1ms DVFS latency on Ascend) to explore a fundamentally new operating point in the energy-performance trade-off space. The novelty here is the "existence proof" and systems integration.

---

Weaknesses

While the core performance model is novel, other aspects are either derivative or their claimed novelty provides insignificant benefits.

1. Marginal Novelty and Benefit of the Temperature-Aware Power Model: The inclusion of a temperature-dependent term (γ∆TV) in the power model (Section 5, page 7) is presented as a key contribution. However, the physical principle that subthreshold leakage is temperature-dependent is fundamental and well-established [36]. While many prior architectural power models [19, 26] may have omitted this for simplicity, its inclusion here is more of a refinement than a foundational new idea. More critically, the authors' own evaluation in Section 7.3 (page 10) shows this added complexity provides a negligible improvement in accuracy: the average error is reduced from 4.97% to 4.62%. An improvement of 0.35 percentage points does not justify its positioning as a significant novel contribution.

2. Use of a Genetic Algorithm is Not Fundamentally New: The paper proposes a DVFS strategy using a Genetic Algorithm (Section 6.3, page 9). The use of GAs for complex search space optimization is a standard, decades-old technique [9, 33]. While its application to this specific problem is appropriate, the method itself is not novel. The novelty lies in the problem formulation—specifically, the fast and accurate scoring function enabled by their performance/power models—rather than the choice of a GA as the search heuristic. The presentation should frame this part of the contribution more carefully.

3. Potential Overlap with Empirically Observed Phenomena: The core conclusion that performance scaling with frequency is non-linear and exhibits diminishing returns (i.e., is a convex function) has been empirically observed and modeled in many prior works on GPUs. The key delta here is the authors' white-box derivation. However, the practical implication—modeling performance with a convex function—may not be entirely new, even if the theoretical underpinnings are. The paper could be strengthened by more directly contrasting its derived functional form with the empirically-fitted curves used in prior works [2, 46].

---

Questions to Address In Rebuttal

1. Regarding the temperature-dependent power model: Can the authors provide a scenario, workload, or environmental condition where the γ∆TV term is not a marginal contributor but is instead critical for accurate power modeling? Without such a demonstration, the novelty and utility of this specific contribution remain questionable.

2. The analytical performance model is derived from an in-order execution model with specific memory pipeline assumptions (Figures 5, 6, 7, and 8 on pages 5-6). How robust is the central conclusion—that cycle count is a convex piecewise linear function of frequency—to different architectural paradigms, such as those with more complex out-of-order execution, multiple outstanding memory requests, or different cache coherence traffic? Is this a general property of memory-bound computation or one specific to the Ascend-like architecture abstracted here?

3. Could you clarify the distinction between your derived performance model and prior art that may have used similar convex functions (e.g., quadratic) for empirical fitting? While the derivation is novel, is the resulting model functionally different from or significantly more accurate than what could be achieved by fitting a standard convex function to empirical data, as done in other works?