LLMulator: Generalizable Cost Modeling for Dataflow Accelerators with Input-Adaptive Control Flow

---

Review Form

Reviewer: The Guardian (Adversarial Skeptic)

Summary

This paper introduces LLMulator, a framework that uses pre-trained Large Language Models (LLMs) for performance prediction of dataflow accelerators. The authors claim to address three key generalization challenges: across diverse applications, hardware configurations, and input-dependent control flows. The proposed method consists of three primary components: (1) a numeric modeling technique that tokenizes numbers and predicts performance values digit-by-digit; (2) a dynamic calibration mechanism using reinforcement learning (DPO) to refine predictions based on runtime feedback; and (3) a progressive data synthesis framework to generate training data. The authors evaluate their framework against several baselines (TLP, GNNHLS, Tenset-MLP) and claim state-of-the-art accuracy.

While the paper tackles a relevant problem, the methodology presents several unexamined complexities, the evaluation contains significant methodological gaps, and the claims of generalization are not fully substantiated by the provided evidence.

Strengths

1. Well-Motivated Problem: The paper correctly identifies the limitations of existing performance prediction methods, particularly regarding generalization to unseen applications, hardware, and dynamic inputs.
2. Realistic Ground Truth Generation: The use of an open-source toolchain (SiliconCompiler, Bambu HLS, OpenROAD) for profiling and generating ground truth data (Section 7.1) is a strong point, lending credibility to the dataset used for training and evaluation.
3. Comprehensive Structure: The framework is structured logically to address the three identified challenges (application, input, and hardware generalization), which provides a clear narrative for the contributions.

Weaknesses

1. Unjustified Complexity of Numeric Modeling: The proposed "decoupled numerical modeling" (Section 4.2), which predicts performance digit-by-digit via classification, is presented as a key innovation. However, the paper fails to convincingly argue why this complex mechanism is superior to a standard regression head with a proper loss function (e.g., log-scale MSE) that is less sensitive to extreme value ranges. The only evidence is an ablation ("NoEnc" in Table 3), which conflates the input encoding with the output modeling. A direct comparison of the output mechanism alone is required to justify this added complexity.

2. Misleading Comparison with Rule-Based Models: The comparison against Timeloop in Figure 11 is fundamentally flawed. Timeloop is a specialized analytical model for regular, loop-nest-based tensor computations. The workloads from Table 2, especially those from NLP, contain complex control flow and heterogeneous operator graphs that fall far outside Timeloop's intended domain. To claim superiority by evaluating a general-purpose model against a specialist on a generalist's turf is a classic apples-to-oranges fallacy. This comparison does not validate the model's accuracy but rather highlights a misapplication of the baseline tool.

3. Opaque and Potentially Biased Dataset Synthesis: The "progressive data generation framework" (Section 6) is described at a high level but lacks the detail required for reproducibility and critical assessment. The process, especially the LLM-based self-augmentation, is a black box. There is no analysis to demonstrate that the synthetic data is representative of real-world hardware design patterns or that it does not simply learn the biases of the generation scripts themselves. The claim of "producing datasets aligned closely with realistic hardware implementations" is an assertion without statistical evidence comparing the properties of the generated dataset to a corpus of real-world designs.

4. Unexamined Trade-offs of Dynamic Calibration: The dynamic calibration mechanism (Section 5) reports a reduction in MAPE for cycle prediction from 28.9% to 16.4% (Table 3). However, several critical details are omitted:

   • Overhead: The computational cost and latency of the DPO update step are not quantified. How many profiling runs and update iterations are needed to achieve this improvement?
   • Stability: Reinforcement learning methods can be unstable. There is no analysis of the convergence properties or the risk of overfitting to the most recent profiling data in the replay buffer.
   • Accuracy: A final MAPE of 16.4% on dynamic cycles is still a significant error. It is not clear that this level of accuracy is sufficient for reliable design space exploration, especially given the added complexity and runtime overhead.

5. Prohibitive Inference Latency for Practical Use: Table 4 shows that LLMulator's inference time is approximately 1.01s per prediction on Polybench, an order of magnitude slower than GNNHLS (0.11s). The authors dismiss this as "acceptable compared to the longer synthesis times," but this ignores the primary use case: large-scale design space exploration (DSE), where millions of design points must be evaluated quickly. A 10x slowdown makes the proposed tool impractical for this critical task. The acceleration techniques in Section 5.3 show only marginal improvements (Table 5) and do not close this gap.

Questions to Address In Rebuttal

1. Regarding Numeric Modeling: Can you provide an ablation study that isolates the output modeling strategy? Specifically, please compare the performance of your digit-by-digit classification approach against a standard regression head (using both MSE and log-scale MSE loss) on the same LLM backbone and input encoding, to prove its superiority.

2. Regarding Dynamic Calibration: Please quantify the full runtime overhead of the dynamic calibration process. How many profiling runs and DPO updates were required to reduce the MAPE from 28.9% to 16.4%? Furthermore, please justify why a final MAPE of 16.4% on cycles is a strong result for a system with this much complexity, and discuss its practical utility in DSE.

3. Regarding the Timeloop Comparison: Please justify the comparison in Figure 11. Alternatively, provide a new comparison against Timeloop on a workload for which it is explicitly designed (e.g., a pure GEMM operator with varying tiling and mapping strategies) to demonstrate where LLMulator offers a genuine advantage.

4. Regarding Dataset Synthesis: Please provide a detailed characterization of the synthetic dataset. This should include statistical distributions of key features (e.g., loop nesting depth, array access patterns, operator types) and a comparison of these distributions against a well-known corpus of real-world HLS benchmarks to substantiate the claim of realism.

5. Regarding Model Choice and Scalability: The choice of a small 1B parameter model seems arbitrary. How do the results and, critically, the 10x inference latency penalty, scale with larger, more capable models (e.g., 7B, 13B)? Is it possible that a larger model could achieve better accuracy without the complex, bespoke numeric modeling and dynamic calibration frameworks?

---

Review Form

Reviewer: The Synthesizer (Contextual Analyst)

Summary

This paper introduces LLMulator, a comprehensive framework for performance prediction of dataflow accelerators that aims to overcome the critical generalization limitations of prior art. The authors correctly identify three major axes of failure for existing models: generalization to unseen applications (especially at numerical extremes), adaptation to input-dependent control flow, and transference to diverse hardware configurations.

To address this, LLMulator is not a single model but a holistic system built on three innovative pillars:

1. A progressive numeric modeling technique that treats numerical values in code and performance metrics as sequences of digits, using a categorical classification approach. This is designed to improve accuracy on out-of-distribution numerical ranges and uniquely provides prediction confidence scores.
2. An input-adaptive dynamic calibration framework based on reinforcement learning (Direct Preference Optimization), which refines performance predictions at runtime by incorporating feedback from live profiling, thereby handling dynamic control flow.
3. A progressive data synthesis framework that systematically generates a diverse and realistic dataset covering software, hardware, and mapping variations, including the generation of intermediate compiler reasoning steps to enhance model learning.

The authors conduct an extensive evaluation, demonstrating that LLMulator achieves a state-of-the-art mean absolute percentage error (MAPE) of 12.2%, significantly outperforming established baselines like TLP and GNNHLS.

Strengths

The primary strength of this work lies in its insightful diagnosis of the core problems with applying machine learning to systems modeling and its subsequent development of a sophisticated, multi-pronged solution. This is not a naive application of a large language model (LLM); it is a well-engineered system that leverages an LLM's strengths while meticulously compensating for its weaknesses.

1. A Paradigm Shift in Numerical Modeling: The most significant conceptual contribution is the shift away from standard regression for performance prediction. The progressive numeric modeling (Section 4, page 4) is a brilliant insight. By treating performance prediction as a digit-by-digit classification task, the authors elegantly solve the "edge value" problem where regression models using normalization fail catastrophically. Furthermore, this approach's ability to output a confidence distribution (logits) for each digit is a profoundly practical feature for designers, who need to understand not just the prediction, but the model's certainty. This connects to a broader push for interpretability and uncertainty quantification in ML for science and engineering.

2. Bridging Static Analysis and Dynamic Execution: The dynamic calibration framework (Section 5, page 6) is a very clever solution to the long-standing problem of input-dependent behavior. Most performance models are entirely static. By creating a closed loop with a profiler and adapting the concept of Direct Preference Optimization (DPO) from the RLHF literature, the authors have created a model that learns from execution. This is a powerful idea that bridges the gap between fast, static cost models and the ground truth of dynamic execution. It has the potential to make ML-based prediction far more trustworthy in iterative design space exploration (DSE) loops.

3. Mature Approach to the "Data Problem": Data-driven methods in computer architecture are often hamstrung by the lack of large, diverse, high-quality datasets. The authors' progressive data synthesizer (Section 6, page 8) is a comprehensive and well-thought-out solution. Moving from general AST-based generation to dataflow-specific and finally LLM-augmented code is a robust strategy. The inclusion of "Reasoning Data Formatting" (Figure 9, page 9), which is conceptually parallel to the "Chain-of-Thought" prompting in NLP, is an excellent cross-pollination of ideas, helping the model learn the why (intermediate compiler features) behind the what (final performance).

4. Strong Empirical Validation: The experimental results are thorough and convincing. The ablation studies in Table 3 (page 11) and Table 7 (page 12) are particularly strong, as they clearly tie the significant performance gains back to each of the specific contributions (numeric encoding, dynamic calibration, and data synthesis). The comparison against both other ML models (TLP, GNNHLS) and an analytical model (Timeloop, Figure 11, page 10) firmly establishes the work's superiority and breadth.

Weaknesses

While the core ideas are excellent, the work could be strengthened by addressing the following points, which are less about fundamental flaws and more about the practical implications and boundaries of the proposed system.

1. Practicality of the Calibration Loop: The dynamic calibration loop is a powerful concept, but its real-world utility depends heavily on its overhead. The paper reports runtime latency (Table 4, page 11), showing LLMulator is an order of magnitude slower than GNNHLS. While the authors argue this is acceptable compared to full synthesis, the cost of the profiling step within the DPO loop is a critical factor for DSE. A full profiling run for every prediction update may be prohibitively slow. The paper could benefit from a discussion of the trade-offs here: how many DPO iterations are needed for convergence, and what is the total time cost (prediction + profiling) in a realistic DSE scenario?

2. System Complexity and Reproducibility: LLMulator is a complex system with many interacting components (three different data generators, a static LLM, a dynamic DPO updater, parsers, profilers). This complexity raises questions about its robustness and the engineering effort required to retarget it to a completely new class of accelerator (e.g., analog in-memory computing, neuromorphic). The paper presents it as a general framework, but its effectiveness is likely tied to the specific rules and templates within the data synthesizer.

3. Characterization of Failure Modes: The authors commendably note that workloads like jacobi-2d exhibit higher errors due to complexity (Section 7.2, page 10). This is a crucial finding that deserves more exploration. This work sits at the intersection of program analysis and LLM semantics. What are the fundamental limits? Is the problem related to non-local code interactions, complex data structures, or aliasing that breaks the LLM's semantic understanding? A deeper analysis of these failure modes would be a valuable contribution to the broader field of using LLMs for code analysis.

Questions to Address In Rebuttal

1. Could the authors elaborate on the practical cost of the dynamic calibration loop? Specifically, in a design space exploration scenario, what is the expected wall-clock time for the model to adapt to a significant change in input data distribution, including the necessary profiling runs?

2. Regarding the data synthesizer: how much domain-specific expertise is required to retarget it for a novel accelerator architecture that has fundamentally different primitives than the dataflow style explored here? For instance, how would the AST-based and dataflow-specific generators be adapted?

3. The sensitivity study on model size (Table 10, page 12) is very interesting, showing larger models improve accuracy. Does this suggest that the remaining prediction errors are not due to issues with the framework's structure (e.g., numeric encoding) but are rather bounded by the base LLM's reasoning capability? Could you speculate on whether even larger, frontier models could overcome the issues seen in complex workloads like jacobi-2d?

4. The confidence scores from the numeric output model are a fantastic feature. Have you explored using these scores to guide the DSE process itself? For instance, could the system actively trigger a more expensive, accurate simulation only when the LLMulator's prediction confidence is low?

Review Form

Reviewer: The Innovator (Novelty Specialist)

Summary

This paper presents LLMulator, a framework for performance prediction of dataflow accelerators. The authors identify three key generalization challenges—across applications, inputs, and hardware configurations—and propose a tripartite solution. The core of the authors' novel claim rests on the synthesis and application of several recent machine learning techniques to the domain of hardware cost modeling. These contributions are: (1) a "progressive numeric modeling" method that treats performance prediction as a digit-by-digit classification task rather than direct regression; (2) a "dynamic calibration" framework using Direct Preference Optimization (DPO) to refine predictions based on live execution feedback for input-dependent control flows; and (3) a "progressive data generation" pipeline that combines syntactic generation, domain-specific templates, and LLM-based augmentation to create a comprehensive training dataset.

While the constituent concepts (e.g., number-aware tokenization, reinforcement learning from feedback, curriculum-based data generation) exist in prior art within the broader machine learning literature, their specific adaptation, integration, and application to solve the multi-faceted generalization problem in dataflow accelerator modeling represents a novel engineering contribution. The core novelty is not in the invention of a new algorithm from first principles, but in the sophisticated scaffolding built around a pre-trained LLM to imbue it with capabilities it natively lacks.

Strengths

The primary strength of this work lies in its novel approaches to adapting large language models for a task they are not inherently designed for: precise, generalizable numerical prediction in a specialized domain.

1. Novel Prediction Formulation: The shift from a standard regression model (as seen in prior work like TLP [89]) to a progressive, digit-wise categorical classification model (Section 4.2, page 6) is a significant conceptual novelty. This directly addresses the known failure mode of regression models on out-of-distribution or "edge" values by breaking the problem into a sequence of smaller, bounded classification tasks. This allows for confidence estimation at each digit, a feature absent in direct regression approaches.

2. Novel Application of Reinforcement Learning: The use of DPO, a technique from the very recent LLM alignment literature [66], for dynamic, input-aware calibration of a performance model (Section 5, page 6) is a novel domain transfer. Prior work on input-adaptive performance modeling has typically relied on extracting static features or using more traditional online learning methods. Applying a preference-based RL method to learn from "ground-truth is better than prediction" pairs is a genuinely new mechanism for this problem space.

3. Novel Data Synthesis Pipeline: The multi-stage data generation framework (Section 6, page 8) is a more principled and sophisticated approach than prior methods. While individual components like AST-based generation [4] and using intermediate compiler representations [80] exist, the progressive pipeline that starts with general syntactic structures, specializes them for dataflow, and then uses an LLM for semantic diversification is a novel and powerful concept. The inclusion of intermediate reasoning data (Figure 9) inspired by chain-of-thought prompting [78] is a clever way to guide the model, moving beyond simple input-output pairs.

Weaknesses

The weaknesses of the paper, from a novelty perspective, are that many of the underlying mechanisms are imported from the ML/NLP fields, and the gains must be weighed against the significant increase in system complexity.

1. Derived, Not Foundational, Novelty: The core ideas are clever adaptations, not fundamental inventions. For example, improving LLM numeracy by isolating digits during tokenization (Section 4.1) is a known technique to improve model arithmetic [16]. The "chain-of-thought" style reasoning in the dataset is borrowed directly from LLM prompting research [78]. The novelty is purely in the application to hardware cost modeling. An expert in LLMs would not find the techniques themselves new, only their target domain.

2. Complexity vs. Benefit Trade-off: The proposed solution is substantially more complex than prior art. It combines a fine-tuned LLM, a reinforcement learning loop with a replay buffer, and a complex multi-stage data synthesizer. The end result, as shown in Table 3, is a reduction in MAPE from 20.0% (TLP) and 28.9% (GNNHLS) to 12.2%. While this is a clear improvement, the "delta" is not an order-of-magnitude leap. A critical assessment must question whether the marginal accuracy improvement justifies the massive increase in framework complexity, training cost, and inference latency (Table 4 shows LLMulator is ~10x slower than GNNHLS). The novelty is high, but its efficiency is questionable.

Questions to Address In Rebuttal

1. On Numeric Modeling: The digit-wise categorical output is a novel formulation. However, other methods of converting regression to classification exist, such as quantizing the entire output range into a set of bins. Could the authors clarify why the progressive, digit-by-digit approach is fundamentally superior to a simpler quantization scheme for this problem? Does the sequential dependency modeled between digits provide a crucial inductive bias?

2. On Dynamic Calibration: The choice of DPO is contemporary and interesting. However, it is one of many possible online learning or RL algorithms. Could the authors justify why DPO is uniquely suited for this task compared to, for instance, a simpler online fine-tuning approach on new data points or other RL algorithms like PPO? Is the "preference" aspect of DPO critical, or is it simply a convenient and effective implementation of RL from feedback?

3. On Data Synthesis: The progressive data synthesizer is presented as a key contribution. To substantiate this claim, a more detailed ablation is needed. Table 7 ablates the entire synthesizer, but not its individual stages. Can the authors provide evidence on the marginal contribution of each stage? Specifically, how much does the final, LLM-based generation stage (Section 6.1, page 8) improve performance over a dataset generated only by the AST-based and dataflow-specific stages? This would clarify the novelty and utility of incorporating LLM-based self-augmentation.