OmniSim: Simulating Hardware with C Speed and RTL Accuracy for High-Level Synthesis Designs

---

Review Form

Reviewer: The Guardian (Adversarial Skeptic)

Summary

The authors present OmniSim, a C-level High-Level Synthesis (HLS) simulation framework. It purports to achieve RTL-level accuracy at near-C speeds for a class of "complex dataflow designs" (termed Type B and C) that are purportedly unsupported by existing commercial and academic tools. The central mechanism involves a multi-threaded simulation model where individual dataflow modules are run in separate "Func Sim" threads. Their execution is orchestrated by a centralized "Perf Sim" thread that resolves hardware timing dependencies (specifically, non-blocking FIFO accesses) by consulting shared FIFO timing tables. The authors evaluate this framework on a set of self-developed benchmarks and claim significant speedups over C/RTL co-simulation and the prior state-of-the-art, LightningSim.

Strengths

1. Problem Taxonomy: The paper's primary strength is the proposed taxonomy of dataflow designs into Type A, B, and C based on module dependency, FIFO access type, and program behavior (Sec. 3, p. 4). This classification provides a useful conceptual framework for discussing the limitations of existing HLS simulation methodologies.

2. Problem Identification: The authors correctly identify a significant and well-known limitation in existing HLS C-level simulation flows, namely their inability to correctly model the functional and performance implications of non-blocking I/O and cyclic dependencies. Commercial tool manuals, as cited by the authors (p. 2), explicitly warn against this.

3. Core Insight: The fundamental insight that functionality and performance simulations are inseparable for Type C designs is sound. The recognition that the functional outcome of a non-blocking access depends on precise cycle timing is a correct diagnosis of the problem.

Weaknesses

1. Evaluation on Self-Curated Benchmarks: The most significant methodological flaw lies in the evaluation of Type B and C designs (Sec. 8.1, p. 10). The authors evaluate their tool on a benchmark suite of their own creation, designed specifically to exhibit the features that OmniSim is built to handle. This introduces a high risk of confirmation bias. The benchmarks, such as fig4_ex2, fig4_ex3, etc., appear to be small, synthetic kernels. Without evaluation on large-scale, pre-existing, and independently developed hardware designs (e.g., from open-source networking, video processing, or accelerator projects), the claims of general applicability are unsubstantiated. The work fails to demonstrate that its solution is robust beyond the tailored test cases.

2. Insufficient Proof of "RTL Accuracy": The paper's core claim of "RTL accuracy" relies on C/RTL co-simulation as the ground truth "oracle". For Type C designs, where behavior is cycle-dependent, this is problematic. A simple match of final output values and total cycle counts (as shown in Table 3 and Fig. 8a, p. 11) is insufficient proof of true cycle-accurate behavioral equivalence. Minor discrepancies in OmniSim's timing model could lead to a different, yet functionally valid, execution path that coincidentally produces the same final result. The paper provides no evidence, such as cycle-by-cycle event trace comparisons, to prove that OmniSim's internal state and module interactions precisely mirror the RTL simulation throughout the entire execution.

3. Unexamined Scalability of the Orchestration Mechanism: The proposed multi-thread orchestration, with a single, central Perf Sim thread processing a global request queue (Fig. 7, p. 8), is a potential architectural bottleneck. As the number of concurrent dataflow modules (and thus Func Sim threads) and the frequency of their communication increase, this single thread is likely to become a serialization point, negating the benefits of parallel simulation. The paper presents a "multicore" benchmark with 34 modules, but provides no sensitivity analysis of how simulation performance degrades as module count or communication density increases. The scalability of the core mechanism is asserted but not proven.

4. Ambiguous Source of Speedup Over LightningSim: The authors claim a 1.26x geomean speedup over LightningSimV2 on its own benchmark suite, which consists of Type A designs (Table 5, p. 12). For these designs, the complex orchestration mechanism of OmniSim should, in theory, represent pure overhead compared to LightningSim's simpler, decoupled approach. The paper attributes the speedup to its "multithreaded architecture" but fails to provide a convincing rationale. It is more plausible that the performance gains stem from unrelated implementation-level optimizations (e.g., the improved graph representation mentioned in Sec. 7.3.1, p. 10) rather than a fundamental architectural advantage for this class of designs. This confounds the evaluation of the paper's core contribution with general implementation quality.

Questions to Address In Rebuttal

1. Benchmark Representativeness: Can the authors provide evidence that the custom Type B and C benchmarks are representative of real-world, complex HLS designs? To substantiate the claims of general utility, can the authors demonstrate OmniSim's correctness and performance on at least one large, publicly available, complex dataflow HLS project that was not developed by the authors?

2. Verification of Accuracy: Beyond matching final cycle counts and output values, what specific steps were taken to verify that the sequence of all FIFO events and state transitions in OmniSim perfectly matches the C/RTL co-simulation on a cycle-by-cycle basis for a complex Type C design like fig4_ex5 or branch? Please provide data (e.g., from a trace diff) to support this.

3. Orchestration Scalability: Please provide data showing how OmniSim's simulation time scales as a function of (a) the number of concurrent modules and (b) the rate of non-blocking FIFO accesses. At what point does the central Perf Sim thread become a bottleneck, and what is the performance characteristic of this bottleneck?

4. Incremental Simulation Efficacy: The incremental simulation capability (Sec. 7.2, p. 10) is a key feature. In a realistic design space exploration (DSE) scenario sweeping through dozens of FIFO size configurations for a complex design, what percentage of incremental runs succeed versus failing due to a "constraint violation" that forces a full re-simulation? An assertion of capability is insufficient; data on its practical effectiveness is required.

Review Form:

Reviewer: The Synthesizer (Contextual Analyst)

Summary

This paper introduces OmniSim, a simulation framework for High-Level Synthesis (HLS) that aims to provide the cycle-accuracy of RTL simulation at speeds approaching native C execution. The authors identify a critical gap in existing HLS flows: the inability to correctly and efficiently simulate complex "dataflow" designs at the C-level, particularly those involving non-blocking FIFO accesses, cyclic dependencies, and data-dependent control flow (which they classify as Type B and C designs).

The core contribution is a novel simulation methodology that tightly couples functionality and performance simulation. It employs a multi-threaded architecture where dedicated "Func Sim" threads simulate individual hardware modules, and a central "Perf Sim" thread orchestrates their execution. This orchestration is key; it maintains hardware-accurate timing information in shared "FIFO tables," allowing functional threads to query the exact state of the system at a specific hardware cycle, thereby resolving the ambiguity that plagues traditional C simulation and OS-based thread scheduling. The authors demonstrate that OmniSim can successfully simulate a suite of designs that are explicitly unsupported by existing commercial and academic tools, achieving significant speedups over the traditional C/RTL co-simulation flow and even outperforming the state-of-the-art LightningSim on its own benchmarks.

Strengths

1. Excellent Problem Formulation and Taxonomy: The paper’s most significant intellectual contribution, beyond the tool itself, is the clear taxonomy of dataflow designs into Type A, B, and C (Section 3, pages 3-5). This classification provides a principled and much-needed framework for understanding why C-level simulation is so challenging. It brilliantly articulates the progression from concurrency-independence (Type A) to concurrency-dependence (Type B) and finally to full cycle-dependence for functionality (Type C). This framing contextualizes the entire field of HLS simulation and makes the need for a solution like OmniSim self-evident.

2. Elegant and Novel Simulation Architecture: The proposed solution—a centrally orchestrated, multi-threaded simulation model—is an elegant answer to the challenges laid out in the taxonomy. Instead of treating functionality and performance as decoupled phases (as in prior work like LightningSim), OmniSim recognizes their inherent coupling in complex designs. The concept of a Perf Sim thread serving as a "source of truth" for hardware timing, which Func Sim threads can query on demand (Figures 6 and 7, pages 7-8), is a powerful abstraction that resolves the fundamental mismatch between software scheduling and hardware timing.

3. Compelling and Direct Evaluation: The evaluation is highly effective because it directly targets the claimed contribution. By developing a benchmark suite of previously "unsimulatable" designs (Table 4, page 11) and showing that OmniSim produces correct results where standard C-sim fails or crashes (Table 3, page 11), the authors provide irrefutable evidence of their system's extended capability. Furthermore, demonstrating a significant performance advantage over LightningSim on its own established benchmarks (Table 5, page 12) is a masterstroke; it proves that OmniSim's more general and powerful architecture does not come at the cost of performance for simpler designs—in fact, it enhances it.

4. Practical, Real-World Relevance: The work addresses a well-known and painful bottleneck for HLS practitioners. The productivity gains promised by HLS are often squandered in slow, cumbersome RTL verification loops. By enabling rapid and accurate design space exploration before RTL generation for a wider class of designs, OmniSim has the potential to fundamentally improve the HLS workflow. Features like deadlock detection (Section 7.1, page 10) and efficient incremental simulation (Section 7.2, page 10) further underscore the authors' focus on practical utility.

Weaknesses

1. Benchmark Representativeness: The authors rightly note that finding real-world, open-source Type B and C designs is difficult precisely because poor tool support discourages their creation. While the handcrafted benchmarks are perfectly suited to demonstrate the mechanism and prove correctness, the paper would be strengthened by a discussion of how these patterns manifest in larger, more systemic applications. The multicore example is a good start, but the work's impact will ultimately be judged by its ability to handle, for instance, a complete network-on-chip with adaptive routing or a processor with a non-trivial out-of-order execution core.

2. Scalability of the Centralized Orchestrator: The Perf Sim thread acts as a centralized bottleneck by design, processing requests from a single queue to ensure correctness. While this is clearly effective for the designs presented, it raises questions about scalability. In a hypothetical future design with hundreds or thousands of concurrently active dataflow modules, could this centralized Perf Sim thread become the limiting factor for simulation performance? The paper does not explore the performance limits of this architectural choice.

3. Positioning Relative to Discrete-Event Simulation: The underlying mechanism of OmniSim bears a strong resemblance to established principles of Parallel Discrete-Event Simulation (PDES). The Perf Sim thread acts as a global event queue manager, and the Func Sim threads are logical processes. While the application to HLS C-level simulation is novel and highly valuable, framing the work within this broader simulation literature could provide deeper theoretical grounding and potentially suggest further optimizations (e.g., optimistic execution).

Questions to Address In Rebuttal

1. Regarding the scalability of the Perf Sim thread: Have the authors characterized the overhead of the query/resolve mechanism? At what number of concurrent modules or frequency of non-blocking accesses do you project the centralized orchestration becoming a performance bottleneck, and are there potential paths to parallelizing the Perf Sim thread's logic in the future?

2. The ability to simulate Type C designs opens the door to using HLS for more dynamic hardware architectures. Could the authors elaborate on a specific, large-scale application (e.g., a cache coherence protocol, a network router) that is currently infeasible to design in HLS due to verification challenges, and walk through how OmniSim would specifically enable its development?

3. The quality of debugging is paramount for productivity. When OmniSim detects a functional error or a design deadlock, what kind of feedback and state information does it provide to the designer? How does this compare to the (often cryptic) experience of debugging a hung C/RTL co-simulation?

---

Review Form

Reviewer: The Innovator (Novelty Specialist)

Summary

The authors present OmniSim, a framework for High-Level Synthesis (HLS) simulation that aims to provide RTL-level accuracy at near-C speeds, specifically for complex dataflow designs involving non-blocking (NB) FIFO accesses and cyclic dependencies. The central novelty claim lies in its simulation architecture: a tightly-coupled, multi-threaded model where functionality threads (representing HLS modules) are orchestrated by a dedicated performance simulation thread that maintains cycle-accurate hardware state via a set of "FIFO tables." This architecture is designed to correctly resolve the functional behavior of designs where correctness is dependent on the precise cycle of an operation (e.g., an NB FIFO access), a problem explicitly identified as a limitation in prior state-of-the-art simulators like LightningSim and commercial HLS tools.

My analysis concludes that this architectural approach, while having conceptual parallels to classic discrete-event simulation, is a novel and significant contribution in the specific context of HLS C-level simulation. The key innovation is the mechanism for on-demand, cycle-accurate state querying from concurrent software threads to correctly model hardware concurrency, which has not been demonstrated in prior HLS simulation literature.

Strengths

1. Core Architectural Novelty: The primary strength of this paper is the novelty of the core simulation engine. The state-of-the-art, exemplified by LightningSim [19, 20], relies on a fully decoupled, two-phase approach (trace generation then performance analysis). This is fundamentally incapable of simulating designs where functionality depends on runtime performance (i.e., cycle timing). OmniSim's proposal to "flexibly couple" these phases via a dedicated orchestrator thread (the "Perf Sim thread," detailed in Sections 5 and 6) is a genuinely new architecture in this space. It moves beyond post-hoc trace analysis to a live, interactive simulation model.

2. Novel Problem Formulation: The taxonomy of dataflow designs into Type A, B, and C (Section 3, Figure 4) is a valuable and novel contribution in its own right. It provides a clear, systematic framework for understanding why certain HLS designs are hard to simulate at the C-level. To my knowledge, such a formal classification and its direct mapping to simulation requirements (concurrency- and cycle-dependence) has not been previously published. This taxonomy effectively carves out the design space where OmniSim's novelty is required.

3. Mechanism for Resolving Cycle-Dependent Functionality: The specific mechanism of using a centralized Perf Sim thread to maintain FIFO R/W Tables and resolve queries from Func Sim threads (Figure 7) is the concrete implementation of the architectural novelty. This is not merely an application of multi-threading; it is a sophisticated orchestration that correctly serializes access to shared state (FIFO status) based on a modeled hardware timeline, not the host OS's arbitrary thread scheduling. This solution directly addresses the core challenge illustrated in Figure 2.

Weaknesses

My concerns are not with a lack of novelty, but rather with the paper's positioning of its novel ideas relative to broader, established concepts in computer science.

1. Absence of Context within Classical Simulation Paradigms: The paper does not explicitly position its multi-threaded orchestration mechanism within the broader context of classical simulation paradigms, such as Discrete-Event Simulation (DES). The proposed Perf Sim thread acts as a central event scheduler, processing a queue of requests and advancing a logical clock (cycle count), which is a core concept in DES. While the application to HLS is novel and the specific implementation with live software threads is unique, the lack of discussion on these conceptual underpinnings is a missed opportunity. Situating OmniSim within this established theoretical framework would strengthen the paper's contribution by highlighting how it specializes or adapts these general principles for the HLS domain.

2. Scalability of the Centralized Orchestrator: The reliance on a single, centralized Perf Sim thread to resolve all queries raises questions about the scalability of this novel approach. For designs with hundreds or thousands of concurrent HLS modules (e.g., large-scale network-on-chip simulations), this single thread could become a significant performance bottleneck, as it must serially process requests from all other threads. While the approach is novel, its practical limits are not explored. The novelty of the solution introduces a new potential performance characteristic that warrants analysis.

3. Incremental Simulation Novelty Overstated: The paper claims "incremental simulation" as a feature (Section 7.2). While the technique of memoizing query outcomes ("constraints") and re-validating them is clever, its novelty is a delta on top of the incremental simulation capability already present in LightningSim. LightningSim's ability to reuse a simulation graph for new FIFO sizes is the foundational idea. OmniSim's contribution is to make this work for its more complex, coupled simulation model. This is an important engineering extension, but it is not as fundamentally novel as the core simulation architecture.

Questions to Address In Rebuttal

1. Please discuss the relationship between OmniSim's orchestration model and established paradigms like Discrete-Event Simulation (DES). How does your contribution differ from or specialize these general concepts for the HLS domain?

2. The Perf Sim thread appears to be a serialization point in your novel architecture. Could you provide an analysis or data on its potential to become a performance bottleneck for designs with a very large number of concurrent modules and high-frequency non-blocking accesses?

3. The concept of "FIFO tables" (Section 6.2, structure D in Figure 7) is central to your implementation. How novel is this specific data structure for tracking hardware state in a software simulation? Are there analogous structures in other concurrent system simulators (e.g., in architectural simulation or parallel system modeling) that you can compare it against to better highlight the novelty of your specific formulation?

Rishov (the main author) and I (a colleague and frequent collaborator of Rishov and user of LightningSim/OmniSim) have read this AI review for the first time and have already identified several critical issues with these AI reviews, even in the context of being used as a starting point for reviewers. However, this write up is mainly by me (Stefan) and mostly includes my own biases, analysis, and opinions.

---

1. LLM Denial-of-Effort Attack on the Author

One main criticism I have is that the LLMs are effectively performing a "denial-of-effort" attack on the human author. What I mean by this is that the AI can effectively list endless criticisms (including detailed criticism) of a work or aspects of the work the author did not include in the paper but could potentially expand upon.

For example, an AI reviewer can suggest ablations of aspects of the work or more detailed explanations to cover aspects not included in the paper. This might also include things like more data, more figures, more discussion, or more thorough theoretical analysis of any given aspect of the work. With this bias, a human reviewer might look at an LLM's analysis of a paper with a list of these "missing" experiments, results, or analysis and be biased toward the idea that the authors missed critical data or analysis.

In reality, there are practical limitations that make this impossible, such as page limits or an author's own assessment of what ideas are important to discuss, analyze, and present with experimental results, and which are not. Human authors deliberately curate which results they believe are sufficient and comprehensive to support their claims, and this editorial judgment is a core part of scientific writing. An LLM, by contrast, lacks the deep contextual understanding to know which findings are truly critical versus tangential. In practice, this can lead to a situation where authors have written a really high quality paper, the LLM provides an initial review with a list of details, analysis, and experiments the authors could add or might have missed, and a reviewer using this LLM analysis without reading the work critically makes a poor judgment of the paper.

This effect is magnified by the "The Guardian" persona, which is explicitly prompted to be overly critical and biased to always list things that authors might not have included. The same is still somewhat true for other personas, but they are still biased toward discussing limitations which might not be true limitations or relevant to the scope and page limit of a work.

Rather than having a human critically analyze a work and determine what it does and does not cover and whether that is sufficient for the ideas presented, this responsibility is shifted to an LLM to provide an initial review which is always biased, and then further biases a human who relies on this LLM review to make the same assessment of scope and missing/included implications and whether they are relevant.

1. Overly Harsh Criticism and Assertive Statements That Are Not True

LLM reviews of this work tend to make assertive statements and overly critical claims about certain aspects of the work that are not true, both from "The Guardian" persona but also somewhat from other personas.

As an example, one claim made in the LLM analysis of this work is as follows: "Without evaluation on large-scale, pre-existing, and independently developed hardware designs (e.g., from open-source networking, video processing, or accelerator projects), the claims of general applicability are unsubstantiated. The work fails to demonstrate that its solution is robust beyond the tailored test cases." The claim that this limitation of the evaluation means "the work fails to demonstrate that its solution is robust beyond the tailored test cases" is untrue. Although Rishov in this case had to construct a manually curated limited set of synthesized designs of type B and C to evaluate, this does not mean the work does not generalize beyond these cases. In the paper, Rishov explicitly discusses this limitation, noting that is is practically difficult to find application-level HLS designs and/or HLS benchmarks of type B and C that are open source / publicly available. What is the alternative to what is proposed in the paper? Perform no evaluation of type B and C designs at all? This criticism from the LLM makes no sense.

Furthermore, even with the evaluation of synthetically curated designs of type B and C, we can generalize with high confidence that the tool will generalize to other type B and C designs in the real world. One way to look at this point: OmniSim is a program analysis and simulation tool, designed to be able to analyze and simulate arbitrary HLS programs of certain classes. In this regard, it would be silly to assume that if we support HLS programs of type B and C, this would apply only to the specific set of designs curated in a paper, not to all possible HLS programs of that type. This is like saying "my tool analyzes Python programs of a certain type, but it only works on these 10 Python programs of that type and we can't claim anything about other programs of that type." However, this is what the LLM analysis is doing, and it is wrong in this case.

Furthermore, like the first point, even with the LLM analysis as a starting point, this will bias reviewers who see these claims and do not analyze the paper fully or critically. We cannot allow reviewers to be biased in this manner or even rely on LLMs to review works, since it incentivizes reviews to take these claims at face value or use them to bias their reading of a work.

1. Confidently Misunderstanding the Authors' Work and Critical Technical Details

LLM reviewers also have fundamental misunderstandings of the authors' work, technical context, and technical details of the work, and in turn extrapolate this misunderstanding to points made in their analysis.

For OmniSim, one aspect that "The Synthesizer" and "The Innovator" reviewers misunderstand is the underlying context for how OmniSim does its simulation and how that relates to discrete event simulation. In OmniSim and the series of LightningSim work in general, one of the core ideas is using a graph representation of modules and FIFOs to perform timing analysis and deadlock analysis of the design being analyzed and simulated. This graph is combined with a trace of the design to then compute the final hierarchical latencies and deadlock conditions. Although some aspects of this could possibly be related to discrete event simulation, this is not a discrete event simulator and does not really share core ideas of discrete event simulation approaches. The LLMs then suggest that there should be more included discussion and analysis of how this relates to discrete event simulation, when in reality it likely only merits very brief discussion and maybe a single citation. And in some cases, the link might be interesting, yet it is better for the author to focus the narrative and main text on other points of motivation and discussion.

Like with previous criticism, this kind of analysis and misunderstanding of the authors' context and technical details can bias and mislead reviewers into thinking authors are not doing proper background analysis or are misunderstanding their technical context.

---

It is clear from this initial prototype of this AI review system being applied to this work that LLMs should not be used for peer review, even as initial analysis or supplemental analysis for human reviewers. LLMs make fundamental flaws, as pointed out earlier, that amplify biases toward reviewers about work on aspects that cannot fit within the limitations of a work, are not true, or are misunderstandings of the technical context and core concepts of the work. These systems should not be used for real peer review unless all these issues can be fully and comprehensively resolved, which I do not think will be reasonable in the near future even with further development and refinement of systems like the one here.

It is hard to criticize and denigrate these AI reviewer or reviewer assistant systems without providing a solution or inviting feedback about being hypocritical. However, this does not excuse the fact that the criticisms are still valid and we should not use systems like this, even if we do not have alternatives. The point still stands that these systems should not be used based on our analysis.

Nonetheless, I will offer the idea that rather than focusing efforts on building systems like the one prototyped here, we should put that same effort and development into improving the peer review system in other aspects. For example, requiring things like reciprocal reviewer requirements for submitting authors, better artifact evaluation requirements and requirements for certain kinds of works, self-ranking of submissions for authors with multiple submissions, better reviewer attribute and credit, and better tracking of reviewer participation across multiple conferences and years in the community. I think looking into different ways to improve the peer review process beyond just AI-assisted reviews can provide much more significant improvements to the quality and amount of peer review that can be done in both the near and long term, providing more value and benefits to the research community overall.