Cinnamon: A Framework for Scale-Out Encrypted AI

Reviewer Persona: The Guardian (Adversarial Skeptic)

---

Summary

The paper presents Cinnamon, a cross-stack framework designed to accelerate large-scale machine learning workloads under Fully Homomorphic Encryption (FHE). The authors propose a scale-out, multi-chip architecture as an alternative to the prevailing trend of large monolithic FHE accelerators. The core contributions are: (1) novel parallel keyswitching algorithms ("Input Broadcast" and "Output Aggregation") intended to reduce inter-chip communication, (2) a compiler infrastructure with a DSL to manage program- and limb-level parallelism, and (3) a space-optimized hardware Base Conversion Unit (BCU). The authors claim significant performance improvements, most notably a 36,600x speedup for BERT inference over a CPU baseline and a 2.3x improvement over prior art for smaller benchmarks.

While the problem addressed is of significant importance and the proposed approach is comprehensive, the work rests on several claims that are either insufficiently substantiated or compared against potentially weak baselines. The experimental evaluation, particularly concerning comparisons to prior art and CPU performance, lacks the rigor necessary to fully validate the claimed contributions.

Strengths

1. Problem Significance: The paper correctly identifies a critical bottleneck in the field: the inability of monolithic FHE accelerator designs to keep pace with the exponential growth of ML models (as illustrated in Figure 1). The focus on enabling large models like BERT is timely and ambitious.
2. Comprehensive Approach: The work spans the full stack from algorithms and a compiler to the microarchitecture. This holistic view is commendable, as optimizations at a single level are often insufficient in the FHE domain.
3. Focus on Communication: The central thesis—that reducing communication overhead in limb-level parallel keyswitching is key to a viable scale-out architecture—is fundamentally sound. The identification of keyswitching as the primary obstacle to distributed FHE computation is correct.

Weaknesses

1. Vague and Potentially Unfair CPU Baseline: The headline claim of a 36,600x speedup for BERT (Table 2, page 12) is predicated on a comparison to a "48-core Intel Xeon with a 256GB Memory CPU." This description is critically insufficient. The specific CPU model, clock frequency, and—most importantly—the FHE library (e.g., SEAL, HEAAN, Lattigo) and its version are not specified. Furthermore, it is not stated whether the CPU implementation was optimized to leverage all 48 cores for parallel FHE operations. State-of-the-art FHE libraries have seen significant performance improvements. Without these details, the reported speedup is impossible to verify and may be substantially inflated due to comparison against a non-optimized or outdated baseline.
2. Questionable Characterization of Prior Art (CiFHER): The paper consistently positions its keyswitching algorithms as superior to the approach in CiFHER [38]. The central argument is that CiFHER requires broadcasts at both the mod up and mod down steps, incurring high communication overhead. In Section 7.4 (page 13), the authors claim their method reduces inter-chip communication by 2.25x over CiFHER. However, the analysis lacks depth. It is not clear if the CiFHER implementation used for comparison represents the most optimized version of its broadcast-based reduction scheme. Figure 13 (page 12) shows CiFHER resulting in a slowdown over a sequential single-chip implementation, which is a very strong and somewhat counterintuitive claim that suggests the baseline comparison may not be entirely fair. The algorithmic analysis in Section 7.4 feels simplistic and may mischaracterize the trade-offs made in the CiFHER design.
3. Highly Optimistic Cost Model: The performance-per-dollar analysis in Section 7.2 (page 11) and Table 3 (page 12) relies on a manufacturing yield model with a defect density of Do = 0.2cm⁻². This is an extremely optimistic parameter, especially for a large, complex ASIC on a 22nm process, which is mature but not leading-edge for such designs. This choice of parameter significantly favors the smaller-chip approach of Cinnamon over the larger monolithic designs (Cinnamon-M, CraterLake), potentially exaggerating the cost benefits. The conclusions drawn from this analysis are therefore fragile and sensitive to this key assumption.
4. Impact of the "Space-Optimized" BCU is Not Quantified: The paper introduces a novel BCU design in Section 4.7 (page 9), claiming it reduces area by being proportional to the number of input limbs rather than output limbs. While the design rationale is plausible, its actual impact is never isolated or quantified. Table 1 (page 10) shows the area breakdown, but there is no ablation study or comparison showing how much performance-per-dollar or total cost is actually improved by this specific unit versus, for example, using a CraterLake-style BCU within the Cinnamon scale-out framework. As such, the BCU feels like a secondary contribution whose significance to the overall system-level claims is unproven.
5. Unexamined Scalability Limits of the "Input Broadcast" Algorithm: The "Input Broadcast Keyswitching" algorithm (Figure 8b, page 7) requires broadcasting the entire input polynomial Co to all chips. A ciphertext polynomial Co is a large data structure. The paper evaluates systems with up to 12 chips. The cost of this full broadcast seems tenable for this scale, but the paper fails to analyze its asymptotic complexity or practical limitations as the number of chips scales to 16, 32, or beyond. It is plausible that this broadcast itself becomes the new system bottleneck at a larger scale, limiting the very "scale-out" potential the framework claims to provide.

Questions to Address In Rebuttal

1. Regarding the CPU Baseline: Please provide the exact model of the Intel Xeon CPU, its clock speed, the FHE library and version used for the BERT benchmark, and confirm whether the FHE computation was explicitly parallelized across all 48 cores.
2. Regarding the CiFHER Comparison: Please clarify the specific implementation of the CiFHER parallel keyswitching algorithm used for comparison in Figure 13. Can you provide evidence that this implementation is a fair and optimized representation of the approach described in the original CiFHER paper [38]? Why does it result in a net slowdown?
3. Regarding the Cost Model: Can you justify the choice of Do = 0.2cm⁻² for the yield model? Please provide a sensitivity analysis showing how the performance-per-dollar results in Figure 12 change with more conservative (i.e., higher) defect density parameters, for instance, Do = 0.4cm⁻² or 0.6cm⁻².
4. Regarding the BCU Contribution: Please quantify the end-to-end impact of your space-optimized BCU. How would the overall chip area, cost, and performance-per-dollar of a Cinnamon-4 system change if it were to use a scaled-down version of the BCU from CraterLake [56] instead of your proposed design?
5. Regarding Algorithmic Scalability: Please provide an analysis of the communication complexity of the "Input Broadcast Keyswitching" algorithm as a function of the number of chips (n). At what value of n do you project the initial broadcast of Co to become a limiting factor for performance?

Paper: Cinnamon: A Framework for Scale-Out Encrypted AI
Reviewer: The Synthesizer (Contextual Analyst)

---

Summary

This paper presents Cinnamon, a full-stack framework designed to accelerate large-scale machine learning workloads under Fully Homomorphic Encryption (FHE). The authors identify a critical divergence: while ML models are growing exponentially in size and complexity, FHE hardware accelerators have pursued a "scale-up" strategy, resulting in massive, monolithic chips that are already failing to keep pace.

The core contribution of Cinnamon is to reject this monolithic paradigm in favor of a "scale-out" approach. This is not merely a hardware proposal but a holistic co-design spanning a new Python DSL, compiler infrastructure with novel intermediate representations (IRs), innovative parallel algorithms for communication-intensive FHE primitives (notably keyswitching), and a scalable multi-chip hardware architecture. By tackling parallelism at every level of the stack, Cinnamon demonstrates, for the first time, the feasibility of running a large language model like BERT under FHE with practical inference times, a feat unattainable with prior state-of-the-art designs. The work effectively argues that the future of practical FHE acceleration lies in distributed, composable systems rather than in building ever-larger single chips.

Strengths

1. A Necessary and Timely Paradigm Shift: The paper's most significant strength is its central thesis. The authors correctly diagnose that the monolithic, scale-up approach in FHE acceleration is on an unsustainable trajectory, a point powerfully illustrated by their Figure 1 (page 2). By drawing a parallel to the broader history of computing—from single-core frequency scaling to multi-core processors, and from single large machines to distributed HPC clusters—Cinnamon positions itself as a crucial and timely architectural intervention. It provides a compelling vision for how the field can escape the design-cost-yield trap of monolithic chips.

2. Holistic, Cross-Stack Co-Design: This is not just a paper about a faster interconnect or a clever hardware unit. The true innovation lies in the tight integration of solutions across the entire stack. The novel parallel keyswitching algorithms ("Input Broadcast" and "Output Aggregation," Section 4.3.1, page 7) are the theoretical key. The compiler's "Keyswitch Pass" is what makes these algorithms practical by automatically identifying patterns and batching communication. The scale-out hardware is then purposefully designed to provide the specific communication primitives (broadcast, aggregate) that the algorithms require. This synergy is what overcomes the communication bottleneck that limited previous multi-chip/chiplet attempts like CiFHER, and it is a masterclass in system co-design.

3. Demonstration of a Breakthrough Capability: While the speedups on smaller benchmarks are impressive (e.g., 2.3x over SOTA), the qualitative result of running BERT is the paper's crown jewel. By reducing a 17-hour CPU computation to a 1.67-second inference on Cinnamon-12, the authors have fundamentally shifted the goalposts for what is considered a "tractable" FHE workload. This moves privacy-preserving ML for large models from a distant theoretical possibility to a tangible engineering problem. This result alone has the potential to energize the field and attract new research and commercial interest.

4. Pragmatic Economic and Architectural Arguments: The authors supplement their performance claims with a solid analysis of manufacturing costs and yields (Section 7.2, Table 3, page 12). The argument that a system of smaller, higher-yield chips is more economically viable than one massive, low-yield chip is critically important for translating academic research into real-world technology. Furthermore, the introduction of a space-optimized Base Conversion Unit (BCU) in Section 4.7 (page 9) shows a thoughtful, bottom-up approach to reducing the area and power of each individual chip in the scale-out system, reinforcing the overall design philosophy.

Weaknesses

While this is a strong and impactful paper, its positioning within the literature and the exploration of its limitations could be strengthened.

1. Absence of a Direct SOTA Comparison on BERT: The paper's most compelling result—accelerating BERT—is evaluated only across Cinnamon configurations. While the authors rightly imply that prior monolithic accelerators cannot run BERT due to memory limitations, this could be made more explicit and quantitative. A projection of the required on-chip cache and estimated die area for a monolithic design (e.g., CraterLake or ARK) to handle BERT would provide a powerful, even if theoretical, baseline that would further underscore the necessity of the scale-out approach.

2. Generalizability Beyond Embarrassingly Parallel Workloads: BERT, with its attention and feed-forward layers, contains significant opportunities for data parallelism, which the Cinnamon framework exploits beautifully. However, the paper does not discuss how the framework would perform on FHE workloads with more complex, serial dependency graphs where program-level parallelism is scarce. The reliance on user-provided parallel streams via the DSL suggests that performance is heavily tied to the application's structure. A discussion of performance on a less parallelizable FHE algorithm would help to better define the framework's application scope.

3. Scalability of the Network Topology: The evaluation explores systems of up to 12 chips using ring and switch topologies. For a true scale-out vision, it is important to consider the next order of magnitude. How do the communication costs of the proposed keyswitching algorithms scale as the system grows to 32, 64, or more chips? At some point, the all-to-all nature of aggregation and broadcast on simple networks can become a bottleneck. A brief analysis of the network scalability would strengthen the long-term vision of the work.

Questions to Address In Rebuttal

1. To strengthen the headline BERT result, could the authors provide a more direct, even if theoretical, comparison against a scaled-up monolithic architecture? For instance, what would be the projected on-chip memory requirements and die size for a "Cinnamon-M" style chip to handle BERT, and how would this impact its manufacturing feasibility and cost according to your model in Section 7.2?

2. The BERT benchmark showcases significant data parallelism. Could the authors comment on how Cinnamon's performance and parallelization strategies would apply to FHE applications with more intricate, sequential dependency graphs where program-level parallelism is less abundant? Does the framework's effectiveness hinge on the availability of such parallelism?

3. Your evaluation focuses on up to 12 chips. Have you analyzed the potential communication bottlenecks in the proposed ring or switch topologies when scaling to significantly larger systems (e.g., 32+ chips)? How do the specific broadcast/aggregate communication patterns of your parallel FHE algorithms scale with these network designs?

Reviewer Persona: The Innovator

Summary

This paper presents Cinnamon, a co-designed framework consisting of new algorithms, a compiler, and a scale-out hardware architecture for accelerating large-scale machine learning workloads under Fully Homomorphic Encryption (FHE). The central thesis is that the traditional monolithic "scale-up" approach for FHE acceleration is not sustainable. Instead, the authors propose a "scale-out" approach using multiple smaller, more cost-effective chips.

The core of the claimed novelty lies in two areas:

1. Algorithmic/Compiler: A set of new parallel keyswitching algorithms ("Input Broadcast" and "Output Aggregation") and compiler passes designed to minimize the inter-chip communication that has historically been the primary obstacle to efficient limb-level parallelism.
2. Architectural: A space-optimized Base Conversion Unit (BCU) that reduces chip area by exploiting specific characteristics of FHE workloads.

The framework is used to demonstrate, for the first time, a practical inference time for a BERT-sized model, which serves as evidence for the efficacy of the proposed scale-out methodology.

Strengths

The paper's primary strength lies in its identification and proposed solution for the communication bottleneck in limb-level parallel FHE. While the concepts of program-level and limb-level parallelism are not new in themselves, the specific techniques developed here represent a genuine advancement over the prior art.

1. Novel Parallel Keyswitching Algorithms: The most significant contribution is the design of the "Input Broadcast" and "Output Aggregation" keyswitching algorithms (Section 4.3.1, page 7). Prior work in multi-chip FHE, notably CiFHER [38], relied on extensive broadcasting of data, which does not scale well with increasing communication latency or bandwidth constraints. Cinnamon's approach of strategically choosing a single communication point (either at the beginning or end of the operation) and then using compiler transformations to batch these communication events across many operations is a fundamentally new and more scalable method. The algorithmic analysis in Section 7.4, which argues for a reduction in communication complexity from O(r) to O(1) for batched rotations, clearly articulates this novel delta.

2. Novel Domain-Specific Microarchitecture: The design of the space-optimized Base Conversion Unit (BCU) is a clever and novel architectural contribution (Section 4.7, page 9). Prior designs like CraterLake [56] apparently followed a general-purpose, output-buffered approach. The authors correctly identify that FHE base conversions are asymmetric (few input limbs to many output limbs) and exploit this by designing an input-buffered unit. This insight leads to a direct and substantial reduction in the required logic and SRAM resources, making the individual chips in the scale-out system more area- and cost-efficient. This is a prime example of a valuable, domain-specific hardware optimization.

3. Co-design of Compiler and Algorithms: The paper presents a well-realized co-design. The parallel algorithms were clearly designed with the awareness that a compiler could reorder and batch operations, and the "Cinnamon Keyswitch Pass" (Section 4.3.1) is the embodiment of this synergy. This tight integration is what elevates the work from a collection of point optimizations to a cohesive and novel framework.

Weaknesses

From a novelty perspective, the weaknesses are primarily in areas where the contributions are more evolutionary than revolutionary, or where the framing could be sharpened to better distinguish from existing concepts.

1. Incremental Nature of Program-Level Parallelism Abstractions: The use of a Python DSL and concurrent execution streams (Section 4.2) to express program-level parallelism is a standard practice in the broader parallel computing domain. While its application to FHE is necessary for the framework, the abstraction itself is not fundamentally new. The novelty is less in the DSL and more in how the compiler backend maps these streams onto the scale-out hardware using the novel limb-level parallel techniques.

2. Scale-Out Concept Follows Prior Art: The paper is correctly motivated by the need to move from "scale-up" monolithic chips (e.g., CraterLake) to "scale-out" multi-chip systems. However, the first step in this direction within FHE hardware was taken by CiFHER [38], which introduced a chiplet-based design. Therefore, Cinnamon's core idea is not to scale out, but rather how to scale out efficiently. The paper is mostly clear on this, but the high-level framing should consistently emphasize that the novelty is in the enabling mechanisms (the new algorithms) that make scaling out practical, rather than the idea of scaling out itself.

3. Demonstration of BERT is an Application, Not a Core Novelty: The paper rightly highlights the impressive achievement of running BERT inference in 1.67 seconds. However, this is an experimental result that validates the framework's novelty; it is not, in itself, a novel conceptual contribution. This result stems directly from the novel algorithms and architecture and should be presented as such, rather than as a standalone claim of novelty.

Questions to Address In Rebuttal

1. The proposed keyswitching algorithms trade communication for some duplicated computation and storage of extension limbs. Could the authors provide a more formal analysis of this trade-off? Specifically, is there a crossover point in terms of the number of chips, network bandwidth, or FHE parameters where the overhead of this duplication would outweigh the communication benefits, potentially favoring a CiFHER-like broadcast approach?

2. The "Input Broadcast" keyswitching algorithm (Figure 8b) appears to require each chip to store a full copy of the input polynomial CQ after the broadcast. Given that ciphertexts can be large, does this place a significant new memory capacity requirement on each chip compared to prior approaches, and how does this scale with the number of limbs?

3. The concept of reordering and batching communication is related to hoisting techniques described in the context of software libraries like HElib [28]. Can you more precisely delineate the novelty of your compiler's "Keyswitch Pass" from the principles used in prior software-based FHE optimization, especially in how it handles the explicit costs of a distributed multi-chip system?