Athena: Accelerating Quantized Convolutional Neural Networks under Fully Homomorphic Encryption

---

Review Form

Reviewer: The Guardian (Adverserial Skeptic)

Summary

The paper proposes "Athena," a framework and co-designed hardware accelerator for executing quantized convolutional neural networks (CNNs) under fully homomorphic encryption (FHE). The core methodological deviation from prior work is the rejection of the conventional CKKS scheme for approximate arithmetic in favor of an integer-based FHE scheme (akin to BFV). Non-linear operations (i.e., activation functions) are handled via a "Functional Bootstrapping" (FBS) mechanism, which is effectively a polynomial evaluation of a Look-Up Table (LUT). The authors claim this approach allows for much smaller cryptographic parameters, resulting in significant speedups (1.5x-2.3x) and EDP improvements (3.8x-9.9x) over state-of-the-art CKKS-based FHE accelerators, with negligible accuracy degradation relative to a quantized plaintext baseline.

Strengths

1. Problem Simplification: The fundamental premise of leveraging model quantization to simplify the underlying cryptographic problem is sound. Moving from the complexities of approximate arithmetic in CKKS to integer arithmetic is a valid research direction that could plausibly reduce overhead.
2. Hardware-Software Co-design: The authors have clearly considered the interplay between their proposed framework and the hardware needed to execute it. The design of specialized units like the FRU to accelerate the specific bottlenecks of their framework (namely FBS) demonstrates a coherent design philosophy.

Weaknesses

My primary concerns with this submission revolve around the rigor of the analysis, the fairness of the experimental comparisons, and the overstatement of the framework's generality and novelty.

1. Insufficient Noise Analysis: The noise analysis presented is superficial and unconvincing. In Section 3.2.2 (p. 5), the authors introduce a rounding noise ems during modulus switching. They claim it has "minimal impact" because it contaminates LSBs. Figure 4 (p. 7) shows that this "minimal" impact results in data error ratios of up to 11% in some layers of ResNet-56. A claim that this level of error has no significant cumulative effect on a deep network is extraordinary and requires extraordinary proof, which is not provided. The analysis lacks any formal treatment of how this error propagates and accumulates across dozens of layers. Relying on final accuracy metrics for just a few models is insufficient to validate the robustness of this noise-handling approach.

2. Misleading Performance Comparisons: The performance evaluation in Section 5.2.2 (p. 10) constitutes a severe apples-to-oranges comparison. The baseline accelerators (CraterLake, ARK, SHARP) are designed to handle the significantly more complex and general-purpose CKKS scheme, which includes computationally intensive operations for bootstrapping and complex number arithmetic. Athena, by design, targets a much simpler cryptographic problem (integer-only arithmetic). It is therefore unsurprising that a specialized accelerator for a simpler problem outperforms a general-purpose accelerator for a harder one. Figure 8 (p. 11), which shows that CKKS accelerators are ill-suited for the Athena workload, does not support the authors' claims of superiority; it merely states the obvious. A fair comparison would involve implementing the Athena framework on a configurable platform against a CKKS implementation on the same platform, or comparing against a hypothetical CKKS accelerator that is also specialized for quantized workloads.

3. Questionable Novelty and Scalability of "Functional Bootstrapping" (FBS): The FBS mechanism, described in Section 3.2.3 (p. 6), is presented as a key innovation. However, it is fundamentally a known technique: evaluating a LUT via polynomial interpolation. The complexity analysis in Table 3 (p. 7) states the complexity of FBS is O(t), where t is the plaintext modulus. For t = 65537, this is computationally massive. While the BSGS optimization (Algorithm 2) reduces ciphertext multiplications to O(√t), the number of scalar multiplications and additions remains a direct function of t. The paper's impressive performance results seem to be achieved not by an algorithmically superior method, but by applying brute-force hardware (a 16-block FRU array) to this high-complexity problem. This raises serious questions about scalability.

4. Limited Generality and Fragile Parameterization: The entire framework hinges on the plaintext modulus t being large enough to contain all intermediate inner product results. The authors select t = 65537, which Figure 4 shows is just sufficient for the tested benchmarks. The paper provides no methodology for selecting t for an arbitrary new model, nor does it analyze the sensitivity of the framework to this choice. If a deeper or wider network architecture requires a larger dynamic range, t would need to increase, causing the complexity of FBS to explode and likely rendering the approach impractical. The claim that Athena can "support any type of activation functions" is thus unsubstantiated, as any function requiring a larger LUT or higher precision would break the current parameterization.

Questions to Address In Rebuttal

The authors must address the following points directly to make a case for this paper's acceptance:

1. Regarding Noise Propagation: Please provide a formal analysis of the propagation and accumulation of the ems error introduced during modulus switching. Demonstrate, either through formal proof or extensive empirical evaluation on significantly deeper networks (e.g., >100 layers), that this error remains bounded and does not catastrophically degrade accuracy.
2. Regarding Baselines: Please justify the fairness of comparing your specialized, integer-only accelerator against general-purpose CKKS accelerators. To make a convincing case, provide a comparison against a more appropriate baseline, for instance, an implementation of your framework on a general-purpose FHE accelerator or a theoretical analysis against a CKKS flow optimized for quantized inference.
3. Regarding FBS Scalability: Given the O(t) complexity of the underlying FBS operation, how does the system's performance and hardware cost scale as t increases? What is the practical upper bound on t before the FBS latency and area make the accelerator infeasible?
4. Regarding Generality: How would a user of the Athena framework determine the required value for the plaintext modulus t for a new, arbitrary CNN model? What happens if this required t is larger than the 17-bit value used in this work?
5. Regarding Complex Functions: The paper briefly mentions a three-step process for Softmax. This appears to involve multiple FBS evaluations and a costly ciphertext-ciphertext multiplication. Please provide a detailed breakdown of the latency, noise growth, and complexity for this operation, as it is a critical component in many classification networks.

---

Review Form

Reviewer: The Synthesizer (Contextual Analyst)

Summary

This paper introduces "Athena," a novel co-designed framework and hardware accelerator for performing Convolutional Neural Network (CNN) inference under Fully Homomorphic Encryption (FHE). The core contribution is a paradigm shift away from the prevailing approach of using approximate arithmetic schemes like CKKS on floating-point models. Instead, Athena leverages the mature field of Quantized CNNs (QCNNs), whose integer-based arithmetic is a natural fit for integer-based FHE schemes like BFV.

This fundamental pivot allows for significantly smaller cryptographic parameters, which in turn leads to dramatically reduced ciphertext sizes and on-chip memory requirements. The framework handles the critical non-linear activation functions not with inaccurate polynomial approximations, but with an exact "functional bootstrapping" (FBS) mechanism. This mechanism, reminiscent of TFHE's programmable bootstrapping, performs a lookup-table operation that can implement any arbitrary activation function and simultaneously handles the re-quantization (remapping) step. The authors present a full-stack solution, from the five-step cryptographic loop (Section 3.1, page 4) to a specialized accelerator architecture designed to handle the framework's unique computational bottlenecks, particularly the FBS step. The result is a system that claims near-plaintext accuracy with significant performance and efficiency gains over state-of-the-art CKKS-based accelerators.

Strengths

1. Elegant Core Idea and Problem Reframing: The primary strength of this work lies in its insightful connection between two distinct domains: Quantized Neural Networks and Integer-based FHE. Instead of trying to force the square peg of floating-point neural networks into the round hole of approximate FHE (CKKS), the authors correctly identify that the integer-only nature of QCNNs is a perfect match for schemes like BFV. This reframing of the problem is the paper's most significant intellectual contribution and the source of all subsequent benefits.

2. Practicality and Feasibility: The consequences of this pivot are profound. As shown in Table 1 (page 3) and Table 8 (page 12), the required ciphertext and key sizes are drastically reduced, leading to a >4x reduction in on-chip scratchpad memory compared to leading FHE accelerators like CraterLake and ARK. This is not an incremental improvement; it is a step-change in hardware feasibility and potential cost, making private ML inference a much more tangible reality.

3. Generalized and Accurate Non-Linearity Handling: The use of functional bootstrapping (FBS) is a powerful choice that solves a major pain point in FHE-based ML. The reliance on Taylor/Chebyshev polynomial approximations in CKKS-based systems is a notorious source of error and requires expert tuning (as shown in Figure 1, page 3). Athena's FBS approach is general—it can implement ReLU, Sigmoid, and even complex pooling operations with perfect precision within the quantized domain. Merging the activation function and the remapping step into a single LUT operation is a particularly clever co-design choice.

4. Strong Full-Stack Co-Design: The paper presents a convincing end-to-end solution. The software framework's five-step loop is directly reflected in the hardware design. The identification of FBS as the new performance bottleneck (as opposed to NTT in traditional designs) and the subsequent design of the versatile FRU and the pipelined two-region dataflow (Section 4.3, page 9) demonstrates a deep understanding of the entire stack. The results in Figure 8 (page 11), where the Athena framework is simulated on prior hardware, effectively argue for the necessity of this specialized accelerator.

Weaknesses

While the work is strong, its focus is sharp, leaving some broader contextual questions open. These are not so much flaws as they are opportunities for strengthening the paper.

1. Limited Discussion on the Quantization Process: The paper assumes the availability of a pre-trained QCNN. However, the process of creating a high-accuracy QCNN, often through Quantization-Aware Training (QAT), is non-trivial. More importantly, the choices made in the FHE framework (e.g., the plaintext modulus t in Section 3.3, page 7) are deeply intertwined with the quantization strategy (e.g., the bit-width and range of intermediate values). A brief discussion on this interplay would strengthen the paper by showing how the two domains must be co-designed, not just cascaded.

2. Architectural Generality: The work is presented and evaluated exclusively on CNNs. While this is a critical workload, the field of deep learning is rapidly expanding, with Transformers becoming dominant in many areas. Transformers also rely heavily on quantization for efficient inference. How would the Athena framework and accelerator adapt to the different computational patterns of a Transformer (e.g., large matrix multiplications and the Softmax function)? Addressing this would elevate the contribution from a "solution for CNNs" to a more general "framework for quantized models."

Questions to Address In Rebuttal

1. Could you elaborate on the interplay between the quantization strategy and the selection of FHE parameters? Specifically, how does the choice of the plaintext modulus t (65537 in your work) constrain or inform the quantization process (e.g., the 7-bit w7a7 scheme)? Is there a systematic way to co-optimize these parameters?

2. The paper's evaluation focuses entirely on CNNs. Could the authors comment on the applicability of the Athena framework to other quantized architectures like Transformers? The Softmax operation, in particular, seems like a perfect candidate for the FBS mechanism. Would the accelerator's dataflow, designed for convolutions, be efficient for the large matrix multiplications in a Transformer's attention and feed-forward layers?

3. In the performance comparison in Table 6 (page 10), the paper compares against baselines running CKKS-based models. Could you provide more detail on how the "computational complexity of other benchmarks" was normalized to that of ResNet-20 for a fair comparison, particularly for ResNet-56 which was not reported by all baselines?

4. Step 4 of the Athena framework ("Packing") involves a "homomorphic decryption" of LWE ciphertexts. This is a powerful but potentially costly primitive. In the execution time breakdown (Figure 9, page 11), this operation does not appear to be explicitly separated. Could you clarify where its cost is accounted for and its relative significance to the overall latency?

Review Form

Reviewer: The Innovator (Novelty Specialist)

---

Summary

This paper introduces "Athena," a framework and co-designed hardware accelerator for performing inference on quantized convolutional neural networks (QCNNs) under fully homomorphic encryption (FHE). The central claim of novelty lies not in the creation of a new cryptographic primitive, but in the formulation of a new end-to-end framework that departs from the dominant CKKS-based paradigm for FHE-based machine learning.

The core idea is to build a processing loop around integer-based FHE (specifically, BFV-like operations) tailored for QCNNs. This loop systematically manages computation and noise through five steps: (1) a linear operation using coefficient encoding, (2) modulus switching to reduce noise, (3) ciphertext conversion from RLWE to LWE, (4) homomorphic decryption and packing back into RLWE, and (5) a "Functional Bootstrapping" (FBS) step. The most significant aspect of this proposed framework is that the FBS step unifies three traditionally separate operations: the noise-clearing bootstrap, the non-linear activation function evaluation, and the integer re-quantization/remapping. The authors claim this synthesis allows for smaller cryptographic parameters and higher accuracy compared to prior CKKS-based approaches, and they present a hardware architecture specifically designed to accelerate the bottlenecks of this new workflow.

---

Strengths

1. A Novel Conceptual Framework: The primary strength and most significant novel contribution is the deliberate departure from the CKKS-based paradigm for deep learning inference. While prior work has focused on approximating real-number arithmetic with CKKS, Athena embraces the discrete nature of quantized networks and builds a native integer-based FHE pipeline. This represents a distinct and valuable alternative direction in the design space.

2. Unification of Operations in Functional Bootstrapping: The application of functional bootstrapping to simultaneously perform bootstrapping, non-linear activation, and remapping (as described in Section 3.2.3, Page 6) is a highly novel synthesis of existing ideas. Prior works have used bootstrapping to manage noise and separate techniques (e.g., polynomial approximation) for activations. Merging these, along with the crucial remapping step required for quantization, into a single LUT-based operation is a clever and powerful simplification of the inference flow. This is the paper's most compelling technical insight.

3. A Coherent, Self-Contained Workflow: The proposed five-step loop (Figure 2, Page 4) is a well-defined and repeatable process for executing QCNN layers under FHE. It presents a structured methodology for managing noise and evaluating functions that is distinct from the ad-hoc parameter tuning often required in leveled CKKS schemes. This structured workflow itself can be considered a novel contribution to the field.

4. Novelty in Hardware Co-Design: The accelerator architecture is not merely a collection of standard FHE components. The design of the versatile "FBS and RNS Base changing unit" (FRU) and the two-region dataflow (Section 4.3, Page 9) is a direct and novel consequence of the proposed software framework. The design correctly identifies FBS as the new system bottleneck (a shift away from NTT in many prior works) and dedicates significant, specialized resources to it.

---

Weaknesses

1. Novelty is in Synthesis, Not Primitives: The paper's novelty is almost entirely based on the clever combination and application of pre-existing techniques. The authors appropriately cite prior art for the core FBS primitive ([29]), coefficient encoding for convolutions ([16, 21]), and RLWE-to-LWE conversions ([12]). While the synthesis is novel, the paper could be more explicit in delineating what is being adopted versus what is being created. The contribution is the recipe, not the ingredients.

2. Incremental Novelty in Encoding: The paper contrasts its coefficient encoding scheme with Cheetah [16] (Table 2, Page 5). The claimed improvement appears to be a different batching strategy (prioritizing output channels) to improve data locality for subsequent steps, rather than a fundamentally new encoding method. The novelty here is an optimization tailored to their specific framework, which, while valuable, is an incremental advancement over prior art.

3. Complexity vs. Benefit Justification: The proposed five-step loop introduces significant complexity, involving two forms of ciphertext (RLWE, LWE) and multiple conversions between them (Steps 3, 4, and the final S2C step). While the results are impressive, the justification for this specific complex pathway over a simpler, purely BFV-based or TFHE-based approach could be stronger. The benefit is clear, but it comes at the cost of a non-trivial pipeline that requires specialized hardware (like the SE unit) to remain efficient.

---

Questions to Address In Rebuttal

1. The core of your non-linear evaluation rests on functional bootstrapping, which essentially performs a LUT lookup. Prior works such as PEGASUS [30] and TOTA [42] have also proposed using TFHE-style bootstrapping for LUT-based evaluation of non-linear functions in FHE. Please clearly articulate the delta between Athena's approach and these prior works. Is the primary novelty the integration of the remapping step into the LUT, the specific five-step pipeline in which it is embedded, or another factor?

2. Regarding the coefficient encoding for linear layers (Section 3.2.1, Page 5), please clarify the precise novel contribution over the methods used in Cheetah [16] and NeuJeans [21]. Is the novelty in the encoding itself, or is it strictly in the batching strategy that arranges data to benefit the subsequent sample extraction step?

3. The proposed framework unifies activation and remapping within the FBS step. How does the framework handle layers that do not have a non-linear activation (e.g., a convolution followed directly by a pooling or batch normalization layer)? Does this require an "identity-with-remapping" LUT, and if so, what is the performance and complexity overhead compared to a more direct remapping operation? This would help clarify the generality of the proposed novel framework.