AiF: Accelerating On-Device LLM Inference Using In-Flash Processing

Paper Title: AiF: Accelerating On-Device LLM Inference Using In-Flash Processing
Reviewer: The Guardian

---

Summary

This paper proposes "Accelerator-in-Flash" (AiF), an in-flash processing (IFP) architecture designed to accelerate on-device Large Language Model (LLM) inference. The core contribution is a pair of flash-level optimizations: 1) "charge-recycling read" (cr-read), a technique to reduce latency for sequential wordline accesses, and 2) "bias-error encoding" (be-enc), a Vth state reconfiguration scheme to improve the reliability of pages storing LLM parameters. The authors claim that these techniques enable a 4x increase in internal flash bandwidth, which, when integrated into a full system, results in a 14.6x throughput improvement over baseline SSD offloading and even a 1.4x improvement over a high-end, in-memory (DRAM) system. The evaluation is conducted using a modified version of the NVMeVirt SSD emulator integrated with the llama.cpp inference engine.

---

Strengths

1. Well-Motivated Problem: The paper correctly identifies the memory capacity and bandwidth limitations as critical barriers to on-device LLM deployment. The analysis in Section 3, particularly Figure 4, effectively establishes the performance bottlenecks and reliability requirements that motivate the work.

2. Conceptually Sound Primitives: The two core technical ideas are well-conceived. The concept of cr-read (Section 4.2) leverages the known spatial locality of model parameter access to optimize the flash read sequence. Similarly, be-enc (Section 4.3) is a clever application of reconfiguring Vth states to create a heterogeneous reliability profile within a single TLC block, prioritizing the LLM data.

3. Holistic Approach: The work commendabley attempts to address both performance (bandwidth via cr-read) and reliability (error rates via be-enc) simultaneously. This is a crucial and often overlooked aspect of practical IFP system design.

---

Weaknesses

My primary concerns with this work lie in the significant gap between the proposed concepts and the evidence provided to support their practicality, scalability, and system-level viability. The evaluation relies on a chain of optimistic assumptions and limited validations that undermine the paper's extraordinary claims.

1. Validation of cr-read is Fundamentally Insufficient: The claim that cr-read is functionally sound and yields a 2.8x bandwidth improvement (Section 4.2.2, page 7) rests on SPICE simulations and experiments on a "fabricated CTF cell array" of 9x9 WLs/BLs (footnote 5, page 6). A 9x9 array is a laboratory-scale toy, not a proxy for a modern 3D NAND device with thousands of wordlines, complex peripheral circuitry, and significant parasitic capacitances. Real-world phenomena like read disturb, wordline-to-wordline coupling, and thermal effects, which are critical at scale, are not captured. Presenting this as sufficient validation for a production-level technique is a significant overstatement.

2. System-Level Side Effects of be-enc are Ignored: The be-enc scheme (Section 4.3) introduces performance and reliability asymmetry, degrading the non-LSB pages used for "general data." The paper dismisses this impact with a cursory analysis in Figure 18 (page 12), showing only a minor drop in random read IOPS. This analysis is critically incomplete. It fails to address:

   • Garbage Collection (GC) and Write Amplification: How does a flash translation layer (FTL) manage blocks containing both high-endurance LSB pages (for LLM data) and low-endurance CSB/MSB pages (for general data)? This heterogeneity would drastically complicate wear-leveling and GC, likely leading to increased write amplification and premature device wear, none of which is modeled or discussed.
   • Write/Erase Performance: The paper only analyzes the read path. Does reconfiguring the Vth states for be-enc impact program/erase times or program disturb characteristics? This is not mentioned.

3. The Reliability Premise of ECCLITE is Fragile: The entire justification for the lightweight ECCLITE decoder hinges on the characterization in Figure 13(b) (page 8), which shows a maximum of 9 bit errors per 1-KiB for be-enc LSB pages. The proposed ECCLITE corrects up to 10 bits. This leaves a safety margin of a single bit error. In real-world flash deployment, error rates are a distribution, not a fixed maximum. This razor-thin margin provides virtually no resilience against process variation, variable retention times, or higher-than-expected cell degradation near the device's end-of-life. A robust system would require a much larger ECC margin.

4. The Evaluation Model for Concurrency is Overly Optimistic: The claim of surpassing in-memory performance (1.4x) is predicated on the parallel host-AiFSSD execution model (Figure 15c, page 9). The NVMeVirt-based evaluation (Section 6.1) simulates the delay of the AiFSSD computation. However, it is highly unlikely that this model accurately captures the true overhead of the proposed fine-grained, tightly-coupled interaction. Each GEMV offload requires command submission, potential context switching, interrupt handling, and DMA setup/teardown. In a real system, this control-plane overhead for frequent, small operations could easily dominate the data-plane latency, nullifying the benefits of parallelism. The simulation appears to model an idealized best-case scenario.

5. Contradictory Claims on Bottlenecks: The authors motivate the work by claiming memory bandwidth is the primary bottleneck. However, their own scalability analysis (Figure 17b, page 12) shows sub-linear performance scaling when doubling or quadrupling internal bandwidth. They attribute this to "multiple vector arithmetic operations" and "NVMe protocol overheads." This is a crucial admission that their solution only addresses a part of the problem, and that other components (left on the host) become the new bottleneck. This weakens the central thesis that simply maximizing internal flash bandwidth is the definitive solution.

---

Questions to Address In Rebuttal

The authors must provide convincing evidence and clarification on the following critical points:

1. Regarding cr-read Validation: How can the results from an 81-cell array be extrapolated to a commercial multi-Gb flash die? Please provide analysis or data on how cr-read would behave in the presence of scaled-up parasitics, read disturb, and other real-world, dynamic flash array effects.

2. Regarding be-enc System Impact: Provide a detailed analysis of how be-enc would interact with a modern FTL. Specifically, quantify its impact on garbage collection efficiency, write amplification, and the endurance of blocks that mix IFP and general-purpose data.

3. Regarding ECCLITE Reliability: Please justify the decision to use an ECC scheme with only a 1-bit error correction margin over your characterized maximum. What is the expected failure rate of this scheme at the device's certified P/E cycles and retention period, considering statistical variations in error rates?

4. Regarding the Concurrency Model: Can you provide evidence that your NVMeVirt timing model accurately accounts for the full software stack and hardware overhead (interrupts, DMA, protocol overhead) of the frequent host-SSD synchronization required by your parallel execution scheme? Please break down the latency components of a single offloaded GEMV operation in your model versus a real system.

5. Regarding Performance Scaling: Given the admitted sub-linear scaling, what is the theoretical peak throughput of the AiF system as internal bandwidth approaches infinity? At what model size do the non-GEMV operations left on the host become the primary bottleneck, rendering further flash acceleration ineffective?

Reviewer: The Synthesizer (Contextual Analyst)

Summary

This paper addresses the critical challenge of running large language models (LLMs) on personal devices with limited DRAM. The authors identify the primary bottleneck of existing SSD offloading techniques: the low external read bandwidth of flash storage, which severely limits token generation rates for memory-bound LLM inference.

The core contribution is Accelerator-in-Flash (AiF), an in-flash processing (IFP) architecture that moves the dominant matrix-vector multiplication (GEMV) operations directly into the NAND flash chips. This approach is designed to leverage the immense aggregate internal bandwidth of the chips, bypassing the external channel and PCIe bottlenecks. The novelty of this work lies not just in proposing IFP for AI, but in developing two co-designed, flash-aware techniques that make it practical for the unique demands of LLMs:

1. Charge-Recycling Read (cr-read): A novel read command that speeds up sequential reads within a block by eliminating redundant precharge and discharge steps, boosting effective read bandwidth.
2. Bias-Error Encoding (be-enc): A clever VTH state encoding scheme that makes one page type (LSB) exceptionally fast and reliable at the cost of others, allowing LLM parameters to be stored with high fidelity while enabling a highly compact on-chip error correction engine (ECCLITE).

Through a comprehensive evaluation using a full-system simulator integrated with the llama.cpp engine, the authors demonstrate that AiF can provide a 14.6x throughput improvement over conventional SSD offloading and even surpass a high-end in-memory system by 1.4x, all while significantly reducing the host memory footprint.

Strengths

1. High Potential Impact and Significance: This work addresses a timely and significant problem. The inability to run large, capable LLMs locally on consumer devices is a major barrier to privacy, low latency, and offline accessibility. AiF presents a compelling hardware-based solution that could fundamentally alter the landscape of edge AI. Rather than an incremental improvement, this work proposes a system that could enable models (e.g., 40B+ parameters) that are currently infeasible on personal devices to run at interactive speeds. This is exactly the kind of ambitious, problem-driven research the community needs.

2. Excellent Cross-Layer Co-Design: The true strength of this paper lies in its vertical integration of insights from the application layer down to the device physics layer. The authors recognize that LLM inference is (a) dominated by GEMV on static weights and (b) highly sensitive to bit errors (as shown in Figure 4b, page 4). Instead of a generic accelerator, they have designed cr-read and be-enc by fundamentally rethinking the NAND flash read protocol itself to meet these specific requirements. This deep understanding, connecting algorithmic needs to the manipulation of VTH states (Section 4.3, page 7) and read sequence timing (Section 4.2, page 5), is the hallmark of outstanding systems research.

3. Holistic and Plausible System Integration: The authors go beyond the core accelerator idea and consider the full system stack. They outline the necessary extensions to the NVMe protocol (aif_post, aif_gemv), discuss the system software and application-level requirements (Section 5.2, page 9), and even architect a parallel execution model to overlap host and AiFSSD computation (Figure 15c, page 9). The evaluation, which uses a modified full-system SSD emulator (NVMeVirt) and a real inference engine (llama.cpp), lends significant credibility to their performance claims. This end-to-end thinking makes the proposed system feel less like a theoretical concept and more like a blueprint for a real-world product.

Weaknesses

While the core ideas are strong, the paper could benefit from addressing the following points, which are less flaws in the work itself and more contextual limitations.

1. Practical Path to Adoption: The proposed changes are significant, requiring modification to both the internal logic of NAND flash chips and the firmware of the SSD controller. This represents a substantial re-architecting effort for semiconductor manufacturers who are typically conservative. While the paper argues for the feasibility of the required changes (e.g., modifying timer codes, leveraging existing dynamic VTH configuration), the business and engineering inertia to implement such a specialized feature is immense. The work would be stronger if it acknowledged this difficult path to commercialization more directly.

2. Handling Dynamic Data and Competing Workloads: The proposed system is exquisitely optimized for the "write-once, read-many" nature of LLM parameters stored in dedicated "IFP blocks." However, modern SSDs are general-purpose devices. The paper mentions that garbage collection (GC) in a footnote (footnote 9, page 10) but does not fully explore the performance implications. If a user is performing heavy I/O operations concurrently with LLM inference, how would background SSD tasks like GC and wear-leveling, which might need to move IFP blocks, impact inference latency and consistency? This interaction between the specialized "compute" workload and general storage tasks is a critical aspect of any practical in-storage processing system.

3. Limited Operational Scope: The work rightly focuses on GEMV as the primary bottleneck. However, other operations in an LLM (e.g., Softmax, LayerNorm) still run on the host, requiring data to move back and forth. The parallel execution scheme (Section 5.1) mitigates this, but as model architectures evolve, the Amdahl's Law effect from the non-accelerated portions could become more significant. This is not a weakness of the current work but a natural question about its extensibility and future-proofing.

Questions to Address In Rebuttal

1. Regarding Practical Implementation: Can the authors elaborate on the non-recurring engineering (NRE) cost and complexity for a NAND manufacturer to implement cr-read and be-enc? Are these changes that could be accomplished primarily through microcode/firmware patches to the on-chip scheduler and voltage generators, or do they require significant redesign of the analog peripheral circuitry?

2. Regarding System Interference: Could you elaborate on the performance impact of background garbage collection on a running LLM inference task? If an IFP block becomes the target of a GC operation, would this introduce significant, unpredictable latency (i.e., jitter) into the token generation pipeline? How does your proposed system manage or isolate the compute-dedicated blocks from the effects of general storage I/O?

3. Regarding Model Management: The aif_post command is used to initially place the model into the optimized layout. What is the performance overhead of this setup process for a large model (e.g., a 70B parameter model)? How does the system envision handling model updates or switching between different fine-tuned versions of a model, which are common user scenarios?

Reviewer: The Innovator (Novelty Specialist)

Summary

This paper proposes Accelerator-in-Flash (AiF), an in-flash processing (IFP) architecture designed to accelerate on-device large language model (LLM) inference. The authors correctly identify that existing SSD offloading techniques are bottlenecked by external I/O bandwidth and that prior IFP solutions fail to meet the unique high-bandwidth and high-reliability demands of LLMs. The core of the proposed contribution is not the general idea of IFP, but rather two specific, low-level flash read techniques designed to make IFP practical for this workload. The first, "charge-recycling read" (cr-read), modifies the NAND read sequence to skip precharge and discharge steps for sequential reads within a block, boosting read speed. The second, "bias-error encoding" (be-enc), reconfigures the threshold voltage (VTH) state mapping in TLC NAND to create an ultra-reliable and fast LSB page type, where LLM parameters are exclusively stored. This increased reliability allows for a lightweight on-chip ECC decoder (ECCLITE), mitigating the area and power overhead that would otherwise make IFP infeasible. The authors evaluate this system via simulation, demonstrating significant throughput gains over both baseline SSD offloading and in-memory inference.

Strengths

The primary strength of this work lies in its specific, non-obvious proposals for modifying the physical operation of NAND flash to serve a high-level application need. The novelty is not in the high-level concept but in the enabling mechanisms:

1. Novel Co-design: The paper presents a compelling cross-layer co-design. Instead of simply positing an accelerator inside a flash chip, the authors have identified the fundamental physical limitations (read latency, error rates, ECC cost) and proposed concrete solutions (cr-read, be-enc) at the device level. The tight coupling between be-enc reducing error rates and ECCLITE reducing hardware cost is particularly novel and well-conceived.

2. Specific Low-Level Contributions: Both cr-read and be-enc appear to be novel in their specific formulation and application. While the principle of optimizing sequential access or leveraging the differing reliability of pages in MLC/TLC flash may have conceptual precedents, the proposed mechanisms are distinct:

   • cr-read (Section 4.2, page 5) as a specific modification to the read state machine (bypassing precharge/discharge) is a clever circuit-level optimization.
   • be-enc (Section 4.3, page 7) goes beyond passively using the LSB page; it proposes actively reconfiguring the VTH state encoding (from (2,3,2) to (1,3,3) coding) to intentionally create a privileged, high-performance page type specifically for IFP data. This is a significant conceptual step beyond prior work that merely partitions data based on existing page characteristics.

3. Problem-Driven Innovation: The work is well-motivated. The authors clearly establish in Section 3.3 (page 4) why existing IFP is insufficient for LLMs, pointing to the dual challenges of raw bandwidth and the prohibitive cost of robust on-chip ECC. Their proposed solutions directly and elegantly address these two specific, well-articulated problems.

Weaknesses

From a novelty perspective, the primary weakness is that the paper frames its contribution under the broad umbrella of "In-Flash Processing," a concept with extensive prior art. The true novelty is more nuanced and lies deep within the flash controller and cell programming logic, which could be emphasized more clearly.

1. Incremental Nature of cr-read: The core idea of cr-read—reusing charged line voltages to accelerate subsequent accesses to a physically local region—is a common optimization principle in hardware design. While its application to the NAND read sequence is new in this context, it feels like an incremental, albeit clever, engineering optimization rather than a fundamentally new concept. The paper would be strengthened by a more thorough comparison to any existing "fast sequential read" modes or similar optimizations that may exist in NAND flash manufacturer datasheets or patents, which often contain proprietary, non-public access modes.

2. Overstated Novelty of the General Approach: The paper's narrative implies that IFP for ML is a new direction. However, as the authors' own related work section (Section 7, page 12) points out, numerous works have explored in-storage and in-flash processing for DNNs and other data-intensive workloads ([38], [45], [78]). The key distinction of AiF is its focus on LLM inference and the specific physical-level techniques to overcome the associated challenges, a distinction that should be made more central to the paper's claims.

Questions to Address In Rebuttal

1. Prior Art for cr-read: Can the authors elaborate on the novelty of the cr-read technique relative to prior art in low-level flash memory operation? Are there functionally similar "burst" or "fast sequential" read modes that have been proposed or implemented by memory manufacturers, even if not for the purpose of general-purpose computation? The concept of skipping reset phases for sequential operations is not new in principle; please clarify what makes this specific application to the NAND read sequence fundamentally novel.

2. Prior Art for be-enc: The concept of pages within a multi-level cell exhibiting different reliability characteristics is well-established. Prior work has proposed leveraging this by, for example, placing critical metadata on more reliable pages. The core novelty of be-enc seems to be the active reconfiguration of VTH levels to further enhance the reliability of LSB pages for IFP workloads. Can you confirm if this specific idea—dynamically changing the VTH encoding scheme on a per-block basis to create a privileged data partition for an IFP accelerator—has been proposed before?

3. Complexity vs. Practicality: The proposed modifications, especially cr-read, require altering the fundamental state machine of a NAND flash chip's read path. This represents a significant deviation from the standard ONFI/Toggle interface and would require deep co-design with a NAND manufacturer. Given the extremely high cost and risk associated with modifying this core IP, is the proposed solution a purely academic exploration, or do the authors see a realistic path to adoption? The justification for novelty must also consider the feasibility of the proposed complexity.