ByteFS: System Support for (CXL-based) Memory-Semantic Solid-State Drives

Paper Title: ByteFS: System Support for (CXL-based) Memory-Semantic Solid-State Drives
Reviewer: The Guardian (Adversarial Skeptic)

---

Summary

The authors present ByteFS, a file system co-designed with custom SSD firmware to support memory-semantic SSDs (M-SSDs). The core contribution is a system that utilizes a dual byte/block interface to optimize I/O accesses. Key techniques include: 1) adapting the I/O interface (byte vs. block) based on the data structure and access pattern, 2) modifying the SSD firmware to manage the on-device DRAM as a log-structured write buffer to coalesce byte-granular writes before flushing to NAND flash, and 3) a coordinated caching scheme that prioritizes the host page cache for read caching to reserve SSD DRAM for writes. The authors implement and evaluate ByteFS on both an FPGA-based SSD prototype and an emulator, demonstrating significant throughput improvements (up to 2.7x) and write traffic reduction (up to 5.1x) compared to existing block-based (Ext4, F2FS) and persistent memory file systems (NOVA, PMFS).

Strengths

The paper is technically dense and presents a comprehensive system-level effort, spanning from file system design to firmware modification and hardware prototyping. This level of vertical integration is commendable.

1. Hardware Prototype: The implementation and evaluation on a real, programmable OpenSSD FPGA board (Section 4.9, page 9) lends significant credibility to the performance results, moving beyond pure emulation which can often hide real-world system complexities.
2. Problem Motivation: The quantitative study in Section 3 (pages 3-5) effectively illustrates the well-known problem of I/O amplification in existing file systems, providing a solid, data-driven motivation for the need for a dual-interface approach.
3. Performance Breakdown: The ablation study presented in Figure 12 (page 12) is a crucial piece of analysis. It successfully disentangles the performance contributions of the three main design components (dual interface, log-structured firmware, and adaptive data I/O), which strengthens the authors' claims about the efficacy of each component.

Weaknesses

Despite the strengths, the work rests on several questionable design choices and the evaluation lacks the rigor to fully substantiate its claims. The core assumption that the proposed complexity is a net-win is not convincingly proven.

1. Prohibitive Overhead of Adaptive Data I/O: The mechanism for selecting the data interface, detailed in Section 4.6 (page 8), is a critical flaw. The use of a copy-on-write (CoW) mechanism within the page cache, followed by an XOR comparison to detect modified cache lines, introduces unacceptable overheads. The authors admit that "duplicated pages occupy 16% of the entire page cache size on average," which is a substantial and potentially prohibitive memory overhead, especially in memory-constrained environments. Furthermore, the CPU cost of performing XOR on entire pages, while benchmarked in isolation, is not evaluated as a system-level overhead that consumes cycles that could be used by the application. The marginal gains shown for this mechanism in Figure 12 for workloads like OLTP do not appear to justify this complexity and cost.
2. Insufficient Evaluation of Background Work: The log-structured SSD DRAM is central to the design, yet the impact of its background log cleaning process (Section 4.3, page 7 and Algorithm 1) is not adequately evaluated. The authors admit that cleaning can involve read-modify-write patterns, leading to higher flash traffic than baselines in some cases (Section 5.3, page 12). They dismiss this by stating it occurs "in the background," but background work is not free; it consumes internal device bandwidth and controller resources, which can create I/O interference and introduce significant tail latency. The evaluation only presents p95 latency (Figure 7, page 11), which is insufficient to expose the impact of such garbage collection-like activities. A rigorous evaluation must include p99 and p999 latencies to demonstrate that log cleaning does not introduce performance cliffs.
3. Ambiguous Persistence Guarantees and Overhead: The mechanism for ensuring the persistence of byte-granular writes relies on a clflush/clwb followed by a "write-verify read" (a zero-byte read) to flush PCIe transaction buffers (Section 4.2, page 6). This is a known technique, but it effectively serializes dependent operations at the PCIe root complex. The cost of this serialization is not measured. For workloads with high-concurrency, small synchronous writes (common in databases and journals), this serialization could become a major bottleneck, undermining the benefits of the byte interface. The paper lacks a microbenchmark that isolates and quantifies this persistence overhead under concurrent load.
4. Conflated Contributions in Baseline Comparison: The primary performance results (Figure 6, page 10) compare ByteFS running with its custom, log-structured firmware against baseline file systems running on the M-SSD with a standard caching firmware. This is not a fair, apples-to-apples comparison of the file systems themselves. It conflates the benefits of the FS design with the benefits of a superior firmware design. The performance breakdown in Figure 12 helps, but the headline claims are based on a comparison that is fundamentally skewed. The baselines are not given the opportunity to run on firmware that is optimized for this class of device.

Questions to Address In Rebuttal

1. Regarding the CoW/XOR mechanism for adaptive data I/O: Can you provide a detailed analysis of the trade-offs? Specifically, what is the measured CPU overhead (as a percentage of total CPU time) and memory overhead under each of the evaluated macro-benchmarks? At what modified page ratio R does the overhead of this mechanism outweigh the benefit of using byte-granular writes?
2. The paper's evaluation of the log-structured DRAM is incomplete. Please provide tail latency data at the 99th and 99.9th percentiles for the YCSB workloads to demonstrate that the background log cleaning process does not introduce unacceptable latency spikes. Furthermore, can you measure the internal SSD bandwidth consumed by the cleaning process and show its impact on foreground I/O performance?
3. Please provide microbenchmark results that specifically measure the throughput and latency of small (e.g., 64-byte) synchronous persistent writes using your clflush/read-verify mechanism. The benchmark should vary the number of concurrent threads to demonstrate how the serialization point at the root complex affects scalability.
4. To create a fairer comparison, could you implement a simplified version of the log-structured write buffer in the firmware and run a traditional file system like Ext4 on top of it? This would help to more clearly isolate the performance gains attributable solely to the ByteFS file system's dual-interface management versus the gains from the superior firmware design.

Reviewer: The Synthesizer (Contextual Analyst)

Summary

This paper introduces ByteFS, a novel file system co-designed with SSD firmware for emerging memory-semantic solid-state drives (M-SSDs). These devices, often enabled by interconnects like CXL, offer a dual-mode interface: fast, byte-granular memory-mapped access and traditional, high-throughput block-based access. The core contribution of this work lies in its holistic approach to embracing this duality, rather than forcing the device into a purely memory-like or purely block-like model.

ByteFS intelligently partitions filesystem operations, using the byte-addressable interface for small, latency-sensitive metadata updates (e.g., inode fields, bitmap flips) and the block interface for bulk data transfers. To bridge the fundamental mismatch between the byte-accessible host interface and the page-granular nature of internal NAND flash, the authors propose a crucial firmware modification: managing the SSD's internal DRAM as a log-structured write buffer. This allows small byte-writes to be coalesced efficiently before being written to flash, significantly reducing I/O amplification. The system is evaluated on both an FPGA prototype and an emulator, demonstrating substantial performance gains and reductions in write traffic compared to state-of-the-art file systems designed for either block devices or persistent memory.

Strengths

1. Timeliness and Strategic Relevance: The paper addresses a critical and timely problem. With the discontinuation of Intel Optane, the industry is actively seeking practical alternatives for storage-class memory. CXL-attached, memory-semantic SSDs are a leading candidate. This work provides one of the first comprehensive system software blueprints for this new class of hardware, moving the conversation from "can we build it?" to "how should we use it?" It's a forward-looking paper that is well-positioned to influence future system design.

2. Excellent Problem Diagnosis: The quantitative study in Section 3 (p. 3-4) is a standout feature. By meticulously dissecting the I/O patterns of individual filesystem data structures (Table 3), the authors provide a compelling, data-driven justification for their dual-interface design. This analysis clearly shows that a one-size-fits-all approach (either pure byte or pure block) is suboptimal and lays a strong foundation for the design of ByteFS.

3. Pragmatic Hardware/Software Co-Design: The paper's strength is its recognition that the problem cannot be solved in the host software alone. The proposed firmware modifications—specifically, the log-structured DRAM cache (Section 4.3, p. 6)—are the linchpin of the entire system. This co-design elegantly resolves the impedance mismatch between the host's view of the device and the physical reality of its NAND media. It provides a practical path forward that acknowledges the constraints of both hardware and software.

4. Connecting Disparate Concepts: The design of ByteFS is a masterful synthesis of ideas from different domains. It borrows the fine-grained access patterns from persistent memory file systems (like NOVA), the robustness of traditional block-based systems (like Ext4), and the write-efficiency of log-structured systems (like F2FS), but reapplies these concepts in a new context. The decision to implement logging within the device firmware is particularly insightful, as it hides flash-related overheads from the host and simplifies crash consistency logic.

Weaknesses

While the core ideas are strong, the paper could be strengthened by a deeper exploration of the following aspects:

1. Exploration of Design-Space Trade-offs: The paper presents a set of well-motivated heuristics for interface selection (e.g., the 512B threshold for direct I/O, the CoW mechanism for buffered I/O in Section 4.6, p. 8). While the evaluation shows these work well, the paper would benefit from a discussion of the sensitivity to these choices. How does performance change as these thresholds are varied? This would provide a richer understanding of the design space and offer guidance for tuning on different hardware.

2. Scalability of the Recovery Mechanism: The crash recovery process (Section 4.7, p. 9) relies on scanning the in-device transaction log. The paper reports a fast recovery time of 4.2 seconds (Section 5.5, p. 12). However, as CXL devices evolve to include multi-gigabyte DRAM caches, this linear scan could become a bottleneck. A brief discussion of how the recovery mechanism could be scaled—perhaps through checkpointing or more structured indexing within the log—would add to the work's long-term relevance.

3. Positioning Relative to CXL.mem Coherency: The authors make a design choice to use a custom persistence protocol (clflush + write-verify read) rather than relying on the full CXL.mem cache coherency protocol. This is a reasonable choice for compatibility and simplicity. However, the paper misses an opportunity to discuss this trade-off more explicitly. A deeper analysis of the performance, complexity, and hardware-dependency implications of their approach versus a fully coherent one would provide valuable context for other researchers building on this work.

Questions to Address In Rebuttal

1. Regarding the adaptive policies for interface selection (e.g., the 512B threshold in Section 4.6, p. 8): Could the authors elaborate on the sensitivity of the system's performance to this threshold? Is there a case for a more dynamic or workload-aware policy beyond the static threshold and CoW-based ratio?

2. The recovery process described in Section 4.7 (p. 9) involves scanning the log region. While the measured recovery time is short on the prototype, could the authors comment on how this approach scales with a much larger in-device DRAM and log region, as might be common in future CXL devices?

3. The paper chooses a custom persistence mechanism (clflush + write-verify read). Given the CXL context, could the authors provide more rationale for this choice over leveraging CXL.mem's native coherency protocols? What are the key performance or implementation trade-offs that motivated this design decision?

Reviewer: The Innovator (Novelty Specialist)

Summary

This paper presents ByteFS, a novel file system designed for memory-semantic SSDs (M-SSDs) that feature dual byte-addressable (via MMIO) and block-addressable (via NVMe) interfaces. The core contribution is a software/hardware co-design that spans the file system and the SSD firmware. The authors claim novelty in three main areas: (1) an adaptive policy within the file system to dynamically select the appropriate interface (byte or block) based on the access pattern, (2) a firmware-level, log-structured management of the SSD's internal DRAM to efficiently coalesce byte-granular writes into page-granular flash writes, and (3) a coordinated caching scheme that dedicates SSD DRAM to this write log while offloading read caching to the host page cache. The evaluation, performed on a real FPGA-based prototype and an emulator, shows significant performance gains and I/O reduction compared to both traditional block-based file systems and existing persistent memory file systems.

Strengths

The primary strength of this work lies in its holistic, co-designed approach to a compelling new hardware target. The paper correctly identifies that neither existing block-based file systems nor persistent memory file systems are a natural fit for M-SSDs. The novelty of the proposed solution is significant:

1. Novel Co-design for Granularity Mismatch: The central novel idea is the tight coupling between the host file system and the device firmware to resolve the byte-host vs. page-flash access granularity mismatch. While SSDs internally buffer writes, ByteFS makes this buffer an explicit, transactionally-consistent log that is directly coordinated with the host file system via custom commands (e.g., COMMIT(TxID) as discussed in Section 4.3, page 7). This elevates a standard FTL optimization into a first-class primitive for system software.

2. Novel Heuristic for Interface Selection: The mechanism for choosing the access granularity for dirty pages in the buffered I/O path is particularly novel. Using Copy-on-Write (CoW) to track changes and XORing the original and modified pages to quantify the "dirtiness" (Section 4.6, page 8) is a clever, concrete heuristic. This provides a data-driven policy for when to expend byte-granular MMIO writes versus a more efficient block-granular NVMe write, a problem unique to this class of device.

3. Novel Caching Policy: The coordinated caching policy is simple but conceptually novel in this context. The decision to forgo read caching in the SSD DRAM and dedicate that precious resource entirely to a persistent write log (Section 4.3, page 6) is a strong design choice that directly addresses the performance characteristics of the underlying flash media (writes are slow and benefit from coalescing). It avoids the redundancy of caching the same data blocks in both the host page cache and the device DRAM, a clear win.

Weaknesses

My analysis focuses exclusively on novelty. While the overall system is a novel composition of ideas, some of the constituent concepts have appeared in recent literature, which slightly tempers the novelty of the individual components, though not the system as a whole.

1. Overlapping Concept of Firmware-Level Logging: The core idea of using a log-structured buffer in the SSD's DRAM to handle the granularity mismatch for CXL-attached SSDs has been explored in prior work. Specifically, "Overcoming the memory wall with CXL-Enabled SSDs" (Yang et al., USENIX ATC '23) [49] also proposes a firmware-level write log to coalesce writes and hide flash latency. While ByteFS's contribution is the full-fledged POSIX file system built on top of this idea, the foundational firmware concept is not entirely de novo. The paper would be strengthened by explicitly positioning its contribution as the system software integration of this emerging device architecture, differentiating it more clearly from device-level proposals like Yang et al.

2. Incremental Novelty on Dual-Interface Hardware: The concept of a dual-interface byte/block SSD was previously introduced by "2B-SSD" (Bae et al., ISCA '18) [12]. ByteFS is a significant and necessary step forward by providing the file system logic to actually exploit such a device. However, the claim of novelty should be carefully scoped to the software system and co-design, rather than the underlying hardware concept itself. The paper does cite this work, but the delta should be framed as enabling a general-purpose file system, which is a substantial but specific type of advancement over the prior art.

Questions to Address In Rebuttal

1. The work by Yang et al. (USENIX ATC '23) [49] also proposes a firmware-level log in SSD DRAM for CXL SSDs to bridge the granularity gap. Could the authors please clarify the novelty of their firmware design in light of this prior work? Is the novelty primarily in the host-side file system's ability to leverage such a feature, or are there fundamental differences in the firmware log design itself (e.g., the indexing structure, transaction management)?

2. The CoW and XOR mechanism for selecting the writeback interface for dirty pages is an interesting heuristic. What is the novelty of this specific technique? Have similar bitwise comparison techniques been used in other contexts (e.g., data deduplication, differential backup) to guide policy decisions in a file or storage system, and if so, how does your application of it represent a novel contribution?

3. The coordinated caching policy is a key part of the design. Can you elaborate on whether this is a fundamentally new idea, or an application of known cache-coordination principles to the specific, novel context of M-SSDs? The novelty seems to stem from the context; please confirm if that is the correct interpretation.