LongSight: Compute-Enabled Memory to Accelerate Large-Context LLMs via Sparse Attention

---

Review Form

Reviewer: The Guardian (Adversarial Skeptic)

Summary

This paper presents LongSight, an algorithm-hardware co-design that repurposes a CXL-based dense retrieval accelerator, DReX, for sparse attention in large-context LLMs. The core mechanism is a hybrid attention model: a dense sliding window of recent tokens is handled by the host GPU, while attention over the long-tail historical context is offloaded to DReX. This offload leverages a multi-stage filtering process, initiated by a sign-based filter (SCF) accelerated in-memory, to retrieve a top-k set of relevant Key-Value pairs. The authors claim this system can support context lengths up to one million tokens on a single GPU and achieves significant throughput improvements over dense baselines.

However, the work rests on a series of optimistic assumptions regarding the generalization of its sparse approximation, the practicality of its complex hyperparameter tuning, and the performance of an emulated hardware interface. The evaluation relies heavily on projections for its most impressive claims, leaving the robustness and real-world viability of the system in question.

Strengths

1. Pragmatic System Partitioning: The hybrid dense-sparse approach, which keeps recent tokens in GPU HBM for dense attention while offloading the historical context, is a logical and practical design choice. It correctly identifies that recent tokens are often most important, providing an accuracy backstop for the sparse approximation.
2. Detailed Hardware-Aware Data Layout: The paper provides a well-considered mapping of the logical data hierarchy (Key Blocks, Context Slices, User Partitions) onto the physical hardware of DReX (Section 7.3.3, page 9). This demonstrates a clear understanding of the need to co-design data structures with the memory system to maximize parallelism and bandwidth.
3. Repurposing Existing Hardware: The central idea of extending a dense retrieval accelerator for sparse attention is resourceful. It proposes a path to broader utility for specialized hardware, which is a commendable system-level goal.

Weaknesses

1. Core Claims are Based on Extrapolation, Not Measurement: The headline claim of supporting and accelerating 1M token contexts is not empirically demonstrated. As noted in the caption of Figure 7 (page 12), performance for context lengths above 128K is "projected based on performance at 128K context." A projection is not a result. For a systems paper making bold performance claims, the lack of measured data at the most challenging scales undermines the entire contribution. The scaling of system bottlenecks (e.g., CXL overhead, data structure management) may not be linear as assumed.
2. The Algorithmic Foundation is Potentially Fragile: The entire performance gain hinges on the effectiveness of the ITQ-enhanced SCF. The methodology for this is questionable:
   • The ITQ rotation matrix is trained on a mere 1K-token sequence (Section 5.4, page 6) but is expected to generalize and remain effective across a 1,000,000-token context. This is a heroic assumption. There is no evidence that the statistical properties of Key/Query vectors remain stable enough for this to hold.
   • The evaluation relies solely on perplexity. While a useful intrinsic metric, it is not a substitute for performance on downstream, long-context reasoning tasks. A small hit in perplexity can sometimes translate to a catastrophic failure in tasks requiring retrieval of specific, non-obvious facts from deep within the context.
3. Impractical Hyperparameter Sensitivity: The authors themselves concede in Section 9.3 (page 13) that "optimal parameters (i.e., window size, k, and SCF thresholds) are heavily context-dependent and impact end-to-end performance." This is a critical flaw for a general-purpose accelerator. It implies that for any new model, dataset, or even target context length, a user must engage in a complex, multi-dimensional tuning sweep to achieve the reported performance. This severely limits the system's practical usability and suggests the proposed method is more of a brittle proof-of-concept than a robust solution.
4. Insufficient and Misleading Baseline Comparisons: The primary baselines are 1-GPU and 2-GPU dense attention. It is a foregone conclusion that a sparse method will outperform a dense one at extreme context lengths. The more scientifically rigorous comparisons would be against state-of-the-art software-based sparse attention techniques (e.g., highly optimized block-sparse kernels) running on the same GPU hardware. The paper discusses such methods in the background (Section 3.1) but fails to compete against them in the evaluation (Section 9). The comparison against a simple sliding window attention (Figure 10, page 13) is a weak benchmark.
5. Hardware Performance is Based on Emulation: The evaluation "emulate[s] the CXL interface using a dual-socket Intel Xeon...platform" (Section 8.2, page 11). CXL is a complex interconnect, and its real-world performance involves subtle effects of protocol overhead, contention, and NUMA effects that are difficult to capture with such an emulation. A paper proposing a CXL-based hardware solution must be held to a higher standard of fidelity in its performance model. Relying on an emulation for the critical data path injects significant uncertainty into the final performance numbers.

Questions to Address In Rebuttal

1. Please provide a rigorous justification for projecting performance from 128K to 1M tokens. What evidence supports the assumption that no new system bottlenecks emerge at these larger scales? Can you provide data from a scaled-down model or hardware configuration that validates this linear scaling assumption?
2. The ITQ matrix is trained on a 1K token sequence. Please provide an ablation study showing how the model's accuracy (in perplexity and, ideally, a downstream task like needle-in-a-haystack) degrades as the context length increases from 1K to 128K. How can the authors be confident this approach does not fail catastrophically at 1M tokens?
3. Given that the system requires extensive, context-dependent hyperparameter tuning, what is the proposed methodology for a practitioner to deploy LongSight on a novel LLM or for a new application? Please quantify the tuning overhead.
4. Why were state-of-the-art software-only sparse attention methods, which require no specialized hardware, omitted as primary performance baselines in Figure 7? A fair comparison would show where LongSight provides a benefit over what is achievable on the commodity GPU alone.
5. How does your CXL emulation model account for protocol overhead and contention from multiple concurrent requests targeting the DReX device? Please quantify the sensitivity of your results to a 2x or 5x increase in CXL tail latency, which can be common in real-world systems under load.

---

Review Form

Reviewer: The Synthesizer (Contextual Analyst)

Summary

This paper introduces LongSight, an algorithm-hardware co-design framework to accelerate large language model (LLM) inference for extremely long contexts. The central and most compelling idea is the repurposing of a compute-enabled CXL memory expander called DReX, originally designed for dense retrieval, to serve as an active offloading device for the LLM's Key-Value (KV) cache. LongSight implements a hybrid attention mechanism: a conventional dense attention is performed on a sliding window of recent tokens stored in the GPU's local HBM, while sparse attention for the vast historical context is handled by DReX. The GPU offloads query vectors to DReX, which leverages its in- and near-memory processing capabilities to efficiently perform a top-k similarity search, returning only the most relevant Keys and Values. This approach dramatically reduces the memory and compute burden on the GPU, enabling a single-GPU system to efficiently handle context lengths of up to one million tokens, a scale that is currently only feasible with large multi-GPU setups.

Strengths

The primary strength of this work lies in its elegant synthesis of ideas from several distinct but converging fields of research.

1. Novel Connection of Problem Domains: The authors make a brilliant conceptual leap by identifying the architectural similarity between top-k dense retrieval (the problem DReX was built for) and the core operation of a common form of sparse attention (finding the keys with the highest dot-product similarity to a query). By repurposing a specialized accelerator, they provide a powerful, hardware-grounded solution to the sparse attention problem, rather than just proposing another software-based heuristic. This connection is the paper's core intellectual contribution and is genuinely insightful.

2. Addressing a Critical System Bottleneck: The paper tackles one of the most significant challenges in modern AI: the memory and computational explosion associated with large context windows. Their claim of enabling 1M token contexts on a single GPU (as shown in Figure 7, Page 12) is not merely an incremental improvement; it represents an order-of-magnitude increase in accessibility for this class of models. This has the potential to democratize research and application development for use cases that are currently out of reach for most, such as processing entire books, legal archives, or extensive codebases in a single inference pass.

3. A Compelling Use Case for Modern Hardware Trends: LongSight serves as a powerful "killer app" for emerging architectural trends. It demonstrates the tangible benefits of:

   • Compute Express Link (CXL): The low-latency, coherent memory sharing provided by CXL is essential for the fine-grained interaction between the GPU and DReX.
   • Processing-in-Memory (PIM): The use of PIM for the initial sign-bit filtering stage (Section 7.1, Page 8) is a perfect application of the technology—a simple, highly parallelizable operation performed directly where the data resides, avoiding massive data movement.
   • Disaggregated/Tiered Memory: The paper provides a concrete vision for a smarter memory hierarchy, where the CXL-attached tier is not just for passive capacity expansion but is an active computational partner to the main processor.

4. Holistic Algorithm-Hardware Co-Design: The solution is thoughtfully designed across the entire stack. The algorithm (hybrid dense-sparse with ITQ-enhancement) is tailored to the hardware's capabilities (sign-based PIM filtering, near-memory dot-products). The data layout within DReX is carefully orchestrated to maximize parallelism (Figure 6, Page 10). This comprehensive, system-level thinking is commendable.

Weaknesses

While the core idea is powerful, its current presentation is tightly coupled to a specific research artifact, which raises questions about its generalizability.

1. Dependence on the DReX Architecture: The work is presented as an extension of DReX [34]. While this makes for a concrete and well-evaluated system, it leaves the reader wondering about the fundamental principles. The paper would be significantly strengthened by abstracting away from DReX and defining the core requirements for a "LongSight-capable" memory device. What are the necessary PIM capabilities, the required CXL bandwidth, and the near-memory compute primitives that make this approach viable? Without this, the work risks being seen as a bespoke solution rather than a generalizable architectural paradigm.

2. Complexity of Hyperparameter Management: The authors acknowledge that the system's performance and accuracy depend on a set of interdependent hyperparameters: the dense window size (W), the number of sparse tokens (k), and the per-head SCF thresholds. The pareto frontier in Figure 10 (Page 13) shows that tuning is critical for optimal performance. This introduces a layer of complexity for practitioners. A more robust system would ideally feature a method for automatically and dynamically adapting these parameters.

3. Limited Comparison to Alternative Sparsity Patterns: The paper focuses exclusively on top-k similarity as the criterion for sparsity. This is a reasonable and popular choice, but the field has explored other patterns (e.g., block-sparse, strided, global tokens as in Longformer [2]). A discussion of whether the DReX hardware could be programmed or adapted to accelerate other sparsity patterns would help contextualize the flexibility and limitations of their proposed hardware.

Questions to Address In Rebuttal

1. The work is tightly coupled with the DReX architecture [34]. Can the authors elaborate on the core architectural requirements for a compute-enabled memory device to effectively implement the LongSight approach? For instance, what are the minimum PIM capabilities (e.g., is sign-bit XOR sufficient?) and CXL bandwidth needed? This would help readers understand how the concept could transcend this specific hardware implementation.

2. The performance of LongSight appears sensitive to several hyperparameters (window size, k, thresholds), which may depend on the context length and task (Section 9.3, Page 13). Could the authors discuss the potential for automating this tuning process? Is there a risk that the overhead of finding optimal parameters could negate the performance benefits in some practical, dynamic deployment scenarios?

3. The paper elegantly repurposes a dense retrieval accelerator for sparse attention. This concept of offloading key primitives to specialized CXL memory seems very powerful. Are there other computationally expensive primitives in modern transformer models (e.g., Mixture-of-Experts routing, speculative decoding verification) that could be similarly offloaded and accelerated by a DReX-like device? A brief discussion on the broader applicability of this "accelerated memory" paradigm would significantly enhance the paper's impact.

---

Review Form

Reviewer: The Innovator (Novelty Specialist)

Summary

This paper presents LongSight, an algorithm-hardware co-design for accelerating large-context LLM inference. The core idea is to repurpose DReX [34], a compute-enabled CXL memory expander originally designed for dense retrieval (e.g., for RAG), to accelerate the sparse attention component of LLM inference. The authors propose a hybrid attention algorithm where a GPU handles a dense sliding window of recent tokens, while the vast history of the KV cache is offloaded to DReX. Within DReX, attention is treated as a top-k vector similarity search, leveraging DReX's in-memory sign-bit filtering and near-memory dot-product accelerators to efficiently find the most relevant Key vectors. The claimed novelty lies in this specific repurposing and the co-design of the algorithm, system integration, and hardware scheduling to enable dynamic, low-overhead sparse attention at a massive scale.

Strengths

The primary strength of this work is its novel synthesis of existing components to solve a well-defined and challenging problem. While individual elements are not entirely new, their combination is.

1. Novel Application of a Specialized Accelerator: The central novel claim—repurposing a dense retrieval accelerator (DReX) for dynamic LLM attention—is a compelling one. Prior art in hardware-accelerated attention (e.g., NeuPIMs [9], AttAcc [29]) has largely focused on accelerating dense attention computations. LongSight's approach of treating attention as a hardware-accelerated retrieval problem is a significant conceptual departure.

2. Addressing the Dynamic Update Challenge: The most significant point of novelty compared to other retrieval-based attention methods (e.g., Squeezed Attention [12], which uses standard ANNS) is how it handles the dynamic nature of the KV cache. Conventional ANNS methods require expensive index rebuilding upon data insertion, making them unsuitable for the per-token updates in autoregressive generation. LongSight’s use of DReX's index-free, sign-concordance filtering (SCF) mechanism provides a novel and practical solution to this critical limitation. This is the key "delta" that makes the contribution significant.

3. Novel System-Level Co-design: The contributions detailed in Section 7, particularly the extensions to the DReX CXL Controller (DCC) and the hierarchical data layout scheme (Figure 6), represent tangible, novel system design. This is not a simple "plug-and-play" use of DReX; it requires new hardware logic and a sophisticated software mapping to handle the granularity of multi-head, multi-layer, and multi-user attention requests.

Weaknesses

The work's novelty is somewhat constrained by its heavy reliance on a foundation of pre-existing concepts and a specific, previously proposed architecture.

1. Incremental Algorithmic Novelty: The hybrid dense-sparse attention algorithm itself is conceptually similar to established patterns. The idea of combining a dense sliding window for recency with a retrieval mechanism for long-term memory is present in various forms in prior work (e.g., Longformer [2], StreamingLLM [41]). The novelty is therefore not in the algorithm's high-level structure, but in its specific hardware-aware implementation.

2. Reliance on Closely-Related Prior Art: The proposed system is fundamentally an application built on top of DReX [34], a system proposed in a recent paper by many of the same authors. While repurposing an architecture is a valid contribution, it frames the work more as an extension or a new use case for DReX rather than a fundamentally new hardware architecture. The paper is transparent about this, but it bounds the scope of the novelty.

3. Synthesis vs. Fundamental Breakthrough: The work is a masterful piece of engineering and synthesis, combining ideas from sparse attention, vector databases, and near-data processing. However, it does not introduce a new, fundamental primitive. Its contribution is the clever and effective integration of existing primitives (CXL, PIM for filtering, near-memory acceleration) into a new system configuration.

Questions to Address In Rebuttal

1. The core idea of treating attention as a top-k retrieval problem is gaining traction, with Squeezed Attention [12] being a notable contemporary work. Your paper states that prior work supports a "fixed long context" (Section 4, page 4). Can you elaborate further on why standard ANNS methods are insufficient for the dynamic KV cache and how DReX's architectural features (specifically, the index-free filtering) are uniquely essential to overcoming this limitation? A more direct comparison would strengthen the novelty claim.

2. Your co-design is deeply tied to the specific architecture of DReX [34], including its two-stage filtering/scoring pipeline and PFU design. To what extent is the LongSight framework generalizable? Could the core principles be applied to other near-data or processing-in-memory architectures that may not feature the exact sign-concordance filtering mechanism? Or is the success of LongSight entirely contingent on the unique properties of DReX?

3. The use of Iterative Quantization (ITQ) [7] is presented as a key enabler for filter efficiency (Section 5.4, page 6). While effective, ITQ is a known technique for improving quantization performance. Is there any novelty in how ITQ is applied or integrated within the LongSight framework, or is its application here standard practice?