Multi-Dimensional ML-Pipeline Optimization in Cost-Effective Disaggregated Datacenter

---

Review Form

Reviewer: The Guardian (Adversarial Skeptic)

Summary

The paper proposes an auto-tuning framework for optimizing multi-stage ML pipelines on NUMA systems, including those with emulated CXL memory. The framework utilizes an eBPF-based monitor to collect performance data with low overhead and a user-space core that employs Bayesian Optimization (BO) to navigate the configuration space of thread counts and memory interleaving ratios. The stated goal is to maximize throughput under SLA latency constraints, with a secondary phase using Pareto analysis for power efficiency. The authors claim significant throughput improvements (up to 48%) and search cost reductions (up to 77%) over existing methods.

However, the work is predicated on a fundamentally flawed CXL emulation methodology and relies on modeled, rather than measured, power data. These weaknesses significantly undermine the credibility of the paper's core performance and efficiency claims, particularly those related to CXL environments.

Strengths

• The problem statement is well-defined and highly relevant to current trends in datacenter architecture and ML deployment.
• The use of eBPF for performance monitoring (Section 3.1.2) is a methodologically sound choice for achieving low-overhead, in-kernel data collection, avoiding the significant performance degradation common with traditional daemon-based tools.
• The evaluation includes comparisons against relevant and recent academic work, specifically TPP [40] and Caption [69], which is commendable.

Weaknesses

1. Invalid CXL Emulation: The paper's entire premise of optimizing for CXL-based disaggregated memory is built on a weak foundation. In Section 2.5, the authors state they "emulated a CXL... memory pool... by disabling all local sockets and treating remote DRAM as a dedicated memory pool." This is an oversimplification that borders on being incorrect. A remote NUMA link is not CXL. This emulation completely ignores critical CXL-specific characteristics such as protocol overhead from the CXL.mem transaction layer, the behavior of the device-side CXL controller and host-side Home Agent (HA), and potential coherency traffic differences. The authors' unsubstantiated claim of a "discrepancy within less than 5% of those observed in our emulation" (Section 2.5.1, Page 4) is presented without any supporting data, experimental setup, or reference to the "actual CXL hardware" used for this validation. Without this evidence, all results pertaining to CXL are speculative at best.

2. Power Claims are Not Empirically Validated: The power optimization phase (Section 4.2) and the associated power savings claims (up to 14.3% in the abstract) are not based on hardware measurements. The authors explicitly state they "estimate the CXL power using two different models" (Section 4, Page 9), dubbed "CXL-preferred" and "DRAM-preferred." These models are based on high-level assumptions about DDR4/DDR5 refresh rates and supply voltages. Consequently, the Pareto frontiers shown in Figure 11 are the result of a simulation, not an empirical observation of a real system. Such claims of power savings are meaningless without physical measurement.

3. Insufficient Justification for Bayesian Optimization: While BO is a powerful technique, the authors fail to demonstrate that its complexity is warranted for this problem. The primary claim is a 77% reduction in search cost (Abstract). This is compared to a pseudo-exhaustive search, which is a strawman baseline. The more relevant comparison is the time-to-solution versus a simpler, robust heuristic. In Figure 6, Caption [69] appears to find a strong configuration very quickly (at low "Search Cost") before its performance degrades. Why does Caption degrade? The paper offers no analysis, simply stating it "falls to local minima." This is an insufficient explanation. The authors must provide a rigorous analysis of why the simpler heuristic fails and demonstrate that BO's overhead is justified by consistently finding a superior solution within a practical time budget.

4. Questionable Baseline Behavior: The performance of the Caption baseline in Figure 6 is highly suspect. The algorithm is designed to incrementally adjust allocation and hill-climb towards a better configuration. The consistent and sharp decline in throughput as search cost increases suggests either a flawed implementation of the baseline or an experimental artifact that the authors have not investigated or explained. A system designed to improve performance should not actively make it worse over time unless it is unstable, which would be a critical finding in itself.

5. Ambiguity in Overhead Measurement: The claim of "over 5× less overhead than traditional Linux daemon-based tools" and the specific figure of 4.7% overhead for the eBPF monitor (Section 4.1.1, Page 9) lacks context. Was this 4.7% overhead measured on an idle system or under the full load of the benchmark workloads? System monitoring overhead is often non-linear and can become significantly more pronounced under high resource contention. The paper must clarify the conditions under which this overhead was measured to validate its "low-overhead" claim.

Questions to Address In Rebuttal

1. Please provide the complete data and methodology for the validation of your CXL emulation against "actual CXL hardware," as mentioned in Section 2.5.1. This should include the specific hardware used, the workloads run, and the performance metrics that showed a <5% discrepancy. Without this, how can any of the CXL-related conclusions be trusted?

2. Given that the power analysis is based entirely on a model, please justify how you can make concrete claims of power savings (e.g., "7.3% power by sacrificing around 4.5% in QPS" in Section 4.2). At a minimum, the authors should rephrase these as purely theoretical findings and explicitly state they are not based on hardware measurements.

3. Provide a detailed analysis explaining the performance degradation of the Caption baseline in Figure 6. Why does a hill-climbing algorithm consistently result in a worse configuration over time? Is this a limitation of the original algorithm or an artifact of your implementation/environment?

4. Please clarify the exact system load conditions under which the 4.7% eBPF monitor overhead was measured.

5. The comparison with TPP (Section 4.1.4) shows significant latency improvements. TPP is primarily designed for transparent capacity tiering. Were your experiments configured in a way that memory capacity was a bottleneck? If not, please justify why TPP is an appropriate performance baseline for a bandwidth-tuning framework.

---

Review Form

Reviewer: The Synthesizer (Contextual Analyst)

Summary

This paper presents a novel, adaptive auto-tuning framework for optimizing multi-stage Machine Learning (ML) inference pipelines in datacenters with disaggregated memory architectures, specifically those enabled by Compute eXpress Link (CXL). The core problem addressed is the combinatorial explosion of configuration parameters—such as memory allocation ratios between local DRAM and CXL, and thread-to-core mappings—that arises in these new, heterogeneous systems. The authors propose a holistic, two-phase optimization approach. Phase 1 uses Bayesian Optimization (BO) to navigate the high-dimensional search space and maximize system throughput under Service Level Agreement (SLA) latency constraints. Phase 2 further refines the set of high-performing configurations by using Pareto optimality to identify the most power-efficient options. A key technical contribution is the use of an eBPF-based kernel module for low-overhead, vendor-agnostic performance monitoring, which feeds the user-space optimization core. The experimental results, conducted on a range of ML workloads, demonstrate significant improvements over default and state-of-the-art configurations, achieving up to a 48% throughput increase while simultaneously reducing search costs and enabling substantial power savings.

Strengths

The primary strength of this work lies in its timely and insightful synthesis of several key trends in modern computing. It provides a cohesive and practical solution to a critical, emerging problem.

1. High Relevance to an Emerging Hardware Paradigm: The paper is exceptionally well-timed. As CXL-enabled servers move from research prototypes to production deployments, the industry will urgently need intelligent management systems. This work directly addresses the fundamental challenge that CXL introduces: a vastly expanded and more complex memory hierarchy. It moves beyond simple page placement heuristics seen in prior work (e.g., "Caption" [69], "TPP" [40]) by embracing the multi-dimensional nature of the problem, making it a forward-looking and highly relevant contribution.

2. Holistic and Principled System Design: The authors have designed a system that is both powerful and pragmatic. The choice of eBPF for monitoring is astute, providing a low-overhead, transparent, and—most importantly—vendor-agnostic solution that overcomes the limitations of prior platform-specific tools (as noted in Challenge C2, page 2). The coupling of this monitor with a Bayesian Optimization core is a natural fit; BO is an ideal tool for optimizing expensive-to-evaluate black-box functions, which perfectly describes tuning a live datacenter workload. The two-phase approach, separating performance and power optimization, is also a very practical design choice, reflecting real-world operational priorities.

3. Connecting Research to Real-World Economics: A significant strength is the paper's grounding in the total cost of ownership (TCO) of datacenter operations. The analysis extends beyond raw performance metrics (QPS, latency) to include power consumption (Section 4.2, page 11) and a CAPEX/OPEX comparison against GPU-based solutions (Section 4.3, page 11). This demonstrates a deep understanding of what matters in practice and elevates the work from a purely academic exercise to a solution with a clear value proposition for datacenter operators.

4. Strong Empirical Foundation: The characterization study in Section 2.5 (page 3) effectively motivates the entire work. Figure 2, in particular, clearly illustrates that no single static memory configuration is optimal across all loads, justifying the need for a dynamic, adaptive tuner. The subsequent evaluation is comprehensive, using a diverse set of modern ML pipeline benchmarks and comparing against multiple relevant baselines, including a state-of-the-art CXL management system. The results convincingly demonstrate the superiority of the proposed BO-based approach.

Weaknesses

The weaknesses identified are not fundamental flaws but rather areas where the current implementation and evaluation could be expanded to address the full complexity of production environments.

1. Oversimplified Pipeline Stage Management: The paper describes a mechanism for managing pipeline stages using POSIX semaphores and tracking kmalloc events with eBPF to detect the end of a stage's memory allocation (Section 3.1.3, page 6). While clever, this approach seems potentially fragile. It assumes a well-behaved application structure where stages are cleanly separated and their memory allocation phases are distinct. It is unclear how this would generalize to more complex, asynchronous pipelines or those written in higher-level languages where memory management is less explicit.

2. Unexplored Dynamics of Optimization Convergence: The framework's value proposition depends on its ability to find an optimal configuration in a reasonable amount of time. The authors report search times of "5 to 17 minutes" (page 10), which is excellent for a stable workload. However, datacenter load is often dynamic, with characteristics that can change on shorter timescales. The paper does not explore the system's reactivity. For instance, if a workload's input data characteristics shift dramatically, how quickly can the framework abandon its current model and re-converge on a new optimum? The current evaluation focuses on a static optimization problem rather than a continuous, dynamic one.

3. Limited Scope (Single-Tenant Focus): The current evaluation is conducted in a single-application, single-tenant context. While the authors briefly discuss extending the framework to multi-tenancy in the Discussion (Section 5.4, page 12), this is a non-trivial extension. In a multi-tenant environment, the optimization for one workload could negatively impact another (the "noisy neighbor" problem). The BO's objective function would need to be reformulated to account for fairness, QoS guarantees for multiple tenants, and global resource contention, which presents a far more complex optimization landscape.

Questions to Address In Rebuttal

1. Regarding the pipeline stage management mechanism: Could the authors elaborate on the robustness of using kmalloc tracking and semaphores? Have they considered alternative pipelines where memory allocation is interleaved with computation, and if so, how would the framework handle such cases? What are the practical limitations on the types of application pipelines the system can currently support?

2. Regarding the real-time adaptability of the system: Can the authors comment on the trade-off between the exploration budget (search cost) of the Bayesian Optimization and the framework's ability to react to dynamic changes in workload behavior? For example, how would the system perform if the workload trace from Figure 7 experienced a sudden, sustained spike, fundamentally changing the latency-throughput curve?

3. While multi-tenancy is noted as future work, could the authors provide a more concrete vision for this extension? Specifically, how might the Bayesian Optimization framework be adapted? Would it involve a multi-objective optimization problem (e.g., balancing the throughput of all tenants), or would it require adding fairness constraints directly into the acquisition function? How would the system ensure performance isolation while optimizing for global efficiency?

---

Review Form

Reviewer: The Innovator (Novelty Specialist)

Summary

The paper presents a framework for optimizing the performance and power consumption of multi-stage Machine Learning (ML) pipelines on NUMA systems with heterogeneous memory, specifically including CXL-attached memory. The core of the proposed solution is an adaptive auto-tuner that operates in two phases. The first phase uses Bayesian Optimization (BO) to maximize throughput by exploring a multi-dimensional configuration space of per-stage memory allocation ratios and thread counts. The second phase uses Pareto optimality to select a configuration from the high-performing set that also minimizes power consumption. A key element of the framework's architecture is its use of an eBPF-based kernel module for low-overhead, hardware-agnostic performance monitoring, which feeds metrics into the user-space optimization core.

Strengths

The primary novel contribution of this work lies in the architectural synthesis of its components to create a practical, low-overhead, and portable auto-tuning system. My analysis identifies two key areas of novelty:

1. The Monitoring Mechanism as a Basis for Auto-Tuning: The decision to use an eBPF-based kernel module (Section 3.1, page 5) for real-time monitoring is a distinct and valuable departure from prior art. Many related works, such as Caption [69] and other vendor-specific studies [81, 85], rely on periodically sampling hardware performance counters (e.g., Intel PCM) or daemon-based tools. These approaches suffer from either a lack of portability across different CPU vendors or significant overhead due to context switching, as the authors correctly identify (Section 1, C2, page 2). By integrating monitoring directly and efficiently into the kernel via eBPF, the authors present a genuinely novel approach in this specific context that addresses both portability and overhead, which their own measurements confirm (Section 4.1.1, page 9).

2. Sophistication of the Search Strategy: The work moves beyond the simpler search heuristics seen in closely related work. For instance, Caption [69] employs a binary search-like algorithm to tune a single dimension (the CXL memory ratio). This paper's use of Bayesian Optimization to navigate a multi-dimensional space (per-stage memory ratios and thread counts) is a significant step up in capability. While applying BO to systems optimization is not new in a general sense, its application to this specific, complex problem of per-stage ML pipeline tuning on CXL is a novel application that demonstrably reduces search cost compared to exhaustive methods and finds better optima than simpler heuristics.

Weaknesses

While the overall framework is novel in its composition, its individual building blocks are well-established. My primary critique is that the paper presents a significant engineering contribution by cleverly integrating existing technologies, but it does not introduce a fundamentally new optimization algorithm or a new theoretical insight into system performance.

1. Component-level Novelty is Limited: Bayesian Optimization is a standard technique for black-box function optimization. Pareto optimality is a classic method for multi-objective decision-making. Using eBPF for system tracing and monitoring is also a widely adopted practice. The novelty here is exclusively in the combination and application of these tools to the problem of ML pipeline tuning on disaggregated memory. The paper should be careful not to overstate the novelty of the constituent parts.

2. Incremental Advance over Conceptually Similar Ideas: The core idea of a closed-loop system that monitors performance and adjusts resource allocation is not new. The delta here is in the "how": using eBPF instead of Perf/PCM, and BO instead of hill-climbing/binary-search. While the results show this delta is impactful (e.g., Figure 6, page 10), the conceptual framework of "monitor -> decide -> actuate" is familiar. The paper's contribution is a much more effective implementation of this concept, not the invention of the concept itself.

3. Scoped Novelty: The paper's novelty is scoped almost exclusively to CPU-based inference on NUMA/CXL systems. This is a relevant domain, but the broader trend in large-scale ML involves heterogeneous systems with hardware accelerators (GPUs, TPUs). While the authors suggest a potential extension to XPUs (Section 5.2, page 12), this remains speculative. The demonstrated novelty is confined to a specific, albeit important, hardware class.

Questions to Address In Rebuttal

To strengthen the paper's claims of novelty, I would ask the authors to address the following:

1. Clarifying the Conceptual Delta: The paper successfully argues that its framework is superior to prior art like Caption [69] and TPP [40]. However, can the authors articulate the single most significant conceptual leap this work makes? Is it the move from single-dimensional to multi-dimensional search, the adoption of kernel-level monitoring for this specific feedback loop, or another factor? A more precise articulation of the core inventive step, beyond just being a more effective integration, would be valuable.

2. Justification of Bayesian Optimization: The rationale for choosing Bayesian Optimization is that it is well-suited for expensive black-box functions. The paper's own results in Figure 6 (page 10) show that a Genetic Algorithm (GA) also achieves strong performance (84-89% of optimal), sometimes appearing more stable than PSO. Could the authors provide a more rigorous justification for why the additional complexity and specific modeling assumptions of a Gaussian Process in BO are fundamentally better for this problem space than other global search metaheuristics? Is the 5-10% performance delta over GA worth the implementation and computational overhead of BO?

3. eBPF Hooking Strategy: The use of eBPF is the strongest novel component. Could the authors provide more technical detail on the specific kernel events, tracepoints, or kprobes they hook into to monitor the start and end of a stage's execution and memory allocation (kmalloc events mentioned in Section 3.1.3)? Discussing the robustness of these hooks across different kernel versions would also strengthen the claim of portability.