Dilu: Enabling GPU Resourcing-on-Demand for Serverless DL Serving via Introspective Elasticity

Paper Title: Dilu: Enabling GPU Resourcing-on-Demand for Serverless DL Serving via Introspective Elasticity
Reviewer: The Guardian

---

Summary

The paper presents Dilu, a serverless DL serving system designed to mitigate GPU fragmentation and improve resource utilization. The authors introduce "introspective elasticity," a concept realized through a 2D co-scaling mechanism that combines fast, intra-instance vertical scaling (adjusting SM quotas) with lazy, inter-instance horizontal scaling (launching/terminating instances). The system is comprised of three main components: a multi-factor profiler to determine resource requirements, a resourcing-complementary scheduler to collocate tasks efficiently, and the adaptive co-scaling mechanism to manage resources dynamically. The authors claim that Dilu significantly reduces GPU fragmentation, increases throughput, and maintains QoS guarantees compared to existing baselines.

Strengths

1. Problem Motivation: The paper correctly identifies a critical and timely problem. GPU fragmentation in serverless DL systems, driven by static allocation policies and workload dynamism, is a well-known issue that erodes the cost-efficiency promises of the serverless paradigm. The motivation presented in Section 1 and Figure 2 is clear.

2. Conceptual Approach: The high-level idea of combining vertical and horizontal scaling (termed "2D co-scaling") is sound. Addressing elasticity at both the intra-instance (fine-grained) and inter-instance (coarse-grained) levels is a logical approach to tackling the multifaceted nature of resource management in this domain.

3. Evaluation Breadth: The authors have conducted an extensive set of experiments in Section 5, comparing Dilu against multiple relevant baselines (Exclusive, MPS, FaST-GS, TGS) across various collocation scenarios (training-inference, inference-inference, etc.) and workload patterns. The inclusion of an ablation study (Section 5.4) is commendable.

Weaknesses

My primary concerns with this paper lie in the insufficient validation of core mechanisms, a lack of clarity on crucial implementation details, and potential overstatement of novelty.

1. Unquantified Overhead of the Core Mechanism: The entire vertical scaling capability of Dilu hinges on the Real-time CUDA Kernel Manager (RCKM), which intercepts CUDA API calls via LD_PRELOAD (Section 3.4.1, page 7). This is a highly invasive technique. The paper, however, provides zero quantification of the intrinsic performance overhead of this interception layer itself. The "Vertical scaling overhead" analysis in Figure 11 is misleadingly titled; it measures the impact of collocation on application performance, not the base overhead of the RCKM framework on a single, unimpeded instance. Without this crucial data point, it is impossible to assess the net benefit of Dilu. A mechanism that saves 10% of resources but introduces a 15% performance penalty is a net loss.

2. Insufficient Differentiation from Prior Art: The RCKM mechanism, as described, appears functionally very similar to the temporal sharing mechanisms in prior work, particularly TGS [47], which is cited and used as a baseline. Both use a centralized manager and client-side interception to control kernel execution. The paper fails to articulate the fundamental technical novelty of its token-based vertical scaling mechanism over these existing approaches. The novelty seems to lie in the control logic (Algorithm 2), but the underlying architecture is not new, and this is not adequately discussed.

3. Lack of Rigor in Profiling Validation: The profiler (Section 3.2, page 5) is the foundation upon which all subsequent scheduling and scaling decisions are built. However, the evaluation of the profiler in Table 2 focuses exclusively on efficiency (i.e., the number of iterations to find a configuration). It presents no evidence of accuracy or optimality. How do the authors validate that the "optimal" <SMR, IBS> configuration found by their Hybrid Growth Search Strategy is indeed the ground-truth optimal, or even close to it? A fast profiler that finds a suboptimal configuration compromises the entire system's performance. This is a critical omission.

4. Vague Description of the Co-Scaling Coordination: The paper claims a key contribution is the adaptive 2D co-scaling, yet the coordination logic between the "fast" vertical scaling and "lazy" horizontal scaling is poorly defined. The description in Section 3.4.2 (page 8) is high-level and relies on arbitrary-seeming window sizes (size=40s) and thresholds (φ_out, φ_in). What happens in a scenario of sustained high load that exceeds the capacity of vertical scaling? How is the "lazy" delay determined, and how does the system avoid severe SLO violations during this delay? The mechanism feels more like a collection of heuristics than a robust, principled control system. The claims of a "smooth transition" are not substantiated by a rigorous explanation of the control loop.

5. Potential for Unfair Baseline Comparisons: While the set of baselines is good, the paper provides insufficient detail on their configuration and tuning. For instance, systems like FaST-GS [19] and TGS [47] have their own internal heuristics and parameters. Were these baselines tuned to perform optimally for the specific workloads used in this evaluation? Without this information, there is a risk that the performance gains attributed to Dilu are partially an artifact of sub-optimally configured baselines. Specifically for TGS, it is a temporal scheduler; its performance is highly dependent on how priorities are assigned to jobs. This is not discussed.

Questions to Address In Rebuttal

The authors must address the following points directly and with concrete data to salvage this submission.

1. Please provide a microbenchmark that quantifies the intrinsic latency and throughput overhead imposed by the RCKM interception library on various CUDA kernel calls, independent of any collocation effects. This should be measured on a single instance running without contention.

2. Beyond the control logic in Algorithm 2, what are the precise technical and architectural novelties of the RCKM mechanism when compared directly to the temporal sharing framework presented in TGS [47]?

3. Please provide evidence for the accuracy of the profiling strategies. For at least one representative model, compare the configuration found by your profiler against a ground truth established by an exhaustive grid search to demonstrate how close to optimal your heuristic gets.

4. Could you provide a more detailed algorithm or state machine diagram that clearly illustrates the coordination logic and state transitions between fast vertical scaling-up and lazy horizontal scaling-out, especially under conditions of sustained high load? Please justify the choice of parameters like the 40s window.

5. The affinity-first scheduling principle (Section 3.3, page 6) aims to reduce the "barrel effect." However, how does this principle avoid creating a different form of fragmentation, where certain GPUs become specialized for specific function types, leaving stranded resources that cannot be used by newly arriving, non-affine functions?

Reviewer: The Synthesizer (Contextual Analyst)

Summary

This paper introduces "Introspective Elasticity (IE)," a novel two-dimensional co-scaling paradigm designed to address the significant problem of GPU fragmentation in serverless Deep Learning (DL) serving. The authors argue that traditional serverless systems, which rely solely on horizontal scaling (adding/removing instances), are ill-suited for the dynamic and resource-intensive nature of DL workloads, leading to overprovisioning and inefficiency.

The core of their contribution is a system called Dilu, which materializes IE through a cross-layer architecture. Dilu combines:

1. Fast, fine-grained vertical scaling: Dynamically adjusting the GPU compute (SM) quotas allocated to running instances on a sub-second timescale to handle short-term workload bursts. This is managed by a token-based runtime mechanism called RCKM.
2. Lazy, coarse-grained horizontal scaling: Making slower, more deliberate decisions to launch or terminate entire instances to adapt to sustained changes in workload.

This 2D co-scaling approach is supported by a multi-factor profiler to determine optimal resource quotas (<request, limit>) for DL tasks and a resource-complementary scheduler that collocates heterogeneous functions to maximize GPU utilization. The experimental results demonstrate significant improvements over existing approaches, showing reduced fragmentation, higher aggregate throughput, and lower SLO violation rates.

Strengths

1. A Compelling and Timely Core Concept: The paper's central thesis—that serverless DL requires a more sophisticated, two-dimensional elasticity model—is both insightful and highly relevant. The rise of LLMs has made GPU efficiency a first-order concern for cloud providers. The proposed "Introspective Elasticity" provides a clear conceptual framework for solving the impedance mismatch between the slow, disruptive nature of horizontal scaling (with its associated cold starts) and the highly bursty, sub-second reality of inference workloads. The idea of using fast vertical scaling as a first line of defense to absorb bursts is elegant and powerful.

2. Excellent Synthesis of Ideas: This work sits at the crossroads of several research domains—serverless computing, GPU virtualization/sharing, and DL systems scheduling—and does an admirable job of synthesizing ideas from each. It takes the elasticity principle from serverless, combines it with fine-grained temporal GPU sharing mechanisms seen in cluster computing (e.g., TGS, Antman), and applies it to the specific problem of heterogeneous DL task collocation. The result is a cohesive system that is more than the sum of its parts. The paper effectively bridges the gap between high-level serverless orchestration and low-level GPU resource management.

3. Strong Systems Contribution and Implementation: The authors have not just proposed an idea; they have built and evaluated a complete system. The architecture presented in Figure 3 (page 4) is well-reasoned, with clear separation of concerns across the control, scaling, and serving planes. The design of the Real-time CUDA Kernel Manager (RCKM) in Section 3.4.1 (page 7) is a practical approach to implementing dynamic, fine-grained control without modifying the GPU driver. The comprehensive evaluation across various workloads and collocation scenarios provides strong evidence for the system's effectiveness.

Weaknesses

While the work is strong, its positioning and some practical considerations could be strengthened.

1. Positioning of the Core Abstraction: The term "Introspective Elasticity" is new and catchy, but the paper could do more to contextualize it within the broader history of adaptive and autonomic computing. The idea of a multi-dimensional control loop that adjusts resources at different granularities and timescales is not entirely new. The novelty here lies in its specific, cross-layer application to the serverless GPU context. A more nuanced discussion of how IE relates to or evolves from prior concepts in adaptive resource management would help solidify its place in the literature, moving it from a system-specific name to a more generalizable principle.

2. Security Implications in a Multi-Tenant Environment: The paper briefly mentions in Section 3.4.1 (page 8) that it relies on container isolation for security. This is insufficient for a system intended for a multi-tenant serverless platform. The fine-grained temporal sharing of a physical GPU, managed by the RCKM, opens a potential surface for timing-based side-channel attacks between functions owned by different tenants. A malicious function could potentially infer information about a co-located victim's workload by observing perturbations in its own kernel execution latencies. This is a well-known concern in shared resource environments and warrants a more serious discussion of the security model and potential mitigation strategies.

3. Practicality of the Runtime Control Mechanism: The RCKM intercepts CUDA API calls to manage kernel execution via a token system. While clever, this introduces overhead on the critical path of every kernel launch. The paper claims this overhead is "negligible" (Section 5.2, page 11), but this is asserted rather than demonstrated with microbenchmarks. For workloads with very high frequencies of small kernels (a common pattern in some models), this interception and communication overhead could become significant. A more detailed analysis of this overhead would increase confidence in the mechanism's real-world viability.

Questions to Address In Rebuttal

1. Could the authors please elaborate on the conceptual novelty of "Introspective Elasticity" compared to prior work on multi-dimensional or hierarchical resource management in cluster systems? How is IE fundamentally different from, for example, a system that combines cluster-level auto-scaling with node-level CPU frequency scaling or I/O throttling?

2. The RCKM mechanism introduces a control loop into every kernel launch. Can you provide microbenchmark data quantifying this overhead? Specifically, what is the latency added to a single cuLaunchKernel call, and how does this impact the end-to-end performance of workloads characterized by a high frequency of short-duration kernels?

3. Beyond relying on containerization, what is your security model for preventing information leakage between co-located functions from different tenants? Have you considered the potential for timing-based side-channel attacks through the shared RCKM and GPU execution pipeline, and what mitigations might be possible within your framework?

Reviewer: The Innovator (Novelty Specialist)

Summary

The authors present Dilu, a serverless DL system centered around a concept they term "Introspective Elasticity" (IE). The core idea is the coordinated, two-dimensional co-scaling of GPU resources. This is achieved by combining: 1) fast, fine-grained, intra-instance vertical scaling (dynamically adjusting an instance's SM quota) to handle immediate workload fluctuations, and 2) lazy, inter-instance horizontal scaling (adding/removing instances) to manage sustained changes in load. The system is realized through a cross-layer architecture featuring multi-factor profiling, a resource-complementary scheduler, and a real-time CUDA kernel manager (RCKM) for vertical scaling. The paper's central claim is that this holistic co-scaling approach can significantly reduce GPU fragmentation, improve throughput, and guarantee QoS in serverless DL environments, advancing beyond prior art which focuses on only one scaling dimension.

Strengths

The primary strength of this work lies in its novel architectural synthesis. My analysis focuses on the degree to which the core ideas advance the state of the art.

1. Novelty of the Core "Introspective Elasticity" Concept: The central contribution is the co-design of fast, intra-instance vertical scaling (scale-up/down) with lazy, inter-instance horizontal scaling (scale-out/in). This departs from prior serverless systems like INFless [51] or FaST-GS [19], which are limited to horizontal scaling over statically-partitioned GPU resources (via MPS). It also advances beyond prior GPU sharing work, such as TGS [47] or Antman [49], which focuses on single-node resource management (temporal sharing) without integrating it into a broader, cluster-wide serverless auto-scaling framework. The explicit "fast-up, lazy-out" dynamic described in Section 3.4 (pages 7-8) is a direct and novel outcome of this architectural synthesis, designed specifically to mitigate cold-start overheads while maintaining elasticity.

2. Advancement in Dynamic Resource Provisioning: The paper makes a significant leap from the static spatial partitioning of MPS [38], which underpins many existing systems [19, 51], to a truly dynamic vertical scaling mechanism. The RCKM (Section 3.4.1, Figure 6) provides continuous, on-demand adjustment of compute quotas without the overhead or limitations of reconfiguring MPS partitions. While the use of LD_PRELOAD to intercept CUDA calls is not new in itself, its application in a token-based system that dynamically adjusts allocations between competing serverless instances based on kernel execution rates represents a meaningful step forward from discrete time-slicing or static priority-based schemes.

Weaknesses

While the overall architecture is novel, it is important to deconstruct the novelty of its constituent parts to accurately place the work in the context of prior art.

1. Precedent for Component-Level Mechanisms:

   • The mechanism for vertical scaling, a token-based kernel throttling system implemented via LD_PRELOAD (RCKM, Section 3.4.1), is conceptually similar to prior work in temporal GPU sharing. Systems like TGS [47] and other container-based GPU sharing frameworks [20, 52] also employ monitor-and-control architectures to manage kernel execution. The novelty here is not in the interception mechanism itself, but in its specific control logic (driven by Kernel Launch Cycle changes) and its tight integration with the horizontal scaler. This distinction should be made clearer.
   • Similarly, the resourcing-complementary scheduling (Section 3.3) is a well-reasoned application of 2D bin-packing heuristics to the problem of co-locating DL tasks. The principles of affinity and complementarity are known in cluster scheduling. The contribution here is an effective implementation for this specific problem domain, rather than a fundamentally new scheduling theory.

2. Marginal Novelty of Profiling Strategy: The Hybrid Growth Search strategy for profiling (Section 3.2, page 5) is an efficiency improvement over exhaustive search or model-based prediction. While effective, it is an incremental advancement—a clever heuristic for navigating a search space. It supports the main contribution but is not, in itself, a significant conceptual leap. The core novelty of the paper would stand even with a less efficient, brute-force profiling method.

Questions to Address In Rebuttal

1. The vertical scaling mechanism (RCKM) shares conceptual underpinnings with systems like TGS [47] which also timeshare the GPU. Could the authors elaborate on the fundamental differences in the control logic (e.g., token-based vs. priority-based time-slicing) that make Dilu's approach uniquely suited for guaranteeing tight latency SLOs in a serverless DL context?

2. The "lazy" horizontal scaling paradigm is predicated on the vertical scaler's ability to absorb transient workload bursts. What are the empirical limits of this absorption capability? At what point does a workload burst become too large or sustained for vertical scaling to handle alone, forcing a reactive (non-lazy) horizontal scale-out and potentially negating some of the claimed benefits of avoiding cold starts?

3. The paper argues that its dynamic vertical scaling is superior to the static partitions of MPS. However, MPS provides strong performance isolation between processes. What, if any, performance interference (e.g., memory bandwidth contention, L2 cache pollution) was observed between co-located instances under Dilu's dynamic token-issuing scheme, and how does the system mitigate it beyond throttling SM access? Is the isolation provided by containers sufficient?