HardHarvest: Hardware-Supported Core Harvesting for Microservices

Title: HardHarvest: Hardware-Supported Core Harvesting for Microservices
Reviewer: The Guardian

---

Summary

The authors propose HardHarvest, a hardware architecture designed to support core harvesting in microservice environments. The paper identifies two primary overheads in existing software-based approaches: the latency of hypervisor-driven core reassignment and the performance penalty from flushing/invalidating private caches and TLBs. To address this, HardHarvest introduces three main hardware features: 1) a hardware request scheduler with per-VM queues to accelerate core reassignment, 2) way-partitioning of private caches/TLBs into "Harvest" and "Non-Harvest" regions to preserve Primary VM state, and 3) a novel replacement algorithm that attempts to steer shared application data into the preserved "Non-Harvest" region. The evaluation, conducted via full-system simulation, claims that HardHarvest significantly reduces Primary VM tail latency by 6.0x and increases Harvest VM throughput by 1.8x compared to a state-of-the-art software baseline.

Strengths

1. Well-Defined Problem: The paper correctly identifies a significant and timely problem. The overheads associated with software-based core harvesting, particularly in the context of latency-sensitive microservices, are substantial and well-articulated in Section 3. The motivational analysis is convincing in establishing the potential for a hardware-based solution.
2. Comprehensive Design: The proposed solution is multifaceted, addressing both the control plane (core reassignment) and data plane (cache state) aspects of the problem. This shows a thorough understanding of the challenges involved.
3. Detailed Breakdown of Gains: The cumulative breakdown of performance improvements in Figure 12 is a methodologically sound way to attribute gains to specific architectural features, which is commendable.

Weaknesses

My primary role is to ensure the rigor and validity of new claims against the established state-of-the-art. In this capacity, I have identified several critical weaknesses that challenge the paper's core conclusions.

1. Potentially Unfair Baseline Comparison: The central claim of a 6.0x tail latency reduction rests on the comparison against "state-of-the-art software core harvesting" (Harvest-Term), which is modeled after SmartHarvest [88]. However, a crucial feature of SmartHarvest for mitigating tail latency is its use of an "emergency buffer" of idle cores that can be reclaimed instantly without software overhead. The authors mention this feature in Section 4 (Page 4) but it is entirely unclear whether their simulated software baseline (Harvest-Term) actually implements this buffer. Without it, the baseline is significantly weakened, and the comparison is arguably one against a strawman. The dramatic latency reduction could simply be an artifact of crippling the baseline.

2. Conflation of General-Purpose and Harvesting-Specific Gains: The results in Figure 11 show that HardHarvest not only improves upon software harvesting but also achieves significantly lower tail latency than the NoHarvest baseline. The authors attribute this to "improved cache/TLB replacement and request queuing" (Section 6.1, Page 11). This is a critical issue. It implies that a substantial portion of the claimed benefit is derived from a general-purpose hardware scheduler and a new cache policy, not from the harvesting mechanism itself. The paper is framed as a solution for harvesting, but the evidence suggests it is a paper about a new scheduler that also happens to do harvesting. This conflates two separate contributions and overstates the benefits attributable to the novel harvesting architecture.

3. The Fragility of the Cache Replacement Heuristic: The proposed replacement algorithm (Algorithm 1, Section 4.2.3) is entirely dependent on a heuristic to classify pages as "shared" or "private." The paper proposes a simple temporal heuristic: memory allocated before server.serve() is shared, and memory allocated after is private. This is a fragile assumption that is unlikely to hold for many real-world microservices that might perform lazy initialization, use dynamic data structures, or rely on just-in-time compilation. The paper presents no sensitivity analysis on the accuracy of this classifier. If pages are frequently misclassified, this "smart" replacement policy could easily degrade into a worst-case scenario, polluting the protected Non-Harvest region with transient data or evicting critical shared data. The entire benefit of cache partitioning rests on this unvalidated heuristic.

4. Understated Hardware Complexity and Cost: The paper proposes a non-trivial centralized hardware controller (Figure 9) with dedicated network links, multiple Queue Managers, and VM State Registers. The cost analysis in Section 6.8, based on McPAT, focuses almost exclusively on the storage overhead of the request queue and extra cache bits, concluding a mere 0.19% area overhead. This seems to grossly underestimate the true cost. The analysis likely neglects the area and power of the complex control logic for the scheduler, the cross-VM interrupt mechanism, the dynamic queue management, and the priority multiplexers for the new replacement policy. Furthermore, the centralized design presents a potential scalability bottleneck for future processors with hundreds of cores, a concern that is not addressed.

Questions to Address In Rebuttal

The authors must address the following points directly and with specific evidence from their experiments to convince me of the validity of their work.

1. On the Baseline's Fidelity: Please confirm whether your simulated software baseline (Harvest-Term) implements the emergency core buffer from SmartHarvest [88]. If it does not, please justify why this is still a fair "state-of-the-art" comparison and how your conclusions would change if it were included.
2. On Deconflating Contributions: Can you provide an evaluation that isolates the performance benefit derived purely from the harvesting-specific mechanisms (i.e., cache partitioning and the associated replacement policy) from the benefits of the general-purpose hardware request scheduler and queues? For example, by comparing NoHarvest against a NoHarvest+HardHarvestScheduler configuration.
3. On the Replacement Heuristic's Robustness: What is the performance impact on a Primary VM if the shared/private page classification heuristic has a high error rate (e.g., 25% or 50% misclassification)? Please provide a sensitivity analysis or a more robust defense of why this simple heuristic is sufficient for a general-purpose hardware mechanism.
4. On Hardware Cost Realism: Can you provide a more detailed breakdown of the hardware cost analysis that includes the control logic for the scheduler, queue managers, and replacement policy, not just the storage elements? Please also comment on the scalability of the centralized controller design to processors with significantly more than 36 cores.

Paper: HardHarvest: Hardware-Supported Core Harvesting for Microservices
Reviewer: The Synthesizer

---

Summary

This paper presents HardHarvest, a novel hardware architecture designed to enable efficient core harvesting in microservice environments. The authors identify a critical bottleneck in existing software-based harvesting techniques (e.g., SmartHarvest): the high overhead of reassigning cores between Virtual Machines (VMs), which involves hypervisor calls and extensive cache/TLB flushes. These overheads, while tolerable for monolithic applications, are prohibitive for latency-sensitive microservices that operate on sub-millisecond timescales.

HardHarvest's core contribution is a holistic, hardware-first solution that tackles these overheads directly. It proposes a two-pronged approach: 1) a hardware-based request scheduling system with per-VM queues that allows cores to be reassigned between Primary (latency-critical) and Harvest (batch) VMs without hypervisor intervention, and 2) a hardware-managed partitioning of private caches and TLBs into "Harvest" and "Non-Harvest" regions. This partitioning scheme preserves the Primary VM's "hot" state in the Non-Harvest region during harvesting, dramatically reducing the cold-start penalty upon core reclamation. The evaluation, performed via full-system simulation, demonstrates significant benefits, most notably reducing the tail latency of Primary VMs by 6.0x compared to software harvesting, to the point where it is even better than a non-harvesting baseline, while simultaneously increasing Harvest VM throughput by 1.8x.

Strengths

This work's primary strength lies in its excellent synthesis of a critical industry problem with a well-motivated and elegant architectural solution.

1. Timely and Significant Problem: The paper addresses a problem of immense practical and economic importance. The tension between provisioning for peak load (leading to low average utilization) and meeting strict tail-latency SLOs is a central challenge in modern cloud infrastructure. The authors correctly identify that existing software solutions for resource harvesting, developed in the context of longer-running applications, are a poor fit for the microsecond-scale world of microservices. This work is therefore not just an academic exercise but a direct response to a real-world architectural gap.

2. A Conceptual Leap in Core Harvesting: The move from software to hardware for this task is a significant conceptual advance. While prior work has proposed hardware support for microservice scheduling (e.g., µManycore [76], RPCValet [15]), HardHarvest is, to my knowledge, the first to propose a comprehensive hardware framework for resource elasticity at this granularity. It connects the dots between datacenter economics and microarchitecture in a compelling way.

3. Holistic and Co-Designed Solution: The true novelty is not in any single component but in the co-design of the two main features. The hardware scheduler solves the core reassignment latency, while the cache partitioning solves the resulting microarchitectural cold-start problem. This integrated approach shows a deep understanding of the full performance stack, from the hypervisor down to the cache replacement policy. The proposed replacement algorithm (Algorithm 1, page 9), which attempts to steer shared vs. private pages, is a particularly thoughtful refinement that demonstrates this depth.

4. Strong Quantitative Motivation: The "Motivation" section (Section 3, pages 3-4) is exemplary. By systematically measuring and presenting the overheads of hypervisor calls (Figure 4) and cache flushes (Figure 5), the authors build an undeniable case for a hardware-level intervention. This data-driven motivation makes the subsequent design feel necessary rather than contrived.

Weaknesses

The weaknesses of the paper are primarily related to the broader systems context and potential complexities that are not fully explored.

1. System Complexity and Interaction: The paper presents HardHarvest as a self-contained module. In a real processor, this logic would need to interact with a host of other complex features: hardware security enforcers, performance monitoring units, NUMA optimizations, and emerging protocols like CXL. For instance, how would the hardware scheduler's decisions interact with the OS/hypervisor's broader power management or process scheduling policies? The paper could be strengthened by acknowledging and briefly discussing these integration challenges.

2. Security Implications Beyond Flushing: The security model is predicated on flushing the "Harvest" region of the caches to prevent direct data leakage. While this is a necessary first step, the design—which intentionally preserves a Primary VM's state on a core while a Harvest VM executes—creates a novel scenario for side-channel analysis. Could rapid, hardware-managed switching create new timing channels related to resource contention in the core's backend or the control logic of the cache partitioning itself? The brief mention of adding a delay to prevent timing channels (page 8) feels insufficient for a system operating at this level of intimacy between tenants.

3. Robustness of Workload Assumptions: The efficacy of the advanced cache replacement policy hinges on the ability to reliably distinguish between "shared" (e.g., application code, read-only data) and "private" (request-specific) memory pages. The proposed heuristic (based on allocation time relative to a server.serve() call) is clever and likely works for the C++/gRPC-style services evaluated. However, its applicability to microservices built in other ecosystems (e.g., Java with a JIT, Go with its own runtime scheduler) is not obvious. The performance of the system could be sensitive to the accuracy of this classification, making this a potential point of fragility.

Questions to Address In Rebuttal

1. On System Integration: Could the authors comment on the potential interactions between the HardHarvest controller and other modern SoC features, such as Intel's Thread Director or system-wide power management policies? Is there a risk of the HardHarvest hardware scheduler and the OS/hypervisor schedulers working at cross-purposes?

2. On Side-Channel Security: The paper's security model relies on flushing the Harvest region. Could the authors elaborate on potential timing channels or other side-channels that might arise from a Primary VM's state being preserved in the Non-Harvest region while a Harvest VM is executing on the same core? For example, could the Harvest VM infer information by observing the performance effects of the replacement policy (Algorithm 1) acting on the Primary VM's hidden state?

3. On the Generality of the Replacement Policy: The efficacy of the specialized replacement policy hinges on accurately identifying shared pages. How sensitive are the performance benefits to the accuracy of this heuristic? Have the authors considered how this heuristic would apply to microservices built with different frameworks (e.g., those using garbage collection or JIT compilation) where the distinction between shared and private allocations may be less clear?

Reviewer: The Innovator (Novelty Specialist)

---

Summary

This paper proposes HardHarvest, a hardware architecture designed to support core harvesting specifically for microservice environments. The authors' central claim is that this is the first hardware-based solution to this problem, addressing the prohibitive overheads of existing software-based approaches. The core of the proposed architecture consists of two main components: 1) a hardware-based request scheduler that manages core re-assignment between "Primary" (latency-critical) and "Harvest" (batch) VMs without hypervisor intervention, and 2) a hardware mechanism for partitioning private caches/TLBs, coupled with a novel replacement algorithm (Algorithm 1, page 9) that aims to preserve the Primary VM's "warm" state by prioritizing shared pages in a protected partition. The authors claim their solution significantly improves core utilization and Harvest VM throughput while reducing Primary VM P99 tail latency by 6.0x compared to state-of-the-art software harvesting.

---

Strengths

The primary strength of this paper lies in its novel synthesis of architectural concepts to solve a well-motivated problem. While individual components of the solution have roots in prior work, their combination and specific application to inter-VM core harvesting for microservices appears to be genuinely new.

1. Novel Problem-Architecture Mapping: The core idea of creating a dedicated hardware architecture for core harvesting is, to my knowledge, novel. Prior art has focused on software techniques for harvesting (e.g., SmartHarvest [88]) or hardware acceleration for RPC scheduling (e.g., µManycore [76], ALTOCUMULUS [96]), but not the explicit, hardware-managed, dynamic lending and reclamation of cores between different VM types.

2. Novel Cache/TLB Replacement Policy: The most distinct and novel technical contribution is the proposed cache/TLB replacement algorithm (Section 4.2.3). The idea of partitioning the cache is not new (e.g., Intel CAT), but the policy itself is. It introduces awareness of both the region type (Harvest vs. Non-Harvest) and the page type (shared vs. private) into the victim selection process. This fine-grained, policy-driven management to selectively preserve a Primary VM's working set across preemption is a clever microarchitectural technique that I have not seen proposed in prior work.

3. Significant Delta Over Prior Art: The proposed hardware scheduler extends prior work on RPC schedulers by adding logic specific to harvesting: it is aware of VM types (Primary/Harvest), manages core "loans," and handles the preemption and reclamation protocol (Section 4.1.5). This is a non-trivial delta that makes it fundamentally different from a simple request scheduler.

---

Weaknesses

My critique focuses on precisely delineating the novel contributions from the integration of existing ideas, and questioning the robustness of the assumptions upon which the novelty rests.

1. Conflation of Novelty with Integration: The paper presents HardHarvest as a monolithic novel architecture. However, it is more accurately an integration of several concepts, some novel and some drawn directly from prior work. For example, the fast context switching is attributed to µManycore [76], and the need for an efficient flush/invalidate mechanism is acknowledged as a known problem with existing solutions [30, 51]. The impressive 6.0x tail latency reduction is therefore a result of the entire system, and it is difficult to isolate the benefit derived purely from the novel components (the VM-aware scheduler and the replacement policy) versus the benefit from simply implementing previously-proposed hardware accelerations for context switching and cache management. The breakdown in Figure 12 (page 11) is cumulative, which makes it hard to assess the standalone value of the truly new ideas.

2. Novelty Reliant on a Heuristic: The effectiveness of the novel replacement policy (Algorithm 1) is entirely dependent on a software heuristic for classifying pages as "shared" or "private" (Section 4.2.2, page 8). The paper suggests a simple rule: data allocated before server.serve() is shared. While plausible for the frameworks studied, this is not a hardware novelty and may be fragile. The paper does not sufficiently explore the generalizability of this heuristic. If the heuristic fails, the core benefit of the novel replacement algorithm is nullified, even if the hardware is implemented perfectly.

3. Complexity vs. Benefit of the Novel Components: The proposed hardware is significant, requiring a centralized controller, dedicated network, and modifications to every core's cache/TLB replacement logic. While the overall performance gain is large, it is unclear how much of that gain is attributable to the new, complex parts of the design. A simpler design that implements hardware context switching and static cache partitioning (using existing techniques) might achieve a substantial portion of these gains. The paper lacks an ablation study to justify the added complexity of its novel scheduler and replacement policy over simpler hardware extensions.

---

Questions to Address In Rebuttal

1. The central claim is that HardHarvest is the "first" architecture for core harvesting in hardware. While it appears novel in the context of mainstream CPU architecture, can the authors elaborate on whether similar concepts of hardware-managed resource borrowing/lending have been explored in adjacent fields, such as reconfigurable computing (FPGAs) or specialized manycore processors, even if not explicitly termed "core harvesting"?

2. To better isolate the paper's novel contributions, can the authors provide a more direct "apples-to-apples" comparison? For instance, what would the performance be with a baseline that includes previously-proposed hardware context switching and efficient flushing, but uses a simpler hardware scheduler (not VM-aware) and standard cache partitioning (e.g., static ways via CAT) without the novel replacement policy? This would help quantify the specific benefit of the new ideas presented in this work.

3. The novel replacement policy's success is predicated on the shared/private page classification heuristic. How would the performance of HardHarvest degrade if this heuristic were, for instance, only 50% accurate? A sensitivity analysis on the accuracy of this software-level assumption would strengthen the claims about the hardware's utility.