Characterizing the Efficiency of Distributed Training: A Power, Performance, and Thermal Perspective

---

Review Form

Reviewer: The Guardian (Adversarial Skeptic)

Summary

The authors present a characterization study of distributed Large Language Model (LLM) training. They evaluate performance, power, and thermal behavior across three modern GPU platforms (NVIDIA H100/H200, AMD MI250) using several dense and sparse models. The study examines the effects of different parallelism strategies (Tensor, Pipeline, Data, Expert) and common software optimizations like activation recomputation and compute-communication overlap. The authors conclude with several insights, including the nuanced trade-offs between scale-up and scale-out systems, inefficiencies in hybrid parallelism schemes, the limits of microbatch scaling, and the performance impact of thermal imbalances. Finally, they attempt to project their findings to datacenter-scale systems via simulation.

Strengths

1. Experimental Testbed: The study is conducted on an impressive and relevant set of modern hardware (H200, H100, MI250). Access to such platforms for a systematic study is a notable strength.
2. Breadth of Experiments: The authors have undertaken a significant experimental effort, covering multiple models, parallelism configurations, and optimization techniques. This provides a broad, albeit thin, survey of the design space.
3. Inclusion of Thermal Analysis: The focus on thermal imbalance (Section 6) and its concrete impact on clock throttling (Figure 17) is a valuable contribution. It highlights a practical, physical-layer constraint often overlooked in purely algorithmic performance studies.

Weaknesses

The paper's ambition is commendable, but its execution lacks the necessary rigor, leading to shallow analyses and insufficiently substantiated claims.

1. Insufficient Methodological Detail: The foundation of any characterization study is its measurement methodology, which is inadequately described here.

   • The authors state they use a "modified version of Zeus" (Page 4, Section 3.1) for energy and telemetry. The nature and impact of these modifications are not specified. What is the measurement overhead? How was the modified tool validated against ground truth? Without this information, the fidelity of all power, thermal, and utilization data is questionable.
   • For the AMD MI250 platform, the paper states that "smaller versions of GPT-3 and Llama-3" were used due to memory constraints (Page 5, Section 3.2). The process of scaling down these models is not detailed. Architectural changes during scaling (e.g., number of layers vs. hidden dimension) can drastically alter the compute-to-communication ratio, making the cross-platform comparison to the full-size models on NVIDIA hardware potentially invalid. The claim of providing "valuable cross-platform insights" is therefore weakened.

2. Superficial Analysis and Overstated Insights: The paper identifies several well-known phenomena but fails to provide deep, quantitative explanations for their root causes.

   • Scale-up vs. Scale-out (Section 4.1): The conclusion that the optimal strategy "depends on model size, sparsity, and parallelism strategy" is not a novel insight. The analysis attributes performance differences to "communication locality" and "inter-node traffic" (Page 5), but fails to provide a quantitative breakdown. For instance, in the GPT3-175B case where H200 excels, what precise percentage of the performance gain is due to avoiding inter-node AllReduce versus exploiting higher intra-node NVLink bandwidth for Tensor Parallelism? The kernel breakdowns in Figure 3 are a start, but the narrative connecting them to the high-level claims is tenuous.
   • Limits of Microbatch Scaling (Section 5): The paper correctly observes that increasing microbatch size can harm performance but attributes this to vague causes like "communication bandwidth saturation" and "pipeline-induced stalls" (Page 10, Insight box). Which specific communication fabric is saturating (PCIe, InfiniBand)? Figure 15 shows that AllReduce and SendRecv time increases, but provides no evidence of why. Is this due to increased message size leading to network congestion, or is it a tail-latency effect from stragglers? The analysis stops short of identifying the true bottleneck.

3. Unsupported Extrapolation and Speculation:

   • The projection to 8K-GPU systems in Section 7.1 is the paper's most significant flaw. The authors switch from empirical measurement on at most 64 GPUs to simulation with Astra-Sim. The paper provides zero detail on how the simulator was parameterized or calibrated against their real-system measurements. Network simulators are notoriously difficult to configure accurately, and without a rigorous calibration methodology, the results presented in Figure 22 are purely speculative and cannot be considered a valid extension of the paper's empirical findings. This section undermines the work's grounding in real-world characterization.

4. Inclusion of Underdeveloped and Contradictory Results:

   • The "thermal-aware pipeline parallelism strategy" presented at the end of Section 6 and in Figure 21 is a premature and distracting inclusion. It is presented as a solution, yet the results are mixed: a meager 4% efficiency gain for Llama3 is contrasted with a 7% efficiency degradation for GPT3-175B. The paper glosses over this negative result. Such a preliminary and inconclusive experiment does not belong in a characterization paper and weakens its focus and credibility.

Questions to Address In Rebuttal

The authors must provide precise, data-driven answers to the following questions:

1. Regarding your "modified version of Zeus" (Page 4): What specific modifications were made? Provide data on the validation of this tool and quantify its measurement overhead on the system.
2. Regarding the scaled-down models for the MI250 (Page 5): Detail the exact methodology used to scale down GPT-3 and Llama-3. Provide evidence that these smaller variants maintain the same fundamental bottleneck characteristics (e.g., compute-bound vs. memory-bound, communication patterns) as their full-sized counterparts, thereby justifying the cross-platform comparison.
3. Regarding the claim of "communication bandwidth saturation" with larger microbatches (Page 10): Provide specific data from your telemetry (e.g., PCIe bus utilization, InfiniBand NIC throughput) that directly demonstrates saturation of a specific hardware resource. Correlate this saturation point with the observed performance degradation.
4. Regarding the thermal-aware scheduling experiment (Page 12, Figure 21): Explain the 7% performance degradation observed for GPT3-175B. Given this negative result, what is the justification for including this experiment as a positive contribution?
5. Regarding the 8K GPU extrapolation (Page 12, Section 7.1): Provide a complete and detailed account of the calibration process for Astra-Sim. How were kernel latencies, network parameters, and collective communication models from your 32/64-GPU empirical measurements translated to the simulator to ensure the validity of the 8K-GPU projections?

---

Review Form

Reviewer: The Synthesizer (Contextual Analyst)

Summary

This paper presents a comprehensive, multi-faceted characterization of distributed training for large language models (LLMs). The authors move beyond traditional performance metrics (e.g., throughput) to provide a holistic analysis that incorporates power consumption, hardware utilization, and thermal behavior. By conducting experiments on a diverse set of modern hardware platforms (NVIDIA H100/H200, AMD MI250) and across various models and parallelism strategies (TP, PP, DP, EP), the authors empirically investigate the complex, second-order effects that arise at scale.

The core contribution is the assertion, backed by extensive data, that optimizing large-scale training requires a "full-stack" perspective. The paper demonstrates that software-level decisions—such as the choice of parallelism strategy, microbatch size, and optimizations like activation recomputation—have profound and often non-intuitive interactions with the physical realities of the underlying hardware, including network topology, power limits, and thermal dissipation. Key findings include the nuanced trade-offs between scale-up and scale-out systems, the communication inefficiencies of certain parallelism combinations (TP+PP), the performance pitfalls of excessive microbatch scaling, and the significant impact of thermal imbalance on system reliability and throughput.

Strengths

This is an excellent and timely study that provides a much-needed bridge between the domains of ML systems software and computer architecture/datacenter operations. Its primary strengths are:

1. Holistic and Grounded Perspective: The single most important strength of this paper is its commitment to analyzing the entire stack. While many papers focus on optimizing parallelism strategies in the abstract (e.g., Megatron-LM, DeepSpeed), and others focus on datacenter efficiency, this work is one of the first to rigorously connect the two. It moves the conversation from idealized performance models to the messy, physical realities of running these workloads, which is where the next set of bottlenecks lies. The focus on thermal throttling and power draw (Sections 5 & 6, pages 9-11) is particularly novel and significant for the training domain.

2. Methodological Rigor and Relevance: The experimental setup is state-of-the-art and highly relevant. The use of H100, H200, and MI250 GPUs covers the most important accelerator architectures in the market today. The choice of workloads, from dense models like GPT-3 and Llama3 to sparse MoE models like Mixtral, ensures the findings are broadly applicable. The fine-grained telemetry collection provides a solid empirical foundation for the paper's claims.

3. Revealing Non-Obvious Interactions: The paper excels at uncovering insights that challenge conventional wisdom. For example:

   • The finding that higher-memory "scale-up" systems (32xH200) can outperform "scale-out" systems (64xH100) in communication-heavy regimes (Section 4.1, page 5) highlights that raw aggregate compute is not the only factor in performance.
   • The demonstration that increasing microbatch size can harm performance due to communication saturation and thermal stress (Section 5, page 9, Figure 13) provides a crucial, practical guideline that contradicts the simplistic "bigger is better" assumption.
   • The clear visualization of thermal imbalance due to physical node layout (Section 6, page 10, Figure 17) and its direct link to performance throttling is a powerful demonstration of how physical constraints impact distributed algorithms.

4. Connecting to Broader Research Agendas: This work provides foundational data that can inform multiple research communities. It implicitly challenges automated parallelism frameworks (e.g., Alpa) to incorporate physical constraints into their cost models. It provides concrete motivation for work on topology-aware communication collectives (e.g., TACCL, TopoOpt). Finally, it extends the investigation of power- and thermal-aware scheduling, previously explored for inference (e.g., TAPAS, DynamoLLM), into the synchronous, long-running, and highly-coupled domain of LLM training.

Weaknesses

The weaknesses of the paper are primarily related to its framing and the scope of its conclusions, rather than fundamental flaws in the methodology or results.

1. Characterization vs. Solution: The paper is, at its heart, a characterization study. It does an excellent job of identifying and quantifying problems. While the brief exploration of thermal-aware pipeline scheduling is a step in the right direction (Section 7, page 12), it feels preliminary compared to the depth of the problem analysis. The paper would be strengthened if the authors were more explicit in framing their work as foundational characterization that motivates the need for new solutions, rather than presenting a complete solution itself.

2. Generalizability of Topology-Specific Findings: The results are derived from specific cluster topologies (e.g., NVIDIA HGX nodes connected via InfiniBand). While these are common, the authors could do more to discuss how their findings might change in systems with different network topologies (e.g., Dragonfly, optical circuit switches) or cooling systems (e.g., direct liquid cooling). For instance, the severe thermal imbalance shown in Figure 17 might be less pronounced in a liquid-cooled system, which would alter the trade-off calculus for different parallelism strategies.

3. Depth of Causal Analysis: The paper establishes strong correlations between software choices and physical effects (e.g., PP-heavy configurations lead to higher peak power). However, a deeper microarchitectural analysis explaining why these patterns emerge would be beneficial. For example, what specific resource contention (e.g., on memory controllers, PCIe bus) during compute-communication overlap leads to the observed thermal stress? While likely outside the primary scope, adding some discussion here would further solidify the paper's contribution.

Questions to Address In Rebuttal

1. The thermal-aware pipeline stage placement experiment (Section 7, page 12) is very promising. Could you elaborate on the limitations of this approach? For instance, how does the strategy adapt if the "hot" and "cold" GPUs change during a long training run, and how much of the performance gain is tied to the specific model architecture (e.g., number of layers being divisible by the number of stages)?

2. Your work compellingly demonstrates the importance of physical node layout and network topology. How do you foresee your key insights—particularly regarding scale-up vs. scale-out and TP+PP inefficiencies—translating to future disaggregated or chiplet-based systems where memory, compute, and networking resources may be composed in more flexible ways?

3. The industry is increasingly moving towards advanced cooling solutions like direct liquid cooling to manage the thermal density of modern accelerators. How would such a technology alter the conclusions of your study? Would it eliminate the thermal bottleneck entirely, or would it simply shift the bottleneck to another system component (e.g., power delivery, interconnect bandwidth)?

---

Review Form

Reviewer: The Innovator (Novelty Specialist)

Summary

The authors present a comprehensive characterization study of distributed Large Language Model (LLM) training. The work's central thesis is that traditional performance-centric analysis is insufficient, and a holistic perspective incorporating power consumption and thermal behavior is necessary to understand system efficiency. The authors evaluate various parallelism strategies (Tensor, Pipeline, Data, Expert) and common optimizations (activation recomputation, compute-communication overlap) across modern hardware platforms (NVIDIA H100/H200, AMD MI250).

The paper does not propose a new algorithm, hardware architecture, or training technique. Its claim to novelty rests entirely on the insights derived from its multi-faceted characterization. Specifically, it claims to be the first to systematically and quantitatively link specific software-level parallelism and optimization choices to their physical, second-order consequences, such as thermal imbalance, power excursions, and clock throttling, in the context of state-of-the-art LLM training systems.

Strengths

The primary contribution of this work is the rigorous documentation of system behaviors that, while perhaps suspected by practitioners, have not been systematically studied and published in an academic venue. The novelty is not in the invention of a new method, but in the elucidation of previously unquantified, non-obvious system interactions.

Specific novel insights include:

1. The Counter-Intuitive Limits of Microbatch Scaling: While conventional wisdom suggests larger microbatches are better if memory allows, this paper provides concrete evidence that beyond an optimal point, they create bursty execution patterns that lead to higher peak power and worsened thermal throttling, ultimately degrading performance (Section 5, page 9, Figure 13). This is a significant finding that challenges common heuristics.

2. Quantification of Thermal Imbalance Impact: The paper moves beyond acknowledging thermal issues to showing a direct causal link between server airflow design (Figure 16, page 10), GPU placement, the chosen parallelism strategy (high-PP configurations), and persistent clock throttling on specific GPUs (Figure 17, page 11). This demonstrates that applying uniform software optimizations to physically non-uniform hardware is a flawed strategy.

3. Inefficiency of Combined Parallelism Strategies: The analysis revealing that the combination of Tensor Parallelism and Pipeline Parallelism (TP+PP) leads to underutilization of PCIe bandwidth due to sparse, uncoordinated SendRecv calls is a specific and novel finding (Section 4.2, page 7). This identifies a concrete inefficiency in current software frameworks that was not previously highlighted.

Weaknesses

My main critique stems from the definition of novelty. This is fundamentally a characterization study, an established research methodology. The work synthesizes existing measurement tools and techniques to analyze existing software on existing hardware.

1. Lack of a Constructive Contribution: The paper is diagnostic, not prescriptive. It identifies and quantifies numerous problems but stops short of proposing and evaluating a novel mechanism to solve them. For example, after demonstrating the negative impact of thermal imbalance, the authors offer recommendations but do not implement a novel thermal-aware scheduler. The brief experiment on thermal-aware placement in the discussion (Section 7, page 12) is a simple heuristic (placing heavy stages on cold GPUs) and is presented more as a proof-of-concept than a fully-fledged novel algorithm. The core of the paper remains observational.

2. Conceptual Overlap with Prior Art: While the authors focus on LLM training, the core concept of co-designing software with an awareness of power and thermal constraints is not new. Prior work in HPC and datacenter management has long explored thermal-aware job scheduling. Google's technical blog post, "Balance of power: A full-stack approach to power and thermal fluctuations in ML infrastructure" [16], which the authors cite, discusses this exact problem space at a high level. This paper's contribution is the granular, quantitative analysis for specific LLM parallelism strategies, which makes it an incremental, albeit valuable, advancement over the conceptual state-of-the-art rather than a paradigm shift.

3. Dependence on Established Optimizations: The techniques analyzed—activation recomputation [28], compute-communication overlap [75], FSDP [82]—are all well-established. The paper's contribution is to map their secondary effects, not to introduce a new optimization.

In essence, the paper provides a new, high-resolution map of a known territory. The map is useful and reveals details not seen before, but it is not the discovery of a new continent.

Questions to Address In Rebuttal

1. The paper's core novelty lies in its insights. Could the authors consolidate their findings and explicitly state which of their documented system interactions they believe were truly unknown to the community (both academic and industrial) prior to this work, versus those that were "common knowledge" but previously unquantified?

2. The work expertly diagnoses the sub-optimality of applying uniform optimizations to thermally heterogeneous hardware. The lack of a proposed novel mechanism to address this is a significant limitation. Can the authors justify why they chose not to propose and evaluate a new scheduling algorithm based on their findings? Does the brief thermal-aware placement experiment (page 12) contain a novel algorithmic component that was not fully elaborated upon?

3. How does this work's contribution differ fundamentally from the full-stack power/thermal analysis described in prior industrial reports, such as Google's blog post [16]? Please precisely articulate the delta beyond being a more detailed, academic study. Is the novelty simply the level of granularity, or is there a conceptual advance?