Characterizing Job Power Elasticity for Power-Flexible AI Training

2609.11542 ↗ · 2026-09-10 · cs.AI

Philip Colangelo, Charles Dawson, Shayan Sengupta, Ayse Coskun, Varun Sivaram

The finding

Headline claim

LLM training jobs vary systematically in how much throughput they keep when GPU power is cut, and a simple telemetry-derived index (PFI) can predict that flexibility online and steer power allocation so clusters shed power cheaply during grid peaks.

The test

131 controlled training runs (plus 24 validation and 34 matched H100 runs) on H200 clusters of 8, 16, and 32 GPUs on GCP, sweeping per-GPU power caps from 200W to 700W across four open-weight models (dense and MoE, up to 70B), pretraining and LoRA fine-tuning; PFI computed per sweep group, predictors fit on DCGM telemetry with leave-one-out and leave-one-cluster-out evaluation.

Effect size & scope

PFI ranges 1.10 to 2.67 across jobs: dense pretraining jobs are least flexible (median 1.27), MoE fine-tuning most (median 2.10), with the MoE fine-tuning difference statistically significant (corrected p<0.05) but MoE pretraining not separable from the others. Under a 30% power cut, PFI-aware allocation recovers about 1.5k tokens/s per job, 63% of the gap between equal-weight and an oracle allocation. The mechanism is that H200 power caps throttle SM clocks but not memory clocks, so memory-bound jobs shed power cheaply.

Limitations

Authors concede n=25 independent PFI samples (25 sweep groups), too few to justify nonlinear models or more features; no jobs with PFI<1 were observed so the elasticity spectrum is truncated; H100 validation is limited; the mechanism's generality across accelerator architectures is untested; PFI is hardware-dependent and not comparable across chip generations. Unstated: this is training only, not inference (which dominates deployed compute); the orchestration demo is on simulated workloads; all runs are on one cloud, one accelerator family, models up to 70B at up to 32 GPUs, far below frontier training scale; the paper is by a power-flexibility vendor (Emerald AI), a possible conflict of interest.

Counterpoint

A skeptic would note the sample is tiny (25 independent observations) on a single accelerator generation at a single cloud, that 'first systematic characterization' claims rest on a narrow job matrix, that the linear predictor's out-of-sample R2 is modest (LOCO R2 about 0.25), and that the throughput-recovery headline comes from simulation rather than a real grid event. Vendor-authored infrastructure papers deserve a prior discount until independently replicated.

Economy implication

In the next 12 to 24 months, if it generalizes, this softens the binding constraint of [2.1]: AI clusters that can shed maybe 30% of GPU power for a few peak hours with limited throughput loss could interconnect faster and lower their effective power cost, easing the power bottleneck on training capacity. But the evidence is at 32-GPU lab scale, so it says little yet about frontier data-center economics.

Who this matters for

Market & Valuation: Power interconnection is a top constraint on AI capacity build-out; a credible demand-flexibility primitive changes how much compute a given grid connection can host, relevant to data-center siting and capex plans.

Technical Capability: Establishes that training throughput under power constraint is a measurable, predictable workload property, an infrastructure capability that could become a requirement for large clusters.

Bears on (advisory · papers never write evidence)

[2.1] Grid interconnection / power timelines exceed data-center construction timelines in major US markets.

[2.4] Data-center demand is measurably pressuring regional power prices / emissions.

Abstract

Large language model (LLM) training is among the fastest-growing sources of electricity demand in modern data centers, and power availability is a primary bottleneck to continued AI infrastructure growth. Making the power consumption of these workloads flexible could unlock additional power for AI growth, limit increases in electricity prices, and improve the utilization of existing grid infrastructure. However, to realize this flexibility, we must first understand how the performance of training workloads changes when GPU power is reduced. This paper presents the first systematic characterization of \emph{job power elasticity} (the sensitivity of throughput to power reductions) in LLM training. To quantify elasticity, we introduce the \emph{Power Flexibility Index (PFI)}, a normalized metric that quantifies the performance cost of power reductions and provides a control primitive for SLA-aware power flexibility. We collect data from 131 LLM training runs on H200 (plus 24 H200 validation runs and 34 matched H100 runs), including both dense and mixture-of-experts models, pretraining and fine-tuning tasks, and up to 32 GPUs. We find that LLM training jobs exhibit substantial but variable power elasticity, and we identify telemetry signals that predict PFI at runtime. Finally, we demonstrate that PFI-aware power allocation maximizes total tokens/second throughput under power constraints. Under a 30\% power reduction, PFI-aware power allocation recovers ~1.5k tokens/s per job, 63\% of the performance gap between an equal-weight allocation and an oracle with perfect information. Our results establish power elasticity as a measurable property of training jobs and provide a foundation for power-aware, grid-responsive AI infrastructure.