EasyRider: Mitigating Power Transients in Datacenter-Scale Training Workloads
This page was written by an AI model to give a high-level overview of the paper. Please check the paper for detail: arXiv.
Large training jobs run thousands of GPUs in lockstep: they all compute, all pause to exchange gradients, then all compute again. Each pause drops every GPU's power at the same moment. An NVIDIA H100 falls from 700 W to 140 W within milliseconds, so a 50,000-GPU job like Meta's Llama-3 swings between about 35 MW and 7 MW. These swings repeat every training iteration (typically 1 to 10 Hz) and at every checkpoint, restart and collective stall.
The power grid cannot follow that. Supply must equal demand at every instant, and generators are spinning machines whose output changes slowly; the paper cites 10 to 20 MW per minute even for fast gas units. When load moves faster, grid voltage and frequency leave their safe ranges, and protection equipment disconnects loads, which is itself destabilizing at gigawatt scale. Existing mitigations each cover part of the problem: GPU "burn" kernels fill power dips but waste energy and depend on the training stack; platform capacitors (NVIDIA GB300) handle only short events on one platform; software-triggered rack batteries are limited by telemetry delay and battery chemistry; and site-level batteries sit above the datacenter's internal power distribution.
EasyRider treats the problem as filtering. A grid operator's limits on how fast load may ramp, and on how much of the load may oscillate at fast time scales, are a low-pass filter specification. EasyRider is a rack power distribution unit (PDU) that sits between one GPU rack and the row's DC busbar and applies that filter to the rack's power draw in hardware, with no software in the transient path and no change to the training job.
It has three hardware stages. A passive inductor-capacitor (LC) input filter removes the fastest content. A DC-DC regulator holds the rack's voltage constant. A battery behind a bidirectional converter absorbs energy when rack power drops and supplies it when rack power rises, under a control law that bounds the grid-side ramp rate by construction. A small software controller runs on a slower timescale only to keep the battery's state of charge near its midpoint; if it fails, the hardware still smooths transients.
The authors built a 10 kW, 400 V DC prototype with a lithium iron phosphate (LiFePO4) battery. Driven by a published normalized training trace, it kept the grid-side ramp rate within ±10% of rated power per second and pushed the power spectrum above 2 Hz below the assumed grid limit. On a 2-GPU Titan X server, a software burn baseline that smooths the same job consumed 19% more total energy than the rack with EasyRider. Cluster-scale behavior (a 40 MW cluster) is shown by simulation, by scaling the per-rack result.
Key takeaways
- Synchronous training makes every GPU's power drop and rise together, every iteration; at 50,000 H100s that is a swing between about 35 MW and 7 MW within milliseconds.
- Grid operators' limits (maximum ramp rate, limited fast oscillation) can be written as a low-pass filter specification on each rack's power draw, and per-rack limits add up to campus-level limits.
- EasyRider is a rack PDU: a passive LC filter for fast content, a DC-DC regulator for rack voltage, and a battery whose control law bounds the grid-side ramp rate. No software is in the transient path and training code is unchanged.
- A slow software loop only manages the battery's state of charge (a mid-band target during training, a lower one during long idle periods); a crash lets charge drift but does not stop smoothing.
- On a 10 kW, 400 V DC prototype: grid-side ramp within ±10% of rated power per second, spectrum above 2 Hz below the assumed limit, and a GPU burn baseline that used 19% more energy on a 2-GPU testbed. Cluster scale is simulated.
A per-rack low-pass filter for training power, built from an LC stage, a battery and a slow controller
Why training swings power, what the grid tolerates, how the limits become a filter specification, the three hardware stages, the battery controller, and the prototype results.
1 · Why synchronous training makes power swing
In synchronous data-parallel training, every GPU computes gradients on its share of the batch and then waits while gradients are exchanged with collective communication. Because all GPUs reach the communication phase together, their power drops together. The paper cites peak-to-idle ratios of 5:1 to 20:1 for current accelerators:
| Scale | Compute phase | Communication phase | Time to swing |
|---|---|---|---|
| One H100 | 700 W | 140 W | milliseconds |
| One B200 | 1000 W | 50 W | milliseconds |
| 10,000 GPUs | cluster power drops by 5 to 15 MW | hundreds of milliseconds | |
| 50,000 GPUs (Llama-3 scale) | 35 MW | 7 MW | milliseconds |
All digital hardware has transients, but regulators on motherboards, GPUs and power supplies already smooth those shorter than about 1 ms (above 1 kHz). Training adds events in the 100 ms to 10 s range (0.1 to 10 Hz): iterations (typically 1 to 10 Hz), checkpoints, restarts, collective stalls and job start and stop. That range overlaps the bulk power system's oscillation modes, where grid equipment has little damping and protection is most sensitive. Ordinary datacenters avoid this because they run many small, unsynchronized jobs, and schedulers such as Kubernetes or Borg stagger their starts.
2 · What the grid and the datacenter can tolerate
Supply and demand on the grid must balance at every instant. Controllable generators (gas, coal, nuclear, hydro) adjust output to follow load, but they are heavy rotating machines: the paper gives 10 to 20 MW per minute as the ramp rate of even fast gas units, and minutes to days to bring new generators online. If load changes faster, the mismatch shows up as voltage and frequency moving out of range (US residential service must stay within 114 to 126 V RMS and 59.9 to 60.1 Hz). Protection equipment then disconnects parts of the network, and at datacenter scale the sudden loss of load is itself a disturbance.
- Texas. ERCOT (the Texas grid operator) has identified that simultaneous disconnection of 2.0 to 2.6 GW of datacenter load could destabilize the grid; in 2023 the trip of a 1.5 GW load caused system-wide frequency deviations.
- Northern Virginia. In July 2024 a transmission fault made 1.5 GW of datacenters disconnect at once, requiring emergency grid management.
- Oscillation. NERC's analysis of a 2019 event found a load oscillating at 0.25 Hz propagated across interconnections and damaged generators hundreds of miles away.
- Siting. Some new datacenter projects have been denied over the instability training can cause.
Inside the datacenter, power arrives at 13.8 kV from the utility, is stepped down by an on-site substation and switchgear, passes through uninterruptible power supplies (UPSes) and row-level PDUs, and reaches racks. Racks have historically used 48 V DC at about 30 kW; AI racks can exceed 100 kW, roadmaps include 1 MW racks, and datacenters are moving to 400 V DC. EasyRider is designed for that 400 V DC regime and sits at the rack end of this chain.
3 · Grid limits as a filter specification
The paper views the power trace P(t) the grid sees as a sum of sinusoids, using a Discrete Fourier Transform (DFT). S(f) is the normalized magnitude at frequency f, readable as the fraction of the power at that rate. In the test trace the largest dips come about every 22 s, so S(1/22 Hz) ≈ 0.1. Grid operators impose two kinds of limits:
| Limit | Form | Meaning | Value assumed in the evaluation |
|---|---|---|---|
| Frequency content | S(f) ≤ α for all f ≥ fc | Above a cutoff, only a small fraction of power may oscillate | α = 10−4, fc = 2 Hz |
| Ramp rate | |dP/dt| ≤ β for all t | Power may change by at most a fraction β of rated power per second | β = 0.1 (10% per second) |
The actual values are set by local grid operators; the paper chose these as in line with prior work and with NERC's 0.1 to 10 Hz band of concern. Two consequences follow. First, the job is a low-pass filter: remove fast content, pass slow behavior. Second, if each rack's power delivery meets per-rack limits that sum to the campus budget, the datacenter meets the campus limits in aggregate, so no per-job coordination is needed.
4 · Where existing approaches sit
The paper's Table 1 compares mitigations by where they are inserted and whether the fast path depends on software:
| Approach | Placement | High frequency | Low frequency | Software/firmware dependence | Main limitation |
|---|---|---|---|---|---|
| GPU burn | GPU | none | work injection | training stack | energy waste; no hardware protection |
| GB300 support | power shelf | capacitors | power cap / burn | platform firmware | platform-specific |
| Software-controlled batteries | rack | battery, software-triggered | battery + cap + burn | telemetry + software | fast path limited by telemetry |
| Site BESS | substation | none | site battery | site controller | does not protect internal DC distribution |
| EasyRider | rack PDU | passive LC | local battery | none for transient mitigation | rack-local only |
Two details from the text: GB300's capacitors cover short (≤ 60 ms) transients, and lithium-ion batteries cannot respond to transients faster than tens to hundreds of milliseconds without accelerated degradation. Scheduling methods that fill pipeline bubbles reduce some swings and are complementary, but give no electrical guarantee.
5 · The three hardware stages
- Input filter. A second-order passive filter (inductor LF, capacitor CF) between the DC busbar and the rest of the unit, with a resistive damping leg that suppresses the LC pair's resonance during transients and is inactive at steady power. Above its cutoff ff, it attenuates fluctuations by up to 100× per 10× in frequency; the implementation uses ff ≈ 4 Hz. Inductors and capacitors store little energy, so this stage handles only fast content, including switching noise from the converters.
- DC-DC converter. Holds the rack's output voltage constant. Its control is entirely in hardware, corrects errors in under a millisecond, and keeps the rack voltage within 0.7% of rated even at rack ramp rates of ±200 kW per second.
- Auxiliary energy storage. A battery bank behind a bidirectional converter, attached on the rack side. High-bandwidth sensors detect rack power changes and drive current into the battery when rack power drops, and out of it when rack power rises, so the grid-side current stays smooth over longer timescales. The prototype uses high-power LiFePO4 cells for their high power-to-capacity ratio and low cost per unit energy; supercapacitors or a mix would also work. It needs minutes of capacity, not the hours of a grid battery.
The two filters multiply. The battery path acts as a first-order filter with cutoff fb = β/2π (attenuating 10× per decade above it), and the LC filter adds 100× per decade above ff:
6 · How the battery bounds the ramp rate
The battery current iB is controlled so that it obeys diB/dt + β·iB + diR/dt = 0, where iR is the rack current (the paper's Appendix A.1, equation 2). In words: any sudden change in rack current is immediately taken up by the battery, and the battery's share then decays toward zero with time constant 1/β. The paper shows this keeps the grid-side ramp at or below β times rated power even if the rack turns off entirely.
The same law sizes the parts (Appendix A.1). With ε the largest swing as a fraction of rated power, the net energy the battery must hold is at most ε·Prated/β, and if only a fraction γ of the battery's capacity may be used (for example a 40 to 60% state-of-charge window), capacity must be at least ε·Prated/(γβ). The battery must also charge or discharge at ε·Prated. For the 10 kW prototype with ε = 0.8, β = 0.1 and γ = 0.2, that is 400 kJ, about 111 Wh, at 8 kW (this summary's arithmetic). The LC filter's cutoff follows from the standard ff = 1/(2π√(LC)).
7 · The software controller
Charging and discharging each lose a little energy, so over hours of training the battery's state of charge (SoC) drifts, upward or downward depending on which losses dominate. Left alone it would hit a safe bound and lose the headroom it needs for the next transient, and sitting at high SoC also speeds aging. A controller with two loops counteracts this:
- Outer loop (every few minutes, and on regime changes). Picks the SoC target S*. During training it is a mid-band value Smid that leaves equal room to charge and discharge (0.5 in the experiment). During long idle periods (in the prototype, gaps over 4 hours) it lowers the target to reduce idle-time aging, but only as far as it can still recharge to Smid at maximum rate before the idle window ends.
- Inner loop (every 5 s). Reads SoC from the battery management system and solves a small convex quadratic program over a receding horizon. The objective trades off distance from S*, corrective-current size, and smoothness of the command. Only the first step is applied, and inside a small deadband around S* the current is set to zero. The solve takes under 10 ms on a Raspberry Pi 5.
- Isolation from the fast path. The corrective current is orders of magnitude smaller than the rack's transient current, so a wrong command cannot disturb filtering. If the software crashes or loses connectivity, smoothing continues and only SoC drift resumes, which is corrected on restart with no cold-start penalty.
- Configuration. All parameters come once from the battery datasheet and the desired correction timescale, with no per-workload tuning.
8 · Prototype and evaluation setup
The prototype (the paper's Figure 8) is rated to deliver 10 kW to a 400 V DC load, with a 74 Ah battery bank and a maximum discharge rate of 2.4C. Its thermal design limits load current to 25 A, so it cannot deliver the full 10 kW at lower voltages. It cost about $3,500, or $0.35 per watt; the battery pack is intentionally oversized relative to the Appendix A.1 requirement, and the authors expect pooled batteries and integrated power stages to cost less in production. Workloads:
- Published trace. Cluster-scale traces of frontier training jobs are not public, so the prototype drives a DC load with an existing normalized training trace from Choukse et al.
- Own testbed. A GPT-style 125M-parameter language model trained on a decommissioned 2-GPU NVIDIA Titan X server, used for the comparison with software burn.
- Software burn baseline. GEMM (matrix multiply) kernels calibrated to map duty cycle to GPU power, used to ramp power up at the start, ramp it down at the end, and fill the power dip on other GPUs while rank 0 writes a checkpoint.
9 · Results
| Question | Setup | Result |
|---|---|---|
| Ramp rate (Figure 9) | prototype, normalized trace | grid-side ramp stays within ±10% of rated power per second, through each communication swing and the abrupt drop at job end; the conditioned trace also has a lower peak |
| Frequency content (Figure 10) | same run, spectrum | the whole EasyRider spectrum above fc = 2 Hz is below α = 10−4, while the raw rack trace has significant content there |
| Energy vs. software burn (Figure 11) | 2-GPU Titan X | software burn consumed 19% more total energy than rack plus EasyRider, and needed a warm-up phase (the raw trace was delayed by about 41 s to account for it) |
| Battery SoC control (Figure 12) | prototype, inner loop only | from about 62% SoC after hours without control, the controller reached Smid = 0.5 in about 20 minutes, monotonically; without control SoC kept drifting toward the upper bound |
| Cluster scale (Figure 13, Appendix D) | simulation, 40 MW cluster, H100 trace scaled | the raw trace falls at 193.7 MW/s during a computation fault near 400 s; with EasyRider on every rack the grid-side power is smoothed for β = 12.5%, 5% and 1% |
The cluster result rests on an argument, not a measurement: in synchronous training all racks draw nearly the same trace, so cluster power and its spectrum are the per-rack ones scaled by the number of racks, and normalized per-rack results carry over. The paper notes the fault in Figure 13 would be hard to predict for a scheduled burn, while EasyRider smooths it because it does not rely on telemetry.
10 · Limitations and scope the paper states
- Prototype scale. 10 kW, a proof of concept; it does not by itself handle grid-scale power. Cluster-scale behavior is shown by simulation and the scaling argument.
- Traces. Frontier-scale traces are not public; the evaluation uses one normalized published trace and a 2-GPU testbed.
- What is validated. The SoC experiment validates the inner loop in isolation. The outer loop's idle-time storage mode uses the same mechanism with a different target and was not separately validated.
- Operating envelope. Guarantees hold for any power trace within the rack's maximum provisioned power, and with the battery within a reasonable SoC. Multi-hour software downtime may age the battery slightly faster.
- Physical placement. The filter and converter must sit in or right next to the rack to keep the characterized electrical behavior, which brings mechanical and thermal design work but no substation changes.
- Rack-local. Table 1 lists "rack-local only" as EasyRider's main limitation; it conditions each rack's load and is complementary to cluster-level power management.
FAQ
Q·What does the name mean?▾
The paper says EasyRider lets a rack "ride through" a training transient without circuits or systems at the datacenter or grid scale. The rack sees the power it needs immediately; the grid sees a slow change.
Q·Why can't the grid simply follow the load?▾
Because generators are large rotating machines whose output is limited by how fast they can change fuel input and accelerate or slow their turbines. The paper gives 10 to 20 MW per minute even for fast gas units, while a large training job can swing by tens of megawatts within milliseconds. When load changes faster than supply, voltage and frequency move outside their safe ranges and protection equipment disconnects loads.
Q·How is this different from GPU burn or power capping?▾
Burn fills power dips by running extra work, such as matrix multiplies, on the GPUs. It spends energy the whole time it runs, ties protection to the training stack, and exposes the transient if detection or control fails. EasyRider leaves the job alone and reshapes its power draw electrically. On the paper's 2-GPU testbed, the burn baseline consumed 19% more total energy and needed a warm-up phase. The paper also notes that an unpredicted event, like the computation fault in its 40 MW simulation, is hard to cover with scheduled burn.
Q·Why not one large battery at the substation?▾
A site battery energy storage system (BESS) smooths what the grid sees at the interconnection point, but it sits above the datacenter's internal row and rack distribution. Rack transients still travel through that internal equipment before they are absorbed. EasyRider absorbs them at the rack PDU.
Q·Why both a passive filter and a battery?▾
Each covers what the other cannot. Inductors and capacitors respond instantly but store little energy, so a passive filter large enough to smooth transients lasting more than a few milliseconds would be prohibitively large. Lithium-ion batteries store plenty of energy but, per the paper, cannot respond to transients faster than tens to hundreds of milliseconds without accelerated degradation. So the LC stage takes the fast content and the battery takes the slow swings, and their responses multiply.
Q·What happens if the software crashes?▾
Smoothing continues. The fast path (LC filter, DC-DC regulator and battery current control) is entirely hardware. The software only nudges the battery's state of charge with a current orders of magnitude smaller than the rack's transients. If it stops, charge slowly drifts; when it restarts it re-optimizes from the current reading with no cold-start penalty. Multi-hour outages may age the battery slightly faster.
Q·Does it slow training or require code changes?▾
No. EasyRider requires no changes to software, models, schedulers or GPU firmware, and the rack always sees a regulated voltage. The guarantee holds for any power trace within the rack's maximum provisioned power, so operators can change workloads freely within that envelope.
Q·And when rack power suddenly rises?▾
The same mechanism runs in reverse: the battery discharges to supply the extra power at once, and the grid-side draw rises gradually, bounded by the same ramp limit, until grid supply catches up.
Q·Does a per-rack limit really give a datacenter-wide limit?▾
The paper argues yes. Datacenter power is the sum of rack powers, so if every rack's grid-side ramp is at most β of its rating, the aggregate ramp is bounded the same way, and per-rack spectral limits sum likewise. In synchronous training the racks draw nearly identical traces, so normalized per-rack results scale directly (Appendix D). This is what allows rack-by-rack, incremental deployment across rows and halls without substation or medium-voltage retrofits.
Q·How big does the battery need to be?▾
Minutes of capacity, not hours. Appendix A.1 bounds the net energy per transient at ε·Prated/β, where ε is the largest swing as a fraction of rated power; dividing by the usable fraction γ of the battery gives the capacity. For a 1 MW rack with an 80% swing, β = 0.1 and a 40 to 60% usable window, that is 40 MJ, about 11 kWh (this summary's arithmetic). The paper estimates the prototype's cost at $0.35 per watt with an intentionally oversized battery.
Q·Does EasyRider waste energy?▾
Some: each charge and discharge cycle loses roughly 1 − ηcηd of the energy exchanged, plus losses in the power electronics. The paper describes these as a small additional energy sink over weeks of training, in contrast to burn, which spends energy throughout the job.
Q·What does the evaluation not show?▾
It does not show a deployment at datacenter scale. The hardware is a 10 kW prototype driven by one normalized published trace and a 2-GPU testbed; the 40 MW cluster result is a simulation built on the per-rack scaling argument. The battery controller's idle-time storage mode was not separately validated, and the evaluation does not report long-term battery aging.