Rethinking Compression for CXL Memory Expanders at Hyperscale
This page was written by an AI model to give a high-level overview of the paper. Please check the paper for detail: PDF.
DRAM accounts for up to 50% of a datacenter server's total cost of ownership, and AI-driven supply shortages are pushing its price higher. One response is memory tiering: put infrequently accessed (cold) pages in a slower, cheaper tier. The Open Compute Project (OCP) Hyperscale CXL Tiered Memory Expander, proposed by Google and Meta, builds that tier from legacy DRAM taken out of retired servers and attached through a Compute Express Link (CXL) card. The expander compresses cold data in hardware to hold more of it. That has two costs: decompression adds latency, and because data is compressed in blocks (for example 4 KiB), serving one 64-byte host request means fetching and decompressing a whole block (read amplification). An on-device SRAM block cache holds recently decompressed blocks to soften both.
Two design choices decide how much capacity the expander gains and how much latency it adds: the compression algorithm and the block size. Today's answers are inherited from software. The OCP specification sets LZ4, a fast and simple algorithm, as the baseline, and early commercial expanders implement only simple compression. That choice made sense for software far memory such as zswap, where decompression runs in software behind a page fault and the fastest algorithm is the only viable one. The specification and commercial devices propose 4 KiB and 1 KiB blocks but give no guidance on how block size affects performance. The paper also finds that the benchmarks commonly used to evaluate tiered memory behave very differently from production cold memory, so they cannot settle the question.
The paper's central observation is that in a hardware expander, decompression time and DRAM fetch time are of similar scale, and the expander's DRAM bandwidth is often the bottleneck. A more complex algorithm such as Zstd takes longer to decompress a block, but it compresses better, so each cache miss reads fewer bytes from DRAM and the memory controller queues fewer requests. The paper captures this with bandwidth inflation, the ratio of bytes read from the expander's DRAM to bytes the host actually requested. When inflation is high, memory bandwidth efficiency matters more than decompression speed.
The evaluation uses ten cold-memory access traces from a production hyperscaler fleet, each from a server running hundreds of co-located workloads. Each block in a trace is assigned a compressed size and decompression latency drawn from HyperCompressBench, a corpus whose compressibility matches production far memory, and the traces drive a gem5 model of a commercially representative expander (four DDR4 channels and a 16 MB block cache). Because fleet traces are scarce and cannot be shared, the paper also builds COLA (COld memory LocAlity), a model that reduces a trace's locality to a small set of interpretable parameters, plus a generator that synthesizes new traces from any point in that parameter space.
On the fleet traces, averaged across all ten, Zstd with 2 KiB blocks gives 30.0% more effective capacity and 12.1% lower average access latency than LZ4 with 4 KiB blocks; Zstd with 4 KiB blocks gives 35.8% more capacity and 11.5% lower latency. LZ4 with 512 B blocks has the lowest latency (33.8% lower) but 24.4% less capacity. The widely used benchmarks all favored LZ4 on latency, the opposite conclusion. Across 2,304 COLA-generated traces, Zstd beats the LZ4/4 KiB baseline on both capacity and latency for 75% of them; LZ4 keeps a latency edge only when locality is very high.
Key takeaways
- CXL memory expanders compress cold data in blocks to gain capacity; the costs are decompression latency and read amplification, partly hidden by an SRAM block cache.
- Expanders today default to LZ4, a choice inherited from software far memory, where decompression sits behind a page fault. In hardware, decompression and DRAM fetch take similar time, and DRAM bandwidth is often the bottleneck.
- On ten production cold-memory traces, Zstd with 2 KiB blocks gives 30.0% more capacity and 12.1% lower average latency than LZ4 with 4 KiB blocks, because it reads fewer bytes per miss and queues less.
- Common tiered-memory benchmarks access cold memory far more often and with different locality than the fleet, keep block-cache hit rates above 85%, and all favor LZ4, so they mislead this design decision.
- COLA models cold-memory locality with a few interpretable parameters and generates synthetic traces; across 2,304 of them, Zstd beats the LZ4/4 KiB baseline on both capacity and latency for 75%.
Why stronger compression lowers latency in a bandwidth-bound expander
The expander's read path, what block size and algorithm trade, the fleet-trace evaluation, why benchmarks disagree, and the COLA locality model.
1 · The expander and its read path
The OCP Hyperscale Tiered Memory Expander is a CXL device holding DRAM, a compression engine, a translation table and an SRAM block cache. The host reaches it with ordinary loads and stores over CXL, with no page faults, and the expander stores only cold pages: pages the operating system has seen inactive for a while or predicts will be accessed rarely. Writes can complete asynchronously, so the paper treats read latency as the key performance metric. Its Figure 1 traces the worst-case read.
Step 4 is where read amplification comes from: one 64 B request reads and decompresses an entire block. For a 4 KiB block that needs only one line, the amplification can approach 64×. A burst of misses can then saturate the expander's DRAM bandwidth and add queuing delay.
2 · What block size and algorithm trade
Section III measures raw compression on HyperCompressBench for seven block sizes (64 B to 4 KiB, in powers of two; 4 KiB is the largest unit the OS guarantees to be physically contiguous) and four algorithms. Decompression latency is measured with FPGA-based Vitis compression kernels at 300 MHz and scaled to ASIC speed by comparison with Intel's IAA and QAT hardware decompressors.
| Algorithm | Design goal | What the measurements show |
|---|---|---|
| LZ4, Snappy | Low latency | Similar to each other in size and speed; the paper treats them as interchangeable |
| Zstd | Higher compression ratio | Compressed size 13 to 37% smaller than LZ4 and Snappy (together with Deflate); about 2.5× LZ4's decompression latency; a fixed minimum of 370 ns, more than ten times the others for small blocks |
| Deflate | Higher compression ratio | Fast for tiny blocks; for large blocks, more than 2.5× the latency of LZ4 and Snappy; its latency rises with compressed size 27× more steeply than LZ4's |
- Bigger blocks compress better, with diminishing returns. All algorithms compress larger blocks more, but gains flatten beyond 1 KiB, and tiny blocks save little space.
- Bigger blocks are slower to serve on a miss. Decompression emits cache lines in order, so a line deep in a large block waits longer, and a larger compressed block takes longer to fetch.
- Latency is linear in compressed size. The paper fits 1.1 ns per percentage point of compressed size for LZ4 (R² 0.83) and 4.3 ns per point for Zstd (R² 0.87), at 4 KiB.
- Incompressible blocks skip decompression. If a block compresses to more than 80% of its size, the expander stores it uncompressed. Smaller blocks and simpler algorithms produce more such blocks: for LZ4 the rate rises from 4.8% at 4 KiB to 18.5% at 512 B, which cuts decompression latency by 18%.
- Chunks waste space. Compressed blocks are stored in fixed-size chunks, so each block wastes part of its last chunk. Smaller blocks waste more. In the paper's example for 512 B LZ4 blocks, 27% of blocks compress to just over 128 B and must occupy 192 B with 64 B chunks or 256 B with 128 B chunks.
3 · Evaluation setup
| Part | What the paper uses |
|---|---|
| Access traces | Ten cold-memory traces (Trace-0 to Trace-9) from a far-memory system deployed in a hyperscaler fleet, each from a randomly selected server running a very different mix of hundreds of workloads. They record physical address and timestamp of accesses to cold pages; CPU cache and host DRAM accesses are excluded. |
| Compressibility | Each block gets a (compressed size, decompression latency) pair sampled from the HyperCompressBench results. Collecting real compressibility alongside the traces would add overhead and privacy risk on production servers. |
| Expander model | gem5, configured to match the Marvell Structera device and the OCP specification: four 8 GB DDR4-2600 DIMMs, one per channel; a 16 MB 8-way set-associative LRU block cache with 30 ns access; enough decompression engines to saturate DRAM bandwidth. |
| Validation | Idle latency within 0.00% (cache hit), 2.59% (incompressible block) and −3.59% (compressed block) of the real device. |
| Configurations | LZ4 and Zstd, block sizes 64 B to 4 KiB. Baseline: LZ4 with 4 KiB blocks. |
4 · The main result
Averaged over all ten traces, relative to LZ4 with 4 KiB blocks (the paper's Table I):
| Block size | LZ4 latency reduction | LZ4 capacity change | Zstd latency reduction | Zstd capacity change |
|---|---|---|---|---|
| 4 KiB | 0% (baseline) | 0% | 11.5% | +35.8% |
| 2 KiB | 16.1% | −4.73% | 12.1% | +30.0% |
| 1 KiB | 27.3% | −10.7% | 5.26% | +21.8% |
| 512 B | 33.8% | −24.4% | −31.0% | −2.27% |
| 256 B | 33.4% | −36.8% | −51.3% | −28.0% |
| 128 B | 32.5% | −53.4% | −82.3% | −47.8% |
| 64 B | 30.5% | −55.5% | −55.6% | −55.5% |
LZ4 trades capacity for latency as blocks shrink. Zstd at 2 KiB and 4 KiB improves both. Tails follow the same pattern (Table II, P95/P99 in µs): LZ4 at 4 KiB is 1.42/1.81, Zstd at 4 KiB 1.22/1.46 and Zstd at 2 KiB 1.04/1.11, so LZ4/4 KiB has 24% and 63% higher P99 latency than Zstd at 4 KiB and 2 KiB. For small blocks the order flips: at 512 B, LZ4 is 0.30/0.55 and Zstd 0.79/0.94.
5 · Why the slower decompressor wins
Section IV-C splits access latency into hit latency, miss latency (decompression, memory fetch and queuing) and hit rate. Between 1 KiB and 4 KiB, hit rate and hit latency are about the same for both algorithms, so the difference is in misses, and specifically in queuing. To reason about queuing the paper defines bandwidth inflation: total bytes read from the expander's DRAM divided by total bytes the host requested. Unlike read amplification, it credits the extra hits served from one decompressed block.
- LZ4 at 4 KiB. On average a 4 KiB block compresses to 44.5% of its size, so each miss reads 44.5% × 4 KiB / 64 B ≈ 28 lines' worth of data. With a 71% block-cache hit rate, about 3.4 lines are served per decompressed block, and bandwidth inflation is 28 × (1 − 0.71) = 8.12.
- Zstd at 4 KiB. The same block compresses further, and bandwidth inflation is 6.0. Fewer bytes per miss means fewer requests queued at the DRAM controller, which more than offsets Zstd's longer decompression.
- Larger LZ4 blocks get slower overall. Hit rate rises with block size, but miss latency rises faster (longer decompression and fetch), and bandwidth inflation also rises. Hit latency grows too, from delayed hits: a request for a block that is still being decompressed counts as a hit but waits for the rest of the decompression.
- Small Zstd blocks turn compute-bound. Halving the block from 1 KiB raises the number of blocks to decompress by 16% (the miss rate rises) while each block's decompression time falls only 4%, so the decompression engines, not DRAM, become the queue. At 128 B to 64 B every block is incompressible, which removes that load again.
- Matching throughput is not enough. The OCP guideline is to match average decompression throughput to memory bandwidth. Because decompression time varies with content and requests arrive out of order, queues still form; doubling the number of decompression engines lowers Zstd's hit and miss latency for small blocks (Figure 6).
6 · Sensitivity checks
- Block cache size. Hit rate on the fleet traces plateaus beyond about 10 MiB. A 4× larger cache raises hit rate by 4.4% on average, and Zstd at 4 KiB and 2 KiB still has 5% and 7% lower access latency, so its advantage holds.
- Compressibility. With compressed-size distributions of smaller or larger spread, and with biased pairings in which frequently missed blocks compress more or less, Zstd at 2 KiB stays 8.5 to 19% faster than LZ4 at 4 KiB. The paper explains why: latency rises gently with compressed size for both algorithms, and over 95% of blocks are accessed fewer than 10 times, leaving few high-miss blocks for a bias to act on.
- DRAM bandwidth. More DRAM bandwidth would shrink Zstd's advantage. But four DDR4 channels already match the most found in commercial CXL expanders, and removing DRAM queuing at the observed 5 to 10× bandwidth inflation would take roughly an order of magnitude more channels, which die area and cost rule out. The paper argues the bandwidth-bound regime is inherent to a cost-driven cold tier.
7 · Why benchmarks reach the opposite conclusion
The paper runs six tiered-memory benchmarks (TPC-C on Silo, YCSB on Redis, PageRank in GAPBS, Liblinear, and SPEC CPU2017's 603.bwaves and 654.roms) plus a mix of all of them, with CXL memory as the cold tier, AutoNUMA-tiering deciding which pages go there, and CXL-Tracer recording the accesses that reach it.
- Far more traffic. Benchmarks put 15× more total accesses and 4.5× more accesses per page on cold memory than the fleet traces. Fleet traces are dominated by pages touched a few times, plus a few very hot outliers; benchmarks show a gradual distribution without such outliers.
- Different locality. Projected into a two-dimensional locality space (Figure 9), benchmarks and fleet traces form separate clusters. Benchmarks have higher temporal locality and denser, scan-like access within pages; fleet traces are sparser, consistent with pointer chasing. Running benchmarks together averages their locality but does not reach the fleet cluster.
- Different design conclusion. Block-cache hit rates on benchmarks consistently exceed 85% and LZ4's bandwidth inflation never exceeds 5, so bandwidth pressure stays low and LZ4's faster decompression dominates. Every benchmark favored LZ4 over Zstd on latency.
8 · COLA: describing cold-memory locality with a few parameters
Fleet traces are scarce, tied to one configuration and workload mix, and cannot be shared. COLA reduces a trace to a small parameter set with physical meaning, so architects can compare traces in one space and ask "what if" questions by moving parameters. The paper argues reuse-distance models, the usual tool, miss what matters here: two traces with the same reuse-distance distribution can have different per-block access counts (so different bandwidth inflation) and different miss orderings (so different queuing). COLA keeps four locality profiles:
- Reuse-distance clustering. The sequence of reuse distances is a first-order Markov chain over 12 bins: bin 0 is reuse distance 0, bin 11 is a first access to a block, and bins 1 to 10 cover two logarithmic steps per decade (bin 4 is 32 to 99). The transition matrix captures bursts and loops. Its density sits in the bin-0 column, the bin-11 column and the diagonal, so COLA fits it with seven 2D Gaussians (one per structure plus four fixed corners): 24 parameters.
- Block access frequency. Real traces are skewed: most blocks are touched a few times and a handful thousands of times. COLA records the number of pages and total accesses in logarithmically scaled count intervals and reconstructs the full distribution with an integer linear program.
- Multi-granular locality. The Markov chain and frequencies are kept only at 4 KiB. For smaller blocks, a bit flip matrix gives, for each reuse-distance bin and address bit, the probability that the bit flips between two accesses to the same page (bit 11 decides which 2 KiB half). Three 2D Gaussians plus four shared sigmas describe it in 16 numbers.
- Inter-access time. Arrivals come from two Poisson distributions, one inside bursts and one between them, and a quadratic function maps each reuse-distance bin to the probability that an access belongs to a burst.
9 · Does COLA reproduce the traces, and do the results generalize?
Synthesized traces are compared with the originals on locality and on expander behavior:
| Check | Result for reduced-parameter synthesis |
|---|---|
| Reuse-distance CDFs (Earth Mover's Distance) | 0.008 to 0.07 per trace, 0.03 on average |
| Footprint (distinct sub-blocks touched per page) | 4% average error, 15% maximum |
| Block-cache hit rate | 3 percentage points average absolute error |
| Access latency | within 6.1 to 21% (9% on average); same best block size and algorithm as the originals |
The paper attributes the larger latency errors to traces that are more or less bursty than its timestamp method captures. It then sweeps five parameters to generate 2,304 traces: working-set size (0.5× to 4× the fleet mean), temporal locality (eight levels, M0 to M7, spanning 10% to 90% hit rate), spatial locality (B0 to B7, average bit flip rate 0.2 to 0.8), access-count distribution, and burstiness. The fleet traces sit near the middle levels. Each trace runs with LZ4 and Zstd at 1 KiB, 2 KiB and 4 KiB.
- Zstd beats the baseline on both metrics for 75% of traces. LZ4 keeps lower latency only where locality is high and accesses rarely queue. At M7/B7, bandwidth inflation falls to 1.83× and hit rate reaches 95%; even there Zstd gives 35% more effective capacity.
- Locality points to the configuration. Higher spatial locality favors larger blocks. Higher temporal locality favors larger blocks and faster algorithms. Access counts and burstiness change these trends less.
- A rule of thumb. Below M5, LZ4's bandwidth inflation at 4 KiB exceeds roughly 10, and Zstd wins on latency for most traces there. The paper suggests bandwidth inflation as a lightweight indicator: above about 10, the expander is bandwidth-bound and Zstd's better compression pays off.
10 · Scope and limits the paper states
- Compressibility is synthetic. Block sizes and latencies come from HyperCompressBench, not production data, for privacy and overhead reasons. The paper reports that results depend on its average compressibility, which matches production zswap far memory, more than on the detailed distribution.
- Ten traces are a small sample. They represent a small subset of a fleet with millions of servers and thousands of workload types, which is why the paper builds COLA.
- Simulated device. Results come from a gem5 model validated on idle latency against a commercial device, configured for near-term deployment: repurposed DDR4 and minimal hardware changes to the OCP design.
- DRAM bandwidth. A configuration with much more expander bandwidth would reduce Zstd's benefit; the paper argues such configurations are impractical for a cost-driven cold tier.
- Artifact. The released artifact regenerates fleet-like traces from the released COLA parameters, so it reproduces the paper's trends and conclusions rather than its exact numbers.
FAQ
Q·How is a compressed CXL expander different from zswap or other software far memory?▾
In software far memory such as Google's zswap-based system, a compressed page is reached through a page fault and decompressed by software, so latency is already large and the fastest algorithm is the only viable choice. The expander decompresses in hardware behind a direct load/store interface, with no page fault, and caches decompressed blocks on the device. Its latency is an order of magnitude lower, and decompression time becomes comparable to DRAM fetch time. That is what changes the algorithm tradeoff.
Q·How can a slower decompressor give lower average latency?▾
Each block-cache miss reads the whole compressed block from the expander's DRAM. Zstd's blocks are smaller, so each miss reads fewer bytes and fewer requests wait at the DRAM controller. On the fleet traces, bandwidth inflation at 4 KiB is 8.12 for LZ4 and 6.0 for Zstd. The queuing saved is larger than the extra decompression time, so average miss latency drops. When locality is high and misses are rare, queuing disappears and LZ4's faster decompression wins instead.
Q·Is this a new expander design?▾
No. The paper studies configurations that the OCP architecture already supports or can adopt with minimal hardware changes: which algorithm the compression engine runs and what block size it uses. The OCP specification already provisions both LZ4 and Zstd. The contributions are the evaluation, the benchmark comparison and the COLA model and generator.
Q·Why not just use LZ4 with small blocks, which had the lowest latency?▾
LZ4 with 512 B blocks lowers average latency by 33.8% but loses 24.4% of effective capacity relative to LZ4 with 4 KiB blocks, and capacity is the reason the expander compresses at all. Small blocks compress worse and waste more space in fixed-size chunks. The paper presents LZ4/512 B as the peak-latency point and Zstd/2 KiB as the configuration that improves both metrics.
Q·What about tail latency?▾
Tails favor Zstd at large blocks even more. P99 latency is 1.81 µs for LZ4 at 4 KiB, against 1.46 µs for Zstd at 4 KiB and 1.11 µs for Zstd at 2 KiB, so LZ4/4 KiB is 24% and 63% higher. Tail latency grows faster with block size for LZ4 than for Zstd. At 512 B and below, LZ4 has the lower tails.
Q·Were real compressed production pages used?▾
No. Measuring compression on live production data was judged an unacceptable privacy risk, and recording compressibility alongside access traces would add overhead on production servers. Each block in a fleet trace is instead assigned a compressed size and decompression latency sampled from HyperCompressBench, which datacenter operators recommended because its average compressibility matches production zswap far memory. The sensitivity study finds the result depends mainly on that average.
Q·Is the evaluation on real hardware?▾
The expander is simulated in gem5, configured to match the Marvell Structera device and the OCP specification, and validated against the real device's idle latency (within 0.00%, 2.59% and −3.59% for a cache hit, an incompressible block and a compressed block). Decompression latencies come from FPGA compression kernels scaled to ASIC speed. The access traces themselves are from production servers.
Q·Why do benchmarks such as YCSB on Redis or TPC-C give a different answer?▾
They load the cold tier much harder and with different locality. Compared with the fleet traces they have 15× more total cold-memory accesses and 4.5× more per page, higher temporal locality and denser access within pages. Their block-cache hit rates exceed 85% and LZ4's bandwidth inflation never exceeds 5, so bandwidth never becomes the bottleneck and LZ4's faster decompression wins. A fleet trace aggregates hundreds of co-running workloads, which no single benchmark, or mix of benchmarks, reproduces.
Q·Why not model locality with reuse distance, as cache studies do?▾
A reuse-distance distribution predicts LRU cache miss rates, but it does not say which block each reuse belongs to or in what order misses arrive. Two traces with the same distribution can differ in per-block access counts, which changes bandwidth inflation, and in how misses cluster in time, which changes queuing. COLA adds those profiles: a Markov chain over reuse-distance bins for ordering and bursts, a per-block access-frequency distribution, a bit flip matrix for sub-page locality, and a two-rate Poisson timing model.
Q·Can others reproduce the results without Google's traces?▾
Partly. COLA is open source, and the artifact includes reduced COLA parameters derived from the ten fleet traces, synthetic fleet-like traces generated from them, benchmark cold-memory traces, and the gem5 simulator. Raw fleet traces are not included and cannot be recovered from the parameters. The paper states that experiments on fleet-like traces reproduce its methodology, trends and conclusions, not its exact numbers.
Q·How much does a Zstd engine cost in hardware?▾
Citing prior work, the paper puts a Zstd engine at 36% more area and 76% more power than an LZ4 engine with the same throughput. It argues this is negligible against the memory Zstd saves and under 1% of an expander controller, whose area is dominated by the SRAM block cache and the CXL and DDR physical interfaces.
Q·When should an architect still choose LZ4?▾
In the COLA sweep, LZ4 has lower latency only when temporal and spatial locality are both high, so misses are infrequent and queuing rare. The paper offers bandwidth inflation as a practical indicator: when LZ4's inflation at 4 KiB is above about 10, the expander is bandwidth-bound and Zstd pays off. Even in the region most favorable to LZ4, Zstd gives 35% more effective capacity.