Capacity-Latency Tradeoffs in CXL Memory Expander at Hyperscale
This page was written by an AI model to give a high-level overview of the paper. Please check the paper for detail: IEEE Xplore.
DRAM now dominates the total cost of ownership (TCO) of hyperscale datacenter servers, and its price is rising further as manufacturing capacity shifts toward high-bandwidth memory (HBM) for AI datacenters. One way to cut that cost is to reuse older, slower DDR modules from retired servers by attaching them through a Compute Express Link (CXL) memory expander, such as the Open Compute Project (OCP) device proposed by Google and Meta. The OCP expander compresses data in hardware to raise its effective capacity. Because the CXL interface and decompression add latency, servers use it to hold cold pages, which are accessed infrequently.
When the host moves a page to the expander, the device splits it into blocks, compresses each block, and stores it in its DRAM. When the host later reads one cache line, the expander fetches the whole compressed block, decompresses it into an SRAM block cache, and returns the line. Two design choices trade capacity against latency: the compression algorithm (more complex algorithms compress better but decompress more slowly) and the block size (larger blocks compress better but take longer to decompress). Both also change read amplification, the amount read from DRAM to serve one cache line, which can saturate the expander's memory bandwidth. Commercial expanders use LZ4, a very simple algorithm, with 4 KiB blocks, choices inherited from software memory compression such as zswap. Expanders differ from software systems in three ways: no software page faults, compression in hardware, and much lower latencies. Whether they should make the same choices is the question this letter studies.
The key finding is counter-intuitive: a slower, more complex algorithm can lower access latency. Zstd takes longer per block than LZ4, but it compresses better, so each block cache miss reads fewer bytes from DRAM and the memory controller queues fewer requests. In an expander, memory bandwidth and queueing can dominate end-to-end latency, so the reduction in DRAM traffic more than offsets the extra decompression time. The paper captures this with bandwidth inflation: bytes the expander reads from DRAM to load a block, divided by the bytes of that block the host requests.
The study has two parts. First, it measures compressed size and decompression latency for four algorithms (LZ4, Snappy, Zstd, Deflate) and seven block sizes (64 B to 4 KiB) on HyperCompressBench, a benchmark built to reflect the compressibility seen in hyperscaler fleets, with latencies from FPGA designs scaled to ASIC speed. Second, it drives a gem5-based simulation of a commercial expander (four DDR4 channels, a 16 MB block cache) with ten cold-memory access traces from servers in Google's fleet, each server running hundreds of concurrent workloads.
Averaged over the ten traces and compared with LZ4 at 4 KiB, Zstd with 4 KiB blocks gives 35.8% more effective capacity and 11.5% lower average access latency, and Zstd with 2 KiB blocks gives 30.0% more capacity and 12.1% lower latency. LZ4 with 512 B blocks has the lowest latency (33.8% lower) but 24.4% less capacity. The paper also finds that for small Zstd blocks decompression becomes compute-bound, so the OCP guideline of matching decompression throughput to memory bandwidth is not enough. Its conclusion is that future expander specifications should be designed differently.
Key takeaways
- CXL memory expanders reuse legacy DDR DRAM and compress cold data in hardware; each cache-line miss reads and decompresses a whole block, which an SRAM block cache partly amortizes.
- Commercial expanders use LZ4 with 4 KiB blocks, defaults inherited from software memory compression, even though hardware expanders have no page faults and much lower latencies.
- Better compression reduces DRAM traffic per miss. Because DRAM bandwidth and queueing dominate expander latency, that saving can outweigh slower decompression.
- Averaged over ten fleet traces, Zstd with 2 KiB blocks gives 30.0% more capacity and 12.1% lower latency than LZ4 at 4 KiB (35.8% and 11.5% at 4 KiB blocks); LZ4 at 512 B is fastest but loses 24.4% capacity.
- For small Zstd blocks decompression turns compute-bound, so matching decompression throughput to memory bandwidth, as the OCP guideline does, is not sufficient.
How algorithm and block size set an expander's capacity and latency
The expander's read path, the two knobs in isolation, the fleet-trace simulation, and why DRAM traffic decides the outcome.
1 · How a compressing expander stores and serves data
The OCP expander sits behind a CXL link and holds cold pages in legacy DDR DRAM. Hardware compression is what distinguishes it from CXL memory pooling systems.
- Write a page. When the host moves a page to the expander, the device breaks it into blocks and compresses each block before storing it in DRAM.
- Skip incompressible blocks. If a block's compressed size is more than 80% of its original size, the expander stores it uncompressed. Such a block takes longer to read from DRAM but bypasses decompression.
- Quantize into chunks. The expander manages its DRAM in fixed-size chunks, so each compressed block occupies a whole number of chunks. The rounding lowers effective capacity.
- Read a cache line. On a request, the expander checks its SRAM block cache. On a miss it fetches the whole compressed block from DRAM, decompresses it into the block cache, and supplies the requested line. Later requests to the same block hit in SRAM.
2 · The two knobs in isolation
Section II of the paper measures compressed size and decompression latency on HyperCompressBench for LZ4, Snappy, Zstd and Deflate at seven block sizes (64 B to 4 KiB in powers of 2). LZ4 and Snappy are designed for low latency; Deflate and Zstd for higher compression ratios. Latency comes from FPGA-based decompressor designs, scaled by a factor derived from an ASIC Deflate decompressor.
| Observation | Value in the paper |
|---|---|
| Compressed size, Zstd and Deflate vs LZ4 and Snappy | 13 to 37% smaller; algorithms with the same computational foundations give similar sizes |
| Effect of larger blocks on size | All algorithms compress larger blocks more, with diminishing returns after 1 KiB; tiny blocks save little space |
| Zstd minimum decompression latency | 370 ns, 21× higher than the others |
| Deflate latency | Competitive with LZ4 and Snappy for tiny blocks; for larger blocks Deflate and Zstd are more than 2.5× slower than LZ4 and Snappy |
| Incompressible LZ4 blocks | 18.5% at 512 B, 4.8% at 4 KiB |
| Decompression latency saved by incompressible blocks | 18% at 512 B (the paper's Fig. 3(c)) |
Chunk quantization. Smaller blocks waste more capacity in chunk rounding, and the waste depends on the distribution of compressed sizes. In the paper's example for LZ4 at 512 B, 27% of blocks compress to just over 128 bytes (25% of original size). With 64 B chunks these blocks are stored as 192 B; with 128 B chunks, as 256 B.
3 · The fleet-trace simulation
Raw decompression latency does not translate directly into access latency, for three reasons the paper gives: the CPU can use a line as soon as it is decompressed (so access position within a block matters), the block cache serves later accesses without DRAM traffic, and memory fetches themselves cost substantial latency that better compression reduces. Section III therefore simulates the whole expander with production access patterns.
| Component | Setup |
|---|---|
| Traces | Ten (Trace-0 to Trace-9) from Intel Optane-based far memory in a hyperscaler fleet; each from a different server running a different mix of hundreds of concurrent workloads |
| Sampling | 1% of memory accesses with Linux perf, limited by production overhead. Each record is a physical address and timestamp of a cold-page access; cache and host DRAM accesses are excluded |
| Data compressibility | Each access gets a compressed size and decompression latency pair sampled from the Section II results |
| Simulator | Based on gem5; idle latency matched to a commercial device: 0.00% difference on a cache hit, +2.61% on an incompressible block, −3.59% on a compressed block (Table I) |
| Expander DRAM | Four 8 GB DDR4-2600 DIMMs, one per channel |
| Block cache | 16 MB, 8-way set-associative, LRU, 30 ns access |
| Decompression engines | Enough to match full DRAM bandwidth, per the OCP specification |
| Configurations | LZ4 and Zstd at each block size. The paper treats Snappy as interchangeable with LZ4 and notes Zstd is faster than Deflate at block sizes with sufficient compression |
4 · Results across the ten traces
- Zstd at large blocks wins on both metrics. At 4 KiB it gives 35.8% more capacity and 11.5% lower latency than LZ4 at 4 KiB, despite Zstd's much higher single-block decompression latency. At 2 KiB it gives 30.0% more capacity and 12.1% lower latency.
- LZ4 at small blocks wins on latency only. LZ4 at 512 B has the lowest average latency (33.8% lower) and loses 24.4% capacity. Below 512 B, LZ4 latency stops improving while capacity keeps falling.
- Zstd at small blocks loses on both. From 512 B down, Zstd is slower than the baseline (−31.0% at 512 B, −82.3% at 128 B) and holds less data.
5 · Why a slower algorithm can be faster
To explain the result, the paper defines bandwidth inflation: bytes the expander reads from DRAM to load a block, divided by the bytes of that block the host requests. Read amplification counts the DRAM data read to serve a single cache line; bandwidth inflation also credits the block cache hits that one decompressed block serves. If every line of a compressed block were accessed, bandwidth inflation could fall below 1. The paper's worked example for LZ4 with 4 KiB blocks, on average:
- A 4 KiB block compresses to 44.5% of its size, so a miss reads 44.5% × 64 lines of data: read amplification of 28×.
- The average block cache hit rate is 71%, so 3.4 cache lines are served per block fetched.
- Bandwidth inflation is 64 × 44.5% × 29% = 8.2. Zstd at the same block size, compressing better, has a bandwidth inflation of 6.0.
On the representative trace the paper breaks down (Trace-3, its Fig. 4), larger blocks raise both the block cache hit rate and bandwidth inflation for both algorithms, and Zstd's inflation stays well below LZ4's.
6 · Delayed hits and decompression queueing
Fig. 4 of the paper splits average latency into hit and miss latency for each block size. Two further effects shape the curves:
- Delayed hits. Average hit latency rises with block size for both algorithms, even though an SRAM read does not depend on block size. A request for a block that is still being decompressed counts as a hit but waits for the rest of the decompression. Larger blocks make it more likely that a later access lands in a block in flight.
- LZ4 misses. Miss latency rises with block size because higher bandwidth inflation adds DRAM queueing. Average latency grows more slowly than either, since larger blocks also raise the hit rate.
- Zstd misses at large blocks. Above 512 B, miss latency grows mainly from memory queueing, as with LZ4.
- Zstd misses at small blocks. Below 1 KiB, miss latency rises as blocks shrink. Halving the block from 1 KiB increases the number of blocks to decompress by 16% (a higher miss rate) but cuts per-block decompression time by only 4%. Decompression becomes compute-bound and requests queue at the engines. At 64 B, where all blocks are incompressible, engine load drops again.
7 · More decompression engines
The simulated expander follows the OCP guideline of enough decompression engines to match DRAM bandwidth. Because small Zstd blocks are compute-bound, the paper argues this guideline is insufficient, and its Fig. 5 repeats the Zstd runs with twice as many engines. Engine queueing falls, smaller blocks see much smaller increases in miss latency, and average hit latency also falls because delayed hits finish sooner. Reading the figure, the miss-latency change is visible from 64 B to 512 B, while at 1 KiB and above it is close to zero (this summary's reading of the chart).
8 · Scope of the evaluation
- Sampled traces. Production overhead limited sampling to 1% of memory accesses.
- Assigned compressibility. Performance and privacy concerns prevented measuring compressibility on production servers, so each access gets a size and latency from HyperCompressBench. Biased assignments that favored higher or lower compressibility for high-miss pages changed average latency by only 3 to 5%, leaving the conclusion unchanged.
- Simulated, two algorithms. Results come from a gem5-based model calibrated to a commercial device's idle latency, and only LZ4 and Zstd are simulated.
- One trace in detail. The latency breakdown and bandwidth inflation curves are shown for one representative trace (Trace-3); Table II averages all ten.
FAQ
Q·How can a slower decompressor lower latency?▾
Decompression is only part of a miss. The expander must also read the compressed block from DRAM, and when DRAM bandwidth saturates, requests queue. Zstd compresses better than LZ4, so each miss reads fewer bytes and the queues are shorter. On the fleet traces, that saving outweighs Zstd's longer decompression: Zstd at 4 KiB has 11.5% lower average latency than LZ4 at 4 KiB, even though its single-block decompression latency is much higher.
Q·What is the difference between read amplification and bandwidth inflation?▾
Read amplification is the data read from DRAM to serve one cache line on a miss: for LZ4 at 4 KiB, 44.5% of 64 lines, or 28×. Bandwidth inflation divides the bytes read to load a block by the bytes of that block the host actually requests, so it credits the block cache hits a decompressed block serves. With a 71% hit rate, LZ4 at 4 KiB has a bandwidth inflation of 8.2, and Zstd at 4 KiB has 6.0.
Q·Isn't this the same tradeoff as in zswap?▾
The defaults (LZ4, 4 KiB) come from software memory compression such as zswap, but the paper lists three differences: expanders do not need software-handled page faults, they implement compression in hardware, and they have much lower latencies. When decompression runs in software behind a page fault, the fastest algorithm is the natural choice (inference). In a hardware expander, the paper finds that memory bandwidth and queueing can dominate end-to-end latency, which changes the answer.
Q·Why not use the smallest blocks for the lowest latency?▾
For LZ4, latency stops improving below 512 B (33.8% better at 512 B, 30.5% at 64 B) while capacity keeps falling (−24.4% at 512 B, −55.5% at 64 B). Small blocks compress less, are more often incompressible (18.5% of LZ4 blocks at 512 B vs 4.8% at 4 KiB) and waste more space to chunk rounding. For Zstd, small blocks are worse on both metrics because decompression becomes compute-bound and requests queue at the engines.
Q·What is a delayed hit, and why does it matter here?▾
A delayed hit is a request for a block that is already being fetched and decompressed after an earlier miss. It counts as a block cache hit, but it waits for the remaining decompression time. Larger blocks make delayed hits more likely, which is why average hit latency rises with block size even though an SRAM read takes the same time at any block size.
Q·Where do the traces come from, and what do they contain?▾
From Intel Optane-based far memory deployed in a hyperscaler fleet. There are ten traces, each from a different server running a different mix of hundreds of concurrent workloads. Each records the physical addresses and timestamps of accesses to cold pages, sampled at 1% with Linux perf; cache and host DRAM accesses are excluded. The paper notes that expander access patterns differ from those of individual programs because the OS selects cold pages globally across many workloads.
Q·How was compressibility assigned if it was not measured in production?▾
Each access in a trace is paired with a compressed size and decompression latency sampled from the HyperCompressBench results in Section II, because performance and privacy concerns ruled out measuring compressibility on production servers. As a check, the paper tried biased assignments that gave high-miss pages higher or lower compressibility; average latency changed by only 3 to 5%, and the qualitative conclusion held.
Q·Why are only LZ4 and Zstd simulated?▾
For brevity, based on Section II: Snappy and LZ4 behave interchangeably (algorithms with the same computational foundations give similar compressed sizes), and Zstd is faster than Deflate at the block sizes that offer sufficient compression.
Q·Would more decompression hardware help Zstd?▾
For small blocks, yes. With twice the OCP-recommended number of engines, engine queueing falls, small blocks see much smaller increases in miss latency, and hit latency drops as delayed hits complete sooner (the paper's Fig. 5). At 1 KiB and above, where latency is dominated by memory queueing, the figure shows little change. The paper's point is that matching decompression throughput to memory bandwidth, as the OCP guideline does, is not sufficient.
Q·Are these results from real hardware?▾
No. Compression sizes come from a benchmark corpus, decompression latencies from FPGA designs scaled to ASIC speed, and end-to-end latency from a gem5-based simulator. The simulator's idle latency was matched to a commercial expander: identical on a cache hit, within +2.61% for an incompressible block and −3.59% for a compressed block. The access traces are from production servers.