Paper summary

The Future of Memory: Limits and Opportunities

Samuel Dayo, Shuhan Liu, Peijing Li, Philip Levis, Subhasish Mitra, Thierry Tambe, David Tennenhouse, H.-S. Philip Wong · Stanford University · Independent Researcher

BigMem '25, The International Workshop on Big Memory · Seoul, Republic of Korea · October 13, 2025, co-located with SOSP 2025

This page was written by an AI model to give a high-level overview of the paper. Please check the paper for detail: arXiv.

The big idea

Memory latency, bandwidth, capacity and energy increasingly limit how fast servers run. One response, which revisits the distributed shared memory (DSM) systems of the 1980s and 1990s, is to build huge network-attached memories, many terabytes to petabytes, that are pooled: one flat address space shared by many processors. This short vision paper argues that such designs are untenable because of two practical engineering barriers grounded in physics: scaling and signaling.

Scaling has stopped. The cost per byte of DRAM has been flat for over a decade, SRAM cells no longer get smaller, and the paper sees no roadmap to cut either cost significantly in the next five years. So memory is a growing share of system cost, and in the paper's words "enormous memories will be enormously expensive." Signaling makes distance expensive. Moving a bit from an on-die SRAM costs about 5 fJ, while moving it across solder bumps and circuit-board traces to a DDR DIMM costs about 10,000 fJ, at a small fraction of the bandwidth. DRAM bandwidth per socket has grown, but core counts have grown as fast, so bandwidth per core has stayed flat, and reaching memory over CXL or RDMA adds still more overhead.

The paper's proposal reverses the direction of disaggregation. Instead of one large, shared, homogeneous memory, systems should break memory into smaller slices tightly coupled to compute, which the paper calls physically composable disaggregation. The building block is a compute-memory node that uses 3D integration to stack private local memory on top of compute, as AMD's 3D V-Cache does. Unlike a cache, this memory is explicitly managed and is the only home for node-private data such as execution stacks.

Around the nodes sit two more tiers. State that must be shared across nodes, such as locks, goes in on-package shared memory (for example HBM), which is slower than the private slices but has far better bandwidth and energy efficiency than off-package DRAM. Off-package DRAM remains, but as a bulk, capacity-driven tier for large working sets and cold data rather than a pooled address space. Hardware makes capacities and distances explicit, and software composes the hierarchy: it decides what stays local, what is shared, and what goes to DRAM.

The paper is a position paper and does not evaluate a design. Its evidence is the scaling and integration data it collects: DRAM price per GB and SRAM cell size over time (its Figure 1), DRAM bandwidth per core for Intel and AMD server processors from Skylake and Naples to Granite Rapids and Turin (its Figure 2), and the pitch, energy per bit and bandwidth of four integration methods (its Table 1).

Two ways to build a big memory system
a pooled, shared address space · compute-memory nodes with explicit tiers
Pooled: one shared memory for all CPUCPUCPUCPU network, CXL or RDMA shared memory pool many terabytes to petabytes one flat address space Scaling cost per byte is flat for SRAM and DRAM Signaling energy and bandwidth worsen with distance The paper argues that these two limits make huge pooled memories expensive (Sections 1 to 3). Composable: memory next to compute one package private local memory node-private data private local memory node-private data compute compute on-package shared memory (e.g., HBM) state shared across nodes, such as locks off-package DRAM bulk capacity, large working sets, cold data 1 2 3 1 hybrid bonding: ≈600 fJ/bit, 2.5 TB/s per chip 2 microbump: ≈2,000 fJ/bit, 1.2 TB/s per chip 3 C4 solder bump (DDR): ≈10,000 fJ/bit, 0.1 TB/s
Left panel from the paper's Sections 1 to 3. Right panel from Section 4. The link numbers are the paper's Table 1, matched to each tier by the examples the paper itself gives (hybrid bonding for V-Cache, microbumps for HBM, C4 bumps for DDR). The layout of two nodes in one package is illustrative.

Key takeaways

  • The paper argues against huge pooled memories (many terabytes to petabytes shared by many CPUs) on two physical grounds: scaling and signaling.
  • Scaling: DRAM cost per byte has been flat for over a decade and SRAM cells have stopped shrinking, so large memories stay expensive.
  • Signaling: energy per bit grows and bandwidth shrinks with distance, from about 5 fJ/bit on die to about 10,000 fJ/bit to a DDR DIMM, and DRAM bandwidth per core has been flat.
  • Proposal: physically composable disaggregation. Compute-memory nodes stack explicitly managed private memory on compute, share on-package memory (e.g., HBM) for cross-node state, and use off-package DRAM only for bulk capacity.
  • Software composes the hierarchy and decides what is local, shared or in DRAM. The paper is a vision paper; it presents trend data, not an evaluation.
Mechanism, step by step

From pooled memory to physically composable disaggregation

The design the paper questions, the two physical barriers, the compute-memory node, and what software must do.

1 · The design the paper questions

A large, distributed address space is appealing: applications can grow beyond one host while lower layers handle caching, consistency and placement. Distributed shared memory explored this in the 1980s and 1990s and informed the memory consistency models of today's multi-core and multiprocessor systems. Because memory is increasingly the bottleneck in data center and cloud servers, current research revisits the idea with huge network-attached memories that are pooled, that is, shared, across many processors.

The paper argues that this approach is untenable, and names two barriers, both "practical limits, grounded in physics": scaling (Section 2 below) and signaling (Section 3 below).

2 · Barrier one: memory has stopped scaling

Two-dimensional semiconductor scaling used to deliver denser, cheaper memory. The paper's Figure 1 shows that it has ended for both kinds of memory a server depends on. DRAM's price per GB fell by more than three orders of magnitude from 1995 to about 2012 and has been close to flat since; SRAM cell area fell from about 1 µm² at 90 nm to about 0.02 µm² at 5 nm, with little change at 3 nm and 2 nm (values read off the paper's log-scale plots, approximate). The causes differ:

  1. SRAM. As transistors approach atomic dimensions, manufacturing tolerances limit how well the two cross-coupled inverters in a cell match, which reduces signal margin. Logic does not have this problem, because each logic stage restores the digital signal.
  2. DRAM. The cost is in etching high aspect-ratio capacitors and the complex transistor geometry that keeps leakage low. Newer nodes make cells smaller but not cheaper per cell, so DIMMs can keep growing while their cost per byte does not fall. 3D DRAM promises better bit density, but its cost is unknown.

As logic keeps shrinking (more slowly than before), memory becomes a growing fraction of system cost. DRAM already dominates the cost of scaled-up servers, and on-chip caches cannot grow faster than chip area, which is already large (the paper cites 5,428 mm² for AMD SP5).

The paper's conclusion from scaling: "enormous memories will be enormously expensive." Systems will have to use memory more efficiently instead of provisioning more of it.

3 · Barrier two: signaling cost grows with distance

Signaling is the energy needed to move data between components at a given bandwidth. The closer memory sits to logic, the cheaper and faster each bit moves. Caches show the principle: L1, L2 and L3 use the same SRAM technology, but L1 is faster because its banks are smaller, its access is finer-grained and it sits closer to the core. The paper's Table 1 quantifies four integration methods:

Integration methodExamplePitchEnergy per bitBandwidth per chip
On-die (5 nm)SRAM0.028 µm5 fJ131 TB/s
Hybrid bonding3D V-Cache9 µm≈600 fJ2.5 TB/s
MicrobumpHBM36 µm≈2,000 fJ1.2 TB/s
C4 solder bumpDDR730 µm≈10,000 fJ0.1 TB/s

From the on-die row to the DDR row, energy per bit rises 2,000× and bandwidth per chip falls about 1,300× (this summary's arithmetic). The paper describes the same gradient inside a package: an SRAM line far away on a chiplet costs more than a nearby one, a line on another chiplet costs more still, going to DRAM across circuit-board traces is an order of magnitude more expensive, and CXL or RDMA to remote memory add further overhead.

  1. Pins limit DRAM. DRAM connects through a printed circuit board with a limited number of copper traces and pins: 288 pins for a DDR5 DIMM. HBM moves DRAM dies into the package on a silicon logic base die with through-silicon vias, giving each HBM3E stack 1,024 pins over a shorter distance, which is where its bandwidth advantage comes from.
  2. Faster pins cost energy. With few pins, the only way to raise bandwidth is faster signaling over copper traces, which costs more energy per bit.
  3. Per-core bandwidth is flat. A DDR5-5600 DIMM delivers 358 Gbps, and DIMMs per socket grew from 8 to 12, for 4.3 Tbps per socket. Core counts grew at least as fast, so DRAM bandwidth per core has not improved (figure below). Boards cannot fit more DIMMs, and their pin counts are at practical limits.
DRAM bandwidth per core, server processors
GB/s per core by release year · label: processor and core count · redrawn from the paper's Figure 2
0123 456 201620182020 202220242026 GB/s per core year Skylake 28 Cascade Lake 28 Cooper Lake 28 Ice Lake 40 Sapphire Rapids 60 Emerald Rapids 64 Granite Rapids 128 Naples 32 Rome 64 Milan 64 Genoa 96 Bergamo 128 Turin 192 Intel AMD
Redrawn from the paper's Figure 2 (Section 3). Point positions are read off the paper's chart and are approximate; core counts are as labeled in the paper.

4 · The compute-memory node and its three tiers

The paper proposes "flipping the script" on disaggregation: integrate memory and compute at finer grain and put more weight on memory utilization, even if that sometimes costs a modest decrease in compute utilization. The design shifts the focus of memory hierarchy design from raw capacity to locality, bandwidth and energy efficiency. Its center is the compute-memory node, which uses 3D integration to stack memory on compute, as in AMD's 3D V-Cache design and Milan-X processors.

TierWhere it sitsWhat lives thereHow it is reached
Private local memoryStacked on the node's computeNode-specific data such as execution stacks and other thread-private state; explicitly managed, and the exclusive home for that dataMicrometer-scale links: microbumps, hybrid bonds, through-silicon vias or monolithic wafer-level interconnects
On-package shared memoryIn the package, shared by nodes (e.g., HBM)State that must span nodes, such as locks, mirroring practice in today's multi-chiplet processorsSlower than private slices, but far better bandwidth and energy efficiency than off-package DRAM
Off-package DRAMOn the board, outside the packageLarge working sets and cold data; a bulk, capacity-driven tier, not a pooled, flat, shared address spaceCircuit-board traces, the costliest of the three
Not a cache. The private local memory resembles a stacked cache physically, but the paper specifies that it is explicitly managed and is the only home for node-private data. Hardware does not decide what it holds.

DRAM stays because integration has physical limits, such as thermal dissipation and module size. As the paper puts it, one cannot fit terabytes of DRAM inside a chip package.

5 · What software does, and what the paper leaves open

In this design hardware makes memory capacities and distances explicit, and software composes the memory system:

  1. Decide what stays local: node-exclusive data goes in the node's private memory.
  2. Decide what is shared: state that spans nodes goes in on-package shared memory.
  3. Decide what is relegated to DRAM: large working sets and cold data.
  4. Manage movement explicitly, through abstractions that expose near-zero-distance local memory alongside higher-latency shared tiers.
Scope. This is a three-page vision paper. It argues from technology trends and published figures; it does not build or evaluate a compute-memory node, and it does not specify the software abstractions or placement policies (this summary's reading of the paper). It also notes that the cost of 3D DRAM, which could change the density picture, is unknown.
Common questions

FAQ

Q·What does "physically composable disaggregation" mean?▾

Disaggregation usually means pulling memory out of servers into a shared pool reached over a network. The paper keeps the idea of splitting memory up but goes the other way physically: memory is broken into slices integrated tightly with compute (private stacked memory per node, shared memory in the package, DRAM on the board), and software composes those slices into a memory system by deciding where each piece of data lives.

Q·Is the private local memory just a bigger cache?▾

No. It is built the way a stacked cache such as AMD's 3D V-Cache is built, but the paper says it is explicitly managed and is the exclusive home for node-specific data such as execution stacks and thread-private state. A cache holds copies chosen by hardware; this memory holds the only copy, chosen by software.

Q·Does the paper say CXL or RDMA memory is useless?▾

It says the penalties of reaching remote memory over CXL or RDMA, on top of the already higher cost of off-package DRAM, make remote memory "prohibitively expensive" as the basis for a huge pooled address space. Its target is that pooled, flat, shared design. DRAM itself remains in the proposal, as an off-package capacity tier.

Q·Why not just add more DIMMs to get more bandwidth?▾

The paper's answer is that circuit boards cannot accommodate more DIMMs and their pin counts are at practical limits (288 pins per DDR5 DIMM). Raising signaling speed over copper traces instead costs more energy. DIMMs per socket did grow from 8 to 12, but core counts grew at least as fast, so bandwidth per core stayed flat.

Q·Why has SRAM stopped shrinking when logic still shrinks?▾

An SRAM cell stores a bit in two cross-coupled inverters that must be closely matched. Near atomic dimensions, manufacturing tolerances limit that matching and the cell loses signal margin. Logic gates do not need such matching, because each stage restores the digital signal.

Q·Won't 3D DRAM fix the cost problem?▾

The paper notes that 3D DRAM promises better bit density but that its cost is unknown. Its argument rests on cost per byte, which for planar DRAM has been flat for over a decade because smaller cells have not meant cheaper cells.

Q·Where does data that several nodes need go?▾

In on-package shared memory, for example HBM. The paper's example is locks. This tier is slower than the private slices but has far better bandwidth and energy efficiency than off-package DRAM. HBM gets its advantage from integration: each HBM3E stack has 1,024 pins on a base die in the package, against 288 pins for a DDR5 DIMM on the board.

Q·Why keep off-package DRAM at all?▾

Because integration has physical limits, such as thermal dissipation and module size: terabytes of DRAM do not fit in a chip package. Off-package DRAM becomes a bulk tier for large working sets and cold data, while performance-critical accesses use the on-package memories.

Q·What does software have to do differently?▾

It has to place and move data explicitly: what stays local, what is shared across nodes, and what goes to DRAM. The paper asks for abstractions that expose near-zero-distance local memory alongside higher-latency shared tiers. It does not define those abstractions or a placement policy.

Q·Does the paper evaluate the design?▾

No. It is a short vision paper. Its evidence is trend and technology data: DRAM price per GB and SRAM cell size over time, DRAM bandwidth per core for 13 Intel and AMD server processors, and the pitch, energy and bandwidth of four integration methods. Performance, energy or cost of a compute-memory node system is not measured or simulated.

Background terms
SRAM (static RAM)Memory that stores each bit in a pair of cross-coupled inverters, using several transistors per bit. It is fast and built on the same die as logic, and is used for CPU caches.
DRAM (dynamic RAM)Memory that stores each bit as charge on a capacitor behind one transistor. It is denser and cheaper than SRAM but must be refreshed, and it is the usual main memory of a server.
DIMMA dual in-line memory module: a circuit board carrying DRAM chips that plugs into a memory slot on the motherboard. A DDR5 DIMM has 288 pins.
HBM (High-Bandwidth Memory)A stack of DRAM dies on a logic base die, connected by through-silicon vias and placed inside the processor package. Its many pins (1,024 per HBM3E stack) give it high bandwidth.
Distributed shared memory (DSM)A system that presents the memory of many machines as one shared address space, with lower layers handling caching and consistency. It was studied in the 1980s and 1990s.
Memory disaggregation and poolingSeparating memory from individual servers so that it can be attached over a network or fabric and shared among many processors as one pool.
CXL (Compute Express Link)A cache-coherent interconnect built on the PCIe physical layer that lets a processor attach memory outside its own memory channels.
RDMA (remote direct memory access)A network capability that lets one machine read or write another machine's memory directly, without involving the remote CPU (general knowledge).
ChipletOne of several separately manufactured dies that are packaged together to form a larger processor, as in modern server CPUs.
PitchThe center-to-center spacing between adjacent connections of an integration method. A smaller pitch fits more connections in the same area.
Energy per bitThe energy spent to move one bit of data across a connection, here in femtojoules (fJ, 10-15 joules).
Hybrid bondingA way of joining two dies face to face with direct copper-to-copper bonds at a fine pitch (9 µm in the paper's table), used by AMD's 3D V-Cache.
MicrobumpA small solder bump (36 µm pitch in the paper's table) that connects dies inside a package, such as HBM stacks.
C4 solder bumpControlled-collapse chip connection: the large solder bumps (730 µm pitch in the paper's table) that connect a package to the circuit board and from there to DDR memory.
Through-silicon via (TSV)A vertical electrical connection that passes through a silicon die, linking stacked dies.
3D V-CacheAMD's design that stacks an extra SRAM cache die on top of a CPU die with hybrid bonding, used in Milan-X processors.
Compute-memory nodeThe paper's building block: compute with private local memory integrated on top of it by 3D integration, the memory explicitly managed by software.
Summary of S. Dayo, S. Liu, P. Li, P. Levis, S. Mitra, T. Tambe, D. Tennenhouse and H.-S. P. Wong, "The Future of Memory: Limits and Opportunities," BigMem '25. Numbers are from the paper unless marked as this summary's arithmetic.