The Future of Memory: Limits and Opportunities
This page was written by an AI model to give a high-level overview of the paper. Please check the paper for detail: arXiv.
Memory latency, bandwidth, capacity and energy increasingly limit how fast servers run. One response, which revisits the distributed shared memory (DSM) systems of the 1980s and 1990s, is to build huge network-attached memories, many terabytes to petabytes, that are pooled: one flat address space shared by many processors. This short vision paper argues that such designs are untenable because of two practical engineering barriers grounded in physics: scaling and signaling.
Scaling has stopped. The cost per byte of DRAM has been flat for over a decade, SRAM cells no longer get smaller, and the paper sees no roadmap to cut either cost significantly in the next five years. So memory is a growing share of system cost, and in the paper's words "enormous memories will be enormously expensive." Signaling makes distance expensive. Moving a bit from an on-die SRAM costs about 5 fJ, while moving it across solder bumps and circuit-board traces to a DDR DIMM costs about 10,000 fJ, at a small fraction of the bandwidth. DRAM bandwidth per socket has grown, but core counts have grown as fast, so bandwidth per core has stayed flat, and reaching memory over CXL or RDMA adds still more overhead.
The paper's proposal reverses the direction of disaggregation. Instead of one large, shared, homogeneous memory, systems should break memory into smaller slices tightly coupled to compute, which the paper calls physically composable disaggregation. The building block is a compute-memory node that uses 3D integration to stack private local memory on top of compute, as AMD's 3D V-Cache does. Unlike a cache, this memory is explicitly managed and is the only home for node-private data such as execution stacks.
Around the nodes sit two more tiers. State that must be shared across nodes, such as locks, goes in on-package shared memory (for example HBM), which is slower than the private slices but has far better bandwidth and energy efficiency than off-package DRAM. Off-package DRAM remains, but as a bulk, capacity-driven tier for large working sets and cold data rather than a pooled address space. Hardware makes capacities and distances explicit, and software composes the hierarchy: it decides what stays local, what is shared, and what goes to DRAM.
The paper is a position paper and does not evaluate a design. Its evidence is the scaling and integration data it collects: DRAM price per GB and SRAM cell size over time (its Figure 1), DRAM bandwidth per core for Intel and AMD server processors from Skylake and Naples to Granite Rapids and Turin (its Figure 2), and the pitch, energy per bit and bandwidth of four integration methods (its Table 1).
Key takeaways
- The paper argues against huge pooled memories (many terabytes to petabytes shared by many CPUs) on two physical grounds: scaling and signaling.
- Scaling: DRAM cost per byte has been flat for over a decade and SRAM cells have stopped shrinking, so large memories stay expensive.
- Signaling: energy per bit grows and bandwidth shrinks with distance, from about 5 fJ/bit on die to about 10,000 fJ/bit to a DDR DIMM, and DRAM bandwidth per core has been flat.
- Proposal: physically composable disaggregation. Compute-memory nodes stack explicitly managed private memory on compute, share on-package memory (e.g., HBM) for cross-node state, and use off-package DRAM only for bulk capacity.
- Software composes the hierarchy and decides what is local, shared or in DRAM. The paper is a vision paper; it presents trend data, not an evaluation.
From pooled memory to physically composable disaggregation
The design the paper questions, the two physical barriers, the compute-memory node, and what software must do.
1 · The design the paper questions
A large, distributed address space is appealing: applications can grow beyond one host while lower layers handle caching, consistency and placement. Distributed shared memory explored this in the 1980s and 1990s and informed the memory consistency models of today's multi-core and multiprocessor systems. Because memory is increasingly the bottleneck in data center and cloud servers, current research revisits the idea with huge network-attached memories that are pooled, that is, shared, across many processors.
The paper argues that this approach is untenable, and names two barriers, both "practical limits, grounded in physics": scaling (Section 2 below) and signaling (Section 3 below).
2 · Barrier one: memory has stopped scaling
Two-dimensional semiconductor scaling used to deliver denser, cheaper memory. The paper's Figure 1 shows that it has ended for both kinds of memory a server depends on. DRAM's price per GB fell by more than three orders of magnitude from 1995 to about 2012 and has been close to flat since; SRAM cell area fell from about 1 µm² at 90 nm to about 0.02 µm² at 5 nm, with little change at 3 nm and 2 nm (values read off the paper's log-scale plots, approximate). The causes differ:
- SRAM. As transistors approach atomic dimensions, manufacturing tolerances limit how well the two cross-coupled inverters in a cell match, which reduces signal margin. Logic does not have this problem, because each logic stage restores the digital signal.
- DRAM. The cost is in etching high aspect-ratio capacitors and the complex transistor geometry that keeps leakage low. Newer nodes make cells smaller but not cheaper per cell, so DIMMs can keep growing while their cost per byte does not fall. 3D DRAM promises better bit density, but its cost is unknown.
As logic keeps shrinking (more slowly than before), memory becomes a growing fraction of system cost. DRAM already dominates the cost of scaled-up servers, and on-chip caches cannot grow faster than chip area, which is already large (the paper cites 5,428 mm² for AMD SP5).
3 · Barrier two: signaling cost grows with distance
Signaling is the energy needed to move data between components at a given bandwidth. The closer memory sits to logic, the cheaper and faster each bit moves. Caches show the principle: L1, L2 and L3 use the same SRAM technology, but L1 is faster because its banks are smaller, its access is finer-grained and it sits closer to the core. The paper's Table 1 quantifies four integration methods:
| Integration method | Example | Pitch | Energy per bit | Bandwidth per chip |
|---|---|---|---|---|
| On-die (5 nm) | SRAM | 0.028 µm | 5 fJ | 131 TB/s |
| Hybrid bonding | 3D V-Cache | 9 µm | ≈600 fJ | 2.5 TB/s |
| Microbump | HBM | 36 µm | ≈2,000 fJ | 1.2 TB/s |
| C4 solder bump | DDR | 730 µm | ≈10,000 fJ | 0.1 TB/s |
From the on-die row to the DDR row, energy per bit rises 2,000× and bandwidth per chip falls about 1,300× (this summary's arithmetic). The paper describes the same gradient inside a package: an SRAM line far away on a chiplet costs more than a nearby one, a line on another chiplet costs more still, going to DRAM across circuit-board traces is an order of magnitude more expensive, and CXL or RDMA to remote memory add further overhead.
- Pins limit DRAM. DRAM connects through a printed circuit board with a limited number of copper traces and pins: 288 pins for a DDR5 DIMM. HBM moves DRAM dies into the package on a silicon logic base die with through-silicon vias, giving each HBM3E stack 1,024 pins over a shorter distance, which is where its bandwidth advantage comes from.
- Faster pins cost energy. With few pins, the only way to raise bandwidth is faster signaling over copper traces, which costs more energy per bit.
- Per-core bandwidth is flat. A DDR5-5600 DIMM delivers 358 Gbps, and DIMMs per socket grew from 8 to 12, for 4.3 Tbps per socket. Core counts grew at least as fast, so DRAM bandwidth per core has not improved (figure below). Boards cannot fit more DIMMs, and their pin counts are at practical limits.
4 · The compute-memory node and its three tiers
The paper proposes "flipping the script" on disaggregation: integrate memory and compute at finer grain and put more weight on memory utilization, even if that sometimes costs a modest decrease in compute utilization. The design shifts the focus of memory hierarchy design from raw capacity to locality, bandwidth and energy efficiency. Its center is the compute-memory node, which uses 3D integration to stack memory on compute, as in AMD's 3D V-Cache design and Milan-X processors.
| Tier | Where it sits | What lives there | How it is reached |
|---|---|---|---|
| Private local memory | Stacked on the node's compute | Node-specific data such as execution stacks and other thread-private state; explicitly managed, and the exclusive home for that data | Micrometer-scale links: microbumps, hybrid bonds, through-silicon vias or monolithic wafer-level interconnects |
| On-package shared memory | In the package, shared by nodes (e.g., HBM) | State that must span nodes, such as locks, mirroring practice in today's multi-chiplet processors | Slower than private slices, but far better bandwidth and energy efficiency than off-package DRAM |
| Off-package DRAM | On the board, outside the package | Large working sets and cold data; a bulk, capacity-driven tier, not a pooled, flat, shared address space | Circuit-board traces, the costliest of the three |
DRAM stays because integration has physical limits, such as thermal dissipation and module size. As the paper puts it, one cannot fit terabytes of DRAM inside a chip package.
5 · What software does, and what the paper leaves open
In this design hardware makes memory capacities and distances explicit, and software composes the memory system:
- Decide what stays local: node-exclusive data goes in the node's private memory.
- Decide what is shared: state that spans nodes goes in on-package shared memory.
- Decide what is relegated to DRAM: large working sets and cold data.
- Manage movement explicitly, through abstractions that expose near-zero-distance local memory alongside higher-latency shared tiers.
FAQ
Q·What does "physically composable disaggregation" mean?▾
Disaggregation usually means pulling memory out of servers into a shared pool reached over a network. The paper keeps the idea of splitting memory up but goes the other way physically: memory is broken into slices integrated tightly with compute (private stacked memory per node, shared memory in the package, DRAM on the board), and software composes those slices into a memory system by deciding where each piece of data lives.
Q·Is the private local memory just a bigger cache?▾
No. It is built the way a stacked cache such as AMD's 3D V-Cache is built, but the paper says it is explicitly managed and is the exclusive home for node-specific data such as execution stacks and thread-private state. A cache holds copies chosen by hardware; this memory holds the only copy, chosen by software.
Q·Does the paper say CXL or RDMA memory is useless?▾
It says the penalties of reaching remote memory over CXL or RDMA, on top of the already higher cost of off-package DRAM, make remote memory "prohibitively expensive" as the basis for a huge pooled address space. Its target is that pooled, flat, shared design. DRAM itself remains in the proposal, as an off-package capacity tier.
Q·Why not just add more DIMMs to get more bandwidth?▾
The paper's answer is that circuit boards cannot accommodate more DIMMs and their pin counts are at practical limits (288 pins per DDR5 DIMM). Raising signaling speed over copper traces instead costs more energy. DIMMs per socket did grow from 8 to 12, but core counts grew at least as fast, so bandwidth per core stayed flat.
Q·Why has SRAM stopped shrinking when logic still shrinks?▾
An SRAM cell stores a bit in two cross-coupled inverters that must be closely matched. Near atomic dimensions, manufacturing tolerances limit that matching and the cell loses signal margin. Logic gates do not need such matching, because each stage restores the digital signal.
Q·Won't 3D DRAM fix the cost problem?▾
The paper notes that 3D DRAM promises better bit density but that its cost is unknown. Its argument rests on cost per byte, which for planar DRAM has been flat for over a decade because smaller cells have not meant cheaper cells.
Q·Where does data that several nodes need go?▾
In on-package shared memory, for example HBM. The paper's example is locks. This tier is slower than the private slices but has far better bandwidth and energy efficiency than off-package DRAM. HBM gets its advantage from integration: each HBM3E stack has 1,024 pins on a base die in the package, against 288 pins for a DDR5 DIMM on the board.
Q·Why keep off-package DRAM at all?▾
Because integration has physical limits, such as thermal dissipation and module size: terabytes of DRAM do not fit in a chip package. Off-package DRAM becomes a bulk tier for large working sets and cold data, while performance-critical accesses use the on-package memories.
Q·What does software have to do differently?▾
It has to place and move data explicitly: what stays local, what is shared across nodes, and what goes to DRAM. The paper asks for abstractions that expose near-zero-distance local memory alongside higher-latency shared tiers. It does not define those abstractions or a placement policy.
Q·Does the paper evaluate the design?▾
No. It is a short vision paper. Its evidence is trend and technology data: DRAM price per GB and SRAM cell size over time, DRAM bandwidth per core for 13 Intel and AMD server processors, and the pitch, energy and bandwidth of four integration methods. Performance, energy or cost of a compute-memory node system is not measured or simulated.