Paper summary

Future of Memory: Massive, Diverse, Tightly Integrated with Compute – from Device to Software

Shuhan Liu, Robert M. Radway, Xinxin Wang, Jimin Kwon, Caroline Trippel, Philip Levis, Subhasish Mitra, H.-S. Philip Wong · Stanford University

IEDM 2024, 70th IEEE International Electron Devices Meeting · San Francisco, CA · December 7 to 11, 2024

This page was written by an AI model to give a high-level overview of the paper. Please check the paper for detail: IEEE Xplore.

The big idea

Future computing systems must deliver higher throughput at much lower energy. This vision paper names three challenges that meet at memory: demand, as AI/ML, big data and networked systems need far more memory; the memory wall, the time and energy spent shuttling data between compute and off-chip memory; and the miniaturization wall, the inability to keep gaining device density in 2D because of physical limits and fabrication complexity.

Software generally assumes one random-access address space with uniform latency and bandwidth, but its actual use of memory is far from uniform. Today's hierarchy (SRAM, DRAM, Flash) is already diverse, yet the paper argues it is not enough. Device research often optimizes one attribute at a time, such as a single entry in a comparison table, while software needs a particular set of attributes together. Memory also lives on separate chips for technology, cost and business reasons, behind interfaces (DDR, HBM, PCIe, CXL) that add energy, latency and bandwidth overhead. The paper notes that the memory wall is often wrongly blamed on the von Neumann architecture; it comes from how the memory hierarchy is physically built.

The paper's position is that future memory must be (1) massive, (2) diverse, and (3) tightly integrated with compute, both physically and architecturally, and that (4) the software stack must exploit that integration. It covers points 1 to 3 and leaves 4 out of scope. Rather than expecting a new device to replace existing memory, designers should start from how software uses data and match each use to a technology whose trade-offs fit it.

To do that, the paper characterizes data by read/write frequency, address predictability and data lifetime, and gives three example use cases (Type A, B, C below). Each use case turns the large space of memory technologies into bands, with a specific list of improvements each technology would need to serve it (the paper's Table III). Physically, the memories move onto or atop the compute chip through a continuum of 3D interconnects, from advanced packaging (pin pitch projected below 1 µm) down to tens of nanometers with monolithic 3D integration. For models larger than one chip's memory, the paper points to Illusion Systems: several chips, each with its own on-chip memory, and no off-chip memory at all.

The paper supports the vision with case studies, mostly drawn from the authors' prior and companion work. A hybrid gain cell (HGC) has 3.6× the density of high-current SRAM; in a 5 nm circuit simulation its memory macro had 72% smaller cell size, 15% lower read energy and 24% lower standby power at the same 2 GHz frequency. HGC monolithically integrated with RRAM saves more than 80% energy for AI/ML training and inference. On-chip RRAM's non-volatility gives up to 9× energy benefits for intermittent edge AI/ML (for example MobileBERTTINY) compared with SRAM, off-chip DRAM and NAND Flash systems. Illusion Systems, for example 8× larger than a single chip, come within 5% of the energy and execution time of an ideal single "Dream Chip".

Three ways software uses memory
use case → what can be traded → what each technology still needs
Type A frequent reads, infrequent writes ACCESS PATTERN predictable, often sequential; data written once, read many times EXAMPLES AI/ML inference weights; processor instruction caches WHAT CAN BE TRADED pay more on writes (energy, latency, endurance) for better reads STILL NEEDED (TABLE III, E.G.) SRAM: lower standby power hybrid gain cell: capacity Type B frequent writes, few reads per write ACCESS PATTERN written in (often sequential) blocks, read soon, then discarded EXAMPLES streaming I/O, AI/ML activations, data analytics WHAT CAN BE TRADED short lifetimes let retention be traded for speed, density or energy STILL NEEDED (TABLE III, E.G.) SRAM: density MRAM: lower write energy Type C random reads, sequential writes ACCESS PATTERN Zipf-like random reads; writes across contiguous blocks EXAMPLES file system buffers; live data write logs WHAT CAN BE TRADED read delay can be hidden with multithreading; read energy matters STILL NEEDED (TABLE III, E.G.) SRAM: capacity RRAM, MRAM, PCM, FeRAM: all each data type is mapped to a matched memory class on or atop the compute chip joined by a continuum of 3D interconnects (monolithic 3D or 2.5D/3D packaging) and data buses
From Section II, Figures 2 and 3, and Table III of the paper. The "still needed" lines are two examples per column picked from Table III, which lists the improvements needed by nine memory technologies for each use case.

Key takeaways

  • Three challenges meet at memory: exploding demand, the memory wall (data movement cost) and the miniaturization wall (2D density stalling).
  • Software uses memory in very different ways. The paper argues for matching a set of device attributes to each use case instead of improving one attribute in isolation.
  • Three example use cases (A: read-mostly and predictable; B: write-heavy and short-lived; C: random reads, sequential writes) each define the improvements every technology would need (Table III).
  • Tight physical integration (2.5D/3D packaging down to monolithic 3D) brings memory onto compute; Illusion Systems spread large models across chips with no off-chip memory.
  • Case studies: HGC 3.6× denser than high-current SRAM, HGC plus RRAM saving over 80% energy, RRAM up to 9× energy benefit for intermittent edge AI, Illusion Systems within 5% of a Dream Chip. Software abstractions are left out of scope.
Mechanism, step by step

From software use cases to matched, tightly integrated memories

The device background, the use-case bands, physical integration, compute-in-memory and multi-chip systems, and the case studies.

1 · Device background for a systems reader

For logic, the field-effect transistor (FET) dominates and the paper assumes it will continue to. Memory is different: many devices store a bit using different physics, and each makes different trade-offs. A short guide (general knowledge, not from the paper):

  • Charge-based: SRAM (a 6-transistor latch in the logic process, fast but large), DRAM (a capacitor per bit that leaks and must be refreshed), gain cells (a 2- or 3-transistor cell that stores charge on a transistor gate, denser than SRAM but with limited retention), and Flash (charge trapped in a transistor, dense and non-volatile but slow to write).
  • Resistance- and polarization-based (emerging non-volatile memories, NVMs): RRAM (a filament in an oxide that switches resistance), PCM (a material switched between crystalline and amorphous phases), MRAM (a magnetic tunnel junction), and FeRAM/FeFET (a ferroelectric layer whose polarization stores the bit).

The paper's Table II shows the "traditional" view, each attribute rated in isolation:

SRAMDRAMRRAMMRAM
EnergyLowMedHighHigh
SpeedHighMedLowLow
DensityLowMedHighHigh
EnduranceHighHighLowMed

Its Table I instead rates a wider set (adding Flash, two kinds of gain cell, PCM, FeFET, FeRAM and multi-layer 3D versions) on read and write energy, standby power, read and write latency, endurance, retention, density, and whether they can be built in one or many layers on a logic chip. Two device trends drive the paper's integration argument. First, 2D density is running out: the paper's Fig. 8(a) shows high-density SRAM rising from about 2 Mb/mm² at 65 nm to about 50 Mb/mm² at 3 nm, with little gain over the last few nodes (read from the plot). Second, many of the newer devices can be fabricated in layers above the transistors, so capacity can grow vertically. Density in Table I is "equivalent density after considering multiple layers."

2 · The problem: three walls and a uniform abstraction

  1. Demand. AI/ML, big data and networked systems need massive memory, much of it off-chip today.
  2. Memory wall. Significant time and energy go to moving data between compute and memory chips. Without a change, the paper warns, computing's energy demand could far exceed sustainable energy production (citing published projections).
  3. Miniaturization wall. Device density in 2D is limited by physics and fabrication complexity.
  4. A uniform view of memory. Software assumes uniform latency and bandwidth, while its accesses are far from uniform. Device work that optimizes single table entries misses what each use actually needs.
The paper's diagnosis: the memory wall is not inherent to the von Neumann architecture. It comes from the physical hierarchy: memories built as separate chips (SRAM, DRAM, Flash) for technological, cost and business reasons. Integrating massive, diverse memories onto compute chips is conceivable, so the question becomes what system-level benefit it brings.

3 · From software use cases to memory bands

The paper describes data by three attributes: read/write access frequency, read/write address predictability, and data lifetime (from creation to destruction). Bulk storage is out of scope. Three example use cases, placed by their ratio of reads to writes and access pattern in the paper's Fig. 3:

Use caseBehaviorExamples and scaleOpportunity
Type AFrequent reads, infrequent writes, predictable (often sequential) accessesInference weights; instruction caches. A Zsim simulation shows instruction cache misses below 5×10-6, with thousands of instructions executed billions of times per runTrade write energy, latency and endurance for better read energy, delay and retention; write buffering and prefetching help. Some write capability is still needed (e.g., hourly LLM model updates)
Type BFrequent writes, few reads per write, short data lifetimeStreaming I/O, AI/ML activations, analytics. Blocks of tens of kB read after tens of µs, MBs after ms, and 1 to 128 MB after secondsShort lifetimes let retention be traded for speed, density or energy (gain cells, MRAM, FeRAM)
Type CRandom (e.g., Zipf-like) reads, sequential writes across contiguous blocksFile system buffers; defragmentation and live data write logsRead delay can be hidden with multithreading, but low read energy is critical; on-chip gain cells have far lower access energy than off-chip DRAM

Each use case narrows the space of technologies into a band and a list of targets. The paper's Table III, based on state-of-the-art macro demonstrations, gives the improvement each technology needs per use case. A few rows:

SRAMDRAMHybrid gain cellRRAMMRAM
AStandby powerRead energy, speed, standby powerCapacityWrite and read energy, enduranceWrite and read energy
BDensityRetentionCapacityEndurance, write energyWrite energy
CCapacityRead and write energy, speedCapacity, write speedAllAll
  1. Optimize combinations, not single numbers. Improving MRAM or RRAM endurance alone, without improving write energy, may not help an energy-constrained Type A system, because using the extra write cycles costs a lot of energy. The paper's Fig. 5(a) splits the write energy and endurance space into "ideal", "domain-specific" and "not of interest" (high endurance with high write energy).
  2. Choose a point on the trade-off curve from the application. Gain cells trade retention against speed along a curve set by the transistor's on/off current ratio (Fig. 5(b)). Which threshold voltage to pick should follow the application, and design knobs (voltage), better devices (sharper on/off transition) or circuits (hybrid gain cells) can shift the curve toward the ideal corner.
  3. Integration can matter more than device type. In Table III, 3D V-Cache (SRAM stacked on a processor die, general knowledge) and DRAM fall in the same band, while two kinds of gain cell differ.

4 · Tight physical integration

High-bandwidth memory access needs a high density of connections between memory and compute. The paper's Table IV lists where each memory can sit: in the logic process itself (SRAM), in a package (DRAM DIMMs, Flash), stacked in a package (HBM), or monolithically 3D integrated above the silicon transistors.

A continuum of memory-to-compute connections
farther from compute → on top of compute · connection density rises
Off-chip packages DRAM DIMMs, Flash connection density: low large, high-capacitance package pins, few parallel links; protocols (DDR, PCIe, CXL) add overhead 2.5D/3D stacking HBM DRAM stacked on a logic die or interposer connection density: medium through-silicon vias 5 to 50 µm in size Finer packaging advanced packaging pin pitch is projected to shrink below 1 µm today, packaging is still far below on-chip density Monolithic 3D gain cell, RRAM, MRAM, PCM, FeRAM built in layers above the Si transistors connection density: high vias of tens of nm; needs low-temperature fabrication macroscopic pins 5 to 50 µm below 1 µm (projected) tens of nm connection pitch shrinks from left to right SRAM sits in the logic process itself. Table IV rates connection density low for DIMM DRAM and Flash, medium for HBM, and high for SRAM and every monolithic 3D option.
From Section III and Table IV of the paper. The four stages are this summary's grouping of the paper's integration options; the pitch values are the paper's.

The paper's Fig. 9 shows a concrete example: TEM cross-sections of a gain cell and an RRAM monolithically integrated with silicon FETs on the same 130 nm chip, combining RRAM's non-volatility with the gain cell's high endurance, connected by dense vertical interconnects. The paper's point is that these connection densities make it possible to re-architect the system to use far more bandwidth.

5 · Compute-in-memory and its limits

Compute-in-memory (CIM) performs computation inside memory arrays to cut data movement. The paper weighs two kinds:

Analog CIMDigital CIM
How it computesMatrix-vector multiplication by Ohm's and Kirchhoff's laws: the summed current on a bitline is the resultA small digital unit next to each bitcell; results summed by an adder tree
AccuracyDegraded by NVM device variation, programming non-linearity, limited on/off ratio, drift and IR drop; even chip-in-the-loop fine-tuning fails to keep software accuracy for RRAM CIMFull bit accuracy
EnergyBelow 10 fJ per 4-bit operation at the array level, but above 100 fJ per 8-bit operation: each extra bit needs 4× the capacitance and energy in the ADCSimple logic must fit the cell pitch (e.g., 6T SRAM plus a few transistors), so it may need bit-serial ALUs and longer runtime
Other limitsCostly DACs and ADCs; small kernels underuse the array; large kernels need post-ADC digital accumulationProgrammability across workloads
Both kinds share a limit the paper stresses: AI/ML models also need operations other than matrix-vector multiplication (attention's vector-vector products, depth-wise separable convolutions), and the memory and compute elements are fixed at design time. If the model does not fit on chip, relying on off-chip memory brings back the memory wall regardless of CIM.

6 · Illusion Systems: many chips that act like one

  1. The ideal. A "Dream Chip" holds the whole model in massive on-chip memory next to compute, accessible quickly at low energy.
  2. The practical version. An Illusion System spreads AI/ML execution across N chips, each with its own on-chip memory and compute and mechanisms for quick wakeup and shutdown, connected by a sparsely used inter-chip network. There is no off-chip memory.
  3. What makes it work. Special mappings of the model to the chips minimize inter-chip traffic, co-optimized with per-chip memory size, heterogeneous inter- and intra-chip interconnects, and fine-grained power management of idle chips.
  4. Result from prior work. Digital accelerator systems, for example 8× larger than single chips, reach system-level energy and execution time within 5% of the corresponding Dream Chip. The paper distinguishes this from traditional parallel processing.

7 · Case studies and scope

Case studyResultBasis
Hybrid gain cell (Si read transistor, oxide-semiconductor write transistor) vs SRAM3× the density of high-density SRAM, scalable to FinFET nodes; 3.6× density vs high-current SRAM. Macro: 72% smaller cell size, 15% lower read energy, 24% lower standby power at the same 2 GHzCadence Spectre circuit simulation at 5 nm with the same peripheral circuits and array architecture; an ITO transistor with 40 nm channel demonstrated experimentally
HGC monolithically integrated with RRAMMore than 80% energy savings for AI/ML training and inferenceCompanion IEDM 2024 work
On-chip RRAM vs SRAM-only, SRAM plus off-chip Flash, and off-chip low-power DRAMUp to 9× lower system energy for intermittent edge AI/ML (e.g., MobileBERTTINY); off-chip DRAM costs 1.6× or more energy vs RRAM-based systemsPrior work; system energy plotted against duty cycle (Fig. 10)

The RRAM result comes from non-volatility, not density. RRAM bitcells are 3× denser than 6T SRAM (0.042 vs 0.127 µm² in TSMC N28), but foundry RRAM macros, with their peripherals and controllers, do not deliver the full density benefit at the system level. The energy win is that an RRAM chip can power-gate between inferences without losing its weights. An SRAM chip that idles long enough is better off turning off and re-fetching weights from NAND Flash; low-leakage 8T SRAM moves that crossover to lower duty cycles but costs area.

What the paper does not cover. Point (4) of its vision, exploiting tight integration in the software stack, is beyond scope; the paper says only that diverse memory trade-offs (for example endurance vs retention, energy vs latency) must be exposed to software through appropriate abstractions. Bulk data storage is also out of scope, and Table III excludes 3D Flash and 3D FeFET for that reason. The paper expects demand to keep outpacing capacity, which is why it calls multi-chip co-design such as Illusion Systems critical.
Common questions

FAQ

Q·Is this a new device or a new system?▾

Neither, on its own. It is a four-page vision paper that argues for a direction (massive, diverse memory tightly integrated with compute) and backs it with case studies from the authors' prior and companion work on gain cells, RRAM integration and multi-chip systems.

Q·Isn't the memory wall a consequence of the von Neumann architecture?▾

The paper says no, and calls that attribution often incorrect. Von Neumann wanted an indefinitely large memory with every word immediately available, and was "forced" into a hierarchy because it could not be built. The wall comes from implementing that hierarchy as separate chips, for technological, cost and business reasons. The paper's prescription applies to von Neumann and non-von Neumann architectures alike.

Q·Why not wait for one universal memory that is better at everything?▾

The paper argues against expecting new devices to replace existing memory. Each technology trades attributes against one another, and domain-specific matching gives high efficiency. Its Fig. 4 shows that the needs of different use cases can be subsets of one another, can overlap, or can be disjoint, so no single point satisfies all of them.

Q·Why does write energy matter if endurance improves?▾

For a read-mostly (Type A) system on a tight energy budget, more endurance cycles are only useful if writing is affordable. The paper's example: improving MRAM or RRAM endurance without improving write energy may not be enough, because using the gained cycles can cost a lot of energy. High endurance with high write energy is the "not of interest" corner of its Fig. 5(a).

Q·How short is "short data lifetime" for Type B data?▾

The paper cites blocks of tens of kB read back within tens of µs, MBs within ms, and 1 to 128 MB within seconds, from streaming I/O, AI/ML activations and analytics. Because such data does not need to survive long, a memory can give up retention for speed, density or energy, as gain cells, MRAM and FeRAM can.

Q·How is this related to CXL memory tiering or HBM?▾

The paper places both on its integration spectrum. HBM stacks DRAM using 5 to 50 µm through-silicon vias for higher bandwidth and lower energy. CXL, PCIe, DDR and similar protocols are needed because off-chip devices come from different manufacturers, but they add energy, latency and bandwidth overhead. The paper's direction goes further, to finer-pitch packaging and monolithic 3D integration on the compute chip.

Q·Does compute-in-memory solve the memory wall?▾

Not by itself, in the paper's view. Analog CIM loses accuracy from device non-idealities and loses energy efficiency quickly as precision rises (below 10 fJ per 4-bit operation but above 100 fJ per 8-bit operation at the array level). Digital CIM keeps full accuracy but needs simple logic that fits the cell pitch. Both struggle with operations beyond matrix-vector multiplication, and if the model exceeds on-chip memory, off-chip accesses bring the wall back.

Q·What does software have to do?▾

The paper names this as point (4) of its vision and leaves it out of scope. Its conclusion states the requirement: memory should move from a uniform, random-access view to a collection of different memories, and their trade-offs (for example endurance vs retention, energy vs latency) should be exposed to software through appropriate abstractions. Table IV's caption notes that today's interface protocols are managed by software, from device drivers to the OS, with minimal hardware support such as memory management units.

Q·Is the 9× RRAM energy benefit general?▾

It is stated for infrequent and intermittent edge AI/ML applications, with MobileBERTTINY as the example, and it varies with duty cycle (the paper's Fig. 10). The benefit comes from RRAM's non-volatility: the chip can power-gate between inferences without losing weights. RRAM's 3× bitcell density over SRAM does not fully carry over to the system level once macro peripherals are counted.

Q·Which results are measured and which are simulated?▾

The hybrid gain cell macro comparison (72%, 15%, 24%) is a Cadence Spectre circuit simulation at a 5 nm node. The ITO transistor with a 40 nm channel and the gain cell plus RRAM chip at 130 nm (shown in TEM images) are fabricated. The RRAM system energy, HGC plus RRAM savings and Illusion System results are summarized from the cited prior and companion papers, which hold the methodology.

Background terms
Memory wallThe performance and energy limit caused by moving data between compute and memory that sits on separate chips.
Miniaturization wallThe paper's term for the inability to keep increasing device density in 2D, because of physical limits and fabrication complexity.
SRAMStatic RAM: a bit held in a latch of (usually six) transistors built in the logic process. Fast and durable, but large per bit and leaky when idle (general knowledge).
DRAMDynamic RAM: a bit stored as charge on a capacitor behind one transistor. Denser than SRAM, but the charge leaks, so it must be refreshed (general knowledge).
Gain cellA memory cell of two or three transistors that stores a bit as charge on a transistor's gate and reads it through a second transistor, without a separate capacitor. Retention depends on how little the write transistor leaks.
Hybrid gain cell (HGC)A gain cell with a silicon read transistor and an oxide-semiconductor (OS) write transistor. An OS-OS gain cell uses oxide-semiconductor transistors for both.
Non-volatile memory (NVM)Memory that keeps its data without power, such as Flash, RRAM, PCM, MRAM, FeRAM and FeFET.
RRAMResistive RAM: a cell whose oxide layer is switched between high- and low-resistance states; 1T-1R means one transistor in series with one resistive element (general knowledge).
PCMPhase-change memory: a cell whose material is switched between a crystalline (low-resistance) and an amorphous (high-resistance) phase by heating (general knowledge).
MRAMMagnetoresistive RAM: a cell that stores a bit in the relative magnetization of two layers of a magnetic tunnel junction (general knowledge).
FeRAM and FeFETFerroelectric memories: the bit is the polarization direction of a ferroelectric layer, in a capacitor (FeRAM) or in a transistor's gate stack (FeFET) (general knowledge).
Endurance and retentionEndurance is the number of write cycles a cell survives; retention is how long it holds data without being rewritten.
FEOL and BEOLFront-end-of-line is the transistor layer of a chip; back-end-of-line is the stack of metal wiring layers above it, where some memories can be built (general knowledge).
Monolithic 3D integrationBuilding additional device layers (here, memory) directly on top of a finished chip's transistors in the same wafer, connected by fine vias, rather than stacking separate dies.
2.5D/3D packaging and TSVPackaging that places separate dies side by side on an interposer (2.5D) or stacks them (3D); a through-silicon via (TSV) is a vertical wire through a die that connects the stack.
HBMHigh Bandwidth Memory: stacked DRAM dies connected by TSVs and placed in the same package as the processor.
Compute-in-memory (CIM)An architecture in which arithmetic is performed inside or beside memory arrays, so stored data (such as weights) need not be moved to a separate processor.
Illusion SystemA system of multiple chips, each with its own on-chip memory and compute and no off-chip memory, mapped and power-managed so that together they behave nearly like one large "Dream Chip".
Summary of S. Liu, R. M. Radway, X. Wang, J. Kwon, C. Trippel, P. Levis, S. Mitra and H.-S. P. Wong, "Future of Memory: Massive, Diverse, Tightly Integrated with Compute – from Device to Software," IEDM 2024. Numbers are from the paper unless marked as this summary's arithmetic.