Future of Memory: Massive, Diverse, Tightly Integrated with Compute – from Device to Software
This page was written by an AI model to give a high-level overview of the paper. Please check the paper for detail: IEEE Xplore.
Future computing systems must deliver higher throughput at much lower energy. This vision paper names three challenges that meet at memory: demand, as AI/ML, big data and networked systems need far more memory; the memory wall, the time and energy spent shuttling data between compute and off-chip memory; and the miniaturization wall, the inability to keep gaining device density in 2D because of physical limits and fabrication complexity.
Software generally assumes one random-access address space with uniform latency and bandwidth, but its actual use of memory is far from uniform. Today's hierarchy (SRAM, DRAM, Flash) is already diverse, yet the paper argues it is not enough. Device research often optimizes one attribute at a time, such as a single entry in a comparison table, while software needs a particular set of attributes together. Memory also lives on separate chips for technology, cost and business reasons, behind interfaces (DDR, HBM, PCIe, CXL) that add energy, latency and bandwidth overhead. The paper notes that the memory wall is often wrongly blamed on the von Neumann architecture; it comes from how the memory hierarchy is physically built.
The paper's position is that future memory must be (1) massive, (2) diverse, and (3) tightly integrated with compute, both physically and architecturally, and that (4) the software stack must exploit that integration. It covers points 1 to 3 and leaves 4 out of scope. Rather than expecting a new device to replace existing memory, designers should start from how software uses data and match each use to a technology whose trade-offs fit it.
To do that, the paper characterizes data by read/write frequency, address predictability and data lifetime, and gives three example use cases (Type A, B, C below). Each use case turns the large space of memory technologies into bands, with a specific list of improvements each technology would need to serve it (the paper's Table III). Physically, the memories move onto or atop the compute chip through a continuum of 3D interconnects, from advanced packaging (pin pitch projected below 1 µm) down to tens of nanometers with monolithic 3D integration. For models larger than one chip's memory, the paper points to Illusion Systems: several chips, each with its own on-chip memory, and no off-chip memory at all.
The paper supports the vision with case studies, mostly drawn from the authors' prior and companion work. A hybrid gain cell (HGC) has 3.6× the density of high-current SRAM; in a 5 nm circuit simulation its memory macro had 72% smaller cell size, 15% lower read energy and 24% lower standby power at the same 2 GHz frequency. HGC monolithically integrated with RRAM saves more than 80% energy for AI/ML training and inference. On-chip RRAM's non-volatility gives up to 9× energy benefits for intermittent edge AI/ML (for example MobileBERTTINY) compared with SRAM, off-chip DRAM and NAND Flash systems. Illusion Systems, for example 8× larger than a single chip, come within 5% of the energy and execution time of an ideal single "Dream Chip".
Key takeaways
- Three challenges meet at memory: exploding demand, the memory wall (data movement cost) and the miniaturization wall (2D density stalling).
- Software uses memory in very different ways. The paper argues for matching a set of device attributes to each use case instead of improving one attribute in isolation.
- Three example use cases (A: read-mostly and predictable; B: write-heavy and short-lived; C: random reads, sequential writes) each define the improvements every technology would need (Table III).
- Tight physical integration (2.5D/3D packaging down to monolithic 3D) brings memory onto compute; Illusion Systems spread large models across chips with no off-chip memory.
- Case studies: HGC 3.6× denser than high-current SRAM, HGC plus RRAM saving over 80% energy, RRAM up to 9× energy benefit for intermittent edge AI, Illusion Systems within 5% of a Dream Chip. Software abstractions are left out of scope.
From software use cases to matched, tightly integrated memories
The device background, the use-case bands, physical integration, compute-in-memory and multi-chip systems, and the case studies.
1 · Device background for a systems reader
For logic, the field-effect transistor (FET) dominates and the paper assumes it will continue to. Memory is different: many devices store a bit using different physics, and each makes different trade-offs. A short guide (general knowledge, not from the paper):
- Charge-based: SRAM (a 6-transistor latch in the logic process, fast but large), DRAM (a capacitor per bit that leaks and must be refreshed), gain cells (a 2- or 3-transistor cell that stores charge on a transistor gate, denser than SRAM but with limited retention), and Flash (charge trapped in a transistor, dense and non-volatile but slow to write).
- Resistance- and polarization-based (emerging non-volatile memories, NVMs): RRAM (a filament in an oxide that switches resistance), PCM (a material switched between crystalline and amorphous phases), MRAM (a magnetic tunnel junction), and FeRAM/FeFET (a ferroelectric layer whose polarization stores the bit).
The paper's Table II shows the "traditional" view, each attribute rated in isolation:
| SRAM | DRAM | RRAM | MRAM | |
|---|---|---|---|---|
| Energy | Low | Med | High | High |
| Speed | High | Med | Low | Low |
| Density | Low | Med | High | High |
| Endurance | High | High | Low | Med |
Its Table I instead rates a wider set (adding Flash, two kinds of gain cell, PCM, FeFET, FeRAM and multi-layer 3D versions) on read and write energy, standby power, read and write latency, endurance, retention, density, and whether they can be built in one or many layers on a logic chip. Two device trends drive the paper's integration argument. First, 2D density is running out: the paper's Fig. 8(a) shows high-density SRAM rising from about 2 Mb/mm² at 65 nm to about 50 Mb/mm² at 3 nm, with little gain over the last few nodes (read from the plot). Second, many of the newer devices can be fabricated in layers above the transistors, so capacity can grow vertically. Density in Table I is "equivalent density after considering multiple layers."
2 · The problem: three walls and a uniform abstraction
- Demand. AI/ML, big data and networked systems need massive memory, much of it off-chip today.
- Memory wall. Significant time and energy go to moving data between compute and memory chips. Without a change, the paper warns, computing's energy demand could far exceed sustainable energy production (citing published projections).
- Miniaturization wall. Device density in 2D is limited by physics and fabrication complexity.
- A uniform view of memory. Software assumes uniform latency and bandwidth, while its accesses are far from uniform. Device work that optimizes single table entries misses what each use actually needs.
3 · From software use cases to memory bands
The paper describes data by three attributes: read/write access frequency, read/write address predictability, and data lifetime (from creation to destruction). Bulk storage is out of scope. Three example use cases, placed by their ratio of reads to writes and access pattern in the paper's Fig. 3:
| Use case | Behavior | Examples and scale | Opportunity |
|---|---|---|---|
| Type A | Frequent reads, infrequent writes, predictable (often sequential) accesses | Inference weights; instruction caches. A Zsim simulation shows instruction cache misses below 5×10-6, with thousands of instructions executed billions of times per run | Trade write energy, latency and endurance for better read energy, delay and retention; write buffering and prefetching help. Some write capability is still needed (e.g., hourly LLM model updates) |
| Type B | Frequent writes, few reads per write, short data lifetime | Streaming I/O, AI/ML activations, analytics. Blocks of tens of kB read after tens of µs, MBs after ms, and 1 to 128 MB after seconds | Short lifetimes let retention be traded for speed, density or energy (gain cells, MRAM, FeRAM) |
| Type C | Random (e.g., Zipf-like) reads, sequential writes across contiguous blocks | File system buffers; defragmentation and live data write logs | Read delay can be hidden with multithreading, but low read energy is critical; on-chip gain cells have far lower access energy than off-chip DRAM |
Each use case narrows the space of technologies into a band and a list of targets. The paper's Table III, based on state-of-the-art macro demonstrations, gives the improvement each technology needs per use case. A few rows:
| SRAM | DRAM | Hybrid gain cell | RRAM | MRAM | |
|---|---|---|---|---|---|
| A | Standby power | Read energy, speed, standby power | Capacity | Write and read energy, endurance | Write and read energy |
| B | Density | Retention | Capacity | Endurance, write energy | Write energy |
| C | Capacity | Read and write energy, speed | Capacity, write speed | All | All |
- Optimize combinations, not single numbers. Improving MRAM or RRAM endurance alone, without improving write energy, may not help an energy-constrained Type A system, because using the extra write cycles costs a lot of energy. The paper's Fig. 5(a) splits the write energy and endurance space into "ideal", "domain-specific" and "not of interest" (high endurance with high write energy).
- Choose a point on the trade-off curve from the application. Gain cells trade retention against speed along a curve set by the transistor's on/off current ratio (Fig. 5(b)). Which threshold voltage to pick should follow the application, and design knobs (voltage), better devices (sharper on/off transition) or circuits (hybrid gain cells) can shift the curve toward the ideal corner.
- Integration can matter more than device type. In Table III, 3D V-Cache (SRAM stacked on a processor die, general knowledge) and DRAM fall in the same band, while two kinds of gain cell differ.
4 · Tight physical integration
High-bandwidth memory access needs a high density of connections between memory and compute. The paper's Table IV lists where each memory can sit: in the logic process itself (SRAM), in a package (DRAM DIMMs, Flash), stacked in a package (HBM), or monolithically 3D integrated above the silicon transistors.
The paper's Fig. 9 shows a concrete example: TEM cross-sections of a gain cell and an RRAM monolithically integrated with silicon FETs on the same 130 nm chip, combining RRAM's non-volatility with the gain cell's high endurance, connected by dense vertical interconnects. The paper's point is that these connection densities make it possible to re-architect the system to use far more bandwidth.
5 · Compute-in-memory and its limits
Compute-in-memory (CIM) performs computation inside memory arrays to cut data movement. The paper weighs two kinds:
| Analog CIM | Digital CIM | |
|---|---|---|
| How it computes | Matrix-vector multiplication by Ohm's and Kirchhoff's laws: the summed current on a bitline is the result | A small digital unit next to each bitcell; results summed by an adder tree |
| Accuracy | Degraded by NVM device variation, programming non-linearity, limited on/off ratio, drift and IR drop; even chip-in-the-loop fine-tuning fails to keep software accuracy for RRAM CIM | Full bit accuracy |
| Energy | Below 10 fJ per 4-bit operation at the array level, but above 100 fJ per 8-bit operation: each extra bit needs 4× the capacitance and energy in the ADC | Simple logic must fit the cell pitch (e.g., 6T SRAM plus a few transistors), so it may need bit-serial ALUs and longer runtime |
| Other limits | Costly DACs and ADCs; small kernels underuse the array; large kernels need post-ADC digital accumulation | Programmability across workloads |
6 · Illusion Systems: many chips that act like one
- The ideal. A "Dream Chip" holds the whole model in massive on-chip memory next to compute, accessible quickly at low energy.
- The practical version. An Illusion System spreads AI/ML execution across N chips, each with its own on-chip memory and compute and mechanisms for quick wakeup and shutdown, connected by a sparsely used inter-chip network. There is no off-chip memory.
- What makes it work. Special mappings of the model to the chips minimize inter-chip traffic, co-optimized with per-chip memory size, heterogeneous inter- and intra-chip interconnects, and fine-grained power management of idle chips.
- Result from prior work. Digital accelerator systems, for example 8× larger than single chips, reach system-level energy and execution time within 5% of the corresponding Dream Chip. The paper distinguishes this from traditional parallel processing.
7 · Case studies and scope
| Case study | Result | Basis |
|---|---|---|
| Hybrid gain cell (Si read transistor, oxide-semiconductor write transistor) vs SRAM | 3× the density of high-density SRAM, scalable to FinFET nodes; 3.6× density vs high-current SRAM. Macro: 72% smaller cell size, 15% lower read energy, 24% lower standby power at the same 2 GHz | Cadence Spectre circuit simulation at 5 nm with the same peripheral circuits and array architecture; an ITO transistor with 40 nm channel demonstrated experimentally |
| HGC monolithically integrated with RRAM | More than 80% energy savings for AI/ML training and inference | Companion IEDM 2024 work |
| On-chip RRAM vs SRAM-only, SRAM plus off-chip Flash, and off-chip low-power DRAM | Up to 9× lower system energy for intermittent edge AI/ML (e.g., MobileBERTTINY); off-chip DRAM costs 1.6× or more energy vs RRAM-based systems | Prior work; system energy plotted against duty cycle (Fig. 10) |
The RRAM result comes from non-volatility, not density. RRAM bitcells are 3× denser than 6T SRAM (0.042 vs 0.127 µm² in TSMC N28), but foundry RRAM macros, with their peripherals and controllers, do not deliver the full density benefit at the system level. The energy win is that an RRAM chip can power-gate between inferences without losing its weights. An SRAM chip that idles long enough is better off turning off and re-fetching weights from NAND Flash; low-leakage 8T SRAM moves that crossover to lower duty cycles but costs area.
FAQ
Q·Is this a new device or a new system?▾
Neither, on its own. It is a four-page vision paper that argues for a direction (massive, diverse memory tightly integrated with compute) and backs it with case studies from the authors' prior and companion work on gain cells, RRAM integration and multi-chip systems.
Q·Isn't the memory wall a consequence of the von Neumann architecture?▾
The paper says no, and calls that attribution often incorrect. Von Neumann wanted an indefinitely large memory with every word immediately available, and was "forced" into a hierarchy because it could not be built. The wall comes from implementing that hierarchy as separate chips, for technological, cost and business reasons. The paper's prescription applies to von Neumann and non-von Neumann architectures alike.
Q·Why not wait for one universal memory that is better at everything?▾
The paper argues against expecting new devices to replace existing memory. Each technology trades attributes against one another, and domain-specific matching gives high efficiency. Its Fig. 4 shows that the needs of different use cases can be subsets of one another, can overlap, or can be disjoint, so no single point satisfies all of them.
Q·Why does write energy matter if endurance improves?▾
For a read-mostly (Type A) system on a tight energy budget, more endurance cycles are only useful if writing is affordable. The paper's example: improving MRAM or RRAM endurance without improving write energy may not be enough, because using the gained cycles can cost a lot of energy. High endurance with high write energy is the "not of interest" corner of its Fig. 5(a).
Q·How short is "short data lifetime" for Type B data?▾
The paper cites blocks of tens of kB read back within tens of µs, MBs within ms, and 1 to 128 MB within seconds, from streaming I/O, AI/ML activations and analytics. Because such data does not need to survive long, a memory can give up retention for speed, density or energy, as gain cells, MRAM and FeRAM can.
Q·How is this related to CXL memory tiering or HBM?▾
The paper places both on its integration spectrum. HBM stacks DRAM using 5 to 50 µm through-silicon vias for higher bandwidth and lower energy. CXL, PCIe, DDR and similar protocols are needed because off-chip devices come from different manufacturers, but they add energy, latency and bandwidth overhead. The paper's direction goes further, to finer-pitch packaging and monolithic 3D integration on the compute chip.
Q·Does compute-in-memory solve the memory wall?▾
Not by itself, in the paper's view. Analog CIM loses accuracy from device non-idealities and loses energy efficiency quickly as precision rises (below 10 fJ per 4-bit operation but above 100 fJ per 8-bit operation at the array level). Digital CIM keeps full accuracy but needs simple logic that fits the cell pitch. Both struggle with operations beyond matrix-vector multiplication, and if the model exceeds on-chip memory, off-chip accesses bring the wall back.
Q·What does software have to do?▾
The paper names this as point (4) of its vision and leaves it out of scope. Its conclusion states the requirement: memory should move from a uniform, random-access view to a collection of different memories, and their trade-offs (for example endurance vs retention, energy vs latency) should be exposed to software through appropriate abstractions. Table IV's caption notes that today's interface protocols are managed by software, from device drivers to the OS, with minimal hardware support such as memory management units.
Q·Is the 9× RRAM energy benefit general?▾
It is stated for infrequent and intermittent edge AI/ML applications, with MobileBERTTINY as the example, and it varies with duty cycle (the paper's Fig. 10). The benefit comes from RRAM's non-volatility: the chip can power-gate between inferences without losing weights. RRAM's 3× bitcell density over SRAM does not fully carry over to the system level once macro peripherals are counted.
Q·Which results are measured and which are simulated?▾
The hybrid gain cell macro comparison (72%, 15%, 24%) is a Cadence Spectre circuit simulation at a 5 nm node. The ITO transistor with a 40 nm channel and the gain cell plus RRAM chip at 130 nm (shown in TEM images) are fabricated. The RRAM system energy, HGC plus RRAM savings and Illusion System results are summarized from the cited prior and companion papers, which hold the methodology.