Towards Memory Specialization: A Case for Long-Term and Short-Term RAM
This page was written by an AI model to give a high-level overview of the paper. Please check the paper for detail: ACM Digital Library.
SRAM and DRAM have stopped getting cheaper per byte. DRAM cost trends have stagnated over the past 15 years, SRAM cells have barely shrunk since the 7 nm process node, and DRAM now accounts for over 50% of server hardware cost. Packaging advances such as HBM (High Bandwidth Memory) and LPDDR buy bandwidth or lower power, but they do not change the cost of a memory cell, so memory has become the main cost, power and performance bottleneck of modern systems.
Emerging memory technologies scale better. Gain cell embedded DRAM reaches 2 to 3 times SRAM's density at similar process nodes, and 3D vertical RRAM (V-RRAM) can reach up to 10 times the density of HBM4 at the same node. None of them is a drop-in replacement, though: they read and write asymmetrically and trade retention time against write endurance. RRAM's limited endurance and high write energy rule it out for frequently overwritten data; gain cells need periodic refresh. Each device is good for some data and bad for other data, and exposing every device's quirks directly to software would be unmanageable.
The paper's proposal is to organize emerging devices into just two new memory classes, defined by the data they serve rather than by device physics. Long-term RAM (LtRAM) is for read-heavy data with long lifetimes (minutes or longer): it accepts higher write latency and energy and lower endurance in exchange for dense, low-energy reads and long retention. Short-term RAM (StRAM) is for transient data with sub-second lifetimes: it needs symmetric reads and writes and very high endurance, but not long retention. Together with SRAM, DRAM and NAND flash they form a five-class memory system, and the operating system manages classes, not individual technologies.
The paper then maps workloads onto the classes. Model weights during inference, code pages, search indices and read-mostly caches such as Redis and Memcached fit LtRAM; neural network activations, gradients, message queues and short-lived cache data fit StRAM. Using them well breaks the assumption that memory forms a hierarchy ordered by distance from the processor, so the paper lays out the systems problems that follow: new OS abstractions, placement policies that reason about lifetime and write cost instead of hotness, memory consistency and coherence when StRAM can lose data or LtRAM writes lag, and power and thermal limits in dense, tightly integrated systems.
This is a position paper. It has no prototype or measurements of its own; its contributions are the two class definitions, a survey of device technologies that could implement them, and a list of research questions. The numbers it cites (scaling trends, densities, rack power) come from prior work.
Key takeaways
- SRAM and DRAM have stopped scaling in cost per byte; DRAM is over 50% of server hardware cost, and packaging (HBM, LPDDR) does not fix cell cost.
- Emerging memories (gain cell eDRAM, RRAM, MRAM, FeRAM) are denser but not drop-in replacements: reads and writes are asymmetric, and retention trades against endurance.
- The proposal: two new classes defined by data, not device. LtRAM for read-heavy, long-lived data; StRAM for transient, frequently accessed data. With SRAM, DRAM and NAND flash, a five-class memory system.
- Workloads split naturally: inference weights, code and indices toward LtRAM; activations, gradients, queues and short-lived cache data toward StRAM.
- A position paper with no evaluation. Its research agenda: OS abstractions, lifetime-aware placement and profiling, consistency and coherence for StRAM and LtRAM, and power-aware integration.
From a memory hierarchy to specialized memory classes
Why SRAM and DRAM stalled, what emerging devices offer, the two proposed classes, and the systems questions they raise.
1 · Why SRAM and DRAM stopped getting cheaper
Today's systems use three kinds of memory: SRAM caches on the compute chip, off-chip DRAM as main memory, and NAND flash (or other non-volatile storage) for permanent data. The two volatile ones are hitting physical limits:
- SRAM uses six transistors per bit, is fast, holds its state while powered and integrates easily with logic. Its cell size has stopped shrinking significantly since the 7 nm node (the paper's Figure 1b), because balancing transistor sizing and threshold voltages gets harder at small nodes and low voltages.
- DRAM uses one transistor and one capacitor per bit. The capacitors are narrow and tall, which makes integration with high-performance logic impractical. Charge leaks and reads are destructive, so DRAM must be refreshed (every 32 ms for DDR5), which draws power even when idle. Smaller cells are still feasible, but capacitor manufacturing prevents lower per-cell cost: DRAM cost trends have stagnated over the past 15 years (Figure 1a).
- Packaging helps capacity and bandwidth, not cost per byte. LPDDR reaches 10 Gbps per pin versus 5.6 for conventional DDR but must be soldered to the board. HBM stacks dies on a silicon interposer for much higher bandwidth, but it is substantially more expensive than DDR DRAM, terabyte on-package configurations are not practical, and density gains may stop beyond 20 dies.
2 · What emerging memories offer, and why they are not drop-in
Gain cell embedded DRAM, ferroelectric RAM (FeRAM), magnetoresistive RAM (MRAM) and resistive RAM (RRAM) have different scaling roadmaps. The paper's density examples:
| Technology | Density claim in the paper | Constraint that blocks drop-in use |
|---|---|---|
| Gain cell eDRAM | 2 to 3× SRAM's density at similar process nodes; all-transistor cell allows on-chip integration | Needs periodic refresh: it stores data on transistor capacitance instead of SRAM's feedback loop |
| Planar RRAM | Up to 2× DRAM's density | Limited endurance and high write energy rule out frequently overwritten data |
| 3D V-RRAM | Up to 10× HBM4's density at the same process node; stacks up to 64 layers versus HBM's 16 dies; may keep scaling where DRAM struggles below 11 nm | Same as RRAM |
Two properties decide which data a device can hold. First, for a device with limited endurance, the data's write rate over the device's expected service life must fit within the cell endurance. Second, matching the retention of the cells to the lifetime of the data helps reduce energy consumption.
3 · Two new classes instead of many devices
Specialized hardware cannot stay invisible to software, but making the OS reason about every device is unmanageable. The paper groups emerging devices into two classes that capture most device and workload trade-offs without excessive segmentation:
| StRAM (short-term RAM) | LtRAM (long-term RAM) | |
|---|---|---|
| Optimized for | Transient data that is frequently accessed, or read once and discarded | Persistent, read-heavy data |
| Data lifetime | Short, sub-second | Minutes and longer |
| Reads and writes | Symmetric performance; low write energy | Read performance and read energy first; accepts higher write latency and energy |
| Endurance | Very high or unlimited | Lower, accepted as a trade-off |
| Retention | Not necessarily high | Long |
| Candidate devices | Gain cell eDRAM; 1T1C FeRAM if its endurance can be brought high enough | RRAM, MRAM, FeRAM, managed-retention DRAM variants |
| Where it sits | On-die to near-compute, depending on device and thermal constraints | On-die, 3D-stacked packages or off-chip modules |
LtRAM builds on managed-retention memory (MRM), a prior proposal to repurpose storage-class memories for workloads such as LLM inference by trading retention time and write performance for improved endurance and lower I/O energy. The key insight the paper gives for LtRAM is that long data lifetimes and read-heavy access allow optimizations unsuitable for general-purpose memory, and that today read-intensive data competes for the same DRAM as frequently modified data.
4 · The five-class memory system
The new classes are added to, not substituted for, the existing three. The paper's Table 1:
| Class | Strengths | Weaknesses | Uses |
|---|---|---|---|
| SRAM | Low read/write latency, long retention, low static power | Low density | Fast read/write caches |
| DRAM | Very dense | Off-chip only, read/write energy, high static power, refresh overhead, destructive reads | Large, random-access read/write data |
| NAND flash | Extremely dense, low static power | Off-chip only, low bandwidth, limited endurance, expensive erases, very high latency | Storage, rarely accessed data |
| StRAM | Dense, low write energy, low static power | Short retention, refresh overhead | Fast read/write caches, write-and-read scratchpads |
| LtRAM | Very dense, low read energy, low static power | High write energy, high write latency, limited endurance | Read-mostly data, read-mostly caches |
In the table, "power" is continual leakage and "energy" is the cost of one active read or write.
5 · Which data goes where
The paper walks through three application domains and assigns their data by lifetime and write pattern.
- Server workloads. In-memory stores, CDNs and DNS servers are read-mostly with variable lifetimes and could adapt their caching to a mix of LtRAM and StRAM. Logging and telemetry produce write-once data with short-to-medium lifetimes, which StRAM can buffer before archival to flash. Code pages are read-intensive with very long lifetimes, yet today share DRAM with frequently modified data.
- Machine learning. Inference weights are typically immutable and read in large blocks; the paper notes that HBM and DRAM are over-provisioned in write performance for them. Activations are discarded right after use. Training mixes read-heavy and write-heavy access to weights with write-heavy access to gradients and optimizer states.
- Processor cores. Most L1/L2/L3 cache data lives briefly, so SRAM's indefinite retention is over-provisioned for it; StRAM could give comparable performance at higher density and lower static power.
6 · OS abstractions and data placement
Traditional memory hides device differences behind a flat address space, and placement policies assume a hierarchy where closer means faster. Specialized classes break that assumption: SRAM and StRAM could both be on-chip scratchpads chosen by application need, and LtRAM could be off-chip yet accessed directly for read-heavy, long-lived data despite lower bandwidth. The paper's questions:
- What to expose. How should the OS expose retention, endurance limits and read/write asymmetry while keeping memory programmable? The paper points to NAND flash exposing internal organization and GPUs exposing explicit SRAM instructions as evidence that low-level control pays off, at the cost of code complexity.
- Placement without a hierarchy. Linux NUMA nodes and tiering policies rank memory by access frequency, latency or memory-level parallelism. LtRAM's large read/write gap does not fit that ranking, and it interacts with the fact that reads load into the cache synchronously while writes reach memory asynchronously, on eviction.
- Who decides. Databases and ML frameworks know their buffer pools and dataflow and can place data themselves. Other applications can rely on the OS (hot-cold policies, placing pages mapped read-only in LtRAM, or inference from program behavior). The OS must support transparent management, hints and explicit control.
- Profiling. Page-level hot-cold and bandwidth tracking cannot express lifetime, endurance, caching or read/write trade-offs. Lightweight fine-grained profiling, and a way to pass its results down (compiler annotations or special instructions, as in the Expressive Memory interface), are open.
7 · Memory consistency and coherence
Memory consistency models (MCMs) define which values a load may return, and coherence protocols make all threads agree on the order of writes to one location. Both assume volatile memories with high retention times, low write latency and energy, and high endurance for all data. StRAM and LtRAM each break part of that. The paper's litmus tests (all values start at 0):
| Paper figure | Thread 1 | Thread 2 | What goes wrong |
|---|---|---|---|
| Figure 3 (StRAM) | data = 1 | data is corrupted to 42; read data returns 42 | An "out-of-thin-air" value that was never written, which most modern MCMs forbid |
| Figure 4 (StRAM) | data = 1 | read data returns 1; data is corrupted to 0; read data returns 0 | Threads disagree on the order of writes to one location: a coherence violation |
| Figure 5 (LtRAM) | data = 1; flag = 1 | read flag returns 1; read data returns 0 | Stale data despite an updated flag: forbidden by sequential consistency and total store order, allowed by ARMv8 and RISC-V's weak order (which provide fences) |
- StRAM fixes. Cache controllers can refresh lines before retention runs out or evict them early. Software-initiated invalidations, with or without writebacks, could lower hardware cost. Coherence metadata stored in StRAM faces the same corruption risk.
- LtRAM fixes. Invalidation-based coherence writes cache-line metadata often, which is costly on LtRAM. Update-based protocols could suit read-mostly, few-producer, many-consumer data; alternatively, hardware could keep metadata in write-optimized StRAM and data in read-optimized LtRAM.
- Mixed caches. A single cache level built from SRAM, StRAM and LtRAM parts behaves like a multibanked memory that one thread can access concurrently, which is how the Figure 5 outcome can arise.
8 · Power and thermal limits
The paper argues that power is becoming a first-order design goal and that memory is a large part of it.
Memory draws power through static leakage, refresh and data movement, and data movement energy, which grows superlinearly with interconnect distance, typically dominates. Matching cell retention to data lifetime cuts static power by avoiding unneeded SRAM leakage and DRAM refresh. Specialization therefore has to be co-optimized with interconnect, packaging and data assignment, and dense, tightly integrated arrays may need new cooling, such as microfluidics.
9 · What the paper leaves open
- No evaluation. The paper defines classes and maps workloads to them; it does not build or measure an LtRAM or StRAM system.
- Future devices. It focuses on technologies available today. Monolithic 3D DRAM (multiple cell layers on one wafer) is one emerging option; it needs at least 64 layers to match 2D density. Commercial viability, particularly cost, is uncertain.
- Integration. Unlike DRAM and NAND, which must be off-chip because of process incompatibilities, StRAM and LtRAM can be placed anywhere from on-chip to 2.5D chiplets to 3D stacks, which the paper sees as a chance to co-design cells, interconnect and placement.
- The research agenda in the conclusion: fine-grained access-pattern profiling, lifetime-aware placement policies, memory system abstractions for heterogeneous devices, and coherence protocols for specialized memories, with collaboration from materials science and device physics through architecture and systems.
FAQ
Q·What do "long-term" and "short-term" refer to?▾
Data lifetime, the time data must stay in memory. LtRAM holds data that lives for minutes or longer and is mostly read; StRAM holds data that lives for under a second, such as activations or queue entries. "Memory specialization" in the title means building memory from several device types, each matched to the write patterns and lifetimes of part of a workload.
Q·Why two new classes and not one per technology?▾
The paper's concern is software complexity. Specialized hardware may differ from machine to machine, and operating systems, middleware and applications must keep working as devices change. Two classes give the OS a small set of abstractions while letting each class be built from different devices. The paper acknowledges that a finer continuum of retention and access characteristics is conceivable, but argues that two classes capture most device and workload trade-offs without excessive segmentation.
Q·Does LtRAM have to be non-volatile?▾
The paper does not require it. It describes LtRAM data as "persistent" in the sense of long-lived, and its candidate devices include non-volatile memories (RRAM, MRAM, FeRAM) and managed-retention DRAM variants. What defines the class is long retention, cheap reads and tolerance for costly writes.
Q·How is this different from memory tiering?▾
Tiering policies, including Linux NUMA placement and CXL memory tiering, assume memories can be ranked: a page moves to a slower tier based on access frequency, latency or memory-level parallelism. The paper argues that LtRAM and StRAM break this ranking. LtRAM is fast to read and slow to write, so whether a page fits depends on its write pattern and lifetime, not on how hot it is. Placement must reason about those properties directly, which today's page-level hot-cold tracking cannot express.
Q·How does LtRAM relate to managed-retention memory?▾
LtRAM builds on it. Managed-retention memory (MRM), from co-authors at Microsoft Research, proposes repurposing storage-class memories for workloads such as LLM inference by trading retention time and write performance for improved endurance and lower I/O energy. It found DRAM too slow for that workload and HBM over-provisioned in write performance and endurance. LtRAM generalizes the idea into a memory class that several technologies could implement.
Q·Who decides what goes into LtRAM or StRAM?▾
The paper wants a spectrum. Databases and ML frameworks know their buffer pools and dataflow and could place data explicitly. Other applications would rely on the OS, for example hot-cold policies, placing pages mapped read-only in LtRAM, or inferring behavior from the program. Application hints, compiler annotations or special instructions sit in between. Building the profiling and hint interfaces is an open problem the paper names.
Q·Can StRAM's short retention break program correctness?▾
Yes, if nothing manages it. If a value decays before it is used, a read can return a value that was never written (an "out-of-thin-air" result, forbidden by most consistency models), or two reads of the same location can disagree on the order of writes (a coherence violation). The paper's remedies: cache controllers refresh lines before retention expires or evict them early, or software issues invalidations. Coherence metadata kept in StRAM needs the same protection.
Q·Why would LtRAM affect cache coherence?▾
Invalidation-based coherence protocols write cache-line metadata frequently, and writes are what LtRAM is bad at. The paper suggests update-based protocols for read-mostly data with few producers and many consumers, or keeping metadata in write-optimized StRAM while data stays in read-optimized LtRAM. It also notes that a cache level mixing SRAM, StRAM and LtRAM parts can let a reader see an updated flag but stale data, an outcome some consistency models (sequential consistency, total store order) forbid and others (ARMv8, RISC-V's weak order) allow.
Q·Where would these memories physically sit?▾
Anywhere from on-die to off-chip. StRAM ranges from on-die to near-compute placement depending on device and thermal constraints. LtRAM can be on-die, in 3D-stacked packages or in off-chip modules. The paper sees this flexibility as an advantage over DRAM and NAND, which must be off-chip because their processes are incompatible with logic, and gives the example of LtRAM placed off-chip yet accessed directly for read-heavy data.
Q·Why does power come into it?▾
Rack power is climbing: the paper cites about 20 kW for typical CPU-based servers, up to 120 kW for current AI racks and 600 kW expected for next-generation AI racks in late 2027. In the warehouse-scale breakdown it cites, DRAM is 30% of peak power. Matching retention to data lifetime reduces SRAM leakage and DRAM refresh, but data movement energy typically dominates memory power, so specialization must be co-designed with interconnect and packaging.
Q·Does the paper evaluate LtRAM or StRAM?▾
No. It is a position paper. It defines the two classes, surveys candidate devices with numbers from prior work (scaling trends, densities, power), maps workloads to classes, and lists research questions. There is no prototype, simulation or application measurement.
Q·Where did the LtRAM and StRAM split come from?▾
The acknowledgments credit Shridhar Mukund of EMD with first suggesting LtRAM and StRAM as a useful decomposition, in the context of ML inference.