GainSight: A Unified Framework for Data Lifetime Profiling and Heterogeneous Memory Composition
This page was written by an AI model to give a high-level overview of the paper. Please check the paper for detail: arXiv.
AI accelerators want more on-chip memory, and on-chip memory today means SRAM. SRAM has stopped shrinking: the paper cites a cell size that has plateaued at 0.021 µm² across the 5 nm to 3 nm process nodes, while language model parameter counts grow almost 400× every two years and accelerator memory capacity only doubles. At the same time, much of the data an accelerator keeps on chip is short-lived. In LLM inference, attention score matrices exist only until the softmax consumes them, decoding generates candidate tokens and keeps one, and KV-cache data is staged into SRAM just before the tensor cores use it; the paper estimates that many such structures persist for mere microseconds. An SRAM cell holds its value for as long as it is powered, so for this data it pays, in area and leakage, for retention nobody uses.
Short-term RAM (StRAM) trades retention for density. Gain-cell RAM (GCRAM) and embedded DRAM (eDRAM) store each bit as charge on a small node that slowly leaks away, so a cell holds its value only for a retention time, from about a microsecond to milliseconds depending on the technology. They are 2 to 4× denser than SRAM, with lower leakage. Normally a leaky memory needs refresh (reading and rewriting each cell before it decays), which costs energy. The key observation is that if data dies before the charge does, no refresh is needed. Exploiting this requires knowing, access by access, how long each piece of data actually lives. Existing profilers report kernel-level or aggregated cache statistics, and earlier accelerators with eDRAM or gain cells estimated lifetimes from static dataflow heuristics.
GainSight is a profiling and design-exploration framework that makes data lifetime (the time from a value's first write to its last read before it is overwritten or invalidated) a first-class design metric. Retargetable backends run workloads on modified cycle-accurate simulators (Accel-Sim for GPU L1 and L2 caches, SCALE-Sim for systolic-array scratchpads) and emit a common trace of every on-chip memory access. An architecture-agnostic frontend turns the trace into lifetimes, refresh counts, live capacity and active energy for each cell type in a library of 5 nm models (SRAM, silicon GCRAM, hybrid oxide-silicon GCRAM, oxide eDRAM). It then builds a heterogeneous composition by assigning each access to the cell type with the shortest retention that still exceeds the data's lifetime, and reports the capacity, area and energy of the resulting mix.
On MLPerf Inference and PolyBench workloads, 64.3% of GPU L1 accesses and 18.4% of L2 accesses have lifetimes within Si-GCRAM's 1 µs retention, and 97.9% and 52.0% fit Hybrid-GCRAM's 10 µs. On a 256×256 systolic array running ResNet-50, at least 79.01% of scratchpad accesses live under 1 µs. The generated compositions reduce active energy by an average of 1.89× for GPU L1 caches and 1.3× for L2 caches, with about 3× smaller memory arrays (up to 3.1× energy and 4.3× area across workloads), and by about 3× in energy and 2.4× in area for the systolic-array scratchpads. All results are simulated, against uniform SRAM, from device models rather than fabricated arrays. GainSight is open source.
Key takeaways
- SRAM cell area has stopped scaling, and much accelerator data lives only microseconds, so SRAM's indefinite retention is overprovisioned for it.
- Gain-cell RAM and eDRAM are 2 to 4× denser than SRAM but lose data after a retention time. Data that dies sooner never needs a refresh.
- GainSight measures per-access data lifetimes with retargetable simulator backends (GPU caches, systolic-array scratchpads) and correlates them with 5 nm cell models in an architecture-agnostic frontend.
- Composition rule: each access goes to the cell type with the shortest retention that still exceeds its lifetime; the frontend sizes each type and projects area and active energy.
- Simulated against uniform SRAM: GPU L1 caches average 1.89× lower active energy and L2 caches 1.3×, with about 3× smaller arrays (up to 3.1× and 4.3×); systolic scratchpads for ResNet-50 save about 3× energy and 2.4× area.
From memory access traces to a heterogeneous on-chip memory
The cells, the lifetime metric, GainSight's backends and frontend, and what the two case studies found.
1 · Why uniform SRAM is a poor fit
Accelerators keep weights, activations and intermediate results in on-chip memory to avoid slow, energy-hungry off-chip accesses. Two trends squeeze that memory. Capacity demand grows fast: the paper cites language model parameter counts growing almost 400× every two years while accelerator memory capacity only doubles. Supply stalls: SRAM cell area has plateaued at 0.021 µm² from 5 nm to 3 nm, and even the 2 nm gate-all-around process struggles to restore the earlier trend. On a reticle-limited die, every extra square millimeter of SRAM is taken from compute.
The second half of the argument is about what the data needs. Data lifetime is highly uneven within and across workloads, and a large share of AI data is transient. A memory whose cells hold data forever treats every byte as if it might be needed indefinitely. The paper's phrase for this is that SRAM is "overprovisioned in retention capability."
2 · The memory cells: SRAM, gain cells and eDRAM
For a systems reader, the difference is how each cell stores a bit (general knowledge, consistent with the paper's Section 2.1). An SRAM cell is a pair of cross-coupled inverters that actively hold each other's value, so it never decays while powered but needs six transistors. A gain cell stores the bit as charge on a tiny storage node (the gate of a read transistor); a separate write transistor puts the charge there, and the read transistor amplifies it into a bitline current, so a read does not destroy the value. An eDRAM cell stores the charge on a capacitor behind one access transistor, like commodity DRAM but on the logic die. In both, the stored charge leaks, and the cell's retention time is how long it stays readable. Data kept longer than that must be refreshed (read and rewritten), which costs energy and access slots.
| Cell | Structure | Retention | What the paper says about it |
|---|---|---|---|
| 6T SRAM | 6 transistors, cross-coupled inverters | indefinite while powered | Fast reads and writes; area scaling has stalled; high static power |
| Si-GCRAM (silicon gain cell) | 2 to 3 silicon transistors (the library uses a 3T cell) | µs-scale; 1 µs in the 5 nm case studies | 2 to 4× denser and lower leakage than SRAM, SRAM-comparable latency, standard FinFET process |
| Hybrid-GCRAM | 2T: an oxide (indium tin oxide, ITO) write transistor stacked above a silicon read transistor | longer than Si-GCRAM; 10 µs in the case studies | Denser through 3D stacking in the back-end-of-line layers; retention falls as write frequency rises |
| Oxide eDRAM | 1 transistor and 1 capacitor (1T1C), built in the back-end-of-line | ms-scale | High density; higher active energy than GCRAM because reads share charge with the bitline |
Moving down the StRAM rows, retention gets longer and active energy per access gets higher. That ordering is what the composition step later exploits.
3 · Data lifetime, and when refresh disappears
The paper's Definition 2.1: the lifetime of a data structure at a given memory address is the time between its first write and last read before being overwritten or invalidated. In a scratchpad, that runs from a store to the final read before the next overwrite. In a cache, it runs from a store or a miss fill to the last hit before the line is replaced or written again.
The rule that follows is simple. If a value's lifetime is shorter than the cell's retention, the cell never needs a refresh for it. If the lifetime is longer, the value needs either refresh or a cell with longer retention. The paper's Figure 3 shows this for Llama-3.2-11B operations on a GPU backend: GEMM data lives under Si-GCRAM's retention, tensor-transpose data fits Hybrid-GCRAM, and normalization data outlives both and needs eDRAM or SRAM. The same figure shows Hybrid-GCRAM's retention dropping at high write frequencies, which is why the device models are frequency-dependent.
4 · The framework at a glance
- Capture fine-grained on-chip memory access traces for arbitrary workloads on a retargetable hardware backend.
- Extract lifetime statistics, read and write frequencies, and capacity use from the trace.
- Correlate lifetimes with each cell type's retention and other properties to project array area and active energy.
- Compose a heterogeneous StRAM mix that minimizes energy through refresh-free operation.
- Visualize the distributions so designers can explore them across workloads and kernels.
5 · The backends: GPU and systolic array
Because lifetimes depend on dataflow and memory organization, the same workload can look very different on two architectures, so GainSight uses one backend per architecture and simulation instead of hardware counters.
- GPU backend. CUDA workloads are traced with an NVBit-based tool to capture each kernel's SASS assembly. Representative kernels are then chosen offline with Principal Kernel Analysis (PKA), and simulated in GPGPU-Sim 4.2.1 through a modified Accel-Sim 1.3.0. The modification inserts logging at the interfaces between the streaming multiprocessors (SMs) and the L1 and L2 caches, recording address, access type, time and hit or miss for every cache operation.
- Why sampling. Accel-Sim runs 6 to 7 orders of magnitude slower than native GPU hardware. Deep learning repeats similar layers and identical operations across batches, so a few representative kernels stand in for many.
- Systolic-array backend. An extended SCALE-Sim models the three scratchpad buffers of a systolic array (input feature map or ifmap, weights or filter, output feature map or ofmap), with configurable buffer sizes, array dimensions and dataflow (input, weight or output stationary).
- Bring your own backend. Any simulator or physical accelerator that emits timestamps, address ranges and read/write types in the trace format can plug in. The frontend treats each distinct memory (a cache level, a scratchpad buffer) as an independent subpartition.
The paper's Table 4 measures what PKA sampling costs in accuracy on three workloads:
| Workload | Kernels simulated | Speedup | Lifetime error (MAE) | Write-frequency error (MAE) |
|---|---|---|---|---|
| bert-base-uncased | 8.37% | 10.89× | 0.36 µs | 11.59 MHz |
| llama-3-8b | 0.15% | 412.76× | 3.76 µs | 21.39 MHz |
| resnet-50 | 10.31% | 60.97× | 3.73 µs | 10.09 MHz |
MAE is mean absolute error against the unsampled run. The paper describes the errors in these memory metrics as bounded and sufficient for memory-system analysis.
6 · The frontend: device library, analysis, composition
Device library. All four cells are characterized for TSMC's N5 (5 nm) node. SRAM numbers come from TSMC publications; Si-GCRAM from fabricated arrays (Giterman et al., Pentecost et al.) scaled to 5 nm; Hybrid-GCRAM from post-layout simulations of 256×256 arrays (Liu et al.); oxide eDRAM from Chiang et al., with DRAM operations such as precharge folded into per-bit energy. Each model gives retention time (write-frequency dependent for GCRAM), bit-cell area, and read and write energy per bit. Adding a technology means adding a model.
- Statistics. From the trace: read and write frequencies, unique addresses and operation counts.
- Lifetimes and refreshes. For each write, the time until the data is overwritten or invalidated. The lifetime distribution is compared with each cell's write-frequency-dependent retention; any data object that outlives retention contributes bit-level refresh operations.
- Capacity. The number of unique addresses live at the same time, scaled by block size. This can be well below the physical capacity.
- Active energy. Reads, writes and refreshes weighted by per-bit device costs, with each refresh modeled as an extra read followed by a write.
- Composition. Each access goes to the cell type with the shortest retention that still exceeds its lifetime, preferring Si-GCRAM, then Hybrid-GCRAM, then eDRAM. The resulting share of accesses per type, combined with the capacity analysis, gives each type's capacity and area and the mix's total area and energy.
- Outputs. Interactive lifetime histograms (built with Tableau), a JSON report of refresh counts, energy and area for each single-device design and for the composition, and a composition summary.
7 · Evaluation setup
| GPU case study | Systolic-array case study | |
|---|---|---|
| Backend | Accel-Sim / GPGPU-Sim, L1 and L2 caches | SCALE-Sim, input, weight and output buffers |
| Workloads | PolyBench 2D and 3D convolution; MLPerf Inference v5.0 models: Llama-3.2-1B, Llama-3-8B, Llama-3.2-11B-Vision, ResNet-18, ResNet-50, BERT-base-uncased, GPT-J-6B, Stable Diffusion | ResNet-50, BERT and a 1B-parameter Llama (GEMM operations only); results shown for ResNet-50 |
| Inputs | Text models: 7-word prompts, 20 generated tokens. Vision: 224×224 images, batch 1 | 256-token sequences; 224×224 images, batch 1 |
| Hardware parameters | simulated GPU cache hierarchy | PE arrays from 32×32 to 256×256; 4 kB input and weight buffers, 8 kB output buffer; three dataflows |
| Baseline | The same memories built entirely from SRAM | |
The inputs were chosen to match common inference scenarios while keeping simulation tractable.
8 · Results on the GPU
- Lifetimes. Across all workloads, 64.3% of L1 and 18.4% of L2 accesses fall within Si-GCRAM's 1 µs retention, and 97.9% of L1 and 52.0% of L2 accesses within Hybrid-GCRAM's 10 µs. Some data lives much longer (up to 70 µs) and needs eDRAM or SRAM.
- Which kernels. Short-lived data comes mostly from linear algebra such as GEMM; longer lifetimes appear in memory transfers and nonlinear computations.
- Compositions. Most workloads get Si-GCRAM-dominated L1 caches (over 90% of capacity). Transformer models such as the Llama variants need substantial longer-retention capacity for KV-cache storage. L2 caches need a more balanced mix of technologies.
- Savings over SRAM. Si-GCRAM-dominant L1 caches average 1.89× lower active energy and Hybrid-GCRAM-integrated L2 caches 1.3×. Memory array area improves by 3× across workloads. The best workloads reach up to 3.1× energy and 4.3× area reductions.
9 · Results on the systolic array
- Lifetimes. For ResNet-50 on a 256×256 array, at least 79.01% of accesses across the three buffers live under 1 µs, so they fit 5 nm Si-GCRAM without refresh.
- Dataflow matters for stationary data. Output feature map data is short-lived under every dataflow (nearly all of it fits Hybrid-GCRAM), because its generation is tightly coupled to the PE array's timing. The dataflow mainly changes the maximum lifetime of whichever operand is held stationary, which can sit idle in its buffer once the array is saturated.
- Compositions and savings. The vast majority of scratchpad capacity goes to Si-GCRAM, with little Hybrid-GCRAM or eDRAM. Savings are about 3× in active energy and 2.4× in area under all three dataflows. The paper attributes the consistency to the deterministic access patterns of systolic arrays, in contrast with the more variable GPU results.
- Array size. Larger PE arrays slightly reduce average lifetimes (data is consumed faster) and shrink the maximum lifetime of stationary data, narrowing the distribution. Fewer long-lived outliers means more room for cells with even shorter retention.
10 · Limitations and directions the paper states
- Gain-cell scaling. Smaller process nodes raise leakage and cut retention in both Si-GCRAM and Hybrid-GCRAM, even though gate-all-around transistors reduce off-current. The paper expects GCRAM to stay viable: higher clock rates at smaller nodes shorten real data lifetimes, and circuit techniques can extend retention.
- Long-term RAM is not yet modeled. RRAM and MRAM are excluded because the studied workloads are transient and write-intensive. Evaluating them properly needs endurance modeling (write frequency, spatial patterns, temporal clustering over long periods), which the paper names as a future extension.
- Systolic-array coverage. The SCALE-Sim study covers GEMM operations; nonlinear operations are future extensions.
- Model fidelity. Results are first-order projections from cycle-accurate simulation with kernel sampling and device models, not measurements of fabricated heterogeneous arrays.
- Beyond AI. The paper argues that the same lifetime-aware approach applies to servers (HTTP buffers and session state with µs-scale lifetimes, cached content suited to LtRAM), databases (transaction logs, query results), scientific computing (intermediate arrays) and media pipelines (frame buffers). These are not evaluated.
FAQ
Q·Is GainSight a chip design or a tool?▾
A tool. It profiles workloads on simulated accelerators and, from a library of cell models, recommends how much of each memory technology an on-chip memory should contain and what that would save in area and active energy. No heterogeneous memory is fabricated in the paper. The code is open source (code.stanford.edu/tambe-lab/gainsight), with documentation and interactive visualizations at gainsight.stanford.edu.
Q·How is short-term RAM different from the long-term RAM (LtRAM) work on emerging memories?▾
They sit at opposite ends of the same trade. LtRAM technologies (MRAM, RRAM, FeRAM) are dense and keep data for a long time, but writes are expensive and endurance is limited, so they suit data that is rarely written. StRAM technologies (gain cells, eDRAM) are dense, cheap to write and low in leakage, but forget their contents after microseconds to milliseconds, so they suit data that is written, read and discarded quickly. GainSight focuses on StRAM; the paper names LtRAM evaluation, with endurance modeling, as a future extension.
Q·What happens to data that lives longer than the cell's retention?▾
On a single-technology design, it must be refreshed, and the frontend counts those bit-level refreshes and charges each as an extra read and write. The composition step avoids refresh altogether by sending such data to a cell with longer retention: Hybrid-GCRAM if it lives under 10 µs, otherwise eDRAM (ms-scale) or SRAM. In the GPU study some data lives up to 70 µs.
Q·Why not use eDRAM for everything, since it retains data for milliseconds?▾
Because longer retention costs active energy. The paper notes that Si-GCRAM, Hybrid-GCRAM and eDRAM offer increasing retention at the cost of higher active energy, and eDRAM's charge-sharing reads make it the most expensive of the three per access. Assigning each access to the shortest-retention cell that is still long enough maximizes use of the cheapest cells while keeping operation refresh-free.
Q·Why does write frequency appear in the device models?▾
For gain cells, retention is not a fixed number. The paper cites an inverse trade-off for Hybrid-GCRAM between retention and write frequency, and its Figure 3 shows Hybrid-GCRAM retention dropping at high write frequencies. The frontend therefore compares each workload's lifetimes with a write-frequency-dependent retention, using the write frequencies it measures from the trace.
Q·Why are the GPU results more variable than the systolic-array results?▾
The paper attributes it to determinism. A systolic array moves data through its buffers according to a fixed dataflow, so lifetimes are predictable and the savings are nearly identical across dataflows (about 3× energy and 2.4× area for ResNet-50). GPU kernels have dynamic control flow and cache behavior, so lifetimes, compositions and savings vary more from workload to workload.
Q·Does the right composition depend on the workload?▾
Yes. Most workloads in the GPU study get L1 caches that are over 90% Si-GCRAM, but the Llama variants need substantial longer-retention capacity for KV-cache storage, and L2 caches need a more balanced mix. That is why the paper argues for profiling each target workload set instead of relying on fixed heuristics.
Q·Cycle-accurate GPU simulation is very slow. How is this tractable?▾
Two ways. The GPU backend uses Principal Kernel Analysis to cluster kernels and simulate only representatives, which cut the simulated kernels to as little as 0.15% of the original (Llama-3-8B) with speedups of 10.89× to 412.76× on the three workloads checked; the mean absolute error in lifetime was 0.36 to 3.76 µs. The experiments also use small inputs: 7-word prompts with 20 generated tokens for text models, and single 224×224 images for vision models.
Q·How does this differ from Nsight Compute, perf mem or Cachegrind?▾
Those tools are retrospective: they report counters, sampled misses or cache statistics, often at kernel granularity, for an existing CPU or GPU. The paper's Table 2 shows none of them profiles non-GPU accelerators, correlates accesses with memory-cell properties, or generates a memory composition. GainSight records every cache or scratchpad access with timing on a simulated architecture and turns that into a design recommendation.
Q·Can I add my own accelerator or memory technology?▾
Yes, by design. Any simulator or physical accelerator that can emit a trace with timestamps, address ranges and read/write types can serve as a backend, with each of its memories mapped to a subpartition. A new cell type needs a model of its retention, bit-cell area and per-bit read and write energy.
Q·Why do larger systolic arrays help?▾
A larger PE array consumes data faster, which slightly lowers average lifetimes, and it sharply reduces the maximum lifetime of the stationary operand. With fewer long-lived outliers, a designer can consider cells that trade even more retention for energy efficiency, since outliers would otherwise dominate refresh overhead.
Q·What does the evaluation not cover?▾
All numbers are first-order projections from simulation and device models, without circuit-level effects or fabricated arrays. The systolic-array study covers GEMM operations only and reports detailed results for ResNet-50. Long-term RAM devices are out of scope. The applicability to server, database, scientific and media workloads is argued in the discussion but not measured.