The Kernel's Write: Application Read-Only Memory
This page was written by an AI model to give a high-level overview of the paper. Please check the paper for detail: ACM Digital Library.
DRAM has become the most expensive part of a server. Microsoft Azure reports that it is over half of total server cost, and DRAM's cost per bit has stopped falling the way it used to. Long-term RAM (LtRAM) is a class of emerging memories (RRAM, PCM, FeFET, MRAM) that read at close to DRAM speed and cost less per bit. The catch is that every candidate has the same shape: writes are much slower than reads, writes happen in chunks larger than a 64-byte cache line, and each cell survives only a limited number of writes. That makes LtRAM a good fit for long-lived, read-mostly data such as code, machine learning inference weights, and in-memory caches.
Intel Optane, the only LtRAM-class memory ever sold as a DIMM, hid that shape from software. Its on-DIMM controller made the media look like DRAM using a write-combining buffer, a read buffer, an address remapping table (the AIT) and its own wear leveling. The paper's analysis of prior measurements shows what that cost: a read-modify-write on every small store, an extra media access on most reads, 60 µs latency spikes when the controller moved data for wear leveling, and a bandwidth collapse when reads and writes mix. The paper's conclusion is that the interface was the problem, not the memory technology.
The proposal is to stop pretending LtRAM is DRAM and let the operating system manage it. The design rests on one invariant, Application Read-Only Memory (AROM): applications can read LtRAM pages but never write them, and only the kernel writes LtRAM, a whole 4 KB page at a time, when it migrates a page there. If an application stores to an LtRAM page, the store faults, the kernel copies the page back to DRAM (copy-on-write), and the store runs on the DRAM copy. Because every write is now a page-sized kernel write, the controller can shrink to three jobs (cache-line reads, 4 KB page writes, and reporting each block's health), and the OS takes over placement, migration and wear leveling with the page tables and free lists it already has.
The authors prototype the interface on the Enzian research platform, with a 256 MB NOR flash chip behind an FPGA controller standing in for LtRAM. Projecting the measured controller cost onto a DDR-class bus, three candidate technologies (3D FeFET, 3D V-RRAM, STT-MRAM) would read 26 to 79% faster than Optane per access, ranging from 3.2 times DRAM's latency to 9% faster than DRAM. Application-level benchmarks are left for future work.
Key takeaways
- LtRAM candidates share one shape: near-DRAM reads, slower and coarser writes, and limited endurance. They suit read-mostly data.
- Optane's DRAM-compatible controller (buffers, AIT, hardware wear leveling) is what made it slow and costly; the paper locates the problem in the interface.
- AROM: applications only read LtRAM; the kernel is the only writer, in 4 KB pages, during migration. Application stores trigger a copy-on-write migration back to DRAM.
- The controller shrinks to cache-line reads, 4 KB page writes and block-health reporting. The OS handles placement (a new memory zone, dirty-bit scans) and wear (a token allocator that paces migrations to the endurance budget).
- Prototype: Enzian with NOR flash. For the three projected technologies, per-access reads are 26 to 79% faster than Optane and 0.91× to 3.2× DRAM's latency; application-level results are future work.
From a DRAM-compatible DIMM to a kernel-managed LtRAM zone
What Optane's controller cost, the AROM invariant, what the controller and the OS each do, and the prototype.
1 · The shape every LtRAM candidate shares
The paper surveys four candidate technologies against DRAM (its Table 1). The numbers vary by orders of magnitude with materials and circuit choices, so the paper asks that they be read for shape, not precision. The shape: reads close to DRAM, writes one to two orders of magnitude slower, write units larger than a cache line, and bounded endurance.
| Device | Read | Write | Endurance (writes) | Write granularity | Cost per bit |
|---|---|---|---|---|---|
| DRAM | 80 ns | 80 ns | unlimited | cache line | high |
| 3D V-RRAM | 100 ns | 1 µs | 106 | page | low |
| 3D FeFET | 200 ns | 10 µs | 105 | page | low |
| PCM | 300 ns | 1 µs | 108 | 256 B | mid |
| STT-MRAM | 20 ns | 20 ns | 1015 | word | high |
Because writes are the weak side, the paper targets long-lived, read-mostly data: code pages, machine learning inference weights, and in-memory stores such as Redis and Memcached.
2 · What Optane's controller cost
Optane DC Persistent Memory Modules sat on the memory bus next to DRAM and accepted ordinary 64-byte loads and stores. Its media, 3D XPoint (a phase-change memory), works in 256-byte units Intel calls XPLines. The on-DIMM controller bridged the two with four mechanisms, and the paper (drawing on published measurements) assigns each a cost:
| Mechanism | What it does | Measured cost | Replaced in AROM by |
|---|---|---|---|
| Granularity bridging (write-combining and read buffers) | Merges 64 B writes into 256 B XPLines; fetches whole XPLines on reads | Random 64 B writes need a read-modify-write: 4× write amplification, ~2.3 to ~0.56 GB/s (~75% drop). Random reads also 4× amplified: ~305 to 374 ns vs ~169 ns sequential | Cache-line reads and 4 KB page writes only |
| Hardware wear leveling | Moves data off cells that receive concentrated writes | A migration about every 14,000 XPLine overwrites (~3.4 MB); reads stall ~60 µs, over 160× a typical random read | Wear leveling in the OS; no migration path in hardware |
| Address Indirection Table (AIT) | Maps CPU physical addresses to media locations, for wear leveling and bad-block remapping. Full table in the media; a 16 MB on-DIMM DRAM cache holds recent entries | Reads take 351 ns on an AIT-cache hit and 427 ns on a miss. The cache covers ~8 GB of a 128 GB DIMM, so ~94% of random reads miss | No on-module map |
| Workload-agnostic design | Accepts any read/write mix, as DRAM does | Sequential bandwidth falls from ~30 GB/s (pure reads) to ~10 GB/s at a 50/50 mix | Read-mostly placement; OS-paced writes |
3 · The AROM invariant
Under Application Read-Only Memory, LtRAM is read-only to applications, and the kernel writes LtRAM only when it migrates a page there from DRAM. That one rule changes what every LtRAM write looks like:
- Every write is page-sized. The kernel writes whole 4 KB pages, so no write is ever smaller than the media's write unit and the read-modify-write path disappears.
- Every write comes from one place. The kernel's migration path is the only writer, so it can decide when writes happen and how many happen.
- Every write is to a page believed to be read-mostly. Pages are moved only after they stop being written, which keeps LtRAM traffic mostly reads and avoids the mixed-workload collapse.
- Application stores still work. They are enforced with copy-on-write: a store to an LtRAM page faults, and the kernel migrates the page back to DRAM before the store is applied. The application never sees the difference except as latency on that store.
4 · What the controller still does
With AROM in place, the device-side controller has three responsibilities:
- Reads at cache-line granularity, where LtRAM is competitive with DRAM.
- Writes at 4 KB page granularity. 4 KB is a multiple of every candidate's write unit in Table 1, so writes always align to the media's boundary.
- Report each block's health to the OS, which uses it to seed its own wear-leveling state.
The controller does not wear-level. That removes the AIT and its DRAM cache (and with them the extra ~75 ns media access on the 94% of random reads that missed), and the ~60 µs stalls from hardware-driven migrations. What remains is the media's own access latency plus a thin controller.
5 · What the OS guarantees
- The only writes to LtRAM are 4 KB page writes from the kernel's migration path.
- An application store to an LtRAM page triggers a copy-on-write fault that migrates the page back to DRAM before the store applies.
- Pages migrated into LtRAM are read-mostly, avoiding the mixed-workload bandwidth collapse.
- Write traffic to LtRAM is balanced across cells, replacing the controller's wear leveling.
Guarantees 1 and 2 enforce AROM. Guarantees 3 and 4 move migration policy and wear leveling from the controller into the kernel, which already tracks pages, page tables and free lists.
6 · The life of a page
- Allocation. LtRAM is a new Linux memory zone with its own free-page list. A page goes straight to LtRAM only when the kernel or application can establish that the data is read-only by construction: executable code, shared libraries, read-only memory-mapped files. Everything else starts in DRAM.
- Finding read-mostly pages. Every scan interval T, the kernel walks the target process's page table entries. Pages whose dirty bit is still clear become migration candidates; pages that were written have their dirty bit cleared for the next scan.
- Migration to LtRAM. Candidates move in scan order, one token per migration (Section 7 below). If tokens run out, the rest wait for the next interval. The paper notes this delay is harmless and even helps accuracy: a page that stays read-mostly for longer is a safer bet.
- Migration back to DRAM. All LtRAM pages are write-protected. On a write fault the kernel allocates a DRAM page, copies the LtRAM page, updates the page table and executes the store. The old LtRAM page is freed and its block scheduled for erase when the device is idle.
- When a phase changes. If a large LtRAM region turns write-heavy at once, every store triggers a copy-on-write migration and DRAM's free list can run out. The paper proposes standard back-pressure: admission control on migration and DRAM-occupancy watermarks. Under write-heavy workloads fewer pages qualify for LtRAM, and the system behaves like a DRAM-only machine that swaps colder pages to storage.
7 · Wear leveling by token allocation
Instead of tracking every block's erase count, the kernel paces writes. For an LtRAM with N pages, each rated for E erases, over a deployment of T seconds, tokens are released at
- Every migration into LtRAM consumes one token; if none is available the migration is deferred.
- Tokens can be banked during idle periods, so a quiet workload can spend them later in a burst while the long-run average stays within budget.
- Because the kernel is the only writer and the allocator paces it, the device meets its lifetime budget by construction. Only the running token balance is stored, not per-block erase counts.
- This bounds total wear. It does not make wear uniform across the chip; the paper leaves uniform wear to future work (see Section 10).
8 · The prototype: Enzian with NOR flash
Enzian is a research platform that pairs a server-class ARM CPU (48 ARMv8 cores) with a cache-coherent FPGA. The FPGA acts as the LtRAM memory controller, and the CPU runs Linux 6.8 with the authors' page-migration extensions. The LtRAM is a 256 MB Micron serial NOR flash chip on the FPGA board.
NOR is not a canonical LtRAM technology, but it has the same shape and needs no custom chip: byte-addressable reads, 256 B writes, 4 KB erases, and 105 erase cycles per block. Measured chip-level latencies are 500 ns for a 128 B read (Enzian's cache line), 80 µs for a 256 B write and 18 ms for a 4 KB erase.
- One size for everything. The 4 KB write unit equals both the OS base page and the NOR erase block, so the controller never merges writes.
- Writes do not block reads. Incoming writes land in an SRAM buffer and are acknowledged before they reach the NOR chip, so reads to other regions are not stalled by the slow program operation.
- Erase is the OS's call. The erase command is exposed to the OS instead of being triggered internally, so the OS can schedule erases when read pressure is low.
- What the controller keeps. Only a small bad-block remap table. Its coarse entries (Optane used 64 KB per entry) let it live entirely in SRAM, unlike the AIT.
- What the kernel changes. A new zone at the top of the zone fallback order keeps ordinary allocations off LtRAM unless explicitly requested. The existing copy-on-write fault path is extended so LtRAM pages are copy-on-write by default. The largest change is migration to LtRAM: choosing pages and the token allocator. Applications are unchanged and never learn whether a page is in DRAM or LtRAM.
9 · Projected read latency
The paper projects the latency of one 128 B read through the new interface on a DDR-class bus, split into interconnect, controller and media. The interconnect is 40 ns for every technology: DRAM's 80 ns minus a 30 ns cell access and a 10 ns on-DIMM controller. The prototype's controller costs 180 ns (500 ns measured minus 320 ns of media access), which is 36 cycles at NOR's 200 MHz clock; at a DDR5-5600 clock (2.8 GHz) the same 36 cycles take 13 ns, the value used for the projected technologies.
The three projected technologies improve on Optane by 26 to 79%. The paper is careful about what this shows: these are single-access latencies. In real workloads, caches, prefetching and memory-level parallelism overlap per-access costs, so application-level performance on read-mostly data may come closer to DRAM than these numbers suggest. Application benchmarks are future work.
10 · Open questions the paper raises
- Wear on pages that never move. Read-only pages such as code and model weights can sit on the same LtRAM blocks for a long time, and unmovable pages (kernel code) never rotate at all, so the remaining blocks absorb more wear. Free-list rotation alone may not even out wear. One option: when the wear gap exceeds a threshold, evict pages from the least-worn blocks so those blocks rejoin the rotation. When to trigger it, the threshold, and how to pick pages cheaply are open.
- Huge pages. Code, libraries and databases are often mapped as 2 MB huge pages for TLB efficiency. One write-heavy object makes the whole 2 MB page ineligible; once in LtRAM, one store triggers a 2 MB copy-on-write. Splitting into 4 KB pages restores fine-grained copy-on-write but costs page-table entries and TLB reach, and transparent huge pages (khugepaged) can merge read-mostly pages with a write-heavy one again.
- Application hints. AROM needs no application changes, but an application could hint that a region is read-only or read-mostly (for example with
madvise) so the kernel places it immediately. A richer interface could let systems such as database engines pass their own knowledge of access patterns to placement.
FAQ
Q·What does the title mean?▾
It names the invariant: in an AROM system, a write to LtRAM is always the kernel's write. Applications only read LtRAM; the kernel writes it, in whole pages, when it migrates data there.
Q·Isn't this just memory tiering, like TPP or Memtis on CXL memory?▾
The mechanism looks similar (the OS moves pages between kinds of memory), but the target is different. Tiering systems demote cold pages to a slower tier that sits between DRAM and storage, and both reads and writes to that tier are slower. None of them reason about endurance. AROM places read-mostly pages, which may be hot, in LtRAM as main memory alongside DRAM, and the only extra cost is a copy-on-write migration when a page in LtRAM is written. The paper also warns that tiering's page-selection policies may not carry over, because they select for coldness, not for an absence of writes.
Q·Which prior system is closest?▾
Google's software-defined far memory built on zswap: the OS manages a cheaper tier (compressed RAM) and defers the cost to access time (decompression on a fault). AROM extends that idea into a hardware/software co-design, and its deferred cost is paid only on writes to LtRAM pages, not on both reads and writes.
Q·Do applications have to change?▾
No. A process sees only its virtual address space and never learns whether a page is in DRAM or LtRAM. The kernel decides placement. Hints (for example madvise marking a region read-only or read-mostly) would let the kernel place data sooner, and richer hints from systems such as databases could reduce migrations back to DRAM, but both are optional and left as open design questions.
Q·What happens when a read-mostly page suddenly starts being written?▾
The first store write-faults. The kernel allocates a DRAM page, copies the LtRAM page into it, updates the page table and runs the store; later stores go to DRAM at full speed. The danger is a phase change that turns a large LtRAM region write-heavy at once: every page faults and DRAM's free list can run out. The paper's answer is back-pressure (admission control on migration and DRAM-occupancy watermarks). In a write-heavy period fewer pages qualify for LtRAM, and the system behaves like an ordinary DRAM-only machine.
Q·How does the kernel know a page is read-mostly?▾
In the current design, with the dirty bit in each page table entry. Every scan interval the kernel checks the bits: a clear bit makes the page a candidate, and a set bit is cleared so the next scan can see whether the page is written again. The paper calls this cheap but imprecise and leaves better classifiers, such as a small online predictor, as an open question.
Q·Why fix the write size at 4 KB?▾
Three reasons from the paper. It is a multiple of every candidate technology's write unit, so writes always align to the media. It is the OS's base page size, so the kernel can manage LtRAM with its existing page machinery. And on the NOR prototype it is also the erase block size, so the controller never has to merge writes.
Q·Does the token allocator spread wear evenly?▾
No, and the paper says so. Tokens bound the total number of writes to what the device's endurance allows over its lifetime. They do not control which blocks absorb them. Long-lived read-only pages and unmovable kernel pages stay on the same blocks, so the rest of the device takes more wear. How to even it out (for example by evicting pages from the least-worn blocks past a wear gap) is open.
Q·Why prototype with NOR flash if it is not an LtRAM technology?▾
Availability and cost: a 256 MB Micron serial NOR chip drops onto the Enzian FPGA board, while MRAM or RRAM would need a research tape-out. NOR has the same shape the interface targets: byte-addressable reads, 256 B writes, 4 KB erases and 105 erase cycles per block. It is slower than the projected technologies (500 ns reads), which is why the paper projects latency for the others instead of reporting NOR's.
Q·What about 2 MB huge pages?▾
They are an open problem. A single write-heavy object makes a whole 2 MB page ineligible for LtRAM, and once a huge page is in LtRAM one store triggers a 2 MB copy-on-write. Splitting it into 4 KB pages keeps copy-on-write fine-grained but costs page-table entries and TLB reach, and transparent huge pages can merge pages back together. Prior tiering work suggests splitting pages with skewed access or co-locating hot objects, but when to split or merge is left for future work.
Q·Does the paper show application speedups?▾
No. It reports measured chip-level latencies on the prototype and projects per-access read latency for other technologies. It notes that caches, prefetching and memory-level parallelism hide part of per-access latency, so applications on read-mostly data may come closer to DRAM than the per-access numbers suggest. Application-level evaluation is future work.
Q·Who handles bad blocks?▾
The controller keeps a small bad-block remap table. Its entries are coarse (Optane used 64 KB per entry), so the whole table fits in on-controller SRAM, unlike Optane's AIT, which needed a DRAM cache and media storage.