all you need is MMU

the GPU’s page tables, the system MMU, and one wall for attention
PART 3 OF THE TOKENIUM SERIESArthur Rasmusson, vgpu.io
Director of AI Architecture, Lightbits Labs
November 2026

Purpose and scope

Purpose and scope. This is the series’ one post about the memory management unit. It gathers what Parts 2 and 3 said about translation, drops what belongs elsewhere, and checks each claim against two sources: the Open-IOV project’s documentation of the GPU’s own MMU, the GMMU, and Arm’s architecture for the System MMU, version 2. It makes one argument. The memory that holds a conversation, the KV cache, is protected today by a serving engine’s bookkeeping. It can be protected by the GPU’s page tables instead, and the first part of that needs no new silicon.

How to read it. Technical statements carry one of six labels, given in prose where they matter: existing and documented; existing but platform-specific; proposed Tokenium extension; architectural inference; unverified; and open question. The GMMU details come from Open-IOV’s Turing reference, and Open-IOV warns that newer generations change the entry format and must each be worked out again. Everything said here about bit positions is therefore platform-specific.

This post is only about translation. Allocation, the firmware front end, scheduling and attestation are covered in Parts 4, 5 and 7.

Every safe way to share a machine

Every safe way to share a machine has been a new layer of translation. The MMU let processes share a CPU. Nested paging let virtual machines share a host. The IOMMU let devices share both. Each time the trick was the same: give each tenant its own table of names, and let the hardware refuse any name that is not in the table. A page that is not mapped in your table is not locked against you. It is simply not there. The series calls this isolation by absence, and it is older and cheaper than any check.

An inference server has three of these units, and they do not share a page table. The CPU’s MMU translates a process’s addresses. The GPU’s GMMU translates the addresses every GPU engine uses. The platform’s IOMMU, which Arm systems call the SMMU, translates the addresses a device puts on the bus. They are three instances of one idea, at three points on the same path, programmed by three different pieces of software (existing and documented).

A table comparing the CPU MMU, the GMMU and the IOMMU or SMMU: what each translates, who programs it, what each table belongs to, and what each keeps apart.
Three MMUs, one idea. Adapted from Open-IOV, Introduction to GMMU.

None of them is asked to protect the thing that matters most in inference. A model keeps what it has read of a conversation in the KV cache, keys and values for every token, in fixed-size blocks. vLLM’s paged attention made those blocks the unit of allocation, with a block table for each request that maps its logical blocks to physical ones. That block table is a page table in everything but name, and software enforces it (existing). The hardware would run a kernel that reads another request’s blocks, because to the hardware they are only memory in the same context. When many tenants or agents share one serving engine, the wall between their conversations is a ledger.

The fix this series proposes fits in one sentence. A block becomes a page the MMU can refuse to translate. The rest of this post is what it takes to make that sentence true: what the GMMU already does, what it does not, what the SMMU has to teach, and what can be built first in software.

The GMMU as it is

Everything that touches GPU memory goes through the GMMU. Shaders and CUDA kernels on the graphics engine, the copy engines, the security engine and the video engines all name memory by GPU virtual address, and the GMMU turns each name into a place: an offset in the GPU’s own memory, an address in system memory, or a peer GPU’s memory across a fabric (existing and documented).

On the Turing generation that Open-IOV works from, a GPU virtual address is 49 bits wide and the table is a tree five levels deep. The walk starts at the page directory base named in the channel’s instance block and goes down through four levels of directory, PDE3, PDE2, PDE1 and PDE0, to a leaf entry. Pages come in three sizes: 2 MB, which ends the walk at PDE0, and 64 KB or 4 KB at the leaf. A PDE0 slot can hold two halves, one pointing at a table of 64 KB pages and one at a table of 4 KB pages, which is how one region mixes both sizes (platform-specific; the geometry is inherited from Pascal and Volta).

A diagram of the GMMU page walk: a 49-bit GPU virtual address split into slices that index five levels, from the instance block’s page directory base through PDE3, PDE2, PDE1 and PDE0 to a leaf entry and a page, with a 2 MB page ending at PDE0, and boxes for the TLB, the page-table fetch cache and the fault buffer.
The walk on the Turing reference. Source: Open-IOV.

Each leaf entry is eight bytes. It says whether it is valid, which aperture the page lives in (video memory, a peer, coherent system memory or non-coherent system memory), the physical address, the memory kind, and a handful of permission bits: privileged, read-only, atomics disabled, encrypted (platform-specific). A translation lookaside buffer, the TLB, remembers finished walks so the tree is not walked on every access. When a walk finds no entry, the GMMU reports the access through a fault buffer and the driver decides what happens next. A replayable fault waits while the page is supplied. A fatal one ends the work that caused it (existing).

A diagram of the 64-bit VER2 leaf entry with its fields from kind down to valid, the read-only bit highlighted, a callout explaining the aperture values, and a dashed empty box noting that no tenant, ACL or PASID field exists.
The leaf entry. The read-only bit is in the format; whether the GPU enforces it has not been tested. Source: Open-IOV.

One detail matters more than it looks. The GPU fetches page-table entries in 128-byte lines, sixteen entries at a time, and keeps them. On Open-IOV’s path, which has no invalidation step yet, a write into a line the GMMU has already walked is invisible to the engine. Changing a mapping is therefore a protocol and not a store: write the entry, then invalidate every cache that may hold the old one (platform-specific; observed live by Open-IOV, and not yet a complete invalidation contract).

What the hardware gives, and what it does not

Here is the most important fact in this post. Each GPU context has its own page directory. One context cannot name another context’s mappings, because they are not in its tree. On the reference generation, that per-context page directory is the only proven hardware isolation primitive (existing and documented, Open-IOV).

One island with roads to several page platforms lit teal, many other platforms lit violet with no roads, and one road that stops in the water short of a violet platform, ending at a red lamp.
Isolation by absence. The island has roads to its own pages. The road toward someone else’s page was never built, and a walk that tries it ends in a fault.

Everything finer than a context is not yet proven, and Open-IOV is careful to say so. Its catalogue of what the hardware gives reads like this.

  1. ProvenA five-level tree with 49-bit addresses and 4 KB, 64 KB and 2 MB pages. The leaf fields, including read-only, privileged, encrypted and the memory kind. A separate page directory for each GPU address space.
  2. Proven absentNo tenant, access-control or process-identifier field in any entry. The peer identifier selects a peer on a fabric. It is not a security domain.
  3. HypothesisThat the GPU enforces the read-only bit against writes. It needs a supervised probe before anyone relies on it.
  4. HypothesisThat the encryption domain can serve as a boundary between tenants.
  5. OpenHow invalidation behaves in every case, beyond the fetch-line rule. And the entry format on Ampere, Hopper and Blackwell, which must be worked out for each.

Two things follow. The first is that the wall we can build today is a context wall: one tenant per context, one page directory per tenant. It is coarse, and it is real. The second is that sharing is the hard part. The obvious way to save memory, mapping one physical copy of a shared prompt prefix into many tenants as read-only, depends on the one bit that has not been tested. Until it is, a shared page has to be treated as writable by everyone who can see it (architectural inference).

The SMMU on the same path

A GPU’s access to system memory crosses two translation units in a row. The GMMU, driven by the GPU driver’s tables, decides which guest page an engine touches. Then the platform’s IOMMU or SMMU, programmed by the hypervisor, decides whether this device may reach that page at all, and where it really lives (existing and documented). The two stages are independent. Isolation between GPU contexts is the GPU driver’s job. Isolation between virtual machines is the platform’s. On Arm hosts the SMMU plays the IOMMU’s part.

A diagram showing the GPU’s two-stage path to system memory through the GMMU and then the IOMMU or SMMU, and below it the inside of an SMMUv2: a transaction with a StreamID, a stream mapping table of stream match registers and stream-to-context registers, context banks configured for stage 1, stage 2 or nested translation, and the two stages leading to memory.
Two stages on the GPU’s path, and inside the second one, how an SMMUv2 routes a transaction. Sources: Open-IOV; Arm SMMU architecture version 2.

The SMMU is worth studying closely, because it solved the problem the GMMU now faces: many requesters, one memory, and each requester owed its own view. Version 2 does it in three steps (existing and documented).

First, every transaction carries a StreamID, an identity supplied by the hardware that issued it and not by the software that asked. Second, the SMMU matches that ID against its stream mapping table. Each entry pairs a stream match register, which compares the ID under a mask, with a stream-to-context register, which says whether to translate, to bypass or to fault, and which context to use. Third, the chosen context bank holds a translation table base and its own fault state. A context bank can translate in stage 1, from a virtual address to an intermediate physical one; in stage 2, from intermediate to physical, which is how a hypervisor confines a guest’s devices; or in both, which Arm calls nested.

Dusk at a harbor. Four gatehouses on a quay, each with a lamp of its own color and a basin behind it tinted the same color. Boats flying matching flags approach their gates, and boats with gray flags wait to one side.
Stream matching. Every boat flies its own flag. A gate opens only for the flags it was told to admit, and behind each gate is its own basin. There are only so many gates.

Three lessons carry over to the GPU.

  1. Name the requesterThe SMMU decides who is asking before it decides what they may reach, and the identity comes from hardware. The GPU’s equivalent is the channel and its instance block, which already selects a page directory. A boundary between tenants should hang on that, and not on anything a kernel can claim about itself.
  2. Keep domains in memorySMMUv2 keeps its contexts in registers, so it has only so many of them. Version 1 allows up to 128 stream match registers, and version 2’s extension up to 1024, and Arm’s own architecture notes that register-based configuration cannot support thousands of concurrent contexts. Version 3 moved stream and context state into tables in memory for that reason. The GMMU’s directories already live in memory, which is the right side of that line. Agents will number in the thousands.
  3. Compose the stagesStage 2 lets a hypervisor confine a guest without trusting the guest’s own tables. The same shape fits inside the GPU: an outer table the platform owns, an inner one the tenant may own, and an access must pass both.

One wall for attention

Put those lessons together and the design is not exotic. Open-IOV’s design for multi-tenant protection, written for the case where several virtual machines, and several tenants inside each, share one GPU, offers three granularities of wall and is plain about the status of each (proposed; Open-IOV research track).

Three cards from coarse to fine: a page directory per tenant, labeled proven lever; channels per agent, labeled design; and a tag on every KV block, labeled research.
Three granularities of wall. Adapted from Open-IOV, GMMU Multi-Tenant Protection.

A page directory per tenant is the proven lever, and it can be deployed now, as a contract: one agent per tenant, one tenant per context. Channels per agent come next. They contain faults and stalls, so that one agent’s bad kernel is charged to that agent alone. A tag on every KV block is the finest wall and the one that saves memory, and it is research: it needs page-table policy enforced in software, and sharing a block safely needs the read-only bit proven.

The first version needs no change to the guest’s driver. The serving engine reserves a range of virtual addresses for each agent. A small utility publishes which range belongs to which agent, and the layer that mediates the GPU checks, when a mapping is made and again when work is submitted, that an agent’s work names only its own range (Open-IOV’s Design A; research). A second version adds explicit controls to the resource manager, to bind a tenant, tag a range and share one, and it waits behind its own gate (Design B; proposed).

Two agent domains side by side, each with a kernel, a page directory and its own pages. A dashed red line from agent A1’s kernel toward A2’s pages ends at a red cross labeled no entry in A1’s tree, above boxes for a fault charged to A1, a check at submission, and an owner audit.
A stray access, walked through. Adapted from Open-IOV’s two-tenant example.

Here is what that looks like when something goes wrong. Two agents each own a pool of addresses and their own channels. A kernel from the first names an address that belongs to the second. There is no entry for it in the first agent’s tree, so the GMMU faults, or the submission is refused before it runs, and the fault is charged to the first agent only. The second never notices. A periodic audit compares what each agent owns with what the page tables actually say, with canary pages to catch anything that slips through (proposed).

Two islands, one with teal blocks and one with violet blocks, joined to a round platform in the middle by two railed bridges. The platform holds one gold stack.
One page, two bridges: a shared prefix kept once and reached read-only from two tenants. A hypothesis until the read-only bit is proven.

Seven changes, none of them exotic

Part 2 imagined a machine built for tokens. Its MMU differs from today’s in five ways, and two more are needed by every MMU on the path. All seven are proposed Tokenium extensions, and each has a precedent in a CPU or an SMMU.

  1. Pages the size of a blockLet the page size match the KV block, so that protecting a block wastes no memory and a block never shares a page with someone else’s.
  2. A domain on every pageGive each page a protection domain and each kernel launch a domain, and let the walk check that they match. The VER2 entry has no field for this today.
  3. Read-only aliasesMap one physical page of a shared prefix into many domains as read-only, on condition that its contents hash the same. This is exactly where the untested bit sits.
  4. Residency, not only faultsLet a load ask whether a page is present and get an answer, so a kernel can take other work instead of stalling.
  5. Evidence for the schedulerExport cheap summaries of access for each page, touched bits and coarse reuse, so placement is decided on evidence and not on guesses.
  6. One invalidationWhen a mapping changes, retire it from every cache that might hold it, in the CPU, the SMMU and the GMMU, in one operation with one completion. Today each has its own.
  7. A compatible formatMake tables that more than one engine can walk, the way the stage 1 format of Arm’s SMMU matches the CPU’s. That match is what lets a device share a process’s own tables, which version 3 of the SMMU supports.

Software first

Most of this can be measured before any chip changes, and the order matters (proposed; Open-IOV roadmap).

  1. Own the stateBefore any page-table policy, the software that mediates the GPU must key every channel, address space and translation by its owner, and not by values a guest chose. Two guests can pick the same handles. Open-IOV treats this as the baseline, and it comes first.
  2. Prove two guests apartRun two guests side by side and show that neither can reach the other.
  3. Invalidate and reclaimFinish an invalidation and eviction protocol, with its invariants written down, before anything is shared.
  4. Agent rangesBuild Design A: an address range for each agent, checked at map time and at submit time, with no driver change.
  5. Probe the read-only bitWrite to a read-only page under supervision and record what the GPU does. Until it passes, sharing stays off.
  6. Explicit controlsDesign B and the encryption domain come last, each behind its own gate.

One number decides whether any of it works: reads across a tenant boundary. It must be zero in every run, under attack, and it must stay zero as the work gets faster. Everything else, throughput, memory saved, latency, is open to negotiation.

The MMU does not implement a scheduler, a coherence protocol or a security model. It implements the one thing all of them depend on: a table of names that the hardware will not look past. The GPU already keeps one for every context. The work is to extend the same protection to attention. A block becomes a page the MMU can refuse to translate, first in software that can be measured this year, then in silicon that makes it cheap.

Notes and sources

  1. GMMU factsOpen-IOV, Introduction to GMMU: the three MMUs, the page-tree geometry, the VER2 entry, walking, caching and faults, and the limits, all on the Turing reference.
  2. Multi-tenant designOpen-IOV, GMMU Multi-Tenant Protection, and the canonical design document in the project repository: isolation layers, proven and hypothesis primitives, Designs A and B, the two-tenant example and the roadmap.
  3. SMMUv2Arm, System Memory Management Unit Architecture Specification, SMMU architecture version 2.0, Arm IHI 0062. A practical summary is the OSDev wiki’s ARM SMMU versions 1 and 2.
  4. Paged attentionWoosuk Kwon and others, Efficient Memory Management for Large Language Model Serving with PagedAttention, SOSP 2023. Its block table is the thing this post wants the hardware to enforce.

The series, in order