Skip to content

Run frontier MoE models on hardware you already own β€” pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦

C

E5: moe_submit relies on the residency set instead of per-buffer useResource:

monotophic309f20c9397f
diff --git a/SUMMARY.md b/SUMMARY.md
new file mode 100644
index 00000000..a4b86e36
--- /dev/null
+++ b/SUMMARY.md
@@ -0,0 +1,275 @@
+# E5 β€” MTLResidencySet over the existing malloc'd slabs (experiment branch)
+
+Branch: `e5/metal-residency-set` (cut from `origin/dev` @ `caa49f7`, per spec β€” E4 was cut
+from `main` @ `72d3d37`; `backend_metal.mm`/`.h` are byte-identical between the two bases,
+confirmed via `git diff 72d3d37 origin/dev -- c/backend_metal.mm c/backend_metal.h`).
+
+## The hypothesis
+
+E4 (MTLHeap-backed slabs) proved that batching residency declaration kills the GPU stall
+(25.9s β†’ 3.9s at cap16, βˆ’85%), but changing the *allocation* (heap sub-buffers instead of
+malloc'd host memory) brought a +12–13s expert-disk-load tax, suspected first-touch/lock
+contention on CPU-writes into GPU-owned heap pages. E5 decouples the two: keep the exact
+same malloc'd slabs and per-slab `newBufferWithBytesNoCopy`-wrapped `MTLBuffer`s, and change
+**only** residency bookkeeping β€” declare residency once, ahead of time, on a set attached to
+the command queue, instead of once per command buffer via `useResource:`. If the stall
+reduction survives without the load-path tax (malloc pages never change ownership), E5 wins.
+
+## What changed
+
+Everything is confined to `c/backend_metal.mm`. **No changes to `glm.c` or
+`backend_metal.h`** β€” `coli_metal_register`/`coli_metal_unregister`'s existing signatures and
+every call site in `glm.c` (expert_load, uring_load_add, qalloc, kv_alloc, map_of_fd) are
+untouched; the residency-set bookkeeping lives entirely inside those two functions' existing
+bodies. This is a smaller diff shape than E4, which needed new `backend_metal.h` declarations
+and new `glm.c` call sites because it changed the allocation function itself.
+
+Env-gated `COLI_METAL_RESSET=1`, default OFF, runtime `@available(macOS 15.0, *)` guard with
+a one-line stderr fallback when requested on an older OS or when residency-set creation
+fails. Gate off β‡’ every new branch is skipped and behavior is byte-for-byte the stock path
+(verified by inspection: `g_resset_enabled` starts `false` and nothing sets it except inside
+the `COLI_METAL_RESSET` `getenv` branch in `coli_metal_init`, so `resset_add`/`resset_remove`/
+`resset_flush` are no-ops and `moe_submit`'s `useResource:` loop runs unconditionally).
+
+### Lifecycle (`c/backend_metal.mm`)
+
+- **Init** (`coli_metal_init`, end of the existing pipeline-setup `@autoreleasepool`): if
+ `COLI_METAL_RESSET=1` and `@available(macOS 15.0, *)`, create one
+ `MTLResidencySetDescriptor` (`initialCapacity=4096`, a presize hint only), call
+ `[g_dev newResidencySetWithDescriptor:desc error:&err]`, and `[g_queue addResidencySet:rs]`
+ β€” one set, attached once, for the process lifetime. Failure (old OS or creation error)
+ prints one stderr line and leaves `g_resset_enabled=false` β€” stock path.
+- **`coli_metal_register`**: after wrapping the buffer exactly as today
+ (`newBufferWithBytesNoCopy`), calls `resset_add(b)` under the *same* `g_slab_mtx` that
+ already serializes `g_slabs` mutation from parallel OMP loader threads. `resset_add` calls
+ `[rs addAllocation:b]` and sets a `g_resset_dirty` flag β€” **it does not commit**.
+- **`coli_metal_unregister`**: calls `resset_remove(b)` (also under `g_slab_mtx`) *before*
+ clearing the `g_slabs` entry. `resset_remove` calls `[rs removeAllocation:b]` **and commits
+ immediately** β€” no batching. See UNCERTAINTIES for why this asymmetry is deliberate.
+- **`moe_submit`** (the one function whose `use` list β€” resolved expert weight/scale slabs β€”
+ scales with LRU cache size): calls `resset_flush()` at the top (commits any pending adds
+ from `resset_add`, under `g_slab_mtx`), then, if `g_resset_enabled`, **skips** the
+ `for(auto&b:use) [e useResource:b usage:MTLResourceUsageRead];` loop entirely β€” residency
+ is already guaranteed by the queue-attached set. Every other `useResource:` call site in the
+ file (`bind_gemv`'s weight/scale buffers, `coli_metal_attn_decode`/`coli_metal_layer_decode`'s
+ `Lb`/`Rb`/`kvbW`/`kvbS`/`inB`/`pnB`/`rwB`/`rbB`, `coli_metal_gemm`'s `wb`/`sb`) is
+ **left completely unchanged**, regardless of the flag β€” see "Why only `moe_submit`" below.
+- **Shutdown** (`coli_metal_shutdown`): `[g_queue removeResidencySet:rs]` then clears the
+ globals, ahead of the existing `g_queue=nil; g_dev=nil;`.
+
+### Why only `moe_submit` skips `useResource:`
+
+Apple's own `MTLResidencySet` docs are explicit: *"Residency sets don't support hazard
+tracking, so you need to account for hazards with fences and events."* (confirmed against
+the actual SDK header comment and Apple's "Simplifying GPU resource management with residency
+sets" guide, both read directly for this design β€” see UNCERTAINTIES for the exact quotes and
+how they were obtained). Dropping `useResource:` therefore risks losing whatever
+hazard-tracking value those calls provided. Rather than apply the residency set uniformly and
+argue *in general* that hazard tracking isn't load-bearing, this diff draws the line at the
+one call site the mechanism history actually implicates:
+
+`moe_submit`'s `use` vector holds only **read-only** (`MTLResourceUsageRead`), **indirectly
+referenced** slab buffers β€” the kernel (`moe_gemv`) never touches them via `setBuffer:`; it
+dereferences raw GPU addresses (`waddr[e]`/`saddr[e]`) baked into a separately-bound address
+array (`bag`/`bau`/`bad`/`bsg`/`bsu`/`bsd`), which is exactly the "indirect reference" case
+`useResource:` exists for. No GPU-side write ever touches these buffers, so there is no
+write-after-write/read-after-write hazard for Metal's tracking to have been serializing in
+the first place; the one real hazard β€” a slab unregistered+freed+reused by the CPU while an
+async in-flight `moe_block_begin` command buffer still references it via a baked-in GPU
+address β€” is a **CPU-write race that Metal's hazard tracking never protected against anyway**
+(hazard tracking only covers GPU-side command dependencies visible through the Metal API; a
+raw host-memory write via `pread`/`memcpy` is invisible to it regardless of `useResource:`).
+That race is, and always was, the engine's own responsibility (slot/generation lifecycle: a
+slab isn't freed while an outstanding async handle still owns it) β€” unrelated to E5.
+
+Every other call site (`bind_gemv`, attention K/V cache writes) either doesn't scale with
+cache size (fixed per-layer dense tensors β€” no perf benefit to touching) or has real
+GPU-side write traffic in the same encoder (`Lb`/`Rb` are written by `a_copy` and read by
+`a_score`/`a_clat` within one encoder β€” currently ordered by explicit
+`memoryBarrierWithScope:MTLBarrierScopeBuffers` calls already present in `encode_attention`,
+not by `useResource:`'s hazard tracking, but touching them wasn't needed for the hypothesis
+and was judged not worth the added surface area). Leaving them untouched keeps the diff's
+blast radius matched to the one seam the fix-plan's v5 finding actually names.
+
+### Deferred-commit design (`resset_add` batches; `resset_remove` doesn't)
+
+`coli_metal_register` is called from parallel OpenMP loader threads in tight bursts
+("warmup fan-out" β€” same phrase E4's audit used for the same threads). Committing on every
+single `addAllocation:` would reintroduce a per-slab cost on the load path, which is exactly
+what E4's own +12s regression looked like (mutex held across a live Metal call, serializing
+loader threads). So `resset_add` only marks `g_resset_dirty`; the commit is deferred to the
+next `moe_submit` call, which flushes once via `resset_flush()` before it relies on the set
+for residency.
+
+This is correct β€” not just fast β€” because of an existing invariant the codebase already
+depends on for `resolve()` to work at all: a slab's `coli_metal_register` call (mutex-guarded)
+always completes and releases `g_slab_mtx` before any dispatch that references that slab's
+pointer can call `resolve()` for it (the caller in `glm.c` cannot pass a freshly-loaded
+expert's pointer to a dispatch before the load β€” which registers it β€” returns). Since
+`resset_flush()` also takes `g_slab_mtx`, and runs immediately before `moe_submit`'s own
+`resolve()` calls in program order, any slab a given `moe_submit` invocation will resolve was
+already `addAllocation:`-ed (and marked dirty) strictly before that invocation's
+`resset_flush()` runs β€” so the flush is guaranteed to cover it, regardless of what other
+threads are concurrently registering unrelated slabs.
+
+`resset_remove`, by contrast, commits synchronously and immediately, with no batching,
+because the caller (`glm.c`, in every one of the four slab-realloc call sites, and in
+`kv_alloc`) frees the underlying host memory *right after* `coli_metal_unregister` returns.
+An uncommitted-but-still-set-member allocation pointing at memory the host has already freed
+is a potential use-after-free the GPU could act on β€” deferring that removal is not a
+performance-vs-safety tradeoff, it's just unsafe, so it isn't deferred. (The spec's own
+lifecycle wording backs this reading: "`coli_metal_register` β†’ add allocation + commit
+**(batch commits where call pattern allows)**" carries a batching allowance that
+"`coli_metal_unregister` β†’ remove + commit" does not.)
+
+## Instrumentation parity
+
+No existing counter's semantics changed. `coli_metal_moe_times`/`coli_metal_moe_counts`
+(`g_t_setup`, `g_t_gpu`, `g_t_kernel`, `g_t_scatter`, `g_moe_ok`/`g_moe_fb`/`g_moe_experts`)
+are computed exactly as before β€” `resset_flush()` runs *before* `ts_start = mnow()` in
+`moe_submit`, so its cost (whatever it is) is **outside** `g_t_setup` and therefore invisible
+to the existing setup/gpu/kernel breakdown. This is a deliberate choice: it keeps the
+orchestrator's A/B harness reading the same counters with the same meaning across stock/E4/E5,
+but it also means the harness's existing numbers will not show E5's flush cost if it turns
+out to be non-negligible β€” see UNCERTAINTIES. No new counters were added (no `glm.c`
+`profile_print` changes), unlike E4's `METAL-HEAP: alloc fallbacks` line, because there was no
+`glm.c` touch point to hang a print on without adding one β€” judged not worth the extra diff
+surface for an experiment branch; `[METAL] residency-set: on` / the two fallback stderr lines
+from `coli_metal_init` are the only new observability, sufficient to confirm which path a run
+took.
+
+## Per-seam differences vs E4
+
+| Seam | E4 (`e4/metal-heap`) | E5 (this branch) |
+|---|---|---|
+| Allocation | New: `MTLHeap` sub-buffers via `coli_metal_heap_alloc` | Unchanged: same `posix_memalign` + `newBufferWithBytesNoCopy` |
+| `glm.c` / `backend_metal.h` | Touched (new alloc/free API, 4 call sites + `expert_host_release`) | **Untouched** |
+| Residency scope | Declared once **per command buffer** (`useHeap:`, still inside `moe_submit`) | Declared once **for the process** (queue-attached set), refreshed incrementally at register/unregister |
+| Hazard tracking | Heap sub-buffers forced `MTLHazardTrackingModeUntracked` always (allocation-level) | Untouched at the resource level; `moe_submit` alone stops calling `useResource:` (encoder-level), independent of `COLI_METAL_UNTRACKED` |
+| Per-buffer vs per-set skip | `[b heap]` (Metal's own `MTLResource.heap` property) checked per buffer β€” heterogeneous mixes possible if a slab fell back to malloc | Blanket `if (!g_resset_enabled)` β€” homogeneous by construction, since every registered slab goes through the same `coli_metal_register` path when the gate is on |
+| Availability guard | None needed (`MTLHeap` is old API) | `@available(macOS 15.0, *)`, matching this box's macOS 26.5 but required for portability |
+| Known regression | +12–13s expert-disk load at cap16 (suspected first-touch/lock contention on heap pages) | None expected β€” malloc pages never change ownership; **unverified without a run** |
+
+## What to measure (orchestrator, cap1/cap16, stock vs E4 vs E5)
+
+1. **GPU stall** (`coli_metal_moe_times` gpu/kernel breakdown) β€” success: E5 β‰ˆ E4's
+ βˆ’85%-class reduction vs stock at cap16.
+2. **Expert-disk load path** (existing load/service-time counters) β€” success: E5 β‰ˆ stock,
+ i.e. **no** repeat of E4's +12–13s tax, since allocation is untouched.
+3. **tok/s** β€” should track (1) and (2) together.
+4. **md5 within a fixed dispatch composition** β€” flag on vs off must be byte-identical at a
+ given cap (the "Output-invariant by construction" hard constraint); flag-on vs flag-on
+ across cap1/cap16 may legitimately differ (different dispatch composition, per the
+ fix-plan's "Determinism side-finding").
+5. **`[METAL] residency-set: on` line present in stderr** at flag-on startup, and absent
+ (or the OS<15/create-failed fallback line) otherwise β€” cheap sanity check that a run
+ actually exercised the intended path before trusting its numbers.
+6. If the hypothesis holds (E5 stall β‰ˆ E4, E5 load-path β‰ˆ stock, identical output), E5 becomes
+ the upstream PR candidate and must include the cap-default recalibration flagged in PR
+ #386's CURRENT-STATE CALIBRATION markers, per the spec's validation plan.
+
+## Build
+
+`cd c && make glm METAL=1` and a separate explicit `-Wall -Wextra` compile of
+`backend_metal.mm` (the Makefile's `METALXX` line does not itself pass `-Wall -Wextra`, so
+"clean under `-Wall -Wextra`" was checked with those flags added explicitly), plus
+`cd c && make glm` (plain, non-Metal β€” unaffected, since this diff never touches `glm.c`), and
+`make metal-test` (existing synthetic kernel-correctness unit test β€” no model, no
+`glm52_i4/`, random weights β€” run once with `COLI_METAL_RESSET` unset and once with
+`COLI_METAL_RESSET=1` to numerically exercise `coli_metal_register`/`moe_submit`'s changed
+code path, since the task scope excludes running the real model). Exact results in the final
+report, not here (build results belong to the report per the task's deliverable split, and
+this file is written before the batched build run, per the scheduling constraint).
+
+## UNCERTAINTIES
+
+**Everything below is a judgment call, a seam where the residency-set lifecycle interacts
+with the existing queue/command-buffer structure, or something unverifiable without a real
+model run β€” flagged per the task's hard requirement.**
+
+1. **The central design risk: skipping `useResource:` in `moe_submit` gives up Metal's
+ automatic hazard tracking for that buffer set.** Apple's `MTLResidencySet.h` header
+ (read directly from this box's SDK at
+ `/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/.../Headers/MTLResidencySet.h`)
+ documents the protocol only in terms of residency; Apple's own "Simplifying GPU resource
+ management with residency sets" guide states plainly: *"You don't need to call
+ `useResource`/`useHeap`... for allocations in a residency set,"* and separately, the
+ `MTLResidencySet` class reference states *"Residency sets don't support hazard tracking,
+ so you need to account for hazards with fences and events."* I reasoned through every
+ code path that touches `moe_submit`'s `use` buffers (read-only, indirectly referenced,
+ never concurrently written, freed only after the engine's own slot lifecycle guarantees
+ no outstanding async reference) and concluded removing `useResource:` there specifically
+ is safe β€” but this reasoning is **not the same as having run the model**. If any code
+ path I didn't trace lets a slab get unregistered while an async `moe_block_begin` handle
+ is still in flight and reading it, this change removes a mitigation (weak as it may have
+ been) that existed before. **This is the #1 thing to watch for md5 divergence on**, and
+ the reason the scope was deliberately narrowed to `moe_submit` alone rather than applied
+ uniformly.
+2. **`resset_add`/`resset_remove`/`resset_flush` all run under `g_slab_mtx`, serializing
+ concurrent OMP loader threads' residency-set mutations against each other and against
+ dispatch's flush.** `MTLResidencySet`'s own header says *"all methods are non-threadsafe"*
+ (confirmed directly from the SDK header), so this serialization is required for
+ correctness, not optional β€” but it is structurally the same shape as the bug E4's
+ audit-round-2 found and fixed (mutex held across a live Metal call, serializing loader
+ threads during warmup fan-out, suspected root cause of E4's +12s regression). Apple's
+ guide frames `addAllocation:`/`commit` as lightweight bookkeeping relative to the actual
+ (deferred, async) page-in work ("Metal makes allocations resident when you call `commit()`
+ on the first command buffer using the set" β€” implying `commit()` itself doesn't
+ synchronously page anything in), which is why I judged holding the mutex across these
+ calls acceptable unlike E4's `newBufferWithLength:` (a real allocation call). **This is
+ unverified without profiling a loaded run** β€” if `commit()` or `addAllocation:` turns out
+ to be synchronously expensive on this hardware/OS build, this could reproduce E4's
+ expert-disk-load regression through a different code path, defeating E5's entire premise
+ (decoupling residency from the load path). Orchestrator: this is the single most important
+ number to check E5's load-path timing against stock, not just against E4.
+3. **`resset_flush()`'s cost sits outside `g_t_setup`/the `moe_times` breakdown** (it runs
+ before `ts_start = mnow()`), by design, to keep the harness's existing counters meaningful
+ β€” but this also means if `commit()` is expensive, it will show up as *general* wall-clock
+ slowdown / tok/s regression without a corresponding line in the existing instrumentation
+ pointing at it. No new counter was added for it (see "Instrumentation parity" above) to
+ avoid a `glm.c` touch; if E5 numbers look off without an obvious cause in the existing
+ breakdown, `resset_flush`/`commit()` cost is the first place to add a throwaway probe.
+4. **`initialCapacity = 4096` on the `MTLResidencySetDescriptor` is an unverified guess.**
+ It's documented as a presize hint only (no correctness effect either way), chosen to be
+ "clearly larger than the permanent-weight-tensor + KV-cache + plausible cap16 LRU-slab
+ count" without actually counting those registrations precisely. Too small just means
+ internal array growth; not a correctness concern, flagged only because it's a number I
+ picked without measuring.
+5. **Not calling `requestResidency()` proactively.** Apple's guide frames it as an optional
+ latency-hiding call ("call ahead of time during non-critical moments... to minimize [first
+ command buffer] latency"), and Blender's Cycles PR (the spec's cited reference
+ implementation) doesn't appear to use it either per its PR description. Omitted to keep
+ the lifecycle minimal and match the reference pattern; if profiling shows a
+ first-command-buffer-after-a-load-burst latency spike, this is the documented lever to try
+ next, not implemented here.
+6. **The deferred-commit correctness argument (item in "Deferred-commit design" above) rests
+ on a single-writer-before-single-reader program-order guarantee that is true today by
+ inspection but is not an invariant enforced anywhere in code** (no assertion, no type-level
+ guarantee) β€” it's the same kind of implicit ordering `resolve()` itself already depends on
+ for correctness (a slab must be registered before any dispatch can resolve its pointer),
+ so this diff doesn't introduce a new category of fragility, but it's worth naming
+ explicitly rather than leaving implicit.
+7. **Async `moe_block_begin`/`moe_block_end` overlap with concurrent `register()` calls**
+ (background loader threads registering new/different experts while an unrelated MoE block
+ is still in flight on the GPU) was reasoned through but never exercised in a real
+ concurrent stress scenario β€” the synthetic `metal-test` unit test's `run_moe` calls are
+ single-threaded and synchronous (`coli_metal_moe_block`, not the async `_begin`/`_end`
+ pair), so it does **not** cover this interleaving. The real engine's `PILOT`/prefetch and
+ `moe_block_begin`/`_end` overlap path is exactly the concurrency shape most likely to
+ expose a bug in this design if one exists, and is untested here by construction (out of
+ scope: no model runs).
+8. **`coli_metal_gemm` (prefill path) and `bind_gemv` (attention path) still call
+ `useResource:` unconditionally, so they get no CPU-overhead benefit from the residency set
+ even though their buffers are also set members.** This is deliberate (see "Why only
+ `moe_submit` skips" above) but means E5's win, if any, is scoped to the decode-path MoE
+ dispatch loop specifically β€” prefill and attention timing should be unaffected by the flag,
+ which is itself a testable prediction the orchestrator's harness can check.
+9. **API surface verified against this box's actual SDK headers**
+ (`MTLResidencySet.h`, `MTLDevice.h`, `MTLCommandQueue.h`, `MTLAllocation.h`,
+ `MTLResource.h` β€” all read directly, not from memory) and against Apple's own
+ "Simplifying GPU resource management with residency sets" guide, so the method names/
+ signatures (`newResidencySetWithDescriptor:error:`, `addResidencySet:`,
+ `removeResidencySet:`, `addAllocation:`, `removeAllocation:`, `commit`) are
+ high-confidence. What is **not** independently verified is runtime behavior beyond what
+ the docs state and what the synthetic unit test exercises β€” no substitute for the
+ orchestrator's real cap-sweep battery.
diff --git a/c/backend_metal.mm b/c/backend_metal.mm
index b57fa668..9c10533d 100644
--- a/c/backend_metal.mm
+++ b/c/backend_metal.mm
@@ -743,6 +743,7 @@ static id<MTLCommandBuffer> moe_submit(int nb, int D, int Iinter, int fmt,
const float *xg, const int *xoff, const int *nr, int R,
id<MTLBuffer> xg_buf, id<MTLBuffer> gg_buf, id<MTLBuffer> uu_buf, id<MTLBuffer> hh_buf) {
if (!g_dev || (fmt != 1 && fmt != 2)) return nil;
+ resset_flush(); // E5: commit any pending slab adds before we may skip useResource: below
double ts_start = mnow();
std::vector<uint64_t> ag(nb),au(nb),ad(nb),sgv(nb),suv(nb),sdv(nb);
std::vector<id<MTLBuffer>> use; use.reserve(nb*2);
@@ -764,7 +765,18 @@ static id<MTLCommandBuffer> moe_submit(int nb, int D, int Iinter, int fmt,
memcpy([xg_buf contents], xg, (size_t)R*D*4);
id<MTLCommandBuffer> cb=[g_queue commandBuffer]; id<MTLComputeCommandEncoder> e=[cb computeCommandEncoder];
- for(auto&b:use) [e useResource:b usage:MTLResourceUsageRead];
+ // E5 (COLI_METAL_RESSET=1): the queue-attached MTLResidencySet already guarantees these
+ // buffers are resident, so skip the per-buffer declaration whose count scales with LRU
+ // cache size (mechanism history v5). Residency sets don't do hazard tracking (Apple docs),
+ // but none was load-bearing here: every buffer in `use` is MTLResourceUsageRead-only and
+ // referenced only indirectly (moe_gemv dereferences waddr[]/saddr[] baked into bag/bsg's
+ // contents), so there's no GPU-side write to serialize against; the one real hazard -- a
+ // slab unregistered+freed+reused while an async in-flight CB still reads it -- is a
+ // CPU-write race outside Metal's hazard tracking either way, held by the engine's own slot
+ // lifecycle, not by useResource:. See SUMMARY.md UNCERTAINTIES.
+ if (!g_resset_enabled) {
+ for(auto&b:use) [e useResource:b usage:MTLResourceUsageRead];
+ }
auto gemv=[&](id<MTLBuffer> wa,id<MTLBuffer> sa,id<MTLBuffer> xin,id<MTLBuffer> y,int O,int K,int Kin){
int NT=R*O;
[e setComputePipelineState:g_moe_gemv];