Skip to content

Run frontier MoE models on hardware you already own β€” pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦

C

Quarantine EXPERT_BUDGET: the operating window is measured empty (#303)

JustVugg35f90b9e765e
diff --git a/c/glm.c b/c/glm.c
index ae18c554..d34606f1 100644
--- a/c/glm.c
+++ b/c/glm.c
@@ -2858,8 +2858,13 @@ static void moe(Model *m, Layer *l, int layer, float *x, int S, float *out, int
* only drop from misses. From the misses, keep the highest-aggregate-gate-weight
* ones up to the budget; drop the rest from idxs[] so they're never loaded.
* (MoE-Spec arXiv 2602.16052: top-32 of 64 capture 93% routing weight.)
- * Complementary to TOPP (per-position) β€” this trims cross-position. */
- if(g_expert_budget>0 && nu>g_expert_budget){
+ * Complementary to TOPP (per-position) β€” this trims cross-position.
+ * DECODE-ONLY (S<=4, incl. MTP verify): during prefill S=prompt_len the batch
+ * union nu is 30-100+ experts and capping to 4-8 drops 80-90% of them, each with
+ * non-trivial gate weight -> corrupted prefill hidden state -> wrong KV cache ->
+ * repetitive garbage decode. The budget is only safe token-by-token, where the
+ * prefill KV cache is already correct. (woolcoxm, #292.) */
+ if(g_expert_budget>0 && S<=4 && nu>g_expert_budget){
/* compute aggregate gate weight per unique expert */
float *wsum=falloc(nu); for(int j=0;j<nu;j++) wsum[j]=0;
for(int s=0;s<S;s++) for(int kk=0;kk<keff[s];kk++){
@@ -5759,7 +5764,31 @@ int main(int argc, char **argv){
if(g_mmap) fprintf(stderr,"[MMAP] expert = viste zero-copy nei file (page cache = cache)\n");
g_topk = getenv("TOPK")?atoi(getenv("TOPK")):0;
g_topp = getenv("TOPP")?atof(getenv("TOPP")):0;
+ /* EXPERT_BUDGET e' sotto quarantena: la finestra operativa e' misurata VUOTA.
+ * @bokiko su tre host (#303) e riprodotto qui su un 25 GB / WSL:
+ * - hellaswag 30% a budget=8 contro 90% a budget spento (25% = il caso);
+ * - a budget=4 il decode e' rumore ("The **1...: s2151:");
+ * - accettazione MTP 0%: quali expert sopravvivono al cap dipende dalla
+ * residenza in cache al momento del forward, quindi draft e verify NON
+ * calcolano la stessa funzione -- la stessa invariante che #294 ha appena
+ * stabilito, violata via stato di cache invece che via scelta del kernel;
+ * - 0.13 tok/s contro 0.30 di baseline, con 14.66 expert caricati per layer
+ * contro topk=8: il cap fa piu' I/O di quello che dice di risparmiare, e la
+ * riga "~N GB I/O saved" conta esperti scartati, non byte non letti.
+ * Resta compilato e sviluppabile (EXPERT_BUDGET_EXPERIMENTAL=1) perche' l'idea
+ * -- MoE-Spec, arXiv 2602.16052 -- non e' sbagliata: e' l'implementazione che
+ * finora non ha un punto in cui sia insieme piu' veloce e corretta. Riaccenderlo
+ * di default richiede una misura di qualita' accanto a quella di velocita'. */
g_expert_budget = getenv("EXPERT_BUDGET")?atoi(getenv("EXPERT_BUDGET")):0;
+ if(g_expert_budget>0 && !getenv("EXPERT_BUDGET_EXPERIMENTAL")){
+ fprintf(stderr,"[EXPERT_BUDGET] ignored: measured empty operating window (issue #303).\n"
+ "[EXPERT_BUDGET] every tested setting is either no faster or no longer coherent:\n"
+ "[EXPERT_BUDGET] budget=8 -> hellaswag 30%% (90%% with it off) | budget=4 -> decode is noise\n"
+ "[EXPERT_BUDGET] MTP acceptance 0%% (the cap breaks the draft/verify contract, #294)\n"
+ "[EXPERT_BUDGET] 0.13 tok/s vs 0.30 baseline, loading 14.7 experts/layer vs topk=8\n"
+ "[EXPERT_BUDGET] set EXPERT_BUDGET_EXPERIMENTAL=1 to run it anyway (expect garbage).\n");
+ g_expert_budget=0;
+ }
g_cache_route = getenv("CACHE_ROUTE")?atoi(getenv("CACHE_ROUTE")):0;
g_route_j = getenv("ROUTE_J")?atoi(getenv("ROUTE_J")):2;
g_route_m = getenv("ROUTE_M")?atoi(getenv("ROUTE_M")):12;
diff --git a/issue_budget.md b/issue_budget.md
deleted file mode 100644
index fe0ef86d..00000000
--- a/issue_budget.md
+++ /dev/null
@@ -1,150 +0,0 @@
-## TL;DR
-
-New `EXPERT_BUDGET=N` env var that caps the number of **distinct experts loaded per layer** across the batch-union. When the union exceeds the budget, keeps only the highest-aggregate-gate-weight experts and drops the rest β€” they're never loaded from disk. On a 24 GB RAM host (cache `cap=2`), `EXPERT_BUDGET=4` nearly **doubles decode tok/s** (0.18 β†’ 0.33) and **4x's prefill speed** (38.7s β†’ 8.9s). Based on MoE-Spec (arXiv 2602.16052): "top 32 of 64 experts capture 93% of routing weight."
-
-Branch: `experiment/expert-budget` (based on latest `dev` at `62419af`)
-
----
-
-## The problem
-
-Every expert miss costs ~19 MB of disk I/O. On low-RAM hosts where the LRU cache cap is tiny (e.g. `cap=2` on 24 GB RAM), nearly every routed expert is a miss. With `topk=8` and 75 sparse layers, a single-token decode reads ~8.5 GB of experts from disk. Under MTP with `S=4`, the batch-union can produce 20-32 distinct experts per layer β€” almost all misses β€” multiplying disk reads further.
-
-The README documents this directly:
-> "on a cold cache each verified draft routes to extra experts (~660 β†’ ~1100 expert-loads/token)"
-
-The existing `TOPP` env var trims experts *within a single position's top-K*. But it cannot reduce the **cross-position union** β€” the deduplication across batch positions (prefill, MTP verification) that multiplies disk loads. That's the gap this fills.
-
-## How it works
-
-The batch-union in `moe()` (line ~2306 of `glm.c`) deduplicates routed experts across all `S` positions into `uniq[0..nu)`. After the union is built but before the cache-resolve/load loop, the budget cap kicks in:
-
-```
-if(EXPERT_BUDGET > 0 && nu > EXPERT_BUDGET):
- 1. Compute aggregate gate weight per unique expert
- (sum of ws[s*K+kk] across all positions that route to it)
- 2. Sort experts by descending aggregate weight
- 3. Keep top EXPERT_BUDGET, mark rest as dropped
- 4. Remove dropped experts from each position's idxs[]/keff[]
- (decrement keff, renormalize remaining weights if norm_topk)
- 5. Compact uniq[] to kept experts only
-```
-
-Dropped experts are removed from `idxs[]` entirely, so they're **never resolved, never loaded from disk, never computed**. The downstream code already tolerates `keff[s] < K` (the `TOPP` path produces the same state), so this propagates correctly through resolve, matmul, and LRU promotion.
-
-**Relationship to existing features:**
-- `TOPP` trims within one position's top-K (per-position)
-- `EXPERT_BUDGET` trims across the cross-position union (per-layer)
-- They compose: `TOPP=0.8 EXPERT_BUDGET=12` first trims each position to its top-p mass, then caps the union
-
-## Code change
-
-**`c/glm.c`** β€” 54 lines added, single file:
-
-1. Global + counter (near existing `g_topk`/`g_topp`):
-```c
-static int g_expert_budget=0; /* EXPERT_BUDGET=N */
-static int64_t g_budget_dropped=0; /* total experts dropped */
-```
-
-2. Env var parsing (near existing `TOPK`/`TOPP`):
-```c
-g_expert_budget = getenv("EXPERT_BUDGET")?atoi(getenv("EXPERT_BUDGET")):0;
-```
-
-3. Budget cap in `moe()` batch-union (after `uniq[]` is built, before resolve loop) β€” full logic described above.
-
-4. Stats reporting alongside existing `TOPK`/`TOPP` line:
-```
-EXPERT_BUDGET=4 (dropped 13613 experts, ~257.3 GB I/O saved)
-```
-
-## Measurements
-
-**Test setup:** GLM-5.2 744B int4, 24 GB RAM (cache `cap=2`), Core Ultra 9 185H (AVX-VNNI), MTP=0, single-token decode, 32 tokens generated, same prompt: *"Explain the concept of recursion in programming. Provide a simple example."*
-
-| Config | tok/s | vs baseline | hit rate | prefill | decode | experts dropped | I/O saved |
-|--------|-------|-------------|----------|---------|--------|-----------------|-----------|
-| **Baseline** (budget=0) | 0.18 | β€” | 9.3% | 38.7s | 176.3s | 0 | 0 |
-| EXPERT_BUDGET=12 | 0.19 | +5% | 14.0% | 12.3s | 171.3s | 3,313 | 62.6 GB |
-| EXPERT_BUDGET=6 | 0.26 | +44% | 21.0% | 7.5s | 122.5s | 8,543 | 161.5 GB |
-| **EXPERT_BUDGET=4** | **0.33** | **+83%** | 16.4% | 8.9s | **97.4s** | 13,613 | 257.3 GB |
-
-### Profile breakdown (prefill)
-
-| Metric | Baseline | Budget=4 |
-|--------|----------|----------|
-| expert-disk service | 28.4s | **3.5s** (8x less) |
-| expert-matmul | 7.1s | 1.4s |
-| attention | 2.3s | 2.8s |
-
-### Profile breakdown (decode, 32 tokens)
-
-| Metric | Baseline | Budget=4 |
-|--------|----------|----------|
-| expert-disk service | 133.4s | proportional ~65s |
-| expert-matmul | 23.9s | ~12s |
-| decode total | 176.3s | **97.4s** |
-
-### Key observations
-
-1. **Prefill is the biggest winner.** With S=14 positions, the batch-union has 50+ distinct experts per layer. Budget=12 cuts that to 12, saving 40+ Γ— 19 MB Γ— 75 layers of disk reads. Prefill goes from 38.7s to 8.9s β€” **4.4x faster**.
-
-2. **Decode improvement scales with budget tightness.** Budget=12 barely constrains single-token decode (S=1 β†’ max 8 experts < 12), so decode is barely affected. Budget=4 actually halves the decode load (8 β†’ 4 experts/layer), nearly doubling tok/s.
-
-3. **Hit rate improves** because fewer experts compete for the tiny cache (cap=2). Budget=6 hit 21% vs baseline 9.3% β€” more than doubled.
-
-4. **No crashes or instability** at any budget value.
-
-## Quality impact β€” honest assessment
-
-**Budget=4 keeps only the top-4 of 8 routed experts per layer.** The dropped 4 experts had the lowest gate weights β€” they contributed the least to the output. But this IS a quality trade-off.
-
-On a **cold cache** (our test setup), quality assessment is confounded: both baseline and budget outputs are already garbled from int4 quantization + cold-cache routing. Sample comparison:
-
-- **Baseline output:** *"Hire Some To Take Object-Oriented Programming Assignment"*
-- **Budget=4 output:** *"The world is a dangerous place, not so much because of the small percentage of people who are doing to do the little of people who are doing to"*
-
-Both are incoherent β€” the cold cache at int4 on 24 GB RAM produces poor output regardless of budget. The budget=4 output is **not dramatically worse** than the already-poor baseline β€” they're both garbled, just differently garbled. A proper quality assessment needs a **warm cache** where the baseline produces coherent text, so you can isolate the degradation from dropping experts.
-
-**Expected quality on warm cache:** With `norm_topk=1` (GLM-5.2 renormalizes gate weights), the top-4 experts typically capture ~80-85% of routing weight (the remaining 4 share 15-20%). Output should be mostly coherent with occasional word-choice degradation. Budget=6 (top-6 of 8, ~90%+ weight captured) should be nearly indistinguishable from baseline.
-
-**Recommendation for users:**
-- `EXPERT_BUDGET=6-8` on cold/low-RAM hosts β€” good speedup, minimal quality loss
-- `EXPERT_BUDGET=4` on very-low-RAM hosts where speed matters more than quality
-- Leave OFF on high-RAM hosts where all experts are resident anyway (budget never triggers)
-
-## Safety
-
-- **Default OFF** (`EXPERT_BUDGET=0`) β€” zero behavior change unless explicitly set
-- **Opt-in quality trade-off** β€” same design philosophy as `TOPP`: the user explicitly chooses speed over quality
-- **No output corruption** β€” dropped experts' contributions are omitted and remaining weights renormalized (identical math to `TOPP`)
-- **Compatible with all features** β€” PILOT, MTP, CUDA, CACHE_ROUTE, PIPE, TOPP all work unchanged; they just see fewer experts in the union
-- **No crash risk** β€” no new memory allocation patterns, no new I/O, purely a filter on existing data structures
-
-## Reproduce
-
-```bash
-make ARCH=native
-
-# Baseline:
-SNAP=<model_dir> MTP=0 PROMPT="Explain recursion in programming." NGEN=32 ./glm 64
-
-# With budget:
-SNAP=<model_dir> MTP=0 EXPERT_BUDGET=4 PROMPT="Explain recursion in programming." NGEN=32 ./glm 64
-
-# With MTP (where budget saves the most):
-SNAP=<model_dir> MTP=1 EXPERT_BUDGET=16 PROMPT="Explain recursion in programming." NGEN=32 ./glm 64
-```
-
-The stats line prints `EXPERT_BUDGET=N (dropped X experts, ~Y GB I/O saved)`.
-
-## What's needed before merge
-
-1. **Warm-cache quality A/B test** on a host with enough RAM (cap>=16) to produce coherent baseline output, then compare budget=4/6/8 text quality
-2. **MTP interaction test** β€” verify that budget + MTP composes correctly and doesn't crash when draft verification routes to budgeted-away experts
-3. **Decide on defaults** β€” should this auto-activate on low-RAM hosts (like `cap` auto-lowering)? My recommendation: no, leave it OFF and document the recommended value per RAM tier
-
-## Prior art
-
-**MoE-Spec** (arXiv 2602.16052) β€” "Expert Budgeting for Efficient Speculative Decoding": Training-free expert budgeting at verification time. Key finding: "top 32 of 64 experts capture 93% of routing weight." Our approach applies the same principle but at the batch-union level (not just MTP verification), making it effective for prefill and single-token decode too.
diff --git a/issue_diskio.md b/issue_diskio.md
deleted file mode 100644
index 91d7b92c..00000000
--- a/issue_diskio.md
+++ /dev/null
@@ -1,309 +0,0 @@
-# Disk I/O Minimization β€” Research
-
-Branch: `experiment/diskio-research` (based on `dev` at `62419af`)
-
-## TL;DR
-
-The engine's disk I/O is **already well-engineered on the hottest path** (expert streaming uses coalesced O_DIRECT `pread` + `posix_fadvise` hints + LRU + pin cache + speculative prefetch). There are **4 concrete, bounded opportunities** to shave latency, ranked by ROI:
-
-| # | Opportunity | Where | Frequency | Estimated win |
-|---|---|---|---|---|
-| 1 | **KV-cache write batching** (157 fwrites/token β†’ 1) | `kv_disk_append` | per turn | cuts ~100s of syscalls/turn |
-| 2 | **`/proc/meminfo` fopen storm** | `rss_gb()` | ~every 16 tokens (Linux) | eliminates recurring open/read/close |
-| 3 | **Expert prefetch on Windows** (`PrefetchVirtualMemory`) | `expert_prefetch` | per miss | mmap path is Linux/macOS-only today |
-| 4 | **KV-cache: buffered handle kept open** | `kv_disk_append` | per turn | kills open+fseek+close per turn |
-
-There are also **2 non-opportunities** worth recording so we don't re-investigate: O_DIRECT for experts (correctly used today), and PagedAttention-style file layout (already single-file + indexed).
-
----
-
-## How the engine does disk I/O today
-
-There are **three I/O stacks**, behaving very differently:
-
-| Stack | Mechanism | Frequency | Files |
-|---|---|---|---|
-| **Expert weights** (hottest) | `pread` on kept-open fds + `posix_fadvise`, optional `mmap` | per miss, every token | `st.h`, `glm.c:1328` |
-| **KV cache** (`.coli_kv`) | `fopen` + `fwrite`/`fread` | per turn | `glm.c:3812-3889` |
-| **Everything else** (config, tokenizer, stats, grammar) | `fopen` + `fread` | startup-only | scattered |
-
-### Expert path (the hot path β€” already good)
-
-`expert_load` (`glm.c:1328-1481`) has three sub-paths:
-
-- **Default `pread` path** (`glm.c:1385-1472`): coalesces the 3 contiguous expert tensors (gate/up/down) into **one ~19 MB O_DIRECT `pread`** into a 16K-aligned slab (`glm.c:1447`). Falls back to 3 separate `pread`s only if non-contiguous. Scales are 3 tiny separate `pread`s (kilobytes). `posix_fadvise(DONTNEED)` evicts pages after if `g_drop`. **This is well-batched β€” one syscall for ~19 MB.**
-- **`COLI_MMAP=1` path** (`glm.c:1352-1383`): `mmap` per shard fd (cached), `madvise(WILLNEED)` + synchronous page-touch loop. Zero-copy. **Default OFF, and Linux/macOS/FreeBSD-only** β€” no `MapViewOfFile` on Windows.
-- **Prefetch hints**: `expert_prefetch` (`glm.c:1602-1609`) β†’ `st_prefetch` (`st.h:178`) issues `posix_fadvise(WILLNEED)` β€” readahead hint only, no data read. Called from `moe` next-64-block lookahead, pilot, and SPEC.
-
-### KV cache persistence (per turn β€” opportunity here)
-
-`kv_disk_append` (`glm.c:3834-3855`), called once per turn:
-1. `fopen("r+b")` β€” **reopens the file every turn**
-2. `fseek` to append position
-3. **per-position loop**: for each new token, `fwrite` the token i32, then **2 fwrites per layer** (Lc + Rc) + optional DSA Ic. With 78 layers that's **~157 fwrites per token appended**.
-4. `fflush` (userspace only β€” **no fsync/fdatasync anywhere in the codebase**)
-5. `fseek` back to header + `fwrite` the new nrec counter (crash-safe ordering)
-6. `fclose`
-
-Record size ~182 KB/token. On a long first turn this is **tens of thousands of small fwrites**. stdio buffering coalesces them into fewer `write` syscalls, but the userspace overhead remains.
-
-### Recurring surprise: `/proc/meminfo`
-
-`rss_gb()` (`glm.c:4625`) does `fopen("/proc/meminfo")` + fgets + fclose. Called from every STAT line and every 16-token heartbeat (`glm.c:3473, 3477`). On Linux this is an **open+read+close of procfs ~every 16 tokens**. (Windows uses `compat_meminfo`, no file β€” not affected.)
-
----
-
-## What similar projects do
-
-**llama.cpp** (the reference): `mmap`s the entire model read-only, uses `--mlock` to pin hot pages, streams layers to GPU via partial offload, and issues per-pass readahead of upcoming tensors (`llama-mmap.cpp`). Justine Tunney's mmap work: "load 100Γ— faster using half as memory." Crucial finding from discussion #18758: **for MoE, mmap beats O_DIRECT** when the model fits in ~RAM β€” O_DIRECT takes "at least 10Γ— longer" on repeated loads because it bypasses the page cache that serves re-faults for free.
-
-**The general consensus across llama.cpp, vLLM, AirLLM, PRESERVE, HOBBIT, SolidAttention (FAST '26):**
-- mmap + OS page cache as the backing store for an LRU is the proven recipe
-- prefetch the *next* expert/layer while computing the current one β€” this is where the 0.5ms lives
-- single indexed file (one `open()`) beats one-file-per-expert
-- align tensors to 4KB (preferably 64KB) for clean page-fault boundaries + SSD geometry
-- buffer sweet spot ~1MB; syscall cost ~1-5Β΅s each, so batching matters at high repetition
-
----
-
-## The 4 opportunities (ranked)
-
-### Opportunity 1 β€” KV-cache write batching (HIGH ROI, LOW risk)
-
-**Problem:** `kv_disk_append` does ~157 `fwrite` calls per appended token (1 token i32 + 2Γ—78 layers). stdio buffering hides some of this, but on a long first-turn prefill (hundreds-thousands of tokens) this is tens of thousands of fwrites.
-
-**Fix:** Build one contiguous record in a heap buffer (token + all layers' Lc/Rc/Ic for that position), then **a single `fwrite` per position** (or even one `fwrite` for the whole turn). The data is already laid out contiguously in memory per-layer (`coli_kv_row`), so a layered `memcpy` into a staging buffer + one write is straightforward.
-
-**Win:** ~157Γ— fewer fwrite calls per token. Even with stdio coalescing, the userspace loop overhead is real at scale.
-
-### Opportunity 2 β€” `/proc/meminfo` fopen storm (MEDIUM ROI, trivial)
-
-**Problem:** `rss_gb()` opens, reads, closes `/proc/meminfo` every ~16 tokens on Linux. Each is ~3 syscalls + path resolution.
-
-**Fix:** Either (a) cache the value for N tokens (e.g. re-read at most once per second), or (b) keep the fd open and `rewind`+`fgets`. Trivial change.
-
-### Opportunity 3 β€” Expert prefetch on Windows (MEDIUM ROI, bounded)
-
-**Problem:** The `COLI_MMAP=1` path (which gives zero-copy expert access + free OS-cache re-faults) is **Linux/macOS/FreeBSD-only** β€” `glm.c:1301` guards it. On Windows, experts always go through the `pread` path, and `expert_prefetch` issues `posix_fadvise(WILLNEED)` which is a no-op shim on Windows (`compat.h`).
-
-**Fix:** On Windows, implement the prefetch via `PrefetchVirtualMemory` (the Win32 analog of `MADV_WILLNEED`) on an mmap'd region, or via an async `ReadFile`+`OVERLAPPED` into a scratch buffer. This brings the Windows build closer to parity with the Linux mmap+prefetch story.
-
-**Scope:** This is the largest of the four β€” it touches the Windows I/O path. Worth doing if Windows perf is a goal; skip if Linux is the target.
-
-### Opportunity 4 β€” KV-cache: keep handle open (LOW-MEDIUM ROI, LOW risk)
-
-**Problem:** `kv_disk_append` does `fopen`+...+`fclose` every turn. Handle creation is ~5-15Β΅s of pure overhead (worse on Windows).
-
-**Fix:** Open the KV file once (lazily on first append), keep the `FILE*` for the engine lifetime, just `fseek`+write each turn. Close on shutdown. Pair with Opportunity 1 for the write batching.
-
----
-
-## Non-opportunities (recording so we don't re-investigate)
-
-- **O_DIRECT for experts**: already correctly used (`st.h:83`, `DIRECT=1`). For an LRU+refetch pattern the page cache is your friend, but the engine offers both paths (O_DIRECT pread default + optional mmap) and the O_DIRECT coalesced read is already one syscall for ~19MB. Don't change this.
-- **Single-file layout**: the engine already uses safetensors shards with kept-open fds + offset-indexed tensors (`st.h`). No per-expert open()/close() waste. Don't change this.
-- **PagedAttention**: solves concurrency fragmentation this engine doesn't have (≀16 slots). Not applicable.
-
----
-
-## Next steps
-
-The highest-ROI, lowest-risk starting point is **Opportunity 1 (KV write batching) + Opportunity 4 (keep handle open)** β€” they're in the same function, both low-risk, and together they eliminate the per-turn open/close overhead and the per-token fwrite storm. Opportunity 2 is a trivial 5-minute fix we can bundle in.
-
-Opportunity 3 (Windows prefetch) is the biggest single win but also the largest scope β€” separate effort, gated on whether Windows perf is a priority.
-
-## Sources
-
-- [justine.lol/mmap β€” Edge AI Just Got Faster](https://justine.lol/mmap/)
-- [llama.cpp discussion #18758 β€” Mmap faster than direct I/O for MoE](https://github.com/ggml-org/llama.cpp/discussions/18758)
-- [llama.cpp issue #20757 β€” Two-tier GPU+RAM expert cache](https://github.com/ggml-org/llama.cpp/issues/20757)
-- [FAST '26 β€” Programmable Page Cache for LLM loading](https://www.usenix.org/system/files/fast26-liu-yubo.pdf)
-- [FAST '26 β€” SolidAttention: SSD-based serving](https://www.usenix.org/system/files/fast26-zheng.pdf)
-- [HOBBIT β€” Mixed precision expert offloading](https://arxiv.org/html/2411.01433v2)
-- [posix_fadvise(2) β€” man7.org](https://man7.org/linux/man-pages/man2/posix_fadvise.2.html)
-- [madvise(2) β€” man7.org](https://man7.org/linux/man-pages/man2/madvise.2.html)
-- [Microsoft Learn β€” File Buffering (FILE_FLAG_NO_BUFFERING)](https://learn.microsoft.com/en-us/windows/win32/fileio/file-buffering)
-- [Microsoft Learn β€” PrefetchVirtualMemory](https://learn.microsoft.com/en-us/windows/win32/api/memoryapi/nf-memoryapi-prefetchvirtualmemory)
-- [What makes system calls expensive β€” codingconfessions.com](https://blog.codingconfessions.com/p/what-makes-system-calls-expensive)
-- [Syscall overhead β€” Stack Overflow](https://stackoverflow.com/questions/8247331/syscall-overhead)
-
----
-
-# Windows Implementation β€” branch `windows-optimizations` (2026-07-15)
-
-## What landed (pread path, validated)
-
-Two changes, both on the `pread` expert-load path (no mmap). Measured against the
-existing `bench_budget*.txt` baselines (GLM-5.2 744B int4, 32 GB RAM, Core Ultra 9
-185H, DRAFT=0, 32-token decode):
-
-### 1. `compat_fadvise` WILLNEED cache-warmer (`c/compat.h`)
-
-Replaced the Windows `posix_fadvise` no-op (was a `do{}while(0)` macro) with a real
-readahead: an overlapped `ReadFile` into a throwaway scratch buffer that populates the
-standby page cache, so the later synchronous `pread` faults from RAM not disk. Mirrors
-the macOS `F_RDADVISE` shim (`compat.h:28-37`). DONTNEED stays a no-op (matches macOS;
-Windows standby-list trimming self-regulates under pressure).
-
-This re-arms the existing `expert_prefetch` β†’ `st_prefetch` β†’ `posix_fadvise(WILLNEED)`
-chain on Windows: the next-block readahead in `moe()` and the PILOT cross-layer prefetch
-hints now actually warm the cache instead of being silently discarded.
-
-**Measured effect (budget=4, PIPE on):** hit rate 16.4% β†’ 27.6%.
-
-### 2. PIPE default ON for Windows (`c/glm.c`)
-
-Flipped the async expert-load thread pool from default OFF to default ON on Windows
-(`getenv("PIPE")?:1` under `_WIN32`, unchanged `:0` elsewhere). PIPE dispatches expert
-`pread` loads onto worker threads so they overlap the expert matmul on the forward-pass
-thread, instead of the blocking serial load-then-compute path. `PIPE=0` opts back out.
-
-**Measured effect (budget=4):** expert-disk 65.9s β†’ 54.3s (βˆ’18%), reaching **1.70 s/tok**
-(under the 2 s/tok target; budget=4 baseline was 2.06 s/tok).
-
-### Results table (DRAFT=0, 32-token decode, pread path)
-
-| config | expert-disk | s/tok | hit% | tok/s |
-|---|---|---|---|---|
-| budget=4, no PIPE (existing baseline) | 65.9s | 2.06 | 16.4% | 0.33 |
-| **budget=4 + PIPE (this PR)** | **54.3s** | **1.70** | **27.6%** | **0.34** |
-| budget=6 + PIPE | 77.1s | 2.41 | 21.8% | 0.27 |
-
-budget=4 + PIPE meets the ≀2 s/tok target. budget=6 (more experts/layer, higher quality)
-misses it at 2.41 s/tok β€” the speed/quality tradeoff.
-
-## What was tried and abandoned: Windows mmap (`COLI_MMAP` on `_WIN32`)
-
-The original plan (informed by llama.cpp #18758: "mmap is β‰₯10Γ— faster than O_DIRECT for
-MoE") was to port the mmap expert path to Windows via `CreateFileMapping`/`MapViewOfFile`.
-This was implemented and tested at length. **It was a measured regression and was reverted.**
-
-### The attempt
-
-Added a `_WIN32` branch to `map_of_fd` (`glm.c`) mapping each shard file read-only and
-resolving experts as views into the mapping, mirroring the POSIX path. Also added
-`PrefetchVirtualMemory` readahead and a `VirtualUnlock` eviction mechanism (the Windows
-`posix_fadvise(DONTNEED)` analog β€” see SO#1880714; validated standalone to demote pages
-to the standby list with a 2.3Γ— faster re-fault).
-
-### Why it regressed
-
-**mmap'd expert pages bloat the process working set on Windows, which collapses the
-expert cache.** This is a fundamental Windows-vs-Linux difference:
-
-- On Linux, `mmap(MAP_SHARED)` file pages live in the kernel page cache (`buff/cache`),
- separate from `MemAvailable`, so the cache budget isn't fooled.
-- On Windows, touched `MapViewOfFile` pages count against `ullAvailPhys` (what
- `compat_meminfo` reads for the budget). The CPU matmul touches every weight byte,
- faulting ~12 GB into the working set. `cap_for_ram()` then sees ~no free RAM and
- collapses the LRU cache cap.
-
-Measured (budget=0, DRAFT=0, apples-to-apples):
-
-| config | RAM_GB detected | cache cap | hit% | expert-disk | RSS |
-|---|---|---|---|---|---|
-| baseline (pread) | 21.4 | 1 | 9.3% | 133s | 15.0 GB |
-| mmap, no eviction | **8.0** | 1 | **2.2%** | **240s** | **27.2 GB** |
-| mmap + VirtualUnlock | 24.9 | 2 | 11.8% | 83s | 18.1 GB |
-| mmap + reserve reductions | 24.6 | 4 | 21.8% | 80s | 20.1 GB |
-
-The `VirtualUnlock` eviction recovered the regression (240s→83s), and dropping the
-Linux-specific page-cache/slab reserves under mmap got it to parity with pread. But it
-never clearly *beat* the simpler pread+PIPE path, and it added substantial complexity
-(per-slot eviction tracking, reserve conditionals, `VirtualUnlock` on every slot recycle).
-**The engine already moved off mmap to pread for this exact RSS bug** (`st.h:3-6`), and
-the Windows port re-confirmed that decision.
-
-### What else didn't work
-
-- **Batched `PrefetchVirtualMemory`** for the mmap path: tested as a single batched
- readahead of all 64 missed experts' pages before the matmul. **Blocked instead of
- prefetching async** on this SSD — inflated `t_edisk` (80s→102s). Consistent with
- microsoft/Windows-Dev-Performance#108 ("PrefetchVirtualMemory does not prefetch").
- Reverted.
-- **True I/O/compute overlap on the CPU path**: the Metal path has this ("submit
- resident experts to GPU before loading misses"), but the CPU path loads-then-computes
- serially. `PrefetchVirtualMemory` was the attempt to add it for mmap and failed. The
- pread path gets overlap via PIPE (which works), not via mmap prefetch.
-
-### Conclusion
-
-For this engine on Windows at this RAM budget (~32 GB, 370 GB model), **pread + PIPE +
-compat_fadvise** is the right path. mmap remains valuable on Linux/macOS (where the page
-cache doesn't inflate process RSS) but is not viable on Windows without a fundamentally
-different cache-budget model that excludes mapped-file pages β€” left as future work.
-
-## Sources added
-
-- [SO#1880714 β€” VirtualUnlock releases mapped pages to standby list](https://stackoverflow.com/questions/1880714/createfilemapping-mapviewoffile-how-to-avoid-holding-up-the-system-memory)
-- [Alois Kraus β€” The Mysterious Lost Memory (modified/standby list)](https://aloiskraus.wordpress.com/2017/02/26/the-mysterious-lost-memory-which-belongs-to-no-process/)
-- [microsoft/Windows-Dev-Performance#108 β€” PrefetchVirtualMemory inconsistency](https://github.com/microsoft/Windows-Dev-Performance/issues/108)
-- [llama.cpp #18758 β€” mmap faster than O_DIRECT for MoE (Linux)](https://github.com/ggml-org/llama.cpp/discussions/18758)
-- [HN#35426679 β€” Why MMAP in llama.cpp hides true memory usage](https://news.ycombinator.com/item?id=35426679)
-
----
-
-# CACHE_ROUTE: miss elimination via cache-aware routing (2026-07-15)
-
-## The breakthrough
-
-Adding `CACHE_ROUTE=1 ROUTE_J=2 ROUTE_M=12` to the optimized stack pushed
-throughput to **1.41 tok/s** (4.3Γ— over stock) by directly reducing the
-miss rate from 27% to 17%.
-
-## How it works
-
-The engine's `CACHE_ROUTE` feature (paper: max-rank routing, arXiv 2412.00099)
-steers the MoE router to prefer experts that are already cache-resident:
-
-- `ROUTE_J=2`: keep the top-2 true router picks (always, even if uncached)
-- `ROUTE_M=12`: fill the remaining 2 of 4 slots with the highest-ranked experts
- that are ALREADY in the LRU cache (from the top-12 candidates)
-
-This guarantees 2 of 4 expert slots per layer per token are cache hits. The
-`route_agree=94.6%` metric confirms minimal quality cost β€” 94.6% of cache-steered
-picks match the true top-K the router would have chosen.
-
-## Why this is the right fix for the miss problem
-
-Analysis of route traces showed GLM-5.2's routing is nearly uniform β€” 75 tokens
-use 237 of 256 experts per layer, with the top-4 capturing only 4.3% of selections.
-This means:
-
-- PIN (hot-expert pre-loading) is ineffective β€” there are no hot experts
-- PILOT_REAL prefetch can't keep up under the fast pipe2 GPU pipeline
-- A bigger cache helps marginally but can't cover 237 unique experts per layer
-- CACHE_ROUTE is the only lever that reduces misses without more RAM or faster disk
-
-## Results (pipe2 + full stack + ws_b fix, budget=4, RAM_GB=28)
-
-| metric | without CACHE_ROUTE | with CACHE_ROUTE |
-|---|---|---|
-| tok/s | 1.03 | **1.41** (+37%) |
-| hit rate | 73% | **83%** |
-| expert-disk | 12.4s | **8.5s** (βˆ’31%) |
-| decode | 31.0s | **22.7s** (βˆ’27%) |
-| route_agree | n/a | 94.6% |
-
-## Full optimization journey
-
-| step | tok/s | hit% | disk |
-|---|---|---|---|
-| stock budget=4 | 0.33 | 9% | 65.9s |
-| + disk stack | 0.63 | 72% | 21.5s |
-| + CUDA dense+attn | 0.72 | 75% | 18.4s |
-| + pipe2 GPU pipeline | 0.85 | 57% | 18.2s |
-| + ws_b cache fix | 1.03 | 73% | 12.4s |
-| **+ CACHE_ROUTE** | **1.41** | **83%** | **8.5s** |
-
-**4.3Γ— total speedup.** Disk I/O reduced 7.7Γ— (65.9s β†’ 8.5s).
-
-## Recommended config
-
-```
-EXPERT_BUDGET=4 PIPE=1 RAM_GB=28 PILOT_REAL=1 DIRECT=1
-COLI_CUDA=1 CUDA_DENSE=1 COLI_CUDA_ATTN=1 COLI_CUDA_PIPE=2 CUDA_EXPERT_GB=0
-CACHE_ROUTE=1 ROUTE_J=2 ROUTE_M=12
-```
-
-CACHE_ROUTE is an existing engine feature (no code change) β€” opt-in via env vars.
diff --git a/issue_grouped_quant.md b/issue_grouped_quant.md
deleted file mode 100644
index fb46ce4f..00000000
--- a/issue_grouped_quant.md
+++ /dev/null
@@ -1,185 +0,0 @@
-## TL;DR
-
-The int4 model produces **fluent-but-incoherent output** β€” grammatical English that is completely off-topic (SEO spam, homework-help boilerplate) instead of reasoned responses. The root cause is **per-row quantization scales**: one F32 scale per output row (e.g. 2048 scales for a 2048Γ—6144 matrix), which is 48x coarser than the FP8 source's 128Γ—128 block scales. This destroys fine-grained weight information that handles reasoning and instruction-following while preserving the large-magnitude weights that handle grammar and vocabulary fluency.
-
-Branch: `experiment/grouped-quant` (based on latest `dev` at `62419af`)
-
----
-
-## The problem β€” demonstrated
-
-Prompt: *"Explain the concept of recursion in programming. Provide a simple example."*
-
-**Baseline (no budget, no changes):**
-> "Hire Some To Take Object-Oriented Programming Assignment"
-
-**Budget=4:**
-> "The world is a dangerous place, not so much because of the small percentage of people who are doing to do the little of people who are doing to"
-
-**Budget=6:**
-> "Elite Custom Essays"
-
-**Budget=12:**
-> "Sololearn: Learn to code for FREE! +1 # Explain A concept of recursion..."
-
-This is not random gibberish β€” it's fluent English that is completely off-topic. This is the signature of **activations corrupted by coarse quantization**: the model retains language fluency (large weights survive) but loses instruction-following and reasoning (fine-grained weights are crushed to zero).
-
-## Root cause: per-row int4 scales
-
-### How the current converter works
-
-`convert_fp8_to_int4.py` (line 39-52) quantizes with **one scale per output row**:
-
-```python
-amax = np.abs(w).max(axis=1, keepdims=True) # one max per ROW
-s = np.maximum(amax / qmax, 1e-8) # one scale per ROW
-q = np.clip(np.rint(w / s), -8, qmax) # quantize entire row with that scale
-```
-
-For a 2048Γ—6144 expert weight matrix, that's **2048 scales** β€” one per row, each covering 6144 elements.
-
-### Why this destroys quality
-
-Consider a row of 6144 values where most are small (magnitude ~0.01) but a few are large (~0.5). The per-row scale is `0.5/7 = 0.071`. The small values become `0.01/0.071 = 0.14`, which rounds to **zero**. All fine-grained information in those small weights is lost.
-
-This matters enormously for MoE models: each token activates only 8 of 256 experts. A poorly-quantized expert pollutes every token that routes to it, and there's no averaging from the other 248 experts (they're simply off). The error is **concentrated**, not diluted.
-
-### What the FP8 source does right
-
-The FP8 checkpoint uses **128Γ—128 block scales** β€” the weight matrix is divided into 128-element chunks, each with its own scale. Small values in one chunk don't get crushed by large values in another. For a 2048Γ—6144 matrix: `2048 Γ— 48 = 98,304` scales β€” 48x more granularity.
-
-The converter already dequants FP8 to f32 correctly (lines 196-201), preserving the block-scale information. But then it **throws it all away** by collapsing to a single per-row scale during int4 requantization.
-
-### GLM-5.2 was QAT-trained for int4
-
-The GLM-5 paper (arXiv 2602.15763, Β§2.4.3) states: *"To provide better accuracy at low-precision, we apply INT4 QAT in the SFT stage."* This means the model is **designed** to work at int4 β€” but only if the quantization is fine-grained enough to match what the model saw during training. The FP8 checkpoint's 128Γ—128 block scales are the granularity the model expects. Per-row scaling is 48x coarser.
-
-### Community confirmation
-
-Every other project getting coherent GLM-5.2 int4 output uses calibrated or fine-grained quantization:
-- **ubergarm/GLM-5.1-GGUF** β€” imatrix-calibrated with expert-specific patches
-- **Unsloth/GLM-5-GGUF** β€” dynamic UD-Q4 quants
-- **QuantTrio/GLM-5.2-Int4-Int8Mix** β€” mixed int4/int8 with channel-wise scales
-- **llama.cpp** β€” Q4_K with block-level group scales (typically 32 or 64 elements)
-
-None use naive per-row RTN int4 for production inference.
-
-## The fix: group-scaled int4 (fmt=4)
-
-Add a new quantization format with **one scale per 128 elements** along the input dimension, matching the FP8 source's natural granularity.
-
-### What changed
-
-**`c/glm.c`** β€” 122 lines added:
-
-1. **QT struct** (line 98): Added `int gs` field (group size, 0=per-row for backward compat, 128=grouped).
-
-2. **Format detection** (`qt_from_disk`, line 1064): Auto-detects fmt=4 by checking the `.qs` scale array size. If it has `O * ceil(I/128)` elements instead of `O`, it's grouped. **Old per-row models (fmt=2) work unchanged.**
-
-3. **New kernel** (`matmul_i4_grouped`, line 379): Same AVX2 nibble unpacking as `matmul_i4`, but the accumulator resets at each 128-element group boundary: `dot(x[grp], w[grp]) * scale[grp]`. The scale changes every 8 vector iterations (128/16=8).
-
-4. **Dispatch** (`matmul_qt_ex`, line 813): Routes to `matmul_i4_grouped` when `fmt==4`. Always uses exact kernels (no IDOT approximation β€” the whole point is quality).
-
-5. **Expert loading** (`expert_load`, lines 1418 and 1538): Both the mmap and slab+pread paths detect fmt=4 from scale array size and set `gs=128`.
-
-6. **`qt_bytes`** (line 104): Reports correct memory for fmt=4.
-
-**`c/tools/convert_fp8_to_int4.py`** β€” `quant_int4_grouped()` + `--group-size` arg:
-
-```python
-def quant_int4_grouped(w, bits, gs=128):
- O, I = w.shape
- ngroups = (I + gs - 1) // gs
- wpad = np.zeros((O, ngroups * gs), np.float32)
- wpad[:, :I] = w
- wr = wpad.reshape(O, ngroups, gs) # [O, ngroups, gs]
- amax = np.abs(wr).max(axis=2, keepdims=True) # one max per GROUP
- s = np.maximum(amax / qmax, 1e-8)
- q = np.clip(np.rint(wr / s), -8, qmax)
- # ... same nibble packing as quant_int4 ...
- return packed_nibbles, s.reshape(-1) # [O * ngroups] scales
-```
-
-Same packed-nibble format as existing int4 β€” only the scale array is larger.
-
-### Verification
-
-**Converter round-trip test** (weights with varying group magnitudes):
-
-| Method | Mean relative error | Max abs error |
-|--------|-------------------|---------------|
-| Per-row int4 (current) | 0.2056 | 0.0127 |
-| Grouped int4 (gs=128) | **0.1278** | 0.0127 |
-| **Improvement** | **1.6x lower** | β€” |
-
-**Engine kernel test** (AVX2 `matmul_i4_grouped` vs f32 reference):
-
-| Output | Reference | Kernel | Error |
-|--------|-----------|--------|-------|
-| o=0 | -0.126368 | -0.126368 | 2.98e-08 |
-| o=1 | 0.075913 | 0.075913 | 1.49e-08 |
-| o=2 | 0.077136 | 0.077136 | 7.45e-09 |
-| ... | ... | ... | ... |
-
-Max error: **2.98e-08** β€” matches f32 reference to within float32 epsilon. **PASS.**
-
-### Cost
-
-| Metric | Per-row (current) | Grouped (new) |
-|--------|-------------------|---------------|
-| Expert weight size | ~18.9 MB | ~20.1 MB (+6%) |
-| Scale array per matrix | O Γ— 4 bytes | O Γ— ceil(I/128) Γ— 4 bytes |
-| Total model size | ~370 GB | ~390 GB (+5%) |
-| Disk I/O per miss | ~19 MB | ~20 MB (+6%) |
-| Kernel speed | One scale multiply per row | One scale multiply per 128 elements |
-
-The 6% I/O increase is negligible compared to the quality gain. The grouped kernel has slightly more scale-lookup overhead but remains within AVX2 throughput β€” the bottleneck is disk I/O, not matmul.
-
-### Backward compatibility
-
-- Old per-row models (fmt=2) **work unchanged** β€” format auto-detected from scale array size
-- All existing features (PILOT, EXPERT_BUDGET, CUDA, MTP, PIPE) work identically β€” they don't touch the dequant path
-- The fused gate+up pair path (`matmul_i4_pair`) falls back to separate `matmul_qt` calls for fmt=4 β€” minor perf cost, correctness preserved
-- `--group-size 0` produces per-row output (backward compat for the converter)
-
-## How to reproduce
-
-### Convert with group scales
-
-```bash
-python tools/convert_fp8_to_int4.py \
- --indir /path/to/GLM-5.2-FP8 \
- --outdir /path/to/glm52_i4_grouped \
- --ebits 4 --io-bits 8 --group-size 128
-```
-
-### Test quality
-
-```bash
-# Old model (per-row):
-SNAP=/path/to/glm52_i4 PROMPT="Explain recursion in programming." NGEN=32 ./glm 64
-
-# New model (grouped):
-SNAP=/path/to/glm52_i4_grouped PROMPT="Explain recursion in programming." NGEN=32 ./glm 64
-```
-
-Compare output text coherence.
-
-## What this won't fix
-
-- **Cold cache speed** β€” still 0.18-0.36 tok/s on 24 GB RAM. Quality and speed are independent axes.
-- **All quantization error** β€” int4 is still 4-bit. Some degradation vs FP8 will remain. But it should be coherent degradation (slightly wrong word choices) rather than total reasoning failure (unrelated topics).
-- **The IDOT +12% perplexity** β€” the approximate activation kernel (`IDOT=1`, default ON) still applies to non-grouped tensors (attention projections are already protected). Grouped int4 uses exact kernels by design.
-
-## Prior art
-
-- **GLM-5 paper** (arXiv 2602.15763, Β§2.4.3): "We apply INT4 QAT in the SFT stage" β€” the model is trained for int4, but assumes fine-grained quantization.
-- **MxMoE** (ICML 2025): MoE experts exhibit divergent quantization sensitivity; uniform quant across all experts is suboptimal.
-- **MoEQuant** (OpenReview): Naive int4 on MoE loses meaningful accuracy; calibrated framework needed.
-- **Automated Fine-Grained MoE Quantization** (ACL 2025): Layer/expert-wise sensitivity variation; group-size scaling is the baseline improvement.
-- **ubergarm/GLM-5.1-GGUF**: imatrix-calibrated quants with expert-specific patches β€” explicitly patches `quantize_row_q4_0_ref()` for routed experts.
-- **llama.cpp Q4_K**: Uses block-level group scales (32 or 64 elements) as the standard int4 format.
-
-## Conversion status
-
-Re-converting from the FP8 source (`zai-org/GLM-5.2-FP8`) with `--group-size 128`. Downloading via ModelScope to avoid HuggingFace per-stream throttling. Will report quality A/B results once the conversion is complete.