Run frontier MoE models on hardware you already own β pure C, zero deps, experts streamed from disk. Tiny engine, immense model. π¦
C
Quarantine EXPERT_BUDGET: the operating window is measured empty (#303)
JustVugg35f90b9e765e
diff --git a/c/glm.c b/c/glm.cindex ae18c554..d34606f1 100644--- a/c/glm.c+++ b/c/glm.c@@ -2858,8 +2858,13 @@ static void moe(Model *m, Layer *l, int layer, float *x, int S, float *out, int* only drop from misses. From the misses, keep the highest-aggregate-gate-weight* ones up to the budget; drop the rest from idxs[] so they're never loaded.* (MoE-Spec arXiv 2602.16052: top-32 of 64 capture 93% routing weight.)- * Complementary to TOPP (per-position) β this trims cross-position. */- if(g_expert_budget>0 && nu>g_expert_budget){+ * Complementary to TOPP (per-position) β this trims cross-position.+ * DECODE-ONLY (S<=4, incl. MTP verify): during prefill S=prompt_len the batch+ * union nu is 30-100+ experts and capping to 4-8 drops 80-90% of them, each with+ * non-trivial gate weight -> corrupted prefill hidden state -> wrong KV cache ->+ * repetitive garbage decode. The budget is only safe token-by-token, where the+ * prefill KV cache is already correct. (woolcoxm, #292.) */+ if(g_expert_budget>0 && S<=4 && nu>g_expert_budget){/* compute aggregate gate weight per unique expert */float *wsum=falloc(nu); for(int j=0;j<nu;j++) wsum[j]=0;for(int s=0;s<S;s++) for(int kk=0;kk<keff[s];kk++){@@ -5759,7 +5764,31 @@ int main(int argc, char **argv){if(g_mmap) fprintf(stderr,"[MMAP] expert = viste zero-copy nei file (page cache = cache)\n");g_topk = getenv("TOPK")?atoi(getenv("TOPK")):0;g_topp = getenv("TOPP")?atof(getenv("TOPP")):0;+ /* EXPERT_BUDGET e' sotto quarantena: la finestra operativa e' misurata VUOTA.+ * @bokiko su tre host (#303) e riprodotto qui su un 25 GB / WSL:+ * - hellaswag 30% a budget=8 contro 90% a budget spento (25% = il caso);+ * - a budget=4 il decode e' rumore ("The **1...: s2151:");+ * - accettazione MTP 0%: quali expert sopravvivono al cap dipende dalla+ * residenza in cache al momento del forward, quindi draft e verify NON+ * calcolano la stessa funzione -- la stessa invariante che #294 ha appena+ * stabilito, violata via stato di cache invece che via scelta del kernel;+ * - 0.13 tok/s contro 0.30 di baseline, con 14.66 expert caricati per layer+ * contro topk=8: il cap fa piu' I/O di quello che dice di risparmiare, e la+ * riga "~N GB I/O saved" conta esperti scartati, non byte non letti.+ * Resta compilato e sviluppabile (EXPERT_BUDGET_EXPERIMENTAL=1) perche' l'idea+ * -- MoE-Spec, arXiv 2602.16052 -- non e' sbagliata: e' l'implementazione che+ * finora non ha un punto in cui sia insieme piu' veloce e corretta. Riaccenderlo+ * di default richiede una misura di qualita' accanto a quella di velocita'. */g_expert_budget = getenv("EXPERT_BUDGET")?atoi(getenv("EXPERT_BUDGET")):0;+ if(g_expert_budget>0 && !getenv("EXPERT_BUDGET_EXPERIMENTAL")){+ fprintf(stderr,"[EXPERT_BUDGET] ignored: measured empty operating window (issue #303).\n"+ "[EXPERT_BUDGET] every tested setting is either no faster or no longer coherent:\n"+ "[EXPERT_BUDGET] budget=8 -> hellaswag 30%% (90%% with it off) | budget=4 -> decode is noise\n"+ "[EXPERT_BUDGET] MTP acceptance 0%% (the cap breaks the draft/verify contract, #294)\n"+ "[EXPERT_BUDGET] 0.13 tok/s vs 0.30 baseline, loading 14.7 experts/layer vs topk=8\n"+ "[EXPERT_BUDGET] set EXPERT_BUDGET_EXPERIMENTAL=1 to run it anyway (expect garbage).\n");+ g_expert_budget=0;+ }g_cache_route = getenv("CACHE_ROUTE")?atoi(getenv("CACHE_ROUTE")):0;g_route_j = getenv("ROUTE_J")?atoi(getenv("ROUTE_J")):2;g_route_m = getenv("ROUTE_M")?atoi(getenv("ROUTE_M")):12;diff --git a/issue_budget.md b/issue_budget.mddeleted file mode 100644index fe0ef86d..00000000--- a/issue_budget.md+++ /dev/null@@ -1,150 +0,0 @@-## TL;DR--New `EXPERT_BUDGET=N` env var that caps the number of **distinct experts loaded per layer** across the batch-union. When the union exceeds the budget, keeps only the highest-aggregate-gate-weight experts and drops the rest β they're never loaded from disk. On a 24 GB RAM host (cache `cap=2`), `EXPERT_BUDGET=4` nearly **doubles decode tok/s** (0.18 β 0.33) and **4x's prefill speed** (38.7s β 8.9s). Based on MoE-Spec (arXiv 2602.16052): "top 32 of 64 experts capture 93% of routing weight."--Branch: `experiment/expert-budget` (based on latest `dev` at `62419af`)-------## The problem--Every expert miss costs ~19 MB of disk I/O. On low-RAM hosts where the LRU cache cap is tiny (e.g. `cap=2` on 24 GB RAM), nearly every routed expert is a miss. With `topk=8` and 75 sparse layers, a single-token decode reads ~8.5 GB of experts from disk. Under MTP with `S=4`, the batch-union can produce 20-32 distinct experts per layer β almost all misses β multiplying disk reads further.--The README documents this directly:-> "on a cold cache each verified draft routes to extra experts (~660 β ~1100 expert-loads/token)"--The existing `TOPP` env var trims experts *within a single position's top-K*. But it cannot reduce the **cross-position union** β the deduplication across batch positions (prefill, MTP verification) that multiplies disk loads. That's the gap this fills.--## How it works--The batch-union in `moe()` (line ~2306 of `glm.c`) deduplicates routed experts across all `S` positions into `uniq[0..nu)`. After the union is built but before the cache-resolve/load loop, the budget cap kicks in:--```-if(EXPERT_BUDGET > 0 && nu > EXPERT_BUDGET):- 1. Compute aggregate gate weight per unique expert- (sum of ws[s*K+kk] across all positions that route to it)- 2. Sort experts by descending aggregate weight- 3. Keep top EXPERT_BUDGET, mark rest as dropped- 4. Remove dropped experts from each position's idxs[]/keff[]- (decrement keff, renormalize remaining weights if norm_topk)- 5. Compact uniq[] to kept experts only-```--Dropped experts are removed from `idxs[]` entirely, so they're **never resolved, never loaded from disk, never computed**. The downstream code already tolerates `keff[s] < K` (the `TOPP` path produces the same state), so this propagates correctly through resolve, matmul, and LRU promotion.--**Relationship to existing features:**-- `TOPP` trims within one position's top-K (per-position)-- `EXPERT_BUDGET` trims across the cross-position union (per-layer)-- They compose: `TOPP=0.8 EXPERT_BUDGET=12` first trims each position to its top-p mass, then caps the union--## Code change--**`c/glm.c`** β 54 lines added, single file:--1. Global + counter (near existing `g_topk`/`g_topp`):-```c-static int g_expert_budget=0; /* EXPERT_BUDGET=N */-static int64_t g_budget_dropped=0; /* total experts dropped */-```--2. Env var parsing (near existing `TOPK`/`TOPP`):-```c-g_expert_budget = getenv("EXPERT_BUDGET")?atoi(getenv("EXPERT_BUDGET")):0;-```--3. Budget cap in `moe()` batch-union (after `uniq[]` is built, before resolve loop) β full logic described above.--4. Stats reporting alongside existing `TOPK`/`TOPP` line:-```-EXPERT_BUDGET=4 (dropped 13613 experts, ~257.3 GB I/O saved)-```--## Measurements--**Test setup:** GLM-5.2 744B int4, 24 GB RAM (cache `cap=2`), Core Ultra 9 185H (AVX-VNNI), MTP=0, single-token decode, 32 tokens generated, same prompt: *"Explain the concept of recursion in programming. Provide a simple example."*--| Config | tok/s | vs baseline | hit rate | prefill | decode | experts dropped | I/O saved |-|--------|-------|-------------|----------|---------|--------|-----------------|-----------|-| **Baseline** (budget=0) | 0.18 | β | 9.3% | 38.7s | 176.3s | 0 | 0 |-| EXPERT_BUDGET=12 | 0.19 | +5% | 14.0% | 12.3s | 171.3s | 3,313 | 62.6 GB |-| EXPERT_BUDGET=6 | 0.26 | +44% | 21.0% | 7.5s | 122.5s | 8,543 | 161.5 GB |-| **EXPERT_BUDGET=4** | **0.33** | **+83%** | 16.4% | 8.9s | **97.4s** | 13,613 | 257.3 GB |--### Profile breakdown (prefill)--| Metric | Baseline | Budget=4 |-|--------|----------|----------|-| expert-disk service | 28.4s | **3.5s** (8x less) |-| expert-matmul | 7.1s | 1.4s |-| attention | 2.3s | 2.8s |--### Profile breakdown (decode, 32 tokens)--| Metric | Baseline | Budget=4 |-|--------|----------|----------|-| expert-disk service | 133.4s | proportional ~65s |-| expert-matmul | 23.9s | ~12s |-| decode total | 176.3s | **97.4s** |--### Key observations--1. **Prefill is the biggest winner.** With S=14 positions, the batch-union has 50+ distinct experts per layer. Budget=12 cuts that to 12, saving 40+ Γ 19 MB Γ 75 layers of disk reads. Prefill goes from 38.7s to 8.9s β **4.4x faster**.--2. **Decode improvement scales with budget tightness.** Budget=12 barely constrains single-token decode (S=1 β max 8 experts < 12), so decode is barely affected. Budget=4 actually halves the decode load (8 β 4 experts/layer), nearly doubling tok/s.--3. **Hit rate improves** because fewer experts compete for the tiny cache (cap=2). Budget=6 hit 21% vs baseline 9.3% β more than doubled.--4. **No crashes or instability** at any budget value.--## Quality impact β honest assessment--**Budget=4 keeps only the top-4 of 8 routed experts per layer.** The dropped 4 experts had the lowest gate weights β they contributed the least to the output. But this IS a quality trade-off.--On a **cold cache** (our test setup), quality assessment is confounded: both baseline and budget outputs are already garbled from int4 quantization + cold-cache routing. Sample comparison:--- **Baseline output:** *"Hire Some To Take Object-Oriented Programming Assignment"*-- **Budget=4 output:** *"The world is a dangerous place, not so much because of the small percentage of people who are doing to do the little of people who are doing to"*--Both are incoherent β the cold cache at int4 on 24 GB RAM produces poor output regardless of budget. The budget=4 output is **not dramatically worse** than the already-poor baseline β they're both garbled, just differently garbled. A proper quality assessment needs a **warm cache** where the baseline produces coherent text, so you can isolate the degradation from dropping experts.--**Expected quality on warm cache:** With `norm_topk=1` (GLM-5.2 renormalizes gate weights), the top-4 experts typically capture ~80-85% of routing weight (the remaining 4 share 15-20%). Output should be mostly coherent with occasional word-choice degradation. Budget=6 (top-6 of 8, ~90%+ weight captured) should be nearly indistinguishable from baseline.--**Recommendation for users:**-- `EXPERT_BUDGET=6-8` on cold/low-RAM hosts β good speedup, minimal quality loss-- `EXPERT_BUDGET=4` on very-low-RAM hosts where speed matters more than quality-- Leave OFF on high-RAM hosts where all experts are resident anyway (budget never triggers)--## Safety--- **Default OFF** (`EXPERT_BUDGET=0`) β zero behavior change unless explicitly set-- **Opt-in quality trade-off** β same design philosophy as `TOPP`: the user explicitly chooses speed over quality-- **No output corruption** β dropped experts' contributions are omitted and remaining weights renormalized (identical math to `TOPP`)-- **Compatible with all features** β PILOT, MTP, CUDA, CACHE_ROUTE, PIPE, TOPP all work unchanged; they just see fewer experts in the union-- **No crash risk** β no new memory allocation patterns, no new I/O, purely a filter on existing data structures--## Reproduce--```bash-make ARCH=native--# Baseline:-SNAP=<model_dir> MTP=0 PROMPT="Explain recursion in programming." NGEN=32 ./glm 64--# With budget:-SNAP=<model_dir> MTP=0 EXPERT_BUDGET=4 PROMPT="Explain recursion in programming." NGEN=32 ./glm 64--# With MTP (where budget saves the most):-SNAP=<model_dir> MTP=1 EXPERT_BUDGET=16 PROMPT="Explain recursion in programming." NGEN=32 ./glm 64-```--The stats line prints `EXPERT_BUDGET=N (dropped X experts, ~Y GB I/O saved)`.--## What's needed before merge--1. **Warm-cache quality A/B test** on a host with enough RAM (cap>=16) to produce coherent baseline output, then compare budget=4/6/8 text quality-2. **MTP interaction test** β verify that budget + MTP composes correctly and doesn't crash when draft verification routes to budgeted-away experts-3. **Decide on defaults** β should this auto-activate on low-RAM hosts (like `cap` auto-lowering)? My recommendation: no, leave it OFF and document the recommended value per RAM tier--## Prior art--**MoE-Spec** (arXiv 2602.16052) β "Expert Budgeting for Efficient Speculative Decoding": Training-free expert budgeting at verification time. Key finding: "top 32 of 64 experts capture 93% of routing weight." Our approach applies the same principle but at the batch-union level (not just MTP verification), making it effective for prefill and single-token decode too.diff --git a/issue_diskio.md b/issue_diskio.mddeleted file mode 100644index 91d7b92c..00000000--- a/issue_diskio.md+++ /dev/null@@ -1,309 +0,0 @@-# Disk I/O Minimization β Research--Branch: `experiment/diskio-research` (based on `dev` at `62419af`)--## TL;DR--The engine's disk I/O is **already well-engineered on the hottest path** (expert streaming uses coalesced O_DIRECT `pread` + `posix_fadvise` hints + LRU + pin cache + speculative prefetch). There are **4 concrete, bounded opportunities** to shave latency, ranked by ROI:--| # | Opportunity | Where | Frequency | Estimated win |-|---|---|---|---|---|-| 1 | **KV-cache write batching** (157 fwrites/token β 1) | `kv_disk_append` | per turn | cuts ~100s of syscalls/turn |-| 2 | **`/proc/meminfo` fopen storm** | `rss_gb()` | ~every 16 tokens (Linux) | eliminates recurring open/read/close |-| 3 | **Expert prefetch on Windows** (`PrefetchVirtualMemory`) | `expert_prefetch` | per miss | mmap path is Linux/macOS-only today |-| 4 | **KV-cache: buffered handle kept open** | `kv_disk_append` | per turn | kills open+fseek+close per turn |--There are also **2 non-opportunities** worth recording so we don't re-investigate: O_DIRECT for experts (correctly used today), and PagedAttention-style file layout (already single-file + indexed).-------## How the engine does disk I/O today--There are **three I/O stacks**, behaving very differently:--| Stack | Mechanism | Frequency | Files |-|---|---|---|---|-| **Expert weights** (hottest) | `pread` on kept-open fds + `posix_fadvise`, optional `mmap` | per miss, every token | `st.h`, `glm.c:1328` |-| **KV cache** (`.coli_kv`) | `fopen` + `fwrite`/`fread` | per turn | `glm.c:3812-3889` |-| **Everything else** (config, tokenizer, stats, grammar) | `fopen` + `fread` | startup-only | scattered |--### Expert path (the hot path β already good)--`expert_load` (`glm.c:1328-1481`) has three sub-paths:--- **Default `pread` path** (`glm.c:1385-1472`): coalesces the 3 contiguous expert tensors (gate/up/down) into **one ~19 MB O_DIRECT `pread`** into a 16K-aligned slab (`glm.c:1447`). Falls back to 3 separate `pread`s only if non-contiguous. Scales are 3 tiny separate `pread`s (kilobytes). `posix_fadvise(DONTNEED)` evicts pages after if `g_drop`. **This is well-batched β one syscall for ~19 MB.**-- **`COLI_MMAP=1` path** (`glm.c:1352-1383`): `mmap` per shard fd (cached), `madvise(WILLNEED)` + synchronous page-touch loop. Zero-copy. **Default OFF, and Linux/macOS/FreeBSD-only** β no `MapViewOfFile` on Windows.-- **Prefetch hints**: `expert_prefetch` (`glm.c:1602-1609`) β `st_prefetch` (`st.h:178`) issues `posix_fadvise(WILLNEED)` β readahead hint only, no data read. Called from `moe` next-64-block lookahead, pilot, and SPEC.--### KV cache persistence (per turn β opportunity here)--`kv_disk_append` (`glm.c:3834-3855`), called once per turn:-1. `fopen("r+b")` β **reopens the file every turn**-2. `fseek` to append position-3. **per-position loop**: for each new token, `fwrite` the token i32, then **2 fwrites per layer** (Lc + Rc) + optional DSA Ic. With 78 layers that's **~157 fwrites per token appended**.-4. `fflush` (userspace only β **no fsync/fdatasync anywhere in the codebase**)-5. `fseek` back to header + `fwrite` the new nrec counter (crash-safe ordering)-6. `fclose`--Record size ~182 KB/token. On a long first turn this is **tens of thousands of small fwrites**. stdio buffering coalesces them into fewer `write` syscalls, but the userspace overhead remains.--### Recurring surprise: `/proc/meminfo`--`rss_gb()` (`glm.c:4625`) does `fopen("/proc/meminfo")` + fgets + fclose. Called from every STAT line and every 16-token heartbeat (`glm.c:3473, 3477`). On Linux this is an **open+read+close of procfs ~every 16 tokens**. (Windows uses `compat_meminfo`, no file β not affected.)-------## What similar projects do--**llama.cpp** (the reference): `mmap`s the entire model read-only, uses `--mlock` to pin hot pages, streams layers to GPU via partial offload, and issues per-pass readahead of upcoming tensors (`llama-mmap.cpp`). Justine Tunney's mmap work: "load 100Γ faster using half as memory." Crucial finding from discussion #18758: **for MoE, mmap beats O_DIRECT** when the model fits in ~RAM β O_DIRECT takes "at least 10Γ longer" on repeated loads because it bypasses the page cache that serves re-faults for free.--**The general consensus across llama.cpp, vLLM, AirLLM, PRESERVE, HOBBIT, SolidAttention (FAST '26):**-- mmap + OS page cache as the backing store for an LRU is the proven recipe-- prefetch the *next* expert/layer while computing the current one β this is where the 0.5ms lives-- single indexed file (one `open()`) beats one-file-per-expert-- align tensors to 4KB (preferably 64KB) for clean page-fault boundaries + SSD geometry-- buffer sweet spot ~1MB; syscall cost ~1-5Β΅s each, so batching matters at high repetition-------## The 4 opportunities (ranked)--### Opportunity 1 β KV-cache write batching (HIGH ROI, LOW risk)--**Problem:** `kv_disk_append` does ~157 `fwrite` calls per appended token (1 token i32 + 2Γ78 layers). stdio buffering hides some of this, but on a long first-turn prefill (hundreds-thousands of tokens) this is tens of thousands of fwrites.--**Fix:** Build one contiguous record in a heap buffer (token + all layers' Lc/Rc/Ic for that position), then **a single `fwrite` per position** (or even one `fwrite` for the whole turn). The data is already laid out contiguously in memory per-layer (`coli_kv_row`), so a layered `memcpy` into a staging buffer + one write is straightforward.--**Win:** ~157Γ fewer fwrite calls per token. Even with stdio coalescing, the userspace loop overhead is real at scale.--### Opportunity 2 β `/proc/meminfo` fopen storm (MEDIUM ROI, trivial)--**Problem:** `rss_gb()` opens, reads, closes `/proc/meminfo` every ~16 tokens on Linux. Each is ~3 syscalls + path resolution.--**Fix:** Either (a) cache the value for N tokens (e.g. re-read at most once per second), or (b) keep the fd open and `rewind`+`fgets`. Trivial change.--### Opportunity 3 β Expert prefetch on Windows (MEDIUM ROI, bounded)--**Problem:** The `COLI_MMAP=1` path (which gives zero-copy expert access + free OS-cache re-faults) is **Linux/macOS/FreeBSD-only** β `glm.c:1301` guards it. On Windows, experts always go through the `pread` path, and `expert_prefetch` issues `posix_fadvise(WILLNEED)` which is a no-op shim on Windows (`compat.h`).--**Fix:** On Windows, implement the prefetch via `PrefetchVirtualMemory` (the Win32 analog of `MADV_WILLNEED`) on an mmap'd region, or via an async `ReadFile`+`OVERLAPPED` into a scratch buffer. This brings the Windows build closer to parity with the Linux mmap+prefetch story.--**Scope:** This is the largest of the four β it touches the Windows I/O path. Worth doing if Windows perf is a goal; skip if Linux is the target.--### Opportunity 4 β KV-cache: keep handle open (LOW-MEDIUM ROI, LOW risk)--**Problem:** `kv_disk_append` does `fopen`+...+`fclose` every turn. Handle creation is ~5-15Β΅s of pure overhead (worse on Windows).--**Fix:** Open the KV file once (lazily on first append), keep the `FILE*` for the engine lifetime, just `fseek`+write each turn. Close on shutdown. Pair with Opportunity 1 for the write batching.-------## Non-opportunities (recording so we don't re-investigate)--- **O_DIRECT for experts**: already correctly used (`st.h:83`, `DIRECT=1`). For an LRU+refetch pattern the page cache is your friend, but the engine offers both paths (O_DIRECT pread default + optional mmap) and the O_DIRECT coalesced read is already one syscall for ~19MB. Don't change this.-- **Single-file layout**: the engine already uses safetensors shards with kept-open fds + offset-indexed tensors (`st.h`). No per-expert open()/close() waste. Don't change this.-- **PagedAttention**: solves concurrency fragmentation this engine doesn't have (β€16 slots). Not applicable.-------## Next steps--The highest-ROI, lowest-risk starting point is **Opportunity 1 (KV write batching) + Opportunity 4 (keep handle open)** β they're in the same function, both low-risk, and together they eliminate the per-turn open/close overhead and the per-token fwrite storm. Opportunity 2 is a trivial 5-minute fix we can bundle in.--Opportunity 3 (Windows prefetch) is the biggest single win but also the largest scope β separate effort, gated on whether Windows perf is a priority.--## Sources--- [justine.lol/mmap β Edge AI Just Got Faster](https://justine.lol/mmap/)-- [llama.cpp discussion #18758 β Mmap faster than direct I/O for MoE](https://github.com/ggml-org/llama.cpp/discussions/18758)-- [llama.cpp issue #20757 β Two-tier GPU+RAM expert cache](https://github.com/ggml-org/llama.cpp/issues/20757)-- [FAST '26 β Programmable Page Cache for LLM loading](https://www.usenix.org/system/files/fast26-liu-yubo.pdf)-- [FAST '26 β SolidAttention: SSD-based serving](https://www.usenix.org/system/files/fast26-zheng.pdf)-- [HOBBIT β Mixed precision expert offloading](https://arxiv.org/html/2411.01433v2)-- [posix_fadvise(2) β man7.org](https://man7.org/linux/man-pages/man2/posix_fadvise.2.html)-- [madvise(2) β man7.org](https://man7.org/linux/man-pages/man2/madvise.2.html)-- [Microsoft Learn β File Buffering (FILE_FLAG_NO_BUFFERING)](https://learn.microsoft.com/en-us/windows/win32/fileio/file-buffering)-- [Microsoft Learn β PrefetchVirtualMemory](https://learn.microsoft.com/en-us/windows/win32/api/memoryapi/nf-memoryapi-prefetchvirtualmemory)-- [What makes system calls expensive β codingconfessions.com](https://blog.codingconfessions.com/p/what-makes-system-calls-expensive)-- [Syscall overhead β Stack Overflow](https://stackoverflow.com/questions/8247331/syscall-overhead)-------# Windows Implementation β branch `windows-optimizations` (2026-07-15)--## What landed (pread path, validated)--Two changes, both on the `pread` expert-load path (no mmap). Measured against the-existing `bench_budget*.txt` baselines (GLM-5.2 744B int4, 32 GB RAM, Core Ultra 9-185H, DRAFT=0, 32-token decode):--### 1. `compat_fadvise` WILLNEED cache-warmer (`c/compat.h`)--Replaced the Windows `posix_fadvise` no-op (was a `do{}while(0)` macro) with a real-readahead: an overlapped `ReadFile` into a throwaway scratch buffer that populates the-standby page cache, so the later synchronous `pread` faults from RAM not disk. Mirrors-the macOS `F_RDADVISE` shim (`compat.h:28-37`). DONTNEED stays a no-op (matches macOS;-Windows standby-list trimming self-regulates under pressure).--This re-arms the existing `expert_prefetch` β `st_prefetch` β `posix_fadvise(WILLNEED)`-chain on Windows: the next-block readahead in `moe()` and the PILOT cross-layer prefetch-hints now actually warm the cache instead of being silently discarded.--**Measured effect (budget=4, PIPE on):** hit rate 16.4% β 27.6%.--### 2. PIPE default ON for Windows (`c/glm.c`)--Flipped the async expert-load thread pool from default OFF to default ON on Windows-(`getenv("PIPE")?:1` under `_WIN32`, unchanged `:0` elsewhere). PIPE dispatches expert-`pread` loads onto worker threads so they overlap the expert matmul on the forward-pass-thread, instead of the blocking serial load-then-compute path. `PIPE=0` opts back out.--**Measured effect (budget=4):** expert-disk 65.9s β 54.3s (β18%), reaching **1.70 s/tok**-(under the 2 s/tok target; budget=4 baseline was 2.06 s/tok).--### Results table (DRAFT=0, 32-token decode, pread path)--| config | expert-disk | s/tok | hit% | tok/s |-|---|---|---|---|---|-| budget=4, no PIPE (existing baseline) | 65.9s | 2.06 | 16.4% | 0.33 |-| **budget=4 + PIPE (this PR)** | **54.3s** | **1.70** | **27.6%** | **0.34** |-| budget=6 + PIPE | 77.1s | 2.41 | 21.8% | 0.27 |--budget=4 + PIPE meets the β€2 s/tok target. budget=6 (more experts/layer, higher quality)-misses it at 2.41 s/tok β the speed/quality tradeoff.--## What was tried and abandoned: Windows mmap (`COLI_MMAP` on `_WIN32`)--The original plan (informed by llama.cpp #18758: "mmap is β₯10Γ faster than O_DIRECT for-MoE") was to port the mmap expert path to Windows via `CreateFileMapping`/`MapViewOfFile`.-This was implemented and tested at length. **It was a measured regression and was reverted.**--### The attempt--Added a `_WIN32` branch to `map_of_fd` (`glm.c`) mapping each shard file read-only and-resolving experts as views into the mapping, mirroring the POSIX path. Also added-`PrefetchVirtualMemory` readahead and a `VirtualUnlock` eviction mechanism (the Windows-`posix_fadvise(DONTNEED)` analog β see SO#1880714; validated standalone to demote pages-to the standby list with a 2.3Γ faster re-fault).--### Why it regressed--**mmap'd expert pages bloat the process working set on Windows, which collapses the-expert cache.** This is a fundamental Windows-vs-Linux difference:--- On Linux, `mmap(MAP_SHARED)` file pages live in the kernel page cache (`buff/cache`),- separate from `MemAvailable`, so the cache budget isn't fooled.-- On Windows, touched `MapViewOfFile` pages count against `ullAvailPhys` (what- `compat_meminfo` reads for the budget). The CPU matmul touches every weight byte,- faulting ~12 GB into the working set. `cap_for_ram()` then sees ~no free RAM and- collapses the LRU cache cap.--Measured (budget=0, DRAFT=0, apples-to-apples):--| config | RAM_GB detected | cache cap | hit% | expert-disk | RSS |-|---|---|---|---|---|---|-| baseline (pread) | 21.4 | 1 | 9.3% | 133s | 15.0 GB |-| mmap, no eviction | **8.0** | 1 | **2.2%** | **240s** | **27.2 GB** |-| mmap + VirtualUnlock | 24.9 | 2 | 11.8% | 83s | 18.1 GB |-| mmap + reserve reductions | 24.6 | 4 | 21.8% | 80s | 20.1 GB |--The `VirtualUnlock` eviction recovered the regression (240sβ83s), and dropping the-Linux-specific page-cache/slab reserves under mmap got it to parity with pread. But it-never clearly *beat* the simpler pread+PIPE path, and it added substantial complexity-(per-slot eviction tracking, reserve conditionals, `VirtualUnlock` on every slot recycle).-**The engine already moved off mmap to pread for this exact RSS bug** (`st.h:3-6`), and-the Windows port re-confirmed that decision.--### What else didn't work--- **Batched `PrefetchVirtualMemory`** for the mmap path: tested as a single batched- readahead of all 64 missed experts' pages before the matmul. **Blocked instead of- prefetching async** on this SSD β inflated `t_edisk` (80sβ102s). Consistent with- microsoft/Windows-Dev-Performance#108 ("PrefetchVirtualMemory does not prefetch").- Reverted.-- **True I/O/compute overlap on the CPU path**: the Metal path has this ("submit- resident experts to GPU before loading misses"), but the CPU path loads-then-computes- serially. `PrefetchVirtualMemory` was the attempt to add it for mmap and failed. The- pread path gets overlap via PIPE (which works), not via mmap prefetch.--### Conclusion--For this engine on Windows at this RAM budget (~32 GB, 370 GB model), **pread + PIPE +-compat_fadvise** is the right path. mmap remains valuable on Linux/macOS (where the page-cache doesn't inflate process RSS) but is not viable on Windows without a fundamentally-different cache-budget model that excludes mapped-file pages β left as future work.--## Sources added--- [SO#1880714 β VirtualUnlock releases mapped pages to standby list](https://stackoverflow.com/questions/1880714/createfilemapping-mapviewoffile-how-to-avoid-holding-up-the-system-memory)-- [Alois Kraus β The Mysterious Lost Memory (modified/standby list)](https://aloiskraus.wordpress.com/2017/02/26/the-mysterious-lost-memory-which-belongs-to-no-process/)-- [microsoft/Windows-Dev-Performance#108 β PrefetchVirtualMemory inconsistency](https://github.com/microsoft/Windows-Dev-Performance/issues/108)-- [llama.cpp #18758 β mmap faster than O_DIRECT for MoE (Linux)](https://github.com/ggml-org/llama.cpp/discussions/18758)-- [HN#35426679 β Why MMAP in llama.cpp hides true memory usage](https://news.ycombinator.com/item?id=35426679)-------# CACHE_ROUTE: miss elimination via cache-aware routing (2026-07-15)--## The breakthrough--Adding `CACHE_ROUTE=1 ROUTE_J=2 ROUTE_M=12` to the optimized stack pushed-throughput to **1.41 tok/s** (4.3Γ over stock) by directly reducing the-miss rate from 27% to 17%.--## How it works--The engine's `CACHE_ROUTE` feature (paper: max-rank routing, arXiv 2412.00099)-steers the MoE router to prefer experts that are already cache-resident:--- `ROUTE_J=2`: keep the top-2 true router picks (always, even if uncached)-- `ROUTE_M=12`: fill the remaining 2 of 4 slots with the highest-ranked experts- that are ALREADY in the LRU cache (from the top-12 candidates)--This guarantees 2 of 4 expert slots per layer per token are cache hits. The-`route_agree=94.6%` metric confirms minimal quality cost β 94.6% of cache-steered-picks match the true top-K the router would have chosen.--## Why this is the right fix for the miss problem--Analysis of route traces showed GLM-5.2's routing is nearly uniform β 75 tokens-use 237 of 256 experts per layer, with the top-4 capturing only 4.3% of selections.-This means:--- PIN (hot-expert pre-loading) is ineffective β there are no hot experts-- PILOT_REAL prefetch can't keep up under the fast pipe2 GPU pipeline-- A bigger cache helps marginally but can't cover 237 unique experts per layer-- CACHE_ROUTE is the only lever that reduces misses without more RAM or faster disk--## Results (pipe2 + full stack + ws_b fix, budget=4, RAM_GB=28)--| metric | without CACHE_ROUTE | with CACHE_ROUTE |-|---|---|---|-| tok/s | 1.03 | **1.41** (+37%) |-| hit rate | 73% | **83%** |-| expert-disk | 12.4s | **8.5s** (β31%) |-| decode | 31.0s | **22.7s** (β27%) |-| route_agree | n/a | 94.6% |--## Full optimization journey--| step | tok/s | hit% | disk |-|---|---|---|---|-| stock budget=4 | 0.33 | 9% | 65.9s |-| + disk stack | 0.63 | 72% | 21.5s |-| + CUDA dense+attn | 0.72 | 75% | 18.4s |-| + pipe2 GPU pipeline | 0.85 | 57% | 18.2s |-| + ws_b cache fix | 1.03 | 73% | 12.4s |-| **+ CACHE_ROUTE** | **1.41** | **83%** | **8.5s** |--**4.3Γ total speedup.** Disk I/O reduced 7.7Γ (65.9s β 8.5s).--## Recommended config--```-EXPERT_BUDGET=4 PIPE=1 RAM_GB=28 PILOT_REAL=1 DIRECT=1-COLI_CUDA=1 CUDA_DENSE=1 COLI_CUDA_ATTN=1 COLI_CUDA_PIPE=2 CUDA_EXPERT_GB=0-CACHE_ROUTE=1 ROUTE_J=2 ROUTE_M=12-```--CACHE_ROUTE is an existing engine feature (no code change) β opt-in via env vars.diff --git a/issue_grouped_quant.md b/issue_grouped_quant.mddeleted file mode 100644index fb46ce4f..00000000--- a/issue_grouped_quant.md+++ /dev/null@@ -1,185 +0,0 @@-## TL;DR--The int4 model produces **fluent-but-incoherent output** β grammatical English that is completely off-topic (SEO spam, homework-help boilerplate) instead of reasoned responses. The root cause is **per-row quantization scales**: one F32 scale per output row (e.g. 2048 scales for a 2048Γ6144 matrix), which is 48x coarser than the FP8 source's 128Γ128 block scales. This destroys fine-grained weight information that handles reasoning and instruction-following while preserving the large-magnitude weights that handle grammar and vocabulary fluency.--Branch: `experiment/grouped-quant` (based on latest `dev` at `62419af`)-------## The problem β demonstrated--Prompt: *"Explain the concept of recursion in programming. Provide a simple example."*--**Baseline (no budget, no changes):**-> "Hire Some To Take Object-Oriented Programming Assignment"--**Budget=4:**-> "The world is a dangerous place, not so much because of the small percentage of people who are doing to do the little of people who are doing to"--**Budget=6:**-> "Elite Custom Essays"--**Budget=12:**-> "Sololearn: Learn to code for FREE! +1 # Explain A concept of recursion..."--This is not random gibberish β it's fluent English that is completely off-topic. This is the signature of **activations corrupted by coarse quantization**: the model retains language fluency (large weights survive) but loses instruction-following and reasoning (fine-grained weights are crushed to zero).--## Root cause: per-row int4 scales--### How the current converter works--`convert_fp8_to_int4.py` (line 39-52) quantizes with **one scale per output row**:--```python-amax = np.abs(w).max(axis=1, keepdims=True) # one max per ROW-s = np.maximum(amax / qmax, 1e-8) # one scale per ROW-q = np.clip(np.rint(w / s), -8, qmax) # quantize entire row with that scale-```--For a 2048Γ6144 expert weight matrix, that's **2048 scales** β one per row, each covering 6144 elements.--### Why this destroys quality--Consider a row of 6144 values where most are small (magnitude ~0.01) but a few are large (~0.5). The per-row scale is `0.5/7 = 0.071`. The small values become `0.01/0.071 = 0.14`, which rounds to **zero**. All fine-grained information in those small weights is lost.--This matters enormously for MoE models: each token activates only 8 of 256 experts. A poorly-quantized expert pollutes every token that routes to it, and there's no averaging from the other 248 experts (they're simply off). The error is **concentrated**, not diluted.--### What the FP8 source does right--The FP8 checkpoint uses **128Γ128 block scales** β the weight matrix is divided into 128-element chunks, each with its own scale. Small values in one chunk don't get crushed by large values in another. For a 2048Γ6144 matrix: `2048 Γ 48 = 98,304` scales β 48x more granularity.--The converter already dequants FP8 to f32 correctly (lines 196-201), preserving the block-scale information. But then it **throws it all away** by collapsing to a single per-row scale during int4 requantization.--### GLM-5.2 was QAT-trained for int4--The GLM-5 paper (arXiv 2602.15763, Β§2.4.3) states: *"To provide better accuracy at low-precision, we apply INT4 QAT in the SFT stage."* This means the model is **designed** to work at int4 β but only if the quantization is fine-grained enough to match what the model saw during training. The FP8 checkpoint's 128Γ128 block scales are the granularity the model expects. Per-row scaling is 48x coarser.--### Community confirmation--Every other project getting coherent GLM-5.2 int4 output uses calibrated or fine-grained quantization:-- **ubergarm/GLM-5.1-GGUF** β imatrix-calibrated with expert-specific patches-- **Unsloth/GLM-5-GGUF** β dynamic UD-Q4 quants-- **QuantTrio/GLM-5.2-Int4-Int8Mix** β mixed int4/int8 with channel-wise scales-- **llama.cpp** β Q4_K with block-level group scales (typically 32 or 64 elements)--None use naive per-row RTN int4 for production inference.--## The fix: group-scaled int4 (fmt=4)--Add a new quantization format with **one scale per 128 elements** along the input dimension, matching the FP8 source's natural granularity.--### What changed--**`c/glm.c`** β 122 lines added:--1. **QT struct** (line 98): Added `int gs` field (group size, 0=per-row for backward compat, 128=grouped).--2. **Format detection** (`qt_from_disk`, line 1064): Auto-detects fmt=4 by checking the `.qs` scale array size. If it has `O * ceil(I/128)` elements instead of `O`, it's grouped. **Old per-row models (fmt=2) work unchanged.**--3. **New kernel** (`matmul_i4_grouped`, line 379): Same AVX2 nibble unpacking as `matmul_i4`, but the accumulator resets at each 128-element group boundary: `dot(x[grp], w[grp]) * scale[grp]`. The scale changes every 8 vector iterations (128/16=8).--4. **Dispatch** (`matmul_qt_ex`, line 813): Routes to `matmul_i4_grouped` when `fmt==4`. Always uses exact kernels (no IDOT approximation β the whole point is quality).--5. **Expert loading** (`expert_load`, lines 1418 and 1538): Both the mmap and slab+pread paths detect fmt=4 from scale array size and set `gs=128`.--6. **`qt_bytes`** (line 104): Reports correct memory for fmt=4.--**`c/tools/convert_fp8_to_int4.py`** β `quant_int4_grouped()` + `--group-size` arg:--```python-def quant_int4_grouped(w, bits, gs=128):- O, I = w.shape- ngroups = (I + gs - 1) // gs- wpad = np.zeros((O, ngroups * gs), np.float32)- wpad[:, :I] = w- wr = wpad.reshape(O, ngroups, gs) # [O, ngroups, gs]- amax = np.abs(wr).max(axis=2, keepdims=True) # one max per GROUP- s = np.maximum(amax / qmax, 1e-8)- q = np.clip(np.rint(wr / s), -8, qmax)- # ... same nibble packing as quant_int4 ...- return packed_nibbles, s.reshape(-1) # [O * ngroups] scales-```--Same packed-nibble format as existing int4 β only the scale array is larger.--### Verification--**Converter round-trip test** (weights with varying group magnitudes):--| Method | Mean relative error | Max abs error |-|--------|-------------------|---------------|-| Per-row int4 (current) | 0.2056 | 0.0127 |-| Grouped int4 (gs=128) | **0.1278** | 0.0127 |-| **Improvement** | **1.6x lower** | β |--**Engine kernel test** (AVX2 `matmul_i4_grouped` vs f32 reference):--| Output | Reference | Kernel | Error |-|--------|-----------|--------|-------|-| o=0 | -0.126368 | -0.126368 | 2.98e-08 |-| o=1 | 0.075913 | 0.075913 | 1.49e-08 |-| o=2 | 0.077136 | 0.077136 | 7.45e-09 |-| ... | ... | ... | ... |--Max error: **2.98e-08** β matches f32 reference to within float32 epsilon. **PASS.**--### Cost--| Metric | Per-row (current) | Grouped (new) |-|--------|-------------------|---------------|-| Expert weight size | ~18.9 MB | ~20.1 MB (+6%) |-| Scale array per matrix | O Γ 4 bytes | O Γ ceil(I/128) Γ 4 bytes |-| Total model size | ~370 GB | ~390 GB (+5%) |-| Disk I/O per miss | ~19 MB | ~20 MB (+6%) |-| Kernel speed | One scale multiply per row | One scale multiply per 128 elements |--The 6% I/O increase is negligible compared to the quality gain. The grouped kernel has slightly more scale-lookup overhead but remains within AVX2 throughput β the bottleneck is disk I/O, not matmul.--### Backward compatibility--- Old per-row models (fmt=2) **work unchanged** β format auto-detected from scale array size-- All existing features (PILOT, EXPERT_BUDGET, CUDA, MTP, PIPE) work identically β they don't touch the dequant path-- The fused gate+up pair path (`matmul_i4_pair`) falls back to separate `matmul_qt` calls for fmt=4 β minor perf cost, correctness preserved-- `--group-size 0` produces per-row output (backward compat for the converter)--## How to reproduce--### Convert with group scales--```bash-python tools/convert_fp8_to_int4.py \- --indir /path/to/GLM-5.2-FP8 \- --outdir /path/to/glm52_i4_grouped \- --ebits 4 --io-bits 8 --group-size 128-```--### Test quality--```bash-# Old model (per-row):-SNAP=/path/to/glm52_i4 PROMPT="Explain recursion in programming." NGEN=32 ./glm 64--# New model (grouped):-SNAP=/path/to/glm52_i4_grouped PROMPT="Explain recursion in programming." NGEN=32 ./glm 64-```--Compare output text coherence.--## What this won't fix--- **Cold cache speed** β still 0.18-0.36 tok/s on 24 GB RAM. Quality and speed are independent axes.-- **All quantization error** β int4 is still 4-bit. Some degradation vs FP8 will remain. But it should be coherent degradation (slightly wrong word choices) rather than total reasoning failure (unrelated topics).-- **The IDOT +12% perplexity** β the approximate activation kernel (`IDOT=1`, default ON) still applies to non-grouped tensors (attention projections are already protected). Grouped int4 uses exact kernels by design.--## Prior art--- **GLM-5 paper** (arXiv 2602.15763, Β§2.4.3): "We apply INT4 QAT in the SFT stage" β the model is trained for int4, but assumes fine-grained quantization.-- **MxMoE** (ICML 2025): MoE experts exhibit divergent quantization sensitivity; uniform quant across all experts is suboptimal.-- **MoEQuant** (OpenReview): Naive int4 on MoE loses meaningful accuracy; calibrated framework needed.-- **Automated Fine-Grained MoE Quantization** (ACL 2025): Layer/expert-wise sensitivity variation; group-size scaling is the baseline improvement.-- **ubergarm/GLM-5.1-GGUF**: imatrix-calibrated quants with expert-specific patches β explicitly patches `quantize_row_q4_0_ref()` for routed experts.-- **llama.cpp Q4_K**: Uses block-level group scales (32 or 64 elements) as the standard int4 format.--## Conversion status--Re-converting from the FP8 source (`zai-org/GLM-5.2-FP8`) with `--group-size 128`. Downloading via ModelScope to avoid HuggingFace per-stream throttling. Will report quality A/B results once the conversion is complete.