Run frontier MoE models on hardware you already own β pure C, zero deps, experts streamed from disk. Tiny engine, immense model. π¦
C
docs: inkling and olmoe on the Vulkan expert tier
JustVugg618d4ae66fb3
diff --git a/docs/ENVIRONMENT.md b/docs/ENVIRONMENT.mdindex 442f67f5..65666c74 100644--- a/docs/ENVIRONMENT.md+++ b/docs/ENVIRONMENT.md@@ -15,10 +15,10 @@ what follows, but the sister engines read their own:| `colibri` | `c/colibri.c` | everything below except the three sections named for another engine || `glm53` | `c/glm53.c` | the `GLM53_*` family and `COLI_MAP_EXPERTS`: see [GLM-5.3-Flash engine](#glm-53-flash-engine-glm53) || `kimi_k3` | `c/kimi_k3.c` | the `K3_*` family β see [Kimi K3 engine](#kimi-k3-engine-kimi_k3) |-| `inkling` | `c/inkling.c` | `INK_*`, plus `CTX_MAX`, `PIN_N`, `REP_PEN`, `GPU_DEV`, `NOGPU` β see [Inkling engine](#inkling-engine-inkling) |+| `inkling` | `c/inkling.c` | `INK_*`, plus `CTX_MAX`, `PIN_N`, `REP_PEN`, `GPU_DEV`, `NOGPU`, the Vulkan expert tier's `COLI_VK_TIER*` β see [Inkling engine](#inkling-engine-inkling) || `qwen36` | `c/qwen36.c` | `QWEN_*`, `Q36_*`, its dense/CUDA-tier controls, the `CACHE_ROUTE` family (VRAM tier over RAM cache), and the Vulkan expert tier's `COLI_VK_TIER*` β see [Qwen3.6 engine](#qwen36-engine-qwen36) || `qwen38` | `c/qwen38.c` | `Q38_MAXT`, `Q38_EOS`, `Q38_NATIVE_FP8`, `Q38_NATIVE_BF16`, `Q38_EXPERT_INT4`, `Q38_MTP`, `Q38_PREFILL_BATCH`, `Q38_TRUNK_CPU_INT8`, `Q38_FP8_KERNEL`, `COLI_TIMERS`, the Vulkan expert tier's `COLI_VK_TIER*` β see [Qwen3.8 engine](#qwen38-engine-qwen38) |-| `olmoe` | `c/olmoe.c` | `HOT`, `WIDE`, `SMOOTH`, `CONF_LIMIT`, `MAX_NEW`, `CHAT`, `EXPERT_DROP`, `WARMUP` β see [OLMoE engine](#olmoe-engine-olmoe) |+| `olmoe` | `c/olmoe.c` | `HOT`, `WIDE`, `SMOOTH`, `CONF_LIMIT`, `MAX_NEW`, `CHAT`, `EXPERT_DROP`, `WARMUP`, the Vulkan expert tier's `COLI_VK_TIER*` β see [OLMoE engine](#olmoe-engine-olmoe) || `deepseek_v4` | `c/deepseek_v4.c` | `CTX`, the `V4_*` / `DSV4_*` families and the two `COLI_CUDA_*_BATCH` gates β see [DeepSeek V4 engine](#deepseek-v4-engine-deepseek_v4); note that the CUDA section below describes `colibri.c` knobs (`COLI_CUDA`, `CUDA_DENSE`, ...) which the V4 engine does not read β its GPU switch is `DSV4_CUDA` || `deepseek_v41` | `c/deepseek_v41.c` | the `V41_*` family: see [DeepSeek V4.1 engine](#deepseek-v41-engine-deepseek_v41) || `mimo` | `c/mimo.c` | the `MIMO_*` family: see [MiMo-V2.6 engine](#mimo-v26-engine-mimo) |@@ -244,7 +244,7 @@ With `COLI_VULKAN=1` the engines on the shared tier keep a cache of routed exper| `COLI_VK_TIER_BALANCE` | on | `0`: the device takes every resident expert of a step. On, when a join keeps waiting for the device, the step's resident experts the CPU also holds in RAM go back to the CPU beyond the device's share, which moves with every join (down when the CPU waited more than a tenth of its own time, up when the device finished before 80% of it). || `COLI_VK_TIER_GEMM_ROWS` | `16` | Rows from which an expert of a step takes the tiled GEMM (prefill) instead of the per-row GEMV; `0` never. Below it a row's bits do not depend on how many rows share the dispatch. || `COLI_VK_TIER_QUEUE` | a second queue | `0`: the tier's batches share the main queue with the dense matmuls (they then serialize). Unset: a second queue of the main family, else a compute-only family's (RADV), else shared (Lavapipe). |-| `COLI_VK_DENSE` | on; off on a device sharing the CPU's RAM while the tier is on | Where the dense (non-expert) matrices of qwen36, qwen38, inkling, olmoe, mimo, deepseek_v41, deepseek_v4 and qwenimage run. `0`: on the CPU (with the tier on, the device takes the routed experts only); `1`: on the device whatever the device. Unset: on the device, except on an integrated GPU or a CPU device (Lavapipe) while the engine runs the expert tier (today qwen36 and qwen38), where they stay on the CPU: there the dense matmuls cost more than the tier gains (measured on a Radeon 780M). The `[VK] <engine>: device ready, dense matrices on the ...` line says which and why. The GLM engine reads it through the same rule with its own default, off (see above). |+| `COLI_VK_DENSE` | on; off on a device sharing the CPU's RAM while the tier is on | Where the dense (non-expert) matrices of qwen36, qwen38, inkling, olmoe, mimo, deepseek_v41, deepseek_v4 and qwenimage run. `0`: on the CPU (with the tier on, the device takes the routed experts only); `1`: on the device whatever the device. Unset: on the device, except on an integrated GPU or a CPU device (Lavapipe) while the engine runs the expert tier (today qwen36, qwen38, inkling and olmoe), where they stay on the CPU: there the dense matmuls cost more than the tier gains (measured on a Radeon 780M). The `[VK] <engine>: device ready, dense matrices on the ...` line says which and why. The GLM engine reads it through the same rule with its own default, off (see above). |### Second Vulkan device (opt-in)@@ -437,6 +437,7 @@ Read **only** by `c/inkling.c`.| `COLI_KV_PREFIX` | on, except DeepSeek V4.1 | `0` disables KV-prefix reuse; on `deepseek_v41` reuse is OFF until you set `1`. That engine reads one set of index keys for a prefilled position and another for a decoded one, both the vendor's, so a prefix holding an earlier turn's generated tokens answers differently than the same text read cold. A resumed prefill is exact. || `GPU_DEV` | `0` | CUDA device index for the inkling CUDA backend. || `NOGPU` | unset | If set, skip GPU init entirely (both CUDA and Metal), regardless of the other GPU variables. |+| `COLI_VULKAN` | `0` | `VK=1` build: the dense and shared-expert matrices on the Vulkan device, and the routed experts on the shared expert tier (`COLI_VK_TIER*`, `COLI_VK_DENSE`, see [Vulkan](#vulkan-any-gpu-with-a-vulkan-12-driver)) in the form the cache holds them: the int4 or int8 container, the runtime int8 quantization, f32 at `bits=0`. The warm start reads the cache-warming history (`<snap>/.coli_usage` or `PIN=<path>`) in the generate and serve modes; the `ref.json` oracle reads none. With the CUDA backend or Metal on, the Vulkan tier stays off. |## Qwen3.6 engine (`qwen36`)@@ -560,6 +561,7 @@ Read **only** by `c/olmoe.c`. This is the sister engine used for streaming-cache| `SMOOTH` | `0.3` | EMA factor for routing momentum (gate logits smoothed across tokens). Clamped to [0, 0.95]. || `CONF_LIMIT` | `0.92` | Confidence ceiling for the router prediction. Clamped to [0.1, 1.0]. || `EXPERT_DROP` | `0` (off) | Drop experts below the confidence threshold instead of loading them (quality/speed experiment). |+| `COLI_VULKAN` | `0` | `VK=1` build: attention, router and lm_head on the Vulkan device, and the routed experts (int8 rows) on the shared expert tier (`COLI_VK_TIER*`, `COLI_VK_DENSE`, see [Vulkan](#vulkan-any-gpu-with-a-vulkan-12-driver)). The warm start reads the history `COLI_USAGE` names; without it the tier fills as experts pass by. |---diff --git a/docs/inkling.md b/docs/inkling.mdindex c6616bb7..1c6e35c4 100644--- a/docs/inkling.md+++ b/docs/inkling.md@@ -195,6 +195,34 @@ SNAP=~/Models/inkling_i4 ./c/inkling -f warmup_prompts.txt -n 32| `TOPP=<p>` | adaptive routing: keep routed experts up to cumulative weight `p`, drop the tail. **Trims the routing** β fewer experts read per token (the lever that matters on a disk-bound host), but a different computation from the declared top-k. Off by default; the run reports `[topp] β¦ N/M routed used (X% trimmed)` so the trade is measurable. Same semantics as `TOPP` in `colibri.c` and `K3_TOPP` in `kimi_k3.c` || First positional arg | expert-cache cap per layer (`0` = auto-size from free RAM) |+## Vulkan: the routed experts on the GPU++`make -C c inkling VK=1`, then `COLI_VULKAN=1`: the routed experts go to the shared+Vulkan expert tier ([vulkan.md](vulkan.md#inkling-and-olmoe)), a cache of experts on+the device that fills at startup from the history the cache warming above keeps+(`SNAP/.coli_usage`, or `PIN=<path>`; the `ref.json` oracle reads none) and adapts+while you chat. Every MoE layer sends the resident experts' (token, rank) pairs to the+device as one batch while the CPU computes the rest and the shared experts, then adds+every rank in routing order, so the sum's order does not depend on what was resident.++- The experts go over as the cache holds them, nothing requantized: the int4+ container (packed nibbles, one scale per row), the int8 container, the runtime+ quantization (`bits` 2 to 8) and f32 (`bits=0`). The fused `gate_up` tensor is+ read as two matrices, gate from row 0 and up from row I.+- The dense matrices (attention, dense MLP, shared experts, lm_head) go to the+ device in their RAM form, the `dense-int4g64/` container's int8 and int4-g64+ included, except on a GPU that shares the CPU's RAM while the tier is on: there+ they stay on the CPU unless `COLI_VK_DENSE=1`.+- With the CUDA backend (`CUDA=1` and a device) or Metal on, the Vulkan tier stays+ off: the other backend wins.+- One line per run and serve turn, `[VK] tier inkling run: device N of M routed+ experts ...`; the dashboard's expert map shows a device-resident expert as tier 2.+ The knobs are `COLI_VK_TIER*` in [ENVIRONMENT.md](ENVIRONMENT.md#vulkan-any-gpu-with-a-vulkan-12-driver).++Status: correct, not yet measured for speed. CI checks the tiny fixture on Lavapipe in+every expert format above, under eviction, with a warm start and through serve, and+again under ASan and UBSan; the 975B has not run on it.+## Performance (975B, Ryzen 9 7900 / 24t, 187 GB DDR5, RTX A6000, NVMe)24-token greedy decode, 5-token prompt, commit-tagged runs, single run each.diff --git a/docs/vulkan.md b/docs/vulkan.mdindex 50cce46d..b83f3257 100644--- a/docs/vulkan.md+++ b/docs/vulkan.md@@ -101,14 +101,14 @@ runs the routed-expert tier ([below](#the-routed-expert-tier-vk_tierc)): there tstay on the CPU and the device takes the experts. On a Radeon 780M the dense matmuls,one synchronous call each at the GPU's 800 MHz floor, cost more than the tier gained(Qwen3.8 decode at 2.35 tok/s with them on the device, 3.80 without). Today that-case is qwen36 and qwen38; an engine that moves to the tier inherits it. A discrete-GPU keeps the dense matrices on the device by default. The GLM engine above reads+case is qwen36, qwen38, inkling and olmoe; an engine that moves to the tier inherits+it. A discrete GPU keeps the dense matrices on the device by default. The GLM engine above readsthe same variable through the same function with its own default, off.What these engines put on the device is their **resident** matrices, in the formthey already hold in RAM, uploaded at the first multiply (MiMo uploads them at-startup). Routed experts arrive from disk on every miss; qwen36 and qwen38 keep a-cache of them on the device with the shared expert tier+startup). Routed experts arrive from disk on every miss; qwen36, qwen38, inkling and+olmoe keep a cache of them on the device with the shared expert tier([below](#the-routed-expert-tier-vk_tierc)), MiMo an opt-in one of its own, theothers none yet.@@ -116,8 +116,8 @@ others none yet.|---|---|---|---|| qwen36 (Qwen3.6, Qwen3-Coder, Qwen3.8-27B) | the dense trunk; routed experts on the expert tier | int8 rows; int4-g64 with `COLI_DENSE_BITS=4`; f32 with `COLI_DENSE_I8=0`; experts int4-g64, int4 per row, int8 per row or gs64 | DeltaNet `dn_a`/`dn_b`, vision tower, the experts the tier does not hold || qwen38 (Qwen3.8 Flash Next) | the trunk; routed experts on the expert tier | int8 trunk rows, bf16, f32 (`Q38_NATIVE_BF16=0`); experts int4-g64 (sidecar), FP8 128x128 blocks, bf16 | the MTP head's experts, the experts the tier does not hold |-| inkling | dense and shared-expert matrices | int8 and int4-g64 (dense-int4g64 container), f32, bf16 | routed experts, embedding and audio lookups, CUDA residents; bf16 on CPUs with the AVX512-BF16 dot (see below) |-| olmoe | attention q/k/v/o, router, lm_head | f32 | routed experts, embedding |+| inkling | dense and shared-expert matrices; routed experts on the expert tier | int8 and int4-g64 (dense-int4g64 container), f32, bf16; experts int4 or int8 per row (container or runtime quantization), f32 | embedding and audio lookups, CUDA residents (with CUDA or Metal on, the experts too); bf16 on CPUs with the AVX512-BF16 dot (see below); the experts the tier does not hold |+| olmoe | attention q/k/v/o, router, lm_head; routed experts on the expert tier | f32; experts int8 per row | embedding, the experts the tier does not hold || deepseek_v41 | the trunk, vision included | fp8 in 32x32 ue8m0 tiles, bf16 | routed experts || deepseek_v4 | resident dense layers, head, router, compressors | fp8 in 128x128 blocks, bf16 | routed experts, the indexer's `weights_proj`, DSpark stages, the `--oracle` path || mimo | trunk and vision tower; up to `MIMO_VK_EXPERTS=N` routed experts | native fp8/bf16, int8, f32 (`MIMO_DENSE_BITS`); experts as MXFP4 | router |@@ -308,7 +308,8 @@ Engines on the tier today: **qwen36** (Qwen3.6, Qwen3-Coder, the 2.4T geometry:per row or gs64, int4 per row or gs64 from either expert kernel, the mixed int4/int8container) and **qwen38** (Qwen3.8 Flash Next: the int4-g64 sidecar, the release'sFP8 with 128x128 block scales, BF16). The MTP head's layer of qwen38 stays on the CPU-(its experts are FP8 beside an int4 sidecar). GLM-5.2's `COLI_VK_EXPERTS`, Kimi K3's+(its experts are FP8 beside an int4 sidecar). **inkling** and **olmoe** as well, see+[Inkling and OLMoE](#inkling-and-olmoe). GLM-5.2's `COLI_VK_EXPERTS`, Kimi K3's`K3_VK` and MiMo's `MIMO_VK_EXPERTS` are the older per-engine tiers; the others moveto this one in the next phase, see [Adding an engine](#adding-an-engine-to-the-tier).@@ -512,6 +513,26 @@ converts `expert_ffn.h`'s planar int4 and spreads Qwen3.8's FP8 block scales). Tbackend enables the extension when the device has it; nothing outside the harnessuses it.+### Inkling and OLMoE++Both run their routed experts on this tier with `COLI_VULKAN=1`, in the form their+RAM caches hold them; nothing is requantized.++| Engine | Experts in RAM (`VktSrc`) | Device | History for the warm start | Notes |+|---|---|---|---|---|+| inkling | int4 container `I4U_PAIRS_ROW`; int8 container `I8_ROW`; runtime-quantized int8 rows, `I8_AS_I4_ROW` at 2 to 4 bits and `I8_ROW` above; f32 at `bits=0` `F32`. Gate and up come from the fused `gate_up` tensor, up I rows in | fmt 2, 1, 2 or 1, 10 | `<snap>/.coli_usage`, or `PIN=<path>`, in the generate and serve modes; the ref.json oracle reads none and fills the tier as experts pass by | the two shared experts run beside the batch, on the CPU or, with the dense matrices, on the device; `TOPP`'s trimmed ranks reach neither side; with CUDA or Metal on, the Vulkan tier stays off |+| olmoe | int8 rows `I8_ROW`, gate, up and down as the merged container holds them | fmt 1 | `COLI_USAGE` | `PILOT`'s prefetcher skips experts the device holds; a slot is handed to the tier only under the cache lock, while it still holds that expert |++Each layer step routes every row first, then runs per block of 64 rows: the block's+resident experts go to the device as one batch, the CPU computes the other pairs with+the kernels and the cache rounds it always uses, and every rank joins its row in+routing order. The device multiplies f32 activations, as both engines' default CPU+kernels do; `IDOT=1` (opt-in on both) rounds the CPU's activations and the device's+not. With `COLI_VULKAN` unset, or in a build without `VK=1`, the stdout and the+stderr (timings aside) and every logit vector of 36 configurations per build (each+expert format, caps 1 to 8, `TOPP`, `IDOT`, the generate and serve modes, `PPL`,+`ROUTE_TRACE`) are the bytes of the build before the tier.+## Adding an engine to the tierThe integration steps are in [`c/vk_tier.h`](../c/vk_tier.h); in short:@@ -542,12 +563,10 @@ What each remaining engine needs, from reading its code:|---|---|---|---|---|| colibri.c (GLM-5.2) | int4 per row `I4U_PAIRS_ROW`, int4-gs `I4U_PAIRS_GS`, int3-g64 `I3_G64`; gate, up, down separate | SwiGLU | `moe()`'s Vulkan block replaces the `COLI_VK_EXPERTS` registry | sums per expert in union order today: move to rank order; fmt 6 (E8/IQ3, rotated input) stays on the CPU; the MTP layer is int8 (another format); the block now serves only S <= 4 || glm53 | int4 gs64 `I4U_PAIRS_GS` 64 | SwiGLU with `swiglu_limit` | `ffn_layer` | its CPU clamps even at limit 0 (no guard), the shader treats 0 as no clamp: pass the config's value and check `L > 0` on the CPU side first; shared expert written first |-| inkling | one `gate_up` tensor [2I, D], rows 0..I-1 gate: pass `g = p13`, `u = p13 + I rows`; int4 per row `I4U_PAIRS_ROW` or int8 `I8_ROW` | SwiGLU | `moe` | routed and shared weights normalized together, `route_scale x rgs` already in the weights || kimi_k3 | MXFP4 `MXFP4_E8M0` 32, gate `w1`, up `w3`, down `w2` | SiTU-GLU, a = 4, b = 25 (`VKT_ACT_SITU`) | `moe_forward` / `expert_apply` (experts in the latent space) | replaces `K3_VK` (synchronous, never evicts); the CPU's `K3_IDOT` rounds activations to int8 || deepseek_v41 | MXFP4 `MXFP4_E8M0` 32 | SwiGLU with limit | `moe_run_at` (already sums per (row, rank) in rank order) | DSpark stages have caches of their own || deepseek_v4 | FP4 + ue8m0/32 = `MXFP4_E8M0` 32; pinned experts are repacked rows16 | SwiGLU with limit, **plus** bf16 rounding of gate/up, the route weight applied before down and bf16 rounding of the output | `moe_token_pipeline` (ascending expert id), `v4_moe_batch_union` | needs an activation variant with those roundings and the host's E4M3 rounding of x (as its fp8 dense path does); undo rows16 or keep pinned experts off the tier; hash-routed layers || mimo | MXFP4 `MXFP4_E8M0` 32, stored down, ds, gate, gs, up, us | SwiGLU | `moe` | replaces `MIMO_VK_EXPERTS` (synchronous, never evicts) |-| olmoe | int8 per row `I8_ROW`, one merged tensor g, u, d | SwiGLU | `moe` | history only with `COLI_USAGE` |## Correctness@@ -571,6 +590,11 @@ What each remaining engine needs, from reading its code:in flight. `tests/vulkan_engines.sh qwen` gives the CPU's tokens with the tier onin every qwen36 and qwen38 expert format, under eviction, with MTP and with thetrunk on the CPU; `qwen-sanitize` runs the same under ASan and UBSan.+ `tests/vulkan_engines.sh inkling-olmoe` does the same for inkling (every expert+ format above, under the f32, bf16 and dense-int4g64 snapshots, `TOPP`) and olmoe+ (with `PILOT`'s worker), under eviction, with a warm start, and through both+ engines' serve tests (prefix reuse, the dashboard, Brio); `inkling-olmoe-sanitize`+ runs them under ASan and UBSan.- int4 weights decode as offset-binary (nibbleβ8), byte-identical layout tothe CPU path β no repacking.- Khronos validation layers: the backend never enables them, so the loader