Run frontier MoE models on hardware you already own β pure C, zero deps, experts streamed from disk. Tiny engine, immense model. π¦
C
Merge branch 'feat/vk-tier-kimi-mimo' into feat/vk-tier-all-engines
JustVugg037d436dd0d9
diff --cc c/tests/vulkan_engines.shindex bcfef797,b398dd80..1c894a89--- a/c/tests/vulkan_engines.sh+++ b/c/tests/vulkan_engines.sh@@@ -648,8 -600,9 +777,10 @@@ case "${1:-}" iqwen) family_qwen ;;qwen-sanitize) family_qwen_sanitize ;;inkling-olmoe) family_inkling_olmoe ;;+ inkling-olmoe-sanitize) family_inkling_olmoe_sanitize ;;mimo-qwenimage) family_mimo_qwenimage ;;deepseek) family_deepseek ;;- *) echo "usage: $0 shader|qwen|qwen-sanitize|inkling-olmoe|mimo-qwenimage|deepseek" >&2; exit 2 ;;+ kimi) family_kimi ;;+ kimi-mimo-sanitize) family_kimi_mimo_sanitize ;;+ *) echo "usage: $0 shader|qwen|qwen-sanitize|inkling-olmoe|mimo-qwenimage|deepseek|kimi|kimi-mimo-sanitize" >&2; exit 2 ;;esacdiff --cc docs/ENVIRONMENT.mdindex 65666c74,b332f7d5..02b2bf23--- a/docs/ENVIRONMENT.md+++ b/docs/ENVIRONMENT.md@@@ -14,14 -14,14 +14,14 @@@ what follows, but the sister engines re|---|---|---|| `colibri` | `c/colibri.c` | everything below except the three sections named for another engine || `glm53` | `c/glm53.c` | the `GLM53_*` family and `COLI_MAP_EXPERTS`: see [GLM-5.3-Flash engine](#glm-53-flash-engine-glm53) |- | `kimi_k3` | `c/kimi_k3.c` | the `K3_*` family β see [Kimi K3 engine](#kimi-k3-engine-kimi_k3) |+ | `kimi_k3` | `c/kimi_k3.c` | the `K3_*` family, and the Vulkan expert tier's `COLI_VK_TIER*` β see [Kimi K3 engine](#kimi-k3-engine-kimi_k3) |-| `inkling` | `c/inkling.c` | `INK_*`, plus `CTX_MAX`, `PIN_N`, `REP_PEN`, `GPU_DEV`, `NOGPU` β see [Inkling engine](#inkling-engine-inkling) |+| `inkling` | `c/inkling.c` | `INK_*`, plus `CTX_MAX`, `PIN_N`, `REP_PEN`, `GPU_DEV`, `NOGPU`, the Vulkan expert tier's `COLI_VK_TIER*` β see [Inkling engine](#inkling-engine-inkling) || `qwen36` | `c/qwen36.c` | `QWEN_*`, `Q36_*`, its dense/CUDA-tier controls, the `CACHE_ROUTE` family (VRAM tier over RAM cache), and the Vulkan expert tier's `COLI_VK_TIER*` β see [Qwen3.6 engine](#qwen36-engine-qwen36) || `qwen38` | `c/qwen38.c` | `Q38_MAXT`, `Q38_EOS`, `Q38_NATIVE_FP8`, `Q38_NATIVE_BF16`, `Q38_EXPERT_INT4`, `Q38_MTP`, `Q38_PREFILL_BATCH`, `Q38_TRUNK_CPU_INT8`, `Q38_FP8_KERNEL`, `COLI_TIMERS`, the Vulkan expert tier's `COLI_VK_TIER*` β see [Qwen3.8 engine](#qwen38-engine-qwen38) |-| `olmoe` | `c/olmoe.c` | `HOT`, `WIDE`, `SMOOTH`, `CONF_LIMIT`, `MAX_NEW`, `CHAT`, `EXPERT_DROP`, `WARMUP` β see [OLMoE engine](#olmoe-engine-olmoe) |+| `olmoe` | `c/olmoe.c` | `HOT`, `WIDE`, `SMOOTH`, `CONF_LIMIT`, `MAX_NEW`, `CHAT`, `EXPERT_DROP`, `WARMUP`, the Vulkan expert tier's `COLI_VK_TIER*` β see [OLMoE engine](#olmoe-engine-olmoe) || `deepseek_v4` | `c/deepseek_v4.c` | `CTX`, the `V4_*` / `DSV4_*` families and the two `COLI_CUDA_*_BATCH` gates β see [DeepSeek V4 engine](#deepseek-v4-engine-deepseek_v4); note that the CUDA section below describes `colibri.c` knobs (`COLI_CUDA`, `CUDA_DENSE`, ...) which the V4 engine does not read β its GPU switch is `DSV4_CUDA` || `deepseek_v41` | `c/deepseek_v41.c` | the `V41_*` family: see [DeepSeek V4.1 engine](#deepseek-v41-engine-deepseek_v41) |- | `mimo` | `c/mimo.c` | the `MIMO_*` family: see [MiMo-V2.6 engine](#mimo-v26-engine-mimo) |+ | `mimo` | `c/mimo.c` | the `MIMO_*` family, and the Vulkan expert tier's `COLI_VK_TIER*`: see [MiMo-V2.6 engine](#mimo-v26-engine-mimo) |Setting an `INK_*` variable while running `colibri` does nothing, and viceversa; nothing warns you about it. A few variables are genuinely shared because@@@ -229,7 -229,7 +229,7 @@@ Per-drive byte counts are reported in| `COLI_VK_GEMM_TILE` | measured | `bm,bn,bk,tm,tn[,pf]`: one tile for every width of the fp32 GEMM instead of the measured pair. Tuning only (`COLI_VK_TEST_GEMM_BENCH`). || `COLI_VK_COOP_TILE` | measured | `bm,bn,wm,wn,bk`: one tile for every width of the cooperative-matrix GEMM. Tuning only. |- ### The routed-expert tier (`vk_tier.c`; qwen36, qwen38)-### The routed-expert tier (`vk_tier.c`; qwen36, qwen38, kimi_k3, mimo)++### The routed-expert tier (`vk_tier.c`; qwen36, qwen38, inkling, olmoe, kimi_k3, mimo)With `COLI_VULKAN=1` the engines on the shared tier keep a cache of routed experts on the device that adapts while you chat, and compute the resident ones of each layer step while the CPU computes the rest. See [vulkan.md](vulkan.md#the-routed-expert-tier-vk_tierc). With the CUDA expert tier also on, CUDA wins and this tier stays off.@@@ -244,7 -244,7 +244,7 @@@| `COLI_VK_TIER_BALANCE` | on | `0`: the device takes every resident expert of a step. On, when a join keeps waiting for the device, the step's resident experts the CPU also holds in RAM go back to the CPU beyond the device's share, which moves with every join (down when the CPU waited more than a tenth of its own time, up when the device finished before 80% of it). || `COLI_VK_TIER_GEMM_ROWS` | `16` | Rows from which an expert of a step takes the tiled GEMM (prefill) instead of the per-row GEMV; `0` never. Below it a row's bits do not depend on how many rows share the dispatch. || `COLI_VK_TIER_QUEUE` | a second queue | `0`: the tier's batches share the main queue with the dense matmuls (they then serialize). Unset: a second queue of the main family, else a compute-only family's (RADV), else shared (Lavapipe). |- | `COLI_VK_DENSE` | on; off on a device sharing the CPU's RAM while the tier is on | Where the dense (non-expert) matrices of qwen36, qwen38, inkling, olmoe, mimo, deepseek_v41, deepseek_v4 and qwenimage run. `0`: on the CPU (with the tier on, the device takes the routed experts only); `1`: on the device whatever the device. Unset: on the device, except on an integrated GPU or a CPU device (Lavapipe) while the engine runs the expert tier (today qwen36, qwen38, inkling and olmoe), where they stay on the CPU: there the dense matmuls cost more than the tier gains (measured on a Radeon 780M). The `[VK] <engine>: device ready, dense matrices on the ...` line says which and why. The GLM engine reads it through the same rule with its own default, off (see above). |-| `COLI_VK_DENSE` | on; off on a device sharing the CPU's RAM while the tier is on | Where the dense (non-expert) matrices of qwen36, qwen38, inkling, olmoe, mimo, deepseek_v41, deepseek_v4 and qwenimage run. `0`: on the CPU (with the tier on, the device takes the routed experts only); `1`: on the device whatever the device. Unset: on the device, except on an integrated GPU or a CPU device (Lavapipe) while the engine runs the expert tier (today qwen36, qwen38, kimi_k3 and mimo), where they stay on the CPU: there the dense matmuls cost more than the tier gains (measured on a Radeon 780M). The `[VK] <engine>: device ready, dense matrices on the ...` line says which and why. The GLM engine reads it through the same rule with its own default, off (see above). |++| `COLI_VK_DENSE` | on; off on a device sharing the CPU's RAM while the tier is on | Where the dense (non-expert) matrices of qwen36, qwen38, inkling, olmoe, mimo, deepseek_v41, deepseek_v4 and qwenimage run. `0`: on the CPU (with the tier on, the device takes the routed experts only); `1`: on the device whatever the device. Unset: on the device, except on an integrated GPU or a CPU device (Lavapipe) while the engine runs the expert tier (today qwen36, qwen38, inkling, olmoe, kimi_k3 and mimo), where they stay on the CPU: there the dense matmuls cost more than the tier gains (measured on a Radeon 780M). The `[VK] <engine>: device ready, dense matrices on the ...` line says which and why. The GLM engine reads it through the same rule with its own default, off (see above). |### Second Vulkan device (opt-in)diff --cc docs/vulkan.mdindex b83f3257,aec699de..2913670e--- a/docs/vulkan.md+++ b/docs/vulkan.md@@@ -101,26 -101,26 +101,27 @@@ runs the routed-expert tier ([below](#tstay on the CPU and the device takes the experts. On a Radeon 780M the dense matmuls,one synchronous call each at the GPU's 800 MHz floor, cost more than the tier gained(Qwen3.8 decode at 2.35 tok/s with them on the device, 3.80 without). Today that- case is qwen36, qwen38, inkling and olmoe; an engine that moves to the tier inherits- it. A discrete GPU keeps the dense matrices on the device by default. The GLM engine above reads-case is qwen36, qwen38, kimi_k3 and mimo; an engine that moves to the tier inherits it. A discrete-GPU keeps the dense matrices on the device by default. The GLM engine above reads--the same variable through the same function with its own default, off.++case is qwen36, qwen38, inkling, olmoe, kimi_k3 and mimo; an engine that moves to the++tier inherits it. A discrete GPU keeps the dense matrices on the device by default. The++GLM engine above reads the same variable through the same function with its own++default, off.What these engines put on the device is their **resident** matrices, in the form- they already hold in RAM, uploaded at the first multiply (MiMo uploads them at- startup). Routed experts arrive from disk on every miss; qwen36, qwen38, inkling and- olmoe keep a cache of them on the device with the shared expert tier- ([below](#the-routed-expert-tier-vk_tierc)), MiMo an opt-in one of its own, the- others none yet.+ they already hold in RAM, uploaded at the first multiply (MiMo and Kimi K3 upload+ them at startup). Routed experts arrive from disk on every miss; qwen36, qwen38,-Kimi K3 and MiMo keep a cache of them on the device with the shared expert tier-([below](#the-routed-expert-tier-vk_tierc)), the others none yet.++inkling, olmoe, Kimi K3 and MiMo keep a cache of them on the device with the shared++expert tier ([below](#the-routed-expert-tier-vk_tierc)), the others none yet.| Engine | On the device | Weight formats | Stays on the CPU ||---|---|---|---|| qwen36 (Qwen3.6, Qwen3-Coder, Qwen3.8-27B) | the dense trunk; routed experts on the expert tier | int8 rows; int4-g64 with `COLI_DENSE_BITS=4`; f32 with `COLI_DENSE_I8=0`; experts int4-g64, int4 per row, int8 per row or gs64 | DeltaNet `dn_a`/`dn_b`, vision tower, the experts the tier does not hold || qwen38 (Qwen3.8 Flash Next) | the trunk; routed experts on the expert tier | int8 trunk rows, bf16, f32 (`Q38_NATIVE_BF16=0`); experts int4-g64 (sidecar), FP8 128x128 blocks, bf16 | the MTP head's experts, the experts the tier does not hold |-| inkling | dense and shared-expert matrices | int8 and int4-g64 (dense-int4g64 container), f32, bf16 | routed experts, embedding and audio lookups, CUDA residents; bf16 on CPUs with the AVX512-BF16 dot (see below) |-| olmoe | attention q/k/v/o, router, lm_head | f32 | routed experts, embedding |+| inkling | dense and shared-expert matrices; routed experts on the expert tier | int8 and int4-g64 (dense-int4g64 container), f32, bf16; experts int4 or int8 per row (container or runtime quantization), f32 | embedding and audio lookups, CUDA residents (with CUDA or Metal on, the experts too); bf16 on CPUs with the AVX512-BF16 dot (see below); the experts the tier does not hold |+| olmoe | attention q/k/v/o, router, lm_head; routed experts on the expert tier | f32; experts int8 per row | embedding, the experts the tier does not hold || deepseek_v41 | the trunk, vision included | fp8 in 32x32 ue8m0 tiles, bf16 | routed experts || deepseek_v4 | resident dense layers, head, router, compressors | fp8 in 128x128 blocks, bf16 | routed experts, the indexer's `weights_proj`, DSpark stages, the `--oracle` path |- | mimo | trunk and vision tower; up to `MIMO_VK_EXPERTS=N` routed experts | native fp8/bf16, int8, f32 (`MIMO_DENSE_BITS`); experts as MXFP4 | router |+ | kimi_k3 (Kimi K3) | the shared experts' matrices (one row at a time: decode); routed experts on the expert tier | shared experts int8 rows, int4-g64, f32 (`K3_BITS`); experts MXFP4 with ue8m0 scales (fmt 7), SiTU-GLU in the latent space | KDA, MLA, the latent projections, router, head, prefill's shared experts, the experts the tier does not hold |+ | mimo | trunk and vision tower; routed experts on the expert tier | native fp8/bf16, int8, f32 (`MIMO_DENSE_BITS`); experts MXFP4 with e8m0 scales (fmt 7) | router, the experts the tier does not hold || qwenimage | the DiT's matrices | int8, bf16, f32 (`COLI_IMG_BITS`) | text encoder, VAE, attention |Each engine ends a run, and each serve turn, with@@@ -308,10 -315,18 +316,19 @@@ Engines on the tier today: **qwen36** (per row or gs64, int4 per row or gs64 from either expert kernel, the mixed int4/int8container) and **qwen38** (Qwen3.8 Flash Next: the int4-g64 sidecar, the release'sFP8 with 128x128 block scales, BF16). The MTP head's layer of qwen38 stays on the CPU-(its experts are FP8 beside an int4 sidecar). **kimi_k3** (Kimi K3: the checkpoint's+(its experts are FP8 beside an int4 sidecar). **inkling** and **olmoe** as well, see- [Inkling and OLMoE](#inkling-and-olmoe). GLM-5.2's `COLI_VK_EXPERTS`, Kimi K3's- `K3_VK` and MiMo's `MIMO_VK_EXPERTS` are the older per-engine tiers; the others move- to this one in the next phase, see [Adding an engine](#adding-an-engine-to-the-tier).++[Inkling and OLMoE](#inkling-and-olmoe). **kimi_k3** (Kimi K3: the checkpoint's+ MXFP4 with ue8m0 scales, SiTU-GLU on the device, the experts in the latent space,+ warm-started from its expert history) and **mimo** (MiMo-V2.6 Flash and Pro: the+ release's MXFP4, SwiGLU; no history, so the tier fills as experts pass by) joined+ it from their own tiers, which filled once and never evicted: Kimi K3's `K3_VK`,+ `K3_VK_GB`, `K3_VK_UP` and MiMo's `MIMO_VK_EXPERTS` are now read as this tier's+ switches ([ENVIRONMENT.md](ENVIRONMENT.md)). Both add every pair of a row in the+ CPU run's order (Kimi K3: its union's disk-offset order; MiMo: the row's routing+ order), and offer the experts the CPU computed after the join, so that a promotion+ that displaces a resident frees it at once. GLM-5.2's `COLI_VK_EXPERTS` is the older+ per-engine tier; the others move to this one in the next phase, see [Adding an+ engine](#adding-an-engine-to-the-tier).With the CUDA expert tier built and on (`COLI_CUDA=1`) as well, **CUDA wins**: theVulkan tier stays off and says so (`[VK] tier <engine>: the CUDA expert tier is on@@@ -563,10 -558,10 +580,8 @@@ What each remaining engine needs, from|---|---|---|---|---|| colibri.c (GLM-5.2) | int4 per row `I4U_PAIRS_ROW`, int4-gs `I4U_PAIRS_GS`, int3-g64 `I3_G64`; gate, up, down separate | SwiGLU | `moe()`'s Vulkan block replaces the `COLI_VK_EXPERTS` registry | sums per expert in union order today: move to rank order; fmt 6 (E8/IQ3, rotated input) stays on the CPU; the MTP layer is int8 (another format); the block now serves only S <= 4 || glm53 | int4 gs64 `I4U_PAIRS_GS` 64 | SwiGLU with `swiglu_limit` | `ffn_layer` | its CPU clamps even at limit 0 (no guard), the shader treats 0 as no clamp: pass the config's value and check `L > 0` on the CPU side first; shared expert written first |- | kimi_k3 | MXFP4 `MXFP4_E8M0` 32, gate `w1`, up `w3`, down `w2` | SiTU-GLU, a = 4, b = 25 (`VKT_ACT_SITU`) | `moe_forward` / `expert_apply` (experts in the latent space) | replaces `K3_VK` (synchronous, never evicts); the CPU's `K3_IDOT` rounds activations to int8 |-| inkling | one `gate_up` tensor [2I, D], rows 0..I-1 gate: pass `g = p13`, `u = p13 + I rows`; int4 per row `I4U_PAIRS_ROW` or int8 `I8_ROW` | SwiGLU | `moe` | routed and shared weights normalized together, `route_scale x rgs` already in the weights || deepseek_v41 | MXFP4 `MXFP4_E8M0` 32 | SwiGLU with limit | `moe_run_at` (already sums per (row, rank) in rank order) | DSpark stages have caches of their own || deepseek_v4 | FP4 + ue8m0/32 = `MXFP4_E8M0` 32; pinned experts are repacked rows16 | SwiGLU with limit, **plus** bf16 rounding of gate/up, the route weight applied before down and bf16 rounding of the output | `moe_token_pipeline` (ascending expert id), `v4_moe_batch_union` | needs an activation variant with those roundings and the host's E4M3 rounding of x (as its fp8 dense path does); undo rows16 or keep pinned experts off the tier; hash-routed layers |- | mimo | MXFP4 `MXFP4_E8M0` 32, stored down, ds, gate, gs, up, us | SwiGLU | `moe` | replaces `MIMO_VK_EXPERTS` (synchronous, never evicts) |-| olmoe | int8 per row `I8_ROW`, one merged tensor g, u, d | SwiGLU | `moe` | history only with `COLI_USAGE` |## Correctness@@@ -589,12 -584,12 +604,16 @@@independent of the batch, and the tier pool's budget with frees while a batch isin flight. `tests/vulkan_engines.sh qwen` gives the CPU's tokens with the tier onin every qwen36 and qwen38 expert format, under eviction, with MTP and with the- trunk on the CPU; `qwen-sanitize` runs the same under ASan and UBSan. `kimi` and- the mimo half of `mimo-qwenimage` do it for Kimi K3 and MiMo: Moonshot's and- Xiaomi's oracles with the tier on, the CPU's tokens in every dense format with the- trunk on the device and on the CPU, a budget of two experts that must evict, the- old switches, Kimi K3's warm start; `kimi-mimo-sanitize` runs the tier's- configurations under ASan and UBSan.+ trunk on the CPU; `qwen-sanitize` runs the same under ASan and UBSan.+ `tests/vulkan_engines.sh inkling-olmoe` does the same for inkling (every expert+ format above, under the f32, bf16 and dense-int4g64 snapshots, `TOPP`) and olmoe+ (with `PILOT`'s worker), under eviction, with a warm start, and through both+ engines' serve tests (prefix reuse, the dashboard, Brio); `inkling-olmoe-sanitize`- runs them under ASan and UBSan.++ runs them under ASan and UBSan. `kimi` and the mimo half of `mimo-qwenimage` do it++ for Kimi K3 and MiMo: Moonshot's and Xiaomi's oracles with the tier on, the CPU's++ tokens in every dense format with the trunk on the device and on the CPU, a budget++ of two experts that must evict, the old switches, Kimi K3's warm start;++ `kimi-mimo-sanitize` runs the tier's configurations under ASan and UBSan.- int4 weights decode as offset-binary (nibbleβ8), byte-identical layout tothe CPU path β no repacking.- Khronos validation layers: the backend never enables them, so the loader