Run frontier MoE models on hardware you already own — pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦
C
vulkan: COLI_VK_DENSE=0 keeps the dense chain off too (#1900)
JustVugg04f7e43df715
diff --git a/c/backend_vulkan.c b/c/backend_vulkan.cindex b985e31d..a9d066ce 100644--- a/c/backend_vulkan.c+++ b/c/backend_vulkan.c@@ -2189,6 +2189,9 @@ unsigned long long coli_vk_dense_host_dropped_bytes(void) { return __atomic_load* _ON (every forward) or _PREFILL (forwards of more than two rows: prompts, not decode* steps or an MTP verify).* COLI_VK_CHAIN set and non-empty: 0 off, 2 prefill only, any other number on.+ * Unset, COLI_VK_DENSE=0: off (the trunk stays on the CPU, as that variable says; the+ * chain's device copy of it would also take the expert tier's budget: on an 8 GB card+ * with qwen38 the tier got 1.87 GiB instead of 6 and decode lost 24%, #1900).* Unset, a discrete GPU: on.* Unset, an integrated GPU: `igpu` when the engine runs the expert tier (tier_on),* else off; an engine never timed on one passes COLI_VK_CHAIN_UNMEASURED, which is@@ -2204,13 +2207,16 @@ unsigned long long coli_vk_dense_host_dropped_bytes(void) { return __atomic_load* Unset, a CPU device (Lavapipe): off.* Printed as a [VK] line with an engine name (NULL: silent). */int coli_vk_chain_decide(const char *engine, int tier_on, int igpu) {- const char *e = getenv("COLI_VK_CHAIN");+ const char *e = getenv("COLI_VK_CHAIN"), *dn = getenv("COLI_VK_DENSE");char why[192];int on;if (e && *e) {int v = atoi(e);on = v == 0 ? COLI_VK_CHAIN_OFF : v == 2 ? COLI_VK_CHAIN_PREFILL : COLI_VK_CHAIN_ON;snprintf(why, sizeof why, "COLI_VK_CHAIN=%s", e);+ } else if (dn && *dn == '0') {+ on = COLI_VK_CHAIN_OFF;+ snprintf(why, sizeof why, "COLI_VK_DENSE=0: the trunk on the CPU; COLI_VK_CHAIN=1 turns the chain on");} else if (coli_vk_device_integrated()) {int measured = igpu != COLI_VK_CHAIN_UNMEASURED;on = tier_on && measured ? igpu : COLI_VK_CHAIN_OFF;diff --git a/c/tests/vulkan_engines.sh b/c/tests/vulkan_engines.shindex 5492374b..92139bb4 100755--- a/c/tests/vulkan_engines.sh+++ b/c/tests/vulkan_engines.sh@@ -433,6 +433,7 @@ family_qwen() {tier_gate qwen36 "qwen36 tier mixed int4/int8 cap=$cap" $D COLI_DENSE_I8=0 SNAP=qwen36_tiny64_d8 -- $cap 4 qwen36_tiny64/ref_full.jsondonetier_gate qwen36 "qwen36 tier alone (COLI_VK_DENSE=0)" COLI_VK_DENSE=0 COLI_DENSE_I8=0 SNAP=qwen36_tiny64_c -- 8 4 qwen36_tiny64/ref_full.json+ grep -q 'dense chain off (COLI_VK_DENSE=0' vk.log || { cat vk.log; fail "qwen36 tier alone: COLI_VK_DENSE=0 did not keep the chain off"; }EVICT=1 tier_gate qwen36 "qwen36 tier, a budget of two experts" COLI_VK_TIER_GB=0.00002 COLI_DENSE_I8=0 SNAP=qwen36_tiny_c -- 8 8 qwen36_tiny/ref_full.json# COLI_VK_TIER=0: the dense trunk alone, as before the tier; with no tier the default# puts the trunk on the device, Lavapipe includeddiff --git a/docs/ENVIRONMENT.md b/docs/ENVIRONMENT.mdindex 94e1a2ff..e5ec0114 100644--- a/docs/ENVIRONMENT.md+++ b/docs/ENVIRONMENT.md@@ -263,7 +263,7 @@ With `COLI_VULKAN=1` every MoE engine (qwen36, qwen38, inkling, olmoe, kimi_k3,| `COLI_VK_TIER_STREAM_HALF` | x rows within 32 MiB | Rows of a streaming sub-batch (tests: small ones cut experts into parts). || `COLI_VK_TIER_STREAM_PAR` | on | `0`: a streamed expert's conversion into its slot runs on the engine thread alone. || `COLI_VK_DENSE` | on; off on a device sharing the CPU's RAM while the tier is on | Where the dense (non-expert) matrices of qwen36, qwen38, inkling, olmoe, kimi_k3 (its shared experts), mimo, deepseek_v41, deepseek_v4, glm53, qwenimage, and laya and gliner_decide with `COLI_VK_CHAIN=0`, run. `0`: on the CPU (with the tier on, the device takes the routed experts only); `1`: on the device whatever the device. Unset: on the device, except on an integrated GPU or a CPU device (Lavapipe) while the engine runs the expert tier (every one of them but qwenimage, which has no routed experts), where they stay on the CPU: there the dense matmuls cost more than the tier gains (measured on a Radeon 780M). The `[VK] <engine>: device ready, dense matrices on the ...` line says which and why. The GLM engine reads it through the same rule with its own default, off (see above). |-| `COLI_VK_CHAIN` | on for a discrete GPU; qwen36 and olmoe on, qwen38 off, and mimo, inkling, colibri, glm53, kimi_k3, deepseek_v41 and deepseek_v4 off (not measured) on an integrated GPU with the tier; off on a CPU device | qwen36, qwen38, olmoe, inkling, mimo, colibri (GLM-5.2), glm53 (GLM-5.3 Flash), kimi_k3, deepseek_v41 (DeepSeek V4.1 Flash) and deepseek_v4: every layer's dense chain recorded into one submission per layer, the residual stream and the recurrent and KV state on the device (see [vulkan.md](vulkan.md#the-dense-chain-vk_chainc)). `0` off, `1` on, `2` on for prompts only (decode on the CPU). The `[VK] <engine>: dense chain ...` line says which and why. |+| `COLI_VK_CHAIN` | off with `COLI_VK_DENSE=0`; else on for a discrete GPU; qwen36 and olmoe on, qwen38 off, and mimo, inkling, colibri, glm53, kimi_k3, deepseek_v41 and deepseek_v4 off (not measured) on an integrated GPU with the tier; off on a CPU device | qwen36, qwen38, olmoe, inkling, mimo, colibri (GLM-5.2), glm53 (GLM-5.3 Flash), kimi_k3, deepseek_v41 (DeepSeek V4.1 Flash) and deepseek_v4: every layer's dense chain recorded into one submission per layer, the residual stream and the recurrent and KV state on the device (see [vulkan.md](vulkan.md#the-dense-chain-vk_chainc)). `0` off, `1` on, `2` on for prompts only (decode on the CPU). The `[VK] <engine>: dense chain ...` line says which and why. || `COLI_VK_DENSE_HOST` | auto | Whether the dense weights the device holds (the chain on, or `COLI_VK_DENSE` on) keep a copy in host RAM. `0`: the device only. Each resident dense matrix goes up at start and its host copy is given back; the steps the chain does not run take the device too; the CPU reads a matrix back from disk only when it needs one (a lost device), and the engine's expert cache no longer reserves the released host copies. `coli plan` still charges device allocations to physical RAM on integrated GPUs. `1`: the host copy stays as the CPU fallback, as before. Unset: the device only on an integrated GPU (its memory is the same RAM) and on a discrete GPU whose free memory, less 1 GiB, holds the dense weights; the host copy stays on a CPU device (Lavapipe), on a discrete GPU without room, and whenever the dense part runs on the CPU. The `[VK] <engine>: dense weights ...` line says which and why, a second one how much RAM was given back, and one at exit what was read back and the resident set. See [vulkan.md](vulkan.md#dense-weights-on-the-device-only). || `COLI_VK_CHAIN_ROWS` | from the budget, up to 8192 | Prompt rows per chunk of the chain's prefill. Unset or `auto`: the most rows whose scratch fits half the device memory free at the engine's first forward (also bounded by available physical RAM and Windows commit headroom), up to `COLI_VK_CHAIN_ROWS_MAX`; MiMo, GLM-5.3, Kimi K3 and DeepSeek V4 then hand the chain blocks of that chunk instead of `MIMO_CHUNK`, `GLM53_PREFILL_CHUNK`, `K3_CHUNK` and the 128-row block, when those are unset. Set: that many rows, the engines' own blocks as before. See [vulkan.md](vulkan.md#big-prompt-chunks-and-expert-streaming). || `COLI_VK_CHAIN_ROWS_MAX` | `8192` | The most rows the budget's chunk takes. |diff --git a/docs/vulkan.md b/docs/vulkan.mdindex 4fa7762a..3a883e61 100644--- a/docs/vulkan.md+++ b/docs/vulkan.md@@ -960,7 +960,10 @@ command buffer instead, and keeps the residual stream on the device from one layto the next. qwen36 (Qwen3.6, Qwen3-Coder, Qwen3.8-27B) and qwen38 (Qwen3.8 FlashNext) run it: by default on a discrete GPU, and for qwen36 on an integrated one withthe expert tier (see [the default](#the-chain-on-a-radeon-780m)); `COLI_VK_CHAIN=1`-anywhere. olmoe and inkling run it as well ([OLMoE and Inkling](#olmoe-and-inkling)).+anywhere. `COLI_VK_DENSE=0` keeps it off too, unless `COLI_VK_CHAIN` says otherwise: the+chain's copy of the trunk would take the expert tier's budget (on an 8 GB RTX 4070+Laptop with qwen38, disk-bound, the tier got 1.87 GiB instead of 6.00 and decode went+from 0.84 to 0.64 tok/s, #1900). olmoe and inkling run it as well ([OLMoE and Inkling](#olmoe-and-inkling)).colibri (GLM-5.2), glm53 (GLM-5.3 Flash) and kimi_k3 run it too, with the MLA,KDA and hyper-connection ops ([below](#glm-52-and-glm-53-flash-on-the-chain)), anddeepseek_v41 and deepseek_v4 with DeepSeek's own ([below](#deepseek-v41-flash-and-deepseek-v4-on-the-chain)).