Skip to content

Run frontier MoE models on hardware you already own — pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦

C

vulkan: COLI_VK_DENSE=0 keeps the dense chain off too (#1900)

JustVugg04f7e43df715
diff --git a/c/backend_vulkan.c b/c/backend_vulkan.c
index b985e31d..a9d066ce 100644
--- a/c/backend_vulkan.c
+++ b/c/backend_vulkan.c
@@ -2189,6 +2189,9 @@ unsigned long long coli_vk_dense_host_dropped_bytes(void) { return __atomic_load
* _ON (every forward) or _PREFILL (forwards of more than two rows: prompts, not decode
* steps or an MTP verify).
* COLI_VK_CHAIN set and non-empty: 0 off, 2 prefill only, any other number on.
+ * Unset, COLI_VK_DENSE=0: off (the trunk stays on the CPU, as that variable says; the
+ * chain's device copy of it would also take the expert tier's budget: on an 8 GB card
+ * with qwen38 the tier got 1.87 GiB instead of 6 and decode lost 24%, #1900).
* Unset, a discrete GPU: on.
* Unset, an integrated GPU: `igpu` when the engine runs the expert tier (tier_on),
* else off; an engine never timed on one passes COLI_VK_CHAIN_UNMEASURED, which is
@@ -2204,13 +2207,16 @@ unsigned long long coli_vk_dense_host_dropped_bytes(void) { return __atomic_load
* Unset, a CPU device (Lavapipe): off.
* Printed as a [VK] line with an engine name (NULL: silent). */
int coli_vk_chain_decide(const char *engine, int tier_on, int igpu) {
- const char *e = getenv("COLI_VK_CHAIN");
+ const char *e = getenv("COLI_VK_CHAIN"), *dn = getenv("COLI_VK_DENSE");
char why[192];
int on;
if (e && *e) {
int v = atoi(e);
on = v == 0 ? COLI_VK_CHAIN_OFF : v == 2 ? COLI_VK_CHAIN_PREFILL : COLI_VK_CHAIN_ON;
snprintf(why, sizeof why, "COLI_VK_CHAIN=%s", e);
+ } else if (dn && *dn == '0') {
+ on = COLI_VK_CHAIN_OFF;
+ snprintf(why, sizeof why, "COLI_VK_DENSE=0: the trunk on the CPU; COLI_VK_CHAIN=1 turns the chain on");
} else if (coli_vk_device_integrated()) {
int measured = igpu != COLI_VK_CHAIN_UNMEASURED;
on = tier_on && measured ? igpu : COLI_VK_CHAIN_OFF;
diff --git a/c/tests/vulkan_engines.sh b/c/tests/vulkan_engines.sh
index 5492374b..92139bb4 100755
--- a/c/tests/vulkan_engines.sh
+++ b/c/tests/vulkan_engines.sh
@@ -433,6 +433,7 @@ family_qwen() {
tier_gate qwen36 "qwen36 tier mixed int4/int8 cap=$cap" $D COLI_DENSE_I8=0 SNAP=qwen36_tiny64_d8 -- $cap 4 qwen36_tiny64/ref_full.json
done
tier_gate qwen36 "qwen36 tier alone (COLI_VK_DENSE=0)" COLI_VK_DENSE=0 COLI_DENSE_I8=0 SNAP=qwen36_tiny64_c -- 8 4 qwen36_tiny64/ref_full.json
+ grep -q 'dense chain off (COLI_VK_DENSE=0' vk.log || { cat vk.log; fail "qwen36 tier alone: COLI_VK_DENSE=0 did not keep the chain off"; }
EVICT=1 tier_gate qwen36 "qwen36 tier, a budget of two experts" COLI_VK_TIER_GB=0.00002 COLI_DENSE_I8=0 SNAP=qwen36_tiny_c -- 8 8 qwen36_tiny/ref_full.json
# COLI_VK_TIER=0: the dense trunk alone, as before the tier; with no tier the default
# puts the trunk on the device, Lavapipe included
diff --git a/docs/ENVIRONMENT.md b/docs/ENVIRONMENT.md
index 94e1a2ff..e5ec0114 100644
--- a/docs/ENVIRONMENT.md
+++ b/docs/ENVIRONMENT.md
@@ -263,7 +263,7 @@ With `COLI_VULKAN=1` every MoE engine (qwen36, qwen38, inkling, olmoe, kimi_k3,
| `COLI_VK_TIER_STREAM_HALF` | x rows within 32 MiB | Rows of a streaming sub-batch (tests: small ones cut experts into parts). |
| `COLI_VK_TIER_STREAM_PAR` | on | `0`: a streamed expert's conversion into its slot runs on the engine thread alone. |
| `COLI_VK_DENSE` | on; off on a device sharing the CPU's RAM while the tier is on | Where the dense (non-expert) matrices of qwen36, qwen38, inkling, olmoe, kimi_k3 (its shared experts), mimo, deepseek_v41, deepseek_v4, glm53, qwenimage, and laya and gliner_decide with `COLI_VK_CHAIN=0`, run. `0`: on the CPU (with the tier on, the device takes the routed experts only); `1`: on the device whatever the device. Unset: on the device, except on an integrated GPU or a CPU device (Lavapipe) while the engine runs the expert tier (every one of them but qwenimage, which has no routed experts), where they stay on the CPU: there the dense matmuls cost more than the tier gains (measured on a Radeon 780M). The `[VK] <engine>: device ready, dense matrices on the ...` line says which and why. The GLM engine reads it through the same rule with its own default, off (see above). |
-| `COLI_VK_CHAIN` | on for a discrete GPU; qwen36 and olmoe on, qwen38 off, and mimo, inkling, colibri, glm53, kimi_k3, deepseek_v41 and deepseek_v4 off (not measured) on an integrated GPU with the tier; off on a CPU device | qwen36, qwen38, olmoe, inkling, mimo, colibri (GLM-5.2), glm53 (GLM-5.3 Flash), kimi_k3, deepseek_v41 (DeepSeek V4.1 Flash) and deepseek_v4: every layer's dense chain recorded into one submission per layer, the residual stream and the recurrent and KV state on the device (see [vulkan.md](vulkan.md#the-dense-chain-vk_chainc)). `0` off, `1` on, `2` on for prompts only (decode on the CPU). The `[VK] <engine>: dense chain ...` line says which and why. |
+| `COLI_VK_CHAIN` | off with `COLI_VK_DENSE=0`; else on for a discrete GPU; qwen36 and olmoe on, qwen38 off, and mimo, inkling, colibri, glm53, kimi_k3, deepseek_v41 and deepseek_v4 off (not measured) on an integrated GPU with the tier; off on a CPU device | qwen36, qwen38, olmoe, inkling, mimo, colibri (GLM-5.2), glm53 (GLM-5.3 Flash), kimi_k3, deepseek_v41 (DeepSeek V4.1 Flash) and deepseek_v4: every layer's dense chain recorded into one submission per layer, the residual stream and the recurrent and KV state on the device (see [vulkan.md](vulkan.md#the-dense-chain-vk_chainc)). `0` off, `1` on, `2` on for prompts only (decode on the CPU). The `[VK] <engine>: dense chain ...` line says which and why. |
| `COLI_VK_DENSE_HOST` | auto | Whether the dense weights the device holds (the chain on, or `COLI_VK_DENSE` on) keep a copy in host RAM. `0`: the device only. Each resident dense matrix goes up at start and its host copy is given back; the steps the chain does not run take the device too; the CPU reads a matrix back from disk only when it needs one (a lost device), and the engine's expert cache no longer reserves the released host copies. `coli plan` still charges device allocations to physical RAM on integrated GPUs. `1`: the host copy stays as the CPU fallback, as before. Unset: the device only on an integrated GPU (its memory is the same RAM) and on a discrete GPU whose free memory, less 1 GiB, holds the dense weights; the host copy stays on a CPU device (Lavapipe), on a discrete GPU without room, and whenever the dense part runs on the CPU. The `[VK] <engine>: dense weights ...` line says which and why, a second one how much RAM was given back, and one at exit what was read back and the resident set. See [vulkan.md](vulkan.md#dense-weights-on-the-device-only). |
| `COLI_VK_CHAIN_ROWS` | from the budget, up to 8192 | Prompt rows per chunk of the chain's prefill. Unset or `auto`: the most rows whose scratch fits half the device memory free at the engine's first forward (also bounded by available physical RAM and Windows commit headroom), up to `COLI_VK_CHAIN_ROWS_MAX`; MiMo, GLM-5.3, Kimi K3 and DeepSeek V4 then hand the chain blocks of that chunk instead of `MIMO_CHUNK`, `GLM53_PREFILL_CHUNK`, `K3_CHUNK` and the 128-row block, when those are unset. Set: that many rows, the engines' own blocks as before. See [vulkan.md](vulkan.md#big-prompt-chunks-and-expert-streaming). |
| `COLI_VK_CHAIN_ROWS_MAX` | `8192` | The most rows the budget's chunk takes. |
diff --git a/docs/vulkan.md b/docs/vulkan.md
index 4fa7762a..3a883e61 100644
--- a/docs/vulkan.md
+++ b/docs/vulkan.md
@@ -960,7 +960,10 @@ command buffer instead, and keeps the residual stream on the device from one lay
to the next. qwen36 (Qwen3.6, Qwen3-Coder, Qwen3.8-27B) and qwen38 (Qwen3.8 Flash
Next) run it: by default on a discrete GPU, and for qwen36 on an integrated one with
the expert tier (see [the default](#the-chain-on-a-radeon-780m)); `COLI_VK_CHAIN=1`
-anywhere. olmoe and inkling run it as well ([OLMoE and Inkling](#olmoe-and-inkling)).
+anywhere. `COLI_VK_DENSE=0` keeps it off too, unless `COLI_VK_CHAIN` says otherwise: the
+chain's copy of the trunk would take the expert tier's budget (on an 8 GB RTX 4070
+Laptop with qwen38, disk-bound, the tier got 1.87 GiB instead of 6.00 and decode went
+from 0.84 to 0.64 tok/s, #1900). olmoe and inkling run it as well ([OLMoE and Inkling](#olmoe-and-inkling)).
colibri (GLM-5.2), glm53 (GLM-5.3 Flash) and kimi_k3 run it too, with the MLA,
KDA and hyper-connection ops ([below](#glm-52-and-glm-53-flash-on-the-chain)), and
deepseek_v41 and deepseek_v4 with DeepSeek's own ([below](#deepseek-v41-flash-and-deepseek-v4-on-the-chain)).