Skip to content

Run frontier MoE models on hardware you already own — pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦

C

docs/vulkan: what every engine puts on the device, and what it costs

JustVuggb1d83e80af9b
diff --git a/docs/ENVIRONMENT.md b/docs/ENVIRONMENT.md
index 7d8d1a15..0af9ed84 100644
--- a/docs/ENVIRONMENT.md
+++ b/docs/ENVIRONMENT.md
@@ -214,7 +214,7 @@ Per-drive byte counts are reported in a `MIRROR:` stats line. Combine with `DIRE
| Variable | Default | Effect |
|---|---|---|
-| `COLI_VULKAN` | off | Enable the Vulkan backend. Requires a `make VK=1` build; fails at startup (no silent fallback) if libvulkan or the compiled shaders are missing. |
+| `COLI_VULKAN` | off | Enable the Vulkan backend. Requires a `make VK=1` build. The GLM engine fails at startup (no silent fallback) if libvulkan or the compiled shaders are missing. The other engines (qwen36, qwen38, inkling, olmoe, deepseek_v41, deepseek_v4, mimo, qwenimage) print one line and run on the CPU. What each one puts on the device: [vulkan.md](vulkan.md#the-other-engines). |
| `COLI_VK_DEV` | unset | Select the primary Vulkan physical-device enumeration index. Without it, the backend prefers a discrete GPU, then integrated/virtual devices. |
| `COLI_VK_SHADERS` | auto | Path to the compiled `qmatmul.spv` **or** the directory holding the `.spv` set; the other shaders are found next to it. Unset: `shaders/` next to the binary, then CWD-relative `shaders/qmatmul.spv`. |
| `COLI_VK_EXPERTS` | `320` | Pinned VRAM expert tier size: top-N experts by `.coli_usage` heat uploaded once at startup and served from VRAM with no RAM slot or disk read. `0` disables the tier (experts stay on the CPU path). ~19 MB VRAM per int4 expert. |
diff --git a/docs/vulkan.md b/docs/vulkan.md
index 2b39cacb..6b12b97d 100644
--- a/docs/vulkan.md
+++ b/docs/vulkan.md
@@ -81,6 +81,62 @@ registry holds the same hot experts a RAM pin would, so the pin's RAM is
better spent on the adaptive LRU cache. Keep `PIN` set so AUTOPIN does not
re-pin from history.
+## The other engines
+
+Every engine links the same backend in a `VK=1` build (`make <engine> VK=1`;
+`make deepseek-v4 VK=1` for DeepSeek V4) and opens it with `COLI_VULKAN=1` once its
+weights are loaded. Kimi K3 has its own expert tier (`K3_VK`, see
+[ENVIRONMENT.md](ENVIRONMENT.md)) and glm53 its own section in
+[glm53-flash.md](glm53-flash.md). For the engines below, a missing device or missing
+shaders prints `[VK] <engine>: no usable Vulkan device ..., running on the CPU` and
+the run continues on the CPU. That differs from the GLM engine above, which exits.
+
+What these engines put on the device is their **resident** matrices, in the form
+they already hold in RAM, uploaded at the first multiply (MiMo uploads them at
+startup). Routed experts arrive from disk on every miss and stay on the CPU, with
+one opt-in exception for MiMo.
+
+| Engine | On the device | Weight formats | Stays on the CPU |
+|---|---|---|---|
+| qwen36 (Qwen3.6, Qwen3-Coder, Qwen3.8-27B) | the dense trunk | int8 rows; int4-g64 with `COLI_DENSE_BITS=4`; f32 with `COLI_DENSE_I8=0` | DeltaNet `dn_a`/`dn_b`, vision tower, routed experts |
+| qwen38 (Qwen3.8 Flash Next) | the trunk | int8 trunk rows, bf16, f32 (`Q38_NATIVE_BF16=0`) | routed FP8 experts |
+| inkling | dense and shared-expert matrices | int8 and int4-g64 (dense-int4g64 container), f32, bf16 | routed experts, embedding and audio lookups, CUDA residents; bf16 on CPUs with the AVX512-BF16 dot (see below) |
+| olmoe | attention q/k/v/o, router, lm_head | f32 | routed experts, embedding |
+| deepseek_v41 | the trunk, vision included | fp8 in 32x32 ue8m0 tiles, bf16 | routed experts |
+| deepseek_v4 | resident dense layers, head, router, compressors | fp8 in 128x128 blocks, bf16 | routed experts, the indexer's `weights_proj`, DSpark stages, the `--oracle` path |
+| mimo | trunk and vision tower; up to `MIMO_VK_EXPERTS=N` routed experts | native fp8/bf16, int8, f32 (`MIMO_DENSE_BITS`); experts as MXFP4 | router |
+| qwenimage | the DiT's matrices | int8, bf16, f32 (`COLI_IMG_BITS`) | text encoder, VAE, attention |
+
+Each engine ends a run, and each serve turn, with
+`[VK] <engine>: N matmuls on the GPU`. That count is how you tell a path that ran
+from one that only initialised.
+
+**Arithmetic.** The device reads the same weights the CPU reads and multiplies
+them by f32 activations.
+- Where the CPU's default kernel also uses f32 activations, the two differ only in
+ the order of the sums.
+- Where the CPU kernel rounds activations first, the device result instead matches
+ the CPU's f32-activation setting, so tokens can drift from the CPU default after
+ a few steps. These kernels are:
+ - qwen36's int8 dot (`COLI_DENSE_IDOT`, on by default);
+ - qwen38's int8 trunk;
+ - qwenimage's `COLI_IMG_ACT8`.
+- Two engines keep the CPU's exact arithmetic instead:
+ - deepseek_v4 rounds activations to E4M3 on the host before the call, as its CPU
+ kernel does.
+ - inkling leaves its bf16 matrices on the CPU when the build has the AVX512-BF16
+ dot (Zen 4/5, Sapphire Rapids), because that dot rounds activations to bf16.
+
+**Memory.** The host copy stays as the CPU fallback. On an integrated GPU or APU,
+which shares RAM with the CPU, the resident set is therefore held twice: size
+`RAM_GB`/caps with that in mind. The weight arena does not return freed tensors'
+memory, which is also why MiMo's expert tier never evicts.
+
+**Status.** CI checks every engine above on Lavapipe (`tests/vulkan_engines.sh`, the
+`vulkan-engines` job): each configuration gives the CPU run's tokens, and its matmul
+count is above zero. That proves correctness, not speed. None of these engines has
+been measured on a real GPU yet.
+
## Correctness
- `gcc -O3 -DVK_TEST backend_vulkan.c -o test_vk -lvulkan -lm && ./test_vk