Run frontier MoE models on hardware you already own — pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦
C
Merge dev (the OLMoE, Inkling, MiMo and GLM chains) into the Kimi K3 chain
JustVuggf4ff18dbb42c
diff --cc c/Makefileindex ac51658d,1df7ee54..bb1016d1--- a/c/Makefile+++ b/c/Makefile@@@ -721,8 -721,9 +721,11 @@@ VK_SPV += shaders/chain_norm.spv shadVK_SPV += shaders/chain_mla.spv shaders/chain_hgemv.spv shaders/chain_dsa.spv# Kimi Delta Attention and manifold-constrained hyper-connections (vk_chain.h)VK_SPV += shaders/chain_kda.spv shaders/chain_mhc.spv+# Kimi K3's attention residuals and SiTU-GLU (vk_chain.h)+VK_SPV += shaders/chain_ares.spv+ # inkling's chain ops (made on first use): the short convolution and the attention+ # with its relative-position bias+ VK_SPV += shaders/chain_sconv.spv shaders/chain_relattn.spvendif# Optional AMD XDNA2 (Ryzen AI NPU) lane, Windows. Same opt-in shape as VK=1:diff --cc c/tests/test_vk_chain.cindex 7c896d57,267276f7..7153efbf--- a/c/tests/test_vk_chain.c+++ b/c/tests/test_vk_chain.c@@@ -32,10 -34,12 +34,16 @@@* decay, beta, output norm and gate* mhc hyper_connections.h's split, collapse and write back, the mean, and the* clamped SwiGLU+ * kda k3 Kimi K3's KDA layer (kimi_k3.c's kda_forward): exp(A_log) as the engine+ * keeps it, its order of the l2 norms and the update, over two submissions+ * ares Kimi K3's attention residuals (res_mix) over 0 to 15 block snapshots, and+ * SiTU-GLU on values both sides of its two constants+ * sconv inkling's short convolution (residual inside, the ring carried across+ * calls) and its scalar multiply and divide+ * relattn inkling's attention: the relative-position bias, tau, a sliding window+ * over a ring the step wraps, the step's rows read from its own K/V, and a+ * global layer; the shared experts' weighted add (HC_APPLY over one stream)+ * inkling's hidden size: the GEMV and the norm at D = 6144, the GEMV at I = 24576** make vk-chain-check VK=1 (VK_ICD_FILENAMES=.../lvp_icd.json for Lavapipe) */#include <stdio.h>@@@ -1212,16 -1295,9 +1415,18 @@@ int main(int argc, char **argv)printf("kda done\n");test_mhc(2, 64, 3); test_mhc(4, 96, 20); test_mhc(3, 40, 1);printf("mhc done\n");+ test_kda_k3(2, 16, 4, 1.f); test_kda_k3(3, 32, 4, 1.f); test_kda_k3(2, 128, 4, 1.f); test_kda_k3(1, 64, 2, 1.f);+ test_kda_k3(2, 16, 4, 0.002f);+ printf("kda k3 done\n");+ }+ if (!vkc_ares_ready()) { fails++; printf("FAIL: the AttnRes shader did not load\n"); }+ else {+ test_ares(1, 128, 0); test_ares(3, 128, 1); test_ares(5, 300, 4); test_ares(2, 7168, 8); test_ares(4, 96, 15);+ test_situ(1.f, 1.f); test_situ(4.f, 25.f);+ printf("ares done\n");}+ if (vkc_sconv_ready() && vkc_relattn_ready()) { test_inkling(); printf("inkling done\n"); }+ else CHECK(0, "inkling's ops: chain_sconv.spv or chain_relattn.spv did not load");VkcStats st; vkc_stats(&st);printf("chain: %llu frames, %llu ops, %llu matmuls (%llu GEMM), %llu barriers\n", st.frames, st.ops, st.matmuls, st.gemms, st.barriers);vkc_shutdown();diff --cc c/tests/vulkan_chain_serve.pyindex 706448a5,11d8dc41..81725cda--- a/c/tests/vulkan_chain_serve.py+++ b/c/tests/vulkan_chain_serve.py@@@ -7,10 -7,9 +7,11 @@@ The turns exercise what moves the chaina prompt that diverges and one that starts over, a cache that grows between turns,and the prefill read-out (logprobs=k, ECHO frames). Token ids and texts must be theCPU's exactly; the logprobs printed with them may differ in their last digits (the- device sums in another order), so numbers compare within 1e-4 (CHAIN_SERVE_TOL sets- another bound, for an engine whose fixture amplifies rounding more; the OK line gives- the largest difference seen). CHAIN_SERVE_EXPECT, a regular expression the chain+ device sums in another order), so numbers compare within 1e-4 (CHAIN_SERVE_TOL in the+ environment sets another bound: a fixture whose logits run to the hundreds, as MiMo's-does, prints logprobs whose rounding is larger in absolute terms).++does, prints logprobs whose rounding is larger in absolute terms; the OK line gives the++largest difference seen). CHAIN_SERVE_EXPECT, a regular expression the chain+session's stderr must match (a lost device's rebuild, say, with COLI_VK_CHAIN_FAULT).usage: vulkan_chain_serve.py <engine> <snapshot> [KEY=VALUE ...] (extra environment)COLI_VK_CHAIN in the caller's environment picks the chain's mode (default 1; 2 runsdiff --cc c/tests/vulkan_engines.shindex f4c8b46b,f945b8c0..c5752324--- a/c/tests/vulkan_engines.sh+++ b/c/tests/vulkan_engines.sh@@@ -7,8 -7,9 +7,10 @@@# bash tests/vulkan_engines.sh glm | glm-sanitize # GLM-5.2 (colibri) and GLM-5.3 Flash (glm53)# bash tests/vulkan_engines.sh shader # the qmatmul formats, the expert batch and the tier, no engine# bash tests/vulkan_engines.sh qwen-chain | qwen-chain-sanitize # the dense chain (COLI_VK_CHAIN=1)+ # bash tests/vulkan_engines.sh mimo-chain | mimo-chain-sanitize # MiMo-V2.6's dense chain+ # bash tests/vulkan_engines.sh inkling-olmoe-chain | inkling-olmoe-chain-sanitize# bash tests/vulkan_engines.sh glm-chain | glm-chain-sanitize # the same for colibri and glm53+# bash tests/vulkan_engines.sh kimi-chain | kimi-chain-sanitize # the same for Kimi K3## Needs libvulkan-dev, glslc and mesa-vulkan-drivers, plus the Python packages of# the family's tiny fixtures (see the vulkan-engines job in .github/workflows/ci.yml).@@@ -1729,9 -1900,9 +2085,11 @@@ case "${1:-}" iglm-sanitize) family_glm_sanitize ;;qwen-chain) family_qwen_chain ;;qwen-chain-sanitize) family_qwen_chain_sanitize ;;+ inkling-olmoe-chain) family_inkling_olmoe_chain ;;+ inkling-olmoe-chain-sanitize) family_inkling_olmoe_chain_sanitize ;;glm-chain) family_glm_chain ;;glm-chain-sanitize) family_glm_chain_sanitize ;;- *) echo "usage: $0 shader|qwen|qwen-sanitize|inkling-olmoe|inkling-olmoe-sanitize|mimo-qwenimage|deepseek|deepseek-sanitize|kimi|kimi-mimo-sanitize|glm|glm-sanitize|qwen-chain|qwen-chain-sanitize|mimo-chain|mimo-chain-sanitize|inkling-olmoe-chain|inkling-olmoe-chain-sanitize|glm-chain|glm-chain-sanitize" >&2; exit 2 ;;+ kimi-chain) family_kimi_chain ;;+ kimi-chain-sanitize) family_kimi_chain_sanitize ;;- *) echo "usage: $0 shader|qwen|qwen-sanitize|inkling-olmoe|inkling-olmoe-sanitize|mimo-qwenimage|deepseek|deepseek-sanitize|kimi|kimi-mimo-sanitize|glm|glm-sanitize|qwen-chain|qwen-chain-sanitize|glm-chain|glm-chain-sanitize|kimi-chain|kimi-chain-sanitize" >&2; exit 2 ;;++ *) echo "usage: $0 shader|qwen|qwen-sanitize|inkling-olmoe|inkling-olmoe-sanitize|mimo-qwenimage|deepseek|deepseek-sanitize|kimi|kimi-mimo-sanitize|glm|glm-sanitize|qwen-chain|qwen-chain-sanitize|mimo-chain|mimo-chain-sanitize|inkling-olmoe-chain|inkling-olmoe-chain-sanitize|glm-chain|glm-chain-sanitize|kimi-chain|kimi-chain-sanitize" >&2; exit 2 ;;esacdiff --cc c/vk_chain.hindex 018d8e52,9509dacf..d9e9a71f--- a/c/vk_chain.h+++ b/c/vk_chain.h@@@ -294,20 -295,22 +304,35 @@@ int vkc_mhc_ready(void)typedef struct { int S, H, D, iters, x_off, x_row, m_off, m_row, hp_off, hp_row, y_off, y_row, prm_off, n;float eps, hc_eps, lim; } VkcMhc;int vkc_mhc(int mode, VkcBuf *x, VkcBuf *m, VkcBuf *hp, VkcBuf *prm, VkcBuf *y, const VkcMhc *p);+ /* Inkling's ops, their pipelines made on first use (an engine checks *_ready at setup:+ * a build without the shader keeps every other op, and that engine's chain off).+ * chain_sconv.comp (mode 0: the depthwise causal short convolution, residual inside, in+ * place, its ring carried; mode 1: x *= fc over n floats; mode 2: x /= fc) */+ typedef struct { int mode, S, C, CK, x_off, x_row, w_off, ring_off, n; float fc; } VkcSconv;+ int vkc_sconv_ready(void);+ int vkc_sconv(VkcBuf *x, VkcBuf *w, VkcBuf *ring, const VkcSconv *p);+ /* chain_relattn.comp: attention with a relative-position bias bank, a per-row scale+ * tau and a sliding window over a ring cache, the step's own rows read from kvs */+ typedef struct { int S, H, KVH, hd, pos_base, cap, window, ext, d_rel;+ int q_off, q_row, o_off, o_row, k_off, v_off, ks_off, vs_off, kv_row;+ int r_off, r_row, relp_off, tau_off; float scale; } VkcRelAttn;+ int vkc_relattn_ready(void);+ int vkc_relattn(VkcBuf *q, VkcBuf *kc, VkcBuf *vc, VkcBuf *o, VkcBuf *kvs, VkcBuf *r, VkcBuf *relp, VkcBuf *tau,+ const VkcRelAttn *p);+/* ---- attention residuals and SiTU-GLU (chain_ares.comp) ----------------------------+ * Kimi K3's residual stream (AttnRes) and its activation, for S rows:+ * vkc_ares_mix kimi_k3.c's res_mix: row s mixes the nb block snapshots ([nb][D] at+ * b_off + s*b_row) and the running prefix (x_off + s*x_row) by the+ * softmax of (v . w) / sqrt(mean(v^2) + eps), w at w_off in prm, into+ * y_off + s*y_row; nb <= 15, y apart from x and blk+ * vkc_situ y[i] = b1*tanh(g/b1)*sigmoid(g)*b2*tanh(u/b2), i < n, in the CPU's order */+int vkc_ares_ready(void);+typedef struct { int S, D, nb, x_off, x_row, b_off, b_row, w_off, y_off, y_row; float eps; } VkcAres;+int vkc_ares_mix(VkcBuf *x, VkcBuf *blk, VkcBuf *prm, VkcBuf *y, const VkcAres *p);+typedef struct { int n, g_off, u_off, y_off; float b1, b2; } VkcSitu;+int vkc_situ(VkcBuf *g, VkcBuf *u, VkcBuf *y, const VkcSitu *p);+/* counters, for the engines' [VK] lines */typedef struct {unsigned long long frames, waits, ops, matmuls, gemms, barriers, bytes_up, bytes_down;diff --cc docs/ENVIRONMENT.mdindex 34689be3,9b9865db..7dd158cf--- a/docs/ENVIRONMENT.md+++ b/docs/ENVIRONMENT.md@@@ -245,7 -245,7 +245,7 @@@ With `COLI_VULKAN=1` every MoE engine (| `COLI_VK_TIER_GEMM_ROWS` | `16` | Rows from which an expert of a step takes the tiled GEMM (prefill) instead of the per-row GEMV; `0` never. Below it a row's bits do not depend on how many rows share the dispatch. || `COLI_VK_TIER_QUEUE` | a second queue | `0`: the tier's batches share the main queue with the dense matmuls (they then serialize). Unset: a second queue of the main family, else a compute-only family's (RADV), else shared (Lavapipe). || `COLI_VK_DENSE` | on; off on a device sharing the CPU's RAM while the tier is on | Where the dense (non-expert) matrices of qwen36, qwen38, inkling, olmoe, kimi_k3 (its shared experts), mimo, deepseek_v41, deepseek_v4, glm53 and qwenimage run. `0`: on the CPU (with the tier on, the device takes the routed experts only); `1`: on the device whatever the device. Unset: on the device, except on an integrated GPU or a CPU device (Lavapipe) while the engine runs the expert tier (every one of them but qwenimage, which has no routed experts), where they stay on the CPU: there the dense matmuls cost more than the tier gains (measured on a Radeon 780M). The `[VK] <engine>: device ready, dense matrices on the ...` line says which and why. The GLM engine reads it through the same rule with its own default, off (see above). |- | `COLI_VK_CHAIN` | on for a discrete GPU; qwen36 on, qwen38, colibri, glm53 and kimi_k3 off on an integrated GPU with the tier; off on a CPU device | qwen36, qwen38, colibri (GLM-5.2), glm53 (GLM-5.3 Flash) and kimi_k3: every layer's dense chain recorded into one submission per layer, the residual stream and the recurrent and KV state on the device (see [vulkan.md](vulkan.md#the-dense-chain-vk_chainc)). `0` off, `1` on, `2` on for prompts only (decode on the CPU). The `[VK] <engine>: dense chain ...` line says which and why. |-| `COLI_VK_CHAIN` | on for a discrete GPU; qwen36 and olmoe on, qwen38 off, and mimo, inkling, colibri and glm53 off (not measured) on an integrated GPU with the tier; off on a CPU device | qwen36, qwen38, olmoe, inkling, mimo, colibri (GLM-5.2) and glm53 (GLM-5.3 Flash): every layer's dense chain recorded into one submission per layer, the residual stream and the recurrent and KV state on the device (see [vulkan.md](vulkan.md#the-dense-chain-vk_chainc)). `0` off, `1` on, `2` on for prompts only (decode on the CPU). The `[VK] <engine>: dense chain ...` line says which and why. |++| `COLI_VK_CHAIN` | on for a discrete GPU; qwen36 and olmoe on, qwen38 off, and mimo, inkling, colibri, glm53 and kimi_k3 off (not measured) on an integrated GPU with the tier; off on a CPU device | qwen36, qwen38, olmoe, inkling, mimo, colibri (GLM-5.2), glm53 (GLM-5.3 Flash) and kimi_k3: every layer's dense chain recorded into one submission per layer, the residual stream and the recurrent and KV state on the device (see [vulkan.md](vulkan.md#the-dense-chain-vk_chainc)). `0` off, `1` on, `2` on for prompts only (decode on the CPU). The `[VK] <engine>: dense chain ...` line says which and why. || `COLI_VK_CHAIN_ROWS` | `512` | Prompt rows per chunk of the chain's prefill. || `COLI_VK_CHAIN_GEMV` | on | `0`: the chain's decode matrices take `qmatmul.comp`'s GEMV instead of `chain_gemv.comp`'s. || `COLI_VK_CHAIN_SPIN_US` | `2000` | How long a wait on a chain frame polls before blocking. |diff --cc docs/vulkan.mdindex 883a4a9e,55ac4659..3d370e2a--- a/docs/vulkan.md+++ b/docs/vulkan.md@@@ -673,7 -673,8 +673,8 @@@ command buffer instead, and keeps the rto the next. qwen36 (Qwen3.6, Qwen3-Coder, Qwen3.8-27B) and qwen38 (Qwen3.8 FlashNext) run it: by default on a discrete GPU, and for qwen36 on an integrated one withthe expert tier (see [the default](#the-chain-on-a-radeon-780m)); `COLI_VK_CHAIN=1`- anywhere. colibri (GLM-5.2), glm53 (GLM-5.3 Flash) and kimi_k3 run it too, with the MLA,+ anywhere. olmoe and inkling run it as well ([OLMoE and Inkling](#olmoe-and-inkling)).-colibri (GLM-5.2) and glm53 (GLM-5.3 Flash) run it too, with the MLA,++colibri (GLM-5.2), glm53 (GLM-5.3 Flash) and kimi_k3 run it too, with the MLA,KDA and hyper-connection ops ([below](#glm-52-and-glm-53-flash-on-the-chain)).**What runs where, per layer** (S rows: one at decode, a prompt chunk at prefill):@@@ -741,7 -742,7 +742,7 @@@ f32 throughout, as the CPU's f32 path| Variable | Default | Effect ||---|---|---|- | `COLI_VK_CHAIN` | on for a discrete GPU; on an integrated GPU with the expert tier, what the engine measured (qwen36 on, qwen38 off; colibri, glm53 and kimi_k3 off: not measured); off on a CPU device | `1`: every layer's dense chain on the device; `2`: prompts only (forwards of more than two rows; decode and MTP verifies on the per-matrix path, the state moving between the two); `0`: the per-matrix path. The `[VK] <engine>: dense chain ...` line says which and why. |-| `COLI_VK_CHAIN` | on for a discrete GPU; on an integrated GPU with the expert tier, what the engine measured (qwen36 and olmoe on, qwen38 off; mimo, inkling, colibri and glm53 off: not measured); off on a CPU device | `1`: every layer's dense chain on the device; `2`: prompts only (forwards of more than two rows; decode and MTP verifies on the per-matrix path, the state moving between the two); `0`: the per-matrix path. The `[VK] <engine>: dense chain ...` line says which and why. |++| `COLI_VK_CHAIN` | on for a discrete GPU; on an integrated GPU with the expert tier, what the engine measured (qwen36 and olmoe on, qwen38 off; mimo, inkling, colibri, glm53 and kimi_k3 off: not measured); off on a CPU device | `1`: every layer's dense chain on the device; `2`: prompts only (forwards of more than two rows; decode and MTP verifies on the per-matrix path, the state moving between the two); `0`: the per-matrix path. The `[VK] <engine>: dense chain ...` line says which and why. || `COLI_VK_CHAIN_ROWS` | `512` | Prompt rows per chunk: a longer prompt runs every layer chunk by chunk (the device's scratch is sized for one chunk). || `COLI_VK_CHAIN_GEMV` | on | `0`: the decode matrices take `qmatmul.comp`'s GEMV instead of `chain_gemv.comp`'s. || `COLI_VK_CHAIN_SPIN_US` | `2000` | How long a wait on a chain frame polls the fence before blocking. |@@@ -1001,10 -1108,10 +1192,10 @@@ What each remaining architecture needs|---|---|---|| deepseek_v41, kimi_k3 (MLA layers) | MLA: q_a/kv_a, the latent norms, q_b, RoPE on the rope dims, a latent + rope cache | the [MLA ops](#multi-head-latent-attention-on-the-chain-vkc_mla) take any geometry (q_lora or none, NoPE, the scale and cos/sin table from the engine, a gate on the values for Kimi K3); colibri and glm53 run them ([above](#glm-52-and-glm-53-flash-on-the-chain)). DeepSeek V4.1's attention has sinks and its own sparse selection: those are not in the ops yet || deepseek_v4 | MLA with compressed (CSA) and hierarchical (HCA) KV, mHC | the manifold hyper-connections are `vkc_mhc` (GLM-5.3's, the same arithmetic) and the clamped SwiGLU too; the compressors' rolling windows are rings like the conv's, snapshotted the same way; the CPU rounds activations to E4M3 before its fp8 matmuls, so the chain needs that rounding as an element-wise op to keep the same arithmetic |-| kimi_k3 (KDA layers) | Kimi Delta Attention: a gated delta rule whose decay is a vector over the key channels | `vkc_kda_conv` and `vkc_kda_rec` (GLM-5.3's KDA, `delta_attention.h`'s step); Kimi K3 builds its decay and output gate its own way (a full g_proj, not a low-rank one), which the recurrence's prologue and epilogue would take as options |+| kimi_k3 (KDA layers) | Kimi Delta Attention: a gated delta rule whose decay is a vector over the key channels | on the chain ([above](#kimi-k3-on-the-chain)): `vkc_kda_conv` and `vkc_kda_rec_flags` with Kimi K3's options; its full-rank output gate and low-rank decay are matmuls before the op |- | mimo | sliding-window attention (and full layers) | `chain_attn.comp` with a window (positions from `max(0, pos - W + 1)`: one more push constant), the cache optionally a ring of W rows (the host mirror then indexes `t % W`) |- | inkling | grouped attention, MoE with a shared expert | qwen36's attention and combine as they are; its per-position heads are matmuls of the residual rows |- | olmoe | attention with q/k norm, MoE without a shared expert | qwen36's Qwen3-Coder geometry (all attention, no gate, no shared expert) is the same chain |+ | mimo | sliding-window attention (and full layers) | done: [MiMo-V2.6 on the chain](#mimo-v26-on-the-chain) (`chain_attn.comp` with a window, a ring of W rows, V's own head dim and a sink) |+ | inkling | grouped attention with a relative-position bias, a sliding window and short convolutions; MoE with shared experts | in the chain ([OLMoE and Inkling](#olmoe-and-inkling)): two shaders of its own, `chain_relattn.comp` and `chain_sconv.comp`; the shared experts join through `HC_APPLY` |+ | olmoe | attention with q/k norm, MoE without a shared expert | in the chain ([OLMoE and Inkling](#olmoe-and-inkling)) with qwen36's ops as they are |### Multi-head latent attention on the chain (`vkc_mla`)@@@ -1140,13 -1257,7 +1341,13 @@@ in shortselection active, n-gram and MTP drafts accepted and rejected, glm53's image, KDA stateand swiglu_limit 0, a device lost (glm53's KDA state rebuilt), serve sessions with pinsand two KV slots, and glm53's pin-branch harness; `glm-chain-sanitize` runs them under- ASan and UBSan; `qwen-chain-sanitize`- runs the chain under ASan and UBSan. `kimi-chain` does it for Kimi K3: Moonshot's+ ASan and UBSan.++ `kimi-chain` does it for Kimi K3: Moonshot's+ oracle with the chain on, every dense format, prefill a token at a time and in chunks,+ the tier's eviction, prompts only, a device lost in a prompt, inside a chunked forward+ and mid-decode (the KDA state rebuilt), serve sessions with prefix reuse and+ recurrent-state checkpoints in RAM and on disk; `kimi-chain-sanitize` under ASan and+ UBSan.- int4 weights decode as offset-binary (nibble−8), byte-identical layout tothe CPU path — no repacking.- Khronos validation layers: the backend never enables them, so the loader