Skip to content

Run frontier MoE models on hardware you already own — pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦

C

Merge dev (the OLMoE, Inkling, MiMo and GLM chains) into the Kimi K3 chain

JustVuggf4ff18dbb42c
diff --cc c/Makefile
index ac51658d,1df7ee54..bb1016d1
--- a/c/Makefile
+++ b/c/Makefile
@@@ -721,8 -721,9 +721,11 @@@ VK_SPV += shaders/chain_norm.spv shad
VK_SPV += shaders/chain_mla.spv shaders/chain_hgemv.spv shaders/chain_dsa.spv
# Kimi Delta Attention and manifold-constrained hyper-connections (vk_chain.h)
VK_SPV += shaders/chain_kda.spv shaders/chain_mhc.spv
+# Kimi K3's attention residuals and SiTU-GLU (vk_chain.h)
+VK_SPV += shaders/chain_ares.spv
+ # inkling's chain ops (made on first use): the short convolution and the attention
+ # with its relative-position bias
+ VK_SPV += shaders/chain_sconv.spv shaders/chain_relattn.spv
endif
# Optional AMD XDNA2 (Ryzen AI NPU) lane, Windows. Same opt-in shape as VK=1:
diff --cc c/tests/test_vk_chain.c
index 7c896d57,267276f7..7153efbf
--- a/c/tests/test_vk_chain.c
+++ b/c/tests/test_vk_chain.c
@@@ -32,10 -34,12 +34,16 @@@
* decay, beta, output norm and gate
* mhc hyper_connections.h's split, collapse and write back, the mean, and the
* clamped SwiGLU
+ * kda k3 Kimi K3's KDA layer (kimi_k3.c's kda_forward): exp(A_log) as the engine
+ * keeps it, its order of the l2 norms and the update, over two submissions
+ * ares Kimi K3's attention residuals (res_mix) over 0 to 15 block snapshots, and
+ * SiTU-GLU on values both sides of its two constants
+ * sconv inkling's short convolution (residual inside, the ring carried across
+ * calls) and its scalar multiply and divide
+ * relattn inkling's attention: the relative-position bias, tau, a sliding window
+ * over a ring the step wraps, the step's rows read from its own K/V, and a
+ * global layer; the shared experts' weighted add (HC_APPLY over one stream)
+ * inkling's hidden size: the GEMV and the norm at D = 6144, the GEMV at I = 24576
*
* make vk-chain-check VK=1 (VK_ICD_FILENAMES=.../lvp_icd.json for Lavapipe) */
#include <stdio.h>
@@@ -1212,16 -1295,9 +1415,18 @@@ int main(int argc, char **argv)
printf("kda done\n");
test_mhc(2, 64, 3); test_mhc(4, 96, 20); test_mhc(3, 40, 1);
printf("mhc done\n");
+ test_kda_k3(2, 16, 4, 1.f); test_kda_k3(3, 32, 4, 1.f); test_kda_k3(2, 128, 4, 1.f); test_kda_k3(1, 64, 2, 1.f);
+ test_kda_k3(2, 16, 4, 0.002f);
+ printf("kda k3 done\n");
+ }
+ if (!vkc_ares_ready()) { fails++; printf("FAIL: the AttnRes shader did not load\n"); }
+ else {
+ test_ares(1, 128, 0); test_ares(3, 128, 1); test_ares(5, 300, 4); test_ares(2, 7168, 8); test_ares(4, 96, 15);
+ test_situ(1.f, 1.f); test_situ(4.f, 25.f);
+ printf("ares done\n");
}
+ if (vkc_sconv_ready() && vkc_relattn_ready()) { test_inkling(); printf("inkling done\n"); }
+ else CHECK(0, "inkling's ops: chain_sconv.spv or chain_relattn.spv did not load");
VkcStats st; vkc_stats(&st);
printf("chain: %llu frames, %llu ops, %llu matmuls (%llu GEMM), %llu barriers\n", st.frames, st.ops, st.matmuls, st.gemms, st.barriers);
vkc_shutdown();
diff --cc c/tests/vulkan_chain_serve.py
index 706448a5,11d8dc41..81725cda
--- a/c/tests/vulkan_chain_serve.py
+++ b/c/tests/vulkan_chain_serve.py
@@@ -7,10 -7,9 +7,11 @@@ The turns exercise what moves the chain
a prompt that diverges and one that starts over, a cache that grows between turns,
and the prefill read-out (logprobs=k, ECHO frames). Token ids and texts must be the
CPU's exactly; the logprobs printed with them may differ in their last digits (the
- device sums in another order), so numbers compare within 1e-4 (CHAIN_SERVE_TOL sets
- another bound, for an engine whose fixture amplifies rounding more; the OK line gives
- the largest difference seen). CHAIN_SERVE_EXPECT, a regular expression the chain
+ device sums in another order), so numbers compare within 1e-4 (CHAIN_SERVE_TOL in the
+ environment sets another bound: a fixture whose logits run to the hundreds, as MiMo's
-does, prints logprobs whose rounding is larger in absolute terms).
++does, prints logprobs whose rounding is larger in absolute terms; the OK line gives the
++largest difference seen). CHAIN_SERVE_EXPECT, a regular expression the chain
+session's stderr must match (a lost device's rebuild, say, with COLI_VK_CHAIN_FAULT).
usage: vulkan_chain_serve.py <engine> <snapshot> [KEY=VALUE ...] (extra environment)
COLI_VK_CHAIN in the caller's environment picks the chain's mode (default 1; 2 runs
diff --cc c/tests/vulkan_engines.sh
index f4c8b46b,f945b8c0..c5752324
--- a/c/tests/vulkan_engines.sh
+++ b/c/tests/vulkan_engines.sh
@@@ -7,8 -7,9 +7,10 @@@
# bash tests/vulkan_engines.sh glm | glm-sanitize # GLM-5.2 (colibri) and GLM-5.3 Flash (glm53)
# bash tests/vulkan_engines.sh shader # the qmatmul formats, the expert batch and the tier, no engine
# bash tests/vulkan_engines.sh qwen-chain | qwen-chain-sanitize # the dense chain (COLI_VK_CHAIN=1)
+ # bash tests/vulkan_engines.sh mimo-chain | mimo-chain-sanitize # MiMo-V2.6's dense chain
+ # bash tests/vulkan_engines.sh inkling-olmoe-chain | inkling-olmoe-chain-sanitize
# bash tests/vulkan_engines.sh glm-chain | glm-chain-sanitize # the same for colibri and glm53
+# bash tests/vulkan_engines.sh kimi-chain | kimi-chain-sanitize # the same for Kimi K3
#
# Needs libvulkan-dev, glslc and mesa-vulkan-drivers, plus the Python packages of
# the family's tiny fixtures (see the vulkan-engines job in .github/workflows/ci.yml).
@@@ -1729,9 -1900,9 +2085,11 @@@ case "${1:-}" i
glm-sanitize) family_glm_sanitize ;;
qwen-chain) family_qwen_chain ;;
qwen-chain-sanitize) family_qwen_chain_sanitize ;;
+ inkling-olmoe-chain) family_inkling_olmoe_chain ;;
+ inkling-olmoe-chain-sanitize) family_inkling_olmoe_chain_sanitize ;;
glm-chain) family_glm_chain ;;
glm-chain-sanitize) family_glm_chain_sanitize ;;
- *) echo "usage: $0 shader|qwen|qwen-sanitize|inkling-olmoe|inkling-olmoe-sanitize|mimo-qwenimage|deepseek|deepseek-sanitize|kimi|kimi-mimo-sanitize|glm|glm-sanitize|qwen-chain|qwen-chain-sanitize|mimo-chain|mimo-chain-sanitize|inkling-olmoe-chain|inkling-olmoe-chain-sanitize|glm-chain|glm-chain-sanitize" >&2; exit 2 ;;
+ kimi-chain) family_kimi_chain ;;
+ kimi-chain-sanitize) family_kimi_chain_sanitize ;;
- *) echo "usage: $0 shader|qwen|qwen-sanitize|inkling-olmoe|inkling-olmoe-sanitize|mimo-qwenimage|deepseek|deepseek-sanitize|kimi|kimi-mimo-sanitize|glm|glm-sanitize|qwen-chain|qwen-chain-sanitize|glm-chain|glm-chain-sanitize|kimi-chain|kimi-chain-sanitize" >&2; exit 2 ;;
++ *) echo "usage: $0 shader|qwen|qwen-sanitize|inkling-olmoe|inkling-olmoe-sanitize|mimo-qwenimage|deepseek|deepseek-sanitize|kimi|kimi-mimo-sanitize|glm|glm-sanitize|qwen-chain|qwen-chain-sanitize|mimo-chain|mimo-chain-sanitize|inkling-olmoe-chain|inkling-olmoe-chain-sanitize|glm-chain|glm-chain-sanitize|kimi-chain|kimi-chain-sanitize" >&2; exit 2 ;;
esac
diff --cc c/vk_chain.h
index 018d8e52,9509dacf..d9e9a71f
--- a/c/vk_chain.h
+++ b/c/vk_chain.h
@@@ -294,20 -295,22 +304,35 @@@ int vkc_mhc_ready(void)
typedef struct { int S, H, D, iters, x_off, x_row, m_off, m_row, hp_off, hp_row, y_off, y_row, prm_off, n;
float eps, hc_eps, lim; } VkcMhc;
int vkc_mhc(int mode, VkcBuf *x, VkcBuf *m, VkcBuf *hp, VkcBuf *prm, VkcBuf *y, const VkcMhc *p);
+ /* Inkling's ops, their pipelines made on first use (an engine checks *_ready at setup:
+ * a build without the shader keeps every other op, and that engine's chain off).
+ * chain_sconv.comp (mode 0: the depthwise causal short convolution, residual inside, in
+ * place, its ring carried; mode 1: x *= fc over n floats; mode 2: x /= fc) */
+ typedef struct { int mode, S, C, CK, x_off, x_row, w_off, ring_off, n; float fc; } VkcSconv;
+ int vkc_sconv_ready(void);
+ int vkc_sconv(VkcBuf *x, VkcBuf *w, VkcBuf *ring, const VkcSconv *p);
+ /* chain_relattn.comp: attention with a relative-position bias bank, a per-row scale
+ * tau and a sliding window over a ring cache, the step's own rows read from kvs */
+ typedef struct { int S, H, KVH, hd, pos_base, cap, window, ext, d_rel;
+ int q_off, q_row, o_off, o_row, k_off, v_off, ks_off, vs_off, kv_row;
+ int r_off, r_row, relp_off, tau_off; float scale; } VkcRelAttn;
+ int vkc_relattn_ready(void);
+ int vkc_relattn(VkcBuf *q, VkcBuf *kc, VkcBuf *vc, VkcBuf *o, VkcBuf *kvs, VkcBuf *r, VkcBuf *relp, VkcBuf *tau,
+ const VkcRelAttn *p);
+/* ---- attention residuals and SiTU-GLU (chain_ares.comp) ----------------------------
+ * Kimi K3's residual stream (AttnRes) and its activation, for S rows:
+ * vkc_ares_mix kimi_k3.c's res_mix: row s mixes the nb block snapshots ([nb][D] at
+ * b_off + s*b_row) and the running prefix (x_off + s*x_row) by the
+ * softmax of (v . w) / sqrt(mean(v^2) + eps), w at w_off in prm, into
+ * y_off + s*y_row; nb <= 15, y apart from x and blk
+ * vkc_situ y[i] = b1*tanh(g/b1)*sigmoid(g)*b2*tanh(u/b2), i < n, in the CPU's order */
+int vkc_ares_ready(void);
+typedef struct { int S, D, nb, x_off, x_row, b_off, b_row, w_off, y_off, y_row; float eps; } VkcAres;
+int vkc_ares_mix(VkcBuf *x, VkcBuf *blk, VkcBuf *prm, VkcBuf *y, const VkcAres *p);
+typedef struct { int n, g_off, u_off, y_off; float b1, b2; } VkcSitu;
+int vkc_situ(VkcBuf *g, VkcBuf *u, VkcBuf *y, const VkcSitu *p);
+
/* counters, for the engines' [VK] lines */
typedef struct {
unsigned long long frames, waits, ops, matmuls, gemms, barriers, bytes_up, bytes_down;
diff --cc docs/ENVIRONMENT.md
index 34689be3,9b9865db..7dd158cf
--- a/docs/ENVIRONMENT.md
+++ b/docs/ENVIRONMENT.md
@@@ -245,7 -245,7 +245,7 @@@ With `COLI_VULKAN=1` every MoE engine (
| `COLI_VK_TIER_GEMM_ROWS` | `16` | Rows from which an expert of a step takes the tiled GEMM (prefill) instead of the per-row GEMV; `0` never. Below it a row's bits do not depend on how many rows share the dispatch. |
| `COLI_VK_TIER_QUEUE` | a second queue | `0`: the tier's batches share the main queue with the dense matmuls (they then serialize). Unset: a second queue of the main family, else a compute-only family's (RADV), else shared (Lavapipe). |
| `COLI_VK_DENSE` | on; off on a device sharing the CPU's RAM while the tier is on | Where the dense (non-expert) matrices of qwen36, qwen38, inkling, olmoe, kimi_k3 (its shared experts), mimo, deepseek_v41, deepseek_v4, glm53 and qwenimage run. `0`: on the CPU (with the tier on, the device takes the routed experts only); `1`: on the device whatever the device. Unset: on the device, except on an integrated GPU or a CPU device (Lavapipe) while the engine runs the expert tier (every one of them but qwenimage, which has no routed experts), where they stay on the CPU: there the dense matmuls cost more than the tier gains (measured on a Radeon 780M). The `[VK] <engine>: device ready, dense matrices on the ...` line says which and why. The GLM engine reads it through the same rule with its own default, off (see above). |
- | `COLI_VK_CHAIN` | on for a discrete GPU; qwen36 on, qwen38, colibri, glm53 and kimi_k3 off on an integrated GPU with the tier; off on a CPU device | qwen36, qwen38, colibri (GLM-5.2), glm53 (GLM-5.3 Flash) and kimi_k3: every layer's dense chain recorded into one submission per layer, the residual stream and the recurrent and KV state on the device (see [vulkan.md](vulkan.md#the-dense-chain-vk_chainc)). `0` off, `1` on, `2` on for prompts only (decode on the CPU). The `[VK] <engine>: dense chain ...` line says which and why. |
-| `COLI_VK_CHAIN` | on for a discrete GPU; qwen36 and olmoe on, qwen38 off, and mimo, inkling, colibri and glm53 off (not measured) on an integrated GPU with the tier; off on a CPU device | qwen36, qwen38, olmoe, inkling, mimo, colibri (GLM-5.2) and glm53 (GLM-5.3 Flash): every layer's dense chain recorded into one submission per layer, the residual stream and the recurrent and KV state on the device (see [vulkan.md](vulkan.md#the-dense-chain-vk_chainc)). `0` off, `1` on, `2` on for prompts only (decode on the CPU). The `[VK] <engine>: dense chain ...` line says which and why. |
++| `COLI_VK_CHAIN` | on for a discrete GPU; qwen36 and olmoe on, qwen38 off, and mimo, inkling, colibri, glm53 and kimi_k3 off (not measured) on an integrated GPU with the tier; off on a CPU device | qwen36, qwen38, olmoe, inkling, mimo, colibri (GLM-5.2), glm53 (GLM-5.3 Flash) and kimi_k3: every layer's dense chain recorded into one submission per layer, the residual stream and the recurrent and KV state on the device (see [vulkan.md](vulkan.md#the-dense-chain-vk_chainc)). `0` off, `1` on, `2` on for prompts only (decode on the CPU). The `[VK] <engine>: dense chain ...` line says which and why. |
| `COLI_VK_CHAIN_ROWS` | `512` | Prompt rows per chunk of the chain's prefill. |
| `COLI_VK_CHAIN_GEMV` | on | `0`: the chain's decode matrices take `qmatmul.comp`'s GEMV instead of `chain_gemv.comp`'s. |
| `COLI_VK_CHAIN_SPIN_US` | `2000` | How long a wait on a chain frame polls before blocking. |
diff --cc docs/vulkan.md
index 883a4a9e,55ac4659..3d370e2a
--- a/docs/vulkan.md
+++ b/docs/vulkan.md
@@@ -673,7 -673,8 +673,8 @@@ command buffer instead, and keeps the r
to the next. qwen36 (Qwen3.6, Qwen3-Coder, Qwen3.8-27B) and qwen38 (Qwen3.8 Flash
Next) run it: by default on a discrete GPU, and for qwen36 on an integrated one with
the expert tier (see [the default](#the-chain-on-a-radeon-780m)); `COLI_VK_CHAIN=1`
- anywhere. colibri (GLM-5.2), glm53 (GLM-5.3 Flash) and kimi_k3 run it too, with the MLA,
+ anywhere. olmoe and inkling run it as well ([OLMoE and Inkling](#olmoe-and-inkling)).
-colibri (GLM-5.2) and glm53 (GLM-5.3 Flash) run it too, with the MLA,
++colibri (GLM-5.2), glm53 (GLM-5.3 Flash) and kimi_k3 run it too, with the MLA,
KDA and hyper-connection ops ([below](#glm-52-and-glm-53-flash-on-the-chain)).
**What runs where, per layer** (S rows: one at decode, a prompt chunk at prefill):
@@@ -741,7 -742,7 +742,7 @@@ f32 throughout, as the CPU's f32 path
| Variable | Default | Effect |
|---|---|---|
- | `COLI_VK_CHAIN` | on for a discrete GPU; on an integrated GPU with the expert tier, what the engine measured (qwen36 on, qwen38 off; colibri, glm53 and kimi_k3 off: not measured); off on a CPU device | `1`: every layer's dense chain on the device; `2`: prompts only (forwards of more than two rows; decode and MTP verifies on the per-matrix path, the state moving between the two); `0`: the per-matrix path. The `[VK] <engine>: dense chain ...` line says which and why. |
-| `COLI_VK_CHAIN` | on for a discrete GPU; on an integrated GPU with the expert tier, what the engine measured (qwen36 and olmoe on, qwen38 off; mimo, inkling, colibri and glm53 off: not measured); off on a CPU device | `1`: every layer's dense chain on the device; `2`: prompts only (forwards of more than two rows; decode and MTP verifies on the per-matrix path, the state moving between the two); `0`: the per-matrix path. The `[VK] <engine>: dense chain ...` line says which and why. |
++| `COLI_VK_CHAIN` | on for a discrete GPU; on an integrated GPU with the expert tier, what the engine measured (qwen36 and olmoe on, qwen38 off; mimo, inkling, colibri, glm53 and kimi_k3 off: not measured); off on a CPU device | `1`: every layer's dense chain on the device; `2`: prompts only (forwards of more than two rows; decode and MTP verifies on the per-matrix path, the state moving between the two); `0`: the per-matrix path. The `[VK] <engine>: dense chain ...` line says which and why. |
| `COLI_VK_CHAIN_ROWS` | `512` | Prompt rows per chunk: a longer prompt runs every layer chunk by chunk (the device's scratch is sized for one chunk). |
| `COLI_VK_CHAIN_GEMV` | on | `0`: the decode matrices take `qmatmul.comp`'s GEMV instead of `chain_gemv.comp`'s. |
| `COLI_VK_CHAIN_SPIN_US` | `2000` | How long a wait on a chain frame polls the fence before blocking. |
@@@ -1001,10 -1108,10 +1192,10 @@@ What each remaining architecture needs
|---|---|---|
| deepseek_v41, kimi_k3 (MLA layers) | MLA: q_a/kv_a, the latent norms, q_b, RoPE on the rope dims, a latent + rope cache | the [MLA ops](#multi-head-latent-attention-on-the-chain-vkc_mla) take any geometry (q_lora or none, NoPE, the scale and cos/sin table from the engine, a gate on the values for Kimi K3); colibri and glm53 run them ([above](#glm-52-and-glm-53-flash-on-the-chain)). DeepSeek V4.1's attention has sinks and its own sparse selection: those are not in the ops yet |
| deepseek_v4 | MLA with compressed (CSA) and hierarchical (HCA) KV, mHC | the manifold hyper-connections are `vkc_mhc` (GLM-5.3's, the same arithmetic) and the clamped SwiGLU too; the compressors' rolling windows are rings like the conv's, snapshotted the same way; the CPU rounds activations to E4M3 before its fp8 matmuls, so the chain needs that rounding as an element-wise op to keep the same arithmetic |
-| kimi_k3 (KDA layers) | Kimi Delta Attention: a gated delta rule whose decay is a vector over the key channels | `vkc_kda_conv` and `vkc_kda_rec` (GLM-5.3's KDA, `delta_attention.h`'s step); Kimi K3 builds its decay and output gate its own way (a full g_proj, not a low-rank one), which the recurrence's prologue and epilogue would take as options |
+| kimi_k3 (KDA layers) | Kimi Delta Attention: a gated delta rule whose decay is a vector over the key channels | on the chain ([above](#kimi-k3-on-the-chain)): `vkc_kda_conv` and `vkc_kda_rec_flags` with Kimi K3's options; its full-rank output gate and low-rank decay are matmuls before the op |
- | mimo | sliding-window attention (and full layers) | `chain_attn.comp` with a window (positions from `max(0, pos - W + 1)`: one more push constant), the cache optionally a ring of W rows (the host mirror then indexes `t % W`) |
- | inkling | grouped attention, MoE with a shared expert | qwen36's attention and combine as they are; its per-position heads are matmuls of the residual rows |
- | olmoe | attention with q/k norm, MoE without a shared expert | qwen36's Qwen3-Coder geometry (all attention, no gate, no shared expert) is the same chain |
+ | mimo | sliding-window attention (and full layers) | done: [MiMo-V2.6 on the chain](#mimo-v26-on-the-chain) (`chain_attn.comp` with a window, a ring of W rows, V's own head dim and a sink) |
+ | inkling | grouped attention with a relative-position bias, a sliding window and short convolutions; MoE with shared experts | in the chain ([OLMoE and Inkling](#olmoe-and-inkling)): two shaders of its own, `chain_relattn.comp` and `chain_sconv.comp`; the shared experts join through `HC_APPLY` |
+ | olmoe | attention with q/k norm, MoE without a shared expert | in the chain ([OLMoE and Inkling](#olmoe-and-inkling)) with qwen36's ops as they are |
### Multi-head latent attention on the chain (`vkc_mla`)
@@@ -1140,13 -1257,7 +1341,13 @@@ in short
selection active, n-gram and MTP drafts accepted and rejected, glm53's image, KDA state
and swiglu_limit 0, a device lost (glm53's KDA state rebuilt), serve sessions with pins
and two KV slots, and glm53's pin-branch harness; `glm-chain-sanitize` runs them under
- ASan and UBSan; `qwen-chain-sanitize`
- runs the chain under ASan and UBSan. `kimi-chain` does it for Kimi K3: Moonshot's
+ ASan and UBSan.
++ `kimi-chain` does it for Kimi K3: Moonshot's
+ oracle with the chain on, every dense format, prefill a token at a time and in chunks,
+ the tier's eviction, prompts only, a device lost in a prompt, inside a chunked forward
+ and mid-decode (the KDA state rebuilt), serve sessions with prefix reuse and
+ recurrent-state checkpoints in RAM and on disk; `kimi-chain-sanitize` under ASan and
+ UBSan.
- int4 weights decode as offset-binary (nibble−8), byte-identical layout to
the CPU path — no repacking.
- Khronos validation layers: the backend never enables them, so the loader