Run frontier MoE models on hardware you already own — pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦
C
Merge the OLMoE, Inkling and MiMo chains into the MLA chain
JustVugg9bb60ea799d9
diff --cc c/Makefileindex a60020f5,8a18cd70..1df7ee54--- a/c/Makefile+++ b/c/Makefile@@@ -716,11 -716,9 +716,14 @@@ VK_OBJ += vk_chain.VK_SPV += shaders/chain_norm.spv shaders/chain_rope.spv shaders/chain_attn.spv shaders/chain_dnconv.spv \shaders/chain_dnrec.spv shaders/chain_ew.spv shaders/chain_qsa.spv shaders/chain_ple.spv \shaders/chain_gemv.spv+# the MLA ops (vk_chain.h): attention core, RoPE and LayerNorm rows, per-head weight+# blocks, the DSA indexer's selection+VK_SPV += shaders/chain_mla.spv shaders/chain_hgemv.spv shaders/chain_dsa.spv+# Kimi Delta Attention and manifold-constrained hyper-connections (vk_chain.h)+VK_SPV += shaders/chain_kda.spv shaders/chain_mhc.spv+ # inkling's chain ops (made on first use): the short convolution and the attention+ # with its relative-position bias+ VK_SPV += shaders/chain_sconv.spv shaders/chain_relattn.spvendif# Optional AMD XDNA2 (Ryzen AI NPU) lane, Windows. Same opt-in shape as VK=1:diff --cc c/tests/test_vk_chain.cindex a17d1626,c8f41ee5..267276f7--- a/c/tests/test_vk_chain.c+++ b/c/tests/test_vk_chain.c@@@ -18,20 -20,12 +20,26 @@@* qsa block keys, and the selection with ties broken as the CPU sorts* ple the gate and the dilated convolution with its ring and snapshot* frames vkc_write, vkc_read, a frame left in flight and one ordered after it+ * mla the per-head weight blocks in every format, both ways, with a gate; RoPE+ * interleaved and rotate-half, in place and into a cache row; LayerNorm;+ * the layer op against colibri.c's absorbed attention over several+ * geometries (q latent or none, NoPE, kv_b or the split halves, selection+ * lists with skipped entries, a gate, prefill rows, a long context from+ * kv_start, latents up to 1024)+ * dsa the indexer's top-k selection bit for bit, ties included+ * kpool the k-pooled indexer (GLM-5.3): pooled keys, the pools' rank order and+ * the tail, slot for slot against sparse_index.h+ * kda GLM-5.3's KDA layer around delta_attention.h's step: the convolution+ * and its window, the recurrence and its state over two submissions, the+ * decay, beta, output norm and gate+ * mhc hyper_connections.h's split, collapse and write back, the mean, and the+ * clamped SwiGLU+ * sconv inkling's short convolution (residual inside, the ring carried across+ * calls) and its scalar multiply and divide+ * relattn inkling's attention: the relative-position bias, tau, a sliding window+ * over a ring the step wraps, the step's rows read from its own K/V, and a+ * global layer; the shared experts' weighted add (HC_APPLY over one stream)+ * inkling's hidden size: the GEMV and the norm at D = 6144, the GEMV at I = 24576** make vk-chain-check VK=1 (VK_ICD_FILENAMES=.../lvp_icd.json for Lavapipe) */#include <stdio.h>@@@ -1058,41 -724,8 +1261,43 @@@ int main(int argc, char **argv)test_qsa();test_ple();test_frames();+ if (!vkc_mla_ready()) { fails++; printf("FAIL: the MLA shaders did not load\n"); }+ else {+ int hf[9] = {1, 2, 4, 5, 7, 10, 11, 12, 13};+ for (int k = 0; k < 9; k++) if (hf[k] != 7) { test_hgemv(hf[k], 0); test_hgemv(hf[k], 1); }+ test_mla_rows();+ printf("mla ops done\n");+ /* H Q R V K q_lora D S pos kv_start style split list gate fq fkv fo */+ MlaCase cases[] = {+ {4, 24, 8, 32, 32, 64, 128, 1, 12, 0, 1, 0, 0, 0, 10, 10, 10}, /* colibri's tiny, decode */+ {4, 24, 8, 32, 32, 64, 128, 5, 7, 0, 1, 0, 0, 0, 10, 10, 10}, /* prefill rows after earlier ones */+ {4, 24, 8, 32, 32, 64, 128, 1, 70, 0, 1, 0, 1, 0, 10, 10, 10}, /* a selection list */+ {3, 16, 16, 20, 48, 0, 96, 3, 9, 0, 0, 1, 0, 1, 1, 1, 1}, /* no q latent, rotate-half, split, gate, int8 */+ {2, 64, 0, 64, 128, 64, 128, 4, 66, 0, 0, 1, 1, 0, 4, 4, 4}, /* NoPE, int4-g64, lists */+ {2, 32, 64, 16, 64, 128, 128, 1, 300, 5, 1, 0, 0, 0, 2, 2, 11}, /* a long context from kv_start, int4 rows, bf16 */+ {2, 192, 64, 256, 512, 128, 256, 2, 70, 0, 1, 0, 0, 0, 4, 4, 4}, /* GLM-5.2's heads */+ {1, 16, 8, 16, 1024, 32, 64, 1, 20, 0, 1, 0, 0, 1, 10, 5, 10}, /* the largest latent, int3-g64 */+ };+ for (size_t k = 0; k < sizeof cases / sizeof *cases; k++) test_mla(cases[k]);+ printf("mla done\n");+ test_dsa(1, 30, 2, 16, 8, 0, 0); test_dsa(4, 40, 4, 32, 16, 0, 0); test_dsa(3, 5, 2, 16, 64, 0, 0);+ test_dsa(3, 5, 2, 16, 64, 1, 0); test_dsa(2, 700, 3, 64, 100, 0, 0); test_dsa(1, 9, 64, 64, 4, 0, 0);+ test_dsa(2, 900, 2, 16, 50, 0, 1);+ printf("dsa done\n");+ test_kpool(3, 20, 4, 16, 2, 8, 1); test_kpool(2, 9, 1, 16, 4, 8, 1); test_kpool(4, 300, 4, 64, 4, 64, 1);+ test_kpool(1, 2, 4, 16, 2, 4, 0); test_kpool(2, 30, 16, 16, 2, 6, 1);+ printf("kpool done\n");+ }+ if (!vkc_kda_ready() || !vkc_mhc_ready()) { fails++; printf("FAIL: the KDA or mHC shaders did not load\n"); }+ else {+ test_kda(3, 16, 4, 1.f); test_kda(2, 32, 4, 1.f); test_kda(4, 128, 3, 1.f); test_kda(1, 64, 2, 1.f);+ test_kda(2, 16, 4, 0.002f);+ printf("kda done\n");+ test_mhc(2, 64, 3); test_mhc(4, 96, 20); test_mhc(3, 40, 1);+ printf("mhc done\n");+ }+ if (vkc_sconv_ready() && vkc_relattn_ready()) { test_inkling(); printf("inkling done\n"); }+ else CHECK(0, "inkling's ops: chain_sconv.spv or chain_relattn.spv did not load");VkcStats st; vkc_stats(&st);printf("chain: %llu frames, %llu ops, %llu matmuls (%llu GEMM), %llu barriers\n", st.frames, st.ops, st.matmuls, st.gemms, st.barriers);vkc_shutdown();diff --cc c/tests/vulkan_engines.shindex 850c5afd,a3d11f16..f945b8c0--- a/c/tests/vulkan_engines.sh+++ b/c/tests/vulkan_engines.sh@@@ -7,7 -7,8 +7,9 @@@# bash tests/vulkan_engines.sh glm | glm-sanitize # GLM-5.2 (colibri) and GLM-5.3 Flash (glm53)# bash tests/vulkan_engines.sh shader # the qmatmul formats, the expert batch and the tier, no engine# bash tests/vulkan_engines.sh qwen-chain | qwen-chain-sanitize # the dense chain (COLI_VK_CHAIN=1)+ # bash tests/vulkan_engines.sh mimo-chain | mimo-chain-sanitize # MiMo-V2.6's dense chain+ # bash tests/vulkan_engines.sh inkling-olmoe-chain | inkling-olmoe-chain-sanitize+# bash tests/vulkan_engines.sh glm-chain | glm-chain-sanitize # the same for colibri and glm53## Needs libvulkan-dev, glslc and mesa-vulkan-drivers, plus the Python packages of# the family's tiny fixtures (see the vulkan-engines job in .github/workflows/ci.yml).@@@ -1341,194 -1498,202 +1499,390 @@@ family_qwen_chain_sanitize()make clean >/dev/null 2>&1 || true}+ # inkling's and olmoe's fixtures for their dense chain: the family inkling-olmoe's (the+ # bf16 copy of the f32 snapshot, the dense-int4g64 container, the expert containers and+ # the tokenizers), and Inkling at its real width (--wide: D = 6144, head dim 128).+ inkling_olmoe_chain_fixtures() {+ $PY tools/make_tiny_inkling.py tiny_inkling+ $PY tools/make_tiny_inkling.py tiny_inkling_wide --wide+ mkdir -p tiny_inkling_bf16+ cp tiny_inkling/config.json tiny_inkling/generation_config.json tiny_inkling_bf16/+ $PY - <<'EOF'+ import json, struct, numpy as np+ src, dst = "tiny_inkling/model.safetensors", "tiny_inkling_bf16/model.safetensors"+ raw = open(src, "rb").read(); n = struct.unpack("<Q", raw[:8])[0]; hdr = json.loads(raw[8:8 + n])+ out, blobs, off = {}, [], 0+ for k, v in hdr.items():+ if k == "__metadata__": out[k] = v; continue+ a = np.frombuffer(raw[8 + n + v["data_offsets"][0]: 8 + n + v["data_offsets"][1]], np.float32)+ u = a.view(np.uint32).astype(np.uint64)+ b = ((u + 0x7FFF + ((u >> 16) & 1)) >> 16).astype(np.uint16).tobytes()+ out[k] = {"dtype": "BF16", "shape": v["shape"], "data_offsets": [off, off + len(b)]}+ blobs.append(b); off += len(b)+ h = json.dumps(out).encode(); h += b" " * (-len(h) % 8)+ with open(dst, "wb") as f:+ f.write(struct.pack("<Q", len(h))); f.write(h); [f.write(b) for b in blobs]+ EOF+ rm -rf tiny_inkling_q && cp -r tiny_inkling tiny_inkling_q+ $PY - tools/convert_inkling_dense_int4.py tiny_inkling_q <<'EOF'+ import importlib.util, sys+ spec = importlib.util.spec_from_file_location("conv", sys.argv[1])+ conv = importlib.util.module_from_spec(spec); spec.loader.exec_module(conv)+ conv.MIN_ELEMS = 0+ conv.ATTN_BITS = 4+ base = conv.classify+ conv.classify = lambda n, s, d: "int8" if n.endswith(".mlp.down_proj.weight") else base(n, s, d)+ sys.argv = ["convert_inkling_dense_int4.py", "--dir", sys.argv[2]]+ conv.main()+ EOF+ mkdir -p tiny_inkling_q/dense-int4g64+ mv tiny_inkling_q/dense-int4g64.safetensors tiny_inkling_q/dense-int4g64/dense.safetensors+ $PY tools/make_olmoe_tiny.py --output olmoe_tiny --force+ $PY tools/convert_olmoe_merged.py --model olmoe_tiny --out olmoe_tiny_c+ inkling_olmoe_tier_fixtures+ }++ # chain_gate for a bf16 snapshot: where this CPU's bf16 dot rounds the activations+ # (AVX512-BF16), the per-matrix path keeps bf16 on the CPU and so does the chain, which+ # must then decline cleanly ("bf16 stays on the CPU") with the CPU's tokens.+ chain_gate_bf16() { # <engine> <tag> <env...> -- <argv...>+ local eng=$1 tag=$2; shift 2+ local envs=(); while [ "$1" != "--" ]; do envs+=("$1"); shift; done; shift+ env "${envs[@]}" COLI_VK_TIER_SYNC=1 COLI_VULKAN=1 COLI_VK_CHAIN=1 ./"$eng" "$@" > vk.log 2>&1 || true+ if grep -qa 'bf16 stays on the CPU' vk.log; then+ env "${envs[@]}" ./"$eng" "$@" > cpu.log 2>&1 || true+ same_tokens cpu.log vk.log "$tag"+ echo "OK $tag: tokens = CPU, the chain declined (this CPU's bf16 dot keeps bf16 on the CPU)"+ else+ chain_gate "$eng" "$tag" 1 "${envs[@]}" -- "$@"+ fi+ }++ # lost_gate for olmoe: attention only, so nothing to rebuild; the CPU redoes the step+ # the device was lost in from its first position and runs from there.+ lost_gate_redo() { # <engine> <tag> <frame> <env...> -- <argv...>+ local eng=$1 tag=$2 k=$3; shift 3+ local envs=(); while [ "$1" != "--" ]; do envs+=("$1"); shift; done; shift+ rm -f chain.usage+ env "${envs[@]}" ./"$eng" "$@" > cpu.log 2>&1 || true+ env "${envs[@]}" COLI_VK_CHAIN_FAULT=$k COLI_USAGE=chain.usage COLI_VK_TIER_SYNC=1 COLI_VULKAN=1 COLI_VK_CHAIN=1 \+ ./"$eng" "$@" > vk.log 2>&1 || true+ same_tokens cpu.log vk.log "$tag"+ grep -q "the CPU redoes the step from position [1-9]" vk.log || { cat vk.log; fail "$tag: no step was redone"; }+ echo "OK $tag: tokens = CPU, $(grep -o 'redoes the step from position [0-9]*' vk.log)"+ }++ # The device lost `back` frames before the end of a run (a probe run counts the frames,+ # so the fault lands mid-decode on any device, whatever its setup submits): lost_gate+ # (the state rebuilt on the CPU) or lost_gate_redo (the step redone there).+ lost_late() { # <engine> <tag> <frames back> rebuild|redo <env...> -- <argv...>+ local eng=$1 tag=$2 back=$3 kind=$4; shift 4+ local envs=(); while [ "$1" != "--" ]; do envs+=("$1"); shift; done; shift+ env "${envs[@]}" COLI_VK_TIER_SYNC=1 COLI_VULKAN=1 COLI_VK_CHAIN=1 ./"$eng" "$@" > vk.log 2>&1 || true+ local f; f=$(sed -n "s/^\[VK\] $eng chain: [0-9]* forwards, \([0-9]*\) frames.*/\1/p" vk.log | tail -1)+ { [ -n "$f" ] && [ "$f" -gt "$back" ]; } || { cat vk.log; fail "$tag: the probe run reports no frames"; }+ if [ "$kind" = rebuild ]; then lost_gate "$eng" "$tag" $((f - back)) "${envs[@]}" -- "$@"+ else lost_gate_redo "$eng" "$tag" $((f - back)) "${envs[@]}" -- "$@"; fi+ }++ # The dense chain (COLI_VK_CHAIN=1) of inkling and olmoe on Lavapipe, each configuration+ # against the CPU run's tokens and logits (chain_gate: every forward's logits within 1e-4+ # of the largest): inkling's resident forms (f32, bf16, the dense-int4g64 container's+ # int8 and int4-g64), its expert containers and runtime quantizations, TOPP, the tier+ # off, the trunk's device copies shared with the per-matrix path (COLI_VK_DENSE=1),+ # prefill in chunks of 3, the tiled GEMM and the per-row GEMV inside the chain, its real+ # width (D = 6144, beside the tier's expert batch at that width); olmoe's caps, PILOT's+ # prefetch from the chain's residual rows (cap 1: the slot the forward pass reads is+ # never the prefetcher's victim), an eviction budget and the same chain settings; both+ # with prompts only (COLI_VK_CHAIN=2), the device lost mid-run, and the serve, prefix+ # and dashboard tests with the chain on.+ family_inkling_olmoe_chain() {+ export OMP_NUM_THREADS=2 # tiny models: a wide team only waits for itself (and for Lavapipe's)+ make inkling olmoe tests/test_vk_chain VK=1+ ./tests/test_vk_chain shaders/qmatmul.spv | tee vk_chain.log+ tail -1 vk_chain.log | grep -qx PASS || fail "the chain's ops"+ inkling_olmoe_chain_fixtures+ local fx cap bits R=tiny_inkling/ref_inkling.json OR=olmoe_tiny/ref_olmoe.json+ for fx in tiny_inkling tiny_inkling_q tiny_inkling_x-i4 tiny_inkling_x-i8; do+ for cap in 1 8; do chain_gate inkling "chain inkling $fx cap=$cap" 1 SNAP=$fx -- $cap 0 $R; done+ done+ chain_gate_bf16 inkling "chain inkling bf16" SNAP=tiny_inkling_bf16 -- 8 0 $R+ for bits in 4 8; do chain_gate inkling "chain inkling runtime int$bits" 1 SNAP=tiny_inkling -- 2 $bits $R; done+ chain_gate inkling "chain inkling TOPP" 1 TOPP=0.3 SNAP=tiny_inkling_x-i4 -- 2 0 $R+ chain_gate inkling "chain inkling tier off" 1 COLI_VK_TIER=0 SNAP=tiny_inkling -- 8 0 $R+ chain_gate inkling "chain inkling trunk shared with the per-matrix path" 1 COLI_VK_DENSE=1 SNAP=tiny_inkling_q -- 8 0 $R+ chain_gate inkling "chain inkling prefill in chunks of 3" 1 COLI_VK_CHAIN_ROWS=3 SNAP=tiny_inkling -- 8 0 $R+ chain_gate inkling "chain inkling tiled GEMM" 1 COLI_VK_GEMM_MIN_S=2 SNAP=tiny_inkling -- 8 0 $R+ chain_gate inkling "chain inkling the per-row GEMV" 1 COLI_VK_CHAIN_GEMV=0 SNAP=tiny_inkling -- 8 0 $R+ CHAINMODE=2 chain_gate inkling "chain inkling prompts only" 1 SNAP=tiny_inkling -- 8 0 $R+ # D = 6144: the CPU's tokens through the chain, and through the tier's expert batch+ # (its gate/up shader stages x up to 6144 floats) with the chain off+ chain_gate inkling "chain inkling D=6144" 1 SNAP=tiny_inkling_wide -- 8 0 tiny_inkling_wide/ref_inkling.json+ tier_gate inkling "inkling tier D=6144" COLI_VK_CHAIN=0 SNAP=tiny_inkling_wide -- 8 0 tiny_inkling_wide/ref_inkling.json+ # the device lost mid-decode, in the last step but one: in a shared-expert frame+ # nobody waits for (20 frames before the end) and at a layer's router (19): the+ # state rebuilt on the CPU from the prefix record, the run finished there+ lost_late inkling "chain inkling device lost in a shared-expert frame" 20 rebuild SNAP=tiny_inkling -- 8 0 $R+ lost_late inkling "chain inkling device lost at a router" 19 rebuild SNAP=tiny_inkling -- 8 0 $R++ for cap in 1 8; do chain_gate olmoe "chain olmoe cap=$cap" 1 SNAP=olmoe_tiny_c -- $cap 8 $OR; done+ chain_gate olmoe "chain olmoe 4-bit experts" 1 SNAP=olmoe_tiny_c -- 8 4 $OR+ for cap in 1 2; do chain_gate olmoe "chain olmoe PILOT cap=$cap" 1 PILOT=1 WIDE=2 SNAP=olmoe_tiny_c -- $cap 8 $OR; done+ chain_gate olmoe "chain olmoe PILOT=3" 1 PILOT=3 SNAP=olmoe_tiny_c -- 8 8 $OR+ chain_gate olmoe "chain olmoe tier off" 1 COLI_VK_TIER=0 SNAP=olmoe_tiny_c -- 8 8 $OR+ chain_gate olmoe "chain olmoe trunk shared with the per-matrix path" 1 COLI_VK_DENSE=1 SNAP=olmoe_tiny_c -- 8 8 $OR+ chain_gate olmoe "chain olmoe a budget of three experts" 1 COLI_VK_TIER_GB=0.000025 SNAP=olmoe_tiny_c -- 8 8 $OR+ chain_gate olmoe "chain olmoe prefill in chunks of 3" 1 COLI_VK_CHAIN_ROWS=3 SNAP=olmoe_tiny_c -- 8 8 $OR+ chain_gate olmoe "chain olmoe tiled GEMM" 1 COLI_VK_GEMM_MIN_S=2 SNAP=olmoe_tiny_c -- 8 8 $OR+ chain_gate olmoe "chain olmoe the per-row GEMV" 1 COLI_VK_CHAIN_GEMV=0 SNAP=olmoe_tiny_c -- 8 8 $OR+ CHAINMODE=2 chain_gate olmoe "chain olmoe prompts only" 1 SNAP=olmoe_tiny_c -- 8 8 $OR+ lost_late olmoe "chain olmoe device lost mid-decode" 7 redo SNAP=olmoe_tiny_c -- 8 8 $OR++ # serve: a session of pins, prompt-cache extensions, a divergent prompt and logprobs+ # against the CPU's, frame by frame (and with prompts only); the prefix-reuse and+ # dashboard contracts with the chain on+ $PY tests/vulkan_chain_serve.py ./inkling tiny_inkling INK_PREFIX_LOG=1+ COLI_VK_CHAIN=2 $PY tests/vulkan_chain_serve.py ./inkling tiny_inkling INK_PREFIX_LOG=1+ $PY tests/vulkan_chain_serve.py ./olmoe olmoe_tiny_c+ $PY tests/vulkan_chain_serve.py ./olmoe olmoe_tiny_c PILOT=1 WIDE=2+ COLI_VK_CHAIN=2 $PY tests/vulkan_chain_serve.py ./olmoe olmoe_tiny_c+ # (these compare a warm session's logprobs with a cold one's to the last printed+ # digit: no history file and COLI_VK_TIER_BALANCE=0, so both sessions split their+ # experts between the device and the CPU alike)+ COLI_VULKAN=1 COLI_VK_CHAIN=1 COLI_VK_TIER_SYNC=1 COLI_VK_TIER_BALANCE=0 INKLING_TINY=tiny_inkling \+ $PY -m unittest tests.test_inkling_prefix_serve tests.test_inkling_dashboard_hits+ COLI_VULKAN=1 COLI_VK_CHAIN=1 COLI_VK_TIER_SYNC=1 COLI_VK_TIER_BALANCE=0 OLMOE_TINY=olmoe_tiny_c \+ $PY -m unittest tests.test_olmoe_prefix_serve tests.test_olmoe_dashboard_hits tests.test_brio_serve+ }++ # inkling's and olmoe's chain under ASan and UBSan: memory safety is the gate (a+ # sanitized build vectorizes differently, so tokens are not compared); each run must+ # have run the chain. detect_stack_use_after_return=0: ASan's fake stack does not keep+ # the 64-byte alignment an AVX-512 build gives its locals.+ family_inkling_olmoe_chain_sanitize() {+ make clean >/dev/null 2>&1 || true+ make inkling olmoe tests/test_vk_chain VK=1 EXTRA_CFLAGS="-fsanitize=address,undefined -fno-omit-frame-pointer -g"+ export ASAN_OPTIONS=detect_leaks=0:detect_stack_use_after_return=0 UBSAN_OPTIONS=print_stacktrace=1+ ./tests/test_vk_chain shaders/qmatmul.spv > san.log 2>&1 || true+ if grep -qE "ERROR: AddressSanitizer|runtime error:" san.log || ! tail -1 san.log | grep -qx PASS; then cat san.log; fail "asan: the chain's ops"; fi+ echo "OK asan: the chain's ops"+ inkling_olmoe_chain_fixtures+ csan() { # <engine> <tag> <env and argv...>+ local eng=$1 tag=$2; shift 2+ rm -f chain.usage+ env OMP_NUM_THREADS=2 COLI_USAGE=chain.usage COLI_VK_TIER_SYNC=1 COLI_VULKAN=1 COLI_VK_CHAIN=${CHAINMODE:-1} "$@" > san.log 2>&1 || true+ if grep -qE "ERROR: AddressSanitizer|runtime error:" san.log; then cat san.log; fail "$tag: sanitizer diagnostic"; fi+ [ "$(chain_count "$eng" san.log)" -gt 0 ] || { cat san.log; fail "$tag: the chain never ran"; }+ echo "OK $tag: sanitizers clean, $(chain_count "$eng" san.log) chain forwards"+ }+ local R=tiny_inkling/ref_inkling.json OR=olmoe_tiny/ref_olmoe.json+ csan inkling "asan chain inkling f32 cap=1" SNAP=tiny_inkling ./inkling 1 0 $R+ csan inkling "asan chain inkling dense-int4g64, the trunk shared" COLI_VK_DENSE=1 SNAP=tiny_inkling_q ./inkling 8 0 $R+ csan inkling "asan chain inkling TOPP, chunks of 3" TOPP=0.3 COLI_VK_CHAIN_ROWS=3 SNAP=tiny_inkling_x-i4 ./inkling 2 0 $R+ csan inkling "asan chain inkling D=6144" SNAP=tiny_inkling_wide ./inkling 8 0 tiny_inkling_wide/ref_inkling.json+ CHAINMODE=2 csan inkling "asan chain inkling prompts only" SNAP=tiny_inkling ./inkling 8 0 $R+ csan inkling "asan chain inkling device lost" COLI_VK_CHAIN_FAULT=60 SNAP=tiny_inkling ./inkling 8 0 $R+ csan olmoe "asan chain olmoe PILOT cap=1" PILOT=1 WIDE=2 SNAP=olmoe_tiny_c ./olmoe 1 8 $OR+ csan olmoe "asan chain olmoe eviction, chunks of 3" COLI_VK_TIER_GB=0.000025 COLI_VK_CHAIN_ROWS=3 SNAP=olmoe_tiny_c ./olmoe 8 8 $OR+ csan olmoe "asan chain olmoe device lost" COLI_VK_CHAIN_FAULT=20 SNAP=olmoe_tiny_c ./olmoe 8 8 $OR+ local eng+ for eng in inkling olmoe; do+ local snap=tiny_inkling; [ $eng = olmoe ] && snap=olmoe_tiny_c+ $PY tests/vulkan_chain_serve.py ./$eng $snap OMP_NUM_THREADS=2 INK_PREFIX_LOG=1 > san.log 2>&1 || { cat san.log; fail "asan chain $eng serve"; }+ if grep -qE "ERROR: AddressSanitizer|runtime error:" san.log; then cat san.log; fail "asan chain $eng serve: sanitizer diagnostic"; fi+ echo "OK asan chain $eng serve: $(tail -1 san.log)"+ done+ make clean >/dev/null 2>&1 || true+ }++# The dense chain of the MLA engines: GLM-5.2 (colibri) and GLM-5.3 Flash (glm53).+# mla_gate <engine> <tag> <tol 0|1> <env...> -- <argv...>: the CPU run's token lines+# (colibri: the generated tokens, the teacher-forced predictions and the oracle's+# mismatches; glm53: teacher_forcing and greedy), every logits row within 1e-4 of the+# largest |logit| (tol 1; DUMP= writes them), and a "[VK] <engine> chain: N forwards"+# line with N > 0. FAULT_BACK=k: the device is lost k frames before the end of the same+# run without a fault (COLI_VK_CHAIN_FAULT counted from that run's frames, so the loss+# lands in the same forward on every device, whatever frames its setup took), and the+# run must say so (and, for glm53 with REBUILD=1, rebuild the KDA state of some positions).+mla_toks() { # <engine> <log>+ if [ "$1" = colibri ]; then grep -aE '^GLM C engine|^PREFILL|^\[ORACLE\] mismatch' "$2" | sed 's/ | [0-9.]* pos\/s//'+ else grep -aE '^teacher_forcing|^greedy' "$2"; fi+}+mla_gate() {+ local eng=$1 tag=$2 tol=$3; shift 3+ local envs=(); while [ "$1" != "--" ]; do envs+=("$1"); shift; done; shift+ rm -f chain.usage cpu.f32 vk.f32+ env "${envs[@]}" COLI_USAGE=chain.usage USAGE_SAVE=0 DUMP=cpu.f32 ./"$eng" "$@" > cpu.log 2>&1 || true+ if [ -n "${FAULT_BACK:-}" ]; then+ rm -f chain.usage+ env "${envs[@]}" COLI_USAGE=chain.usage USAGE_SAVE=0 COLI_VK_TIER_SYNC=1 COLI_VULKAN=1 COLI_VK_CHAIN=1 \+ ./"$eng" "$@" > vk.log 2>&1 || true+ local frames; frames=$(sed -n "s/^\[VK\] $eng chain: [0-9]* forwards, \([0-9]*\) frames.*/\1/p" vk.log | tail -1)+ [ -n "$frames" ] && [ "$frames" -gt "$FAULT_BACK" ] || { cat vk.log; fail "$tag: no fault-free run to count frames from"; }+ envs+=("COLI_VK_CHAIN_FAULT=$((frames - FAULT_BACK + 1))")+ fi+ rm -f chain.usage+ env "${envs[@]}" COLI_USAGE=chain.usage USAGE_SAVE=0 DUMP=vk.f32 COLI_VK_TIER_SYNC=1 COLI_VULKAN=1 \+ COLI_VK_CHAIN=${CHAINMODE:-1} ./"$eng" "$@" > vk.log 2>&1 || true+ mla_toks "$eng" cpu.log > cpu.tok; mla_toks "$eng" vk.log > vk.tok+ { [ -s cpu.tok ] && cmp -s cpu.tok vk.tok; } || { cat cpu.tok vk.tok; tail -20 vk.log; fail "$tag: the chain's tokens differ from the CPU"; }+ if [ -n "${FAULT_BACK:-}" ]; then+ grep -q "$eng chain: the device was lost" vk.log || { cat vk.log; fail "$tag: no loss was handled"; }+ if [ "${REBUILD:-0}" = 1 ]; then grep -q "rebuilding the state of [1-9]" vk.log || { cat vk.log; fail "$tag: no state was rebuilt"; }; fi+ else+ [ "$(chain_count "$eng" vk.log)" -gt 0 ] || { cat vk.log; fail "$tag: the chain never ran"; }+ fi+ local lg=""+ if [ "$tol" = 1 ]; then lg=$(logits_close cpu.f32 vk.f32) || { echo "$lg"; fail "$tag: logits"; }; lg=", $lg"; fi+ echo "OK $tag: tokens = CPU$lg, $(chain_count "$eng" vk.log) chain forwards$(grep -o 'rebuilding the state of [0-9]* positions' vk.log | sed 's/^/, /')"+}+glm_chain_fixtures() {+ glm_fixtures+ $PY tools/make_glm_mtp_tiny.py --src glm_tiny --out glm_tiny_mtp > /dev/null && cp ref_glm.json glm_tiny_mtp/+ rm -rf glm_tiny_serve && cp -r glm_tiny glm_tiny_serve+ $PY tools/make_edge_tiny_tokenizer.py --vocab-size 256 ./glm_tiny_serve > /dev/null # serve speaks text+ rm -rf glm53_serve && cp -r glm53_stream-i4 glm53_serve+ $PY tools/make_edge_tiny_tokenizer.py --vocab-size 128 ./glm53_serve > /dev/null+ $PY tools/make_glm53_multimodal_tiny.py --output glm53_mm_tiny > /dev/null+ rm -rf glm53_lim0 && cp -r glm53_stream-i4 glm53_lim0 # swiglu_limit 0: the tier declines, the chain does not+ $PY - <<'PY'+import json+p = "glm53_lim0/config.json"; c = json.load(open(p))+(c["text_config"] if "text_config" in c else c)["swiglu_limit"] = 0.0+json.dump(c, open(p, "w"))+PY+}++# Every configuration the chain takes: colibri in every expert format (IDOT=0 where the+# CPU would round activations to int8: the device matches IDOT=0, as for the tier),+# decode and teacher-forced prefill in chunks, the DSA indexer's selection active+# (DSA_TOPK=4 on the 12-token prompt, and DSA_FORCE), n-gram and MTP drafts accepted and+# rejected (the MTP fixture's head is the last layer's copy), the tier off, the per-matrix+# trunk and attention core beside it, prompts only, a device lost mid-decode and in a+# prompt, serve sessions (pins, the prompt cache, two KV slots, the prefill read-out);+# glm53 with f32, int8 and int4 trunks, streamed and resident experts, an image, prefill+# chunks and chain chunks, swiglu_limit 0, a lost device whose KDA state is rebuilt on+# the CPU, serve sessions, and its pin-branch harness with the chain on.+family_glm_chain() {+ make colibri glm53 tests/test_vk_chain VK=1+ ./tests/test_vk_chain shaders/qmatmul.spv | tee vk_chain.log+ tail -1 vk_chain.log | grep -qx PASS || fail "the chain's ops"+ glm_chain_fixtures+ export OMP_NUM_THREADS=2 CAP_RAISE=0+ local cap b fx d+ for cap in 1 64; do mla_gate colibri "chain colibri f32 cap=$cap" 1 SNAP=glm_tiny REF=ref_glm.json -- $cap 16 16; done+ mla_gate colibri "chain colibri f32 prefill" 1 SNAP=glm_tiny REF=ref_glm.json TF=1 -- 64 16 16+ mla_gate colibri "chain colibri prefill in chunks of 3" 1 SNAP=glm_tiny REF=ref_glm.json TF=1 COLI_VK_CHAIN_ROWS=3 -- 64 16 16+ for b in 8 4 3; do+ mla_gate colibri "chain colibri ${b}-bit trunk and experts" 1 SNAP=glm_tiny REF=ref_glm.json IDOT=0 -- 2 $b $b+ done+ mla_gate colibri "chain colibri int4-g64 experts" 1 SNAP=glm_tiny_fmt4 REF=glm_tiny_fmt4/ref_glm.json -- 2 16 16+ mla_gate colibri "chain colibri E8/IQ3 experts on the CPU" 1 SNAP=glm_tiny_fmt6 REF=glm_tiny_fmt6/ref_glm.json -- 2 16 16+ for fx in i4 i4r i3 d3; do+ mla_gate colibri "chain colibri $fx container" 1 SNAP=glm_tiny_$fx REF=glm_tiny_$fx/ref_glm.json IDOT=0 -- 1 4 4+ done+ mla_gate colibri "chain colibri i4 container, prefill" 1 SNAP=glm_tiny_i4 REF=glm_tiny_i4/ref_glm.json IDOT=0 TF=1 -- 2 4 4+ # the DSA indexer's selection on the device: index_topk 4 against a 32-token context+ mla_gate colibri "chain colibri DSA top-4" 1 SNAP=glm_tiny REF=ref_glm.json DSA_TOPK=4 -- 64 16 16+ mla_gate colibri "chain colibri DSA top-4, prefill in chunks of 5" 1 SNAP=glm_tiny REF=ref_glm.json DSA_TOPK=4 TF=1 COLI_VK_CHAIN_ROWS=5 -- 64 16 16+ mla_gate colibri "chain colibri DSA_FORCE" 1 SNAP=glm_tiny REF=ref_glm.json DSA_FORCE=1 -- 64 16 16+ # drafts: n-gram, and the MTP head at depths 1 to 3 (accepted and rejected)+ mla_gate colibri "chain colibri n-gram drafts" 1 SNAP=glm_tiny REF=ref_glm.json DRAFT=3 -- 64 16 16+ for d in 1 2 3; do mla_gate colibri "chain colibri MTP depth $d" 1 SNAP=glm_tiny_mtp REF=glm_tiny_mtp/ref_glm.json DRAFT=$d -- 64 16 16; done+ mla_gate colibri "chain colibri MTP with DSA top-4" 1 SNAP=glm_tiny_mtp REF=glm_tiny_mtp/ref_glm.json DSA_TOPK=4 -- 64 16 16+ mla_gate colibri "chain colibri MTP, 4-bit" 1 SNAP=glm_tiny_mtp REF=glm_tiny_mtp/ref_glm.json IDOT=0 DRAFT=2 -- 2 4 4+ # around the chain: the tier off, the per-matrix trunk and core, the GEMM and GEMV choices+ mla_gate colibri "chain colibri tier off" 1 SNAP=glm_tiny REF=ref_glm.json COLI_VK_TIER=0 -- 64 16 16+ mla_gate colibri "chain colibri beside COLI_VK_DENSE=1 COLI_VK_ATTN=1" 1 SNAP=glm_tiny_i4 REF=glm_tiny_i4/ref_glm.json IDOT=0 COLI_VK_DENSE=1 COLI_VK_ATTN=1 -- 2 4 4+ # COLI_VK_DEV2 beside the chain: a second logical device holds what the capped tier does not+ SNAP=glm_tiny_i4r REF=glm_tiny_i4r/ref_glm.json IDOT=0 STATS=glm_tiny_i4r/.coli_usage ./colibri 64 4 4 > /dev/null 2>&1 || true+ mla_gate colibri "chain colibri beside COLI_VK_DEV2" 1 SNAP=glm_tiny_i4r REF=glm_tiny_i4r/ref_glm.json IDOT=0 COLI_VK_EXPERTS=4 COLI_VK_DEV2=0 -- 64 4 4+ grep -qaE 'lru \+ [1-9][0-9]* vk /' vk.log || { cat vk.log; fail "chain colibri beside COLI_VK_DEV2: no expert on the devices"; }+ rm -f glm_tiny_i4r/.coli_usage+ mla_gate colibri "chain colibri tiled GEMM from 2 rows" 1 SNAP=glm_tiny REF=ref_glm.json TF=1 COLI_VK_GEMM_MIN_S=2 -- 64 16 16+ mla_gate colibri "chain colibri the per-row GEMV" 1 SNAP=glm_tiny REF=ref_glm.json COLI_VK_CHAIN_GEMV=0 -- 64 16 16+ CHAINMODE=2 mla_gate colibri "chain colibri prompts only, drafts" 1 SNAP=glm_tiny REF=ref_glm.json DRAFT=3 -- 64 16 16+ # the device lost: mid-decode, in the prompt (the teacher-forced pass is one forward),+ # between MTP drafts+ FAULT_BACK=12 mla_gate colibri "chain colibri device lost mid-decode" 1 SNAP=glm_tiny REF=ref_glm.json -- 64 16 16+ FAULT_BACK=3 mla_gate colibri "chain colibri device lost in the prompt" 1 SNAP=glm_tiny REF=ref_glm.json TF=1 -- 64 16 16+ FAULT_BACK=12 mla_gate colibri "chain colibri device lost with MTP" 1 SNAP=glm_tiny_mtp REF=glm_tiny_mtp/ref_glm.json -- 64 16 16+ # serve sessions frame for frame: pins, the prompt cache, the prefill read-out, two KV slots+ CHAIN_SERVE_DIALECT=colibri $PY tests/vulkan_chain_serve.py ./colibri glm_tiny_serve SERVE_BATCH=1 IDOT=0+ CHAIN_SERVE_DIALECT=colibri $PY tests/vulkan_chain_serve.py ./colibri glm_tiny_serve SERVE_BATCH=1 IDOT=0 DSA_TOPK=4 DRAFT=3+ CHAIN_SERVE_SLOTS=2 CHAIN_SERVE_DIALECT=colibri $PY tests/vulkan_chain_serve.py ./colibri glm_tiny_serve SERVE_BATCH=1 IDOT=0 KV_SLOTS=2+ COLI_VK_CHAIN=2 CHAIN_SERVE_DIALECT=colibri $PY tests/vulkan_chain_serve.py ./colibri glm_tiny_serve SERVE_BATCH=1 IDOT=0 DSA_TOPK=4++ # glm53+ local ids bits+ ids=$($PY -c "print(','.join(str((i*37+5)%120+2) for i in range(100)))")+ mla_gate glm53 "chain glm53 decode" 1 GLM53_BITS=32 -- --model glm53_stream-i4 --ids 5,7,9,11,13,17,19,23 --greedy 8+ for bits in 32 8 4; do+ mla_gate glm53 "chain glm53 100-token prompt, ${bits}-bit trunk" 1 GLM53_BITS=$bits -- --model glm53_stream-i4 --ids $ids --greedy 4+ done+ mla_gate glm53 "chain glm53 prefill chunks of 7, one cache slot" 1 GLM53_BITS=4 GLM53_PREFILL_CHUNK=7 GLM53_EXPERT_GB=0.000001 -- --model glm53_stream-i4 --ids $ids --greedy 4+ mla_gate glm53 "chain glm53 chain chunks of 3" 1 GLM53_BITS=32 COLI_VK_CHAIN_ROWS=3 -- --model glm53_stream-i4 --ids $ids --greedy 4+ mla_gate glm53 "chain glm53 resident experts" 1 GLM53_BITS=32 -- --model glm53_tiny --ids $ids --greedy 6+ mla_gate glm53 "chain glm53 tier off" 1 GLM53_BITS=32 COLI_VK_TIER=0 -- --model glm53_stream-i4 --ids $ids --greedy 6+ mla_gate glm53 "chain glm53 an image" 1 GLM53_BITS=32 -- --model glm53_mm_tiny --ids 103,117,268,268,268,268,120,121 --patches glm53_mm_tiny/patches.f32 --grid 4x4 --greedy 4+ mla_gate glm53 "chain glm53 swiglu_limit 0" 1 GLM53_BITS=32 -- --model glm53_lim0 --ids $ids --greedy 4+ grep -qa 'tier glm53: swiglu_limit is 0' vk.log || { cat vk.log; fail "chain glm53 swiglu_limit 0: the tier did not decline"; }+ CHAINMODE=2 mla_gate glm53 "chain glm53 prompts only" 1 GLM53_BITS=32 GLM53_PREFILL_CHUNK=16 -- --model glm53_stream-i4 --ids $ids --greedy 6+ FAULT_BACK=7 REBUILD=1 mla_gate glm53 "chain glm53 device lost, the KDA state rebuilt" 1 GLM53_BITS=32 GLM53_PREFILL_CHUNK=16 -- --model glm53_stream-i4 --ids $ids --greedy 6+ FAULT_BACK=5 REBUILD=1 mla_gate glm53 "chain glm53 device lost after an image" 1 GLM53_BITS=32 -- --model glm53_mm_tiny --ids 103,117,268,268,268,268,120,121 --patches glm53_mm_tiny/patches.f32 --grid 4x4 --greedy 4+ CHAIN_SERVE_DIALECT=numeric $PY tests/vulkan_chain_serve.py ./glm53 glm53_serve GLM53_BITS=32+ CHAIN_SERVE_DIALECT=numeric $PY tests/vulkan_chain_serve.py ./glm53 glm53_serve GLM53_BITS=4 COLI_VK_CHAIN_ROWS=3+ CHAIN_SERVE_SLOTS=2 CHAIN_SERVE_DIALECT=numeric $PY tests/vulkan_chain_serve.py ./glm53 glm53_serve GLM53_BITS=32 KV_SLOTS=2+ COLI_VK_CHAIN=2 CHAIN_SERVE_DIALECT=numeric $PY tests/vulkan_chain_serve.py ./glm53 glm53_serve GLM53_BITS=32+ # a pin restored over rows another branch rewrote: the KDA state goes up from the pin,+ # the MLA rows' watermark comes down+ COLI_VULKAN=1 COLI_VK_CHAIN=1 COLI_USAGE=$PWD/chain.usage $PY tests/glm53_pin_branch_harness.py --binary ./glm53 --fixture glm53_mm_tiny+ unset OMP_NUM_THREADS CAP_RAISE+}++# The same chains under ASan and UBSan: memory safety is the gate; each run must have run+# the chain (or handled the loss). detect_stack_use_after_return=0 as in every sanitized+# family (ASan's fake stack breaks 64-byte aligned AVX-512 locals).+family_glm_chain_sanitize() {+ make clean >/dev/null 2>&1 || true+ make colibri glm53 tests/test_vk_chain VK=1 EXTRA_CFLAGS="-fsanitize=address,undefined -fno-omit-frame-pointer -g"+ export ASAN_OPTIONS=detect_leaks=0:detect_stack_use_after_return=0 UBSAN_OPTIONS=print_stacktrace=1+ ./tests/test_vk_chain shaders/qmatmul.spv > san.log 2>&1 || true+ if grep -qE "ERROR: AddressSanitizer|runtime error:" san.log || ! tail -1 san.log | grep -qx PASS; then cat san.log; fail "asan: the chain's ops"; fi+ echo "OK asan: the chain's ops"+ glm_chain_fixtures+ export OMP_NUM_THREADS=2 CAP_RAISE=0+ msan() { # <engine> <tag> <env and argv...>+ local eng=$1 tag=$2; shift 2+ rm -f chain.usage+ env COLI_USAGE=chain.usage USAGE_SAVE=0 COLI_VK_TIER_SYNC=1 COLI_VULKAN=1 COLI_VK_CHAIN=1 "$@" > san.log 2>&1 || true+ if grep -qE "ERROR: AddressSanitizer|runtime error:" san.log; then cat san.log; fail "$tag: sanitizer diagnostic"; fi+ grep -q "$eng chain: \([1-9][0-9]* forwards\|the device was lost\)" san.log || { cat san.log; fail "$tag: the chain never ran"; }+ echo "OK $tag: sanitizers clean, $(chain_count "$eng" san.log) chain forwards"+ }+ local ids+ ids=$($PY -c "print(','.join(str((i*37+5)%120+2) for i in range(100)))")+ msan colibri "asan chain colibri f32 decode" SNAP=glm_tiny REF=ref_glm.json ./colibri 1 16 16+ msan colibri "asan chain colibri DSA top-4 prefill, chunks of 3" SNAP=glm_tiny REF=ref_glm.json TF=1 DSA_TOPK=4 COLI_VK_CHAIN_ROWS=3 ./colibri 64 16 16+ msan colibri "asan chain colibri i4 container" SNAP=glm_tiny_i4 REF=glm_tiny_i4/ref_glm.json IDOT=0 ./colibri 2 4 4+ msan colibri "asan chain colibri MTP depth 2" SNAP=glm_tiny_mtp REF=glm_tiny_mtp/ref_glm.json DRAFT=2 ./colibri 64 16 16+ msan colibri "asan chain colibri device lost" SNAP=glm_tiny REF=ref_glm.json COLI_VK_CHAIN_FAULT=30 ./colibri 64 16 16+ msan glm53 "asan chain glm53 decode" GLM53_BITS=32 ./glm53 --model glm53_stream-i4 --ids 5,7,9,11,13,17,19,23 --greedy 8+ msan glm53 "asan chain glm53 int4 trunk, chain chunks of 3" GLM53_BITS=4 COLI_VK_CHAIN_ROWS=3 ./glm53 --model glm53_stream-i4 --ids $ids --greedy 4+ msan glm53 "asan chain glm53 an image" GLM53_BITS=32 ./glm53 --model glm53_mm_tiny --ids 103,117,268,268,268,268,120,121 --patches glm53_mm_tiny/patches.f32 --grid 4x4 --greedy 4+ msan glm53 "asan chain glm53 device lost, rebuilt" GLM53_BITS=32 GLM53_PREFILL_CHUNK=16 COLI_VK_CHAIN_FAULT=20 ./glm53 --model glm53_stream-i4 --ids $ids --greedy 6+ local args+ for args in "./colibri glm_tiny_serve SERVE_BATCH=1 IDOT=0 KV_SLOTS=2" "./glm53 glm53_serve GLM53_BITS=32 KV_SLOTS=2"; do+ # shellcheck disable=SC2086+ CHAIN_SERVE_SLOTS=2 CHAIN_SERVE_DIALECT=$([ "${args%% *}" = ./colibri ] && echo colibri || echo numeric) \+ $PY tests/vulkan_chain_serve.py $args > san.log 2>&1 || { cat san.log; fail "asan chain serve $args"; }+ if grep -qE "ERROR: AddressSanitizer|runtime error:" san.log; then cat san.log; fail "asan chain serve $args: sanitizer diagnostic"; fi+ echo "OK asan chain serve ${args%% *}: $(tail -1 san.log)"+ done+ unset OMP_NUM_THREADS CAP_RAISE+ make clean >/dev/null 2>&1 || true+}+case "${1:-}" inshader) shader_formats ;;qwen) family_qwen ;;@@@ -1544,7 -1711,7 +1900,9 @@@glm-sanitize) family_glm_sanitize ;;qwen-chain) family_qwen_chain ;;qwen-chain-sanitize) family_qwen_chain_sanitize ;;+ inkling-olmoe-chain) family_inkling_olmoe_chain ;;+ inkling-olmoe-chain-sanitize) family_inkling_olmoe_chain_sanitize ;;- *) echo "usage: $0 shader|qwen|qwen-sanitize|inkling-olmoe|inkling-olmoe-sanitize|mimo-qwenimage|deepseek|deepseek-sanitize|kimi|kimi-mimo-sanitize|glm|glm-sanitize|qwen-chain|qwen-chain-sanitize|mimo-chain|mimo-chain-sanitize|inkling-olmoe-chain|inkling-olmoe-chain-sanitize" >&2; exit 2 ;;+ glm-chain) family_glm_chain ;;+ glm-chain-sanitize) family_glm_chain_sanitize ;;- *) echo "usage: $0 shader|qwen|qwen-sanitize|inkling-olmoe|inkling-olmoe-sanitize|mimo-qwenimage|deepseek|deepseek-sanitize|kimi|kimi-mimo-sanitize|glm|glm-sanitize|qwen-chain|qwen-chain-sanitize|glm-chain|glm-chain-sanitize" >&2; exit 2 ;;++ *) echo "usage: $0 shader|qwen|qwen-sanitize|inkling-olmoe|inkling-olmoe-sanitize|mimo-qwenimage|deepseek|deepseek-sanitize|kimi|kimi-mimo-sanitize|glm|glm-sanitize|qwen-chain|qwen-chain-sanitize|mimo-chain|mimo-chain-sanitize|inkling-olmoe-chain|inkling-olmoe-chain-sanitize|glm-chain|glm-chain-sanitize" >&2; exit 2 ;;esacdiff --cc c/vk_chain.cindex ee112a1d,6d2e0d65..3fda6c1e--- a/c/vk_chain.c+++ b/c/vk_chain.c@@@ -782,202 -767,46 +791,242 @@@ int vkc_ple(VkcBuf *keys, VkcBuf *hyp,return record(K.pipe[P_PLE], bd, 8, p, sizeof *p, gx, gy, 1);}+/* ---- multi-head latent attention ------------------------------------------------- */+int vkc_mla_ready(void) { return vkc_ready() && K.mla_ok; }++int vkc_mla_core(VkcBuf *qabs, VkcBuf *qr, VkcBuf *lat, VkcBuf *rope, VkcBuf *sel, VkcBuf *clat, const VkcMlaCore *p) {+ K.kind = PK_MLA;+ if (!K.mla_ok || p->S < 1 || p->H < 1 || p->K < 1 || p->K > 1024 || p->R < 0 || p->R > 128 ||+ (p->R > 0 && !rope) || p->pos_base < 0 || p->kv_start < 0 || p->kv_start > p->pos_base) return 0;+ struct { int mode; VkcMlaCore b; } pc = {0, *p};+ if (!sel) pc.b.sel_row = 0;+ VkcBind bd[6] = {B(qabs, 0), B(qr, 0), B(lat, 0), B(p->R > 0 ? rope : NULL, 0), B(pc.b.sel_row > 0 ? sel : NULL, 0),+ B(clat, 1)};+ return record(K.mpipe[PM_MLA], bd, 6, &pc, sizeof pc, (uint32_t)p->H, (uint32_t)p->S, 1);+}+static int mla_rowop(int mode, VkcBuf *x, VkcBuf *aux, VkcBuf *y, const VkcMlaRow *p) {+ if (!K.mla_ok || !open_frame() || K.lost || p->nseg < 0) return 0;+ if (p->nseg == 0) return 1;+ if (mode == 1 && (p->rd < 2 || p->rd > 256 || (p->rd & 1) || p->rd > p->seg_len)) return 0;+ if (mode == 2 && p->seg_len < 1) return 0;+ struct { int mode; VkcMlaRow b; int copy_rest; } pc = {mode, *p, x != y};+ VkcBind bd[6] = {B(x, x == y), B(aux, 0), B(NULL, 0), B(NULL, 0), B(NULL, 0), B(y, 1)};+ uint32_t gx, gy; grid((uint64_t)p->nseg, &gx, &gy);+ return record(K.mpipe[PM_MLA], bd, 6, &pc, sizeof pc, gx, gy, 1);+}+int vkc_mla_rope(VkcBuf *x, VkcBuf *cs, VkcBuf *y, const VkcMlaRow *p) { K.kind = PK_ROPE; return mla_rowop(1, x, cs, y, p); }+int vkc_mla_lnorm(VkcBuf *x, VkcBuf *prm, VkcBuf *y, const VkcMlaRow *p) { K.kind = PK_NORM; return mla_rowop(2, x, prm, y, p); }++int vkc_mla_hgemv(ColiVkTensor *t, VkcBuf *x, VkcBuf *y, VkcBuf *gate, const VkcHgemv *p) {+ K.kind = PK_MLAW;+ ColiVkTensorInfo ti;+ if (!K.mla_ok || !open_frame() || K.lost || !coli_vk_tensor_info(t, &ti) || p->S < 1 || p->S > 65535 ||+ p->H < 1 || p->H > 65535 || p->n < 1 || (p->trans && p->n > 1024)) return 0;+ int f = ti.fmt;+ if (!(f == 1 || f == 2 || f == 4 || f == 5 || f == 7 || f == 10 || f == 11 || f == 12 || f == 13)) return 0;+ if ((f == 4 || f == 7 || f == 12 || f == 13) && ti.gs < 1) return 0;+ if ((int64_t)(p->H - 1) * p->hstride + p->hoff + p->n > (int64_t)ti.O) return 0; /* the rows exist */+ struct { int mode, fmt, I, rowWords, gs, S, H, n, hstride, hoff, x_off, x_row, x_seg, y_off, y_row, y_seg,+ g_off, g_row, has_gate; } pc = {p->trans, f, ti.I, ti.rowWords, ti.gs, p->S, p->H, p->n, p->hstride, p->hoff,+ p->x_off, p->x_row, p->x_seg, p->y_off, p->y_row, p->y_seg, p->g_off, p->g_row, gate && p->has_gate};+ VkcBind bd[5] = {B(x, 0), {(VkBuffer)ti.wbuf, 0, 0, 0}, {(VkBuffer)ti.sbuf, 0, 0, 0}, B(y, 1), B(pc.has_gate ? gate : NULL, 0)};+ if (p->trans) return record(K.mpipe[PM_HGEMV], bd, 5, &pc, sizeof pc, (uint32_t)p->H, (uint32_t)p->S, 1);+ /* rows: a subgroup each, grid-stride (at least four subgroups a workgroup) */+ int64_t rows = (int64_t)p->H * p->n, wg = (rows + 7) / 8;+ if (wg > 4096) wg = 4096;+ return record(K.mpipe[PM_HGEMV], bd, 5, &pc, sizeof pc, (uint32_t)wg, (uint32_t)p->S, 1);+}++int vkc_dsa_select(VkcBuf *iq, VkcBuf *hw, VkcBuf *keys, VkcBuf *sc, VkcBuf *sel, const VkcDsa *p) {+ K.kind = PK_DSA;+ if (!K.mla_ok || p->S < 1 || p->S > 65535 || p->IH < 1 || p->IH > 64 || p->ID < 1 || p->IH * p->ID > 4096 ||+ p->topk < 1 || p->pos_base < 0 || p->sc_row < p->pos_base + p->S ||+ p->sel_row < 1 + (p->topk < p->pos_base + p->S ? p->topk : p->pos_base + p->S)) return 0;+ struct { int mode; VkcDsa b; } pc = {0, *p};+ VkcBind bd[5] = {B(iq, 0), B(hw, 0), B(keys, 0), B(sc, 1), B(sel, 1)};+ return record(K.mpipe[PM_DSA], bd, 5, &pc, sizeof pc, (uint32_t)p->S, 1, 1);+}++/* k-pooling (chain_dsa.comp modes 1 and 2) */+int vkc_dsa_pool_keys(VkcBuf *keys, VkcBuf *gates, VkcBuf *prm, VkcBuf *pk, const VkcDsaPool *p) {+ K.kind = PK_DSA;+ if (!K.mla_ok || p->np < 0 || p->pool < 1 || p->p0 < 0 || p->ID < 1) return 0;+ if (p->np == 0) return open_frame() && !K.lost;+ struct { int mode; VkcDsaPool b; } pc = {1, *p};+ VkcBind bd[8] = {B(NULL, 0), B(NULL, 0), B(keys, 0), B(NULL, 0), B(NULL, 0), B(pk, 1), B(gates, 0), B(prm, 0)};+ return record(K.mpipe[PM_DSA], bd, 8, &pc, sizeof pc, (uint32_t)p->np, 1, 1);+}+int vkc_dsa_pool_select(VkcBuf *iq, VkcBuf *hw, VkcBuf *pk, VkcBuf *sc, VkcBuf *sel, const VkcDsaPick *p) {+ K.kind = PK_DSA;+ if (!K.mla_ok || p->S < 1 || p->S > 65535 || p->IH < 1 || p->IH > 64 || p->ID < 1 || p->IH * p->ID > 4096 ||+ p->pool < 1 || p->topk < p->pool || p->topk % p->pool || p->topk / p->pool > 1024 || p->pos_base < 0 ||+ p->sc_row < (p->pos_base + p->S) / p->pool || p->sel_row < 1 + p->topk + (p->tail ? p->pool - 1 : 0)) return 0;+ struct { int mode; VkcDsaPick b; } pc = {2, *p};+ VkcBind bd[6] = {B(iq, 0), B(hw, 0), B(NULL, 0), B(sc, 1), B(sel, 1), B(pk, 0)};+ return record(K.mpipe[PM_DSA], bd, 6, &pc, sizeof pc, (uint32_t)p->S, 1, 1);+}++/* ---- Kimi Delta Attention (chain_kda.comp) ------------------------------------------ */+int vkc_kda_ready(void) { return vkc_ready() && K.mpipe[PM_KDA]; }+int vkc_kda_conv(VkcBuf *in, VkcBuf *w, VkcBuf *win, VkcBuf *out, const VkcKdaConv *p) {+ K.kind = PK_KDA;+ if (!K.mpipe[PM_KDA] || p->K < 1 || p->K > 8 || p->P < 1 || p->C < 1 || p->S < 1) return 0;+ struct { int mode; VkcKdaConv b; } pc = {0, *p};+ VkcBind bd[4] = {B(in, 0), B(w, 0), B(win, 1), B(out, 1)};+ uint32_t gx, gy; grid(((uint64_t)p->C + 127) / 128, &gx, &gy);+ return record(K.mpipe[PM_KDA], bd, 4, &pc, sizeof pc, gx, gy, 1);+}+static VkPipeline kda_pipe(int KD) {+ for (int i = 0; i < K.nkdarec; i++) if (K.kdarec_kd[i] == KD) return K.kdarec[i];+ if (K.nkdarec == VKC_KDA_MAX || !K.mmod[PM_KDA]) return VK_NULL_HANDLE;+ int32_t kd = KD;+ VkSpecializationMapEntry me = {0, 0, 4};+ VkSpecializationInfo si = {1, &me, 4, &kd};+ VkPipeline p = make_pipe(K.mmod[PM_KDA], &si);+ if (!p) return VK_NULL_HANDLE;+ K.kdarec[K.nkdarec] = p; K.kdarec_kd[K.nkdarec++] = KD;+ return p;+}+int vkc_kda_rec(int KD, VkcBuf *m, VkcBuf *f, VkcBuf *b, VkcBuf *g, VkcBuf *prm, VkcBuf *st, VkcBuf *y, const VkcKdaRec *p) {+ K.kind = PK_KDA;+ if (!K.mpipe[PM_KDA] || KD < 1 || KD > 256 || p->VD < 1 || p->VD > 128 || p->H < 1 || p->S < 1) return 0;+ VkPipeline pipe = kda_pipe(KD);+ if (!pipe) return 0;+ struct { int mode; VkcKdaRec b; } pc = {1, *p};+ VkcBind bd[7] = {B(m, 0), B(prm, 0), B(st, 1), B(y, 1), B(f, 0), B(b, 0), B(g, 0)};+ return record(pipe, bd, 7, &pc, sizeof pc, (uint32_t)p->H, 1, 1);+}++/* ---- manifold-constrained hyper-connections (chain_mhc.comp) ------------------------ */+int vkc_mhc_ready(void) { return vkc_ready() && K.mpipe[PM_MHC]; }+int vkc_mhc(int mode, VkcBuf *x, VkcBuf *m, VkcBuf *hp, VkcBuf *prm, VkcBuf *y, const VkcMhc *p) {+ K.kind = PK_MHC;+ if (!K.mpipe[PM_MHC] || mode < 0 || mode > 4 || p->S < 0 || (mode != 4 && (p->H < 1 || p->H > 8 || p->D < 1))) return 0;+ if (mode == 0 && p->iters < 1) return 0;+ struct { int mode; VkcMhc b; } pc = {mode, *p};+ VkcBind bd[5] = {B(x, 0), B(m, 0), B(hp, mode == 0), B(prm, 0), B(y, mode != 0)};+ uint64_t n = mode == 4 ? (uint64_t)p->n : mode == 2 ? (uint64_t)p->S * p->H * p->D : (uint64_t)p->S * p->D;+ if (mode == 0) return p->S == 0 || record(K.mpipe[PM_MHC], bd, 5, &pc, sizeof pc, (uint32_t)p->S, 1, 1);+ uint32_t gx, gy; grid((n + 255) / 256, &gx, &gy);+ return n == 0 || record(K.mpipe[PM_MHC], bd, 5, &pc, sizeof pc, gx, gy, 1);+}++int vkc_mla_scratch(VkcMlaScratch *s, const VkcMla *m, int rows) {+ if (s->rows >= rows && s->q) return 1;+ size_t r = (size_t)rows, H = (size_t)m->H;+ int ok = vkc_reserve(&s->qa, (m->q_lora > 0 ? r * m->q_lora : 1) * 4, VKC_DEV) &&+ vkc_reserve(&s->q, r * H * (m->Q + m->R) * 4, VKC_DEV) &&+ vkc_reserve(&s->kv, r * (m->K + m->R) * 4, VKC_DEV) &&+ vkc_reserve(&s->qabs, r * H * m->K * 4, VKC_DEV) &&+ vkc_reserve(&s->clat, r * H * m->K * 4, VKC_DEV) &&+ vkc_reserve(&s->ctx, r * H * m->V * 4, VKC_DEV);+ s->rows = ok ? rows : 0;+ return ok;+}+void vkc_mla_scratch_free(VkcMlaScratch *s) {+ vkc_free(s->qa); vkc_free(s->q); vkc_free(s->kv); vkc_free(s->qabs); vkc_free(s->clat); vkc_free(s->ctx);+ memset(s, 0, sizeof *s);+}+/* the tensor is there and has the shape [O x I] */+static int mla_shape(ColiVkTensor *t, int O, int I) {+ ColiVkTensorInfo ti;+ return t && coli_vk_tensor_info(t, &ti) && ti.I == I && ti.O == O;+}+int vkc_mla_qkv(const VkcMla *m, VkcMlaScratch *s, VkcBuf *x, size_t x_off, int S, int pos_base,+ VkcBuf *cs, VkcMlaCache *c, VkcBuf *down, size_t down_off) {+ int H = m->H, QR = m->Q + m->R, KR = m->K + m->R;+ if (!K.mla_ok || S < 1 || S > s->rows || pos_base < 0 || (int64_t)pos_base + S > c->cap || m->R < 0 ||+ (m->R & 1) || m->R > 128 || m->K < 1 || m->K > 1024 || (m->R > 0 && (!cs || !c->rope)) ||+ !mla_shape(m->q_b, H * QR, m->q_lora > 0 ? m->q_lora : m->D) || !mla_shape(m->kv_a, KR, m->D) ||+ (m->q_lora > 0 && !mla_shape(m->q_a, m->q_lora, m->D))) return 0;+ int ok;+ if (m->q_lora > 0) {+ VkcNorm qn = {S, m->q_lora, 1, 0, m->q_lora, m->q_lora, 0, m->q_lora, m->q_lora, (int)m->q_norm, 0, 0, m->eps, 1.f};+ ok = vkc_matmul(m->q_a, x, x_off, s->qa, 0, S) && vkc_norm(s->qa, m->prm, s->qa, &qn) &&+ vkc_matmul(m->q_b, s->qa, 0, s->q, 0, S);+ } else ok = vkc_matmul(m->q_b, x, x_off, s->q, 0, S);+ ok = ok && vkc_matmul(m->kv_a, x, x_off, s->kv, 0, S);+ VkcNorm ln = {S, m->K, 1, 0, KR, KR, pos_base * m->K, m->K, m->K, (int)m->kv_norm, 0, 0, m->eps, 1.f};+ ok = ok && vkc_norm(s->kv, m->prm, c->lat, &ln);+ if (ok && m->R > 0) {+ VkcMlaRow rq = {S * H, H, m->R, m->R, m->rope_style, m->Q, H * QR, QR, m->Q, H * QR, QR, 0, m->R, 0, 0, 0, 0.f};+ VkcMlaRow rk = {S, 1, m->R, m->R, m->rope_style, m->K, KR, 0, pos_base * m->R, m->R, 0, 0, m->R, 0, 0, 0, 0.f};+ ok = vkc_mla_rope(s->q, cs, s->q, &rq) && vkc_mla_rope(s->kv, cs, c->rope, &rk);+ }+ if (ok && down)+ ok = vkc_copy(down, down_off, c->lat, (size_t)pos_base * m->K, (size_t)S * m->K) &&+ (m->R == 0 || vkc_copy(down, down_off + (size_t)S * m->K, c->rope, (size_t)pos_base * m->R, (size_t)S * m->R));+ return ok;+}+int vkc_mla_attn(const VkcMla *m, VkcMlaScratch *s, int S, int pos_base, int kv_start, VkcMlaCache *c,+ VkcBuf *sel, size_t sel_off, int sel_row, VkcBuf *gate, size_t gate_off, VkcBuf *out, size_t out_off) {+ int H = m->H, QR = m->Q + m->R, HK = H * m->K, HV = H * m->V;+ if (!K.mla_ok || S < 1 || S > s->rows || (int64_t)pos_base + S > c->cap) return 0;+ if (m->kv_b ? !mla_shape(m->kv_b, H * (m->Q + m->V), m->K)+ : !(mla_shape(m->k_abs, HK, m->Q) && mla_shape(m->v_abs, HV, m->K))) return 0;+ if (m->o && out && !mla_shape(m->o, m->D, HV)) return 0;+ int ok;+ if (m->kv_b) { /* qa[s][h][i] = sum_d kv_b[h*(Q+V) + d][i] q[s][h][d], d < Q */+ VkcHgemv a = {1, S, H, m->Q, m->Q + m->V, 0, 0, H * QR, QR, 0, HK, m->K, 0, 0, 0};+ ok = vkc_mla_hgemv(m->kv_b, s->q, s->qabs, NULL, &a);+ } else { /* qa[s][h][i] = k_abs[h*K + i] . q[s][h][0..Q) */+ VkcHgemv a = {0, S, H, m->K, m->K, 0, 0, H * QR, QR, 0, HK, m->K, 0, 0, 0};+ ok = vkc_mla_hgemv(m->k_abs, s->q, s->qabs, NULL, &a);+ }+ VkcMlaCore cp = {S, H, m->K, m->R, pos_base, kv_start, 0, HK, m->K, m->Q, H * QR, QR, 0, m->K, 0, m->R,+ (int)sel_off, sel ? sel_row : 0, 0, HK, m->K, m->scale};+ ok = ok && vkc_mla_core(s->qabs, s->q, c->lat, c->rope, sel, s->clat, &cp);+ VkcHgemv v = {0, S, H, m->V, m->kv_b ? m->Q + m->V : m->V, m->kv_b ? m->Q : 0, 0, HK, m->K, 0, HV, m->V,+ (int)gate_off, HV, gate != NULL};+ ok = ok && vkc_mla_hgemv(m->kv_b ? m->kv_b : m->v_abs, s->clat, s->ctx, gate, &v);+ if (ok && m->o && out) ok = vkc_matmul(m->o, s->ctx, 0, out, out_off, S);+ return ok;+}+int vkc_mla(const VkcMla *m, VkcMlaScratch *s, VkcBuf *x, size_t x_off, int S, int pos_base, int kv_start,+ VkcBuf *cs, VkcMlaCache *c, VkcBuf *out, size_t out_off) {+ return vkc_mla_qkv(m, s, x, x_off, S, pos_base, cs, c, NULL, 0) &&+ vkc_mla_attn(m, s, S, pos_base, kv_start, c, NULL, 0, 0, NULL, 0, out, out_off);+}+ /* ---- inkling's ops: chain_sconv.comp and chain_relattn.comp, made on first use ---- */+ static struct { VkShaderModule mod[2]; VkPipeline pipe[2]; int tried[2]; } KX;+ static VkPipeline kx_pipe(int k) {+ static const char *const file[2] = {"chain_sconv.spv", "chain_relattn.spv"};+ if (KX.pipe[k] || KX.tried[k] || !vkc_ready()) return KX.pipe[k];+ KX.tried[k] = 1;+ if ((KX.mod[k] = load_module(K.core.spv_path, file[k]))) KX.pipe[k] = make_pipe(KX.mod[k], NULL);+ return KX.pipe[k];+ }+ static void kx_shutdown(void) {+ for (int k = 0; k < 2; k++) {+ if (KX.pipe[k]) vkDestroyPipeline(K.dev, KX.pipe[k], NULL);+ if (KX.mod[k]) vkDestroyShaderModule(K.dev, KX.mod[k], NULL);+ }+ memset(&KX, 0, sizeof KX);+ }+ int vkc_sconv_ready(void) { return kx_pipe(0) != VK_NULL_HANDLE; }+ int vkc_relattn_ready(void) { return kx_pipe(1) != VK_NULL_HANDLE; }+ int vkc_sconv(VkcBuf *x, VkcBuf *w, VkcBuf *ring, const VkcSconv *p) {+ VkPipeline pipe = kx_pipe(0);+ if (!pipe || (p->mode == 0 && (p->CK < 1 || p->CK > 9))) return 0;+ K.kind = p->mode == 0 ? PK_DNCONV : PK_EW;+ uint32_t gx, gy;+ if (p->mode == 0) {+ VkcBind bd[3] = {B(x, 1), B(w, 0), B(ring, 1)};+ grid(((uint64_t)p->C + 63) / 64, &gx, &gy);+ return record(pipe, bd, 3, p, sizeof *p, gx, gy, 1);+ }+ VkcBind bd[1] = {B(x, 1)};+ grid(((uint64_t)p->n + 63) / 64, &gx, &gy);+ return record(pipe, bd, 1, p, sizeof *p, gx, gy, 1);+ }+ int vkc_relattn(VkcBuf *q, VkcBuf *kc, VkcBuf *vc, VkcBuf *o, VkcBuf *kvs, VkcBuf *r, VkcBuf *relp, VkcBuf *tau,+ const VkcRelAttn *p) {+ VkPipeline pipe = kx_pipe(1);+ if (!pipe || p->hd > 256 || p->d_rel > 64 || p->d_rel < 0 || p->KVH < 1 || p->H % p->KVH || p->cap < 1) return 0;+ K.kind = PK_ATTN;+ VkcBind bd[8] = {B(q, 0), B(kc, 0), B(vc, 0), B(o, 1), B(kvs, 0), B(r, 0), B(relp, 0), B(tau, 0)};+ return record(pipe, bd, 8, p, sizeof *p, (uint32_t)p->H, (uint32_t)p->S, 1);+ }void vkc_stats(VkcStats *st) { *st = K.st; }void vkc_prof_print(void) {diff --cc c/vk_chain.hindex 784b3d2b,d8d730b0..9509dacf--- a/c/vk_chain.h+++ b/c/vk_chain.h@@@ -123,168 -129,21 +133,183 @@@ typedef struct { int mode, S, C, H, CKint vkc_ple(VkcBuf *keys, VkcBuf *hyp, VkcBuf *val, VkcBuf *prm, VkcBuf *gated, VkcBuf *normv,VkcBuf *conv, VkcBuf *ring, const VkcPle *p);+/* ---- multi-head latent attention (MLA) -------------------------------------------+ * The attention of GLM-5.2, GLM-5.3, DeepSeek V3-style and Kimi K3's MLA layers, for S+ * rows (one at decode, a prompt chunk at prefill), with the KV cache on the device. Per+ * head h the query has Q no-position floats and R rotated ones; the cache holds, per+ * position, the normalized latent (K floats, kv_lora) and the rotated shared key (R).+ * Weight absorption: the no-position query enters the latent space once per head+ * (qa = W_k^T q_nope), the scores are qa . latent + q_rot . rope_key, and the value+ * rows apply once to the softmax-weighted latent (ctx = W_v clat). Nothing assumes a+ * model's shapes: H, Q, R (0 = NoPE), V, K, q_lora (0 = no q latent), the RoPE style,+ * the softmax scale (YaRN's mscale^2 included by the caller) and the cos/sin table+ * (YaRN's frequencies and mscale included by the caller) are all the caller's.+ *+ * Two levels. The ops (shaders/chain_mla.comp, chain_hgemv.comp, chain_dsa.comp):+ * vkc_mla_core the attention core over the cache: scores, online softmax, clat;+ * the causal range or a selection list (a DSA indexer's)+ * vkc_mla_hgemv per-head block matmuls of a resident tensor (the absorbed query+ * from kv_b's key rows, transposed, or from a [H*K x Q] key matrix;+ * the value rows; an optional sigmoid gate on the values)+ * vkc_mla_rope RoPE from a host cos/sin table, rotate-half or interleaved pairs in+ * and halves out, in place or into a cache row+ * vkc_mla_lnorm LayerNorm with weight and bias (the indexer's key norm)+ * vkc_dsa_select a token-level DSA indexer's scores and top-k, in the CPU's order+ * and the layer op, vkc_mla_qkv + vkc_mla_attn (or vkc_mla for both): q_a, its norm,+ * q_b (or q straight from the hidden rows), kv_a, the latent norm, RoPE, the new rows+ * into the cache (and a copy for the host's), then the absorbed core, the values, the+ * gate and o_proj. Between the two an engine records what reads the projections (a+ * DSA indexer reads the normalized q latent, s->qa).+ * The cache is the caller's (VkcMlaCache): positions [kv_start, pos_base) must be there+ * before a step from pos_base (the engine keeps a watermark, as for the GQA caches);+ * the step writes [pos_base, pos_base + S).+ * Limits: K <= 1024, R <= 128 and even, Q <= 1024 for kv_b's transposed absorption,+ * IH <= 64 and IH*ID <= 4096 for the indexer. The ops return 0 (nothing recorded) when+ * the MLA shaders are missing or a limit is passed; vkc_mla_ready() says whether the+ * shaders are there. */+int vkc_mla_ready(void);++/* chain_mla.comp mode 0: clat[s][h] (o_off + s*o_row + h*o_seg) from the absorbed query+ * qabs (qa_off + s*qa_row + h*qa_seg, K floats) and the rotated query (qr_off + ..., R),+ * over lat (lat_off + t*lat_row) and rope (rope_off + t*rope_row) at the positions+ * kv_start..pos_base+s or sel's list (sel_row 0: none; sel[sel_off + s*sel_row] = count,+ * -1 the causal range, then the positions, a negative one skipped). rope may be NULL with R 0. */+typedef struct { int S, H, K, R, pos_base, kv_start, qa_off, qa_row, qa_seg, qr_off, qr_row, qr_seg,+ lat_off, lat_row, rope_off, rope_row, sel_off, sel_row, o_off, o_row, o_seg; float scale; } VkcMlaCore;+int vkc_mla_core(VkcBuf *qabs, VkcBuf *qr, VkcBuf *lat, VkcBuf *rope, VkcBuf *sel, VkcBuf *clat, const VkcMlaCore *p);+/* chain_mla.comp modes 1 and 2: nseg segments, segment g at row g / per_row, index+ * g % per_row: x_off + row*x_row + j*x_seg (y likewise; y may be x).+ * RoPE: the first rd floats rotate with the cos/sin pairs at cs_off + row*cs_row, the+ * rest of seg_len is copied when y is elsewhere. LayerNorm: seg_len floats, weight at+ * w_off and bias at b_off (has_b) in prm. */+#define VKC_ROPE_HALF 0 /* a = x[i], b = x[i + rd/2] */+#define VKC_ROPE_INTERLEAVED 1 /* a = x[2i], b = x[2i + 1], written to i and i + rd/2 */+typedef struct { int nseg, per_row, rd, seg_len, style, x_off, x_row, x_seg, y_off, y_row, y_seg,+ cs_off, cs_row, w_off, b_off, has_b; float eps; } VkcMlaRow;+int vkc_mla_rope(VkcBuf *x, VkcBuf *cs, VkcBuf *y, const VkcMlaRow *p);+int vkc_mla_lnorm(VkcBuf *x, VkcBuf *prm, VkcBuf *y, const VkcMlaRow *p);+/* chain_hgemv.comp: head h's block is t's rows h*hstride + hoff + [0, n).+ * trans 0: y[s][h][o] = W[row o] . x[s][h][0..I), o < n (has_gate: times+ * sigmoid(gate[g_off + s*g_row + h*n + o]));+ * trans 1: y[s][h][i] = sum_d W[row d][i] * x[s][h][d], d < n, i < I.+ * x[s][h] at x_off + s*x_row + h*x_seg, y[s][h] at y_off + s*y_row + h*y_seg. */+typedef struct { int trans, S, H, n, hstride, hoff, x_off, x_row, x_seg, y_off, y_row, y_seg, g_off, g_row, has_gate; } VkcHgemv;+int vkc_mla_hgemv(ColiVkTensor *t, VkcBuf *x, VkcBuf *y, VkcBuf *gate, const VkcHgemv *p);+/* chain_dsa.comp mode 0: row s (position pos_base + s) scores its pos_base+s+1 positions+ * as (sum_h [d_h > 0] w_h d_h) * wscale, d_h = (q_h . k_t) * qscale (q at q_off ++ * s*q_row, IH x ID; w at w_off + s*w_row; k_t at k_off + t*k_row), into sc[s*sc_row + t],+ * and writes sel[s*sel_row] = keep, then the positions: those above the keep-th largest+ * score in position order, then those equal to it in position order. With no more+ * positions than topk (and force 0), sel[s*sel_row] = -1: every position. */+typedef struct { int S, pos_base, IH, ID, topk, force, q_off, q_row, w_off, w_row, k_off, k_row, sc_row, sel_row;+ float qscale, wscale; } VkcDsa;+int vkc_dsa_select(VkcBuf *iq, VkcBuf *hw, VkcBuf *keys, VkcBuf *sc, VkcBuf *sel, const VkcDsa *p);++/* chain_dsa.comp with k-pooling (GLM-5.3): mode 1, the pooled keys of the np complete+ * pools from p0 on, each the per-channel softmax mixture (gate logits plus ape, at+ * ape_off in prm, [pool][ID]) of its members' keys, into pk[pk_off + p*ID]; mode 2, row+ * s's selection: the complete pools visible to position pos_base + s scored+ * sum_h [dot > 0] (w_h / wdiv) * dot * scale (dot = q_h . pk[p]), the top topk/pool in+ * rank order filling `pool` slots each, the incomplete tail from slot topk (tail), -1+ * elsewhere; sel[s*sel_row] = topk (+ pool - 1 with the tail). sc: [S][sc_row] scores. */+typedef struct { int np, pool, p0, ID, g_off, g_row, ape_off, pk_off, k_off, k_row; } VkcDsaPool;+int vkc_dsa_pool_keys(VkcBuf *keys, VkcBuf *gates, VkcBuf *prm, VkcBuf *pk, const VkcDsaPool *p);+typedef struct { int S, pos_base, IH, ID, topk, pool, q_off, q_row, w_off, w_row, pk_off, tail, sc_row, sel_row;+ float wdiv, scale; } VkcDsaPick;+int vkc_dsa_pool_select(VkcBuf *iq, VkcBuf *hw, VkcBuf *pk, VkcBuf *sc, VkcBuf *sel, const VkcDsaPick *p);++/* The layer op. Tensors are the resident copies the per-matrix path uploads+ * (coli_vk_tensor_ensure), in any format the backend holds. */+typedef struct {+ int H, Q, R, V, K; /* heads; per head qk_nope, qk_rope (0 = NoPE), v_head; kv_lora */+ int D, q_lora; /* hidden; q_lora 0: no q latent, q_b reads the hidden rows */+ float eps; /* the latent RMSNorms' eps */+ float scale; /* the softmax scale */+ int rope_style; /* VKC_ROPE_HALF or VKC_ROPE_INTERLEAVED */+ ColiVkTensor *q_a; /* [q_lora x D]; NULL with q_lora 0 */+ ColiVkTensor *q_b; /* [H*(Q+R) x (q_lora or D)] */+ ColiVkTensor *kv_a; /* [K+R x D]: the latent, then the shared key's rotated part */+ ColiVkTensor *kv_b; /* [H*(Q+V) x K], per head Q key rows then V value rows; or: */+ ColiVkTensor *k_abs, *v_abs; /* [H*K x Q] (W_k^T per head) and [H*V x K] */+ ColiVkTensor *o; /* [D x H*V]; NULL: the context stays in s->ctx */+ VkcBuf *prm; /* the norm weights: q latent at q_norm (unused with q_lora 0), */+ size_t q_norm, kv_norm; /* kv latent at kv_norm (floats) */+} VkcMla;+typedef struct { VkcBuf *qa, *q, *kv, *qabs, *clat, *ctx; int rows; } VkcMlaScratch;+typedef struct { VkcBuf *lat, *rope; int cap; } VkcMlaCache; /* [cap][K] and [cap][R] (rope NULL with R 0) */+/* scratch for `rows` rows (grows, never shrinks); free releases it */+int vkc_mla_scratch(VkcMlaScratch *s, const VkcMla *m, int rows);+void vkc_mla_scratch_free(VkcMlaScratch *s);+/* x: S rows of D floats at x_off (the layer's normalized input). cs: the cos/sin pairs+ * of the S positions, R floats a row (NULL with R 0). Leaves s->qa = the normalized q+ * latent [S][q_lora] and s->q = the queries [S][H*(Q+R)], rotated; writes the cache's+ * rows [pos_base, pos_base+S) and, when down is given, copies them to down at down_off+ * (S*K latent floats, then S*R rope floats) for the host's cache. */+int vkc_mla_qkv(const VkcMla *m, VkcMlaScratch *s, VkcBuf *x, size_t x_off, int S, int pos_base,+ VkcBuf *cs, VkcMlaCache *c, VkcBuf *down, size_t down_off);+/* The core over positions kv_start..pos_base+s (or sel's lists, sel_row > 0), the values+ * (times sigmoid(gate[gate_off + s*H*V + h*V + v]) when gate is given) into s->ctx, then+ * out[out_off + s*D] = o(ctx) when m->o and out are given. */+int vkc_mla_attn(const VkcMla *m, VkcMlaScratch *s, int S, int pos_base, int kv_start, VkcMlaCache *c,+ VkcBuf *sel, size_t sel_off, int sel_row, VkcBuf *gate, size_t gate_off, VkcBuf *out, size_t out_off);+/* both, the causal range */+int vkc_mla(const VkcMla *m, VkcMlaScratch *s, VkcBuf *x, size_t x_off, int S, int pos_base, int kv_start,+ VkcBuf *cs, VkcMlaCache *c, VkcBuf *out, size_t out_off);++/* ---- Kimi Delta Attention (chain_kda.comp) --------------------------------------------+ * delta_attention.h's coli_kda_step for S rows in order, the state and the short+ * convolution's window on the device (GLM-5.3's linear layers; Kimi K3's KDA).+ * vkc_kda_conv: C = 3P channels (q, k, v), the input of part c / P at in_off + part*in_part+ * + s*in_row + c % P, the window [C][K] at win_off (the last K inputs, oldest first, as+ * the CPU keeps it), taps [C][K] at w_off; out[out_off + s*out_row + c] = silu(sum).+ * vkc_kda_rec: one workgroup per head, the key dim KD a specialization (<= 256), VD <= 128;+ * m = the convolution's output rows (q at h*KD, k at P + h*KD, v at 2P + h*VD); f, b, g+ * the raw decay, beta and output-gate projections; prm at prm_off: A_log[H], dt[P],+ * norm[VD]; alpha = exp(lb * sigmoid(exp(A_log) * (f + dt))), beta = sigmoid(b), q and k+ * l2-normalized with neps inside the root; y = RMSNorm(o, eps) * norm * sigmoid(g).+ * The state [H][KD][VD] at st_off. */+int vkc_kda_ready(void);+typedef struct { int S, C, K, P, in_off, in_row, in_part, out_off, out_row, w_off, win_off; } VkcKdaConv;+int vkc_kda_conv(VkcBuf *in, VkcBuf *w, VkcBuf *win, VkcBuf *out, const VkcKdaConv *p);+typedef struct { int S, H, VD, P, m_off, m_row, f_off, f_row, b_off, b_row, g_off, g_row, y_off, y_row, st_off, prm_off;+ float lb, neps, eps; } VkcKdaRec;+int vkc_kda_rec(int KD, VkcBuf *m, VkcBuf *f, VkcBuf *b, VkcBuf *g, VkcBuf *prm, VkcBuf *st, VkcBuf *y, const VkcKdaRec *p);++/* ---- manifold-constrained hyper-connections (chain_mhc.comp) ------------------------+ * hyper_connections.h for S rows of H <= 8 streams of D floats ([S][H*D] at x_off,+ * x_row apart), GLM-5.3's (and DeepSeek V4's) residual:+ * VKC_MHC_SPLIT m: the raw mix products hc_fn . x ([S][(2+H)*H], from a matmul),+ * scaled by the rows' 1/rms (eps); pre, post and the Sinkhorn-projected+ * comb (iters, hc_eps; scale[3] and base at prm_off in prm) into+ * hp[hp_off + s*hp_row]: H pre, H post, H*H comb+ * VKC_MHC_COLLAPSE y[s][d] = sum_i pre[i] x[s][i*D + d]+ * VKC_MHC_POST y[s][j*D + d] = sum_i comb[i*H + j] x[s][i*D + d] + post[j] m[s][d]+ * VKC_MHC_MEAN y[s][d] = (sum_i x[s][i*D + d]) / H+ * VKC_SWIGLU_CLAMP y[i] = silu(min(x[i], lim)) * clamp(m[i], -lim, lim), i < n */+#define VKC_MHC_SPLIT 0+#define VKC_MHC_COLLAPSE 1+#define VKC_MHC_POST 2+#define VKC_MHC_MEAN 3+#define VKC_SWIGLU_CLAMP 4+int vkc_mhc_ready(void);+typedef struct { int S, H, D, iters, x_off, x_row, m_off, m_row, hp_off, hp_row, y_off, y_row, prm_off, n;+ float eps, hc_eps, lim; } VkcMhc;+int vkc_mhc(int mode, VkcBuf *x, VkcBuf *m, VkcBuf *hp, VkcBuf *prm, VkcBuf *y, const VkcMhc *p);+ /* Inkling's ops, their pipelines made on first use (an engine checks *_ready at setup:+ * a build without the shader keeps every other op, and that engine's chain off).+ * chain_sconv.comp (mode 0: the depthwise causal short convolution, residual inside, in+ * place, its ring carried; mode 1: x *= fc over n floats; mode 2: x /= fc) */+ typedef struct { int mode, S, C, CK, x_off, x_row, w_off, ring_off, n; float fc; } VkcSconv;+ int vkc_sconv_ready(void);+ int vkc_sconv(VkcBuf *x, VkcBuf *w, VkcBuf *ring, const VkcSconv *p);+ /* chain_relattn.comp: attention with a relative-position bias bank, a per-row scale+ * tau and a sliding window over a ring cache, the step's own rows read from kvs */+ typedef struct { int S, H, KVH, hd, pos_base, cap, window, ext, d_rel;+ int q_off, q_row, o_off, o_row, k_off, v_off, ks_off, vs_off, kv_row;+ int r_off, r_row, relp_off, tau_off; float scale; } VkcRelAttn;+ int vkc_relattn_ready(void);+ int vkc_relattn(VkcBuf *q, VkcBuf *kc, VkcBuf *vc, VkcBuf *o, VkcBuf *kvs, VkcBuf *r, VkcBuf *relp, VkcBuf *tau,+ const VkcRelAttn *p);/* counters, for the engines' [VK] lines */typedef struct {diff --cc docs/ENVIRONMENT.mdindex 3178dfc3,b2b584ef..9b9865db--- a/docs/ENVIRONMENT.md+++ b/docs/ENVIRONMENT.md@@@ -245,13 -245,12 +245,13 @@@ With `COLI_VULKAN=1` every MoE engine (| `COLI_VK_TIER_GEMM_ROWS` | `16` | Rows from which an expert of a step takes the tiled GEMM (prefill) instead of the per-row GEMV; `0` never. Below it a row's bits do not depend on how many rows share the dispatch. || `COLI_VK_TIER_QUEUE` | a second queue | `0`: the tier's batches share the main queue with the dense matmuls (they then serialize). Unset: a second queue of the main family, else a compute-only family's (RADV), else shared (Lavapipe). || `COLI_VK_DENSE` | on; off on a device sharing the CPU's RAM while the tier is on | Where the dense (non-expert) matrices of qwen36, qwen38, inkling, olmoe, kimi_k3 (its shared experts), mimo, deepseek_v41, deepseek_v4, glm53 and qwenimage run. `0`: on the CPU (with the tier on, the device takes the routed experts only); `1`: on the device whatever the device. Unset: on the device, except on an integrated GPU or a CPU device (Lavapipe) while the engine runs the expert tier (every one of them but qwenimage, which has no routed experts), where they stay on the CPU: there the dense matmuls cost more than the tier gains (measured on a Radeon 780M). The `[VK] <engine>: device ready, dense matrices on the ...` line says which and why. The GLM engine reads it through the same rule with its own default, off (see above). |- | `COLI_VK_CHAIN` | on for a discrete GPU; qwen36 on, qwen38, colibri and glm53 off on an integrated GPU with the tier; off on a CPU device | qwen36, qwen38, colibri (GLM-5.2) and glm53 (GLM-5.3 Flash): every layer's dense chain recorded into one submission per layer, the residual stream and the recurrent and KV state on the device (see [vulkan.md](vulkan.md#the-dense-chain-vk_chainc)). `0` off, `1` on, `2` on for prompts only (decode on the CPU). The `[VK] <engine>: dense chain ...` line says which and why. |-| `COLI_VK_CHAIN` | on for a discrete GPU; qwen36 and olmoe on, qwen38 off, mimo and inkling off (not measured) on an integrated GPU with the tier; off on a CPU device | qwen36, qwen38, olmoe, inkling and mimo: every layer's dense chain recorded into one submission per layer, the residual stream and the recurrent and KV state on the device (see [vulkan.md](vulkan.md#the-dense-chain-vk_chainc)). `0` off, `1` on, `2` on for prompts only (decode on the CPU). The `[VK] <engine>: dense chain ...` line says which and why. |++| `COLI_VK_CHAIN` | on for a discrete GPU; qwen36 and olmoe on, qwen38 off, and mimo, inkling, colibri and glm53 off (not measured) on an integrated GPU with the tier; off on a CPU device | qwen36, qwen38, olmoe, inkling, mimo, colibri (GLM-5.2) and glm53 (GLM-5.3 Flash): every layer's dense chain recorded into one submission per layer, the residual stream and the recurrent and KV state on the device (see [vulkan.md](vulkan.md#the-dense-chain-vk_chainc)). `0` off, `1` on, `2` on for prompts only (decode on the CPU). The `[VK] <engine>: dense chain ...` line says which and why. || `COLI_VK_CHAIN_ROWS` | `512` | Prompt rows per chunk of the chain's prefill. || `COLI_VK_CHAIN_GEMV` | on | `0`: the chain's decode matrices take `qmatmul.comp`'s GEMV instead of `chain_gemv.comp`'s. || `COLI_VK_CHAIN_SPIN_US` | `2000` | How long a wait on a chain frame polls before blocking. || `COLI_VK_CHAIN_PROF` | off | `1`: device time per kind of chain op, one line per run. |- | `COLI_VK_CHAIN_FAULT` | off | Tests: the n-th chain frame fails as a lost device would; the engine rebuilds the state on the CPU (colibri has none to rebuild) and runs there. |-| `COLI_VK_CHAIN_FAULT` | off | Tests: the n-th chain frame fails as a lost device would; the engine rebuilds the state on the CPU (mimo has none to rebuild: the CPU continues from its host caches). |++| `COLI_VK_CHAIN_FAULT` | off | Tests: the n-th chain frame fails as a lost device would; the engine rebuilds the state on the CPU and runs there (colibri and mimo have none to rebuild: the CPU continues from its host caches). |+| `DUMP` | unset | colibri and glm53 in a `VK=1` build: every logits row the forward computes, appended to this file as raw f32 (the Vulkan gates compare the device's with the CPU's). |### Second Vulkan device (opt-in)diff --cc docs/vulkan.mdindex 8dda85a9,41830dd3..55ac4659--- a/docs/vulkan.md+++ b/docs/vulkan.md@@@ -673,8 -673,7 +673,9 @@@ command buffer instead, and keeps the rto the next. qwen36 (Qwen3.6, Qwen3-Coder, Qwen3.8-27B) and qwen38 (Qwen3.8 FlashNext) run it: by default on a discrete GPU, and for qwen36 on an integrated one withthe expert tier (see [the default](#the-chain-on-a-radeon-780m)); `COLI_VK_CHAIN=1`- anywhere. colibri (GLM-5.2) and glm53 (GLM-5.3 Flash) run it too, with the MLA,+ anywhere. olmoe and inkling run it as well ([OLMoE and Inkling](#olmoe-and-inkling)).++colibri (GLM-5.2) and glm53 (GLM-5.3 Flash) run it too, with the MLA,+KDA and hyper-connection ops ([below](#glm-52-and-glm-53-flash-on-the-chain)).**What runs where, per layer** (S rows: one at decode, a prompt chunk at prefill):@@@ -741,7 -740,7 +742,7 @@@ f32 throughout, as the CPU's f32 path| Variable | Default | Effect ||---|---|---|- | `COLI_VK_CHAIN` | on for a discrete GPU; on an integrated GPU with the expert tier, what the engine measured (qwen36 on, qwen38 off; colibri and glm53 off: not measured); off on a CPU device | `1`: every layer's dense chain on the device; `2`: prompts only (forwards of more than two rows; decode and MTP verifies on the per-matrix path, the state moving between the two); `0`: the per-matrix path. The `[VK] <engine>: dense chain ...` line says which and why. |-| `COLI_VK_CHAIN` | on for a discrete GPU; on an integrated GPU with the expert tier, what the engine measured (qwen36 on, qwen38 off, olmoe on; inkling off, not measured); off on a CPU device | `1`: every layer's dense chain on the device; `2`: prompts only (forwards of more than two rows; decode and MTP verifies on the per-matrix path, the state moving between the two); `0`: the per-matrix path. The `[VK] <engine>: dense chain ...` line says which and why. |++| `COLI_VK_CHAIN` | on for a discrete GPU; on an integrated GPU with the expert tier, what the engine measured (qwen36 and olmoe on, qwen38 off; mimo, inkling, colibri and glm53 off: not measured); off on a CPU device | `1`: every layer's dense chain on the device; `2`: prompts only (forwards of more than two rows; decode and MTP verifies on the per-matrix path, the state moving between the two); `0`: the per-matrix path. The `[VK] <engine>: dense chain ...` line says which and why. || `COLI_VK_CHAIN_ROWS` | `512` | Prompt rows per chunk: a longer prompt runs every layer chunk by chunk (the device's scratch is sized for one chunk). || `COLI_VK_CHAIN_GEMV` | on | `0`: the decode matrices take `qmatmul.comp`'s GEMV instead of `chain_gemv.comp`'s. || `COLI_VK_CHAIN_SPIN_US` | `2000` | How long a wait on a chain frame polls the fence before blocking. |@@@ -827,66 -826,196 +828,256 @@@ prediction of it. Not timed either: conselects blocks instead of attending to all of them), serve sessions, Qwen3-Coder andQwen3.8-27B (no checkpoints on the box); their correctness is the Lavapipe gates'.+ ### MiMo-V2.6 on the chain++ `mimo_chain.h` runs MiMo-V2.6's layers (Flash and Pro) as frames on the device, as+ qwen36's chain does, with what this model has instead of a DeltaNet:++ | | mimo |+ |---|---|+ | device, one frame per MoE layer | the previous layer's routed sum joining the residual (the CPU's `h += out`); the input RMSNorm; the fused qkv projection in its dense form (`MIMO_DENSE_BITS`: the release's FP8 with its 128-column block scales, int8 rows or f32); partial RoPE on q and k (rotate-half on the first `rope_dim` dims, from a host table of the CPU's own `cosf`/`sinf`, one per layer kind: the two kinds have their own theta); the value scale (`chain_ew` SCALE); the new K/V rows into the device's cache; attention with the layer's sliding window and its sink logits (`chain_attn.comp` with a window, a ring and a sink); o_proj; the residual add; the post-attention RMSNorm. The dense layer (layer 0 on the release) runs its MLP there too and has no host step |+ | host | `moe()` as the CPU runs it, on the device's normalized rows: the router (sigmoid scores, the correction bias picks, top-k renormalized) and the routed experts (the tier's batch and the CPU's share, joined in each row's routing order). There is no shared expert, so nothing runs beside it |+ | last frame | the final norm and lm_head on the last row, every row for a read-out (`MIMO_LOGITS`, logprobs) |++ **The caches.** The host's stay canonical, in the layout the CPU keeps: a+ full-attention layer `[ctx][kvh][hd]` (V `[ctx][kvh][vd]`), a windowed layer a ring of+ `min(window, ctx)` rows with position `p` in row `p % rows`. The device mirrors each+ one in exactly that layout (`chain_attn.comp` reads position-major rows and a ring) behind+ a watermark per layer: positions below it (for a ring, the last `rows` of them) equal+ the host's. A step lowers it to its first position and uploads what the window still+ sees; a CPU step lowers it to its own; a restored photo (`pin_state_load` rewrites the+ host's rings) drops the rings' watermarks to 0, so the next step uploads them again.+ Prefix reuse needs nothing else.++ **A windowed layer's attention.** One decode row reads the ring in place: its window+ is exactly the ring's rows. A block of rows cannot (its later rows would overwrite slots+ its earlier rows still see), so it gathers the window's earlier rows from the ring and+ its own new rows into one position-major scratch, as the CPU's `attention()` copies+ them, attends there, and writes its last `rows` rows into the ring. Both visit the+ positions in the same order, so a row gets the same bits either way: with every matrix+ on the per-row GEMV (`COLI_VK_GEMM_MIN_S=0`) and the tier off, the logits of every+ position are the same bytes for a prompt in one block, one token at a time, and in+ blocks of 3, 8 (the fixture's window) and 9 rows.++ **A lost device.** MiMo has no recurrent state, and the new K/V rows reach the host's+ caches only when the step's last frame has completed: a device lost in the middle of a+ step loses nothing the host holds, and the CPU runs the step's remaining rows from where+ the caches end. Nothing is rebuilt, and a turn with a picture recovers too. The line is+ `[VK] mimo chain: the device was lost at position P; ...`.++ **What stays off the device.** The router's top-k and the routed experts the tier does+ not hold, the embedding gather, the vision tower (on the device only with+ `COLI_VK_DENSE=1`, through the per-matrix path), and runs with `MIMO_TRACE` (the CPU's+ per-layer dump), which the chain declines. MiMo has no MTP head. The chain also+ declines a head dim above 256 or a dense matrix that did not reach the device.++ **Correctness, on the tiny fixture** (6 layers: 4 windowed with a window of 8 and a sink+ logit, 2 full, partial RoPE with two thetas, a value scale of 0.707, a dense layer 0, a+ vision tower), on Lavapipe and on the Radeon 780M (`tests/vulkan_engines.sh+ mimo-chain`): Xiaomi's vendor oracle passes with the chain on (greedy and teacher-forced+ in every case, the native FP8/BF16 trunk, prefill in blocks of 3 and of 1, the picture);+ every configuration gives the CPU's tokens and every position's logits within 1e-4 of+ the largest one (measured over 42 configurations at most 2.0e-6 on Lavapipe and 2.7e-6+ on the 780M, both with the picture's tower on the device; 1.7e-6 without it); serve+ sessions give the CPU's frames (logprobs within 2.9e-4 on Lavapipe and 1.8e-4 on the+ 780M, on logits that reach 300); the prefix-reuse and photo tests pass bit for bit+ against a cold engine. With `COLI_VULKAN` unset, the tokens and the logits bytes of 36+ configurations (`MIMO_DENSE_BITS` 32, 0 and 8, blocks of 64, 3 and 1, four cases with+ the picture) are the previous build's; with `COLI_VULKAN=1` and `COLI_VK_CHAIN=0` or+ unset, those of 54 (the tier balanced deterministically, `COLI_VK_TIER_BALANCE=0`, since+ its balance moves bits from run to run on either build).++ **The default.** The chain's speed on a real MiMo model is not measured: no MiMo+ checkpoint is on the test box (the smallest, Flash, has 309B parameters). The chain is+ on for a discrete GPU (the common rule), off on an integrated GPU (`COLI_VK_CHAIN=1`+ turns it on) and on a CPU device:++ ```+ [VK] mimo: dense chain off (an integrated GPU with the expert tier: not measured; COLI_VK_CHAIN=0 off, 1 on, 2 prompts only)+ ```++ ### OLMoE and Inkling++ olmoe (`olmoe_chain.h`) and inkling (`inkling_chain.h`) run the chain too, with the same+ knobs, default rule and `[VK] <engine> chain: ...` line as qwen36.++ | | olmoe | inkling |+ |---|---|---|+ | device, frame A1 | the layer before's routed sum joining the residual; the input norm; q/k/v; OLMoE's q and k norms over the whole projection; RoPE from the CPU's table; the K/V rows into the device mirror; attention; o_proj; the residual add; the post-attention norm; the router logits | the layer before's MoE output joining the residual (the routed sum, then each shared expert times its combine weight, `moe()`'s order, then the MLP's short convolution and the add); the input norm; q/k/v and the bias projection r; the short convolutions on K and V with their rings; the per-head q/k norms; the attention (`chain_relattn.comp`); the K/V rows into the device ring; o_proj; the attention's short convolution; the residual add; the post-attention norm; the router logits (routed and shared). A dense-MLP layer runs its MLP, global scale, short convolution and add in the same frame |+ | host | softmax, top-k, the routed experts (the tier's batch and the CPU's share in rank order: the code `moe()` runs); the K/V rows into the host's cache; `PILOT`'s prefetch (below) | the sigmoid router with its bias, top-k, the joint combine weights, `TOPP`; the routed experts (`moe_ex`: `moe()`'s code, the device's logits handed in, the shared experts left out); the K/V rows into the host's cache |+ | device, frame A2 (not waited for) | none: OLMoE has no shared expert | the shared experts, unweighted |+ | last frame | the final norm and lm_head on the last row | the final norm, the division by the width multiplier, lm_head; the per-position heads (logprobs, teacher forcing) read the final rows on the host |++ Inkling's attention is not qwen36's: a learned relative-position bias (an r projection+ per row mixed through a per-layer bank, one bias per backward distance), the log-length+ scale tau, a sliding window on five layers of six over a ring of K/V rows, and short+ depthwise convolutions (residual inside) on K, V, the attention output and the MLP+ output. Two shaders do it, made on first use so that the qwen chains never depend on+ them: `chain_relattn.comp` (grouped attention with the bias bank mixed in the CPU's+ order, tau from a host table computed as the CPU computes it, the window, the ring, and+ the step's own rows read from its K/V scratch as `attention()` reads them, so a step+ that wraps the ring never loses a row an earlier query of it reads) and+ `chain_sconv.comp` (the convolution with its ring, in `sconv_apply`'s order, plus the+ scalar multiply and divide of the dense MLP's global scale and the logits' width+ multiplier). The shared experts join through `chain_ew.comp`'s `HC_APPLY` over one stream+ (y += w[row] * x), the CPU's `os[d] += w * hh[d]`. Nothing in the chain stages a row+ in shared memory past what the device allows: Inkling's hidden size (6144) and its+ 24576-wide dense MLP run through `chain_gemv.comp` or, past its staging, `qmatmul.comp`'s+ GEMV reading from the buffer; `tools/make_tiny_inkling.py --wide` makes a two-layer+ model at D = 6144 for the tests.++ **State.** Both keep the host's K/V cache canonical and copy each step's rows back.+ olmoe mirrors it behind qwen36's watermark. Inkling's mirror has the host's layout, a+ ring on sliding layers, and a range of positions per layer that the host wrote alone+ (a CPU step): the next chain step uploads the rows of its last `cap` positions first,+ whatever positions they now hold, so a rewind to a pinned snapshot over a wrapped ring+ reads on the device what the CPU reads. Inkling's four convolution states per layer run+ on the device and come back at the end of every chain step (a few KB a layer), so the+ host's copy is always current; a reset, a restored snapshot or a CPU step sends them+ up again before the next chain step.++ **Where they decline** (the per-matrix path, the state marked as the host's): inkling+ under CUDA or Metal; inkling's bf16 matrices on a CPU whose bf16 dot rounds the+ activations (AVX512-BF16), which the per-matrix path keeps on the CPU for the same+ reason (`[VK] inkling chain: ... bf16 stays on the CPU`); a geometry outside the shaders+ (head dim above 256, a bias bank wider than 64, more than 9 taps). olmoe runs `PILOT`:+ the rows after attention come down with the frame and the prefetch reads them, the+ rows after the MoE are those plus the routed sum (the float add the device makes).+ Running PILOT under the chain showed a race of the CPU path's own: the prefetcher could+ take the slot the forward pass was multiplying with as its LRU victim (at cap 1 the only+ slot) and read another expert into it mid-matmul. A slot now counts its readers, and+ neither eviction takes one being read. A device lost mid-step: olmoe redoes the step on+ the CPU (attention only, nothing to rebuild); inkling rebuilds its K/V and convolution+ states on the CPU from the prefix record, as qwen36 does.++ **OLMoE-1B-7B on the Radeon 780M** (`allenai/OLMoE-1B-7B-0924`, converted with+ `tools/convert_olmoe_merged.py`: int8 experts, f32 trunk; cap 64, `OMP_NUM_THREADS=8`,+ 1-min load under 2, no other engine running, the same binary for every arm, every tier+ arm from the same history of an unrelated prompt). Cold: the model files dropped from+ the page cache first (`posix_fadvise`, as the qwen bench does); warm: the run after.+ Decode is 100 steps after a 25-token prompt: the engine's time for 101 new tokens minus+ its time for 1 (the prefill alone), both from `TUNE decode`; prefill is a 512-token+ prompt with one new token. Where a warm cell lists two numbers, they are separate+ rounds.++ | Decode, 100 tokens | cold | warm |+ |---|---|---|+ | CPU | 22.2 tok/s | 23.1, 23.2 tok/s |+ | tier, trunk on the CPU | 12.6 tok/s | 12.8, 12.8 tok/s |+ | tier and chain (`COLI_VK_CHAIN=1`) | 16.8 tok/s | 17.3, 17.2 tok/s |+ | chain without the tier (`COLI_VK_TIER=0`) | 12.0 tok/s | 12.8 tok/s |++ | Prefill, 512 tokens | cold | warm |+ |---|---|---|+ | CPU | 11.7 s | 10.5, 10.6 s |+ | tier, trunk on the CPU | 6.3 s | 6.4, 6.4 s |+ | tier and chain | 5.5 s | 5.5, 5.5 s |+ | chain without the tier | 12.7 s | 11.4 s |++ What the numbers say:+ - **Against the tier alone the chain wins both**: decode 35% faster, the 512-token+ prompt 14% sooner. That is the comparison `coli_vk_chain_decide` makes on an+ integrated GPU with the tier on, so olmoe passes ON there.+ - **Against the CPU, decode loses**: 17.3 against 23.1 tok/s. OLMoE's trunk is f32+ (1.49 GB read every token, more than its eight routed experts' 0.8 GB of int8); the CPU+ reads it faster than the GPU does at the clock it mostly holds. Prefill wins: 5.5+ against 10.5 s. `COLI_VULKAN=1` is opt-in, and on this box the CPU alone is the faster+ way to decode OLMoE; with the device on, the chain is the better of its two modes.+ `COLI_VK_CHAIN_PROF=1` put 95% of the chain's device time in the trunk's f32 GEMVs,+ 34.8 ms a token (about 43 GB/s for its 1.49 GB); the GPU sat at its 800 MHz floor in+ 78% of the clock samples with the chain, 95% with the tier alone.+ - **Without the tier** the chain only moves the f32 trunk to the device and leaves every+ expert on the CPU: slower than the CPU in both. The default keeps it off there.+ - **The text.** Every arm, cold and warm, printed the same 100 decode tokens, and the+ same first token after the 512-token prompt.++ **Not measured**: a discrete GPU; OLMoE in serve sessions, past a 537-token context, with+ `PILOT` (its experts fit in RAM here) or with the dense trunk on the device beside the+ chain (`COLI_VK_DENSE=1`).++ **Inkling: not measured.** No Inkling checkpoint runs on the box (the model is 975B),+ so its integrated-GPU default is off (`COLI_VK_CHAIN_UNMEASURED`: the `[VK]` line says+ "not measured"); `COLI_VK_CHAIN=1` turns it on. Its correctness is the tiny fixtures',+ on Lavapipe and on the 780M (below), including the two-layer model at D = 6144.++ **Tests.** `tests/vulkan_engines.sh inkling-olmoe-chain` gates every configuration on the+ CPU run's tokens and every forward's logits within 1e-4 of the largest (`DUMP=<path>` in+ both engines' ref mode writes them; measured 2e-7 relative on the fixtures, 6e-7 at+ D = 6144): inkling's f32, dense-int4g64 and bf16 snapshots (on an AVX512-BF16 host the+ bf16 one checks the clean decline instead), its int4 and int8 expert containers and+ runtime quantizations, `TOPP`, the tier off, the trunk's device copies shared with the+ per-matrix path, prefill in chunks of 3, the tiled GEMM, the per-row GEMV, prompts only,+ D = 6144 (and the tier's expert batch at that width); olmoe's caps, 4-bit experts,+ `PILOT` at caps 1 and 2 and `PILOT=3`, the tier off, the shared trunk, an eviction+ budget, chunks, the GEMM, the per-row GEMV, prompts only; the device lost mid-decode in+ both (inkling in a shared-expert frame and at a router); `tests/vulkan_chain_serve.py`+ sessions (pins, prompt-cache extensions, a divergent prompt, logprobs) frame for frame+ against the CPU, with the chain and with prompts only; and both engines' prefix-reuse,+ dashboard and Brio tests with the chain on. `inkling-olmoe-chain-sanitize` runs the+ chain's ops, a set of those configurations and a serve session of each engine under+ ASan and UBSan. On the 780M (RADV) every one of those configurations and serve sessions+ gave the CPU's tokens.++### GLM-5.2 and GLM-5.3 Flash on the chain++`glm_chain.h` (colibri) and `glm53_chain.h` (glm53) follow the recipe below with the+[MLA ops](#multi-head-latent-attention-on-the-chain-vkc_mla). What runs where, per+layer:++| | colibri (GLM-5.2) | glm53 (GLM-5.3 Flash) |+|---|---|---|+| device, frame A1 | the previous layer's MoE output joining the residual (routed, then the shared expert, then the add: the CPU's order); the input RMSNorm; the MLA attention: q_a, its norm, q_b, kv_a, the latent norm, interleaved RoPE, the new latent and rope rows into the device cache, on a DSA layer the index key (wk, LayerNorm, RoPE) into its cache and, past `index_topk` (or with `DSA_FORCE`), each row's top-k, reused by the shared layers after it; the absorbed core over the cache or the selection, the value rows, o_proj; the add; the post-attention norm. A dense layer runs its MLP here and has no host step | the previous layer's FFN branch (the routed sum plus the shared expert) written back into the hc_mult streams; every site through mHC (the mix, the split with Sinkhorn, the collapse, the write back); the input RMSNorm; a KDA layer (its projections, the short convolution with its window, the delta rule with its state, the output norm and gate, o) or an MLA layer (the projections into the cache, the k-pooled indexer: index keys and pool gates into their caches, each completed pool's key, every row's pools; the absorbed core over the selection, the values, o); the FFN site's entry and norm; a dense layer's MLP (clamped SwiGLU) |+| host | moe() on the normalized rows without the shared expert: the f32 router with every routing option, the routed experts (the tier's batch and the CPU's share); the new KV rows copied into the host's cache | the router and the routed experts (the tier's batch and the CPU's share); the new MLA rows copied into the host's cache |+| device, frame A2 (not waited for) | the shared expert | the shared expert (clamped SwiGLU) |+| after the last layer | the final rows back to the host, which runs the final norm and lm_head as before | the final streams back to the host, which collapses them and runs the final norm and the head as before |++**The state.** The KV caches (GLM-5.2's latent, rope keys and index keys; GLM-5.3's+latent, index keys and pool gates) stay the host's: each step copies its new rows back,+and the device mirror has a watermark that every host write lowers (the CPU's attention,+`kv_alloc`, a slot adopting another slot's rows, another KV state or session bound, a+pin restored). GLM-5.3's pool keys have a watermark of their own. MLA has no recurrent+state, so a rejected draft is rows the next step rewrites. GLM-5.3's KDA state and+convolution windows stay on the device while the chain runs, for one session at a time:+the host's copy is brought back before a pin or a state capture reads it, before a CPU+forward of the session and when another session takes the device, and goes up after a+pin or a state is restored.++**Drafts and the MTP head.** colibri's speculative decode runs as before: the verify+rows go through the chain, the MTP head stays on the CPU and reads the chain's final+rows, and n-gram drafts work the same way. glm53 has no draft path.++**A lost device.** The forward that failed runs again on the CPU from its input (the+chain keeps the caller's rows untouched until its last chunk is through), and the CPU+runs from there. colibri has nothing to rebuild (its cache is the host's). glm53+rebuilds the KDA state on the CPU from the input rows the chain records since the host's+copy was last current (embedding rows, or the vision tower's), a prefill's worth of CPU+work.++**Declined** (the CPU path runs, the state synced first): a ragged multi-slot decode+batch (a single-slot serve's one-row batch takes the chain), a layer range (a segment),+a quantized KV cache (KV8, KV_TQ), PILOT, LOOKA, the exact verify of+`COLI_EXACT_VERIFY`, the CUDA backend, matrices with no device form (int2, E8/IQ3, fp8+dense matrices), and geometries past the ops' limits. With the chain on, the per-matrix+switches (`COLI_VK_DENSE`, `COLI_VK_ATTN`, `COLI_VK_DEV2`) keep working beside it: the+chain's tensors are their device copies where they made one.++**Arithmetic.** f32 activations throughout, as the CPU's f32 paths: colibri's CPU int8+dot (`IDOT`, on by default for int8 and, from two rows, int4 rows) rounds activations,+so the tests set `IDOT=0` for those trunks, as for the tier. On the fixtures every+configuration gives the CPU's tokens, and every logits row is within 2e-6 of the largest+logit (Lavapipe: 1.9e-6 at worst, colibri's int4-g64 experts on the tier, 6.3e-7 and+below everywhere else).++**The default**: both engines pass `COLI_VK_CHAIN_UNMEASURED`, so the chain is off on+an integrated GPU (`COLI_VK_CHAIN=1` turns it on, `2` for prompts only) and on a+discrete GPU follows the rule above. No GLM checkpoint was run: the 780M box has none,+and both models are hundreds of GB. Speed is not measured; the tests prove the tokens on+the tiny fixtures, on Lavapipe and on the 780M:++```+[VK] colibri: dense chain off (an integrated GPU with the expert tier: not measured; COLI_VK_CHAIN=0 off, 1 on, 2 prompts only)+```+### Adding an engine to the chainThe recipe qwen36_chain.h and qwen38_chain.h follow, for the engines still on the@@@ -915,62 -1044,13 +1106,62 @@@ What each remaining architecture needs| Engine | Attention / mixer | New pieces ||---|---|---|-| colibri.c (GLM-5.2), glm53, deepseek_v41, kimi_k3 (MLA layers) | MLA: q_a/kv_a, the latent norms, q_b, RoPE on the rope dims, a latent + rope cache | the absorb core (`attention_absorb.comp`) already reads a device KV mirror with GLM's watermark (`vk_kv_valid`): record it as a chain op with the o-projection fused (the universal layout's eight bindings hold its seven); DSA's index keys and top-k are `chain_qsa.comp`'s selection with blocks of one position, and the absorb shader takes the selection list as `chain_attn.comp` does; GLM's MTP layer stays on the CPU like qwen38's head, or runs in the chain (MLA has no recurrent state: a rejected draft is a lowered watermark) |-| deepseek_v4 | MLA with compressed (CSA) and hierarchical (HCA) KV, mHC | the manifold hyper-connections are qwen38's stream read/apply with a Sinkhorn normalization (an element-wise op that iterates); the compressors' rolling windows are rings like the conv's, snapshotted the same way; the CPU rounds activations to E4M3 before its fp8 matmuls, so the chain needs that rounding as an element-wise op to keep the same arithmetic |-| kimi_k3 (KDA layers) | Kimi Delta Attention: a gated delta rule whose decay is a vector over the key channels | `chain_dnrec.comp` with the decay per key row (one `exp(g_k)` per row of the column, loaded beside q and k in shared memory) instead of one per head; the short convolution is `chain_dnconv.comp`; its output gate and norm as the gated norm |+| deepseek_v41, kimi_k3 (MLA layers) | MLA: q_a/kv_a, the latent norms, q_b, RoPE on the rope dims, a latent + rope cache | the [MLA ops](#multi-head-latent-attention-on-the-chain-vkc_mla) take any geometry (q_lora or none, NoPE, the scale and cos/sin table from the engine, a gate on the values for Kimi K3); colibri and glm53 run them ([above](#glm-52-and-glm-53-flash-on-the-chain)). DeepSeek V4.1's attention has sinks and its own sparse selection: those are not in the ops yet |+| deepseek_v4 | MLA with compressed (CSA) and hierarchical (HCA) KV, mHC | the manifold hyper-connections are `vkc_mhc` (GLM-5.3's, the same arithmetic) and the clamped SwiGLU too; the compressors' rolling windows are rings like the conv's, snapshotted the same way; the CPU rounds activations to E4M3 before its fp8 matmuls, so the chain needs that rounding as an element-wise op to keep the same arithmetic |+| kimi_k3 (KDA layers) | Kimi Delta Attention: a gated delta rule whose decay is a vector over the key channels | `vkc_kda_conv` and `vkc_kda_rec` (GLM-5.3's KDA, `delta_attention.h`'s step); Kimi K3 builds its decay and output gate its own way (a full g_proj, not a low-rank one), which the recurrence's prologue and epilogue would take as options |- | mimo | sliding-window attention (and full layers) | `chain_attn.comp` with a window (positions from `max(0, pos - W + 1)`: one more push constant), the cache optionally a ring of W rows (the host mirror then indexes `t % W`) |- | inkling | grouped attention, MoE with a shared expert | qwen36's attention and combine as they are; its per-position heads are matmuls of the residual rows |- | olmoe | attention with q/k norm, MoE without a shared expert | qwen36's Qwen3-Coder geometry (all attention, no gate, no shared expert) is the same chain |+ | mimo | sliding-window attention (and full layers) | done: [MiMo-V2.6 on the chain](#mimo-v26-on-the-chain) (`chain_attn.comp` with a window, a ring of W rows, V's own head dim and a sink) |+ | inkling | grouped attention with a relative-position bias, a sliding window and short convolutions; MoE with shared experts | in the chain ([OLMoE and Inkling](#olmoe-and-inkling)): two shaders of its own, `chain_relattn.comp` and `chain_sconv.comp`; the shared experts join through `HC_APPLY` |+ | olmoe | attention with q/k norm, MoE without a shared expert | in the chain ([OLMoE and Inkling](#olmoe-and-inkling)) with qwen36's ops as they are |+### Multi-head latent attention on the chain (`vkc_mla`)++The MLA layers (GLM-5.2, GLM-5.3, the DeepSeek V3 family's attention, Kimi K3's MLA+layers) have chain ops of their own, in `vk_chain.h`, written for any geometry: H heads+of Q no-position and R rotated query floats (R = 0 for NoPE), V value floats, a latent of+K floats (`kv_lora`), q_lora from 0 (q straight from the hidden rows) up, RoPE as rotate+-half or as interleaved pairs, the softmax scale and the cos/sin table from the caller (a+YaRN model passes its scaled frequencies, its mscale on the table and mscale squared on+the scale). The cache on the device holds, per position, the normalized latent and the+rotated shared key; the engine owns it, mirrors the host's rows behind a watermark as the+GQA engines do, and copies each step's new rows back.++| Op | Shader | What it does |+|---|---|---|+| `vkc_mla_qkv` | `chain_norm`, `chain_mla` (mode 1) and `vkc_matmul` | q_a, its RMSNorm and q_b (or q_b alone), kv_a, the latent's RMSNorm into the cache row, RoPE on q's rotated part and on the shared key into the cache row, a copy of the new rows for the host |+| `vkc_mla_attn` | `chain_hgemv`, `chain_mla` (mode 0) | the absorbed query (each head's Q values times its key rows of `kv_b`, read transposed, or a `[H*K x Q]` matrix), the attention core over the cache with an online softmax (the causal range or a selection list with skipped entries), the value rows on the softmax-weighted latent, an optional sigmoid gate, o_proj |+| `vkc_mla_rope`, `vkc_mla_lnorm` | `chain_mla` (modes 1, 2) | RoPE over segments from a host table, in place or into another buffer; LayerNorm with weight and bias (an indexer's key norm) |+| `vkc_dsa_select` | `chain_dsa` | a token-level DSA indexer: each position's score `(sum_h [d_h > 0] w_h d_h) * wscale`, `d_h = (q_h . k_t) * qscale`, and the top-k in the CPU's order (above the k-th score in position order, then the ties in position order), or "every position" while the context is within top-k |++The weights stay in the format the engine already holds them in (int8 and int4 rows,+int4 and int8 and fp8 in groups, int3-g64, MXFP4, f32, bf16): the per-head blocks read+them where the backend uploaded them. Between `vkc_mla_qkv` and `vkc_mla_attn` an engine+records what reads the projections (a DSA indexer reads the normalized q latent the first+leaves in its scratch). Limits: K up to 1024, R up to 128, Q up to 1024 where kv_b's key+rows are read transposed, an indexer of up to 64 heads and 4096 query floats.++GLM-5.3 Flash adds three more things, as ops of their own in `vk_chain.h`, which the+DeepSeek V4 and Kimi K3 chains can take as they are:++| Op | Shader | What it does |+|---|---|---|+| `vkc_dsa_pool_keys`, `vkc_dsa_pool_select` | `chain_dsa` (modes 1, 2) | the k-pooled indexer (`sparse_index.h`): a pool's key, the per-channel softmax mixture of its members' keys under their gate logits plus a position bias, computed once when a step completes the pool; each row's top pools in rank order (score, then the lower pool), their positions, the incomplete tail, -1 in the unused slots |+| `vkc_kda_conv`, `vkc_kda_rec` | `chain_kda` | Kimi Delta Attention (`delta_attention.h`): the short convolution with its window, the gated delta rule with a decay per key row and the state on the device, GLM-5.3's output RMSNorm and sigmoid gate; the l2 sums and the norm's sum in the CPU's order |+| `vkc_mhc` | `chain_mhc` | manifold-constrained hyper-connections (`hyper_connections.h`): the mix logits' split with Sinkhorn, the collapse, the write back, the plain mean of the last collapse; and a clamped SwiGLU |++`make vk-chain-check VK=1` runs them against double-precision references of colibri.c's+absorbed attention: every weight format both ways through the per-head blocks, RoPE in+both styles in place and into a cache row, LayerNorm with and without a bias, and the+layer op over eight geometries (q latent or none, NoPE, `kv_b` or the split halves,+selection lists with skipped entries, a gate, prefill rows after earlier ones, a long+context from a nonzero start, GLM-5.2's head shape, a latent of 1024). Measured on+Lavapipe and on an Intel Iris Xe (Mesa's Dozen): the layer's output within 5e-7 of its+largest value and the new cache rows within 7e-7 (the test's bound is 2e-5). The+indexer's selection is checked bit for bit, ties included, on scores that are exact in+float on both sides. The k-pooled selection is checked slot for slot against `sparse_index.h` the same+way, the KDA layer against `delta_attention.h` over two submissions (the state and the+window carried between them, inputs small enough that the l2 eps counts), the mHC ops+against `hyper_connections.h`; each within 3e-7 on Lavapipe and on the Iris Xe.+## Adding an engine to the tierEvery MoE engine here is on the tier ([the table above](#the-routed-expert-tier-vk_tierc));@@@ -1051,13 -1131,16 +1242,22 @@@ in shortthe last logits within 1e-4 of the largest one where both sides use f32 activations(measured 2e-7 and below on the fixtures), prefill in chunks, an image, MTP draftsrejected, accepted and alternating, a device lost mid-run, the qwen38 oracle targets,- the prefix-reuse contract and serve sessions frame for frame; `glm-chain` does the+ the prefix-reuse contract and serve sessions frame for frame; `qwen-chain-sanitize`+ runs the chain under ASan and UBSan. `mimo-chain` does it for MiMo: the vendor+ oracle with the chain on, the CPU's tokens and every position's logits in every dense+ form, block size, case and tier setting, the window boundary bit for bit across block+ sizes, prompts only, a device lost in four places, the prefix-reuse and photo tests+ and serve sessions; `mimo-chain-sanitize` under ASan and UBSan.+ `inkling-olmoe-chain` and+ `inkling-olmoe-chain-sanitize` do the same for inkling and olmoe+ ([OLMoE and Inkling](#olmoe-and-inkling)); `vk-chain-check` covers their ops+ (`chain_sconv.comp`, `chain_relattn.comp`, the GEMV and the norm at D = 6144).++ `glm-chain` does the+ same for colibri and glm53: every expert format and trunk, prefill in chunks, the DSA+ selection active, n-gram and MTP drafts accepted and rejected, glm53's image, KDA state+ and swiglu_limit 0, a device lost (glm53's KDA state rebuilt), serve sessions with pins+ and two KV slots, and glm53's pin-branch harness; `glm-chain-sanitize` runs them under- ASan and UBSan; `qwen-chain-sanitize`- runs the chain under ASan and UBSan.++ ASan and UBSan.- int4 weights decode as offset-binary (nibble−8), byte-identical layout tothe CPU path — no repacking.- Khronos validation layers: the backend never enables them, so the loader