Skip to content

Run frontier MoE models on hardware you already own — pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦

C

Merge branch 'feat/vk-tier-glm' into feat/vk-tier-all-engines

JustVuggd7a3f0e4bbb3
diff --cc .github/workflows/ci.yml
index b04b635e,ee2ff907..036a071f
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@@ -178,16 -175,12 +178,22 @@@ jobs
- family: deepseek
req: c/tools/requirements-deepseek-v4-tiny.txt
extra: sympy pillow # the V4.1 fixture's reference and its vision tower
+ # the routed-expert tier of deepseek_v41 and deepseek_v4 under ASan and UBSan
+ - family: deepseek-sanitize
+ req: c/tools/requirements-deepseek-v4-tiny.txt
+ extra: sympy pillow
+ # Kimi K3's routed experts on the shared tier, and its and MiMo's tier under
+ # ASan and UBSan (MiMo's tier runs in mimo-qwenimage)
+ - family: kimi
+ req: c/tools/requirements-kimi-k3-tiny.txt
+ - family: kimi-mimo-sanitize
+ req: c/tools/requirements-mimo-tiny.txt
+ # GLM-5.2 (colibri) and GLM-5.3 Flash (glm53) on the routed-expert tier, and
+ # the same under ASan and UBSan
+ - family: glm
+ req: c/tools/requirements-glm53-tiny.txt
+ - family: glm-sanitize
+ req: c/tools/requirements-glm53-tiny.txt
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
diff --cc c/tests/vulkan_engines.sh
index e99ef4e9,7301f604..00276725
--- a/c/tests/vulkan_engines.sh
+++ b/c/tests/vulkan_engines.sh
@@@ -2,8 -2,8 +2,9 @@@
# Every engine's Vulkan path against its own CPU run, on Lavapipe (Mesa's software
# Vulkan), one family per call so CI can run them side by side:
#
-# bash tests/vulkan_engines.sh qwen | qwen-sanitize | inkling-olmoe | mimo-qwenimage | deepseek
+# bash tests/vulkan_engines.sh qwen | qwen-sanitize | inkling-olmoe | inkling-olmoe-sanitize
+# bash tests/vulkan_engines.sh mimo-qwenimage | kimi | kimi-mimo-sanitize | deepseek | deepseek-sanitize
+ # bash tests/vulkan_engines.sh glm | glm-sanitize # GLM-5.2 (colibri) and GLM-5.3 Flash (glm53)
# bash tests/vulkan_engines.sh shader # the qmatmul formats, the expert batch and the tier, no engine
#
# Needs libvulkan-dev, glslc and mesa-vulkan-drivers, plus the Python packages of
@@@ -692,264 -464,205 +693,454 @@@ print("OK deepseek_v4 session: ids = CP
sys.exit(0 if ok else 1)
PY
need_gpu deepseek_v4 v4-vk.err "deepseek_v4 session"
- echo "OK deepseek_v4: $(vk_count deepseek_v4 v4-vk.err) matmuls on the GPU"
+ echo "OK deepseek_v4: $(vk_count deepseek_v4 v4-vk.err) matmuls on the GPU, $(grep -a -o 'device [0-9]* of [0-9]* routed experts' v4-vk.err | tail -1)"
+
+ # The routed-expert tier on deepseek_v4's fp4 experts (fmt 7) with its own
+ # activation (VKT_ACT_SWIGLU_V4: the CPU kernel's bf16 and E4M3 roundings and the
+ # route weight before down, on the device). Every oracle case against the CPU run:
+ # ids equal to the CPU's and the reference's, teacher-forced predictions equal,
+ # routed experts served by the device. The history is the store's own .coli_usage
+ # in the fixture, removed before each run unless WARM=1 (the tier then warm-starts
+ # from the CPU run's): the runs use a copy of the fixture, whose committed history
+ # stays as it is. An 8-expert variant of the fixture has room for pinned hot
+ # experts, which the store keeps in its rows16 layout: the tier gets them unpacked.
+ rm -rf deepseek_v4_tiny_t && cp -r deepseek_v4_tiny deepseek_v4_tiny_t
+ $PY - tools/make_deepseek_v4_tiny.py deepseek_v4_tiny_e8 <<'PY'
+import importlib.util, sys
+spec = importlib.util.spec_from_file_location("gen", sys.argv[1])
+gen = importlib.util.module_from_spec(spec); spec.loader.exec_module(gen)
+gen.EXPERTS = 8 # 8 routed experts a layer: two cache slots a layer for pins
+sys.argv = ["make_deepseek_v4_tiny.py", "--output", sys.argv[2], "--force"]
+gen.main()
+PY
+ v4_tier() { # <tag> <fixture> <case> <env...>
+ local tag=$1 fx=$2 c=$3; shift 3
+ local p mt
+ p=$($PY -c 'import json,sys; c=json.load(open(sys.argv[1]+"/ref.json"))["cases"][sys.argv[2]]; print("".join("<t%03d>" % t for t in c["prompt_ids"]))' $fx $c)
+ mt=$($PY -c 'import json,sys; print(json.load(open(sys.argv[1]+"/ref.json"))["cases"][sys.argv[2]]["max_new_tokens"])' $fx $c)
+ rm -f $fx/.coli_usage
+ env "$@" ./deepseek_v4 ./$fx "$p" --raw-prompt --max-tokens $mt --record-oracle v4-cpu.json > /dev/null 2> v4-cpu.err ||
+ { cat v4-cpu.err; fail "deepseek_v4 tier $tag: CPU run"; }
+ [ "${WARM:-0}" = 1 ] || rm -f $fx/.coli_usage
+ env "$@" COLI_VK_TIER_SYNC=1 COLI_VULKAN=1 ./deepseek_v4 ./$fx "$p" --raw-prompt --max-tokens $mt --record-oracle v4-vk.json > /dev/null 2> v4-vk.err ||
+ { cat v4-vk.err; fail "deepseek_v4 tier $tag: Vulkan run"; }
+ rm -f $fx/.coli_usage
+ $PY - $fx $c <<'PY' || fail "deepseek_v4 tier $tag: the Vulkan session differs from the CPU's"
+import json, sys
+a, b = json.load(open("v4-cpu.json")), json.load(open("v4-vk.json"))
+ref = json.load(open(sys.argv[1] + "/ref.json"))["cases"][sys.argv[2]]["greedy_full_ids"]
+ok = a["full_ids"] == b["full_ids"] == ref and a["tf_pred"] == b["tf_pred"]
+if not ok: print("CPU", a, "VK", b, "ref", ref)
+sys.exit(0 if ok else 1)
+PY
+ [ "$(tier_count deepseek_v4 v4-vk.err)" -gt 0 ] || { cat v4-vk.err; fail "deepseek_v4 tier $tag: no routed expert ran on the device"; }
+ if [ "${EVICT:-0}" = 1 ]; then
+ [ "$(tier_evictions deepseek_v4 v4-vk.err)" -gt 0 ] || { grep '\[VK\] tier' v4-vk.err; fail "deepseek_v4 tier $tag: the budget forced no eviction"; }
+ fi
+ if [ "${WARM:-0}" = 1 ]; then
+ grep -q '^\[VK\] tier deepseek_v4: warm start' v4-vk.err || { grep '\[VK\] tier' v4-vk.err; fail "deepseek_v4 tier $tag: no warm start"; }
+ fi
+ echo "OK deepseek_v4 tier $tag ($c): ids = CPU = reference, $(grep -a -o 'device [0-9]* of [0-9]* routed experts' v4-vk.err | tail -1), $(grep -a -o 'evictions [0-9]*' v4-vk.err | tail -1), $(grep -a -o 'packed_slots=[0-9]*' v4-vk.err) rows16"
+ }
+ for c in short compressed long; do
+ v4_tier "4 experts" deepseek_v4_tiny_t $c
+ v4_tier "8 experts, pinned rows16" deepseek_v4_tiny_e8 $c
+ done
+ EVICT=1 v4_tier "a budget of three experts" deepseek_v4_tiny_e8 long COLI_VK_TIER_GB=0.00009
+ v4_tier "trunk on the device too" deepseek_v4_tiny_t long COLI_VK_DENSE=1
+ v4_tier "the GEMM route from 2 rows" deepseek_v4_tiny_t long COLI_VK_TIER_GEMM_ROWS=2
+ WARM=1 v4_tier "warm start from the CPU run's history" deepseek_v4_tiny_e8 long
+ # A served prompt with every routed expert on the device (warm start from the CPU
+ # run's history): its per-position logprob echoes against the CPU's. Measured on
+ # Lavapipe they are the same bytes; the gate allows 1e-3, for an exp() that rounds
+ # one hidden value to the other bf16 neighbour on another driver.
+ $PY - deepseek_v4_tiny_e8 <<'PY' || fail "deepseek_v4 tier: served logprobs differ from the CPU's"
+import json, os, sys
+from pathlib import Path
+sys.path.insert(0, "tests")
+import test_deepseek_v4_brio as b
+fx = Path(sys.argv[1])
+case = json.load(open(fx / "ref.json"))["cases"]["long"]
+def echoes(vk):
+ os.environ.pop("COLI_VULKAN", None)
+ if vk: os.environ.update(COLI_VULKAN="1", COLI_VK_TIER_SYNC="1")
+ else:
+ try: os.remove(fx / ".coli_usage")
+ except FileNotFoundError: pass
+ s = b.Serve(Path("deepseek_v4").resolve(), fx)
+ try: return s.submit(b.token_prompt(case["prompt_ids"]), 0, logprobs=5).echoes
+ finally:
+ s.close()
+ tier = [l for l in s.process.stderr.read().decode(errors="replace").splitlines() if "tier deepseek_v4 turn:" in l]
+ if vk: print(" ", tier[-1][:90] if tier else "no tier line")
+cpu, dev = echoes(False), echoes(True)
+os.environ.pop("COLI_VULKAN", None)
+same = sum(cpu[p] == dev.get(p) for p in cpu)
+worst = max(abs(cpu[p]["lp"] - dev[p]["lp"]) for p in cpu) if len(cpu) == len(dev) else 1.0
+print(f"OK deepseek_v4 tier served: {same} of {len(cpu)} positions' logprob echoes identical to the CPU's, worst |delta| {worst:.2e}")
+sys.exit(0 if len(cpu) == len(dev) and worst <= 1e-3 else 1)
+PY
+ rm -rf deepseek_v4_tiny_t deepseek_v4_tiny_e8
+}
+
+# The routed-expert tier of both deepseek engines under ASan and UBSan: sanitized VK=1
+# builds, the tier's configurations on Lavapipe (decode and prefill, eviction, DSpark,
+# pinned rows16 experts, the warm start, the trunk on the device). Memory safety is
+# the gate; each run must still put experts on the device.
+family_deepseek_sanitize() {
+ local SAN="-fsanitize=address,undefined -fno-omit-frame-pointer -g"
+ make clean >/dev/null 2>&1 || true
+ make deepseek_v41 VK=1 EXTRA_CFLAGS="$SAN"
+ make deepseek-v4 VK=1 LTO=0 EXTRA_CFLAGS="$SAN" EXTRA_LDFLAGS="$SAN"
+ $PY tools/make_dsv41_tiny.py --out dsv41_tiny --emit-ref dsv41_tiny/ref.json
+ $PY tools/make_dsv41_tiny.py --out dsv41_long --emit-ref dsv41_long/ref.json --prompt-len 40 --max-new 6
+ $PY tools/make_deepseek_v4_tiny.py --output deepseek_v4_tiny_t --force
+ $PY - tools/make_deepseek_v4_tiny.py deepseek_v4_tiny_e8 <<'PY'
+import importlib.util, sys
+spec = importlib.util.spec_from_file_location("gen", sys.argv[1])
+gen = importlib.util.module_from_spec(spec); spec.loader.exec_module(gen)
+gen.EXPERTS = 8
+sys.argv = ["make_deepseek_v4_tiny.py", "--output", sys.argv[2], "--force"]
+gen.main()
+PY
+ san() { # <engine> <tag> <env and argv...>
+ local eng=$1 tag=$2; shift 2
+ rm -f tier.usage deepseek_v4_tiny_t/.coli_usage deepseek_v4_tiny_e8/.coli_usage
+ env ASAN_OPTIONS=detect_leaks=0 UBSAN_OPTIONS=print_stacktrace=1 OMP_NUM_THREADS=2 \
+ COLI_USAGE=tier.usage COLI_VK_TIER_SYNC=1 COLI_VULKAN=1 "$@" > san.log 2>&1 || true
+ if grep -qE "ERROR: AddressSanitizer|runtime error:" san.log; then cat san.log; fail "$tag: sanitizer diagnostic"; fi
+ [ "$(tier_count "$eng" san.log)" -gt 0 ] || { cat san.log; fail "$tag: no routed expert ran on the device"; }
+ echo "OK $tag: sanitizers clean, $(grep -a -o 'device [0-9]* of [0-9]* routed experts' san.log | tail -1), $(grep -a -o 'evictions [0-9]*' san.log | tail -1)"
+ }
+ local cap force p
+ for cap in 1 8; do san deepseek_v41 "asan deepseek_v41 cap=$cap" SNAP=dsv41_long ./deepseek_v41 $cap dsv41_long/ref.json; done
+ for force in 2 4; do san deepseek_v41 "asan deepseek_v41 DSpark spec=$force" SNAP=dsv41_tiny V41_DSPARK=1 V41_SPEC_FORCE=$force ./deepseek_v41 1 dsv41_tiny/ref.json; done
+ san deepseek_v41 "asan deepseek_v41 eviction" COLI_VK_TIER_GB=0.00005 SNAP=dsv41_long ./deepseek_v41 8 dsv41_long/ref.json
+ san deepseek_v41 "asan deepseek_v41 trunk on the device" COLI_VK_DENSE=1 SNAP=dsv41_tiny ./deepseek_v41 2 dsv41_tiny/ref.json
+ p=$($PY -c 'import json; c=json.load(open("deepseek_v4_tiny_t/ref.json"))["cases"]["long"]; print("".join("<t%03d>" % t for t in c["prompt_ids"]))')
+ san deepseek_v4 "asan deepseek_v4" ./deepseek_v4 ./deepseek_v4_tiny_t "$p" --raw-prompt --max-tokens 4 --record-oracle san.json
+ san deepseek_v4 "asan deepseek_v4 pinned rows16" ./deepseek_v4 ./deepseek_v4_tiny_e8 "$p" --raw-prompt --max-tokens 4 --record-oracle san.json
+ san deepseek_v4 "asan deepseek_v4 eviction" COLI_VK_TIER_GB=0.00009 ./deepseek_v4 ./deepseek_v4_tiny_e8 "$p" --raw-prompt --max-tokens 4 --record-oracle san.json
+ san deepseek_v4 "asan deepseek_v4 trunk on the device" COLI_VK_DENSE=1 ./deepseek_v4 ./deepseek_v4_tiny_t "$p" --raw-prompt --max-tokens 4
+ # the warm start: a run that leaves its history, then one that starts from it
+ ./deepseek_v4 ./deepseek_v4_tiny_e8 "$p" --raw-prompt --max-tokens 4 > /dev/null 2>&1 || true
+ env ASAN_OPTIONS=detect_leaks=0 UBSAN_OPTIONS=print_stacktrace=1 OMP_NUM_THREADS=2 COLI_VK_TIER_SYNC=1 COLI_VULKAN=1 \
+ ./deepseek_v4 ./deepseek_v4_tiny_e8 "$p" --raw-prompt --max-tokens 4 > san.log 2>&1 || true
+ if grep -qE "ERROR: AddressSanitizer|runtime error:" san.log; then cat san.log; fail "asan deepseek_v4 warm start: sanitizer diagnostic"; fi
+ grep -q '^\[VK\] tier deepseek_v4: warm start' san.log || { cat san.log; fail "asan deepseek_v4 warm start: no warm start"; }
+ echo "OK asan deepseek_v4 warm start: sanitizers clean, $(grep -a -o 'device [0-9]* of [0-9]* routed experts' san.log | tail -1)"
+ rm -rf deepseek_v4_tiny_t deepseek_v4_tiny_e8 san.json
+ make clean >/dev/null 2>&1 || true
+ make deepseek-v4-clean >/dev/null 2>&1 || true
+}
+
+# Kimi K3: the routed experts on the shared tier (MXFP4 with ue8m0 scales, fmt 7,
+# SiTU-GLU in the latent space), against Moonshot's vendor oracle and the CPU run.
+# K3_IDOT=0 everywhere: the CPU's default int8-activation expert kernel is an
+# approximation the device does not make (the oracle's engine_env sets it too).
+family_kimi() {
+ make kimi_k3 VK=1
+ $PY tools/make_kimi_k3_tiny.py --output ./kimi_k3_tiny --force
+ # Moonshot's oracle (greedy, teacher forcing at every position, determinism, the
+ # bite) with the tier on, the shared experts where the default puts them (the CPU
+ # on Lavapipe) and on the device
+ COLI_USAGE=$PWD/k3.usage USAGE_SAVE=0 COLI_VULKAN=1 COLI_VK_TIER_SYNC=1 \
+ $PY tests/test_kimi_k3_tiny.py --binary ./kimi_k3 --fixture ./kimi_k3_tiny
+ COLI_USAGE=$PWD/k3.usage USAGE_SAVE=0 COLI_VULKAN=1 COLI_VK_TIER_SYNC=1 COLI_VK_DENSE=1 \
+ $PY tests/test_kimi_k3_tiny.py --binary ./kimi_k3 --fixture ./kimi_k3_tiny
+ k3ids() { $PY -c "import json,sys;print(' '.join(map(str,json.load(open('kimi_k3_tiny/ref.json'))['cases'][sys.argv[1]]['prompt_ids'])))" "$1"; }
+ # k3_gate <tag> <env...>: the tokens of the CPU run, the tier served some experts,
+ # the shared experts where they were asked to be (dense_where); EVICT=1: it evicted
+ k3_gate() {
+ local tag=$1 c; shift
+ for c in short chunk long; do
+ rm -f k3.usage
+ env "$@" COLI_USAGE=$PWD/k3.usage USAGE_SAVE=0 ./kimi_k3 kimi_k3_tiny --ids "$(k3ids $c)" --ngen 8 2>/dev/null | sed 's/ *TUNE.*//' > cpu.tok
+ env "$@" COLI_USAGE=$PWD/k3.usage USAGE_SAVE=0 COLI_VK_TIER_SYNC=1 COLI_VULKAN=1 \
+ ./kimi_k3 kimi_k3_tiny --ids "$(k3ids $c)" --ngen 8 2> vk.log | sed 's/ *TUNE.*//' > vk.tok
+ { [ -s cpu.tok ] && cmp -s cpu.tok vk.tok; } || { cat cpu.tok vk.tok vk.log; fail "$tag $c: Vulkan tokens differ from the CPU"; }
+ [ "$(tier_count kimi_k3 vk.log)" -gt 0 ] || { cat vk.log; fail "$tag $c: no routed expert ran on the device"; }
+ if [ "${EVICT:-0}" = 1 ]; then
+ [ "$(tier_evictions kimi_k3 vk.log)" -gt 0 ] || { grep '\[VK\] tier' vk.log; fail "$tag $c: the budget forced no eviction"; }
+ grep -q ' failed 0 ' vk.log || { grep '\[VK\] tier' vk.log; fail "$tag $c: an upload failed"; }
+ fi
+ local where; where=$(dense_where kimi_k3 vk.log "$tag $c" "$@") || { echo "$where"; exit 1; }
+ echo "OK $tag $c: tokens = CPU, $(grep -a -o 'device [0-9]* of [0-9]* routed experts' vk.log | tail -1), $(grep -a -o 'evictions [0-9]*' vk.log | tail -1), shared experts on the $where"
+ done
+ }
+ local O="K3_BITS=32 K3_MLA_BITS=32 K3_HEAD_BITS=32 K3_IDOT=0 COLI_TEMP=0" b
+ k3_gate "kimi_k3 tier f32" $O
+ k3_gate "kimi_k3 tier f32, shared experts on the device" $O COLI_VK_DENSE=1
+ for b in 8 4; do # the shared experts as int8 rows (fmt 1) and int4-g64 (fmt 4) on the device
+ k3_gate "kimi_k3 tier K3_BITS=$b, shared experts on the device" K3_BITS=$b K3_MLA_BITS=$b K3_HEAD_BITS=$b K3_IDOT=0 COLI_TEMP=0 COLI_VK_DENSE=1
+ done
+ k3_gate "kimi_k3 tier, prefill one token at a time" $O K3_CHUNK=1
+ k3_gate "kimi_k3 tier, loads not pipelined" $O K3_PIPE=0
+ EVICT=1 k3_gate "kimi_k3 tier, a budget of two experts" $O COLI_VK_TIER_GB=0.000005
+ # the old switches: K3_VK=1 opens the device as COLI_VULKAN=1 does and K3_VK_GB caps
+ # the tier; K3_VK=0 keeps it closed whatever COLI_VULKAN says
+ rm -f k3.usage
+ env $O COLI_USAGE=$PWD/k3.usage USAGE_SAVE=0 ./kimi_k3 kimi_k3_tiny --ids "$(k3ids long)" --ngen 8 2>/dev/null | sed 's/ *TUNE.*//' > cpu.tok
+ env $O COLI_USAGE=$PWD/k3.usage USAGE_SAVE=0 COLI_VK_TIER_SYNC=1 K3_VK=1 K3_VK_GB=0.000005 \
+ ./kimi_k3 kimi_k3_tiny --ids "$(k3ids long)" --ngen 8 2> vk.log | sed 's/ *TUNE.*//' > vk.tok
+ cmp -s cpu.tok vk.tok || { cat vk.log; fail "kimi_k3 K3_VK=1: tokens differ from the CPU"; }
+ grep -q 'K3_VK=1 read as COLI_VULKAN=1' vk.log && grep -q 'budget [0-9.]* KiB = 2 experts' vk.log &&
+ [ "$(tier_count kimi_k3 vk.log)" -gt 0 ] || { cat vk.log; fail "kimi_k3 K3_VK=1 K3_VK_GB: not the tier it asked for"; }
+ env $O COLI_USAGE=$PWD/k3.usage USAGE_SAVE=0 K3_VK=0 COLI_VULKAN=1 \
+ ./kimi_k3 kimi_k3_tiny --ids "$(k3ids long)" --ngen 8 2> vk.log | sed 's/ *TUNE.*//' > vk.tok
+ cmp -s cpu.tok vk.tok && ! grep -q '^\[VK\]' vk.log || { cat vk.log; fail "kimi_k3 K3_VK=0: the device opened"; }
+ echo "OK kimi_k3 K3_VK=1 / K3_VK_GB / K3_VK=0: the shared tier's switches"
+ # a warm start from the history of the run before: the tier starts full, serves
+ # every routed expert of the same prompt, and the tokens stay the CPU's
+ rm -f k3.usage
+ env $O COLI_USAGE=$PWD/k3.usage COLI_VULKAN=1 COLI_VK_TIER_SYNC=1 ./kimi_k3 kimi_k3_tiny --ids "$(k3ids long)" --ngen 8 > /dev/null 2>&1
+ env $O COLI_USAGE=$PWD/k3.usage USAGE_SAVE=0 COLI_VULKAN=1 COLI_VK_TIER_SYNC=1 \
+ ./kimi_k3 kimi_k3_tiny --ids "$(k3ids long)" --ngen 8 2> vk.log | sed 's/ *TUNE.*//' > vk.tok
+ cmp -s cpu.tok vk.tok || { cat vk.log; fail "kimi_k3 warm start: tokens differ from the CPU"; }
+ grep -q 'tier kimi_k3: warm start, [1-9]' vk.log || { cat vk.log; fail "kimi_k3: no warm start from the history"; }
+ echo "OK kimi_k3 warm start: tokens = CPU, $(grep -a -o 'warm start, [0-9]* experts' vk.log), $(grep -a -o 'device [0-9]* of [0-9]* routed experts' vk.log | tail -1)"
+ # COLI_VK_TIER=0: the shared experts alone on the device, as before the tier
+ env $O COLI_USAGE=$PWD/k3.usage USAGE_SAVE=0 COLI_VK_TIER=0 COLI_VULKAN=1 \
+ ./kimi_k3 kimi_k3_tiny --ids "$(k3ids long)" --ngen 8 2> vk.log | sed 's/ *TUNE.*//' > vk.tok
+ cmp -s cpu.tok vk.tok || { cat vk.log; fail "kimi_k3 COLI_VK_TIER=0: tokens differ from the CPU"; }
+ ! grep -q '^\[VK\] tier' vk.log || { cat vk.log; fail "kimi_k3 COLI_VK_TIER=0: the tier started"; }
+ need_gpu kimi_k3 vk.log "kimi_k3 COLI_VK_TIER=0"
+ echo "OK kimi_k3 COLI_VK_TIER=0: tokens = CPU, no tier, $(vk_count kimi_k3 vk.log) matmuls on the GPU"
+}
+
+# The Kimi K3 and MiMo expert tiers under ASan and UBSan, as qwen-sanitize does for
+# the qwen engines: memory safety is the gate, and each run must still put routed
+# experts on the device.
+family_kimi_mimo_sanitize() {
+ make clean >/dev/null 2>&1 || true
+ make kimi_k3 mimo VK=1 EXTRA_CFLAGS="-fsanitize=address,undefined -fno-omit-frame-pointer -g"
+ $PY tools/make_kimi_k3_tiny.py --output ./kimi_k3_tiny --force
+ $PY tools/make_mimo_tiny.py --output ./mimo_tiny --force --vision
+ ksan() { # <engine> <tag> <env and argv...>
+ local eng=$1 tag=$2; shift 2
+ env ASAN_OPTIONS=detect_leaks=0 UBSAN_OPTIONS=print_stacktrace=1 OMP_NUM_THREADS=2 \
+ COLI_VK_TIER_SYNC=1 COLI_VULKAN=1 "$@" > san.log 2>&1 || true
+ if grep -qE "ERROR: AddressSanitizer|runtime error:" san.log; then cat san.log; fail "$tag: sanitizer diagnostic"; fi
+ [ "$(tier_count "$eng" san.log)" -gt 0 ] || { cat san.log; fail "$tag: no routed expert ran on the device"; }
+ echo "OK $tag: sanitizers clean, $(grep -a -o 'device [0-9]* of [0-9]* routed experts' san.log | tail -1), $(grep -a -o 'evictions [0-9]*' san.log | tail -1)"
+ }
+ local O="K3_BITS=32 K3_MLA_BITS=32 K3_HEAD_BITS=32 K3_IDOT=0 COLI_TEMP=0" K3IDS MIDS IMG GRID
+ K3IDS=$($PY -c "import json;print(' '.join(map(str,json.load(open('kimi_k3_tiny/ref.json'))['cases']['long']['prompt_ids'])))")
+ MIDS=$($PY -c "import json;print(' '.join(map(str,json.load(open('mimo_tiny/ref.json'))['cases']['long']['prompt_ids'])))")
+ IMG=$($PY -c "import json;print(' '.join(map(str,json.load(open('mimo_tiny/ref.json'))['image']['prompt_ids'])))")
+ GRID=$($PY -c "import json;i=json.load(open('mimo_tiny/ref.json'))['image'];print(i['grid_h'],i['grid_w'])")
+ rm -f k3.usage
+ ksan kimi_k3 "asan kimi_k3 tier" $O COLI_USAGE=$PWD/k3.usage ./kimi_k3 kimi_k3_tiny --ids "$K3IDS" --ngen 8
+ ksan kimi_k3 "asan kimi_k3 warm start, shared experts on the device" $O COLI_USAGE=$PWD/k3.usage COLI_VK_DENSE=1 ./kimi_k3 kimi_k3_tiny --ids "$K3IDS" --ngen 8
+ ksan kimi_k3 "asan kimi_k3 eviction, int8 shared experts" K3_BITS=8 K3_IDOT=0 COLI_USAGE=$PWD/k3e.usage USAGE_SAVE=0 COLI_VK_TIER_GB=0.000005 COLI_VK_DENSE=1 ./kimi_k3 kimi_k3_tiny --ids "$K3IDS" --ngen 8
+ ksan kimi_k3 "asan kimi_k3 prefill one token at a time, no pipeline" $O COLI_USAGE=$PWD/k3e.usage USAGE_SAVE=0 K3_CHUNK=1 K3_PIPE=0 ./kimi_k3 kimi_k3_tiny --ids "$K3IDS" --ngen 8
+ ksan mimo "asan mimo tier" MIMO_DENSE_BITS=32 COLI_TEMP=0 ./mimo mimo_tiny --ids "$MIDS" --ngen 6
+ ksan mimo "asan mimo eviction (MIMO_VK_EXPERTS=2)" MIMO_DENSE_BITS=32 COLI_TEMP=0 MIMO_VK_EXPERTS=2 ./mimo mimo_tiny --ids "$MIDS" --ngen 6
+ ksan mimo "asan mimo picture, native dense on the device" MIMO_DENSE_BITS=0 COLI_TEMP=0 COLI_VK_DENSE=1 ./mimo mimo_tiny --ids "$IMG" --ngen 6 --image mimo_tiny/patches.f32 --grid $GRID
+ ksan mimo "asan mimo prefill blocks of 3, cache 4" MIMO_DENSE_BITS=32 COLI_TEMP=0 MIMO_CHUNK=3 MIMO_CAP=4 ./mimo mimo_tiny --ids "$MIDS" --ngen 6
+ make clean >/dev/null 2>&1 || true
}
+ # GLM-5.2 (colibri) and GLM-5.3 Flash (glm53) on the routed-expert tier: the fixtures.
+ # colibri: the bf16 oracle, whose experts the loader quantizes to the bits asked for
+ # (16: f32, fmt 10 on the device; 8: int8, fmt 1; 4: int4 per row, fmt 2; 3: int3-g64,
+ # fmt 5), the int4-g64 and E8/IQ3 containers of the parity gate, and the FP8 oracle
+ # converted to int4-g64 (its 32-wide down rows stay per row), int4 per row, int3-g64 and
+ # int4-g64 with an int3-g64 down. glm53: the int4-gs64 streaming container.
+ glm_fixtures() {
+ $PY tools/make_glm_oracle.py > /dev/null
+ $PY tools/make_glm_oracle.py --fmt4 > /dev/null
+ $PY tools/make_glm_oracle.py --fmt6 > /dev/null
+ mkdir -p glm_fp8 && (cd glm_fp8 && $PY ../tools/make_glm_oracle.py --fp8 > /dev/null)
+ local v
+ for v in "i4:" "i4r:--group-size 0" "i3:--xbits 3" "d3:--down-bits 3"; do
+ # shellcheck disable=SC2086
+ $PY tools/convert_fp8_to_int4.py --indir glm_fp8/glm_tiny --outdir glm_tiny_${v%%:*} \
+ --ebits 4 --io-bits 4 --n-layers 5 --min-free-gb 0 ${v#*:} > /dev/null
+ cp glm_fp8/ref_glm.json glm_tiny_${v%%:*}/
+ done
+ $PY tools/make_glm53_tiny.py --output glm53_tiny --force > /dev/null
+ $PY tools/make_glm53_streaming_pair.py --fixture glm53_tiny --output glm53_stream > /dev/null
+ }
+
+ # glm_tier <tag> <snap> <ref> <env...> -- <argv...>
+ # colibri with the tier against its CPU run: the same greedy tokens (or teacher-forced
+ # predictions, TF=1, mismatches included) and a "[VK] tier colibri run: device N" with
+ # N > 0; EVICT=1: evictions too. The history the tier warms from is <snap>/.coli_usage:
+ # none by default (the tier fills as experts pass by), HIST=1 writes one first from a
+ # greedy run (the warm start, and the device in a teacher-forced prefill). DEV2=1: the
+ # second device's registry served experts ("+ N vk" in the hit-rate line).
+ glm_tier() {
+ local tag=$1 snap=$2 ref=$3; shift 3
+ local envs=() pre=() e; while [ "$1" != "--" ]; do envs+=("$1"); shift; done; shift
+ for e in "${envs[@]}"; do [ "${e%%=*}" = TF ] || pre+=("$e"); done
+ rm -f "$snap/.coli_usage"
+ [ "${HIST:-0}" = 1 ] && { env "${pre[@]}" SNAP=$snap REF=$ref STATS=$snap/.coli_usage ./colibri "$@" > /dev/null 2>&1 || true; }
+ env "${envs[@]}" SNAP=$snap REF=$ref ./colibri "$@" > cpu.log 2>&1 || true
+ env "${envs[@]}" SNAP=$snap REF=$ref COLI_VK_TIER_SYNC=1 COLI_VULKAN=1 ./colibri "$@" > vk.log 2>&1 || true
+ rm -f "$snap/.coli_usage"
+ grep -aE '^GLM C engine|^PREFILL|^\[ORACLE\] mismatch' cpu.log | sed 's/ | [0-9.]* pos\/s//' > cpu.tok || true
+ grep -aE '^GLM C engine|^PREFILL|^\[ORACLE\] mismatch' vk.log | sed 's/ | [0-9.]* pos\/s//' > vk.tok || true
+ { [ -s cpu.tok ] && cmp -s cpu.tok vk.tok; } || { cat cpu.tok vk.tok; tail -20 vk.log; fail "$tag: the tier's tokens differ from the CPU"; }
+ if [ "${DEV2:-0}" = 1 ]; then
+ grep -qaE 'lru \+ [1-9][0-9]* vk /' vk.log || { cat vk.log; fail "$tag: the second device served no expert"; }
+ else
+ [ "$(tier_count colibri vk.log)" -gt 0 ] || { cat vk.log; fail "$tag: no routed expert ran on the device"; }
+ fi
+ if [ "${EVICT:-0}" = 1 ]; then
+ [ "$(tier_evictions colibri vk.log)" -gt 0 ] || { grep '\[VK\] tier' vk.log; fail "$tag: the budget forced no eviction"; }
+ fi
+ echo "OK $tag: tokens = CPU, $(grep -a -o 'device [0-9]* of [0-9]* routed experts' vk.log | tail -1)$(grep -a -o ' + [0-9]* vk' vk.log | tail -1), $(grep -a -o 'evictions [0-9]*' vk.log | tail -1), $(grep -a -o '(fmt [^)]*)' vk.log | head -1)"
+ }
+
+ # g53_tier <tag> <model dir> <env...> -- <argv...>: glm53, the same gate on its
+ # teacher_forcing and greedy lines; the history is a fresh COLI_USAGE file (HIST=1: one
+ # greedy run writes it first).
+ g53_tier() {
+ local tag=$1 model=$2; shift 2
+ local envs=(); while [ "$1" != "--" ]; do envs+=("$1"); shift; done; shift
+ rm -f g53.usage
+ [ "${HIST:-0}" = 1 ] && { env "${envs[@]}" COLI_USAGE=g53.usage ./glm53 --model $model "$@" > /dev/null 2>&1 || true; }
+ env "${envs[@]}" COLI_USAGE=g53.usage USAGE_SAVE=0 ./glm53 --model $model "$@" > cpu.log 2>&1 || true
+ env "${envs[@]}" COLI_USAGE=g53.usage USAGE_SAVE=0 COLI_VK_TIER_SYNC=1 COLI_VULKAN=1 ./glm53 --model $model "$@" > vk.log 2>&1 || true
+ rm -f g53.usage
+ grep -aE '^teacher_forcing|^greedy' cpu.log > cpu.tok || true
+ grep -aE '^teacher_forcing|^greedy' vk.log > vk.tok || true
+ { [ -s cpu.tok ] && cmp -s cpu.tok vk.tok; } || { cat cpu.tok vk.tok; tail -20 vk.log; fail "$tag: the tier's tokens differ from the CPU"; }
+ [ "$(tier_count glm53 vk.log)" -gt 0 ] || { cat vk.log; fail "$tag: no routed expert ran on the device"; }
+ if [ "${EVICT:-0}" = 1 ]; then
+ [ "$(tier_evictions glm53 vk.log)" -gt 0 ] || { grep '\[VK\] tier' vk.log; fail "$tag: the budget forced no eviction"; }
+ fi
+ local where; where=$(dense_where glm53 vk.log "$tag" "${envs[@]}") || { echo "$where"; exit 1; }
+ echo "OK $tag: tokens = CPU, $(grep -a -o 'device [0-9]* of [0-9]* routed experts' vk.log | tail -1), $(grep -a -o 'evictions [0-9]*' vk.log | tail -1), trunk on the $where"
+ }
+
+ # GLM-5.2 and GLM-5.3 Flash on the routed-expert tier, in every expert format the two
+ # engines read, warm and cold, decode and teacher-forced prefill, under eviction (a
+ # budget of two experts, by COLI_VK_TIER_GB and by the deprecated COLI_VK_EXPERTS), with
+ # the trunk and the attention core on the device, beside COLI_VK_DEV2's second
+ # (logical) device, and with the tier off. IDOT=0 gives the CPU's int8 and int4-per-row
+ # experts the f32 activations the device uses (the CPU default rounds them to int8 on
+ # some ISAs), so the two sides differ by summation order only.
+ family_glm() {
+ make colibri glm53 VK=1
+ glm_fixtures
+ export OMP_NUM_THREADS=2 CAP_RAISE=0
+ local cap b
+ # the oracle with f32 experts (fmt 10): decode cold at a one-slot and a full cache,
+ # the teacher-forced prefill from a warm start
+ for cap in 1 64; do glm_tier "colibri f32 experts cap=$cap" glm_tiny ref_glm.json -- $cap 16 16; done
+ HIST=1 glm_tier "colibri f32 experts, prefill from a warm start" glm_tiny ref_glm.json TF=1 -- 64 16 16
+ # quantized at load: int8 (fmt 1), int4 per row (fmt 2), int3-g64 (fmt 5)
+ for b in 8 4 3; do
+ glm_tier "colibri ${b}-bit experts cap=2" glm_tiny ref_glm.json IDOT=0 -- 2 $b 4
+ HIST=1 glm_tier "colibri ${b}-bit experts, prefill warm" glm_tiny ref_glm.json IDOT=0 TF=1 -- 2 $b 4
+ done
+ # the containers: int4-g64 gate/up/down, int4-g64 with a per-row down, int4 per row,
+ # int3-g64, int4-g64 with an int3-g64 down
+ HIST=1 glm_tier "colibri int4-g64 container" glm_tiny_fmt4 glm_tiny_fmt4/ref_glm.json -- 2 16 16
+ HIST=1 glm_tier "colibri int4-g64 container, prefill" glm_tiny_fmt4 glm_tiny_fmt4/ref_glm.json TF=1 -- 2 16 16
+ local fx
+ for fx in i4 i4r i3 d3; do
+ HIST=1 glm_tier "colibri $fx container cap=1" glm_tiny_$fx glm_tiny_$fx/ref_glm.json IDOT=0 -- 1 4 4
+ HIST=1 glm_tier "colibri $fx container, prefill" glm_tiny_$fx glm_tiny_$fx/ref_glm.json IDOT=0 TF=1 -- 2 4 4
+ done
+ # PIPE's asynchronous loads and the PILOT prefetcher beside the tier
+ glm_tier "colibri PIPE=1" glm_tiny_i4 glm_tiny_i4/ref_glm.json IDOT=0 PIPE=1 -- 1 4 4
+ glm_tier "colibri PILOT=1" glm_tiny_i4 glm_tiny_i4/ref_glm.json IDOT=0 PILOT=1 -- 2 4 4
+ # the trunk and the MLA attention core on the device too
+ HIST=1 glm_tier "colibri tier + COLI_VK_DENSE=1 COLI_VK_ATTN=1" glm_tiny_i4 glm_tiny_i4/ref_glm.json IDOT=0 COLI_VK_DENSE=1 COLI_VK_ATTN=1 -- 2 4 4
+ # a budget of two experts must evict as the routing moves: COLI_VK_TIER_GB, and the
+ # deprecated COLI_VK_EXPERTS mapped onto the tier as a cap
+ EVICT=1 glm_tier "colibri, a budget of two experts" glm_tiny ref_glm.json COLI_VK_TIER_GB=0.0001 -- 64 16 16
+ EVICT=1 glm_tier "colibri, COLI_VK_EXPERTS=2" glm_tiny ref_glm.json COLI_VK_EXPERTS=2 -- 64 16 16
+ # COLI_VK_DEV2: a second logical device on Lavapipe holds what the capped tier does not
+ HIST=1 glm_tier "colibri tier + COLI_VK_DEV2" glm_tiny_i4r glm_tiny_i4r/ref_glm.json IDOT=0 COLI_VK_EXPERTS=4 COLI_VK_DEV2=0 -- 64 4 4
+ HIST=1 DEV2=1 glm_tier "colibri COLI_VK_DEV2 alone (COLI_VK_TIER=0)" glm_tiny_i4r glm_tiny_i4r/ref_glm.json IDOT=0 COLI_VK_TIER=0 COLI_VK_DEV2=0 -- 64 4 4
+ # E8/IQ3 (fmt 6) has no device form: the tier declines, the CPU computes them
+ SNAP=glm_tiny_fmt6 REF=glm_tiny_fmt6/ref_glm.json ./colibri 2 16 16 > cpu.log 2>&1 || true
+ SNAP=glm_tiny_fmt6 REF=glm_tiny_fmt6/ref_glm.json COLI_VULKAN=1 ./colibri 2 16 16 > vk.log 2>&1 || true
+ cmp -s <(grep -a '^GLM C engine' cpu.log) <(grep -a '^GLM C engine' vk.log) || { cat vk.log; fail "colibri fmt 6: tokens differ"; }
+ grep -qa 'tier colibri: experts in fmt 6/6/6' vk.log || { cat vk.log; fail "colibri fmt 6: the tier did not decline"; }
+ echo "OK colibri E8/IQ3 (fmt 6): the tier declines, tokens = CPU"
+ # COLI_VK_TIER=0: no tier, the trunk and the attention core on the device as before
+ SNAP=glm_tiny REF=ref_glm.json COLI_VK_DENSE=1 COLI_VK_ATTN=1 ./colibri 2 16 16 > cpu.log 2>&1 || true
+ SNAP=glm_tiny REF=ref_glm.json COLI_VK_DENSE=1 COLI_VK_ATTN=1 COLI_VK_TIER=0 COLI_VULKAN=1 ./colibri 2 16 16 > vk.log 2>&1 || true
+ cmp -s <(grep -a '^GLM C engine' cpu.log) <(grep -a '^GLM C engine' vk.log) || { cat vk.log; fail "colibri COLI_VK_TIER=0: tokens differ"; }
+ ! grep -qa '^\[VK\] tier colibri' vk.log || { cat vk.log; fail "colibri COLI_VK_TIER=0: the tier started"; }
+ echo "OK colibri COLI_VK_TIER=0: tokens = CPU, no tier"
+
+ # glm53: its int4-gs64 streaming container (swiglu_limit 10), dense matrices f32 and int4
+ local ids
+ ids=$($PY -c "print(','.join(str((i*37+5)%120+2) for i in range(100)))")
+ g53_tier "glm53 decode, cold" glm53_stream-i4 GLM53_BITS=32 -- --ids 5,7,9,11,13,17,19,23 --greedy 8
+ HIST=1 g53_tier "glm53 100-token prompt, warm (two blocks of rows)" glm53_stream-i4 GLM53_BITS=32 -- --ids $ids --greedy 4
+ HIST=1 g53_tier "glm53 prefill chunks of 7, one cache slot" glm53_stream-i4 GLM53_BITS=4 GLM53_PREFILL_CHUNK=7 GLM53_EXPERT_GB=0.000001 -- --ids $ids --greedy 4
+ HIST=1 g53_tier "glm53 trunk on the device (COLI_VK_DENSE=1)" glm53_stream-i4 GLM53_BITS=4 COLI_VK_DENSE=1 -- --ids $ids --greedy 4
+ EVICT=1 g53_tier "glm53, a budget of two experts" glm53_stream-i4 GLM53_BITS=32 COLI_VK_TIER_GB=0.00006 -- --ids $ids --greedy 6
+ # COLI_VK_TIER=0: the trunk alone, on the device by default, Lavapipe included
+ GLM53_BITS=4 COLI_USAGE=g53.usage USAGE_SAVE=0 ./glm53 --model glm53_stream-i4 --ids 5,7,9,11 --greedy 4 > cpu.log 2>&1 || true
+ GLM53_BITS=4 COLI_USAGE=g53.usage USAGE_SAVE=0 COLI_VK_TIER=0 COLI_VULKAN=1 ./glm53 --model glm53_stream-i4 --ids 5,7,9,11 --greedy 4 > vk.log 2>&1 || true
+ cmp -s <(grep -aE '^teacher_forcing|^greedy' cpu.log) <(grep -aE '^teacher_forcing|^greedy' vk.log) || { cat vk.log; fail "glm53 COLI_VK_TIER=0: tokens differ"; }
+ ! grep -qa '^\[VK\] tier glm53' vk.log || { cat vk.log; fail "glm53 COLI_VK_TIER=0: the tier started"; }
+ need_gpu glm53 vk.log "glm53 COLI_VK_TIER=0"
+ echo "OK glm53 COLI_VK_TIER=0: tokens = CPU, no tier, $(vk_count glm53 vk.log) matmuls on the GPU"
+ unset OMP_NUM_THREADS CAP_RAISE
+ }
+
+ # The same engines under ASan and UBSan with the tier on: memory safety is the gate, and
+ # each run must still put experts on the device.
+ family_glm_sanitize() {
+ make clean >/dev/null 2>&1 || true
+ make colibri glm53 VK=1 EXTRA_CFLAGS="-fsanitize=address,undefined -fno-omit-frame-pointer -g"
+ glm_fixtures
+ export OMP_NUM_THREADS=2 CAP_RAISE=0
+ gsan() { # <engine> <tag> <history: colibri snap, or - for glm53's COLI_USAGE> <env and argv...>
+ local eng=$1 tag=$2 snap=$3; shift 3
+ rm -f g53.usage; [ "$snap" = - ] || rm -f "$snap/.coli_usage"
+ if [ "${HIST:-0}" = 1 ]; then # a history from a greedy run (TF=1 writes none)
+ local a=() x; for x in "$@"; do [ "$x" = TF=1 ] || a+=("$x"); done
+ if [ "$snap" = - ]; then env COLI_USAGE=g53.usage "${a[@]}" > /dev/null 2>&1 || true
+ else env STATS=$snap/.coli_usage "${a[@]}" > /dev/null 2>&1 || true; fi
+ fi
+ env ASAN_OPTIONS=detect_leaks=0 UBSAN_OPTIONS=print_stacktrace=1 COLI_USAGE=g53.usage \
+ COLI_VK_TIER_SYNC=1 COLI_VULKAN=1 "$@" > san.log 2>&1 || true
+ rm -f g53.usage; [ "$snap" = - ] || rm -f "$snap/.coli_usage"
+ if grep -qE "ERROR: AddressSanitizer|runtime error:" san.log; then cat san.log; fail "$tag: sanitizer diagnostic"; fi
+ [ "$(tier_count $eng san.log)" -gt 0 ] || { cat san.log; fail "$tag: no routed expert ran on the device"; }
+ echo "OK $tag: sanitizers clean, $(grep -a -o 'device [0-9]* of [0-9]* routed experts' san.log | tail -1), $(grep -a -o 'evictions [0-9]*' san.log | tail -1)"
+ }
+ gsan colibri "asan colibri f32 cap=1" glm_tiny SNAP=glm_tiny REF=ref_glm.json ./colibri 1 16 16
+ HIST=1 gsan colibri "asan colibri prefill, warm" glm_tiny SNAP=glm_tiny REF=ref_glm.json TF=1 ./colibri 64 16 16
+ gsan colibri "asan colibri eviction" glm_tiny SNAP=glm_tiny REF=ref_glm.json COLI_VK_EXPERTS=2 ./colibri 64 16 16
+ HIST=1 gsan colibri "asan colibri int4-g64, trunk + attention on the device" glm_tiny_i4 SNAP=glm_tiny_i4 REF=glm_tiny_i4/ref_glm.json COLI_VK_DENSE=1 COLI_VK_ATTN=1 ./colibri 2 4 4
+ HIST=1 gsan colibri "asan colibri int3-g64 prefill" glm_tiny_i3 SNAP=glm_tiny_i3 REF=glm_tiny_i3/ref_glm.json TF=1 ./colibri 2 4 4
+ HIST=1 gsan colibri "asan colibri int3-g64 down, cap=1" glm_tiny_d3 SNAP=glm_tiny_d3 REF=glm_tiny_d3/ref_glm.json ./colibri 1 4 4
+ HIST=1 gsan colibri "asan colibri tier + COLI_VK_DEV2" glm_tiny_i4r SNAP=glm_tiny_i4r REF=glm_tiny_i4r/ref_glm.json COLI_VK_EXPERTS=4 COLI_VK_DEV2=0 ./colibri 64 4 4
+ gsan colibri "asan colibri int4 at load, PIPE=1" glm_tiny SNAP=glm_tiny REF=ref_glm.json PIPE=1 ./colibri 1 4 4
+ gsan colibri "asan colibri int8 at load, PILOT=1" glm_tiny SNAP=glm_tiny REF=ref_glm.json PILOT=1 ./colibri 2 8 8
+ local ids
+ ids=$($PY -c "print(','.join(str((i*37+5)%120+2) for i in range(100)))")
+ gsan glm53 "asan glm53 decode" - GLM53_BITS=32 ./glm53 --model glm53_stream-i4 --ids 5,7,9,11,13,17,19,23 --greedy 20
+ HIST=1 gsan glm53 "asan glm53 100-token prompt, warm" - GLM53_BITS=32 ./glm53 --model glm53_stream-i4 --ids $ids --greedy 4
+ HIST=1 gsan glm53 "asan glm53 one slot, trunk on the device" - GLM53_BITS=4 GLM53_EXPERT_GB=0.000001 COLI_VK_DENSE=1 ./glm53 --model glm53_stream-i4 --ids $ids --greedy 4
+ gsan glm53 "asan glm53 eviction" - GLM53_BITS=32 COLI_VK_TIER_GB=0.00006 ./glm53 --model glm53_stream-i4 --ids $ids --greedy 6
+ unset OMP_NUM_THREADS CAP_RAISE
+ make clean >/dev/null 2>&1 || true
+ }
+
case "${1:-}" in
shader) shader_formats ;;
qwen) family_qwen ;;
qwen-sanitize) family_qwen_sanitize ;;
inkling-olmoe) family_inkling_olmoe ;;
+ inkling-olmoe-sanitize) family_inkling_olmoe_sanitize ;;
mimo-qwenimage) family_mimo_qwenimage ;;
deepseek) family_deepseek ;;
+ deepseek-sanitize) family_deepseek_sanitize ;;
+ kimi) family_kimi ;;
+ kimi-mimo-sanitize) family_kimi_mimo_sanitize ;;
- *) echo "usage: $0 shader|qwen|qwen-sanitize|inkling-olmoe|inkling-olmoe-sanitize|mimo-qwenimage|deepseek|deepseek-sanitize|kimi|kimi-mimo-sanitize" >&2; exit 2 ;;
+ glm) family_glm ;;
+ glm-sanitize) family_glm_sanitize ;;
- *) echo "usage: $0 shader|qwen|qwen-sanitize|inkling-olmoe|mimo-qwenimage|deepseek|glm|glm-sanitize" >&2; exit 2 ;;
++ *) echo "usage: $0 shader|qwen|qwen-sanitize|inkling-olmoe|inkling-olmoe-sanitize|mimo-qwenimage|deepseek|deepseek-sanitize|kimi|kimi-mimo-sanitize|glm|glm-sanitize" >&2; exit 2 ;;
esac
diff --cc c/vk_tier.h
index 61f87e2d,2d7d89f7..2486c8eb
--- a/c/vk_tier.h
+++ b/c/vk_tier.h
@@@ -170,16 -166,12 +173,20 @@@ int vkt_put(int layer, int eid, const
void vkt_put_done(void);
void vkt_note(int layer, int eid, const VktExpertSrc *src);
int vkt_issue(int layer, const float *x, int S, int K, const int *idx, uint8_t *taken);
+/* vkt_issue with the route weights w[S*K] (beside idx), for VKT_ACT_SWIGLU_V4, which
+ * applies them on the device; the other activations ignore them. */
+int vkt_issue_w(int layer, const float *x, int S, int K, const int *idx, const float *w, uint8_t *taken);
+/* 1 when vkt_note would take this expert now (not on the device or on its way, the
+ * promotion rate not spent, room or a colder resident to displace): for an engine
+ * whose RAM form must be converted before vkt_note can read it, so it converts only
+ * the experts the tier will take. No side effects. */
+int vkt_wants(int layer, int eid);
int vkt_join(const float **rows);
int vkt_resident(int layer, int eid);
+ /* The next vkt_issue starts a forward (promotion rate, decay, eviction candidates).
+ * Optional: the tier tells forwards apart by the layer index going back, which a model
+ * with a single MoE layer never does; such an engine calls this at its first MoE layer. */
+ void vkt_begin_forward(void);
void vkt_report(const char *scope, unsigned long long ram_hits, unsigned long long disk_loads);
/* Sizing helpers for engines: bytes one expert takes on the device in a source format. */
size_t vkt_expert_bytes(int hidden, int inter, VktFmt gate_up, VktFmt down);
@@@ -193,10 -185,9 +200,11 @@@ static inline int vkt_put(int l,int e,
static inline void vkt_put_done(void){}
static inline void vkt_note(int l,int e,const VktExpertSrc *s){(void)l;(void)e;(void)s;}
static inline int vkt_issue(int l,const float *x,int S,int K,const int *i,uint8_t *t){(void)l;(void)x;(void)S;(void)K;(void)i;(void)t;return 0;}
+static inline int vkt_issue_w(int l,const float *x,int S,int K,const int *i,const float *w,uint8_t *t){(void)l;(void)x;(void)S;(void)K;(void)i;(void)w;(void)t;return 0;}
+static inline int vkt_wants(int l,int e){(void)l;(void)e;return 0;}
static inline int vkt_join(const float **r){(void)r;return 0;}
static inline int vkt_resident(int l,int e){(void)l;(void)e;return 0;}
+ static inline void vkt_begin_forward(void){}
static inline void vkt_report(const char *s,unsigned long long r,unsigned long long d){(void)s;(void)r;(void)d;}
static inline size_t vkt_expert_bytes(int h,int i,VktFmt a,VktFmt b){(void)h;(void)i;(void)a;(void)b;return 0;}
#endif
diff --cc docs/ENVIRONMENT.md
index 772ef804,602535cf..f4381659
--- a/docs/ENVIRONMENT.md
+++ b/docs/ENVIRONMENT.md
@@@ -214,10 -214,10 +214,10 @@@ Per-drive byte counts are reported in
| Variable | Default | Effect |
|---|---|---|
-| `COLI_VULKAN` | off | Enable the Vulkan backend. Requires a `make VK=1` build. The GLM engine fails at startup (no silent fallback) if libvulkan or the compiled shaders are missing. The other engines (qwen36, qwen38, inkling, olmoe, deepseek_v41, deepseek_v4, mimo, qwenimage) print one line and run on the CPU. What each one puts on the device: [vulkan.md](vulkan.md#the-other-engines). |
+| `COLI_VULKAN` | off | Enable the Vulkan backend. Requires a `make VK=1` build. The GLM engine fails at startup (no silent fallback) if libvulkan or the compiled shaders are missing. The other engines (qwen36, qwen38, inkling, olmoe, deepseek_v41, deepseek_v4, kimi_k3, mimo, qwenimage) print one line and run on the CPU. What each one puts on the device: [vulkan.md](vulkan.md#the-other-engines). |
| `COLI_VK_DEV` | unset | Select the primary Vulkan physical-device enumeration index. Without it, the backend prefers a discrete GPU, then integrated/virtual devices. |
| `COLI_VK_SHADERS` | auto | Path to the compiled `qmatmul.spv` **or** the directory holding the `.spv` set; the other shaders are found next to it. Unset: `shaders/` next to the binary, then CWD-relative `shaders/qmatmul.spv`. |
- | `COLI_VK_EXPERTS` | `320` | Pinned VRAM expert tier size: top-N experts by `.coli_usage` heat uploaded once at startup and served from VRAM with no RAM slot or disk read. `0` disables the tier (experts stay on the CPU path). ~19 MB VRAM per int4 expert. |
+ | `COLI_VK_EXPERTS` | unset | Deprecated. The GLM engine's routed experts are on the shared expert tier (below), sized by its budget. `N`: the tier holds at most N experts (it was the size of a fixed top-N set, 320 by default); `0`: no tier, as `COLI_VK_TIER=0`. ~19 MB per int4 expert. |
| `COLI_VK_DENSE` | `0` in this engine | Run the resident dense matmuls (attention projections, shared expert) on the GPU. The other engines read the same variable with a default of on (see the expert tier below); all of them go through `coli_vk_dense_decide()` in the backend. |
| `COLI_VK_ATTN` | `0` | Run the S≤4 MLA absorb attention core (+ fused o-projection) on the GPU, with a persistent device-side KV mirror. |
| `COLI_VK_QPREP` | `1` (on) | Fuse the Q-prep step (RMSNorm + rope + compress) into one GPU dispatch instead of splitting it, which cost three fences where one suffices. `0` restores the split path; `2` additionally keeps CPU reference copies of Q and comp for A/B comparison. |
@@@ -229,9 -229,9 +229,9 @@@
| `COLI_VK_GEMM_TILE` | measured | `bm,bn,bk,tm,tn[,pf]`: one tile for every width of the fp32 GEMM instead of the measured pair. Tuning only (`COLI_VK_TEST_GEMM_BENCH`). |
| `COLI_VK_COOP_TILE` | measured | `bm,bn,wm,wn,bk`: one tile for every width of the cooperative-matrix GEMM. Tuning only. |
- ### The routed-expert tier (`vk_tier.c`; qwen36, qwen38, inkling, olmoe, kimi_k3, mimo, deepseek_v41, deepseek_v4)
-### The routed-expert tier (`vk_tier.c`; qwen36, qwen38, colibri, glm53)
++### The routed-expert tier (`vk_tier.c`; every MoE engine: qwen36, qwen38, inkling, olmoe, kimi_k3, mimo, deepseek_v41, deepseek_v4, colibri, glm53)
- With `COLI_VULKAN=1` the engines on the shared tier keep a cache of routed experts on the device that adapts while you chat, and compute the resident ones of each layer step while the CPU computes the rest. See [vulkan.md](vulkan.md#the-routed-expert-tier-vk_tierc). With the CUDA expert tier also on, CUDA wins and this tier stays off.
+ With `COLI_VULKAN=1` the engines on the shared tier keep a cache of routed experts on the device that adapts while you chat, and compute the resident ones of each layer step while the CPU computes the rest. See [vulkan.md](vulkan.md#the-routed-expert-tier-vk_tierc), and for the two GLM engines (the formats, `COLI_VK_EXPERTS`, glm53's `swiglu_limit`) [vulkan.md](vulkan.md#glm-52-and-glm-53-flash-on-the-tier). With the CUDA expert tier also on, CUDA wins and this tier stays off.
| Variable | Default | Effect |
|---|---|---|
@@@ -244,7 -244,7 +244,7 @@@
| `COLI_VK_TIER_BALANCE` | on | `0`: the device takes every resident expert of a step. On, when a join keeps waiting for the device, the step's resident experts the CPU also holds in RAM go back to the CPU beyond the device's share, which moves with every join (down when the CPU waited more than a tenth of its own time, up when the device finished before 80% of it). |
| `COLI_VK_TIER_GEMM_ROWS` | `16` | Rows from which an expert of a step takes the tiled GEMM (prefill) instead of the per-row GEMV; `0` never. Below it a row's bits do not depend on how many rows share the dispatch. |
| `COLI_VK_TIER_QUEUE` | a second queue | `0`: the tier's batches share the main queue with the dense matmuls (they then serialize). Unset: a second queue of the main family, else a compute-only family's (RADV), else shared (Lavapipe). |
- | `COLI_VK_DENSE` | on; off on a device sharing the CPU's RAM while the tier is on | Where the dense (non-expert) matrices of qwen36, qwen38, inkling, olmoe, mimo, deepseek_v41, deepseek_v4 and qwenimage run. `0`: on the CPU (with the tier on, the device takes the routed experts only); `1`: on the device whatever the device. Unset: on the device, except on an integrated GPU or a CPU device (Lavapipe) while the engine runs the expert tier (today qwen36, qwen38, inkling, olmoe, kimi_k3, mimo, deepseek_v41 and deepseek_v4), where they stay on the CPU: there the dense matmuls cost more than the tier gains (measured on a Radeon 780M). The `[VK] <engine>: device ready, dense matrices on the ...` line says which and why. The GLM engine reads it through the same rule with its own default, off (see above). |
-| `COLI_VK_DENSE` | on; off on a device sharing the CPU's RAM while the tier is on | Where the dense (non-expert) matrices of qwen36, qwen38, inkling, olmoe, mimo, deepseek_v41, deepseek_v4 and qwenimage run. `0`: on the CPU (with the tier on, the device takes the routed experts only); `1`: on the device whatever the device. Unset: on the device, except on an integrated GPU or a CPU device (Lavapipe) while the engine runs the expert tier (today qwen36, qwen38 and glm53), where they stay on the CPU: there the dense matmuls cost more than the tier gains (measured on a Radeon 780M). The `[VK] <engine>: device ready, dense matrices on the ...` line says which and why. The GLM engine reads it through the same rule with its own default, off (see above). |
++| `COLI_VK_DENSE` | on; off on a device sharing the CPU's RAM while the tier is on | Where the dense (non-expert) matrices of qwen36, qwen38, inkling, olmoe, kimi_k3 (its shared experts), mimo, deepseek_v41, deepseek_v4, glm53 and qwenimage run. `0`: on the CPU (with the tier on, the device takes the routed experts only); `1`: on the device whatever the device. Unset: on the device, except on an integrated GPU or a CPU device (Lavapipe) while the engine runs the expert tier (every one of them but qwenimage, which has no routed experts), where they stay on the CPU: there the dense matmuls cost more than the tier gains (measured on a Radeon 780M). The `[VK] <engine>: device ready, dense matrices on the ...` line says which and why. The GLM engine reads it through the same rule with its own default, off (see above). |
### Second Vulkan device (opt-in)
diff --cc docs/vulkan.md
index 94ad588a,7cdbd638..bd6d0df3
--- a/docs/vulkan.md
+++ b/docs/vulkan.md
@@@ -101,28 -101,27 +101,28 @@@ runs the routed-expert tier ([below](#t
stay on the CPU and the device takes the experts. On a Radeon 780M the dense matmuls,
one synchronous call each at the GPU's 800 MHz floor, cost more than the tier gained
(Qwen3.8 decode at 2.35 tok/s with them on the device, 3.80 without). Today that
- case is qwen36, qwen38, inkling, olmoe, kimi_k3, mimo, deepseek_v41 and deepseek_v4;
-case is qwen36, qwen38 and glm53; an engine that moves to the tier inherits it. A discrete
-GPU keeps the dense matrices on the device by default. The GLM engine above reads
-the same variable through the same function with its own default, off.
++case is every engine below with routed experts, that is all of them but qwenimage;
+an engine that moves to the tier inherits it. A discrete GPU keeps the dense matrices
+on the device by default. The GLM engine above reads the same variable through the
+same function with its own default, off.
What these engines put on the device is their **resident** matrices, in the form
-they already hold in RAM, uploaded at the first multiply (MiMo uploads them at
-startup). Routed experts arrive from disk on every miss; qwen36, qwen38, glm53 and the GLM
-engine above keep a cache of them on the device with the shared expert tier
-([below](#the-routed-expert-tier-vk_tierc)), MiMo an opt-in one of its own, the
-others none yet.
+they already hold in RAM, uploaded at the first multiply (MiMo and Kimi K3 upload
- them at startup). Routed experts arrive from disk on every miss; qwen36, qwen38,
- inkling, olmoe, Kimi K3, MiMo, deepseek_v41 and deepseek_v4 keep a cache of them on
- the device with the shared expert tier ([below](#the-routed-expert-tier-vk_tierc)),
- the others none yet.
++them at startup). Routed experts arrive from disk on every miss; every engine below
++that has them, and the GLM engine above, keeps a cache of them on the device with
++the shared expert tier ([below](#the-routed-expert-tier-vk_tierc)).
| Engine | On the device | Weight formats | Stays on the CPU |
|---|---|---|---|
| qwen36 (Qwen3.6, Qwen3-Coder, Qwen3.8-27B) | the dense trunk; routed experts on the expert tier | int8 rows; int4-g64 with `COLI_DENSE_BITS=4`; f32 with `COLI_DENSE_I8=0`; experts int4-g64, int4 per row, int8 per row or gs64 | DeltaNet `dn_a`/`dn_b`, vision tower, the experts the tier does not hold |
| qwen38 (Qwen3.8 Flash Next) | the trunk; routed experts on the expert tier | int8 trunk rows, bf16, f32 (`Q38_NATIVE_BF16=0`); experts int4-g64 (sidecar), FP8 128x128 blocks, bf16 | the MTP head's experts, the experts the tier does not hold |
+| inkling | dense and shared-expert matrices; routed experts on the expert tier | int8 and int4-g64 (dense-int4g64 container), f32, bf16; experts int4 or int8 per row (container or runtime quantization), f32 | embedding and audio lookups, CUDA residents (with CUDA or Metal on, the experts too); bf16 on CPUs with the AVX512-BF16 dot (see below); the experts the tier does not hold |
+| olmoe | attention q/k/v/o, router, lm_head; routed experts on the expert tier | f32; experts int8 per row | embedding, the experts the tier does not hold |
+| deepseek_v41 | the trunk, vision included; routed experts on the expert tier | fp8 in 32x32 ue8m0 tiles, bf16; experts MXFP4 (fp4, a ue8m0 scale per 32) | the DSpark stages and their experts, the experts the tier does not hold |
+| deepseek_v4 | resident dense layers, head, router, compressors; routed experts on the expert tier | fp8 in 128x128 blocks, bf16; experts MXFP4 (fp4, a ue8m0 scale per 32) with [an activation of its own](#deepseek-v4s-activation) | the indexer's `weights_proj`, DSpark stages, the `--oracle` path's dense layers, the experts the tier does not hold |
+| kimi_k3 (Kimi K3) | the shared experts' matrices (one row at a time: decode); routed experts on the expert tier | shared experts int8 rows, int4-g64, f32 (`K3_BITS`); experts MXFP4 with ue8m0 scales (fmt 7), SiTU-GLU in the latent space | KDA, MLA, the latent projections, router, head, prefill's shared experts, the experts the tier does not hold |
+| mimo | trunk and vision tower; routed experts on the expert tier | native fp8/bf16, int8, f32 (`MIMO_DENSE_BITS`); experts MXFP4 with e8m0 scales (fmt 7) | router, the experts the tier does not hold |
+ | glm53 (GLM-5.3 Flash) | the resident matrices (an f32 checkpoint's experts among them); the streaming container's routed experts on the expert tier | int8 and int4-g64 (`GLM53_BITS`); experts int4-gs64 | f32 matrices (`GLM53_BITS=32`), the streamed experts the tier does not hold, all of them when `swiglu_limit` is 0 |
-| inkling | dense and shared-expert matrices | int8 and int4-g64 (dense-int4g64 container), f32, bf16 | routed experts, embedding and audio lookups, CUDA residents; bf16 on CPUs with the AVX512-BF16 dot (see below) |
-| olmoe | attention q/k/v/o, router, lm_head | f32 | routed experts, embedding |
-| deepseek_v41 | the trunk, vision included | fp8 in 32x32 ue8m0 tiles, bf16 | routed experts |
-| deepseek_v4 | resident dense layers, head, router, compressors | fp8 in 128x128 blocks, bf16 | routed experts, the indexer's `weights_proj`, DSpark stages, the `--oracle` path |
-| mimo | trunk and vision tower; up to `MIMO_VK_EXPERTS=N` routed experts | native fp8/bf16, int8, f32 (`MIMO_DENSE_BITS`); experts as MXFP4 | router |
| qwenimage | the DiT's matrices | int8, bf16, f32 (`COLI_IMG_BITS`) | text encoder, VAE, attention |
Each engine ends a run, and each serve turn, with
@@@ -319,25 -307,15 +319,27 @@@ of them on the device the way a GPU-equ
Engines on the tier today: **qwen36** (Qwen3.6, Qwen3-Coder, the 2.4T geometry: int8
per row or gs64, int4 per row or gs64 from either expert kernel, the mixed int4/int8
- container) and **qwen38** (Qwen3.8 Flash Next: the int4-g64 sidecar, the release's
- FP8 with 128x128 block scales, BF16). The MTP head's layer of qwen38 stays on the CPU
- (its experts are FP8 beside an int4 sidecar). **inkling** and **olmoe** as well, see
+ container), **qwen38** (Qwen3.8 Flash Next: the int4-g64 sidecar, the release's
+ FP8 with 128x128 block scales, BF16), **colibri** (GLM-5.2: f32, int8, int4 per row
+ and int3-g64 experts quantized at load, the int4-gs and int3-g64 containers) and
+ **glm53** (GLM-5.3 Flash: the int4-gs64 streaming container); see
+ [GLM-5.2 and GLM-5.3 Flash](#glm-52-and-glm-53-flash-on-the-tier). The MTP head's layer
+ of qwen38 stays on the CPU (its experts are FP8 beside an int4 sidecar), as does
-GLM-5.2's (int8 beside int4 experts). Kimi K3's `K3_VK` and MiMo's `MIMO_VK_EXPERTS`
-are the older per-engine tiers; the others move to this one in the next phase, see
-[Adding an engine](#adding-an-engine-to-the-tier).
++GLM-5.2's (int8 beside int4 experts). **inkling** and **olmoe** as well, see
+[Inkling and OLMoE](#inkling-and-olmoe). **kimi_k3** (Kimi K3: the checkpoint's
+MXFP4 with ue8m0 scales, SiTU-GLU on the device, the experts in the latent space,
+warm-started from its expert history) and **mimo** (MiMo-V2.6 Flash and Pro: the
+release's MXFP4, SwiGLU; no history, so the tier fills as experts pass by) joined
+it from their own tiers, which filled once and never evicted: Kimi K3's `K3_VK`,
+`K3_VK_GB`, `K3_VK_UP` and MiMo's `MIMO_VK_EXPERTS` are now read as this tier's
+switches ([ENVIRONMENT.md](ENVIRONMENT.md)). Both add every pair of a row in the
+CPU run's order (Kimi K3: its union's disk-offset order; MiMo: the row's routing
+order), and offer the experts the CPU computed after the join, so that a promotion
+that displaces a resident frees it at once. **deepseek_v41** (DeepSeek V4.1 Flash) and
+**deepseek_v4** (DeepSeek V4 Flash) put their fp4 experts (MXFP4 with ue8m0 scales) on
+it as the checkpoint stores them; their DSpark stages keep their own experts on the
+CPU, and deepseek_v4's experts take [an activation of their own](#deepseek-v4s-activation).
- GLM-5.2's `COLI_VK_EXPERTS` is the older
- per-engine tier; the others move to this one in the next phase, see [Adding an
- engine](#adding-an-engine-to-the-tier).
++Every MoE engine is on the tier.
With the CUDA expert tier built and on (`COLI_CUDA=1`) as well, **CUDA wins**: the
Vulkan tier stays off and says so (`[VK] tier <engine>: the CUDA expert tier is on
@@@ -576,26 -517,60 +578,80 @@@ converts `expert_ffn.h`'s planar int4 a
backend enables the extension when the device has it; nothing outside the harness
uses it.
+### Inkling and OLMoE
+
+Both run their routed experts on this tier with `COLI_VULKAN=1`, in the form their
+RAM caches hold them; nothing is requantized.
+
+| Engine | Experts in RAM (`VktSrc`) | Device | History for the warm start | Notes |
+|---|---|---|---|---|
+| inkling | int4 container `I4U_PAIRS_ROW`; int8 container `I8_ROW`; runtime-quantized int8 rows, `I8_AS_I4_ROW` at 2 to 4 bits and `I8_ROW` above; f32 at `bits=0` `F32`. Gate and up come from the fused `gate_up` tensor, up I rows in | fmt 2, 1, 2 or 1, 10 | `<snap>/.coli_usage`, or `PIN=<path>`, in the generate and serve modes; the ref.json oracle reads none and fills the tier as experts pass by | the two shared experts run beside the batch, on the CPU or, with the dense matrices, on the device; `TOPP`'s trimmed ranks reach neither side; with CUDA or Metal on, the Vulkan tier stays off |
+| olmoe | int8 rows `I8_ROW`, gate, up and down as the merged container holds them | fmt 1 | `COLI_USAGE` | `PILOT`'s prefetcher skips experts the device holds; a slot is handed to the tier only under the cache lock, while it still holds that expert |
+
+Each layer step routes every row first, then runs per block of 64 rows: the block's
+resident experts go to the device as one batch, the CPU computes the other pairs with
+the kernels and the cache rounds it always uses, and every rank joins its row in
+routing order. The device multiplies f32 activations, as both engines' default CPU
+kernels do; `IDOT=1` (opt-in on both) rounds the CPU's activations and the device's
+not. With `COLI_VULKAN` unset, or in a build without `VK=1`, the stdout and the
+stderr (timings aside) and every logit vector of 36 configurations per build (each
+expert format, caps 1 to 8, `TOPP`, `IDOT`, the generate and serve modes, `PPL`,
+`ROUTE_TRACE`) are the bytes of the build before the tier.
+
+ ### GLM-5.2 and GLM-5.3 Flash on the tier
+
+ **GLM-5.2 (`colibri`).** With `COLI_VULKAN=1` the routed experts go to the tier, in the
+ form the loader holds them: f32 (`./colibri <cap> 16`), int8 and int4 per row and
+ int3-g64 when a bf16 or FP8 checkpoint is quantized at load, the int4-gs container
+ (`convert_fp8_to_int4.py`'s default, gs 64) and int3-g64. Gate and up share one format,
+ down may have its own (`--down-bits 3`; an int4-g64 container's rows narrower than the
+ group stay per row). E8/IQ3 experts (fmt 6, whose input is rotated), int2 and fp8 have
+ no device form and stay on the CPU (`[VK] tier colibri: experts in fmt 6/6/6 ... stay
+ on the CPU`), and so does the MTP head's layer (int8). The tier serves every row count:
+ decode, the MTP and n-gram verify rows and the batched prefill, by blocks of 64 rows; the
+ fixed set it replaces served S <= 4 only.
+
+ - *The sum.* Without `COLI_VULKAN`, `moe()` adds a token's experts in the order of the
+ batch's union, as before (the default build's stdout and the teacher-forced logits of
+ every fixture configuration are the base commit's bytes). With the tier on, every
+ expert of a row joins it in routing (rank) order, the device's and the CPU's alike,
+ then the shared expert.
+ - *What the CPU computes beside the batch* goes through the same pin set, LRU and disk
+ loads (`PIPE` included) as the CPU path, and every expert it computes is offered to
+ the tier. A device-served expert takes no RAM slot and no disk read.
+ - *Arithmetic.* The device multiplies f32 activations. The CPU's int8 dot (`IDOT`,
+ on by default) rounds the activations to int8 for int8 experts, and for int4-per-row
+ experts from two rows (from one on AVX-512 VNNI and ARM dot-product builds): there
+ the tier's experts match `IDOT=0`, which the tests set.
+ - *The old names.* `COLI_VK_EXPERTS` (the count of the fixed set, 320 by default) is a
+ deprecated alias: `N` caps the tier at N experts, `0` turns it off as `COLI_VK_TIER=0`
+ does, and a line says so. `COLI_VK_RESERVE_GB` (the old reserve, 3 GB) adds what it
+ asks beyond `COLI_VK_TIER_RESERVE_GB`. With `COLI_VK_ATTN=1` the absorb core's KV
+ mirror (`CTX` rows a layer) is reserved too. The dense set (`COLI_VK_DENSE=1`) is
+ uploaded before the tier sizes itself, so the budget is what remains; the trunk's
+ default here stays off. On an integrated GPU the tier takes its default share of RAM
+ after the expert cache's (the startup reservation of the fixed set is gone).
+ - *`COLI_VK_DEV2`*: the second device's registry, fixed at startup, takes the hottest
+ experts of the history that the tier does not hold (with the tier off, the hottest
+ ones), up to `COLI_VK_EXPERTS2`; a step sends them there as one group on a worker
+ thread while the tier's batch and the CPU run.
+
+ **GLM-5.3 Flash (`glm53`).** The streaming container's int4-gs64 experts go to the tier;
+ the resident matrices follow the dense rule above (on a device that shares the CPU's RAM
+ they stay on the CPU while the tier is on; before, `COLI_VULKAN=1` always put them on the
+ device). An f32 checkpoint, whose experts are resident matrices, has no tier. The
+ experts' SwiGLU is clamped (`swiglu_limit`): `swiglu_clamped` clamps at any limit,
+ including 0, where it zeroes every routed expert's output, while the shader clamps only
+ above 0; so the tier runs only for a limit above 0 and says why otherwise. The shared
+ expert is written first, then every rank of every row in routing order. Its tiny
+ fixtures have a single MoE layer, whose forwards the tier cannot tell apart by the
+ layer index going back: the engine marks each forward's start (`vkt_begin_forward`).
+
+ Both print `[VK] tier colibri|glm53 run:` at the end of a run and a `turn` line after
+ each serve turn; `EMAP` shows a tier-resident expert as tier 2. No GPU has run GLM
+ weights on the tier yet: `tests/vulkan_engines.sh glm` proves the tokens on Lavapipe,
+ nothing about speed.
+
## Adding an engine to the tier
The integration steps are in [`c/vk_tier.h`](../c/vk_tier.h); in short:
@@@ -623,13 -595,17 +679,6 @@@
4. **Report**: `vkt_report("run"|"turn", ram_hits, disk_loads)` beside the engine's
`[VK]` line; `vkt_resident(l, e)` gives EMAP its tier 2.
--What each remaining engine needs, from reading its code:
--
--| Engine | Experts in RAM (`VktSrc`) | Activation | Where | To watch |
--|---|---|---|---|---|
- | colibri.c (GLM-5.2) | int4 per row `I4U_PAIRS_ROW`, int4-gs `I4U_PAIRS_GS`, int3-g64 `I3_G64`; gate, up, down separate | SwiGLU | `moe()`'s Vulkan block replaces the `COLI_VK_EXPERTS` registry | sums per expert in union order today: move to rank order; fmt 6 (E8/IQ3, rotated input) stays on the CPU; the MTP layer is int8 (another format); the block now serves only S <= 4 |
- | glm53 | int4 gs64 `I4U_PAIRS_GS` 64 | SwiGLU with `swiglu_limit` | `ffn_layer` | its CPU clamps even at limit 0 (no guard), the shader treats 0 as no clamp: pass the config's value and check `L > 0` on the CPU side first; shared expert written first |
-| inkling | one `gate_up` tensor [2I, D], rows 0..I-1 gate: pass `g = p13`, `u = p13 + I rows`; int4 per row `I4U_PAIRS_ROW` or int8 `I8_ROW` | SwiGLU | `moe` | routed and shared weights normalized together, `route_scale x rgs` already in the weights |
-| kimi_k3 | MXFP4 `MXFP4_E8M0` 32, gate `w1`, up `w3`, down `w2` | SiTU-GLU, a = 4, b = 25 (`VKT_ACT_SITU`) | `moe_forward` / `expert_apply` (experts in the latent space) | replaces `K3_VK` (synchronous, never evicts); the CPU's `K3_IDOT` rounds activations to int8 |
-| deepseek_v41 | MXFP4 `MXFP4_E8M0` 32 | SwiGLU with limit | `moe_run_at` (already sums per (row, rank) in rank order) | DSpark stages have caches of their own |
-| deepseek_v4 | FP4 + ue8m0/32 = `MXFP4_E8M0` 32; pinned experts are repacked rows16 | SwiGLU with limit, **plus** bf16 rounding of gate/up, the route weight applied before down and bf16 rounding of the output | `moe_token_pipeline` (ascending expert id), `v4_moe_batch_union` | needs an activation variant with those roundings and the host's E4M3 rounding of x (as its fp8 dense path does); undo rows16 or keep pinned experts off the tier; hash-routed layers |
-| mimo | MXFP4 `MXFP4_E8M0` 32, stored down, ds, gate, gs, up, us | SwiGLU | `moe` | replaces `MIMO_VK_EXPERTS` (synchronous, never evicts) |
-| olmoe | int8 per row `I8_ROW`, one merged tensor g, u, d | SwiGLU | `moe` | history only with `COLI_USAGE` |
--
## Correctness
- `gcc -O3 -DVK_TEST backend_vulkan.c -o test_vk -lvulkan -lm && ./test_vk
@@@ -651,22 -627,10 +700,25 @@@
independent of the batch, and the tier pool's budget with frees while a batch is
in flight. `tests/vulkan_engines.sh qwen` gives the CPU's tokens with the tier on
in every qwen36 and qwen38 expert format, under eviction, with MTP and with the
- trunk on the CPU; `qwen-sanitize` runs the same under ASan and UBSan. `glm` does the
- same for colibri and glm53 (every expert format, warm and cold, prefill, eviction,
- the trunk and attention core on the device, `COLI_VK_DEV2`), `glm-sanitize` under
- ASan and UBSan.
+ trunk on the CPU; `qwen-sanitize` runs the same under ASan and UBSan.
+ `tests/vulkan_engines.sh inkling-olmoe` does the same for inkling (every expert
+ format above, under the f32, bf16 and dense-int4g64 snapshots, `TOPP`) and olmoe
+ (with `PILOT`'s worker), under eviction, with a warm start, and through both
+ engines' serve tests (prefix reuse, the dashboard, Brio); `inkling-olmoe-sanitize`
+ runs them under ASan and UBSan. `kimi` and the mimo half of `mimo-qwenimage` do it
+ for Kimi K3 and MiMo: Moonshot's and Xiaomi's oracles with the tier on, the CPU's
+ tokens in every dense format with the trunk on the device and on the CPU, a budget
+ of two experts that must evict, the old switches, Kimi K3's warm start;
+ `kimi-mimo-sanitize` runs the tier's configurations under ASan and UBSan.
+ `tests/vulkan_engines.sh deepseek` does it for deepseek_v41 (every tiny oracle, the
+ 40-token prompt, DSpark, eviction) and deepseek_v4 (the three oracle cases on a 4-
+ and an 8-expert fixture with pinned rows16 experts, eviction, the warm start, the
+ served logprobs), and `deepseek-sanitize` under ASan and UBSan. DeepSeek V4's
+ activation has a case of its own in the harness, checked bit for bit, and in
+ `vk-tier-check`.
++ `tests/vulkan_engines.sh glm` does the same for colibri and glm53 (every expert
++ format, warm and cold, prefill, eviction, the trunk and attention core on the
++ device, `COLI_VK_DEV2`), `glm-sanitize` under ASan and UBSan.
- int4 weights decode as offset-binary (nibble−8), byte-identical layout to
the CPU path — no repacking.
- Khronos validation layers: the backend never enables them, so the loader