Run frontier MoE models on hardware you already own — pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦
C
test(vulkan): the partial-glm family, its CI legs and docs
JustVuggb929056e5f26
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.ymlindex 9e9e36ff..9a8b7f0b 100644--- a/.github/workflows/ci.yml+++ b/.github/workflows/ci.yml@@ -297,6 +297,15 @@ jobs:- family: dense-only-glm-sanitizereq: c/tools/requirements-qwen38-tiny.txtextra: pillow+ # the partial chain of colibri and glm53 (the first N layers on the device):+ # COLI_VK_CHAIN_LAYERS, COLI_VK_DEVICE_CAP_MB, an upload failing mid-setup, the+ # forward paths with N < L, and the same under ASan and UBSan+ - family: partial-glm+ req: c/tools/requirements-glm53-tiny.txt+ extra: pillow+ - family: partial-glm-sanitize+ req: c/tools/requirements-glm53-tiny.txt+ extra: pillow- family: dense-only-deepseekreq: c/tools/requirements-deepseek-v4-tiny.txtextra: sympy pillowdiff --git a/c/glm53_chain.h b/c/glm53_chain.hindex 9cd95b1c..7baad590 100644--- a/c/glm53_chain.h+++ b/c/glm53_chain.h@@ -48,7 +48,20 @@* segment), a session above the device's limits, and matrices or a geometry its shaders* do not take (kv_lora above 1024, qk_rope above 0 is not GLM-5.3's and is declined too,* a KDA head above 256 key or 128 value floats, an indexer above 64 heads, 4096 query- * floats or k-pooling below 2). */+ * floats or k-pooling below 2).+ *+ * A partial chain (vk_chain.h, vkc_fit; docs/vulkan.md, "A partial chain"): when the+ * device's free memory does not hold every layer, the chain takes layers 0..N-1 and the+ * CPU runs N..L-1, the final collapse, norm and head (the head stays on the host unless+ * the whole chain and the head fit). N is decided at load, once the device is open and+ * before the expert cache and the tier size themselves (g53c_start, from+ * model_load_range); with COLI_VK_DENSE_HOST only the N layers give their host copies+ * back, each once all of it reached the device. A forward runs every chunk of its rows+ * through the N device layers, then run_layers goes on from layer N with the hc_mult+ * streams (H x D floats a row, once per chunk). The KDA state and windows on the device,+ * the MLA mirror, its watermarks and the KV split cover the N layers; the CPU layers'+ * state is the host's alone, so a lost device rebuilds the device layers' KDA state+ * only. */#include "vk_chain.h"#include "vk_kvsplit.h"@@ -782,7 +795,16 @@ static void g53c_start(GModel *m) {vkc_fit_placed("glm53", &g_g53c_fit);g53_dho_finish(m, g_g53c_fit.n, g_g53c_fit.tail, g_g53_partial);}+/* A partial chain at exit: the matrices the device holds, which must be the ones the+ * setup placed (the per-matrix path put nothing of a CPU layer or the head there). */+static void g53c_held_atexit(void) {+ size_t w = 0, nt = 0;+ coli_vk_mem_info(&w, &nt);+ fprintf(stderr, "[VK] glm53 chain: %d of %d layers held at exit: %zu B of matrices on the device\n",+ g_g53c_fit.n, g_g53c_fit.L, w);+}/* After the tier's: at exit the chain goes before the device. */static void g53c_atexit(void) {if (g_g53c_inited) atexit(vkc_shutdown);+ if (g_g53c_fitted && g_g53_partial) atexit(g53c_held_atexit);}diff --git a/c/glm_chain.h b/c/glm_chain.hindex 9ee5b186..16925de0 100644--- a/c/glm_chain.h+++ b/c/glm_chain.h@@ -674,9 +674,18 @@ static void glmc_start(Model *m) {vkc_fit_placed("colibri", &g_glmc_fit);glm_dho_finish(m);}+/* A partial chain at exit: the matrices the device holds, which must be the ones the+ * setup placed (the per-matrix path put nothing of a CPU layer or the tail there). */+static void glmc_held_atexit(void) {+ size_t w = 0, nt = 0;+ coli_vk_mem_info(&w, &nt);+ fprintf(stderr, "[VK] colibri chain: %d of %d layers held at exit: %zu B of matrices on the device\n",+ g_glmc_fit.n, g_glmc_fit.L, w);+}/* After the tier's: at exit the report runs first, then the chain goes, then the device. */static void glmc_atexit(Model *m) {if (!g_glmc_inited) return;atexit(vkc_shutdown);if (g_vk_chain) { g_glmc_model = m; atexit(glmc_report_atexit); }+ if (g_glmc_fitted && g_glmc_partial) atexit(glmc_held_atexit);}diff --git a/c/tests/vulkan_partial_glm.sh b/c/tests/vulkan_partial_glm.shnew file mode 100644index 00000000..c7d542bd--- /dev/null+++ b/c/tests/vulkan_partial_glm.sh@@ -0,0 +1,401 @@+# Sourced by tests/vulkan_engines.sh: the partial chain (docs/vulkan.md, "A partial chain")+# on GLM-5.2 (colibri) and GLM-5.3 Flash (glm53): the first N layers on the device, the CPU+# the rest and the head. Run as partial-glm and partial-glm-sanitize.+#+# Every configuration runs the glm-chain family's gate (mla_gate: the CPU's token lines,+# every logits row within 1e-4 of the largest, chain forwards) with the partial chain's+# settings on top, then checks the lines (ptl_* helpers in tests/vulkan_engines.sh):+# - COLI_VK_CHAIN_LAYERS=k for k in 0, 1, about L/2, L-1, L: N = k, the matrices on the+# device after setup are those k layers' (the placed line), and with k < L the+# per-matrix path put nothing more there (the held line at exit);+# - COLI_VK_DEVICE_CAP_MB aimed at k layers from a probe's fit line: the rule's N from+# the run's own numbers, and the line's, are k;+# - a staged upload failing inside layer k's setup (COLI_VK_STAGED_FAULT=submit:n, n+# from a probe's per-layer counts): N = k, nothing of layer k on the device;+# - the forward paths with N < L: prompt chunks, expert streaming, the DSA selection+# handed to the CPU at a shared indexer layer, n-gram and MTP drafts, prompts only,+# the per-matrix path beside, the tier off, the KV split, a lost device mid-decode, in+# the prompt, between drafts and after an image (glm53's KDA state rebuilt for the+# device's layers), serve sessions with the prompt cache and two KV slots, glm53's+# image and its pin-branch harness;+# - the dense weights on the device only with N < L: only the N layers' matrices drop+# their host copies (counted from the config), none is read back on a healthy run,+# the device layers' come back after a loss, the RAM the experts get grows with N;+# - everything fitting: the fit takes every layer, the same bits as asking for all.+# The fixtures: glm-chain's, glm_tiny with shared indexer layers (layers 1, 3 and 4 reuse+# the selection of the full layer before them) and a six-layer GLM-5.3 (KDA and MLA+# alternating, so MLA layers run on both sides of the handoff).++# ptl_glm_shx <src> <dst>: a copy of glm_tiny whose layers 1, 3 and 4 reuse the DSA+# selection of the full layer before them (the CPU's engine reads indexer_types)+ptl_glm_shx() {+ rm -rf "$2" && cp -r "$1" "$2"+ $PY - "$2/config.json" <<'PY'+import json, sys+c = json.load(open(sys.argv[1])); c["indexer_types"] = ["full", "shared", "full", "shared", "shared"]+json.dump(c, open(sys.argv[1], "w"))+PY+}+# ptl_g53_l6 <dir>: the GLM-5.3 tiny with six layers, KDA and MLA alternating, the first+# MLP dense; no oracle (the engine's runs are compared with each other)+ptl_g53_l6() {+ $PY - tools/make_glm53_tiny.py "$1" <<'PY' > /dev/null+import importlib.util, sys+spec = importlib.util.spec_from_file_location("gen", sys.argv[1])+gen = importlib.util.module_from_spec(spec); spec.loader.exec_module(gen)+gen.LAYERS = 6+kinds = ["linear_attention", "deepseek_sparse_attention"] * 3+ck, rc = gen.config_kwargs, gen.runtime_config+def config_kwargs():+ k = ck()+ k["layer_types"], k["mlp_layer_types"], k["indexer_types"] = kinds, ["dense"] + ["sparse"] * 5, ["full"] * 6+ return k+def runtime_config(version, fp8_mlp=False):+ r = rc(version, fp8_mlp)+ r["text_config"]["linear_attn_config"]["kda_layers"] = [i for i, t in enumerate(kinds) if t == "linear_attention"]+ return r+gen.config_kwargs, gen.runtime_config = config_kwargs, runtime_config+gen.oracle = lambda *a: {"note": "no reference: the engine's runs are compared with each other"}+sys.argv = ["make_glm53_tiny.py", "--output", sys.argv[2], "--force"]+gen.main()+PY+}+ptl_glm_fixtures() {+ glm_chain_fixtures+ pf_ref glm_tiny/ref_long.json 200 1 250+ ptl_glm_shx glm_tiny glm_tiny_shx+ ptl_g53_l6 glm53_l6+ $PY tools/make_glm53_streaming_pair.py --fixture glm53_l6 --output glm53_l6s > /dev/null+ rm -rf glm53_l6s-deq glm53_l6_serve && cp -r glm53_l6s-i4 glm53_l6_serve+ $PY tools/make_edge_tiny_tokenizer.py --vocab-size 128 ./glm53_l6_serve > /dev/null+}++# ptl_mla_run <out> <engine> <env...> -- <argv...>: one chained run, its stderr kept (a probe)+ptl_mla_run() {+ local out=$1 eng=$2; shift 2+ local envs=(); while [ "$1" != "--" ]; do envs+=("$1"); shift; done; shift+ rm -f chain.usage+ env "${envs[@]}" USAGE_SAVE=0 COLI_USAGE=chain.usage COLI_VK_TIER_SYNC=1 COLI_VULKAN=1 COLI_VK_CHAIN=${CHAINMODE:-1} \+ ./"$eng" "$@" > /dev/null 2> "$out" || true+ rm -f chain.usage+ grep -q "^\[VK\] $eng chain fit:" "$out" || { tail -20 "$out"; fail "probe run of $eng: no fit line"; }+}+# ptl_mla_held <engine> <log> <tag>: N < L, the matrices on the device at exit are the ones+# the setup placed (the held line)+ptl_mla_held() {+ local s x+ s=$(ptl_calc matrices "$1" "$2" | cut -d' ' -f1)+ x=$(sed -n "s/^\[VK\] $1 chain: [0-9]* of [0-9]* layers held at exit: \([0-9]*\) B of matrices on the device$/\1/p" "$2" | tail -1)+ { [ -n "$x" ] && [ "$s" = "$x" ]; } || { grep -a '^\[VK\]' "$2"; fail "$3: $s B of matrices at setup, ${x:-no line} at exit: a CPU layer or the tail went up"; }+}+# ptl_mla_off <engine> <tag> <env...> -- <argv...>: N = 0, the chain off: the CPU's tokens+# and logits, no chain forward, nothing of the trunk on the device at setup or at exit+ptl_mla_off() {+ local eng=$1 tag=$2; shift 2+ local envs=(); while [ "$1" != "--" ]; do envs+=("$1"); shift; done; shift+ rm -f chain.usage cpu.f32 vk.f32+ env "${envs[@]}" COLI_USAGE=chain.usage USAGE_SAVE=0 DUMP=cpu.f32 ./"$eng" "$@" > cpu.log 2>&1 || true+ rm -f chain.usage+ env "${envs[@]}" COLI_USAGE=chain.usage USAGE_SAVE=0 DUMP=vk.f32 COLI_VK_TIER_SYNC=1 COLI_VULKAN=1 \+ COLI_VK_CHAIN=${CHAINMODE:-1} ./"$eng" "$@" > vk.log 2>&1 || true+ rm -f chain.usage+ mla_toks "$eng" cpu.log > cpu.tok; mla_toks "$eng" vk.log > vk.tok+ { [ -s cpu.tok ] && cmp -s cpu.tok vk.tok; } || { cat cpu.tok vk.tok; tail -20 vk.log; fail "$tag: the tokens differ from the CPU"; }+ local lg; lg=$(logits_close cpu.f32 vk.f32) || { echo "$lg"; fail "$tag: logits"; }+ ptl_check_n "$eng" vk.log 0 "$tag"+ [ "$(chain_count "$eng" vk.log)" = 0 ] || { grep -a '^\[VK\]' vk.log; fail "$tag: the chain ran"; }+ ptl_check_placed "$eng" vk.log "$tag"+ ptl_mla_held "$eng" vk.log "$tag"+ [ "$(ptl_calc matrices "$eng" vk.log | cut -d' ' -f1)" = 0 ] || { grep -a '^\[VK\]' vk.log; fail "$tag: a matrix of the trunk on the device with the chain off"; }+ echo "OK $tag: N = 0, the chain off, tokens = CPU, $lg, no matrix of the trunk on the device"+}+# ptl_mla <engine> <tag> <k|-> <env...> -- <argv...>: mla_gate with COLI_VK_CHAIN_LAYERS=k+# (-: the fit decides), then N = PTL_WANT (default k), the placed check, and with N < L+# nothing more on the device at exit than at setup. KV=<rows>:<block>: kv_mla_gate (the+# KV split) instead of mla_gate.+ptl_mla() {+ local eng=$1 tag=$2 k=$3; shift 3+ local want=${PTL_WANT:-$k} e=()+ [ "$k" != - ] && e=(COLI_VK_CHAIN_LAYERS=$k)+ if [ "$want" = 0 ]; then ptl_mla_off "$eng" "$tag" "${e[@]}" "$@"; return; fi+ if [ -n "${KV:-}" ]; then kv_mla_gate "$eng" "$tag" 1 "${KV%%:*}" "${KV##*:}" "${e[@]}" "$@"+ else mla_gate "$eng" "$tag" 1 "${e[@]}" "$@"; fi+ ptl_check_n "$eng" vk.log "$want" "$tag"+ ptl_check_placed "$eng" vk.log "$tag"+ local L; L=$(ptl_L "$eng" vk.log)+ [ "$want" -ge "$L" ] || ptl_mla_held "$eng" vk.log "$tag"+ echo " N = $want of $L, $(ptl_calc matrices "$eng" vk.log | cut -d' ' -f1) B of matrices on the device after setup"+}+# ptl_mla_cap <engine> <tag> <k> <env...> -- <argv...>: COLI_VK_DEVICE_CAP_MB aimed at k+# layers from a probe at 256 MiB with the same settings; the reserve holds the device's+# own scratch beside what the tier takes+ptl_mla_cap() {+ local eng=$1 tag=$2 k=$3; shift 3+ local R=COLI_VK_TIER_RESERVE_GB=0.04 cap pred+ ptl_mla_run ptl-probe.err "$eng" COLI_VK_DEVICE_CAP_MB=256 $R "$@"+ cap=$(PTL_PROBE_CAP_MB=256 ptl_calc cap "$eng" ptl-probe.err "$k") || fail "$tag: no cap from the probe"+ PTL_WANT=$k ptl_mla "$eng" "$tag (cap $cap MiB)" - COLI_VK_DEVICE_CAP_MB="$cap" $R "$@"+ pred=$(ptl_calc predict "$eng" vk.log)+ [ "$pred" = "$k" ] || { grep -a "^\[VK\] $eng chain" vk.log; fail "$tag: the rule gives $pred from the run's own numbers"; }+}+# ptl_mla_fault <engine> <tag> <k> <env...> -- <argv...>: a staged upload fails inside+# layer k's setup (every layer asked for): the chain keeps the layers before it+ptl_mla_fault() {+ local eng=$1 tag=$2 k=$3; shift 3+ local L n+ ptl_mla_run ptl-probe.err "$eng" COLI_VK_STAGED=1 COLI_VK_STAGED_FAULT=submit:1000000 COLI_VK_CHAIN_LAYERS=99 "$@"+ L=$(ptl_L "$eng" ptl-probe.err)+ n=$(ptl_calc fault "$eng" ptl-probe.err "$k") || { grep -a "^\[VK\] $eng chain" ptl-probe.err; fail "$tag: no fault count from the probe"; }+ PTL_WANT=$k ptl_mla "$eng" "$tag (submit #$n)" "$L" COLI_VK_STAGED=1 COLI_VK_STAGED_FAULT=submit:"$n" "$@"+ grep -q "$eng chain: $k of $L layers on the device.*layer $k did not reach the device" vk.log ||+ { grep -a "^\[VK\] $eng chain" vk.log; fail "$tag: no line for layer $k's failure"; }+}+# ptl_mla_droppable <engine> <snapshot> <k>: the matrices of layers 0..k-1 that drop their+# host copies (glm_dho_layer, g53_dho_layer): colibri's five attention matrices, three of+# the MLP or the shared expert, the DSA indexer's three on a full layer; glm53's KDA layer+# twelve, its MLA layer thirteen (the absorbed kv_b's two halves among them). KEEP_KVB=1:+# the chain does not run every forward (prompts only): kv_b and the indexer stay+ptl_mla_droppable() {+ $PY - "$1" "$2/config.json" "$3" "${KEEP_KVB:-0}" <<'PY'+import json, sys+eng, c, k, keep = sys.argv[1], json.load(open(sys.argv[2])), int(sys.argv[3]), sys.argv[4] == "1"+c = c.get("text_config", c)+if eng == "colibri":+ kinds = c.get("indexer_types") or ["full"] * c["num_hidden_layers"]+ print(sum(7 + (0 if keep else 1 + (3 if kinds[i] == "full" else 0)) for i in range(k)))+else:+ print(sum(12 if c["layer_types"][i] == "linear_attention" else 11 + (0 if keep else 2) for i in range(k)))+PY+}+# ptl_mla_dho <engine> <tag> <k> <snapshot> <env...> -- <argv...>: the dense weights on the+# device only with N = k < L: the k layers' matrices alone drop their host copies, the+# placed line says so, none comes back on a healthy run (FAULT_BACK: the device layers' do)+ptl_mla_dho() {+ local eng=$1 tag=$2 k=$3 snap=$4; shift 4+ ptl_mla "$eng" "$tag" "$k" COLI_VK_DENSE_HOST=0 "$@"+ local n want L r+ n=$(dho_dropped vk.log); want=$(ptl_mla_droppable "$eng" "$snap" "$k"); L=$(ptl_L "$eng" vk.log); r=$(dho_reloaded vk.log)+ [ "$n" = "$want" ] || { grep -a '^\[VK\]' vk.log; fail "$tag: $n host copies dropped, the $k layers have $want"; }+ grep -q "dense matrices on the device only, .* (the $k of $L layers on the device; the $((L - k)) on the CPU keep theirs)" vk.log ||+ { grep -a '^\[VK\]' vk.log; fail "$tag: the placed line does not say the $k of $L layers"; }+ if [ -n "${FAULT_BACK:-}" ]; then+ { [ "$r" -gt 0 ] && [ "$r" -le "$n" ]; } || { grep -a '^\[VK\]' vk.log; fail "$tag: $r read back after the loss (the device held $n alone)"; }+ else [ "$r" = 0 ] || { grep -a '^\[VK\]' vk.log; fail "$tag: $r matrices read back from disk on a healthy run"; }; fi+ echo " $n host copies dropped (the $k layers' $want), $r read back"+}+# ptl_mla_same <engine> <tag> <env A> <env B> <env...> -- <argv...>: two chained runs that+# differ only in env A / env B (each one string, KEY=VALUE words): the same tokens and the+# same logits, bit for bit (COLI_VK_TIER_BALANCE=0: the tier's split of the routed experts+# with the CPU follows the timings, and with it which side's bits a row gets)+ptl_mla_same() {+ local eng=$1 tag=$2 a=$3 b=$4; shift 4+ local envs=(); while [ "$1" != "--" ]; do envs+=("$1"); shift; done; shift+ local x+ for x in a b; do+ local v; [ $x = a ] && v=$a || v=$b+ rm -f chain.usage+ # shellcheck disable=SC2086+ env "${envs[@]}" $v COLI_USAGE=chain.usage USAGE_SAVE=0 DUMP=same-$x.f32 COLI_VK_TIER_SYNC=1 COLI_VK_TIER_BALANCE=0 COLI_VULKAN=1 \+ COLI_VK_CHAIN=${CHAINMODE:-1} ./"$eng" "$@" > same-$x.log 2>&1 || true+ mla_toks "$eng" same-$x.log > same-$x.tok+ done+ rm -f chain.usage+ { [ -s same-a.tok ] && cmp -s same-a.tok same-b.tok; } || { cat same-a.tok same-b.tok; fail "$tag: the tokens differ"; }+ cmp -s same-a.f32 same-b.f32 || fail "$tag: the logits differ"+ echo "OK $tag: the same tokens and logits ($a | $b)"+}+# ptl_mla_resident <log>: colibri's "resident dense" MB after the load (what cap_for_ram counts)+ptl_mla_resident() { grep -ao 'resident dense: [0-9.]* MB' "$1" | head -1 | sed 's/[^0-9.]//g'; }++ptl_family_glm() {+ make colibri glm53 VK=1+ ptl_glm_fixtures+ export OMP_NUM_THREADS=2 CAP_RAISE=0+ local G="SNAP=glm_tiny REF=ref_glm.json" X="SNAP=glm_tiny_shx REF=ref_glm.json" T="SNAP=glm_tiny_mtp REF=glm_tiny_mtp/ref_glm.json" k+ # ---- colibri: N forced, k = 0, 1, L/2, L-1, L, and every format+ for k in 0 1 2 4 5; do ptl_mla colibri "partial colibri COLI_VK_CHAIN_LAYERS=$k" $k $G -- 64 16 16; done+ ptl_mla colibri "partial colibri 8-bit trunk, 2 of 5 layers" 2 $G IDOT=0 -- 2 8 8+ ptl_mla colibri "partial colibri 4-bit trunk, 3 of 5 layers" 3 $G IDOT=0 -- 2 4 4+ ptl_mla colibri "partial colibri i4 container, 1 of 5 layers" 1 SNAP=glm_tiny_i4 REF=glm_tiny_i4/ref_glm.json IDOT=0 -- 1 4 4+ # the DSA selection handed over: layer N reuses the device's selection (shared indexer+ # layers 1, 3 and 4), and a full layer N makes its own (2)+ for k in 1 2 3 4; do ptl_mla colibri "partial colibri DSA top-4, shared indexers, $k of 5 layers" $k $X DSA_TOPK=4 -- 64 16 16; done+ ptl_mla colibri "partial colibri DSA top-4 prefill in chunks of 5, 1 of 5 layers" 1 $X DSA_TOPK=4 TF=1 COLI_VK_CHAIN_ROWS=5 -- 64 16 16+ ptl_mla colibri "partial colibri DSA_FORCE, 3 of 5 layers" 3 $X DSA_FORCE=1 -- 64 16 16+ # ---- everything fits: the fit takes every layer, the same bits as asking for all of them+ PTL_WANT=5 ptl_mla colibri "partial colibri the fit, everything fitting" - $G -- 64 16 16+ ptl_mla_same colibri "partial colibri the fit = every layer asked for" "X=1" "COLI_VK_CHAIN_LAYERS=5" $G -- 64 16 16+ # ---- a small device: N from the budget+ for k in 1 4 0; do ptl_mla_cap colibri "partial colibri a device for $k layers" $k $G -- 64 16 16; done+ # ---- a staged upload failing in the middle of the setup+ for k in 0 2 4; do ptl_mla_fault colibri "partial colibri an upload failing in layer $k" $k $G -- 64 16 16; done+ # ---- the forward paths with N < L+ ptl_mla colibri "partial colibri prefill in chunks of 3" 2 $G TF=1 COLI_VK_CHAIN_ROWS=3 -- 64 16 16+ ptl_mla colibri "partial colibri cold experts streamed" 3 $PF_FORCE COLI_VK_TIER_GB=0.0003 COLI_VK_CHAIN_ROWS=64 SNAP=glm_tiny REF=glm_tiny/ref_long.json TF=1 -- 2 16 16+ pf_check colibri vk.log "partial colibri cold experts streamed" forced+ ptl_mla colibri "partial colibri n-gram drafts" 2 $G DRAFT=3 -- 64 16 16+ for k in 1 3; do ptl_mla colibri "partial colibri MTP depth 3, $k of 5 layers" $k $T DRAFT=3 -- 64 16 16; done+ ptl_mla colibri "partial colibri MTP with DSA top-4" 2 $T DSA_TOPK=4 -- 64 16 16+ ptl_mla colibri "partial colibri tier off" 2 $G COLI_VK_TIER=0 -- 64 16 16+ ptl_mla colibri "partial colibri beside COLI_VK_DENSE=1 COLI_VK_ATTN=1" 2 SNAP=glm_tiny_i4 REF=glm_tiny_i4/ref_glm.json IDOT=0 COLI_VK_DENSE=1 COLI_VK_ATTN=1 -- 2 4 4+ CHAINMODE=2 ptl_mla colibri "partial colibri prompts only, drafts" 2 $G DRAFT=3 -- 64 16 16+ KV=8:4 ptl_mla colibri "partial colibri the KV split" 3 $G -- 64 16 16+ KV=8:4 ptl_mla colibri "partial colibri the KV split, DSA top-4 prefill in chunks of 5" 3 $X DSA_TOPK=4 TF=1 COLI_VK_CHAIN_ROWS=5 -- 64 16 16+ FAULT_BACK=12 ptl_mla colibri "partial colibri device lost mid-decode" 2 $G -- 64 16 16+ FAULT_BACK=2 ptl_mla colibri "partial colibri device lost in the prompt" 3 $G TF=1 COLI_VK_CHAIN_ROWS=5 -- 64 16 16+ FAULT_BACK=8 ptl_mla colibri "partial colibri device lost with MTP" 2 $T DRAFT=2 -- 64 16 16+ # ---- the chain against itself with N < L: chunks+ ptl_mla_same colibri "partial colibri chunks of 1 = 5" "COLI_VK_CHAIN_ROWS=1" "COLI_VK_CHAIN_ROWS=5" COLI_VK_CHAIN_LAYERS=2 $X DSA_TOPK=4 TF=1 -- 64 16 16+ # ---- the dense weights on the device only+ for k in 1 3; do ptl_mla_dho colibri "partial colibri device only, $k of 5 layers" $k glm_tiny $G -- 64 16 16; done+ ptl_mla_dho colibri "partial colibri device only, MTP" 2 glm_tiny_mtp $T DRAFT=2 -- 64 16 16+ ptl_mla_dho colibri "partial colibri device only, DSA top-4 shared" 1 glm_tiny_shx $X DSA_TOPK=4 -- 64 16 16+ # prompts only: decode runs per matrix, the device layers' kv_b and indexer stay on the+ # host (the CPU's attention reads them by row)+ CHAINMODE=2 KEEP_KVB=1 ptl_mla_dho colibri "partial colibri device only, prompts only, drafts" 2 glm_tiny $G DRAFT=3 -- 64 16 16+ COLI_VK_STAGED=1 ptl_mla_dho colibri "partial colibri device only, staged" 2 glm_tiny $G -- 64 16 16+ FAULT_BACK=12 ptl_mla_dho colibri "partial colibri device only, device lost mid-decode" 2 glm_tiny $G -- 64 16 16+ # the RAM cap_for_ram counts: the host copies dropped are the N layers', resident dense+ # falls with N (the pins and the cap count only what was dropped)+ local r0 r2 r5+ for k in 0 2 5; do+ rm -f chain.usage+ env $G USAGE_SAVE=0 COLI_USAGE=chain.usage COLI_VK_TIER_SYNC=1 COLI_VULKAN=1 COLI_VK_CHAIN=1 COLI_VK_DENSE_HOST=0 \+ COLI_VK_CHAIN_LAYERS=$k ./colibri 64 16 16 > ptl-ram$k.log 2>&1 || true+ done+ rm -f chain.usage+ r0=$(ptl_mla_resident ptl-ram0.log); r2=$(ptl_mla_resident ptl-ram2.log); r5=$(ptl_mla_resident ptl-ram5.log)+ $PY -c "import sys; a,b,c=map(float,sys.argv[1:]); sys.exit(not (a > b > c))" "$r0" "$r2" "$r5" ||+ { grep -a 'resident dense\|dense matrices' ptl-ram*.log; fail "partial colibri device only: resident dense $r0 / $r2 / $r5 MB with 0 / 2 / 5 layers"; }+ echo "OK partial colibri device only: resident dense $r0 MB with 0 layers on the device, $r2 MB with 2, $r5 MB with 5"+ # serve sessions frame for frame with 2 of 5 layers: pins, the prompt cache, the prefill+ # read-out, drafts, two KV slots (the ragged batch declines the chain), prompts only+ export CHAIN_SERVE_EXPECT='colibri chain: 2 of 5 layers on the device'+ CHAIN_SERVE_DIALECT=colibri $PY tests/vulkan_chain_serve.py ./colibri glm_tiny_serve SERVE_BATCH=1 IDOT=0 COLI_VK_CHAIN_LAYERS=2+ CHAIN_SERVE_DIALECT=colibri $PY tests/vulkan_chain_serve.py ./colibri glm_tiny_serve SERVE_BATCH=1 IDOT=0 DSA_TOPK=4 DRAFT=3 COLI_VK_CHAIN_LAYERS=2+ CHAIN_SERVE_SLOTS=2 CHAIN_SERVE_DIALECT=colibri $PY tests/vulkan_chain_serve.py ./colibri glm_tiny_serve SERVE_BATCH=1 IDOT=0 KV_SLOTS=2 COLI_VK_CHAIN_LAYERS=2+ COLI_VK_CHAIN=2 CHAIN_SERVE_DIALECT=colibri $PY tests/vulkan_chain_serve.py ./colibri glm_tiny_serve SERVE_BATCH=1 IDOT=0 DSA_TOPK=4 COLI_VK_CHAIN_LAYERS=2+ CHAIN_SERVE_EXPECT='colibri: [0-9]+ dense matrices on the device only.*the 2 of 5 layers on the device' COLI_VK_DENSE_HOST=0 \+ CHAIN_SERVE_DIALECT=colibri $PY tests/vulkan_chain_serve.py ./colibri glm_tiny_serve SERVE_BATCH=1 IDOT=0 COLI_VK_CHAIN_LAYERS=2+ unset CHAIN_SERVE_EXPECT++ # ---- glm53: the six-layer fixture (KDA and MLA alternating), N forced+ local ids lids F="--model glm53_l6s-i4" M="--model glm53_mm_tiny --ids 103,117,268,268,268,268,120,121 --patches glm53_mm_tiny/patches.f32 --grid 4x4" bits+ ids=$($PY -c "print(','.join(str((i*37+5)%120+2) for i in range(100)))")+ lids=$(pf_ids 200 2 120)+ for k in 0 1 3 5 6; do ptl_mla glm53 "partial glm53 COLI_VK_CHAIN_LAYERS=$k" $k GLM53_BITS=32 -- $F --ids $ids --greedy 4; done+ for bits in 8 4; do ptl_mla glm53 "partial glm53 ${bits}-bit trunk, 3 of 6 layers" 3 GLM53_BITS=$bits -- $F --ids $ids --greedy 4; done+ ptl_mla glm53 "partial glm53 decode, 2 of 6 layers" 2 GLM53_BITS=32 -- $F --ids 5,7,9,11,13,17,19,23 --greedy 8+ ptl_mla glm53 "partial glm53 resident experts, 4 of 6 layers" 4 GLM53_BITS=32 -- --model glm53_l6 --ids $ids --greedy 6+ ptl_mla glm53 "partial glm53 four layers, 2 on the device" 2 GLM53_BITS=32 -- --model glm53_stream-i4 --ids $ids --greedy 4+ ptl_mla glm53 "partial glm53 swiglu_limit 0, 2 of 4 layers" 2 GLM53_BITS=32 -- --model glm53_lim0 --ids $ids --greedy 4+ # ---- everything fits+ PTL_WANT=6 ptl_mla glm53 "partial glm53 the fit, everything fitting" - GLM53_BITS=32 -- $F --ids $ids --greedy 4+ ptl_mla_same glm53 "partial glm53 the fit = every layer asked for" "X=1" "COLI_VK_CHAIN_LAYERS=6" GLM53_BITS=32 -- $F --ids $ids --greedy 4+ # ---- a small device; an upload failing in the middle of the setup+ for k in 1 4 0; do ptl_mla_cap glm53 "partial glm53 a device for $k layers" $k GLM53_BITS=32 -- $F --ids $ids --greedy 4; done+ for k in 0 2 5; do ptl_mla_fault glm53 "partial glm53 an upload failing in layer $k" $k GLM53_BITS=4 -- $F --ids $ids --greedy 4; done+ # ---- the forward paths with N < L+ ptl_mla glm53 "partial glm53 prefill chunks of 7, one cache slot" 3 GLM53_BITS=4 GLM53_PREFILL_CHUNK=7 GLM53_EXPERT_GB=0.000001 -- $F --ids $ids --greedy 4+ ptl_mla glm53 "partial glm53 chain chunks of 3" 4 GLM53_BITS=32 COLI_VK_CHAIN_ROWS=3 -- $F --ids $ids --greedy 4+ ptl_mla glm53 "partial glm53 cold experts streamed" 3 $PF_FORCE COLI_VK_TIER_GB=0.0002 COLI_VK_CHAIN_ROWS=64 GLM53_BITS=4 GLM53_PREFILL_CHUNK=200 -- $F --ids "$lids" --greedy 4+ pf_check glm53 vk.log "partial glm53 cold experts streamed" forced+ ptl_mla glm53 "partial glm53 tier off" 3 GLM53_BITS=32 COLI_VK_TIER=0 -- $F --ids $ids --greedy 6+ ptl_mla glm53 "partial glm53 beside COLI_VK_DENSE=1" 3 GLM53_BITS=4 COLI_VK_DENSE=1 -- $F --ids $ids --greedy 4+ ptl_mla glm53 "partial glm53 an image, 2 of 4 layers" 2 GLM53_BITS=32 -- $M --greedy 4+ CHAINMODE=2 ptl_mla glm53 "partial glm53 prompts only" 3 GLM53_BITS=32 GLM53_PREFILL_CHUNK=16 -- $F --ids $ids --greedy 6+ KV=16:4 ptl_mla glm53 "partial glm53 the KV split" 4 GLM53_BITS=32 -- $F --ids $ids --greedy 8+ KV=16:4 ptl_mla glm53 "partial glm53 the KV split, chain chunks of 3" 2 GLM53_BITS=32 COLI_VK_CHAIN_ROWS=3 -- $F --ids $ids --greedy 8+ FAULT_BACK=7 REBUILD=1 ptl_mla glm53 "partial glm53 device lost, the device layers' KDA state rebuilt" 3 GLM53_BITS=32 GLM53_PREFILL_CHUNK=16 -- $F --ids $ids --greedy 6+ FAULT_BACK=5 REBUILD=1 ptl_mla glm53 "partial glm53 device lost after an image" 2 GLM53_BITS=32 -- $M --greedy 4+ ptl_mla_same glm53 "partial glm53 chunks of 1 = 7" "COLI_VK_CHAIN_ROWS=1" "COLI_VK_CHAIN_ROWS=7" COLI_VK_CHAIN_LAYERS=4 GLM53_BITS=32 -- $F --ids $ids --greedy 4+ # ---- the dense weights on the device only+ for k in 1 3; do ptl_mla_dho glm53 "partial glm53 device only, $k of 6 layers" $k glm53_l6s-i4 GLM53_BITS=32 -- $F --ids $ids --greedy 4; done+ ptl_mla_dho glm53 "partial glm53 device only, 4-bit trunk" 4 glm53_l6s-i4 GLM53_BITS=4 -- $F --ids $ids --greedy 4+ ptl_mla_dho glm53 "partial glm53 device only, an image" 2 glm53_mm_tiny GLM53_BITS=32 -- $M --greedy 4+ CHAINMODE=2 KEEP_KVB=1 ptl_mla_dho glm53 "partial glm53 device only, prompts only" 4 glm53_l6s-i4 GLM53_BITS=32 GLM53_PREFILL_CHUNK=16 -- $F --ids $ids --greedy 6+ COLI_VK_STAGED=1 ptl_mla_dho glm53 "partial glm53 device only, staged" 3 glm53_l6s-i4 GLM53_BITS=4 -- $F --ids $ids --greedy 4+ FAULT_BACK=7 REBUILD=1 ptl_mla_dho glm53 "partial glm53 device only, device lost" 3 glm53_l6s-i4 GLM53_BITS=32 GLM53_PREFILL_CHUNK=16 -- $F --ids $ids --greedy 6+ # the expert cache sizes itself after the N layers' host copies went+ rm -f chain.usage+ env GLM53_VERBOSE=1 GLM53_BITS=32 USAGE_SAVE=0 COLI_USAGE=chain.usage COLI_VK_TIER_SYNC=1 COLI_VULKAN=1 COLI_VK_CHAIN=1 \+ COLI_VK_DENSE_HOST=0 COLI_VK_CHAIN_LAYERS=3 ./glm53 $F --ids 5,7,9 --greedy 2 > vk.log 2>&1 || true+ rm -f chain.usage+ local given budget+ given=$(grep -n 'dense matrices on the device only' vk.log | head -1 | cut -d: -f1); budget=$(grep -n '^expert budget:' vk.log | head -1 | cut -d: -f1)+ { [ -n "$given" ] && [ -n "$budget" ] && [ "$given" -lt "$budget" ]; } || { cat vk.log; fail "partial glm53 device only: the expert cache sized itself before the host copies went"; }+ echo "OK partial glm53 device only: the expert cache sized after the 3 layers' host copies went ($(grep -a '^expert budget:' vk.log | head -1))"+ # serve sessions with 3 of 6 layers: pins, the prompt cache, read-outs, chain chunks, two+ # sessions taking the device in turn, prompts only, device only+ export CHAIN_SERVE_EXPECT='glm53 chain: 3 of 6 layers on the device'+ CHAIN_SERVE_DIALECT=numeric $PY tests/vulkan_chain_serve.py ./glm53 glm53_l6_serve GLM53_BITS=32 COLI_VK_CHAIN_LAYERS=3+ CHAIN_SERVE_DIALECT=numeric $PY tests/vulkan_chain_serve.py ./glm53 glm53_l6_serve GLM53_BITS=4 COLI_VK_CHAIN_ROWS=3 COLI_VK_CHAIN_LAYERS=3+ CHAIN_SERVE_SLOTS=2 CHAIN_SERVE_DIALECT=numeric $PY tests/vulkan_chain_serve.py ./glm53 glm53_l6_serve GLM53_BITS=32 KV_SLOTS=2 COLI_VK_CHAIN_LAYERS=3+ COLI_VK_CHAIN=2 CHAIN_SERVE_DIALECT=numeric $PY tests/vulkan_chain_serve.py ./glm53 glm53_l6_serve GLM53_BITS=32 COLI_VK_CHAIN_LAYERS=3+ CHAIN_SERVE_EXPECT='glm53: [0-9]+ dense matrices on the device only.*the 3 of 6 layers on the device' COLI_VK_DENSE_HOST=0 \+ CHAIN_SERVE_DIALECT=numeric $PY tests/vulkan_chain_serve.py ./glm53 glm53_l6_serve GLM53_BITS=32 COLI_VK_CHAIN_LAYERS=3+ unset CHAIN_SERVE_EXPECT+ # a pin restored over rows another branch rewrote, 2 of 4 layers on the device+ COLI_VULKAN=1 COLI_VK_CHAIN=1 COLI_VK_CHAIN_LAYERS=2 COLI_USAGE=$PWD/chain.usage $PY tests/glm53_pin_branch_harness.py --binary ./glm53 --fixture glm53_mm_tiny+ rm -rf glm_tiny_shx glm53_l6 glm53_l6s-i4 glm53_l6_serve chain.usage+ rm -f ptl-probe.err ptl-ram*.log same-*.log same-*.tok same-*.f32+ unset OMP_NUM_THREADS CAP_RAISE+}++# The same under ASan and UBSan (the glm-chain-sanitize family's build): no diagnostic, the+# chain ran with the N the line says (or handled the loss).+ptl_mla_san() { # <engine> <tag> <want N> <env and argv...>+ local eng=$1 tag=$2 want=$3; shift 3+ rm -f chain.usage+ env COLI_USAGE=chain.usage USAGE_SAVE=0 COLI_VK_TIER_SYNC=1 COLI_VULKAN=1 COLI_VK_CHAIN=${CHAINMODE:-1} "$@" > san.log 2>&1 || true+ rm -f chain.usage+ if grep -qE "ERROR: AddressSanitizer|runtime error:" san.log; then cat san.log; fail "$tag: sanitizer diagnostic"; fi+ grep -q "$eng chain: \([1-9][0-9]* forwards\|the device was lost\)" san.log || [ "$want" = 0 ] || { cat san.log; fail "$tag: the chain never ran"; }+ ptl_check_n "$eng" san.log "$want" "$tag"+ ptl_check_placed "$eng" san.log "$tag"+ echo "OK $tag: sanitizers clean, N = $want, $(chain_count "$eng" san.log) chain forwards"+}+ptl_family_glm_sanitize() {+ make clean >/dev/null 2>&1 || true+ make colibri glm53 VK=1 EXTRA_CFLAGS="-fsanitize=address,undefined -fno-omit-frame-pointer -g"+ export ASAN_OPTIONS=detect_leaks=0:detect_stack_use_after_return=0 UBSAN_OPTIONS=print_stacktrace=1+ ptl_glm_fixtures+ export OMP_NUM_THREADS=2 CAP_RAISE=0+ local G="SNAP=glm_tiny REF=ref_glm.json" X="SNAP=glm_tiny_shx REF=ref_glm.json" cap n ids F="--model glm53_l6s-i4"+ ids=$($PY -c "print(','.join(str((i*37+5)%120+2) for i in range(100)))")+ ptl_mla_san colibri "asan partial colibri 2 of 5 layers" 2 COLI_VK_CHAIN_LAYERS=2 $G ./colibri 64 16 16+ ptl_mla_san colibri "asan partial colibri DSA top-4 handed over, chunks of 3" 1 COLI_VK_CHAIN_LAYERS=1 $X DSA_TOPK=4 TF=1 COLI_VK_CHAIN_ROWS=3 ./colibri 64 16 16+ ptl_mla_run ptl-probe.err colibri COLI_VK_DEVICE_CAP_MB=256 COLI_VK_TIER_RESERVE_GB=0.04 $G -- 64 16 16+ cap=$(PTL_PROBE_CAP_MB=256 ptl_calc cap colibri ptl-probe.err 3)+ ptl_mla_san colibri "asan partial colibri a device for 3 layers" 3 COLI_VK_DEVICE_CAP_MB=$cap COLI_VK_TIER_RESERVE_GB=0.04 $G ./colibri 64 16 16+ ptl_mla_run ptl-probe.err colibri COLI_VK_STAGED=1 COLI_VK_STAGED_FAULT=submit:1000000 COLI_VK_CHAIN_LAYERS=5 $G -- 64 16 16+ n=$(ptl_calc fault colibri ptl-probe.err 2)+ ptl_mla_san colibri "asan partial colibri an upload failing in layer 2" 2 COLI_VK_STAGED=1 COLI_VK_STAGED_FAULT=submit:$n COLI_VK_CHAIN_LAYERS=5 $G ./colibri 64 16 16+ ptl_mla_san colibri "asan partial colibri device only, MTP" 2 COLI_VK_CHAIN_LAYERS=2 COLI_VK_DENSE_HOST=0 SNAP=glm_tiny_mtp REF=glm_tiny_mtp/ref_glm.json DRAFT=2 ./colibri 64 16 16+ ptl_mla_san colibri "asan partial colibri the KV split" 3 COLI_VK_CHAIN_LAYERS=3 COLI_VK_KV_DEVICE_ROWS=8 COLI_VK_KV_BLOCK=4 $G ./colibri 64 16 16+ ptl_mla_san colibri "asan partial colibri device lost" 2 COLI_VK_CHAIN_LAYERS=2 COLI_VK_CHAIN_FAULT=12 $G ./colibri 64 16 16+ ptl_mla_san glm53 "asan partial glm53 3 of 6 layers, chain chunks of 3" 3 COLI_VK_CHAIN_LAYERS=3 GLM53_BITS=4 COLI_VK_CHAIN_ROWS=3 ./glm53 $F --ids $ids --greedy 4+ ptl_mla_run ptl-probe.err glm53 COLI_VK_DEVICE_CAP_MB=256 COLI_VK_TIER_RESERVE_GB=0.04 GLM53_BITS=32 -- $F --ids $ids --greedy 4+ cap=$(PTL_PROBE_CAP_MB=256 ptl_calc cap glm53 ptl-probe.err 2)+ ptl_mla_san glm53 "asan partial glm53 a device for 2 layers" 2 COLI_VK_DEVICE_CAP_MB=$cap COLI_VK_TIER_RESERVE_GB=0.04 GLM53_BITS=32 ./glm53 $F --ids $ids --greedy 4+ ptl_mla_run ptl-probe.err glm53 COLI_VK_STAGED=1 COLI_VK_STAGED_FAULT=submit:1000000 COLI_VK_CHAIN_LAYERS=6 GLM53_BITS=4 -- $F --ids $ids --greedy 4+ n=$(ptl_calc fault glm53 ptl-probe.err 3)+ ptl_mla_san glm53 "asan partial glm53 an upload failing in layer 3" 3 COLI_VK_STAGED=1 COLI_VK_STAGED_FAULT=submit:$n COLI_VK_CHAIN_LAYERS=6 GLM53_BITS=4 ./glm53 $F --ids $ids --greedy 4+ ptl_mla_san glm53 "asan partial glm53 device only, an image" 2 COLI_VK_CHAIN_LAYERS=2 COLI_VK_DENSE_HOST=0 GLM53_BITS=32 ./glm53 --model glm53_mm_tiny \+ --ids 103,117,268,268,268,268,120,121 --patches glm53_mm_tiny/patches.f32 --grid 4x4 --greedy 4+ ptl_mla_san glm53 "asan partial glm53 the KV split" 4 COLI_VK_CHAIN_LAYERS=4 COLI_VK_KV_DEVICE_ROWS=16 COLI_VK_KV_BLOCK=4 GLM53_BITS=32 ./glm53 $F --ids $ids --greedy 6+ ptl_mla_san glm53 "asan partial glm53 device lost, rebuilt" 3 COLI_VK_CHAIN_LAYERS=3 GLM53_BITS=32 GLM53_PREFILL_CHUNK=16 COLI_VK_CHAIN_FAULT=8 ./glm53 $F --ids $ids --greedy 6+ local args+ for args in "./colibri glm_tiny_serve SERVE_BATCH=1 IDOT=0 KV_SLOTS=2 COLI_VK_CHAIN_LAYERS=2" \+ "./glm53 glm53_l6_serve GLM53_BITS=32 KV_SLOTS=2 COLI_VK_CHAIN_LAYERS=3"; do+ # shellcheck disable=SC2086+ CHAIN_SERVE_SLOTS=2 CHAIN_SERVE_DIALECT=$([ "${args%% *}" = ./colibri ] && echo colibri || echo numeric) \+ $PY tests/vulkan_chain_serve.py $args > san.log 2>&1 || { cat san.log; fail "asan partial serve $args"; }+ if grep -qE "ERROR: AddressSanitizer|runtime error:" san.log; then cat san.log; fail "asan partial serve $args: sanitizer diagnostic"; fi+ echo "OK asan partial serve ${args%% *}: $(tail -1 san.log | cut -c1-120)"+ done+ rm -rf glm_tiny_shx glm53_l6 glm53_l6s-i4 glm53_l6_serve chain.usage+ rm -f ptl-probe.err san.log+ unset OMP_NUM_THREADS CAP_RAISE+ make clean >/dev/null 2>&1 || true+}diff --git a/docs/vulkan.md b/docs/vulkan.mdindex 8c193d17..3b13f94a 100644--- a/docs/vulkan.md+++ b/docs/vulkan.md@@ -1275,6 +1275,31 @@ dense matrices), and geometries past the ops' limits. With the chain on, the perswitches (`COLI_VK_DENSE`, `COLI_VK_ATTN`, `COLI_VK_DEV2`) keep working beside it: thechain's tensors are their device copies where they made one.+**A partial chain** ([the fit](#a-partial-chain)). GLM-5.2's dense part is 9.9 GB, more+than an 8 or 12 GB card holds. Both engines decide N right after the load, before+colibri's pins and `cap_for_ram`, glm53's expert cache and the tier size themselves, and+place layers 0..N-1 one at a time (`COLI_VK_CHAIN_LAYERS=n` forces N). A forward runs all+its chunks through the N device layers; then the residual rows (colibri) or the hc_mult+streams (glm53) come back to the host once per chunk, and the CPU's own layer loop runs+layers N..L-1, the final norm and the head on them. When colibri's layer N is a shared+DSA indexer layer, the selection made by the device's last full layer comes back with+the rows (1 + `index_topk` ints a row) into the CPU's, so layer N reads the same positions+it would on the device. The KV mirror, its watermarks, the KV split and glm53's KDA state+on the device cover the N layers; the CPU layers' KV rows and KDA state are the host's+alone. After a lost device colibri runs the forward again on the CPU, and glm53 rebuilds+the KDA state of the device's layers only (the CPU layers ran every position already).+With `COLI_VK_DENSE_HOST=0` only the N layers drop their host copies, each once all of it+reached the device; colibri's `resident dense`, and with it the pins and `cap_for_ram`,+counts only what was dropped, and glm53's expert cache sizes itself after them. lm_head,+and colibri's MTP layer and eh_proj, go to the device only with every layer there and+room for them. Until then the per-matrix path (`COLI_VK_DENSE`, `COLI_VK_ATTN`) multiplies+on the device only what the chain placed, and the line+`[VK] <engine> chain: N of L layers held at exit: M B of matrices on the device` says so at+exit. The `partial-glm` and `partial-glm-sanitize` families+(`tests/vulkan_partial_glm.sh`) check it on the tiny fixtures, with glm_tiny's shared+indexer layers and a six-layer GLM-5.3 whose KDA and MLA layers alternate, so MLA layers+run on both sides of the handoff.+**Arithmetic.** f32 activations throughout, as the CPU's f32 paths: colibri's CPU int8dot (`IDOT`, on by default for int8 and, from two rows, int4 rows) rounds activations,so the tests set `IDOT=0` for those trunks, as for the tier. On the fixtures every