Skip to content

Run frontier MoE models on hardware you already own — pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦

C

feat(vulkan): integrate bounded memory, expert streaming and speculative verification

JustVugg2f33f211ea6e
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index a4662b74..ef862ef9 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -183,6 +183,13 @@ jobs:
req: c/tools/requirements-qwen38-tiny.txt
- family: qwen-chain-sanitize
req: c/tools/requirements-qwen38-tiny.txt
+ # speculative decoding with deeper verifies (MTP drafts up to 3, prompt-lookup
+ # drafts up to 5) on qwen36 and qwen38: against plain decoding on the CPU and
+ # in the dense chain, and the same under ASan and UBSan
+ - family: qwen-spec
+ req: c/tools/requirements-qwen38-tiny.txt
+ - family: qwen-spec-sanitize
+ req: c/tools/requirements-qwen38-tiny.txt
# staged device-local uploads (COLI_VK_STAGED=1): the tier and the chain of qwen36
# and qwen38, and an engine choosing staging on its own under an emulated window
- family: qwen-staged
@@ -253,6 +260,82 @@ jobs:
req: c/tools/requirements-kimi-k3-tiny.txt
- family: kimi-chain-sanitize
req: c/tools/requirements-kimi-k3-tiny.txt
+ # the dense weights on the device only (COLI_VK_DENSE_HOST=0): the host copies
+ # given back, the CPU's tokens, a lost device reading them back from disk;
+ # one leg per engine group, staged, and the same under ASan and UBSan
+ - family: dense-only-qwen38
+ req: c/tools/requirements-qwen38-tiny.txt
+ - family: dense-only-qwen38-staged
+ req: c/tools/requirements-qwen38-tiny.txt
+ - family: dense-only-qwen38-sanitize
+ req: c/tools/requirements-qwen38-tiny.txt
+ - family: dense-only-qwen36-olmoe
+ req: c/tools/requirements-qwen38-tiny.txt
+ - family: dense-only-qwen36-olmoe-sanitize
+ req: c/tools/requirements-qwen38-tiny.txt
+ - family: dense-only-inkling-mimo-kimi
+ req: c/tools/requirements-kimi-k3-tiny.txt
+ extra: pillow
+ - family: dense-only-inkling-mimo-kimi-sanitize
+ req: c/tools/requirements-kimi-k3-tiny.txt
+ extra: pillow
+ - family: dense-only-glm
+ req: c/tools/requirements-qwen38-tiny.txt
+ extra: pillow
+ - family: dense-only-glm-sanitize
+ req: c/tools/requirements-qwen38-tiny.txt
+ extra: pillow
+ - family: dense-only-deepseek
+ req: c/tools/requirements-deepseek-v4-tiny.txt
+ extra: sympy pillow
+ - family: dense-only-deepseek-sanitize
+ req: c/tools/requirements-deepseek-v4-tiny.txt
+ extra: sympy pillow
+ # big prompt chunks and expert streaming (docs/vulkan.md): every chain engine on a
+ # prompt past its usual block, the chunk from the budget, a forced small chunk and
+ # streaming threshold, staged, the feature off, and the forced runs under ASan
+ # and UBSan
+ - family: prefill-qwen
+ req: c/tools/requirements-qwen38-tiny.txt
+ - family: prefill-qwen-sanitize
+ req: c/tools/requirements-qwen38-tiny.txt
+ - family: prefill-inkling-olmoe
+ req: c/tools/oracle-requirements.txt
+ - family: prefill-mimo-kimi
+ req: c/tools/requirements-mimo-tiny.txt
+ - family: prefill-glm
+ req: c/tools/requirements-glm53-tiny.txt
+ - family: prefill-deepseek
+ req: c/tools/requirements-deepseek-v4-tiny.txt
+ extra: sympy pillow
+ # the KV cache split between the device and the host past the device's budget
+ # (COLI_VK_KV_DEVICE_ROWS emulates a small device): every chain engine against its
+ # CPU run, prefix reuse, serve sessions and a lost device across the split, and the
+ # same under ASan and UBSan; DeepSeek's in legs of their own (their fixtures pin
+ # another transformers)
+ - family: kv-split
+ req: c/tools/requirements-qwen38-tiny.txt
+ extra: numpy pillow
+ - family: kv-split-sanitize
+ req: c/tools/requirements-qwen38-tiny.txt
+ extra: numpy pillow
+ - family: kv-split-deepseek
+ req: c/tools/requirements-deepseek-v4-tiny.txt
+ extra: sympy pillow
+ - family: kv-split-deepseek-sanitize
+ req: c/tools/requirements-deepseek-v4-tiny.txt
+ extra: sympy pillow
+ # the decision engines: Laya's and GLiNER2.5-Decide's forward on the device and
+ # matrix by matrix, Clef's DECIDE on qwen36's paths in f32, int8 and f16, the
+ # tiny oracles, the serve tests and TypeSafe's SDKs through the device; and the
+ # same (but the SDKs) under ASan and UBSan
+ - family: decide
+ req: c/tools/requirements-jev-sdk.txt
+ extra: -r c/tools/oracle-requirements.txt
+ npm: c/tools/jev-sdk
+ - family: decide-sanitize
+ req: c/tools/requirements-jev-sdk.txt
+ extra: -r c/tools/oracle-requirements.txt
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
@@ -266,6 +349,9 @@ jobs:
sudo apt-get install -y --no-install-recommends \
libvulkan-dev glslc mesa-vulkan-drivers
pip install -r ${{ matrix.req }} ${{ matrix.extra }}
+ - name: The pinned TypeScript SDK
+ if: ${{ matrix.npm }}
+ run: cd ${{ matrix.npm }} && npm ci
- name: CPU against Vulkan, engine by engine
run: bash c/tests/vulkan_engines.sh ${{ matrix.family }}
diff --git a/.gitignore b/.gitignore
index ceeb4090..9c981856 100644
--- a/.gitignore
+++ b/.gitignore
@@ -142,6 +142,11 @@ c/qwen38_tiny/
c/dsv41_tiny/
c/qwen36_tiny/
c/qwen36_tiny_c/
+# tests/spec_drafts_harness.py: the ASCII-vocabulary fixture whose replies round-trip as text
+c/qwen36_tiny_v128/
+c/qwen36_tiny_v128_c/
+c/qwen36_tiny64_v128/
+c/qwen36_tiny64_v128_c/
# tools/make_clef_tiny.py and its container; clef_tiny/ref.json is committed
c/clef_tiny/*
!c/clef_tiny/ref.json
diff --git a/c/Makefile b/c/Makefile
index 42442964..28e674dd 100644
--- a/c/Makefile
+++ b/c/Makefile
@@ -728,6 +728,14 @@ VK_SPV += shaders/chain_ares.spv
# inkling's chain ops (made on first use): the short convolution and the attention
# with its relative-position bias
VK_SPV += shaders/chain_sconv.spv shaders/chain_relattn.spv
+# blocked attention for prompt chunks (made on first use; vk_chain.c, chain_attnb.comp)
+VK_SPV += shaders/chain_attnb.spv
+# the KV cache split between the device and the host past the device's budget
+# (vk_chain.h: vkc_kvs_*; vk_kvsplit.h)
+VK_SPV += shaders/chain_kvs.spv
+# the decision engines' encoders (laya, gliner_decide): bidirectional attention over row
+# ranges with DeBERTa's relative terms, LayerNorm, GeGLU, RoPE (made on first use)
+VK_SPV += shaders/chain_enc.spv
endif
# Optional AMD XDNA2 (Ryzen AI NPU) lane, Windows. Same opt-in shape as VK=1:
@@ -1292,8 +1300,8 @@ ifneq ($(EXE),)
.PHONY: laya
laya: laya$(EXE)
endif
-laya$(EXE): laya.c .build-config
- $(CC) $(CFLAGS) laya.c -o laya$(EXE) $(LDFLAGS)
+laya$(EXE): laya.c $(VK_OBJ) $(VK_SPV) .build-config
+ $(CC) $(CFLAGS) laya.c $(VK_OBJ) -o laya$(EXE) $(LDFLAGS)
# GLiNER2.5-Decide (fastino), a decision model: DeBERTa-v3 encoder + GLiNER2's
# classification head, served on POST /v1/systemone through the DECIDE command
@@ -1303,8 +1311,8 @@ ifneq ($(EXE),)
.PHONY: gliner_decide
gliner_decide: gliner_decide$(EXE)
endif
-gliner_decide$(EXE): gliner_decide.c .build-config
- $(CC) $(CFLAGS) gliner_decide.c -o gliner_decide$(EXE) $(LDFLAGS)
+gliner_decide$(EXE): gliner_decide.c $(VK_OBJ) $(VK_SPV) .build-config
+ $(CC) $(CFLAGS) gliner_decide.c $(VK_OBJ) -o gliner_decide$(EXE) $(LDFLAGS)
# Tiny Laya checkpoint + the laya package's own answers on it. The generator is
# numpy and deterministic; laya_tiny/ref.json is committed and comes from the
@@ -1532,6 +1540,51 @@ qwen38-tiny-mtp-check: qwen38-tiny-mtp-generate qwen38$(EXE)
$(PYTHON) tests/qwen38_mtp_harness.py --engine ./qwen38$(EXE) --bf16 ./qwen38_tiny_mtp \
--fp8 ./qwen38_tiny_fp8_mtp --int4 ./qwen38_tiny_int4_mtp
+# Speculative decoding with deeper verifies (docs/speculative.md): the MTP head's
+# drafts up to 3 per verify (Q38_MTP_DRAFTS) and prompt-lookup drafts up to 5
+# (COLI_LOOKUP=1, spec_draft.h), on qwen38 and qwen36. tests/test_spec_draft
+# checks the lookup and the gate on their own; tests/spec_drafts_harness.py gates
+# every depth and every rejected row against plain decoding (tokens and last logits
+# byte-identical, the forced modes' counts exact), and serve sessions with prefix
+# reuse, pins and turns that end inside a verify. The qwen36 fixtures have an ASCII
+# vocabulary (--vocab 128), so a reply resent as text reads back as the ids the
+# engine generated. SPEC_ENV passes more settings to every run (the Vulkan family
+# in tests/vulkan_engines.sh runs the same gates in the dense chain).
+.PHONY: qwen36-tiny-v128-generate spec-drafts-check spec-drafts-run
+qwen36-tiny-v128-generate:
+ $(PYTHON) tools/make_qwen36_tiny.py --out ./qwen36_tiny_v128 --ref-mode full --vocab 128 \
+ --emit-ref ./qwen36_tiny_v128/ref_full.json
+ $(PYTHON) tools/convert_qwen36.py --model ./qwen36_tiny_v128 --out ./qwen36_tiny_v128_c --ebits 8
+ $(PYTHON) tools/make_qwen36_tiny.py --out ./qwen36_tiny64_v128 --ref-mode full --vocab 128 --inter 64 \
+ --emit-ref ./qwen36_tiny64_v128/ref_full.json
+ $(PYTHON) tools/convert_qwen36.py --model ./qwen36_tiny64_v128 --out ./qwen36_tiny64_v128_c --ebits 4 --gs 64
+
+tests/test_spec_draft$(EXE): tests/test_spec_draft.c
+ $(CC) $(CFLAGS) $< -o $@ $(LDFLAGS)
+
+tests/test_qwen38_spec_alloc$(EXE): tests/test_qwen38_spec_alloc.c qwen38.c $(VK_OBJ)
+ $(CC) $(CFLAGS) $< $(VK_OBJ) -o $@ $(LDFLAGS)
+
+SPEC_HARNESS = $(PYTHON) tests/spec_drafts_harness.py
+spec-drafts-check: qwen38-tiny-mtp-generate qwen36-tiny-v128-generate
+ $(MAKE) spec-drafts-run
+
+# the gates alone, on fixtures already generated (spec-drafts-check makes them)
+spec-drafts-run: qwen38$(EXE) qwen36$(EXE) tests/test_spec_draft$(EXE) tests/test_qwen38_spec_alloc$(EXE)
+ ./tests/test_spec_draft$(EXE)
+ ./tests/test_qwen38_spec_alloc$(EXE)
+ $(SPEC_HARNESS) --engine ./qwen38$(EXE) --kind qwen38 --fixture ./qwen38_tiny_mtp --mtp --serve $(SPEC_ENV)
+ $(SPEC_HARNESS) --engine ./qwen38$(EXE) --kind qwen38 --fixture ./qwen38_tiny_fp8_mtp --mtp \
+ --env Q38_PREFILL_BATCH=0 $(SPEC_ENV)
+ $(SPEC_HARNESS) --engine ./qwen38$(EXE) --kind qwen38 --fixture ./qwen38_tiny_int4_mtp \
+ --ref ./qwen38_tiny_int4_mtp/ref_int4.json --mtp --env Q38_EXPERT_INT4=1 --env Q38_TRUNK_MIN_KB=0 $(SPEC_ENV)
+ $(SPEC_HARNESS) --engine ./qwen36$(EXE) --kind qwen36 --fixture ./qwen36_tiny_v128_c \
+ --ref ./qwen36_tiny_v128/ref_full.json --cap 8 --n-new 64 --serve --env COLI_DENSE_I8=0 $(SPEC_ENV)
+ $(SPEC_HARNESS) --engine ./qwen36$(EXE) --kind qwen36 --fixture ./qwen36_tiny_v128_c \
+ --ref ./qwen36_tiny_v128/ref_full.json --cap 1 --n-new 64 $(SPEC_ENV)
+ $(SPEC_HARNESS) --engine ./qwen36$(EXE) --kind qwen36 --fixture ./qwen36_tiny64_v128_c \
+ --ref ./qwen36_tiny64_v128/ref_full.json --cap 8 --n-new 64 $(SPEC_ENV)
+
# Context-size gates: KV layout, growth across requests, and the attention
# capacity. Includes qwen36.c directly, so no model file is needed.
# Same tier objects as the engine: the test includes qwen36.c, so with CUDA=1
@@ -1540,6 +1593,9 @@ qwen38-tiny-mtp-check: qwen38-tiny-mtp-generate qwen38$(EXE)
tests/test_qwen36_ctx$(EXE): tests/test_qwen36_ctx.c qwen36.c $(QWEN36_TIER_OBJ) $(CUDA_OBJ) $(VK_OBJ)
$(CC) $(CFLAGS) $< $(QWEN36_TIER_OBJ) $(CUDA_OBJ) $(VK_OBJ) -o $@ $(LDFLAGS)
+tests/test_qwen36_weight_alloc$(EXE): tests/test_qwen36_weight_alloc.c qwen36.c $(QWEN36_TIER_OBJ) $(CUDA_OBJ) $(VK_OBJ)
+ $(CC) $(CFLAGS) $< $(QWEN36_TIER_OBJ) $(CUDA_OBJ) $(VK_OBJ) -o $@ $(LDFLAGS)
+
# CACHE_ROUTE: route_select() against a residency table, no model needed.
tests/test_qwen36_cache_route$(EXE): tests/test_qwen36_cache_route.c qwen36.c $(QWEN36_TIER_OBJ) $(CUDA_OBJ) $(VK_OBJ)
$(CC) $(CFLAGS) $< $(QWEN36_TIER_OBJ) $(CUDA_OBJ) $(VK_OBJ) -o $@ $(LDFLAGS)
diff --git a/c/backend_vulkan.c b/c/backend_vulkan.c
index 12c7dfc2..bfbe0124 100644
--- a/c/backend_vulkan.c
+++ b/c/backend_vulkan.c
@@ -34,6 +34,9 @@ struct ColiVkTensor {
VkaRange wr, sr; /* where the rows and the scales sit in the pool's blocks */
struct ColiVkTensor *next_free; /* deferred-free list (a free while async work is in flight) */
uint8_t *img; /* staged uploads: the host image a tier tensor is filled in, until committed */
+ uint8_t *wmap, *smap; /* mapped memory: the rows' and the scales' mapping (coli_vk_tensor_refill) */
+ VkDeviceMemory imp; /* coli_vk_tensor_import: the host memory its rows are read from, in place */
+ struct ColiVkTensor *imp_next; /* the live imports, freed at shutdown */
};
/* ---- memory placement without Resizable BAR (docs/vulkan.md) ------------------------
@@ -435,7 +438,7 @@ static size_t cpu_row_bytes(int fmt, int I) {
return fmt == 1 || fmt == 12 || fmt == 13 ? (size_t)I // int8, fp8: one byte per weight
: fmt == 5 ? ((size_t)I + 63) / 64 * 24 // int3-g64: 24B per 64-group
: fmt == 10 ? (size_t)I * 4
- : fmt == 11 ? (size_t)I * 2
+ : fmt == 11 || fmt == 14 ? (size_t)I * 2 // bf16, f16
: (size_t)(I + 1) / 2;
}
static int rowwords(int fmt, int I) {
@@ -444,7 +447,7 @@ static int rowwords(int fmt, int I) {
/* Scale floats per tensor: per-row formats carry O, int3-g64 carries O*ceil(I/64)
* (one f32 per 64-input group). upload_tensor and tensor_free must agree on this. */
static size_t scale_floats(int fmt, int I, int O, int gs) {
- if (fmt == 10 || fmt == 11) return 1; // unused by the shader; bound anyway
+ if (fmt == 10 || fmt == 11 || fmt == 14) return 1; // unused by the shader; bound anyway
if (fmt == 5) return (size_t)O * (((size_t)I + 63) / 64);
if (fmt == 4 || fmt == 7 || fmt == 12 || fmt == 13)
return (size_t)O * (((size_t)I + gs - 1) / gs); // per-group [O,ng]
@@ -1302,6 +1305,7 @@ static ColiVkTensor *tensor_alloc(VkWPool *P, int fmt, int I, int O, int gs, voi
} else {
*wptr = wp; *sptr = sp;
memset(*wptr, 0, t->wbytes);
+ t->wmap = wp; t->smap = sp;
}
__atomic_add_fetch(&P->bytes, t->wbytes + sbytes, __ATOMIC_RELAXED);
__atomic_add_fetch(&P->tensors, 1, __ATOMIC_RELAXED);
@@ -1314,7 +1318,7 @@ static ColiVkTensor *tensor_alloc(VkWPool *P, int fmt, int I, int O, int gs, voi
return t;
}
static int fmt_uploadable(int fmt, int gs) {
- return fmt == 1 || fmt == 2 || fmt == 5 || fmt == 10 || fmt == 11 || /* fmt=4/7: word-aligned groups only */
+ return fmt == 1 || fmt == 2 || fmt == 5 || fmt == 10 || fmt == 11 || fmt == 14 || /* fmt=4/7: word-aligned groups only */
((fmt == 4 || fmt == 7) && gs >= 8 && gs % 8 == 0) ||
((fmt == 12 || fmt == 13) && gs >= 4 && gs % 4 == 0); /* fp8 / int8: 4 per word */
}
@@ -1333,7 +1337,7 @@ static int upload_tensor_pool(VkWPool *P, ColiVkTensor **out, const void *weight
VkUp *u = &g_up[P->dev];
pthread_mutex_lock(&u->mx);
int ok = up_add(u, P->dev, t->wbuf, 0, t->wbytes, up_fill_rows, &rows) &&
- up_add(u, P->dev, t->sbuf, 0, sb, up_fill_bytes, fmt == 10 || fmt == 11 ? (const void *)&one : scales);
+ up_add(u, P->dev, t->sbuf, 0, sb, up_fill_bytes, fmt == 10 || fmt == 11 || fmt == 14 ? (const void *)&one : scales);
ok = up_finish(u, P->dev) && ok;
pthread_mutex_unlock(&u->mx);
if (!ok) { /* the matrix stays on the CPU (with the device, if it was lost) */
@@ -1353,7 +1357,7 @@ static int upload_tensor_pool(VkWPool *P, ColiVkTensor **out, const void *weight
memcpy((uint8_t *)wptr + (size_t)o * stride,
(const uint8_t *)weights + (size_t)o * cpu_rb, cpu_rb);
size_t sfl = scale_floats(fmt, I, O, gs); // fmt=5: O*ceil(I/64) group scales
- if (fmt == 10 || fmt == 11) ((float *)sptr)[0] = 1.0f; /* float weights: no scales */
+ if (fmt == 10 || fmt == 11 || fmt == 14) ((float *)sptr)[0] = 1.0f; /* float weights: no scales */
else memcpy(sptr, scales, sfl * sizeof(float));
*out = t;
return 1;
@@ -1370,6 +1374,99 @@ int coli_vk_tensor_ensure(ColiVkTensor **tensor, const void *weights, const floa
return upload_tensor(tensor, weights, scales, fmt, I, O, grp);
}
+/* ---- resident rows read in place (VK_EXT_external_memory_host) ----------------------
+ * On a device that shares the CPU's RAM a resident copy is a second copy of the same
+ * bytes: a 27B trunk held twice does not fit beside the host's on a 64 GB box. An import
+ * hands the device the host's own rows instead: the pages are wrapped in a VkDeviceMemory
+ * of the type vkGetMemoryHostPointerPropertiesEXT allows and a buffer is bound to it at
+ * offset 0, so the shaders read the matrix where the engine keeps it. The scales (a few
+ * floats a row) are copied into the weight pool as usual. Only where the rows need no
+ * padding (the shaders read rowWords words a row: the CPU's row length must already be a
+ * whole number of words) and the allocation is aligned to the device's import alignment
+ * and spans whole units of it. Not with staged uploads: those are for a card whose memory
+ * is not the host's. */
+static ColiVkTensor *g_imports; /* live imports, freed at shutdown */
+static pthread_mutex_t g_imports_mx = PTHREAD_MUTEX_INITIALIZER;
+static size_t g_import_bytes;
+size_t coli_vk_import_alignment(void) {
+#ifdef VK_EXT_external_memory_host
+ if (!G.ready || !G.has_hostmem || g_up[0].on) return 0;
+ static size_t al;
+ if (!al) {
+ VkPhysicalDeviceExternalMemoryHostPropertiesEXT hp = {.sType = VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_EXTERNAL_MEMORY_HOST_PROPERTIES_EXT};
+ VkPhysicalDeviceProperties2 p2 = {.sType = VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_PROPERTIES_2, .pNext = &hp};
+ vkGetPhysicalDeviceProperties2(G.phys, &p2);
+ al = hp.minImportedHostPointerAlignment ? (size_t)hp.minImportedHostPointerAlignment : 4096;
+ }
+ return al;
+#else
+ return 0;
+#endif
+}
+size_t coli_vk_imported_bytes(void) { return __atomic_load_n(&g_import_bytes, __ATOMIC_RELAXED); }
+int coli_vk_tensor_import(ColiVkTensor **tensor, const void *weights, size_t alloc_bytes, const float *scales,
+ int fmt, int I, int O, int gs) {
+ if (*tensor) return (*tensor)->fmt == fmt && (*tensor)->I == I && (*tensor)->O == O;
+#ifdef VK_EXT_external_memory_host
+ size_t al = coli_vk_import_alignment();
+ if (!al || !fmt_uploadable(fmt, gs) || O < 1 || I < 1) return 0;
+ size_t rb = cpu_row_bytes(fmt, I);
+ if (rb != (size_t)rowwords(fmt, I) * 4) return 0; /* rows the shaders would read padded */
+ size_t wbytes = rb * (size_t)O, sz = (wbytes + al - 1) / al * al;
+ if ((uintptr_t)weights % al || sz > alloc_bytes || wbytes > G.ssbo_range) return 0;
+ static PFN_vkGetMemoryHostPointerPropertiesEXT gp;
+ if (!gp) gp = (PFN_vkGetMemoryHostPointerPropertiesEXT)vkGetDeviceProcAddr(G.dev, "vkGetMemoryHostPointerPropertiesEXT");
+ VkMemoryHostPointerPropertiesEXT mp = {.sType = VK_STRUCTURE_TYPE_MEMORY_HOST_POINTER_PROPERTIES_EXT};
+ if (!gp || gp(G.dev, VK_EXTERNAL_MEMORY_HANDLE_TYPE_HOST_ALLOCATION_BIT_EXT, weights, &mp) != VK_SUCCESS ||
+ !mp.memoryTypeBits) return 0;
+ VkExternalMemoryBufferCreateInfo eb = {.sType = VK_STRUCTURE_TYPE_EXTERNAL_MEMORY_BUFFER_CREATE_INFO,
+ .handleTypes = VK_EXTERNAL_MEMORY_HANDLE_TYPE_HOST_ALLOCATION_BIT_EXT};
+ VkBufferCreateInfo bi = {.sType = VK_STRUCTURE_TYPE_BUFFER_CREATE_INFO, .pNext = &eb, .size = wbytes,
+ .usage = VK_BUFFER_USAGE_STORAGE_BUFFER_BIT, .sharingMode = VK_SHARING_MODE_EXCLUSIVE};
+ VkBuffer buf;
+ if (vkCreateBuffer(G.dev, &bi, NULL, &buf) != VK_SUCCESS) return 0;
+ VkMemoryRequirements req;
+ vkGetBufferMemoryRequirements(G.dev, buf, &req);
+ uint32_t bits = mp.memoryTypeBits & req.memoryTypeBits, mt = 0;
+ if (!bits) { vkDestroyBuffer(G.dev, buf, NULL); return 0; }
+ while (!(bits & (1u << mt))) mt++;
+ VkImportMemoryHostPointerInfoEXT imp = {.sType = VK_STRUCTURE_TYPE_IMPORT_MEMORY_HOST_POINTER_INFO_EXT,
+ .handleType = VK_EXTERNAL_MEMORY_HANDLE_TYPE_HOST_ALLOCATION_BIT_EXT, .pHostPointer = (void *)weights};
+ VkMemoryAllocateInfo ai = {.sType = VK_STRUCTURE_TYPE_MEMORY_ALLOCATE_INFO, .pNext = &imp, .allocationSize = sz,
+ .memoryTypeIndex = mt};
+ VkDeviceMemory mem;
+ if (vkAllocateMemory(G.dev, &ai, NULL, &mem) != VK_SUCCESS) { vkDestroyBuffer(G.dev, buf, NULL); return 0; }
+ if (vkBindBufferMemory(G.dev, buf, mem, 0) != VK_SUCCESS) {
+ vkDestroyBuffer(G.dev, buf, NULL); vkFreeMemory(G.dev, mem, NULL); return 0;
+ }
+ ColiVkTensor *t = calloc(1, sizeof(*t));
+ size_t sbytes = scale_floats(fmt, I, O, gs) * sizeof(float);
+ void *sp = NULL;
+ if (!t || !pool_suballoc(&g_wpool, sbytes, &t->sbuf, &sp, &t->sr) || !sp) {
+ if (t && t->sbuf) { vkDestroyBuffer(G.dev, t->sbuf, NULL); pool_free_range(&g_wpool, t->sr); }
+ free(t); vkDestroyBuffer(G.dev, buf, NULL); vkFreeMemory(G.dev, mem, NULL); return 0;
+ }
+ t->fmt = fmt; t->I = I; t->O = O; t->rowWords = rowwords(fmt, I);
+ t->gs = (fmt == 4 || fmt == 7 || fmt == 12 || fmt == 13) ? gs : 0;
+ t->dev = 0; t->pool = &g_wpool; t->wbuf = buf; t->wbytes = wbytes; t->imp = mem;
+ if (fmt == 10 || fmt == 11 || fmt == 14) ((float *)sp)[0] = 1.0f;
+ else memcpy(sp, scales, sbytes);
+ __atomic_add_fetch(&g_wpool.bytes, sbytes, __ATOMIC_RELAXED);
+ __atomic_add_fetch(&g_wpool.tensors, 1, __ATOMIC_RELAXED);
+ __atomic_add_fetch(&G.used_bytes, sbytes, __ATOMIC_RELAXED);
+ __atomic_add_fetch(&G.tensor_count, 1, __ATOMIC_RELAXED);
+ __atomic_add_fetch(&g_import_bytes, wbytes, __ATOMIC_RELAXED);
+ pthread_mutex_lock(&g_imports_mx);
+ t->imp_next = g_imports; g_imports = t;
+ pthread_mutex_unlock(&g_imports_mx);
+ *tensor = t;
+ return 1;
+#else
+ (void)tensor; (void)weights; (void)alloc_bytes; (void)scales; (void)fmt; (void)I; (void)O; (void)gs;
+ return 0;
+#endif
+}
+
/* Sync-path fence wait. A blocked vkWaitForFences pays a scheduler wake on
* signal (~50-150 us) — and the engine fences ~2 sync submits per layer per
* token, so the wakes alone cost seconds per run. Spin on vkGetFenceStatus for
@@ -1608,6 +1705,109 @@ int coli_vk_dense_decide(const char *engine, int tier_on, int def) {
}
int coli_vk_dense(void) { return g_dense_on; }
+/* The dense weights' host copies (coli_vk_dense_host_decide): device only or not, and
+ * what the engine dropped and read back since. Counters are touched from parallel
+ * regions (a CPU fallback may reload inside one): atomics. */
+static int g_dho, g_dho_atexit, g_dho_on_device;
+static char g_dho_engine[32];
+static unsigned long long g_dho_drop_n, g_dho_drop_b, g_dho_load_n, g_dho_load_b;
+static double dho_gib(unsigned long long b) { return (double)b / 1073741824.0; }
+/* A size for the [VK] lines: GiB from 1 GiB, MiB below (a test fixture's KiB show as 0.0 MiB). */
+static const char *dho_size(unsigned long long b, char *buf, size_t n) {
+ if (b >= 1073741824ull) snprintf(buf, n, "%.2f GiB", dho_gib(b));
+ else snprintf(buf, n, "%.1f MiB", (double)b / 1048576.0);
+ return buf;
+}
+/* This process's resident set in GiB (Linux; -1 elsewhere). */
+static double dho_rss_gib(void) {
+#ifdef __linux__
+ FILE *f = fopen("/proc/self/statm", "r");
+ long pages = -1, rss = -1;
+ if (f) { if (fscanf(f, "%ld %ld", &pages, &rss) != 2) rss = -1; fclose(f); }
+ long pg = sysconf(_SC_PAGESIZE);
+ return rss >= 0 && pg > 0 ? (double)rss * (double)pg / 1073741824.0 : -1.0;
+#else
+ return -1.0;
+#endif
+}
+static void dho_exit_report(void) {
+ double rss = dho_rss_gib();
+ char r[48] = "";
+ if (rss >= 0) snprintf(r, sizeof r, "; RSS %.2f GiB", rss);
+ if (!g_dho) { /* the same line with the host copies kept, for a comparison of the two */
+ fprintf(stderr, "[VK] %s: dense weights at exit: on the device and in host RAM%s\n", g_dho_engine, r);
+ return;
+ }
+ char a[32], b[32];
+ fprintf(stderr, "[VK] %s: dense weights at exit: %llu matrices on the device only (%s of host copies "
+ "dropped), %llu read back from disk for the CPU (%s)%s\n", g_dho_engine,
+ __atomic_load_n(&g_dho_drop_n, __ATOMIC_RELAXED), dho_size(__atomic_load_n(&g_dho_drop_b, __ATOMIC_RELAXED), a, sizeof a),
+ __atomic_load_n(&g_dho_load_n, __ATOMIC_RELAXED), dho_size(__atomic_load_n(&g_dho_load_b, __ATOMIC_RELAXED), b, sizeof b), r);
+}
+static int dho_choice(int dense_on_device, size_t dense_bytes, char *why, size_t n) {
+ const char *e = getenv("COLI_VK_DENSE_HOST");
+ if (!dense_on_device) {
+ snprintf(why, n, "the dense part runs on the CPU%s", e && *e && atoi(e) == 0 ? ", so COLI_VK_DENSE_HOST=0 has nothing to drop" : "");
+ return 0;
+ }
+ if (e && *e) { snprintf(why, n, "COLI_VK_DENSE_HOST=%s", e); return atoi(e) == 0; }
+ if (coli_vk_device_integrated()) {
+ snprintf(why, n, "an integrated GPU: its memory is the CPU's RAM, the host copy would hold them twice; "
+ "COLI_VK_DENSE_HOST=1 keeps it");
+ return 1;
+ }
+ if (coli_vk_device_shares_ram()) {
+ snprintf(why, n, "a CPU device; COLI_VK_DENSE_HOST=0 drops the host copies");
+ return 0;
+ }
+ /* a discrete GPU: device only when the dense weights fit what it has free, with the
+ * expert tier's default reserve (1 GiB) left for scratch, mirrors and the driver */
+ double used = 0, budget = 0, fit;
+ if (coli_vk_mem_budget(&used, &budget)) fit = (budget - used) * 1e9;
+ else fit = (double)coli_vk_device_local_bytes();
+ fit -= 1073741824.0;
+ if ((double)dense_bytes <= fit) {
+ snprintf(why, n, "a discrete GPU with room for them: %.2f of %.2f GiB free; COLI_VK_DENSE_HOST=1 keeps the host copy",
+ dho_gib(dense_bytes), fit > 0 ? fit / 1073741824.0 : 0.0);
+ return 1;
+ }
+ snprintf(why, n, "a discrete GPU without room for all of them (%.2f GiB, %.2f GiB free): the host copy stays the "
+ "fallback; COLI_VK_DENSE_HOST=0 drops it", dho_gib(dense_bytes), fit > 0 ? fit / 1073741824.0 : 0.0);
+ return 0;
+}
+int coli_vk_dense_host_decide(const char *engine, int dense_on_device, size_t dense_bytes) {
+ char why[256];
+ g_dho = G.ready && dho_choice(dense_on_device, dense_bytes, why, sizeof why);
+ if (!G.ready) snprintf(why, sizeof why, "no device");
+ snprintf(g_dho_engine, sizeof g_dho_engine, "%s", engine ? engine : "engine");
+ if (engine) fprintf(stderr, "[VK] %s: dense weights %s (%s)\n", engine,
+ g_dho ? "on the device only" : "on the device and in host RAM", why);
+ g_dho_on_device = G.ready && dense_on_device;
+ if (g_dho_on_device && !g_dho_atexit) { g_dho_atexit = 1; atexit(dho_exit_report); }
+ return g_dho;
+}
+int coli_vk_dense_device_only(void) { return g_dho; }
+void coli_vk_dense_host_dropped(size_t bytes) {
+ __atomic_add_fetch(&g_dho_drop_n, 1, __ATOMIC_RELAXED);
+ __atomic_add_fetch(&g_dho_drop_b, (unsigned long long)bytes, __ATOMIC_RELAXED);
+}
+void coli_vk_dense_host_reloaded(size_t bytes) {
+ __atomic_add_fetch(&g_dho_load_n, 1, __ATOMIC_RELAXED);
+ __atomic_add_fetch(&g_dho_load_b, (unsigned long long)bytes, __ATOMIC_RELAXED);
+}
+void coli_vk_dense_host_placed(const char *engine, const char *kept) {
+ if (!g_dho) return;
+ double rss = dho_rss_gib();
+ char r[48] = "";
+ if (rss >= 0) snprintf(r, sizeof r, "; RSS now %.2f GiB", rss);
+ char a[32];
+ fprintf(stderr, "[VK] %s: %llu dense matrices on the device only, %s of host RAM given back%s%s%s\n",
+ engine, __atomic_load_n(&g_dho_drop_n, __ATOMIC_RELAXED),
+ dho_size(__atomic_load_n(&g_dho_drop_b, __ATOMIC_RELAXED), a, sizeof a), kept && *kept ? "; kept on the host: " : "",
+ kept && *kept ? kept : "", r);
+}
+unsigned long long coli_vk_dense_host_dropped_bytes(void) { return __atomic_load_n(&g_dho_drop_b, __ATOMIC_RELAXED); }
+
/* Where a layer's dense chain runs (vk_chain.c): the whole layer recorded into one
* submission with the residual stream and the recurrent/KV state on the device, the
* alternative to one synchronous coli_vk_matmul per matrix. Returns COLI_VK_CHAIN_OFF,
@@ -2592,6 +2792,25 @@ int coli_vk_attention_absorb_project(ColiVkTensor **kvb, const void *w, const fl
* reclaim the host struct and counters only. */
static void tensor_release(ColiVkTensor *t) {
VkWPool *P = t->pool;
+ if (t->imp) { /* rows imported in place: their memory object, the scales' range */
+ pthread_mutex_lock(&g_imports_mx);
+ for (ColiVkTensor **pp = &g_imports; *pp; pp = &(*pp)->imp_next) if (*pp == t) { *pp = t->imp_next; break; }
+ pthread_mutex_unlock(&g_imports_mx);
+ size_t sb = scale_floats(t->fmt, t->I, t->O, t->gs) * sizeof(float);
+ if (G.ready) {
+ vkDestroyBuffer(G.dev, t->wbuf, NULL);
+ vkFreeMemory(G.dev, t->imp, NULL);
+ if (t->sbuf) vkDestroyBuffer(G.dev, t->sbuf, NULL);
+ pool_free_range(P, t->sr);
+ }
+ __atomic_sub_fetch(&P->bytes, sb, __ATOMIC_RELAXED);
+ __atomic_sub_fetch(&P->tensors, 1, __ATOMIC_RELAXED);
+ __atomic_sub_fetch(&G.tensor_count, 1, __ATOMIC_RELAXED);
+ __atomic_sub_fetch(&G.used_bytes, sb, __ATOMIC_RELAXED);
+ __atomic_sub_fetch(&g_import_bytes, t->wbytes, __ATOMIC_RELAXED);
+ free(t);
+ return;
+ }
if (t->dev == 1 ? G2.ready : G.ready) {
VkDevice dev = pool_device(P);
if (t->wbuf) vkDestroyBuffer(dev, t->wbuf, NULL);
@@ -2748,15 +2967,17 @@ static struct {
int D, I, act; float limit, a, b;
VkCommandPool cpool; VkCommandBuffer cmd; VkFence fence;
VkQueryPool qpool; int has_ts;
- VkShaderModule sh_act, sh_gu, sh_mv, sh_mm;
+ VkShaderModule sh_act, sh_gu, sh_mv, sh_mm, sh_coop;
VkDescriptorSetLayout dsl6, dsl4, dsl3;
VkPipelineLayout pl6, pl4, pl3;
- VkPipeline p_gu, p_mv, p_act, p_mm[VK_GEMM_SLOTS];
+ VkPipeline p_gu, p_mv, p_act, p_mm[VK_GEMM_SLOTS], p_coop[VK_COOP_SLOTS];
VkDescriptorPool *dpools; int ndpools;
VkDescriptorPool act_pool; VkDescriptorSet act_set;
XbBuf x, g, u, h, y; /* x host-written, g/u/h device-only, y host-read */
size_t align;
int gemm_rows; /* rows from which an expert takes the GEMM route (0 = never) */
+ int coop_rows;
+ unsigned long long cooperative_matmuls;
ColiVkExpert *live;
/* the batch in flight: where each assignment row's output sits */
size_t *yoff; int nrows, cyoff;
@@ -2766,6 +2987,12 @@ static struct {
* row weights in the x buffer after each expert's rows (xb_v4_init) */
VkShaderModule sh_v4; VkDescriptorSetLayout dsl_v4; VkPipelineLayout pl_v4; VkPipeline p_v4;
VkDescriptorPool v4_pool; VkDescriptorSet v4_set;
+ /* sub-batches (coli_vk_xb_sub_*, the tier's big prefill steps): two halves of the
+ * scratch, each with its own command buffer, fence and timestamps, up to two in flight */
+ struct { VkCommandBuffer cmd; VkFence fence; VkQueryPool qp; int inflight, nrows, cyoff; size_t *yoff; } sub[2];
+ size_t sub_x, sub_i, sub_y; /* a half's bytes of each scratch (0: not reserved) */
+ unsigned long long sub_batches, sub_experts, sub_rows, sub_gemm;
+ double sub_ms;
} XB;
struct PCV4 { int rows, n; };
@@ -2833,16 +3060,19 @@ static void xb_write_v4_set(void) {
/* Make every scratch region hold `xr`/`ir`/`yr` bytes (offsets of a batch stay below
* its region; the buffer is twice that, so offset + window always fits). Grows by
* half again at least; rewrites every live expert's sets when a buffer moved. */
+static int xb_sizes(size_t xr, size_t ir, size_t yr, size_t *nx, size_t *ni, size_t *ny) {
+ *nx = xr > XB.x.region ? xb_up(xr + xr / 2, 4096) : XB.x.region;
+ *ni = ir > XB.g.region ? xb_up(ir + ir / 2, 4096) : XB.g.region;
+ *ny = yr > XB.y.region ? xb_up(yr + yr / 2, 4096) : XB.y.region;
+ if (*nx > G.ssbo_range) *nx = G.ssbo_range & ~(size_t)4095;
+ if (*ni > G.ssbo_range) *ni = G.ssbo_range & ~(size_t)4095;
+ if (*ny > G.ssbo_range) *ny = G.ssbo_range & ~(size_t)4095;
+ return xr <= *nx && ir <= *ni && yr <= *ny;
+}
static int xb_reserve(size_t xr, size_t ir, size_t yr) {
if (xr <= XB.x.region && ir <= XB.g.region && yr <= XB.y.region) return 1;
- size_t nx = XB.x.region, ni = XB.g.region, ny = XB.y.region;
- if (xr > nx) nx = xb_up(xr + xr / 2, 4096);
- if (ir > ni) ni = xb_up(ir + ir / 2, 4096);
- if (yr > ny) ny = xb_up(yr + yr / 2, 4096);
- if (nx > G.ssbo_range) nx = G.ssbo_range & ~(size_t)4095;
- if (ni > G.ssbo_range) ni = G.ssbo_range & ~(size_t)4095;
- if (ny > G.ssbo_range) ny = G.ssbo_range & ~(size_t)4095;
- if (xr > nx || ir > ni || yr > ny) return 0; /* one window cannot address it */
+ size_t nx, ni, ny;
+ if (!xb_sizes(xr, ir, yr, &nx, &ni, &ny)) return 0; /* one window cannot address it */
int ok = 1;
if (nx != XB.x.region) ok &= xb_buf(&XB.x, nx, G.memtype, 1);
if (ni != XB.g.region) ok &= xb_buf(&XB.g, ni, G.memtype_dev, 0) && xb_buf(&XB.u, ni, G.memtype_dev, 0) &&
@@ -2882,6 +3112,69 @@ static int xb_pipeline(VkShaderModule sh, VkPipelineLayout pl, const VkSpecializ
return 1;
}
+static void xb_coop_init(void) {
+#ifdef VK_KHR_cooperative_matrix
+ const char *e = getenv("COLI_VK_TIER_COOP");
+ if (!G.pipe_coop[0] || !XB.p_mm[0] || (e && !atoi(e))) return;
+ char path[1100]; derive_sibling(G.spv_path, "_coop.spv", path, sizeof path);
+ XB.sh_coop = load_spv(G.dev, path);
+ if (!XB.sh_coop) return;
+ VkPhysicalDeviceProperties pp; vkGetPhysicalDeviceProperties(G.phys, &pp);
+ for (int k = 0; k < VK_COOP_SLOTS; k++) {
+ if (!G.pipe_coop[k]) continue;
+ VkCoopTile t = G.coop_t[k];
+ int nw = (t.bm / t.wm) * (t.bn / t.wn), nt = nw * G.coop_sg;
+ size_t shared = (size_t)(t.bm + 2 * t.bn) * (t.bk + 8) * 2 +
+ (size_t)t.bm * 4 + (size_t)nw * 16 * 17 * 4 + (size_t)(nt + t.bn) * 4;
+ if (shared > pp.limits.maxComputeSharedMemorySize || nt % (nt < t.bn ? 1 : nt / t.bn)) continue;
+ int32_t sv[8] = {t.bm, t.bn, t.wm, t.wn, t.bk, G.coop_sg, nt, 1};
+ VkSpecializationMapEntry me[8];
+ for (int i = 0; i < 8; i++) me[i] = (VkSpecializationMapEntry){(uint32_t)i, (uint32_t)(i * 4), 4};
+ VkSpecializationInfo si = {8, me, sizeof sv, sv};
+ VkPipelineShaderStageRequiredSubgroupSizeCreateInfoEXT rss = {
+ .sType = VK_STRUCTURE_TYPE_PIPELINE_SHADER_STAGE_REQUIRED_SUBGROUP_SIZE_CREATE_INFO_EXT,
+ .requiredSubgroupSize = (uint32_t)G.coop_sg};
+ VkComputePipelineCreateInfo ci = {.sType = VK_STRUCTURE_TYPE_COMPUTE_PIPELINE_CREATE_INFO,
+ .stage = {.sType = VK_STRUCTURE_TYPE_PIPELINE_SHADER_STAGE_CREATE_INFO, .pNext = &rss,
+ .flags = VK_PIPELINE_SHADER_STAGE_CREATE_REQUIRE_FULL_SUBGROUPS_BIT_EXT,
+ .stage = VK_SHADER_STAGE_COMPUTE_BIT, .module = XB.sh_coop, .pName = "main",
+ .pSpecializationInfo = &si}, .layout = XB.pl4};
+ if (vkCreateComputePipelines(G.dev, VK_NULL_HANDLE, 1, &ci, NULL, &XB.p_coop[k]) != VK_SUCCESS)
+ XB.p_coop[k] = VK_NULL_HANDLE;
+ }
+ e = getenv("COLI_VK_TIER_COOP_ROWS");
+ XB.coop_rows = e && *e ? atoi(e) : 64;
+ if (XB.coop_rows < 1) XB.coop_rows = 1;
+#endif
+}
+
+/* The narrow tile wins on per-row and integer-group weights in expert shapes.
+ * FP8/MXFP4 benefit from wider tiles, but rounding an awkward row count up to
+ * another wide tile can cost more than the FP32 GEMM. Keep that fallback when
+ * the cooperative tile would process more padded rows. */
+static int xb_coop_slot(const ColiVkTensor *t, int rows, int gemm) {
+ int k = t->fmt == 1 || t->fmt == 2 || t->fmt == 4 || t->fmt == 5 ? 0 : coop_slot(rows);
+ if (!XB.coop_rows || rows < XB.coop_rows || !XB.p_coop[k] || !coop_fmt(t)) return -1;
+ if (t->fmt == 7 || t->fmt == 12) {
+ int cbn = G.coop_t[k].bn, fbn = G.gemm_t[gemm].bn;
+ if ((rows + cbn - 1) / cbn * cbn > (rows + fbn - 1) / fbn * fbn) return -1;
+ }
+ return k;
+}
+
+/* Keep each projection's format check: mixed gate/up/down quantization may make
+ * only some projections eligible. Small verifies retain their scalar GEMV path. */
+static VkPipeline xb_mm_pipeline(const ColiVkTensor *t, int rows, int slot,
+ int *bm, int *bn, int *cooperative) {
+ int k = xb_coop_slot(t, rows, slot);
+ if (k >= 0) {
+ *bm = G.coop_t[k].bm; *bn = G.coop_t[k].bn; (*cooperative)++;
+ return XB.p_coop[k];
+ }
+ *bm = G.gemm_t[slot].bm; *bn = G.gemm_t[slot].bn;
+ return XB.p_mm[slot];
+}
+
/* COLI_VK_ACT_SWIGLU_V4's pass (expert_act_v4.spv), made the first time that
* activation is asked for; 0 when its shader is missing. */
static int xb_v4_init(void) {
@@ -2904,7 +3197,7 @@ static int xb_v4_init(void) {
int coli_vk_xb_init(int D, int I, int act, float limit, float a, float b) {
if (XB.ready) { /* the geometry stays; the activation rides the push constants */
- if (XB.D != D || XB.I != I || XB.inflight) return 0;
+ if (XB.D != D || XB.I != I || XB.inflight || XB.sub[0].inflight || XB.sub[1].inflight) return 0;
if (act == COLI_VK_ACT_SWIGLU_V4 && !xb_v4_init()) return 0;
XB.act = act; XB.limit = limit > 0.f ? limit : 0.f; XB.a = a; XB.b = b;
return 1;
@@ -2944,6 +3237,7 @@ int coli_vk_xb_init(int D, int I, int act, float limit, float a, float b) {
const char *e = getenv("COLI_VK_TIER_GEMM_ROWS");
XB.gemm_rows = XB.p_mm[0] && XB.p_act ? (e && *e ? atoi(e) : 16) : 0;
if (XB.gemm_rows < 0) XB.gemm_rows = 0;
+ xb_coop_init();
VkCommandPoolCreateInfo cp = {.sType = VK_STRUCTURE_TYPE_COMMAND_POOL_CREATE_INFO,
.flags = VK_COMMAND_POOL_CREATE_RESET_COMMAND_BUFFER_BIT, .queueFamilyIndex = G.tq_fam};
VKCHECK(vkCreateCommandPool(G.dev, &cp, NULL, &XB.cpool), "xb cmd pool");
@@ -2974,8 +3268,9 @@ int coli_vk_xb_init(int D, int I, int act, float limit, float a, float b) {
int coli_vk_xb_ready(void) { return XB.ready && G.ready; } /* a device lost elsewhere stops the tier too */
int coli_vk_xb_queue_shared(void) { return G.tq_shared; }
+static int xb_busy(void);
ColiVkExpert *coli_vk_xb_expert(ColiVkTensor *g, ColiVkTensor *u, ColiVkTensor *d) {
- if (!XB.ready || XB.inflight || !g || !u || !d) return NULL;
+ if (!XB.ready || xb_busy() || !g || !u || !d) return NULL;
if (g->I != XB.D || g->O != XB.I || u->I != XB.D || u->O != XB.I || g->fmt != u->fmt || g->gs != u->gs ||
d->I != XB.I || d->O != XB.D) return NULL;
ColiVkExpert *e = calloc(1, sizeof(*e));
@@ -3031,52 +3326,42 @@ static void xb_barrier(VkCommandBuffer c) {
0, 1, &mb, 0, NULL, 0, NULL);
}
-static int xb_issue(ColiVkExpert *const *ex, const int *rows, int count, const float *const *xrows,
- const float *wrows) {
- if (!XB.ready || XB.inflight || count < 1) return 0;
- const int D = XB.D, I = XB.I;
- const size_t a = XB.align, dr = (size_t)D * 4, ir = (size_t)I * 4;
- /* COLI_VK_ACT_SWIGLU_V4: each expert's row weights follow its x rows */
- const int v4 = XB.act == COLI_VK_ACT_SWIGLU_V4 && XB.p_v4;
- size_t *off = malloc((size_t)count * 3 * sizeof(size_t));
- if (!off) return 0;
- size_t xo = 0, io = 0, yo = 0; int total = 0;
+/* One batch's layout in the scratch: each expert's x, hidden and y offsets from the bases
+ * x0, i0, y0 (its x rows, then with the V4 activation its row weights), the ends and the
+ * row total. 0 when an expert is missing or a row count is out of range. */
+static int xb_plan(ColiVkExpert *const *ex, const int *rows, int count, int v4, size_t x0, size_t i0, size_t y0,
+ size_t *off, size_t *xe, size_t *ie, size_t *ye, int *total) {
+ const size_t a = XB.align, dr = (size_t)XB.D * 4, ir = (size_t)XB.I * 4;
+ size_t xo = x0, io = i0, yo = y0; int t = 0;
for (int c = 0; c < count; c++) {
- if (!ex[c] || rows[c] < 1 || rows[c] > 65535) { free(off); return 0; }
+ if (!ex[c] || rows[c] < 1 || rows[c] > 65535) return 0;
off[3 * c] = xo; off[3 * c + 1] = io; off[3 * c + 2] = yo;
xo += xb_up((size_t)rows[c] * dr, a); io += xb_up((size_t)rows[c] * ir, a); yo += xb_up((size_t)rows[c] * dr, a);
if (v4) xo += xb_up((size_t)rows[c] * 4, a);
- total += rows[c];
- }
- if (!xb_reserve(xo, io, yo)) { free(off); return 0; }
- if (total > XB.cyoff) {
- size_t *n = realloc(XB.yoff, (size_t)total * sizeof(*n));
- if (!n) { free(off); return 0; }
- XB.yoff = n; XB.cyoff = total;
+ t += rows[c];
}
- /* the activation rows, packed per expert; where each output row will be */
- int j = 0;
- for (int c = 0; c < count; c++)
- for (int r = 0; r < rows[c]; r++, j++) {
- memcpy((uint8_t *)XB.x.ptr + off[3 * c] + (size_t)r * dr, xrows[j], dr);
- XB.yoff[j] = off[3 * c + 2] + (size_t)r * dr;
- if (v4) ((float *)((uint8_t *)XB.x.ptr + off[3 * c] + xb_up((size_t)rows[c] * dr, a)))[r] = wrows ? wrows[j] : 1.0f;
- }
- XB.nrows = total;
-
- VkCommandBuffer cmd = XB.cmd;
- if (vkResetCommandBuffer(cmd, 0) != VK_SUCCESS) { free(off); return 0; }
+ *xe = xo; *ie = io; *ye = yo; *total = t;
+ return 1;
+}
+/* Record one batch laid out at off[] (its x rows already written) into cmd, with its
+ * timestamps in qp when the device has them, and submit it with fence on the tier queue.
+ * *ngemm: how many experts took the GEMM route. 0 = nothing was submitted. */
+static int xb_record_submit(VkCommandBuffer cmd, VkFence fence, VkQueryPool qp, ColiVkExpert *const *ex,
+ const int *rows, int count, const size_t *off, int v4, int *ngemm_out) {
+ const int D = XB.D, I = XB.I;
+ const size_t a = XB.align, dr = (size_t)D * 4;
+ if (vkResetCommandBuffer(cmd, 0) != VK_SUCCESS) return 0;
VkCommandBufferBeginInfo bi = {.sType = VK_STRUCTURE_TYPE_COMMAND_BUFFER_BEGIN_INFO,
.flags = VK_COMMAND_BUFFER_USAGE_ONE_TIME_SUBMIT_BIT};
vkBeginCommandBuffer(cmd, &bi);
if (XB.has_ts) {
- vkCmdResetQueryPool(cmd, XB.qpool, 0, 2);
- vkCmdWriteTimestamp(cmd, VK_PIPELINE_STAGE_TOP_OF_PIPE_BIT, XB.qpool, 0);
+ vkCmdResetQueryPool(cmd, qp, 0, 2);
+ vkCmdWriteTimestamp(cmd, VK_PIPELINE_STAGE_TOP_OF_PIPE_BIT, qp, 0);
}
/* the route of each expert: its GEMM slot, or -1 for the GEMVs */
int *route = malloc((size_t)count * sizeof(int));
- if (!route) { free(off); vkEndCommandBuffer(cmd); return 0; }
- int ngemm = 0;
+ if (!route) { vkEndCommandBuffer(cmd); return 0; }
+ int ngemm = 0, cooperative = 0;
for (int c = 0; c < count; c++) {
int k = XB.gemm_rows && rows[c] >= XB.gemm_rows ? gemm_slot(rows[c]) : -1;
route[c] = k >= 0 && XB.p_mm[k] ? k : -1;
@@ -3094,21 +3379,22 @@ static int xb_issue(ColiVkExpert *const *ex, const int *rows, int count, const f
vkCmdDispatch(cmd, (uint32_t)((I + 7) / 8), (uint32_t)rows[c], 1);
}
if (ngemm) {
- int slot = -1;
+ VkPipeline bound = VK_NULL_HANDLE;
for (int c = 0; c < count; c++) {
if (route[c] < 0) continue;
const ColiVkExpert *e = ex[c];
int k = route[c];
- if (k != slot) { vkCmdBindPipeline(cmd, VK_PIPELINE_BIND_POINT_COMPUTE, XB.p_mm[k]); slot = k; }
- VkGemmTile t = G.gemm_t[k];
const ColiVkTensor *gu[2] = {e->g, e->u};
VkDescriptorSet su[2] = {e->s_g, e->s_u};
for (int q = 0; q < 2; q++) {
+ int bm, bn;
+ VkPipeline pipe = xb_mm_pipeline(gu[q], rows[c], k, &bm, &bn, &cooperative);
+ if (pipe != bound) { vkCmdBindPipeline(cmd, VK_PIPELINE_BIND_POINT_COMPUTE, pipe); bound = pipe; }
uint32_t dyn[2] = {(uint32_t)off[3 * c], (uint32_t)off[3 * c + 1]};
vkCmdBindDescriptorSets(cmd, VK_PIPELINE_BIND_POINT_COMPUTE, XB.pl4, 0, 1, &su[q], 2, dyn);
struct PC pc = {gu[q]->fmt, rows[c], D, I, gu[q]->rowWords, gu[q]->gs};
vkCmdPushConstants(cmd, XB.pl4, VK_SHADER_STAGE_COMPUTE_BIT, 0, sizeof(pc), &pc);
- vkCmdDispatch(cmd, (uint32_t)((I + t.bm - 1) / t.bm), (uint32_t)((rows[c] + t.bn - 1) / t.bn), 1);
+ vkCmdDispatch(cmd, (uint32_t)((I + bm - 1) / bm), (uint32_t)((rows[c] + bn - 1) / bn), 1);
}
}
xb_barrier(cmd);
@@ -3138,33 +3424,70 @@ static int xb_issue(ColiVkExpert *const *ex, const int *rows, int count, const f
xb_barrier(cmd);
}
/* phase 3: down of every expert */
- int bound = -2;
+ VkPipeline bound = VK_NULL_HANDLE;
for (int c = 0; c < count; c++) {
const ColiVkExpert *e = ex[c];
int k = route[c], gemm = k >= 0;
- if (k != bound) { vkCmdBindPipeline(cmd, VK_PIPELINE_BIND_POINT_COMPUTE, gemm ? XB.p_mm[k] : XB.p_mv); bound = k; }
+ int bm = 8, bn = 1;
+ VkPipeline pipe = gemm ? xb_mm_pipeline(e->d, rows[c], k, &bm, &bn, &cooperative) : XB.p_mv;
+ if (pipe != bound) { vkCmdBindPipeline(cmd, VK_PIPELINE_BIND_POINT_COMPUTE, pipe); bound = pipe; }
uint32_t dyn[2] = {(uint32_t)off[3 * c + 1], (uint32_t)off[3 * c + 2]};
vkCmdBindDescriptorSets(cmd, VK_PIPELINE_BIND_POINT_COMPUTE, XB.pl4, 0, 1, &e->s_dn, 2, dyn);
struct PC pc = {e->d->fmt, rows[c], I, D, e->d->rowWords, e->d->gs};
vkCmdPushConstants(cmd, XB.pl4, VK_SHADER_STAGE_COMPUTE_BIT, 0, sizeof(pc), &pc);
if (gemm) {
- VkGemmTile t = G.gemm_t[k];
- vkCmdDispatch(cmd, (uint32_t)((D + t.bm - 1) / t.bm), (uint32_t)((rows[c] + t.bn - 1) / t.bn), 1);
+ vkCmdDispatch(cmd, (uint32_t)((D + bm - 1) / bm), (uint32_t)((rows[c] + bn - 1) / bn), 1);
} else vkCmdDispatch(cmd, (uint32_t)((D + 7) / 8), (uint32_t)rows[c], 1);
}
/* the host reads y after the fence */
VkMemoryBarrier hb = {.sType = VK_STRUCTURE_TYPE_MEMORY_BARRIER,
.srcAccessMask = VK_ACCESS_SHADER_WRITE_BIT, .dstAccessMask = VK_ACCESS_HOST_READ_BIT};
vkCmdPipelineBarrier(cmd, VK_PIPELINE_STAGE_COMPUTE_SHADER_BIT, VK_PIPELINE_STAGE_HOST_BIT, 0, 1, &hb, 0, NULL, 0, NULL);
- if (XB.has_ts) vkCmdWriteTimestamp(cmd, VK_PIPELINE_STAGE_BOTTOM_OF_PIPE_BIT, XB.qpool, 1);
- free(off); free(route);
+ if (XB.has_ts) vkCmdWriteTimestamp(cmd, VK_PIPELINE_STAGE_BOTTOM_OF_PIPE_BIT, qp, 1);
+ free(route);
if (vkEndCommandBuffer(cmd) != VK_SUCCESS) return 0;
VkSubmitInfo si = {.sType = VK_STRUCTURE_TYPE_SUBMIT_INFO, .commandBufferCount = 1, .pCommandBuffers = &cmd};
- if (vkResetFences(G.dev, 1, &XB.fence) != VK_SUCCESS) return 0;
- if (vk_submit(0, G.tqueue, &si, XB.fence) != VK_SUCCESS) {
+ if (vkResetFences(G.dev, 1, &fence) != VK_SUCCESS) return 0;
+ if (vk_submit(0, G.tqueue, &si, fence) != VK_SUCCESS) {
fprintf(stderr, "[VK] expert batch: submit failed, the tier stops\n");
XB.ready = 0; return 0;
}
+ *ngemm_out = ngemm;
+ XB.cooperative_matmuls += (unsigned long long)cooperative;
+ return 1;
+}
+/* the single batch or a sub-batch in flight: the scratch and the experts' sets are in use */
+static int xb_busy(void) { return XB.inflight || XB.sub[0].inflight || XB.sub[1].inflight; }
+
+static int xb_issue(ColiVkExpert *const *ex, const int *rows, int count, const float *const *xrows,
+ const float *wrows) {
+ if (!XB.ready || xb_busy() || count < 1) return 0;
+ const size_t a = XB.align, dr = (size_t)XB.D * 4;
+ /* COLI_VK_ACT_SWIGLU_V4: each expert's row weights follow its x rows */
+ const int v4 = XB.act == COLI_VK_ACT_SWIGLU_V4 && XB.p_v4;
+ size_t *off = malloc((size_t)count * 3 * sizeof(size_t));
+ if (!off) return 0;
+ size_t xo, io, yo; int total;
+ if (!xb_plan(ex, rows, count, v4, 0, 0, 0, off, &xo, &io, &yo, &total)) { free(off); return 0; }
+ if (!xb_reserve(xo, io, yo)) { free(off); return 0; }
+ if (total > XB.cyoff) {
+ size_t *n = realloc(XB.yoff, (size_t)total * sizeof(*n));
+ if (!n) { free(off); return 0; }
+ XB.yoff = n; XB.cyoff = total;
+ }
+ /* the activation rows, packed per expert; where each output row will be */
+ int j = 0;
+ for (int c = 0; c < count; c++)
+ for (int r = 0; r < rows[c]; r++, j++) {
+ memcpy((uint8_t *)XB.x.ptr + off[3 * c] + (size_t)r * dr, xrows[j], dr);
+ XB.yoff[j] = off[3 * c + 2] + (size_t)r * dr;
+ if (v4) ((float *)((uint8_t *)XB.x.ptr + off[3 * c] + xb_up((size_t)rows[c] * dr, a)))[r] = wrows ? wrows[j] : 1.0f;
+ }
+ XB.nrows = total;
+ int ngemm = 0;
+ int ok = xb_record_submit(XB.cmd, XB.fence, XB.qpool, ex, rows, count, off, v4, &ngemm);
+ free(off);
+ if (!ok) return 0;
XB.inflight = 1; async_begin(0);
XB.batches++; XB.experts += (unsigned long long)count; XB.rows += (unsigned long long)total;
XB.gemm_experts += (unsigned long long)ngemm;
@@ -3207,11 +3530,174 @@ void coli_vk_xb_stats(ColiVkXbStats *st) {
st->device_ms = XB.dev_ms; st->timestamps = XB.has_ts; st->queue_shared = G.tq_shared;
st->gemm_rows = XB.gemm_rows;
st->scratch_bytes = 2 * (XB.x.region + 3 * XB.g.region + XB.y.region);
+ st->sub_batches = XB.sub_batches; st->sub_experts = XB.sub_experts; st->sub_rows = XB.sub_rows;
+ st->sub_gemm = XB.sub_gemm; st->sub_ms = XB.sub_ms;
+ st->cooperative_matmuls = XB.cooperative_matmuls;
+}
+
+/* ---- sub-batches: a big prefill step as several batches, two in flight ----------------
+ * The single batch above holds a whole step in the scratch at once. A big prefill step
+ * (vk_tier.c's streaming: thousands of rows, every expert of the layer) instead runs as
+ * a sequence of bounded batches: batch k uses half k % 2 of each scratch, so the host
+ * writes the rows of the next while the device computes the one before, and the scratch
+ * stays the size of two halves whatever the step's size (a card without Resizable BAR
+ * keeps its x rows inside the host-visible window). Each half has its own command buffer,
+ * fence and timestamps on the tier queue; its outputs are copied out when it is joined.
+ * The recording is the single batch's (xb_record_submit). */
+static int xb_sub_make(int h) {
+ if (XB.sub[h].cmd) return 1;
+ VkCommandBufferAllocateInfo cb = {.sType = VK_STRUCTURE_TYPE_COMMAND_BUFFER_ALLOCATE_INFO,
+ .commandPool = XB.cpool, .level = VK_COMMAND_BUFFER_LEVEL_PRIMARY, .commandBufferCount = 1};
+ VkFenceCreateInfo fi = {.sType = VK_STRUCTURE_TYPE_FENCE_CREATE_INFO};
+ if (vkAllocateCommandBuffers(G.dev, &cb, &XB.sub[h].cmd) != VK_SUCCESS) { XB.sub[h].cmd = VK_NULL_HANDLE; return 0; }
+ if (vkCreateFence(G.dev, &fi, NULL, &XB.sub[h].fence) != VK_SUCCESS) {
+ vkFreeCommandBuffers(G.dev, XB.cpool, 1, &XB.sub[h].cmd); XB.sub[h].cmd = VK_NULL_HANDLE; XB.sub[h].fence = VK_NULL_HANDLE;
+ return 0;
+ }
+ if (XB.has_ts) {
+ VkQueryPoolCreateInfo qi = {.sType = VK_STRUCTURE_TYPE_QUERY_POOL_CREATE_INFO,
+ .queryType = VK_QUERY_TYPE_TIMESTAMP, .queryCount = 2};
+ if (vkCreateQueryPool(G.dev, &qi, NULL, &XB.sub[h].qp) != VK_SUCCESS) XB.sub[h].qp = VK_NULL_HANDLE;
+ }
+ return 1;
+}
+/* Logical bytes of one half; xb_reserve adds its descriptor window and growth. */
+static void xb_sub_sizes(int rows, int experts, size_t *hx, size_t *hi, size_t *hy) {
+ const size_t a = XB.align, dr = (size_t)XB.D * 4, ir = (size_t)XB.I * 4, pad = (size_t)experts * a;
+ const int v4 = XB.act == COLI_VK_ACT_SWIGLU_V4 && XB.p_v4;
+ *hx = xb_up((size_t)rows * dr + pad + (v4 ? (size_t)rows * 4 + pad : 0), a);
+ *hi = xb_up((size_t)rows * ir + pad, a);
+ *hy = xb_up((size_t)rows * dr + pad, a);
+}
+/* Choose before allocating: an OOM in xb_reserve disables the batch backend.
+ * Account for both halves, the doubled descriptor windows, growth by 1.5 and
+ * all five buffers. xb_buf frees an old buffer before replacing it, so only
+ * positive growth must fit the remaining budget. Reused scratch costs zero. */
+int coli_vk_xb_sub_fit(int rows, int experts, size_t extra_budget) {
+ if (!XB.ready || xb_busy() || rows < 1 || experts < 1) return 0;
+ if (rows > 65535) rows = 65535;
+ VkPhysicalDeviceMemoryProperties2 mp = {.sType = VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_MEMORY_PROPERTIES_2};
+#ifdef VK_EXT_memory_budget
+ VkPhysicalDeviceMemoryBudgetPropertiesEXT bud = {.sType = VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_MEMORY_BUDGET_PROPERTIES_EXT};
+ if (G.has_budget) mp.pNext = &bud;
+#endif
+ vkGetPhysicalDeviceMemoryProperties2(G.phys, &mp);
+ size_t room[VK_MAX_MEMORY_HEAPS];
+ for (uint32_t h = 0; h < mp.memoryProperties.memoryHeapCount; h++) {
+ room[h] = (size_t)mp.memoryProperties.memoryHeaps[h].size / 8;
+#ifdef VK_EXT_memory_budget
+ if (G.has_budget) room[h] = bud.heapBudget[h] > bud.heapUsage[h] ? (size_t)((bud.heapBudget[h] - bud.heapUsage[h]) / 2) : 0;
+#endif
+ }
+ uint32_t xheap = mp.memoryProperties.memoryTypes[G.memtype].heapIndex;
+ uint32_t iheap = mp.memoryProperties.memoryTypes[G.memtype_dev].heapIndex;
+ uint32_t yheap = mp.memoryProperties.memoryTypes[G.memtype_cached].heapIndex;
+ int lo = 0, hi = rows;
+ while (lo < hi) {
+ int mid = lo + (hi - lo + 1) / 2;
+ size_t hx, hh, hy, nx, ni, ny;
+ xb_sub_sizes(mid, experts, &hx, &hh, &hy);
+ int ok = xb_sizes(2 * hx, 2 * hh, 2 * hy, &nx, &ni, &ny);
+ if (ok) {
+ size_t dx = 2 * (nx - XB.x.region), di = 6 * (ni - XB.g.region), dy = 2 * (ny - XB.y.region);
+ size_t need[VK_MAX_MEMORY_HEAPS] = {0};
+ need[xheap] += dx; need[iheap] += di; need[yheap] += dy;
+ ok = dx + di + dy <= extra_budget;
+ for (uint32_t h = 0; ok && h < mp.memoryProperties.memoryHeapCount; h++) ok = need[h] <= room[h];
+ }
+ if (ok) lo = mid; else hi = mid - 1;
+ }
+ return lo;
+}
+int coli_vk_xb_sub_reserve(int rows, int experts) {
+ if (!XB.ready || xb_busy() || rows < 1 || experts < 1) return 0;
+ if (!xb_sub_make(0) || !xb_sub_make(1)) return 0;
+ if (XB.has_ts && (!XB.sub[0].qp || !XB.sub[1].qp)) return 0;
+ size_t hx, hi, hy;
+ xb_sub_sizes(rows, experts, &hx, &hi, &hy);
+ if (!xb_reserve(2 * hx, 2 * hi, 2 * hy)) return 0;
+ XB.sub_x = hx; XB.sub_i = hi; XB.sub_y = hy;
+ return 1;
+}
+int coli_vk_xb_sub_busy(int h) { return h >= 0 && h < 2 && XB.sub[h].inflight; }
+int coli_vk_xb_sub_issue(int h, ColiVkExpert *const *ex, const int *rows, int count, const float *const *xrows,
+ const float *wrows) {
+ if (!XB.ready || h < 0 || h > 1 || XB.inflight || XB.sub[h].inflight || !XB.sub_x || count < 1) return 0;
+ const size_t a = XB.align, dr = (size_t)XB.D * 4;
+ const int v4 = XB.act == COLI_VK_ACT_SWIGLU_V4 && XB.p_v4;
+ size_t *off = malloc((size_t)count * 3 * sizeof(size_t));
+ if (!off) return 0;
+ size_t x0 = h * XB.sub_x, i0 = h * XB.sub_i, y0 = h * XB.sub_y, xe, ie, ye; int total;
+ if (!xb_plan(ex, rows, count, v4, x0, i0, y0, off, &xe, &ie, &ye, &total) ||
+ xe > x0 + XB.sub_x || ie > i0 + XB.sub_i || ye > y0 + XB.sub_y) { free(off); return 0; }
+ if (total > XB.sub[h].cyoff) {
+ size_t *n = realloc(XB.sub[h].yoff, (size_t)total * sizeof(*n));
+ if (!n) { free(off); return 0; }
+ XB.sub[h].yoff = n; XB.sub[h].cyoff = total;
+ }
+ /* the activation rows, packed per expert (the copies in parallel: thousands of rows) */
+ int *first = malloc((size_t)count * sizeof(int));
+ if (!first) { free(off); return 0; }
+ for (int c = 0, j = 0; c < count; j += rows[c], c++) first[c] = j;
+ size_t *yoff = XB.sub[h].yoff;
+ uint8_t *xp = XB.x.ptr;
+ #pragma omp parallel for schedule(dynamic, 4) if (total >= 256)
+ for (int c = 0; c < count; c++)
+ for (int r = 0; r < rows[c]; r++) {
+ int j = first[c] + r;
+ memcpy(xp + off[3 * c] + (size_t)r * dr, xrows[j], dr);
+ yoff[j] = off[3 * c + 2] + (size_t)r * dr;
+ if (v4) ((float *)(xp + off[3 * c] + xb_up((size_t)rows[c] * dr, a)))[r] = wrows ? wrows[j] : 1.0f;
+ }
+ free(first);
+ XB.sub[h].nrows = total;
+ int ngemm = 0;
+ int ok = xb_record_submit(XB.sub[h].cmd, XB.sub[h].fence, XB.sub[h].qp, ex, rows, count, off, v4, &ngemm);
+ free(off);
+ if (!ok) return 0;
+ XB.sub[h].inflight = 1; async_begin(0);
+ XB.sub_batches++; XB.sub_experts += (unsigned long long)count; XB.sub_rows += (unsigned long long)total;
+ XB.sub_gemm += (unsigned long long)ngemm;
+ return 1;
+}
+int coli_vk_xb_sub_join(int h, float *const *yout, double *device_ms) {
+ if (device_ms) *device_ms = 0;
+ if (h < 0 || h > 1 || !XB.sub[h].inflight) return 0;
+ XB.sub[h].inflight = 0;
+ VkResult r = vk_fence_wait_gemm(G.dev, XB.sub[h].fence);
+ async_end(0);
+ if (r != VK_SUCCESS) {
+ fprintf(stderr, "[VK] expert batch: fence wait failed (%d), the tier stops\n", r);
+ XB.ready = 0; return 0;
+ }
+ double ms = 0;
+ if (XB.has_ts && XB.sub[h].qp) {
+ uint64_t ts[2];
+ if (vkGetQueryPoolResults(G.dev, XB.sub[h].qp, 0, 2, sizeof ts, ts, sizeof ts[0], VK_QUERY_RESULT_64_BIT) == VK_SUCCESS) {
+ uint64_t mask = G.tq_ts >= 64 ? ~0ull : ((1ull << G.tq_ts) - 1);
+ ms = (double)((ts[1] - ts[0]) & mask) * G.ts_period / 1e6;
+ }
+ }
+ XB.sub_ms += ms;
+ if (device_ms) *device_ms = ms;
+ const size_t dr = (size_t)XB.D * 4;
+ const size_t *yoff = XB.sub[h].yoff;
+ const uint8_t *yp = XB.y.ptr;
+ int n = XB.sub[h].nrows;
+ #pragma omp parallel for schedule(static) if (n >= 256)
+ for (int j = 0; j < n; j++) memcpy(yout[j], yp + yoff[j], dr);
+ return 1;
}
static void xb_shutdown(void) {
if (!G.dev) return;
if (XB.inflight) { vk_fence_wait(G.dev, XB.fence); XB.inflight = 0; async_end(0); }
+ for (int h = 0; h < 2; h++) {
+ if (XB.sub[h].inflight) { vk_fence_wait(G.dev, XB.sub[h].fence); XB.sub[h].inflight = 0; async_end(0); }
+ if (XB.sub[h].fence) vkDestroyFence(G.dev, XB.sub[h].fence, NULL);
+ if (XB.sub[h].qp) vkDestroyQueryPool(G.dev, XB.sub[h].qp, NULL);
+ free(XB.sub[h].yoff);
+ }
while (XB.live) coli_vk_xb_expert_free(XB.live);
XbBuf *bufs[5] = {&XB.x, &XB.g, &XB.u, &XB.h, &XB.y};
for (int k = 0; k < 5; k++)
@@ -3226,6 +3712,8 @@ static void xb_shutdown(void) {
if (XB.sh_gu) vkDestroyShaderModule(G.dev, XB.sh_gu, NULL);
if (XB.sh_mv) vkDestroyShaderModule(G.dev, XB.sh_mv, NULL);
if (XB.sh_mm) vkDestroyShaderModule(G.dev, XB.sh_mm, NULL);
+ for (int k = 0; k < VK_COOP_SLOTS; k++) if (XB.p_coop[k]) vkDestroyPipeline(G.dev, XB.p_coop[k], NULL);
+ if (XB.sh_coop) vkDestroyShaderModule(G.dev, XB.sh_coop, NULL);
if (XB.pl6) vkDestroyPipelineLayout(G.dev, XB.pl6, NULL);
if (XB.pl4) vkDestroyPipelineLayout(G.dev, XB.pl4, NULL);
if (XB.pl3) vkDestroyPipelineLayout(G.dev, XB.pl3, NULL);
@@ -3280,6 +3768,24 @@ int coli_vk_tensor_commit(ColiVkTensor *const *t, int n) {
for (int k = 0; k < n; k++) if (t[k]) { free(t[k]->img); t[k]->img = NULL; }
return ok;
}
+/* A tier tensor filled again in place (a staging slot of the tier's streaming): mapped
+ * memory hands its mapping back (the padding past each row is still the zeros of its
+ * allocation, and the caller writes the rows only), staged uploads a fresh zeroed host
+ * image that coli_vk_tensor_commit copies over and frees. Not while a batch reads it. */
+int coli_vk_tensor_refill(ColiVkTensor *t, uint8_t **rows, size_t *stride, float **scales) {
+ if (!G.ready || !t || t->pool != &g_tpool) return 0;
+ if (g_up[t->dev].on) {
+ size_t sb = scale_floats(t->fmt, t->I, t->O, t->gs) * sizeof(float);
+ free(t->img);
+ if (!(t->img = calloc(1, t->wbytes + sb))) return 0;
+ *rows = t->img; *scales = (float *)(t->img + t->wbytes);
+ } else {
+ if (!t->wmap || !t->smap) return 0;
+ *rows = t->wmap; *scales = (float *)t->smap;
+ }
+ *stride = (size_t)t->rowWords * 4;
+ return 1;
+}
int coli_vk_staged(void) { return G.ready && g_up[0].on; }
size_t coli_vk_tensor_row_bytes(int fmt, int I) { return cpu_row_bytes(fmt, I); }
size_t coli_vk_buffer_alignment(void) { return G.buf_align ? G.buf_align : 256; }
@@ -3363,6 +3869,13 @@ void coli_vk_shutdown(void) {
}
g_async_inflight[0] = 0; /* idle after vkDeviceWaitIdle: the deferred frees run now */
tensor_reap(0);
+ pthread_mutex_lock(&g_imports_mx); /* imported rows: their memory objects go before the device */
+ for (ColiVkTensor *t = g_imports; t; t = t->imp_next) {
+ vkDestroyBuffer(G.dev, t->wbuf, NULL); vkFreeMemory(G.dev, t->imp, NULL);
+ t->wbuf = VK_NULL_HANDLE; t->imp = VK_NULL_HANDLE;
+ }
+ g_imports = NULL;
+ pthread_mutex_unlock(&g_imports_mx);
pool_destroy(&g_wpool); /* weight blocks: unmapped/freed with the device */
pool_destroy(&g_tpool);
if (PW.buf) { vkDestroyBuffer(G.dev, PW.buf, NULL); vkFreeMemory(G.dev, PW.mem, NULL); }
@@ -3432,10 +3945,10 @@ static void digest(const void *p, size_t n) {
}
static size_t ref_rowbytes(int fmt, int I) {
return fmt == 1 || fmt == 12 || fmt == 13 ? (size_t)I : fmt == 5 ? (size_t)((I + 63) / 64) * 24
- : fmt == 10 ? (size_t)I * 4 : fmt == 11 ? (size_t)I * 2 : (size_t)(I + 1) / 2;
+ : fmt == 10 ? (size_t)I * 4 : fmt == 11 || fmt == 14 ? (size_t)I * 2 : (size_t)(I + 1) / 2;
}
static size_t ref_scales(int fmt, int I, int O) { // scale COUNT (per-group for fmt 4/5)
- if (fmt == 10 || fmt == 11) return 1;
+ if (fmt == 10 || fmt == 11 || fmt == 14) return 1;
if (fmt == 5) return (size_t)O * (size_t)((I + 63) / 64);
if (fmt == 4 || fmt == 7 || fmt == 12 || fmt == 13) return (size_t)O * (size_t)((I + g_ref_gs - 1) / g_ref_gs);
return (size_t)O;
@@ -3449,6 +3962,10 @@ static float deq(const uint8_t *row, int fmt, int i) {
return (b & 0x80) ? -v : v; }
if (fmt == 11) { uint16_t h; memcpy(&h, row + (size_t)i * 2, 2); uint32_t u = (uint32_t)h << 16;
float f; memcpy(&f, &u, 4); return f; }
+ if (fmt == 14) { uint16_t h; memcpy(&h, row + (size_t)i * 2, 2); /* f16: normals and subnormals */
+ int e = (h >> 10) & 31, m = h & 1023;
+ float v = e == 0 ? ldexpf((float)m, -24) : ldexpf(1.0f + m / 1024.0f, e - 15);
+ return (h & 0x8000) ? -v : v; }
if (fmt == 5) { // int3-g64: 16B low plane (2 bits) + 8B high plane (1 bit), v+4
const uint8_t *lo = row + (size_t)(i >> 6) * 24, *hi = lo + 16; int j = i & 63;
unsigned u = ((lo[j >> 2] >> ((j & 3) * 2)) & 3u) | (((hi[j >> 3] >> (j & 7)) & 1u) << 2);
@@ -3474,7 +3991,7 @@ static void cpu_ref(float *y, const float *x, const uint8_t *w, const float *sc,
y[s * O + o] = (float)sum;
} else {
for (int i = 0; i < I; i++) sum += x[s * I + i] * deq(row, fmt, i);
- y[s * O + o] = (float)(fmt == 10 || fmt == 11 ? sum : sum * sc[o]);
+ y[s * O + o] = (float)(fmt == 10 || fmt == 11 || fmt == 14 ? sum : sum * sc[o]);
}
}
}
@@ -3514,6 +4031,19 @@ static void case_fill(int fmt, int S, int I, int O, float **x, uint8_t **w, floa
if (fmt == 10) memcpy(*w + i * 4, &f, 4);
else { uint32_t u; memcpy(&u, &f, 4); uint16_t h = (uint16_t)(u >> 16); memcpy(*w + i * 2, &h, 2); }
}
+ if (fmt == 14) /* f16: the same values as 10 and 11, rounded to f16 */
+ for (size_t i = 0; i < (size_t)I * O; i++) {
+ float f = (float)((rand() % 2001 - 1000) / 1000.0), a = fabsf(f);
+ int e; float m = frexpf(a, &e); /* a = m 2^e, m in [0.5, 1) */
+ uint16_t h = 0;
+ if (a >= ldexpf(1.0f, -14)) { /* normal: 10 bits after the leading one */
+ int q = (int)lrintf(ldexpf(m, 11)); /* in [1024, 2048] */
+ if (q == 2048) { q = 1024; e++; }
+ h = (uint16_t)(((e - 1 + 15) << 10) | (q - 1024));
+ } else h = (uint16_t)lrintf(ldexpf(a, 24)); /* subnormal */
+ if (f < 0) h |= 0x8000;
+ memcpy(*w + i * 2, &h, 2);
+ }
for (size_t o = 0; o < nsc; o++) (*sc)[o] = 0.01f + (rand() % 100) / 10000.0f;
}
@@ -4009,6 +4539,112 @@ static int run_qprep(int fmt, int S, int I, int Oqa, int Okva, int Oqb) {
return mq > 1e-2f || mk > 1e-3f;
}
+/* f16 weights (fmt 14) and rows imported in place (coli_vk_tensor_import), after the
+ * digest above, which stays the one of the formats before them (tests/vulkan_engines.sh
+ * staged compares it across builds), with a digest line of their own. The imports: an
+ * int8, an f32 and an f16 matrix in page-aligned host memory, through the GEMV and the
+ * fp32 GEMM, against the reference and bit for bit against the same matrix uploaded as
+ * a copy; a device without the extension (or staged uploads) says so and skips them. */
+static int run_import_case(int fmt, int S, int I, int O) {
+ size_t al = coli_vk_import_alignment();
+ if (!al) { printf("import fmt=%d: not possible on this device, skipped\n", fmt); return 0; }
+ float *x, *sc; uint8_t *w0;
+ case_fill(fmt, S, I, O, &x, &w0, &sc);
+ size_t rb = ref_rowbytes(fmt, I), bytes = rb * O, sz = (bytes + al - 1) / al * al;
+ void *w = NULL;
+ if (posix_memalign(&w, al, sz)) { printf("import: out of memory\n"); return 1; }
+ memset(w, 0, sz); memcpy(w, w0, bytes);
+ float *yi = malloc((size_t)S * O * 4), *yc = malloc((size_t)S * O * 4), *yr = malloc((size_t)S * O * 4);
+ ColiVkTensor *ti = NULL, *tc = NULL;
+ int bad = 0, keep_s = G.gemm_min_s, keep_so = G.gemm_min_so;
+ if (S > 1) { G.gemm_min_s = 2; G.gemm_min_so = 1; } /* a block of rows: the fp32 GEMM */
+ size_t before = coli_vk_imported_bytes();
+ if (!coli_vk_tensor_import(&ti, w, sz, sc, fmt, I, O, 0)) { printf("import fmt=%d: refused\n", fmt); bad = 1; }
+ else if (coli_vk_imported_bytes() != before + bytes) { printf("import fmt=%d: the bytes were not counted\n", fmt); bad = 1; }
+ else if (!coli_vk_matmul(&ti, yi, x, w, sc, fmt, S, I, O, 0) || !coli_vk_matmul(&tc, yc, x, w0, sc, fmt, S, I, O, 0)) {
+ printf("import fmt=%d: matmul failed\n", fmt); bad = 1;
+ } else {
+ digest(yi, (size_t)S * O * 4);
+ cpu_ref(yr, x, w0, sc, fmt, S, I, O);
+ double maxrel = 0;
+ for (int i = 0; i < S * O; i++)
+ if (fabs(yr[i]) > 1e-2) { double r = fabs(yi[i] - yr[i]) / fabs(yr[i]); if (r > maxrel) maxrel = r; }
+ int same = !memcmp(yi, yc, (size_t)S * O * 4);
+ printf("import fmt=%d S=%d I=%d O=%d %s | maxrel=%.4g | the copy's bits: %s\n", fmt, S, I, O,
+ G.bound_gemm ? "gemm" : "gemv", maxrel, same ? "same" : "DIFFERENT");
+ bad = maxrel > 1e-3 || !same;
+ }
+ G.gemm_min_s = keep_s; G.gemm_min_so = keep_so;
+ coli_vk_tensor_free(ti); coli_vk_tensor_free(tc);
+ if (coli_vk_imported_bytes() != before) { printf("import fmt=%d: the bytes were not given back\n", fmt); bad = 1; }
+ free(w); free(w0); free(x); free(sc); free(yi); free(yc); free(yr);
+ return bad;
+}
+static int run_gemm_case(int fmt, int S, int I, int O, int gs);
+/* COLI_VK_TEST_IMPORT_BENCH=1: one Qwen3.8-27B MLP matrix (I 5120, O 17408) in int8 and
+ * f16, as a copy in the weight pool and imported in place, at S = 1 (the GEMV) and S =
+ * 300 (a decision's prompt, the tiled GEMM): the wall time of a call, best of 5. */
+static void bench_import(void) {
+ size_t al = coli_vk_import_alignment();
+ if (!al) { printf("import bench: imports not possible on this device\n"); return; }
+ const int I = 5120, O = 17408, Ss[2] = {1, 300}, fm[2] = {1, 14};
+ for (int f = 0; f < 2; f++) {
+ int fmt = fm[f];
+ float *x, *sc; uint8_t *w0;
+ case_fill(fmt, 300, I, O, &x, &w0, &sc);
+ size_t bytes = ref_rowbytes(fmt, I) * O, sz = (bytes + al - 1) / al * al;
+ void *w = NULL;
+ if (posix_memalign(&w, al, sz)) return;
+ memcpy(w, w0, bytes);
+ float *y = malloc((size_t)300 * O * 4);
+ ColiVkTensor *tc = NULL, *ti = NULL;
+ if (!coli_vk_tensor_ensure(&tc, w0, sc, fmt, I, O, 0) || !coli_vk_tensor_import(&ti, w, sz, sc, fmt, I, O, 0)) {
+ printf("import bench fmt=%d: no tensor\n", fmt); return;
+ }
+ for (int k = 0; k < 2; k++) {
+ double best[2] = {1e30, 1e30};
+ int keep_s = G.gemm_min_s, keep_so = G.gemm_min_so;
+ if (Ss[k] > 1) { G.gemm_min_s = 2; G.gemm_min_so = 1; } /* the harness keeps GEMVs; a prompt takes the GEMM */
+ for (int which = 0; which < 2; which++) {
+ ColiVkTensor **t = which ? &ti : &tc;
+ coli_vk_matmul(t, y, x, which ? w : w0, sc, fmt, Ss[k], I, O, 0);
+ for (int r = 0; r < 5; r++) {
+ double t0 = now();
+ coli_vk_matmul(t, y, x, which ? w : w0, sc, fmt, Ss[k], I, O, 0);
+ double ms = (now() - t0) * 1000;
+ if (ms < best[which]) best[which] = ms;
+ }
+ }
+ printf("import bench fmt=%d S=%d (%s): copy %.2f ms, imported %.2f ms (%.1f MB of rows)\n", fmt, Ss[k],
+ G.bound_gemm == 2 ? "coop" : G.bound_gemm ? "gemm" : "gemv", best[0], best[1], bytes / 1e6);
+ G.gemm_min_s = keep_s; G.gemm_min_so = keep_so;
+ }
+ coli_vk_tensor_free(tc); coli_vk_tensor_free(ti);
+ free(w); free(w0); free(x); free(sc); free(y);
+ }
+}
+static int run_late_cases(void) {
+ if (getenv("COLI_VK_TEST_IMPORT_BENCH") && atoi(getenv("COLI_VK_TEST_IMPORT_BENCH"))) bench_import();
+ uint64_t keep = g_digest;
+ g_digest = 1469598103934665603ULL;
+ int bad = 0;
+ bad |= run_case(14, 1, 6144, 1536, 20);
+ bad |= run_case(14, 8, 2048, 512, 10);
+ bad |= run_case(14, 1, 16384, 512, 5);
+ bad |= run_case(14, 3, 37, 5, 5); // odd I: the last word holds one f16
+ bad |= run_gemm_case(14, 16, 1001, 193, 64);
+ bad |= run_gemm_case(14, 64, 1536, 97, 64);
+ bad |= run_gemm_case(14, 512, 520, 131, 64);
+ bad |= run_import_case(1, 1, 2048, 384);
+ bad |= run_import_case(1, 64, 2048, 384);
+ bad |= run_import_case(10, 3, 1024, 100);
+ bad |= run_import_case(14, 1, 4096, 257);
+ bad |= run_import_case(14, 96, 4096, 257);
+ printf("f16 and import digest %016llx\n", (unsigned long long)g_digest);
+ g_digest = keep;
+ return bad;
+}
+
/* ---- the expert tier's device side: weight pool and async expert batch ----------
* Reference dot of one weight row in any format, its scales applied the way the
* shader does (per row for 1/2, per gs-group for 4/7/12/13, per 64 for 5, none for
@@ -4055,7 +4691,9 @@ static void xmat_drop(XMat *m) { free(m->w); free(m->s); m->w = NULL; m->s = NUL
* experts. */
static int run_xbatch(int fmt, int dfmt, int gs, int D, int I, int act, float limit) {
enum { K = 5 };
- static const int rows[K] = {1, 3, 2, 20, 1};
+ const char *large = getenv("COLI_VK_TEST_XB_LARGE");
+ int rows[K] = {1, 3, 2, 20, 1};
+ if (large && atoi(large)) { rows[2] = 64; rows[3] = 257; rows[4] = 20; }
XMat mg[K], mu[K], md[K]; ColiVkExpert *ex[K];
XB.act = act; XB.limit = limit; XB.a = 4.0f; XB.b = 25.0f;
for (int c = 0; c < K; c++) {
@@ -4072,6 +4710,13 @@ static int run_xbatch(int fmt, int dfmt, int gs, int D, int I, int act, float li
int j = 0;
for (int c = 0; c < K; c++)
for (int r = 0; r < rows[c]; r++, j++) {
+ if (large && atoi(large) && c >= 1) {
+ /* Device-side scaling must preserve zero and tiny rows as well
+ * as values that would overflow an unscaled fp16 conversion. */
+ static const int powers[] = {-40, 0, 10, 20};
+ for (int d = 0; d < D; d++)
+ x[(size_t)j * D + d] = r ? ldexpf(x[(size_t)j * D + d], powers[(r + 3) % 4]) : 0;
+ }
const float *xs = x + (size_t)j * D;
size_t grb = ref_rowbytes(fmt, D), drb = ref_rowbytes(dfmt, I);
for (int o = 0; o < I; o++) {
@@ -4082,18 +4727,32 @@ static int run_xbatch(int fmt, int dfmt, int gs, int D, int I, int act, float li
for (int d = 0; d < D; d++) ref[(size_t)j * D + d] = (float)xref_dot(hid, md[c].w + (size_t)d * drb, md[c].s, d, dfmt, I, gs);
}
int bad = 0; double dms = 0;
+ unsigned long long coop_before = XB.cooperative_matmuls, coop_expected = 0;
+ for (int c = 0; c < K; c++) {
+ int slot = gemm_slot(rows[c]);
+ if (XB.gemm_rows && rows[c] >= XB.gemm_rows && XB.p_mm[slot])
+ coop_expected += (xb_coop_slot(mg[c].t, rows[c], slot) >= 0) +
+ (xb_coop_slot(mu[c].t, rows[c], slot) >= 0) +
+ (xb_coop_slot(md[c].t, rows[c], slot) >= 0);
+ }
if (!coli_vk_xb_issue(ex, rows, K, xr) || !coli_vk_xb_join(yr, &dms)) { printf("xbatch fmt=%d/%d: issue/join failed\n", fmt, dfmt); bad = 1; }
+ unsigned long long coop_done = XB.cooperative_matmuls - coop_before;
+ if (coop_done != coop_expected) { printf("xbatch: cooperative matmuls %llu, expected %llu\n", coop_done, coop_expected); bad = 1; }
for (int jj = 0; !bad && jj < total; jj++) digest(yr[jj], (size_t)D * 4);
- double maxrel = 0, scale = 0;
- for (int i = 0; i < total * D; i++) if (fabs(ref[i]) > scale) scale = fabs(ref[i]);
- for (int jj = 0; !bad && jj < total; jj++)
+ double maxrel = 0;
+ for (int jj = 0; !bad && jj < total; jj++) {
+ double scale = 0;
+ for (int d = 0; d < D; d++) if (fabs(ref[(size_t)jj * D + d]) > scale) scale = fabs(ref[(size_t)jj * D + d]);
for (int d = 0; d < D; d++) {
double r = ref[(size_t)jj * D + d], e = fabs(yr[jj][d] - r);
+ if (!isfinite(yr[jj][d])) { bad = 1; break; }
+ if (scale == 0) { if (yr[jj][d] != 0) bad = 1; continue; }
/* relative to the row's own magnitude where it is not tiny; cancellation-heavy
* random rows make single elements near zero meaningless */
double rel = e / (fabs(r) > 1e-2 * scale ? fabs(r) : 1e-2 * scale);
if (rel > maxrel) maxrel = rel;
}
+ }
/* the first expert's row alone, then the same row as the 2nd of 3: same bits */
float *y1 = malloc((size_t)D * 4);
int same = 1;
@@ -4107,8 +4766,9 @@ static int run_xbatch(int fmt, int dfmt, int gs, int D, int I, int act, float li
else if (!bad) same &= !memcmp(y1, yr[1], (size_t)D * 4);
}
const char *an = act == COLI_VK_ACT_SITU ? "situ" : limit > 0 ? "swiglu-limit" : "swiglu";
- printf("xbatch fmt=%d/%d gs=%d D=%d I=%d %s, rows 1/3/2/20/1 (GEMM from %d) | maxrel=%.3g | row bits independent of the batch: %s | device %.3f ms\n",
- fmt, dfmt, gs, D, I, an, XB.gemm_rows, maxrel, same ? "yes" : "NO", dms);
+ printf("xbatch fmt=%d/%d gs=%d D=%d I=%d %s, rows %d/%d/%d/%d/%d (GEMM from %d, %llu cooperative matmuls) | maxrel=%.3g | row bits independent of the batch: %s | device %.3f ms\n",
+ fmt, dfmt, gs, D, I, an, rows[0], rows[1], rows[2], rows[3], rows[4], XB.gemm_rows,
+ coop_done, maxrel, same ? "yes" : "NO", dms);
if (maxrel > 3e-3 || !same) bad = 1;
for (int c = 0; c < K; c++) { coli_vk_xb_expert_free(ex[c]); xmat_drop(&mg[c]); xmat_drop(&mu[c]); xmat_drop(&md[c]); }
free(x); free(ref); free(hid); free(xr); free(yr); free(y1);
@@ -4336,6 +4996,8 @@ static int run_xbatch_all(void) {
bad |= run_xbatch(fm[f], fm[f], gs, 320, 160, COLI_VK_ACT_SWIGLU, 0.f);
}
bad |= run_xbatch(4, 1, 64, 320, 160, COLI_VK_ACT_SWIGLU, 0.f); /* int4 gate/up, int8 down */
+ bad |= run_xbatch(4, 10, 64, 320, 160, COLI_VK_ACT_SWIGLU, 0.f); /* cooperative gate/up, fp32 down */
+ bad |= run_xbatch(10, 1, 64, 320, 160, COLI_VK_ACT_SWIGLU, 0.f); /* fp32 gate/up, cooperative down */
bad |= run_xbatch(4, 4, 64, 320, 160, COLI_VK_ACT_SWIGLU, 0.7f); /* the clamped SwiGLU */
bad |= run_xbatch(7, 7, 32, 320, 160, COLI_VK_ACT_SITU, 0.f); /* SiTU-GLU on MXFP4 */
xb_shutdown();
@@ -4577,6 +5239,7 @@ int main(int argc, char **argv) {
* take most of the time. */
if (getenv("COLI_VK_TEST_MATMUL_ONLY") && atoi(getenv("COLI_VK_TEST_MATMUL_ONLY"))) {
printf("outputs digest %016llx\n", (unsigned long long)g_digest);
+ bad |= run_late_cases();
printf(bad ? "FAIL\n" : "PASS\n");
coli_vk_shutdown();
return bad;
@@ -4697,6 +5360,7 @@ int main(int argc, char **argv) {
g_absorb_rewind = 0;
}
printf("outputs digest %016llx\n", (unsigned long long)g_digest);
+ bad |= run_late_cases();
printf(bad ? "FAIL\n" : "PASS\n");
coli_vk_shutdown();
return bad;
diff --git a/c/backend_vulkan.h b/c/backend_vulkan.h
index 1ed41d6e..a76f0405 100644
--- a/c/backend_vulkan.h
+++ b/c/backend_vulkan.h
@@ -70,8 +70,21 @@ int coli_vk_expert_group_issue(ColiVkTensor *const *gates, ColiVkTensor *const
int coli_vk_expert_group_take(float *y);
/* Upload a resident tensor without computing (expert tier: gate/up/down uploaded once,
- * then driven by coli_vk_expert_group). Returns 0 on failure/unsupported fmt. */
+ * then driven by coli_vk_expert_group). Returns 0 on failure/unsupported fmt.
+ * fmt 14 = f16 weights (low half = even column), no scales, like 10 and 11. */
int coli_vk_tensor_ensure(ColiVkTensor **tensor, const void *weights, const float *scales, int fmt, int I, int O, int grp);
+/* The same tensor with its rows read where the host keeps them (VK_EXT_external_memory_host),
+ * no device copy: for a device that shares the CPU's RAM, where a copy would hold the
+ * matrix twice. weights must be aligned to coli_vk_import_alignment() and its allocation
+ * (alloc_bytes) must cover the rows rounded up to that alignment; the rows must need no
+ * padding (cpu row bytes a multiple of 4). The host memory must outlive the tensor
+ * (coli_vk_tensor_free first). 0 = not possible here: the caller uploads a copy instead.
+ * coli_vk_import_alignment() is 0 where imports are not possible (no extension, staged
+ * uploads); coli_vk_imported_bytes() the rows read in place now. */
+int coli_vk_tensor_import(ColiVkTensor **tensor, const void *weights, size_t alloc_bytes, const float *scales,
+ int fmt, int I, int O, int gs);
+size_t coli_vk_import_alignment(void);
+size_t coli_vk_imported_bytes(void);
/* SECOND DEVICE (COLI_VK_DEV2): a self-contained context on another Vulkan GPU that
* hosts ONLY tier experts and runs ONLY the async expert-group path. devidx: -1 =
@@ -213,8 +226,31 @@ typedef struct {
double device_ms; /* summed batch device time (timestamps) */
int timestamps, queue_shared, gemm_rows;
size_t scratch_bytes;
+ unsigned long long sub_batches, sub_experts, sub_rows, sub_gemm; /* coli_vk_xb_sub_* */
+ double sub_ms;
+ unsigned long long cooperative_matmuls; /* all projections, including sub-batches */
} ColiVkXbStats;
void coli_vk_xb_stats(ColiVkXbStats *st);
+/* Sub-batches, for a prefill step too big for one batch (vk_tier.c's streaming): batch k
+ * of a step goes to half k % 2 of the scratch, so two run at once and the host fills one
+ * while the device computes the other. _reserve sizes each half for a batch of up to
+ * `rows` rows over at most `experts` experts (not while any batch is in flight; it may
+ * move the scratch). _issue submits one into half h and returns (0: nothing submitted);
+ * _join waits for it and copies the D outputs of its row j to yout[j], same order as
+ * xrows; *device_ms its device time when timestamps exist. The single batch above and
+ * the sub-batches do not overlap: neither starts while the other is in flight. */
+/* Largest sub-batch up to rows that can grow all five buffers within extra_budget
+ * and half each heap's free budget; does not allocate. Zero means no row fits. */
+int coli_vk_xb_sub_fit(int rows, int experts, size_t extra_budget);
+int coli_vk_xb_sub_reserve(int rows, int experts);
+int coli_vk_xb_sub_issue(int h, ColiVkExpert *const *ex, const int *rows, int count, const float *const *xrows,
+ const float *wrows);
+int coli_vk_xb_sub_join(int h, float *const *yout, double *device_ms);
+int coli_vk_xb_sub_busy(int h);
+/* A tier tensor (coli_vk_tier_tensor) to fill again in place: rows and scales as
+ * coli_vk_tier_tensor hands them out, then coli_vk_tensor_commit. 0 = no memory. Not
+ * while a batch that reads it is in flight. */
+int coli_vk_tensor_refill(ColiVkTensor *t, uint8_t **rows, size_t *stride, float **scales);
/* 1 if the selected device is an integrated GPU (shares physical memory with
* the host), 0 otherwise or when no device is selected. */
@@ -243,6 +279,25 @@ int coli_vk_init_env_tier(const char *engine, int tier_on);
* an engine name it is printed as a [VK] line when it changes (NULL: silent). */
int coli_vk_dense_decide(const char *engine, int tier_on, int def);
int coli_vk_dense(void); /* the last decision; 1 before any */
+/* Whether the dense weights the device holds keep a host copy (docs/vulkan.md, "Dense
+ * weights on the device only"). With the dense part on the device (dense_on_device: the
+ * chain on, or the per-matrix path on), an engine in this mode uploads each resident
+ * dense matrix once, drops its host copy, and reads it back from disk only when the CPU
+ * needs it (a lost device, a step the device declines). COLI_VK_DENSE_HOST set and
+ * non-empty: 0 device only, any other number keeps the host copies. Unset: device only
+ * on an integrated GPU (its memory is the same RAM) and on a discrete GPU whose free
+ * memory, less 1 GiB, holds dense_bytes; host copies kept on a CPU device (Lavapipe)
+ * and when the dense part runs on the CPU. Printed as a [VK] line with an engine name;
+ * with the dense part on the device, one more at exit: in the mode, what was dropped and
+ * read back, and the resident set either way. Returns 1 for device only. */
+int coli_vk_dense_host_decide(const char *engine, int dense_on_device, size_t dense_bytes);
+int coli_vk_dense_device_only(void); /* the last decision; 0 before any */
+void coli_vk_dense_host_dropped(size_t bytes); /* a host copy given back after its upload */
+void coli_vk_dense_host_reloaded(size_t bytes);/* a host copy read back from disk for the CPU */
+/* One [VK] line once the engine has placed its dense matrices: how many are on the device
+ * only, the RAM given back, and what stays on the host (kept: NULL or a short note). */
+void coli_vk_dense_host_placed(const char *engine, const char *kept);
+unsigned long long coli_vk_dense_host_dropped_bytes(void);
/* COLI_VK_SHADERS (the .spv or its directory), else shaders/ next to the binary, else
* shaders/ in the working directory. buf holds the result when it is not a literal. */
const char *coli_vk_shader_path(char *buf, size_t n);
diff --git a/c/colibri.c b/c/colibri.c
index f80c8964..d6620c5d 100644
--- a/c/colibri.c
+++ b/c/colibri.c
@@ -70,6 +70,7 @@
#include "tok.h"
#include "tier.h"
#include "grammar.h" /* metodo F: draft grammaticali (#48) */
+#include "spec_draft.h" /* COLI_LOOKUP=1: the n-gram drafts from spec_draft.h, gated */
#include "abl.h" /* per-expert causal-ablation harness — inert unless g_abl.mode set (ABLATE_SCORE=<manifest>) */
#include "evidence_digest.h" /* SHA-256 over the bytes an evidence mode consumed */
#include "schema_gbnf.h" /* SCHEMA=: JSON-Schema -> GBNF for method F */
@@ -265,6 +266,10 @@ typedef struct {
#endif
#ifdef COLI_VULKAN
ColiVkTensor *vk; int vk_eligible; /* resident on the Vulkan expert tier */
+ /* the dense weights on the device only (COLI_VK_DENSE_HOST): the tensor's name and the
+ * bits qt_load quantized it to, to read it back (qt_dho_reload); vk_gone = 1 while the
+ * device holds it alone (no host copy) */
+ char *vk_name; int vk_bits, vk_gone;
#endif
#ifdef COLI_XDNA
/* Derived, disposable BF16 host image for the XDNA lane. NULL for an
@@ -755,6 +760,17 @@ static void qt_vk_reset(QT *t){
* int8/int4/int3-g64 weight once (t->vk), then reuses it. Returns 0 (caller runs the CPU
* matmul) when VK-dense is off, the format is unsupported, or upload/compute fails. */
#define VK_FMT_OK(t) ((t)->fmt==1||(t)->fmt==2||(t)->fmt==5||((t)->fmt==4&&(t)->gs>=8&&(t)->gs%8==0))
+/* The dense weights on the device only (COLI_VK_DENSE_HOST, glm_dho_start below): a QT
+ * whose host copy was given back (vk_gone) is multiplied by its device copy wherever the
+ * CPU would have read it (matmul_qt_ex), from the engine's thread outside any parallel
+ * region; anywhere else, and once the device is lost, it is read back from disk first. */
+static void qt_dho_reload(QT *t);
+static pthread_t g_dho_thread;
+static int qt_vk_fmt(const QT *t){ return t->fmt==0 ? 10 : t->fmt; }
+static int qt_dho_matmul(QT *t, float *y, const float *x, int S){
+ if(omp_in_parallel() || !pthread_equal(pthread_self(), g_dho_thread) || !t->vk) return 0;
+ return coli_vk_matmul(&t->vk, y, x, NULL, NULL, qt_vk_fmt(t), S, t->I, t->O, t->fmt==4 ? t->gs : 0);
+}
static int vk_matmul_qt(QT *t, float *y, const float *x, int S){
if(!g_vk_dense || !VK_FMT_OK(t)) return 0;
const void *w = t->fmt==1 ? (const void*)t->q8 : (const void*)t->q4;
@@ -1246,6 +1262,12 @@ static inline int pq_want(const QT *g,const QT *u,int nr){
* projections need it: IDOT's int8 activation quantization costs +0.117 nats/token there
* (~+12% perplexity), measured. Every other prefill matmul keeps IDOT as before. */
static void matmul_qt_ex(float *y, const float *x, QT *w, int S, int allow_idot){
+#ifdef COLI_VULKAN
+ if(__atomic_load_n(&w->vk_gone, __ATOMIC_ACQUIRE)){
+ if(qt_dho_matmul(w, y, x, S)) return;
+ qt_dho_reload(w);
+ }
+#endif
#ifdef COLI_METAL
/* fmt=8 (fp8 passthrough) deliberately absent from this allowlist, same as fmt=5/6:
* the S>=g_metal_gemm_min batched prefill GEMM path (coli_metal_gemm) only knows
@@ -1397,6 +1419,13 @@ static float g_route_alpha=1.f; /* ROUTE_ALPHA: scale gate mass of CACHE_ROUTE s
static int g_route_agree=0; /* ROUTE_AGREE=1: footer overlap% + mean KL vs true top-K */
static int expert_is_resident(Model *m, int layer, int eid); /* pin∪LRU; defined near pilot */
static int g_spec=1; /* metodo C: SPEC=0 disabilita il prefetch speculativo cross-layer */
+/* COLI_LOOKUP=1 (no MTP head): the n-gram source proposes from spec_draft.h's prompt
+ * lookup -- the longest suffix of 4 down to 2 tokens found earlier in the context, its
+ * most recent occurrence -- instead of the last bigram, and its gate decides how many of
+ * the DRAFT proposals a verify carries, from the measured acceptance by position and
+ * the measured forward time by rows (COLI_SPEC_GATE=0 drafts every proposal in full).
+ * Unset: the bigram source as before. */
+static int g_lookup=0; static SpecGate g_lookup_gate;
static int g_draft=0; /* metodo E: DRAFT=n token auto-speculati per forward via n-gram lookup
* (0=off). LOSSLESS: verifica = output identico al greedy. Default OFF:
* misurato sul run reale (2026-07-03) acceptance ~5% -> ogni draft
@@ -2295,6 +2324,10 @@ static int qt_load_mmap(Model *m, const char *name, int O, int I, QT *t){
static QT qt_load(Model *m, const char *name, int O, int I, int bits){
QT t; memset(&t,0,sizeof(t)); qt_from_disk(m,name,O,I,bits,0,&t);
+#ifdef COLI_VULKAN
+ t.vk_bits=bits; /* to read it back, should the device end up holding it alone */
+ if(!(t.vk_name=strdup(name))){ fprintf(stderr,"OOM tensor name\n"); exit(1); }
+#endif
#ifdef COLI_CUDA
if(g_cuda_enabled&&g_cuda_dense){
t.cuda_eligible=1;
@@ -4001,6 +4034,9 @@ static float fp8_block_scale(float sc, int64_t blkO, int64_t bi, const char *who
}
static void qt_addrow(const QT *t, int row, float coef, float *acc){
+#ifdef COLI_VULKAN
+ if(__atomic_load_n(&t->vk_gone, __ATOMIC_ACQUIRE)) qt_dho_reload((QT *)t); /* the CPU's attention reads its rows */
+#endif
int I=t->I;
if(t->fmt==0){ const float *w=t->qf+(int64_t)row*I; for(int i=0;i<I;i++) acc[i]+=coef*w[i]; return; }
/* fmt=4 PRIMA del calcolo di c: s[] e' [O,ng] per-gruppo, s[row] sarebbe la scala
@@ -4074,6 +4110,9 @@ static void qt_addrow(const QT *t, int row, float coef, float *acc){
}
/* y[0..n) = W[r0+j,:]·x (matvec su una FETTA di righe del QT) */
static void qt_matvec_rows(const QT *t, int r0, int n, const float *x, float *y){
+#ifdef COLI_VULKAN
+ if(__atomic_load_n(&t->vk_gone, __ATOMIC_ACQUIRE)) qt_dho_reload((QT *)t);
+#endif
int I=t->I;
for(int j=0;j<n;j++){ int row=r0+j; double a=0;
if(t->fmt==0){ const float *w=t->qf+(int64_t)row*I; for(int i=0;i<I;i++) a+=(double)w[i]*x[i]; }
@@ -5549,9 +5588,11 @@ static int moe_vk_on(Model *m,int layer){
* from the pin set, the LRU or a working-set slot loaded from disk (PIPE or the blocking
* parallel load), computed once for all its rows into ctb[i]; the block's misses then
* promoted into the LRU. note: offer each computed expert to the tier. */
+static void vk_stream_promote(Model *m);
static void vk_cpu_pairs(Model *m,int layer,const float *x,int rows,int K,const int *ib,const uint8_t *want,
float *ctb,float *xg,float *gg,float *uu,float *hh,int note){
Cfg *c=&m->c; int D=c->hidden, I=c->moe_inter, E=c->n_experts, n=rows*K;
+ vk_stream_promote(m); /* the streamed misses still in the working set join the LRU first */
int *uq=xalloc((size_t)n*sizeof(int),"vk uq"), *rws=xalloc((size_t)rows*sizeof(int),"vk rws"), nu=0;
unsigned char *seen=xzalloc((size_t)E,"vk seen");
for(int i=0;i<n;i++) if(want[i] && ib[i]>=0 && !seen[ib[i]]){ seen[ib[i]]=1; uq[nu++]=ib[i]; }
@@ -5604,7 +5645,7 @@ static void vk_cpu_pairs(Model *m,int layer,const float *x,int rows,int K,const
static void moe_vk(Model *m,int layer,const float *x,int S,float *out,const int *idxs,const float *ws,
const int *keff,float *xg,float *gg,float *uu,float *hh){
Cfg *c=&m->c; int D=c->hidden, K=c->topk, E=c->n_experts;
- int B = S<GLM_VK_ROWS ? S : GLM_VK_ROWS, nmax=B*K, dev2=g_vk_reg_n2>0;
+ int B = vkt_step_rows(S,GLM_VK_ROWS), nmax=B*K, dev2=g_vk_reg_n2>0; /* a whole prompt chunk when the tier streams */
int *ib=xalloc((size_t)nmax*sizeof(int),"vk ib"), *r2=xalloc((size_t)nmax*sizeof(int),"vk r2");
uint8_t *taken=xalloc((size_t)nmax,"vk taken"), *want=xalloc((size_t)nmax,"vk want"), *on2=xzalloc((size_t)nmax,"vk on2");
const float **dev=xalloc((size_t)nmax*sizeof(*dev),"vk dev");
@@ -8134,6 +8175,11 @@ static int spec_decode(Model *m, int *all, int kv, int n_new, int eos, float *lo
}
if(!g && g_draft>0 && !(m->has_mtp && gd_pause>0)){
if(m->has_mtp){ g=mtp_draft(m,next,kv,g_draft,draft); m->mtp_prop+=g; if(g)gsrc=2; }
+ else if(g_lookup){
+ int n=spec_lookup(all,kv+1,2,4,g_draft,draft);
+ g=n>0?spec_gate_pick(&g_lookup_gate,SPEC_SRC_LOOKUP,n,NULL):0;
+ if(g)gsrc=4;
+ }
else { g=ngram_draft(all,kv+1,g_draft,draft); if(g)gsrc=2; }
}
if(g>n_new-emitted) g=n_new-emitted;
@@ -8141,9 +8187,11 @@ static int spec_decode(Model *m, int *all, int kv, int n_new, int eos, float *lo
if(g<0) g=0;
if(gsrc==1) g_grd.prop+=(uint64_t)g;
int S=1+g; int batch[64]; batch[0]=next; memcpy(batch+1,draft,g*sizeof(int));
- double tf0=g_prof?now_s():0;
+ int lk=g_lookup&&!m->has_mtp;
+ double tf0=g_prof||lk?now_s():0;
float *lo=step_all(m,batch,S,kv); m->n_fw++;
if(g_prof) prof_lat(now_s()-tf0);
+ if(lk) spec_gate_forward(&g_lookup_gate,S,now_s()-tf0);
int k=0; /* verifica: accetta finche' coincide */
if(g>0 && getenv("MTP_DEBUG")){ int veri=argmax_v(lo,V);
fprintf(stderr,"[mtpdbg] draft0=%d verified=%d %s\n", draft[0], veri, draft[0]==veri?"HIT":"miss"); }
@@ -8158,6 +8206,7 @@ static int spec_decode(Model *m, int *all, int kv, int n_new, int eos, float *lo
gr_feed(&g_grd,draft[k]); k++;
}
if(gsrc==1) g_grd.acc+=(uint64_t)k;
+ else if(gsrc==4) spec_gate_result(&g_lookup_gate,SPEC_SRC_LOOKUP,k<g&&emitted<n_new&&!done?k+1:k,k);
else if(gsrc==2 && m->has_mtp) m->mtp_acc+=k;
else if(gsrc==3) g_corp_acc+=(uint64_t)k;
if(m->has_mtp && k>=1) mtp_absorb(m, all+kv+1, m->h_all, k, kv); /* KV MTP in sync coi verificati */
@@ -10751,6 +10800,43 @@ static int vk_in_ram(void *ctx,int layer,int eid){
Model *m=ctx;
return pin_indexed(m,layer,eid)!=NULL || ecache_indexed(m,layer,eid,0)!=NULL;
}
+/* The tier's streaming (a big prompt chunk's cold experts on the device): a group of
+ * experts as vk_cpu_pairs resolves them -- the pin set, the LRU, the misses read into
+ * the working-set slots in parallel -- each valid until the tier gives it back. The
+ * misses then join the LRU as vk_cpu_pairs's do, when the group's last one is given
+ * back (before anything else can reuse the working set). */
+static int g_vks_layer=-1, g_vks_nmiss=0, g_vks_held=0;
+static void vk_stream_promote(Model *m){
+ if(g_vks_nmiss>0 && g_vks_layer>=0) ecache_promote_ws(m,g_vks_layer,g_vks_nmiss);
+ g_vks_nmiss=0; g_vks_layer=-1; g_vks_held=0;
+}
+static int vk_load_batch(void *ctx,int layer,const int *e,int n,VktExpertSrc *srcs,void **h){
+ Model *m=ctx;
+ vk_stream_promote(m);
+ if(n<1) return 0;
+ if(n>64) n=64;
+ ESlot *use[64]; int miss[64], nmiss=0;
+ for(int j=0;j<n;j++){
+ use[j]=pin_indexed(m,layer,e[j]);
+ if(use[j]){ m->hits++; m->hit_pin++; continue; }
+ use[j]=ecache_indexed(m,layer,e[j],0);
+ if(use[j]){ m->hits++; m->hit_ecache++; use[j]->used=(uint64_t)__atomic_add_fetch(&m->eclock,1,__ATOMIC_RELAXED); continue; }
+ use[j]=&m->ws[nmiss]; miss[nmiss++]=j; m->miss++;
+ }
+ if(nmiss){ double t0=now_s();
+ #pragma omp parallel for schedule(dynamic,1)
+ for(int q=0;q<nmiss;q++) expert_load(m,layer,e[miss[q]],&m->ws[q],1,1);
+ m->t_ewait += now_s()-t0; }
+ int k=0;
+ for(;k<n;k++){ if(!vk_slot_src(use[k],&srcs[k])) break; h[k]=m; }
+ g_vks_layer=layer; g_vks_nmiss=nmiss; g_vks_held=k;
+ if(!k) vk_stream_promote(m);
+ return k;
+}
+static int vk_load(void *ctx,int layer,int e,VktExpertSrc *src,void **h){
+ return vk_load_batch(ctx,layer,&e,1,src,h)==1;
+}
+static void vk_unhold(void *ctx,void *h){ (void)h; if(g_vks_held>0 && --g_vks_held==0) vk_stream_promote((Model *)ctx); }
/* Is this model's expert set quantized at load (no .qs companions: qt_from_disk's
* qt_alloc buffers, one per tensor) rather than read from a container (a slab, or
* views of a COLI_MMAP mapping that own nothing)? */
@@ -10805,7 +10891,8 @@ static void vk_tier_start(Model *m){
.ram_reserve=(size_t)((double)m->ecap*expert_cache_row_bytes(m,m->ebits)),
.dense_bytes=(size_t)dense,
.in_ram=vk_in_ram, .ram_ctx=m,
- .max_experts=g_vk_experts>0 ? g_vk_experts : 0};
+ .max_experts=g_vk_experts>0 ? g_vk_experts : 0,
+ .load=vk_load, .release=vk_unhold, .load_ctx=m, .load_batch=vk_load_batch};
atexit(coli_vk_shutdown); /* before vkt_init, which makes the expert batch's pipelines and can still refuse (no room): the device goes at exit either way, after the tier's teardown */
if(!vkt_init(&vc,m->eusage)) return;
g_vkt.on=1; g_vkt.gu_fmt=f[0]; g_vkt.gu_gs=gs[0]; g_vkt.dn_fmt=f[2]; g_vkt.dn_gs=gs[2];
@@ -12116,6 +12203,121 @@ static double coli_ssd_probe_cached(const char *snap_dir){
#endif /* __APPLE__ */
#endif /* (COLI_METAL && __APPLE__) || COLI_SSD_PROBE_TEST */
+#ifdef COLI_VULKAN
+/* ---- the dense weights on the device only (COLI_VK_DENSE_HOST; docs/vulkan.md) ------
+ * With the dense part on the device (the chain, or COLI_VK_DENSE=1), every resident dense
+ * QT with a device form is uploaded right after the load, before the pins and the RAM
+ * cap, and its host copy is given back: resident_bytes drops by what went, so autopin and
+ * cap_for_ram give that RAM to the experts. The per-matrix path is forced on (the CPU has
+ * no copy): what the chain does not run (lm_head, the MTP head, eh_proj, a declined step)
+ * multiplies the same device copies, through matmul_qt_ex when no device call was there.
+ * What keeps its host copy:
+ * - the embedding (its rows are gathered on the CPU), the routers, norms and biases;
+ * - the MTP layer's kv_b: its attention runs on the CPU (the absorb core takes the main
+ * layers only);
+ * - when the chain will not run every forward (COLI_VK_CHAIN=2 or off, KV8, KV_TQ,
+ * PILOT, LOOKA, COLI_EXACT_VERIFY, a multiplexed serve with KV_SLOTS>1, CUDA on), every
+ * layer's kv_b and DSA indexer, which the CPU's attention reads row by row; with PILOT
+ * or LOOKA the shared experts too (the prefetcher and the predictor multiply them);
+ * - the formats with no device form (int2, E8/IQ3, fp8, planar int4).
+ * A lost device, or a read anywhere else (a parallel region, another thread), reads the QT
+ * back from the checkpoint first (qt_from_disk at the bits qt_load used: the same bytes),
+ * and the copy stays from there on. */
+static Model *g_dho_model;
+static pthread_mutex_t g_dho_mx=PTHREAD_MUTEX_INITIALIZER;
+static void qt_dho_reload(QT *t){
+ pthread_mutex_lock(&g_dho_mx);
+ if(__atomic_load_n(&t->vk_gone,__ATOMIC_ACQUIRE)){
+ if(!g_dho_model||!t->vk_name){
+ fprintf(stderr,"[VK] colibri: a dense matrix the device held alone cannot be read back\n"); exit(1);
+ }
+ QT n; memset(&n,0,sizeof n);
+ qt_from_disk(g_dho_model,t->vk_name,t->O,t->I,t->vk_bits,0,&n);
+ if(n.fmt!=t->fmt||n.gs!=t->gs||n.O!=t->O||n.I!=t->I){
+ fprintf(stderr,"[VK] colibri: %s came back from disk in another form (fmt %d, was %d)\n",t->vk_name,n.fmt,t->fmt);
+ exit(1);
+ }
+ t->qf=n.qf; t->q8=n.q8; t->q4=n.q4; t->s=n.s;
+ __atomic_store_n(&t->vk_gone,0,__ATOMIC_RELEASE);
+ coli_vk_dense_host_reloaded((size_t)qt_bytes(t));
+ }
+ pthread_mutex_unlock(&g_dho_mx);
+}
+/* The device form glm_chain.h's glmc_tensor gives a QT (same fmt and geometry, so the
+ * chain and the per-matrix path find this copy); 0 = none. */
+static int qt_dho_form(const QT *t){
+ if(!t->vk_name||t->mmap_view||t->O<=0||t->I<=0) return 0;
+ switch(t->fmt){
+ case 0: return t->qf!=NULL;
+ case 1: return t->q8!=NULL;
+ case 2: return !t->planar&&t->q4;
+ case 4: return !t->planar&&t->gs>=8&&t->gs%8==0&&t->q4;
+ case 5: return t->q4!=NULL;
+ default: return 0;
+ }
+}
+typedef struct { int keep_kvb, keep_shared, n; int64_t bytes; } DhoPass;
+static void qt_dho_visit(QT *t, int keep, int drop, DhoPass *p){
+ if(keep||!qt_dho_form(t)||t->vk_gone) return;
+ if(!drop){ p->n++; p->bytes+=qt_bytes(t); return; }
+ const void *w=t->fmt==0?(const void*)t->qf:t->fmt==1?(const void*)t->q8:(const void*)t->q4;
+ if(!coli_vk_tensor_ensure(&t->vk,w,t->fmt==0?NULL:t->s,qt_vk_fmt(t),t->I,t->O,t->fmt==4?t->gs:0)) return;
+ int64_t b=qt_bytes(t);
+ free(t->qf); free(t->q8); free(t->q4); free(t->s);
+ t->qf=NULL; t->q8=NULL; t->q4=NULL; t->s=NULL;
+ __atomic_store_n(&t->vk_gone,1,__ATOMIC_RELEASE);
+ coli_vk_dense_host_dropped((size_t)b);
+ p->n++; p->bytes+=b;
+}
+static void glm_dho_pass(Model *m, int drop, DhoPass *p){
+ Cfg *c=&m->c;
+ for(int i=0;i<=c->n_layers;i++){
+ if(i==c->n_layers&&!m->has_mtp) break;
+ Layer *l = i<c->n_layers ? &m->L[i] : &m->mtpL;
+ int mtp = i==c->n_layers;
+ qt_dho_visit(&l->q_a,0,drop,p); qt_dho_visit(&l->q_b,0,drop,p); qt_dho_visit(&l->kv_a,0,drop,p);
+ qt_dho_visit(&l->kv_b,mtp||p->keep_kvb,drop,p); qt_dho_visit(&l->o,0,drop,p);
+ if(l->sparse){ qt_dho_visit(&l->sh_gate,p->keep_shared,drop,p); qt_dho_visit(&l->sh_up,p->keep_shared,drop,p);
+ qt_dho_visit(&l->sh_down,p->keep_shared,drop,p); }
+ else { qt_dho_visit(&l->gate_proj,0,drop,p); qt_dho_visit(&l->up_proj,0,drop,p); qt_dho_visit(&l->down_proj,0,drop,p); }
+ if(!mtp&&m->ix_wq&&m->has_dsa&&c->idx_type[i]){
+ qt_dho_visit(&m->ix_wq[i],p->keep_kvb,drop,p); qt_dho_visit(&m->ix_wk[i],p->keep_kvb,drop,p);
+ qt_dho_visit(&m->ix_wp[i],p->keep_kvb,drop,p);
+ }
+ }
+ if(m->has_mtp) qt_dho_visit(&m->eh_proj,0,drop,p);
+ qt_dho_visit(&m->lm_head,0,drop,p);
+}
+/* After the load, before the pins and cap_for_ram: the decision (the chain's own, made
+ * silently here and printed by glmc_start), the uploads and the RAM given back. */
+static void glm_dho_start(Model *m){
+ if(!g_vulkan) return;
+ int tier_on = vkt_wanted() && g_vk_experts!=0 && m->c.n_experts>0;
+ int chain = coli_vk_chain_decide(NULL, tier_on, COLI_VK_CHAIN_UNMEASURED), cuda = 0;
+#ifdef COLI_CUDA
+ cuda = g_cuda_enabled;
+#endif
+ if(cuda || g_kv8 || g_tq || g_pilot) chain = COLI_VK_CHAIN_OFF; /* glmc_start keeps it off */
+ int slots = getenv("KV_SLOTS") ? atoi(getenv("KV_SLOTS")) : 1;
+ DhoPass p = {0};
+ const char *xv = getenv("COLI_EXACT_VERIFY"); /* exact_verify_on() prints its line later, where it always did */
+ p.keep_kvb = chain!=COLI_VK_CHAIN_ON || g_looka || g_pilot || (xv && atoi(xv)) || slots>1;
+ p.keep_shared = g_pilot || g_looka;
+ glm_dho_pass(m, 0, &p);
+ if(!coli_vk_dense_host_decide("colibri", !cuda && (chain!=COLI_VK_CHAIN_OFF || g_vk_dense), (size_t)p.bytes)) return;
+ g_dho_model = m; g_dho_thread = pthread_self();
+ g_vk_dense = 1; /* the steps the chain does not run take the device too: the CPU has no copy */
+ p.n = 0; p.bytes = 0;
+ glm_dho_pass(m, 1, &p);
+ m->resident_bytes -= p.bytes; /* what cap_for_ram and autopin count in RAM */
+ char kept[256];
+ snprintf(kept, sizeof kept, "the embedding, routers and norms, the MTP layer's kv_b%s%s",
+ p.keep_kvb ? ", every kv_b and DSA indexer (the chain will not run every forward: the CPU's attention reads them)" : "",
+ p.keep_shared ? ", the shared experts (PILOT/LOOKA multiply them on the host)" : "");
+ coli_vk_dense_host_placed("colibri", kept);
+}
+#endif
+
/* Expert-cache-cap precedence (#379, S2): explicit CLI positional > explicit
* CAP env > platform default (Metal + darwin + fast SSD) > historic default.
* `cli_given` distinguishes a bare invocation (no positional at all -> the
@@ -12366,6 +12568,8 @@ int main(int argc, char **argv){
g_mlock = getenv("MLOCK")?atoi(getenv("MLOCK")):-1; /* -1 auto (ON macOS), 0 off, 1 force / auto (ON macOS), 0 off, 1 force */
g_spec = getenv("SPEC")?atoi(getenv("SPEC")):1;
g_draft = getenv("DRAFT")?atoi(getenv("DRAFT")):-1;
+ g_lookup = getenv("COLI_LOOKUP") && getenv("COLI_LOOKUP")[0]=='1';
+ spec_gate_init(&g_lookup_gate, getenv("COLI_SPEC_GATE") && getenv("COLI_SPEC_GATE")[0]=='0');
g_no_fused_pair = getenv("COLI_NO_FUSED_PAIR")?atoi(getenv("COLI_NO_FUSED_PAIR")):0; /* -1 = auto: 3 se MTP, 0 senza */
g_looka = getenv("LOOKA")?atoi(getenv("LOOKA")):0; /* 1 = misura predicibilita' routing */
g_pilot = getenv("PILOT")?atoi(getenv("PILOT")):0; /* 1 = prefetch pilotato dal router */
@@ -12837,8 +13041,12 @@ int main(int argc, char **argv){
#else
g_draft = m.has_mtp ? 1 : 0;
#endif
+ if(!m.has_mtp && g_lookup) g_draft = 5; /* COLI_LOOKUP=1: up to 5 lookup drafts, the gate picks */
}
if(getenv("DSA_TOPK")) m.c.index_topk=atoi(getenv("DSA_TOPK")); /* override per test */
+#ifdef COLI_VULKAN
+ glm_dho_start(&m); /* COLI_VK_DENSE_HOST: the trunk on the device only, before the pins and the RAM cap */
+#endif
/* Il path MUX (SERVE_BATCH=1, cioe' `coli serve`) forza g_draft=0 sotto —
* la speculazione non e' ragged-safe nel batch multi-slot. Segnalarlo QUI,
* altrimenti "MTP active (draft=8)" mentirebbe: il messaggio e' stampato
@@ -13108,6 +13316,16 @@ int main(int argc, char **argv){
double tot=m.hits+m.miss;
printf("N-gram speculation (DRAFT=%d): %.2f tokens/forward (%llu forwards per %llu tokens)\n",
g_draft, m.n_fw?(double)m.n_emit/m.n_fw:1.0, (unsigned long long)m.n_fw, (unsigned long long)m.n_emit);
+ if(g_lookup && !m.has_mtp){
+ unsigned long long lp=0, lh=0;
+ for(int j=0;j<SPEC_MAX_DRAFTS;j++){ lp+=g_lookup_gate.prop[SPEC_SRC_LOOKUP][j]; lh+=g_lookup_gate.hit[SPEC_SRC_LOOKUP][j]; }
+ char gd[384]; spec_gate_describe(&g_lookup_gate,SPEC_SRC_LOOKUP,gd,sizeof gd);
+ fprintf(stderr,"[colibri lookup] acceptance %.1f%% (%llu/%llu judged drafts in %llu verifies) | gate %s, "
+ "%llu declined, %llu probes | %s\n", lp?100.0*lh/lp:0.0, lh, lp,
+ (unsigned long long)g_lookup_gate.verifies[SPEC_SRC_LOOKUP], g_lookup_gate.off?"off":"on",
+ (unsigned long long)g_lookup_gate.declined[SPEC_SRC_LOOKUP],
+ (unsigned long long)g_lookup_gate.probes[SPEC_SRC_LOOKUP], gd);
+ }
char vkhits[48]=""; /* experts a Vulkan device served (#336's split) */
if(m.hit_vk) snprintf(vkhits,sizeof vkhits," + %llu vk",(unsigned long long)m.hit_vk);
printf("Expert cache hit rate: %.1f%% (%llu pin + %llu lru%s / %llu miss) | RSS: %.2f GB | %.1f tok/s\n",
diff --git a/c/decide_vk.h b/c/decide_vk.h
new file mode 100644
index 00000000..be0cb2b8
--- /dev/null
+++ b/c/decide_vk.h
@@ -0,0 +1,207 @@
+/* decide_vk.h -- the decision engines' encoders on a Vulkan device (COLI_VULKAN=1):
+ * laya.c (ModernBERT and its decision head) and gliner_decide.c (DeBERTa-v3).
+ * docs/vulkan.md, "The decision engines".
+ *
+ * A decision is one forward over a prompt-sized batch of rows and no generation, so it
+ * is matrix products end to end. With COLI_VULKAN=1 the device opens once the weights
+ * are loaded, and a forward runs one of two ways:
+ * - on the device (COLI_VK_CHAIN, dvk_decide_chain below): the engine records the
+ * whole encoder (and Laya's head layers) as frames of the dense chain (vk_chain.h:
+ * vkc_matmul and the chain_enc.comp ops), the activations staying on the device from
+ * the embedding rows to the rows the scorer reads; one submission per forward;
+ * - matrix by matrix (COLI_VK_CHAIN=0): every dense matrix through coli_vk_matmul (the
+ * tiled GEMM from the backend's threshold), the norms, the attention and the rest on
+ * the CPU (COLI_VK_DENSE=0 keeps the matrices there too).
+ * Both read the same resident copy of a matrix, uploaded during setup as f32 (fmt 10)
+ * or exact f16 values (fmt 14), through the backend's staged path where
+ * the card needs it. Activations stay f32 on both, so a forward differs from the CPU's
+ * only in the order of the sums: the decisions are the CPU's, the probabilities within
+ * rounding (the tests hold them to the CPU's; tests/vulkan_engines.sh decide).
+ * A device that fails mid-forward (lost) leaves the forward to the CPU, which recomputes
+ * it from the ids: a decision has no state to recover.
+ *
+ * Without COLI_VULKAN in the build, or unset at run time, none of this exists or runs. */
+#ifndef COLI_DECIDE_VK_H
+#define COLI_DECIDE_VK_H
+#ifdef COLI_VULKAN
+#include <stdint.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include "backend_vulkan.h"
+#include "vk_chain.h"
+
+typedef struct { const void *w; ColiVkTensor *t; int off, fmt; } DvkSlot;
+#define DVK_SLOTS 2048
+static struct {
+ int ready; /* the device is open */
+ int dense; /* matrices one by one through coli_vk_matmul (COLI_VK_CHAIN=0) */
+ int chain; /* the whole forward on the device */
+ int f16; /* the checkpoint stores f16: matrices go up as f16 where every value is one */
+ const char *engine;
+ unsigned n_f16, n_f32;
+ DvkSlot slot[DVK_SLOTS];
+ unsigned long long fwd_dev, fwd_cpu, rows_dev;
+ double dev_ms;
+} g_dvk;
+
+static double dvk_now_ms(void)
+{
+ struct timespec t;
+ clock_gettime(CLOCK_MONOTONIC, &t);
+ return t.tv_sec * 1e3 + t.tv_nsec / 1e6;
+}
+
+/* COLI_VK_CHAIN set: 0 matrix by matrix, any other number the forward on the device.
+ * Unset: on a GPU, discrete or integrated (measured on a Radeon 780M: docs/vulkan.md);
+ * off on a CPU device (Lavapipe), where the device is the CPU. */
+static int dvk_decide_chain(const char *engine)
+{
+ const char *e = getenv("COLI_VK_CHAIN");
+ char why[160];
+ int on;
+ if (e && *e) { on = atoi(e) != 0; snprintf(why, sizeof why, "COLI_VK_CHAIN=%s", e); }
+ else if (coli_vk_device_shares_ram() && !coli_vk_device_integrated()) {
+ on = 0; snprintf(why, sizeof why, "a CPU device; COLI_VK_CHAIN=1 turns it on");
+ } else {
+ on = 1;
+ snprintf(why, sizeof why, "%s; COLI_VK_CHAIN=0 matrix by matrix",
+ coli_vk_device_integrated() ? "an integrated GPU: measured faster" : "a discrete GPU");
+ }
+ fprintf(stderr, "[VK] %s: forward on the device %s (%s)\n", engine, on ? "on" : "off", why);
+ return on;
+}
+
+static void dvk_init(const char *engine)
+{
+ memset(&g_dvk, 0, sizeof g_dvk);
+ g_dvk.engine = engine;
+ if (!coli_vk_init_env(engine)) return; /* COLI_VULKAN unset, or no device: the CPU */
+ g_dvk.ready = 1;
+ atexit(coli_vk_shutdown);
+ g_dvk.chain = dvk_decide_chain(engine);
+ if (g_dvk.chain && !(vkc_init() && vkc_enc_ready())) {
+ fprintf(stderr, "[VK] %s: the chain's shaders are missing (chain_enc.spv): matrix by matrix\n", engine);
+ g_dvk.chain = 0;
+ }
+ if (g_dvk.chain) atexit(vkc_shutdown); /* registered after the backend's: runs before it */
+ g_dvk.dense = !g_dvk.chain && coli_vk_dense();
+}
+
+/* f32 -> f16 bits when the value is exactly an f16 (normal, subnormal or zero), else -1 */
+static int dvk_f16_exact(float f)
+{
+ uint32_t x;
+ memcpy(&x, &f, 4);
+ uint32_t sign = (x >> 16) & 0x8000u, mant = x & 0x7fffffu;
+ int e = (int)((x >> 23) & 0xff) - 127;
+ if ((x & 0x7fffffffu) == 0) return (int)sign;
+ if (e > 15 || e < -24 || ((x >> 23) & 0xff) == 0xff) return -1;
+ if (e >= -14) { /* normal: 10 mantissa bits */
+ if (mant & 0x1fffu) return -1;
+ return (int)(sign | (uint32_t)((e + 15) << 10) | (mant >> 13));
+ }
+ int shift = -14 - e; /* subnormal: 1.m >> shift, 10 bits */
+ uint32_t m = mant | 0x800000u;
+ if (m & ((1u << (13 + shift)) - 1)) return -1;
+ return (int)(sign | (m >> (13 + shift)));
+}
+
+/* The device copy of W, uploaded on first use: f16 (fmt 14) when the engine says its
+ * checkpoint is f16 and every value of W is exactly one (the same values, half the
+ * bytes the device reads), else the f32 rows (fmt 10). NULL when it cannot be (that
+ * matrix then stays on the CPU). */
+static ColiVkTensor *dvk_tensor(const QiMat *W)
+{
+ if (!g_dvk.ready || W->fmt != QI_F32 || W->ld || !W->w) return NULL;
+ uintptr_t k = (uintptr_t)W->w;
+ unsigned h = (unsigned)((k >> 4) * 2654435761u) % DVK_SLOTS;
+ for (int n = 0; n < DVK_SLOTS; n++, h = (h + 1) % DVK_SLOTS) {
+ DvkSlot *s = &g_dvk.slot[h];
+ if (s->w == W->w) return s->off ? NULL : s->t;
+ if (s->w) continue;
+ s->w = W->w;
+ const float *f = (const float *)W->w;
+ size_t n_ = (size_t)W->N * W->K;
+ uint16_t *hb = g_dvk.f16 && W->K % 2 == 0 ? (uint16_t *)malloc(n_ * sizeof(uint16_t)) : NULL;
+ int exact = hb != NULL;
+ for (size_t i = 0; exact && i < n_; i++) {
+ int b = dvk_f16_exact(f[i]);
+ if (b < 0) exact = 0; else hb[i] = (uint16_t)b;
+ }
+ s->fmt = exact ? 14 : 10;
+ int ok = coli_vk_tensor_ensure(&s->t, exact ? (const void *)hb : W->w, NULL, s->fmt, W->K, W->N, 0);
+ free(hb);
+ if (!ok) { s->off = 1; return NULL; }
+ if (exact) g_dvk.n_f16++; else g_dvk.n_f32++;
+ return s->t;
+ }
+ return NULL;
+}
+
+/* Y = X W^T + bias through the device, matrix by matrix; 0 = the caller runs it. */
+static int dvk_gemm(float *Y, const float *X, int M, const QiMat *W, const float *bias)
+{
+ if (!g_dvk.dense || M < 1 || M > 65535) return 0;
+#ifdef _OPENMP
+ if (omp_in_parallel()) return 0;
+#endif
+ ColiVkTensor *t = dvk_tensor(W);
+ ColiVkTensorInfo ti;
+ if (!t || !coli_vk_tensor_info(t, &ti) || !coli_vk_matmul(&t, Y, X, W->w, NULL, ti.fmt, M, W->K, W->N, 0)) return 0;
+ if (bias) {
+ const int N = W->N;
+ #pragma omp parallel for schedule(static)
+ for (int r = 0; r < M; r++)
+ for (int j = 0; j < N; j++) Y[(int64_t)r * N + j] += bias[j];
+ }
+ return 1;
+}
+
+/* The norms', biases' and tables' floats, packed for one upload. */
+typedef struct { float *v; size_t n, cap; int failed; } DvkPrm;
+static int dvk_prm_add(DvkPrm *P, const float *v, size_t n)
+{
+ if (!v) return -1;
+ if (P->n + n > P->cap) {
+ size_t cap = P->cap ? P->cap : 4096;
+ while (cap < P->n + n) cap *= 2;
+ float *g = (float *)realloc(P->v, cap * sizeof(float));
+ if (!g) { P->failed = 1; return -1; }
+ P->v = g; P->cap = cap;
+ }
+ if (P->n + n > (size_t)INT32_MAX) { P->failed = 1; return -1; }
+ memcpy(P->v + P->n, v, n * sizeof(float));
+ P->n += n;
+ return (int)(P->n - n);
+}
+/* a device buffer holding P's floats (one upload, in a frame of its own) */
+static VkcBuf *dvk_prm_upload(DvkPrm *P)
+{
+ if (P->failed || !P->n) return NULL;
+ VkcBuf *b = vkc_buf(P->n * sizeof(float), VKC_DEV);
+ if (!b) return NULL;
+ if (!(vkc_begin() && vkc_write(b, 0, P->v, P->n * sizeof(float)) && vkc_submit(1))) { vkc_free(b); return NULL; }
+ return b;
+}
+
+/* At the end of a request: the line the tests read (N > 0 shows the device really ran). */
+static void dvk_report(void)
+{
+ if (!g_dvk.ready) return;
+ VkcStats st;
+ memset(&st, 0, sizeof st);
+ if (g_dvk.chain) vkc_stats(&st);
+ size_t bytes = 0, tensors = 0;
+ coli_vk_mem_info(&bytes, &tensors);
+ fprintf(stderr, "[VK] %s: %llu matmuls on the GPU (%zu matrices resident, %.1f MiB)\n", g_dvk.engine,
+ coli_vk_matmul_calls() + st.matmuls, tensors, bytes / 1048576.0);
+ if (g_dvk.chain)
+ fprintf(stderr, "[VK] %s chain: %llu forwards on the device (%llu rows, %.1f ms), %llu on the CPU, "
+ "%llu frames (%llu ops, %llu matmuls, %llu tiled GEMM), %.1f ms waiting, %.1f MiB of chain buffers\n",
+ g_dvk.engine, g_dvk.fwd_dev, g_dvk.rows_dev, g_dvk.dev_ms, g_dvk.fwd_cpu,
+ st.frames, st.ops, st.matmuls, st.gemms, st.wait_ms, st.dev_bytes / 1048576.0);
+ if (g_dvk.chain) vkc_prof_print();
+}
+#endif
+#endif
diff --git a/c/deepseek_v4.c b/c/deepseek_v4.c
index 60484576..5725f0b4 100644
--- a/c/deepseek_v4.c
+++ b/c/deepseek_v4.c
@@ -602,6 +602,12 @@ int coli_st_prefetch_at_rep(const ColiSafetensorsIndex *index, int shard,
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
+#ifdef _WIN32
+#include <windows.h>
+#else
+#include <sys/mman.h>
+#include <unistd.h>
+#endif
static int set_error(char *error, size_t size, const char *format, ...) {
if (error && size) {
@@ -776,11 +782,69 @@ int coli_v4_layer_validate(const ColiDeepSeekV4LayerPlan *plan,
return 0;
}
+/* Dense weights on the Vulkan device only: a droppable tensor of a resident layer is
+ * read into an anonymous mapping of its own, so that its pages can go back to the system
+ * while its address stays reserved (the device copies are found by it, and no later
+ * allocation can take it), and come back at the same address if the CPU needs it. */
+static int g_v4_layer_map; /* the reference loader maps droppable tensors (resident, device only) */
+static size_t v4_map_len(size_t bytes) {
+#ifdef _WIN32
+ SYSTEM_INFO si; GetSystemInfo(&si);
+ size_t page = si.dwAllocationGranularity ? si.dwAllocationGranularity : 65536;
+#else
+ long ps = sysconf(_SC_PAGESIZE);
+ size_t page = ps > 0 ? (size_t)ps : 4096;
+#endif
+ return (bytes + page - 1) / page * page;
+}
+static void *v4_map_alloc(size_t bytes) {
+ size_t len = v4_map_len(bytes ? bytes : 1);
+#ifdef _WIN32
+ return VirtualAlloc(NULL, len, MEM_RESERVE | MEM_COMMIT, PAGE_READWRITE);
+#else
+ void *p = mmap(NULL, len, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
+ return p == MAP_FAILED ? NULL : p;
+#endif
+}
+static int v4_map_drop(void *data, size_t bytes) { /* pages back, the range reserved and unreadable */
+ size_t len = v4_map_len(bytes ? bytes : 1);
+#ifdef _WIN32
+ return VirtualFree(data, len, MEM_DECOMMIT) ? 0 : -1;
+#else
+ int flags = MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED;
+#ifdef MAP_NORESERVE
+ flags |= MAP_NORESERVE;
+#endif
+ return mmap(data, len, PROT_NONE, flags, -1, 0) == MAP_FAILED ? -1 : 0;
+#endif
+}
+static int v4_map_restore(void *data, size_t bytes) { /* zeroed, writable pages at the same address */
+ size_t len = v4_map_len(bytes ? bytes : 1);
+#ifdef _WIN32
+ return VirtualAlloc(data, len, MEM_COMMIT, PAGE_READWRITE) == data ? 0 : -1;
+#else
+ return mprotect(data, len, PROT_READ | PROT_WRITE);
+#endif
+}
+static void v4_map_free(void *data, size_t bytes) {
+ if (!data) return;
+#ifdef _WIN32
+ (void)bytes;
+ VirtualFree(data, 0, MEM_RELEASE);
+#else
+ munmap(data, v4_map_len(bytes ? bytes : 1));
+#endif
+}
+
void coli_v4_layer_free(ColiV4Engine *engine,
ColiDeepSeekV4LayerWeights *weights) {
(void)engine;
if (!weights) return;
- for (size_t i = 0; i < weights->plan.tensor_count; i++) free(weights->data[i]);
+ for (size_t i = 0; i < weights->plan.tensor_count; i++) {
+ if (weights->mapped[i])
+ v4_map_free(weights->data[i], (size_t)coli_v4_dense_tensor_bytes(&weights->plan.tensors[i]));
+ else free(weights->data[i]);
+ }
memset(weights, 0, sizeof(*weights));
}
@@ -858,8 +922,11 @@ int coli_v4_layer_load(ColiV4Engine *engine,
const ColiSafetensorsTensor *tensor = coli_st_find(index, spec->name);
size_t resident_bytes = tensor->dtype == COLI_ST_F8_E8M0
? (size_t)tensor->numel * sizeof(float) : (size_t)tensor->nbytes;
- weights->data[i] = malloc(resident_bytes);
+ weights->mapped[i] = g_v4_layer_map && coli_v4_dense_device_only_tensor(spec) &&
+ resident_bytes == coli_v4_dense_tensor_bytes(spec);
+ weights->data[i] = weights->mapped[i] ? v4_map_alloc(resident_bytes) : malloc(resident_bytes);
if (!weights->data[i]) {
+ weights->mapped[i] = 0;
coli_v4_layer_free(NULL, weights);
return set_error(error, error_size, "out of memory loading: %s", spec->name);
}
@@ -952,6 +1019,52 @@ static int resident_enabled_v2(ColiV4Engine *engine) {
return engine && engine->runtime.dense_resident;
}
+/* Dense weights on the Vulkan device only (deepseek_v4_internal.h): NULL in every link
+ * but the engine binary built with VK=1. */
+int (*coli_v4_dense_place)(ColiV4Engine *engine, int layer);
+int (*coli_v4_dense_device_lost)(void);
+void (*coli_v4_dense_reread)(uint64_t bytes);
+
+int coli_v4_layer_host_drop(ColiV4Engine *engine, int layer, size_t i) {
+ if (!engine || layer < 0 || layer >= COLI_V4_RESIDENT_MAX_LAYERS_V2 ||
+ !engine->dense_resident.ready[layer]) return -1;
+ ColiDeepSeekV4LayerWeights *w = &engine->dense_resident.layers[layer];
+ if (i >= w->plan.tensor_count || !w->mapped[i]) return -1;
+ if (w->host_gone[i]) return 0;
+ uint64_t bytes = coli_v4_dense_tensor_bytes(&w->plan.tensors[i]);
+ if (v4_map_drop(w->data[i], (size_t)bytes)) return -1;
+ w->host_gone[i] = 1;
+ engine->dense_resident.dropped_bytes += bytes;
+ return 0;
+}
+
+int coli_v4_layer_host_restore(ColiV4Engine *engine, const void *data, uint64_t *bytes_out) {
+ if (!engine || !data || !engine->dense_resident.device_only) return 0;
+ const ColiSafetensorsIndex *index = engine->dense_resident.index;
+ for (int layer = 0; layer < COLI_V4_RESIDENT_MAX_LAYERS_V2; layer++) {
+ if (!engine->dense_resident.ready[layer]) continue;
+ ColiDeepSeekV4LayerWeights *w = &engine->dense_resident.layers[layer];
+ for (size_t i = 0; i < w->plan.tensor_count; i++) {
+ if (!w->host_gone[i]) continue;
+ const ColiDeepSeekV4TensorSpec *spec = &w->plan.tensors[i];
+ uint64_t bytes = coli_v4_dense_tensor_bytes(spec);
+ uintptr_t p = (uintptr_t)data, b = (uintptr_t)w->data[i];
+ if (p < b || p - b >= bytes) continue;
+ /* back at the same address, read and laid out exactly as the first load did */
+ const ColiSafetensorsTensor *tensor = index ? coli_st_find(index, spec->name) : NULL;
+ if (!tensor || v4_map_restore(w->data[i], (size_t)bytes) ||
+ coli_st_read_tensor(index, tensor, w->data[i]) != 0) return -1;
+ if (spec->dtype == COLI_ST_F8_E4M3 && spec->packed_rows8 &&
+ v4_fp8_pack_rows8_inplace(w->data[i], spec->shape[0], spec->shape[1]) <= 0) return -1;
+ w->host_gone[i] = 0;
+ engine->dense_resident.dropped_bytes -= bytes;
+ if (bytes_out) *bytes_out = bytes;
+ return 1;
+ }
+ }
+ return 0;
+}
+
int coli_v4_layer_load(ColiV4Engine *engine,
ColiDeepSeekV4LayerWeights *weights,
const ColiDeepSeekV4Config *config,
@@ -965,6 +1078,17 @@ int coli_v4_layer_load(ColiV4Engine *engine,
if (!resident_enabled_v2(engine))
return coli_v4_layer_resident_reference_load(
NULL, weights, effective_config, index, layer, error, error_size);
+ if (engine->dense_resident.device_only && coli_v4_dense_device_lost &&
+ coli_v4_dense_device_lost()) {
+ /* the device that held the dense layers is gone: the CPU reads them per forward,
+ * as a low-memory plan does (its two-layer reserve is in the plan) */
+ if (coli_v4_layer_resident_reference_load(
+ NULL, weights, effective_config, index, layer, error, error_size)) return -1;
+ for (size_t i = 0; coli_v4_dense_reread && i < weights->plan.tensor_count; i++)
+ if (coli_v4_dense_device_only_tensor(&weights->plan.tensors[i]))
+ coli_v4_dense_reread(coli_v4_dense_tensor_bytes(&weights->plan.tensors[i]));
+ return 0;
+ }
if (engine->dense_resident.index && engine->dense_resident.index != index) {
if (error && error_size)
snprintf(error, error_size,
@@ -973,9 +1097,12 @@ int coli_v4_layer_load(ColiV4Engine *engine,
}
engine->dense_resident.index = index;
if (!engine->dense_resident.ready[layer]) {
- if (coli_v4_layer_resident_reference_load(
+ g_v4_layer_map = engine->dense_resident.device_only; /* droppable tensors in mappings of their own */
+ int failed = coli_v4_layer_resident_reference_load(
NULL, &engine->dense_resident.layers[layer], effective_config, index,
- layer, error, error_size)) return -1;
+ layer, error, error_size);
+ g_v4_layer_map = 0;
+ if (failed) return -1;
#ifdef COLI_V4_GPU_TIER
if (coli_v4_gpu_layer_upload(engine, layer,
&engine->dense_resident.layers[layer]) < 0)
@@ -985,6 +1112,10 @@ int coli_v4_layer_load(ColiV4Engine *engine,
engine->dense_resident.ready[layer] = 1;
engine->dense_resident.total_bytes +=
engine->dense_resident.layers[layer].stats.total_bytes;
+ /* device only: the layer's matrices to the device and their host pages back,
+ * before the next layer is read (one layer in RAM at a time) */
+ if (engine->dense_resident.device_only && coli_v4_dense_place)
+ coli_v4_dense_place(engine, layer);
if (layer == effective_config->num_hidden_layers - 1)
fprintf(stderr, "v4_dense_resident layers=%d bytes=%.3fGiB\n",
effective_config->num_hidden_layers,
@@ -1410,6 +1541,23 @@ static int build_runtime_plan(ColiV4Engine *engine,
}
+/* Dense weights on the Vulkan device only (deepseek_v4_internal.h): set by the engine
+ * binary built with VK=1, NULL in every other link. */
+ColiV4DenseDeviceDecide coli_v4_dense_device_decide;
+
+/* The bytes of the dense layers a device may hold alone (coli_v4_dense_device_only_tensor). */
+static uint64_t v4_dense_device_bytes(const ColiV4Engine *engine) {
+ uint64_t total = 0;
+ for (int layer = 0; layer < engine->config.num_hidden_layers; layer++) {
+ ColiDeepSeekV4LayerPlan plan;
+ if (coli_v4_layer_plan(&plan, &engine->config, layer, NULL, 0)) return 0;
+ for (size_t i = 0; i < plan.tensor_count; i++)
+ if (coli_v4_dense_device_only_tensor(&plan.tensors[i]))
+ total += coli_v4_dense_tensor_bytes(&plan.tensors[i]);
+ }
+ return total;
+}
+
int coli_v4_expert_store_open_planned(
ColiV4Engine *engine,
const ColiDeepSeekV4Config *config,
@@ -1431,6 +1579,18 @@ int coli_v4_expert_store_open_planned(
double open_plan_s = v4_open_now_s() - open_plan_t0;
uint64_t fixed = plan.system_reserve_bytes + plan.runtime_reserve_bytes;
+ /* Dense weights on a Vulkan device only: its matrices count as not in RAM, only
+ * what stays on the host (norms, scales, the router...) does, and the RAM they
+ * would have taken goes to the experts' cache. */
+ uint64_t device_bytes = 0;
+ if (coli_v4_dense_device_decide && dense_bytes) {
+ uint64_t candidate = v4_dense_device_bytes(engine);
+ if (candidate && candidate < dense_bytes &&
+ coli_v4_dense_device_decide(engine, candidate, dense_bytes - candidate)) {
+ device_bytes = candidate;
+ dense_bytes -= candidate;
+ }
+ }
ColiDeepSeekV4ResidentTierPlan tiers;
ColiDeepSeekV4ResidentTierInputs tier_inputs = {
plan.planner_available_bytes, fixed, dense_bytes,
@@ -1440,6 +1600,14 @@ int coli_v4_expert_store_open_planned(
error, error_size)) return -1;
dense_bytes = tiers.dense_bytes;
runtime->dense_resident = tiers.dense_resident;
+ if (device_bytes && !tiers.dense_resident) {
+ fprintf(stderr, "ram_tiers: even the dense layers' host part (%.2f GiB) does not fit: "
+ "they are read per forward and stay off the device\n",
+ tier_inputs.dense_bytes / (double)GIB);
+ device_bytes = 0;
+ }
+ engine->dense_resident.device_only = device_bytes != 0;
+ engine->dense_resident.device_bytes = device_bytes;
if (dense_bytes > plan.planner_available_bytes - fixed) {
snprintf(error, error_size, "resident V4 tiers exceed available RAM");
return -1;
@@ -1469,6 +1637,17 @@ int coli_v4_expert_store_open_planned(
plan.expert_cache_bytes + (resident_head ? head_bytes : 0);
if (resident_head && coli_v4_head_cache_load(
engine, engine->target_index, error, error_size)) return -1;
+ if (device_bytes)
+ fprintf(stderr,
+ "ram_tiers available=%.2fGiB dense=device(%.2fGiB on the device, %.2fGiB in RAM) "
+ "target_slots=%d target_cache=%.2fGiB head=%s projected=%.2fGiB\n",
+ plan.planner_available_bytes / (double)GIB,
+ device_bytes / (double)GIB, dense_bytes / (double)GIB,
+ slots,
+ plan.expert_cache_bytes / (double)GIB,
+ resident_head ? "resident-bf16" : "streamed-bf16",
+ plan.projected_bytes / (double)GIB);
+ else
fprintf(stderr,
"ram_tiers available=%.2fGiB dense=%s(%.2fGiB) "
"target_slots=%d target_cache=%.2fGiB head=%s projected=%.2fGiB\n",
@@ -3364,6 +3543,8 @@ int coli_v4_compressor_step(ColiDeepSeekV4CompressorState *state,
return compressor_pool_and_emit(state, output, produced, position,
error, error_size);
}
+ coli_v4_vk_host_ensure(wkv); /* a device-only matrix comes back for the CPU */
+ coli_v4_vk_host_ensure(wgate);
#endif
#pragma omp parallel for
for (int row = 0; row < projection; row++) {
@@ -6322,7 +6503,9 @@ int coli_v4_moe_routed(float *routed, const ColiDeepSeekV4LayerWeights *weights,
const ColiDeepSeekV4Config *config, ColiExpertStore *store,
const float *inputs, const int *tokens, int rows,
char *error, size_t error_size) {
- if (!routed || !weights || !config || !store || !inputs || !tokens || rows < 1 || rows > 128)
+ /* a chunk of up to 128 rows as the CPU's prefill block takes, or with a chunk from the
+ * budget (vkc_chunk_auto) up to the chain's (the union path takes any batch) */
+ if (!routed || !weights || !config || !store || !inputs || !tokens || rows < 1 || rows > 65535)
return set_error(error, error_size, "invalid chained MoE arguments");
int d = config->hidden_size, result = 0;
v4_moe_routed_only = 1;
@@ -6657,6 +6840,8 @@ int coli_v4_compressor_step(ColiDeepSeekV4CompressorState *state,
return compressor_pool_and_emit(state, output, produced, position,
error, error_size);
}
+ coli_v4_vk_host_ensure(wkv); /* a device-only matrix comes back for the CPU */
+ coli_v4_vk_host_ensure(wgate);
#endif
#pragma omp parallel for
for (int row = 0; row < projection; row++) {
@@ -12465,6 +12650,7 @@ typedef struct {
const void *data;
int fmt, rows, columns, refused;
ColiVkTensor *tensor;
+ int dho; /* placed by v4_dho_place: the device holds the matrix alone */
} V4VkEntry;
static V4VkEntry *g_v4_vk_map;
static size_t g_v4_vk_cap, g_v4_vk_used;
@@ -12501,7 +12687,7 @@ static V4VkEntry *v4_vk_insert(const void *data, int fmt, int rows, int columns)
}
size_t at = v4_vk_hash(data, g_v4_vk_cap);
while (g_v4_vk_map[at].data) at = (at + 1) & (g_v4_vk_cap - 1);
- g_v4_vk_map[at] = (V4VkEntry){data, fmt, rows, columns, 0, NULL};
+ g_v4_vk_map[at] = (V4VkEntry){data, fmt, rows, columns, 0, NULL, 0};
g_v4_vk_used++;
return &g_v4_vk_map[at];
}
@@ -12552,12 +12738,16 @@ static int v4_vk_matmul_impl(int fmt, const void *data, const float *scales,
if (fmt != 11 && fmt != 12) return -1;
if (fmt == 12 && (!scales || columns % 128 || (rows8 && rows % 8))) return -1;
V4VkEntry *e = v4_vk_find(data, fmt, rows, columns);
+ /* With the dense weights on the device only the hook is on whatever COLI_VK_DENSE
+ * says, for the matrices the device holds alone; the rest stay where it puts them. */
+ if (!coli_vk_dense() && !(e && e->dho)) return -1;
if (e && e->refused) return -1;
if (e && e->tensor)
return coli_vk_matmul(&e->tensor, output, input, NULL, NULL, fmt, batch,
columns, rows, fmt == 12 ? 128 : 0) ? 0 : -1;
size_t bytes = (size_t)rows * columns * (fmt == 11 ? 2 : 1);
if (!v4_vk_resident(data, bytes)) return -1;
+ coli_v4_vk_host_ensure(data); /* a view the placement did not make: its rows back first */
if (!e && !(e = v4_vk_insert(data, fmt, rows, columns))) return -1;
/* First use: the device copy is made from a row-major matrix and one f32
* scale per 128 inputs of every row, the layout fmt 12 reads. */
@@ -12666,6 +12856,48 @@ static void v4_vkt_note(const ColiExpertView *view) {
free(buffer);
}
+/* The tier's streaming (a big prompt chunk's cold experts on the device): an expert as
+ * the store leases it (its RAM cache, else the disk), a pinned rows16 one unpacked to
+ * rows as v4_vkt_note does, held until the tier has copied it. */
+typedef struct { ColiExpertView view; unsigned char *buffer; } V4VkHold;
+static int v4_vkt_load(void *ctx, int layer, int expert, VktExpertSrc *src, void **h) {
+ ColiExpertStore *store = ctx;
+ if (!v4_vkt_thread()) return 0;
+ V4VkHold *hold = calloc(1, sizeof *hold);
+ if (!hold) return 0;
+ if (coli_expert_lookup(store, (ColiExpertKey){layer, expert}, &hold->view)) { free(hold); return 0; }
+ const ColiTensorView *m[3] = {&hold->view.gate, &hold->view.up, &hold->view.down};
+ if (m[0]->block_rows != 16 && m[1]->block_rows != 16 && m[2]->block_rows != 16) {
+ *src = (VktExpertSrc){m[0]->data, m[1]->data, m[2]->data, m[0]->scales, m[1]->scales, m[2]->scales};
+ *h = hold;
+ return 1;
+ }
+ size_t total = 0;
+ for (int k = 0; k < 3; k++) total += m[k]->data_bytes + m[k]->scale_bytes;
+ unsigned char *at = hold->buffer = malloc(total), *data[3], *scales[3];
+ if (!at) { coli_expert_release(store, &hold->view); free(hold); return 0; }
+ for (int k = 0; k < 3; k++) {
+ size_t rows = (size_t)m[k]->rows;
+ data[k] = at; at += m[k]->data_bytes;
+ scales[k] = at; at += m[k]->scale_bytes;
+ if (m[k]->block_rows == 16) {
+ v4_vkt_unpack16(data[k], m[k]->data, rows, m[k]->data_bytes / rows);
+ v4_vkt_unpack16(scales[k], m[k]->scales, rows, m[k]->scale_bytes / rows);
+ } else {
+ memcpy(data[k], m[k]->data, m[k]->data_bytes);
+ memcpy(scales[k], m[k]->scales, m[k]->scale_bytes);
+ }
+ }
+ *src = (VktExpertSrc){data[0], data[1], data[2], scales[0], scales[1], scales[2]};
+ *h = hold;
+ return 1;
+}
+static void v4_vkt_unhold(void *ctx, void *h) {
+ V4VkHold *hold = h;
+ if (!hold) return;
+ coli_expert_release((ColiExpertStore *)ctx, &hold->view);
+ free(hold->buffer); free(hold);
+}
static void v4_vk_tier_start(const ColiV4Engine *engine) {
const ColiDeepSeekV4Config *c = &engine->config;
ColiExpertStore *store = engine->experts;
@@ -12686,9 +12918,12 @@ static void v4_vk_tier_start(const ColiV4Engine *engine) {
.act_limit = c->swiglu_limit, .max_rows = 128 * c->num_experts_per_tok,
.ram_reserve = st.capacity_bytes > st.resident_bytes
? (size_t)(st.capacity_bytes - st.resident_bytes) : 0,
+ /* what the device has still to take: matrices placed already
+ * (the dense weights on the device only) are in its usage */
.dense_bytes = coli_vk_dense()
- ? (size_t)(engine->dense_resident.total_bytes + engine->head_cache.bytes) : 0,
- .in_ram = v4_vkt_in_ram, .ram_ctx = store};
+ ? (size_t)(engine->dense_resident.total_bytes - engine->dense_resident.dropped_bytes + engine->head_cache.bytes) : 0,
+ .in_ram = v4_vkt_in_ram, .ram_ctx = store,
+ .load = v4_vkt_load, .release = v4_vkt_unhold, .load_ctx = store};
/* The device goes before the drivers unload, after the tier's teardown (atexit runs
* last-registered first), and whether or not the tier starts: vkt_init makes the
* expert batch's pipelines before it can refuse (no room). */
@@ -12741,26 +12976,164 @@ static void v4c_atexit(void);
static void v4c_close(void);
#endif
-static void v4_vk_open(const ColiV4Engine *engine) {
#ifdef COLI_VULKAN
+/* ---- dense weights on the device only (COLI_VK_DENSE_HOST) ------------------------
+ * The RAM plan asks (coli_v4_dense_device_decide) before the engine exists as a whole,
+ * so the device opens there, early, with the chain's decision: COLI_VK_DENSE_HOST=0,
+ * or unset by coli_vk_dense_host_decide's rule (an integrated GPU; a discrete one with
+ * room for the matrices), with the dense part on the device (the chain, or
+ * COLI_VK_DENSE). Then the plan counts the matrices out of RAM and keeps the layers
+ * resident, every resident layer is uploaded as it is read and its fp8 matrices and
+ * compressor projections give their pages back (v4_dho_place), all of it before the
+ * expert tier sizes its budget. The CPU reads such a matrix again from disk at its own
+ * address (v4_dho_host), and a lost device turns the resident layers into per-forward
+ * reads, as a low-memory plan does (v4_dho_lost). */
+static int g_v4_vk_opened, g_v4_dho_asked;
+static pthread_mutex_t g_v4_dho_mx = PTHREAD_MUTEX_INITIALIZER;
+static int v4c_decide(const ColiV4Engine *engine); /* deepseek_v4_chain.h */
+
+static void v4_vk_device_open(const ColiV4Engine *engine) {
+ if (g_v4_vk_opened) return;
+ g_v4_vk_opened = 1;
g_v4_vk_thread = pthread_self();
g_v4_vk_engine = engine;
/* the device opens knowing whether the routed-expert tier will be tried, so the
- * dense matrices get their default place (coli_vk_dense_decide) */
- int tier = vkt_wanted() && engine->experts && !engine->experts->gpu &&
+ * dense matrices get their default place (coli_vk_dense_decide); asked by the RAM
+ * plan, the expert store is not open yet */
+ int tier = vkt_wanted() && !(engine->experts && engine->experts->gpu) &&
engine->config.n_routed_experts > 0;
g_v4_vk_ready = coli_vk_init_env_tier("deepseek_v4", tier);
+}
+static int v4_dho_decide(ColiV4Engine *engine, uint64_t device_bytes, uint64_t host_bytes) {
+ (void)host_bytes;
+ const char *on = getenv("COLI_VULKAN"), *keep = getenv("COLI_VK_DENSE_HOST");
+ if (!on || !atoi(on)) return 0;
+ if (keep && *keep && atoi(keep) != 0) return 0; /* kept: said in v4_vk_open, the device opened there as before */
+#ifdef COLI_V4_GPU_TIER
+ const char *cuda = getenv("DSV4_CUDA"); /* the CUDA tier mirrors the host layers */
+ if (!(cuda && atoi(cuda) == 0)) return 0;
+#endif
+ v4_vk_device_open(engine);
+ if (!g_v4_vk_ready) return 0;
+ g_v4_dho_asked = 1;
+ int chain = v4c_decide(engine);
+ return coli_vk_dense_host_decide("deepseek_v4", chain != 0 || coli_vk_dense(), (size_t)device_bytes);
+}
+/* The CPU needs a matrix the device holds alone: read back from disk at its address. */
+static int v4_dho_host(const void *data) {
+ ColiV4Engine *engine = (ColiV4Engine *)g_v4_vk_engine;
+ if (!engine || !engine->dense_resident.device_only) return 0;
+ uint64_t bytes = 0;
+ pthread_mutex_lock(&g_v4_dho_mx);
+ int r = coli_v4_layer_host_restore(engine, data, &bytes);
+ pthread_mutex_unlock(&g_v4_dho_mx);
+ if (r < 0) {
+ fprintf(stderr, "[VK] deepseek_v4: a dense matrix the device held alone could not be read back from disk\n");
+ exit(1);
+ }
+ if (r > 0) coli_vk_dense_host_reloaded((size_t)bytes);
+ return 0;
+}
+static void v4_dho_reread(uint64_t bytes) { coli_vk_dense_host_reloaded((size_t)bytes); }
+static int v4_dho_lost(void) {
+ static int said;
+ if (coli_vk_available()) return 0;
+ if (!said) {
+ said = 1;
+ fprintf(stderr, "[VK] deepseek_v4: the device that held the dense layers is gone; the CPU reads them "
+ "from disk per forward from here on, as a low-memory plan does\n");
+ }
+ return 1;
+}
+/* One resident layer just read: each droppable tensor up as every lookup will ask for it
+ * (fp8 as fmt 12 whole, wo_a per output group; the compressors' bf16 as fmt 11), then its
+ * pages back. A tensor the device refuses keeps its host copy. */
+static ColiVkTensor *v4c_tensor(int fmt, const void *data, const float *scales, int rows8, int rows, int columns);
+static int v4_dho_up(int fmt, const void *data, const float *scales, int rows8, int rows, int columns) {
+ if (!v4c_tensor(fmt, data, scales, rows8, rows, columns)) return 0;
+ V4VkEntry *e = v4_vk_find(data, fmt, rows, columns);
+ if (e) e->dho = 1;
+ return e != NULL;
+}
+static unsigned g_v4_dho_kept;
+static int v4_dho_place(ColiV4Engine *engine, int layer) {
+ ColiDeepSeekV4LayerWeights *w = &engine->dense_resident.layers[layer];
+ const ColiDeepSeekV4Config *c = &engine->config;
+ for (size_t i = 0; i < w->plan.tensor_count; i++) {
+ const ColiDeepSeekV4TensorSpec *spec = &w->plan.tensors[i];
+ if (!w->mapped[i] || w->host_gone[i]) continue;
+ int O = (int)spec->shape[0], I = (int)spec->shape[1], ok = 1;
+ if (spec->dtype == COLI_ST_F8_E4M3) {
+ char sname[COLI_V4_MAX_TENSOR_NAME];
+ size_t n = strlen(spec->name);
+ snprintf(sname, sizeof sname, "%.*s.scale", (int)(n > 7 ? n - 7 : 0), spec->name); /* ".weight" -> ".scale" */
+ const float *scales = coli_v4_layer_data(w, sname, NULL);
+ const unsigned char *data = w->data[i];
+ size_t tail = strlen(".attn.wo_a.weight");
+ if (!scales) ok = 0;
+ else if (n > tail && !strcmp(spec->name + n - tail, ".attn.wo_a.weight") && c->o_groups > 1) {
+ int rows = c->o_lora_rank, sc = (I + 127) / 128, sr = (rows + 127) / 128;
+ for (int g = 0; ok && g < c->o_groups; g++)
+ ok = v4_dho_up(12, data + (size_t)g * rows * I, scales + (size_t)g * sr * sc,
+ spec->packed_rows8, rows, I);
+ } else ok = v4_dho_up(12, data, scales, spec->packed_rows8, O, I);
+ } else ok = v4_dho_up(11, w->data[i], NULL, 0, O, I);
+ if (ok && !coli_v4_layer_host_drop(engine, layer, i))
+ coli_vk_dense_host_dropped((size_t)coli_v4_dense_tensor_bytes(spec));
+ else g_v4_dho_kept++;
+ }
+ return 0;
+}
+#endif
+
+static void v4_vk_open(const ColiV4Engine *engine) {
+#ifdef COLI_VULKAN
+ v4_vk_device_open(engine); /* opened already when the RAM plan asked */
+ g_v4_vk_engine = engine;
+ ColiV4Engine *mut = (ColiV4Engine *)engine;
+ if (g_v4_vk_ready && !g_v4_dho_asked) { /* COLI_VK_DENSE_HOST=1: said here, as every engine says it */
+ const char *keep = getenv("COLI_VK_DENSE_HOST");
+ if (keep && *keep && atoi(keep) != 0)
+ coli_vk_dense_host_decide("deepseek_v4", v4c_decide(engine) != 0 || coli_vk_dense(),
+ (size_t)engine->dense_resident.total_bytes);
+ }
+ int device_only = g_v4_vk_ready && engine->dense_resident.device_only;
+ if (device_only) coli_v4_vk_host = v4_dho_host;
v4c_start(engine); /* COLI_VK_CHAIN: the trunk on the device, before the tier sizes itself */
+ if (device_only) { /* every layer up now (the chain has read them if it runs): before the tier */
+ char err[256] = {0};
+ for (int i = 0; i < engine->config.num_hidden_layers; i++) {
+ ColiDeepSeekV4LayerWeights w;
+ if (coli_v4_layer_load(mut, &w, &engine->config, engine->target_index, i, err, sizeof err)) {
+ fprintf(stderr, "[VK] deepseek_v4: layer %d did not load (%s)\n", i, err);
+ break;
+ }
+ }
+ coli_vk_dense_host_placed("deepseek_v4", g_v4_dho_kept
+ ? "norms, block scales, the router, weights_proj, the mHC mixes, the head; and matrices the device refused"
+ : "norms, block scales, the router, the indexer's weights_proj, the mHC mixes, the head");
+ }
v4_vk_tier_start(engine);
v4c_atexit(); /* after the tier's: the chain goes before the device */
if (g_v4_vk_ready && !vkt_ready() && !coli_vk_dense())
coli_vk_dense_decide("deepseek_v4", 0, 1); /* no tier after all: the trunk to the device */
- if (g_v4_vk_ready && coli_vk_dense()) coli_v4_vk_matmul = v4_vk_matmul_impl; /* COLI_VK_DENSE=0: no hook */
+ /* COLI_VK_DENSE=0: no hook, except for the matrices the device holds alone */
+ if (g_v4_vk_ready && (coli_vk_dense() || device_only)) coli_v4_vk_matmul = v4_vk_matmul_impl;
#else
(void)engine;
#endif
}
+/* Before the engine opens: the RAM plan may put the dense layers on the device only. */
+static void v4_vk_hooks(void) {
+#ifdef COLI_VULKAN
+ coli_v4_dense_device_decide = v4_dho_decide;
+ coli_v4_dense_place = v4_dho_place;
+ coli_v4_dense_device_lost = v4_dho_lost;
+ coli_v4_dense_reread = v4_dho_reread;
+#endif
+}
+
/* Once, at the end: what ran on the device, then the device copies go before the
* engine frees the weights they mirror. */
static void v4_vk_close(void) {
@@ -12782,6 +13155,7 @@ static void v4_vk_close(void) {
fprintf(stderr, "[VK] deepseek_v4: %zu resident matrices on the device, %.1f MiB "
"(fp8 as fmt 12, bf16 as fmt 11)\n", tensors, bytes / 1048576.0);
coli_v4_vk_matmul = NULL;
+ coli_v4_vk_host = NULL;
for (size_t i = 0; i < g_v4_vk_cap; i++)
coli_vk_tensor_free(g_v4_vk_map[i].tensor);
free(g_v4_vk_map);
@@ -15595,6 +15969,7 @@ static int v4_serve_main(void) {
if (ram && atof(ram) > 0.0)
open_options.memory_limit_bytes =
(uint64_t)(atof(ram) * 1073741824.0);
+ v4_vk_hooks(); /* the RAM plan may put the dense layers on the Vulkan device only */
if (coli_v4_engine_open(&engine, &open_options, error, sizeof(error))) {
fprintf(stderr, "%s\n", error);
return 1;
@@ -15732,6 +16107,7 @@ int main(int argc, char **argv) {
if (cli.memory_gib > 0.0)
open_opts.memory_limit_bytes =
(uint64_t)(cli.memory_gib * 1073741824.0);
+ v4_vk_hooks(); /* the RAM plan may put the dense layers on the Vulkan device only */
if (coli_v4_engine_open(&engine, &open_opts, error, sizeof(error))) {
fprintf(stderr, "%s\n", error);
goto cleanup;
@@ -16872,6 +17248,12 @@ fail:
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
+#ifdef _WIN32
+#include <windows.h>
+#else
+#include <sys/mman.h>
+#include <unistd.h>
+#endif
static int set_error(char *error, size_t size, const char *format, ...) {
if (error && size) {
@@ -17046,11 +17428,69 @@ int coli_v4_layer_validate(const ColiDeepSeekV4LayerPlan *plan,
return 0;
}
+/* Dense weights on the Vulkan device only: a droppable tensor of a resident layer is
+ * read into an anonymous mapping of its own, so that its pages can go back to the system
+ * while its address stays reserved (the device copies are found by it, and no later
+ * allocation can take it), and come back at the same address if the CPU needs it. */
+static int g_v4_layer_map; /* the reference loader maps droppable tensors (resident, device only) */
+static size_t v4_map_len(size_t bytes) {
+#ifdef _WIN32
+ SYSTEM_INFO si; GetSystemInfo(&si);
+ size_t page = si.dwAllocationGranularity ? si.dwAllocationGranularity : 65536;
+#else
+ long ps = sysconf(_SC_PAGESIZE);
+ size_t page = ps > 0 ? (size_t)ps : 4096;
+#endif
+ return (bytes + page - 1) / page * page;
+}
+static void *v4_map_alloc(size_t bytes) {
+ size_t len = v4_map_len(bytes ? bytes : 1);
+#ifdef _WIN32
+ return VirtualAlloc(NULL, len, MEM_RESERVE | MEM_COMMIT, PAGE_READWRITE);
+#else
+ void *p = mmap(NULL, len, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
+ return p == MAP_FAILED ? NULL : p;
+#endif
+}
+static int v4_map_drop(void *data, size_t bytes) { /* pages back, the range reserved and unreadable */
+ size_t len = v4_map_len(bytes ? bytes : 1);
+#ifdef _WIN32
+ return VirtualFree(data, len, MEM_DECOMMIT) ? 0 : -1;
+#else
+ int flags = MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED;
+#ifdef MAP_NORESERVE
+ flags |= MAP_NORESERVE;
+#endif
+ return mmap(data, len, PROT_NONE, flags, -1, 0) == MAP_FAILED ? -1 : 0;
+#endif
+}
+static int v4_map_restore(void *data, size_t bytes) { /* zeroed, writable pages at the same address */
+ size_t len = v4_map_len(bytes ? bytes : 1);
+#ifdef _WIN32
+ return VirtualAlloc(data, len, MEM_COMMIT, PAGE_READWRITE) == data ? 0 : -1;
+#else
+ return mprotect(data, len, PROT_READ | PROT_WRITE);
+#endif
+}
+static void v4_map_free(void *data, size_t bytes) {
+ if (!data) return;
+#ifdef _WIN32
+ (void)bytes;
+ VirtualFree(data, 0, MEM_RELEASE);
+#else
+ munmap(data, v4_map_len(bytes ? bytes : 1));
+#endif
+}
+
void coli_v4_layer_free(ColiV4Engine *engine,
ColiDeepSeekV4LayerWeights *weights) {
(void)engine;
if (!weights) return;
- for (size_t i = 0; i < weights->plan.tensor_count; i++) free(weights->data[i]);
+ for (size_t i = 0; i < weights->plan.tensor_count; i++) {
+ if (weights->mapped[i])
+ v4_map_free(weights->data[i], (size_t)coli_v4_dense_tensor_bytes(&weights->plan.tensors[i]));
+ else free(weights->data[i]);
+ }
memset(weights, 0, sizeof(*weights));
}
@@ -17071,8 +17511,11 @@ int coli_v4_layer_load(ColiV4Engine *engine,
const ColiSafetensorsTensor *tensor = coli_st_find(index, spec->name);
size_t resident_bytes = tensor->dtype == COLI_ST_F8_E8M0
? (size_t)tensor->numel * sizeof(float) : (size_t)tensor->nbytes;
- weights->data[i] = malloc(resident_bytes);
+ weights->mapped[i] = g_v4_layer_map && coli_v4_dense_device_only_tensor(spec) &&
+ resident_bytes == coli_v4_dense_tensor_bytes(spec);
+ weights->data[i] = weights->mapped[i] ? v4_map_alloc(resident_bytes) : malloc(resident_bytes);
if (!weights->data[i]) {
+ weights->mapped[i] = 0;
coli_v4_layer_free(NULL, weights);
return set_error(error, error_size, "out of memory loading: %s", spec->name);
}
@@ -18187,6 +18630,7 @@ static int fp8_matvec_compute(float *output, const ColiTensorView *weight,
* rounded to E4M3 per 128, so the device multiplies exactly what the CPU kernel
* would; only the order of the sums differs. */
ColiV4VkMatmul coli_v4_vk_matmul;
+ColiV4VkHost coli_v4_vk_host;
#endif
int coli_fp8_matvec_ref(float *output, const ColiTensorView *weight,
@@ -18213,6 +18657,7 @@ int coli_fp8_matvec_ref(float *output, const ColiTensorView *weight,
}
#ifdef COLI_VULKAN
if (coli_v4_vk_fp8(output, weight, activation, 1)) return 0;
+ coli_v4_vk_host_ensure(weight->data);
#endif
return fp8_matvec_compute(output, weight, activation);
}
@@ -18235,6 +18680,7 @@ int coli_fp8_matvec_pre(float *output, const ColiTensorView *weight,
#endif
#ifdef COLI_VULKAN
if (coli_v4_vk_fp8(output, weight, activation, 1)) return 0;
+ coli_v4_vk_host_ensure(weight->data);
#endif
return fp8_matvec_compute(output, weight, activation);
}
@@ -18349,6 +18795,8 @@ int coli_fp8_dual_matvec_ref(float *output_a, float *output_b,
#ifdef COLI_VULKAN
if (coli_v4_vk_fp8(output_a, a, activation, 1) &&
coli_v4_vk_fp8(output_b, b, activation, 1)) return 0;
+ coli_v4_vk_host_ensure(a->data);
+ coli_v4_vk_host_ensure(b->data);
#endif
#ifdef __AVX2__
if (a->block_rows == 8) {
@@ -18472,6 +18920,7 @@ int coli_fp8_matmul_batch_ref(float *outputs, const ColiTensorView *weight,
}
#ifdef COLI_VULKAN
if (coli_v4_vk_fp8(outputs, weight, activations, batch)) return 0;
+ coli_v4_vk_host_ensure(weight->data);
#endif
return fp8_batch_compute(outputs, weight, activations, batch);
}
@@ -18494,6 +18943,7 @@ int coli_fp8_matmul_batch_pre(float *outputs, const ColiTensorView *weight,
#endif
#ifdef COLI_VULKAN
if (coli_v4_vk_fp8(outputs, weight, activations, batch)) return 0;
+ coli_v4_vk_host_ensure(weight->data);
#endif
return fp8_batch_compute(outputs, weight, activations, batch);
}
diff --git a/c/deepseek_v41.c b/c/deepseek_v41.c
index 3f0eedfb..7d1a11bb 100644
--- a/c/deepseek_v41.c
+++ b/c/deepseek_v41.c
@@ -52,6 +52,10 @@
#if defined(__APPLE__) || defined(__linux__) || defined(__FreeBSD__)
#include <sys/resource.h>
#endif
+#if defined(COLI_VULKAN) && !defined(_WIN32)
+#include <sys/mman.h>
+#include <unistd.h>
+#endif
#ifdef _OPENMP
#include <omp.h>
#endif
@@ -322,6 +326,107 @@ static void *xmalloc(size_t bytes, const char *what) {
return p;
}
+#ifdef COLI_VULKAN
+/* ---- dense weights on the device only (COLI_VK_DENSE_HOST) -------------------------
+ * With the trunk on the device (the chain, or COLI_VK_DENSE) and this mode on (decided
+ * by v41_dho_open in model_load, before the layers are read), each trunk matrix (the
+ * fp8 rows of every W8 of a layer, the compressor's, indexer's and engram's projections;
+ * not the router, the head, the embedding, the norms, DSpark's stages or the vision
+ * tower) is read into an anonymous mapping of its own, uploaded as every lookup will ask
+ * for it, and its pages given back with the address kept: the device copies are found
+ * by that address (vk_entry), and a matrix the CPU needs again (a lost device, a step
+ * the device declines) is read back from disk at it (v41_dho_host), and stays. */
+typedef struct {
+ unsigned char *base; size_t bytes;
+ char name[256];
+ int kind, O, I; /* 8: the e4m3 rows of a W8 (tiles below), 16: a WB */
+ const uint8_t *tiles;
+ int placed, gone;
+} V41Home;
+static V41Home *g_v41_home;
+static size_t g_v41_home_n, g_v41_home_cap;
+static int g_v41_dho, g_v41_map, g_vk_opened;
+static shards *g_v41_shards;
+static pthread_mutex_t g_v41_home_mx = PTHREAD_MUTEX_INITIALIZER;
+static size_t v41_map_len(size_t bytes) {
+#ifdef _WIN32
+ SYSTEM_INFO si; GetSystemInfo(&si);
+ size_t page = si.dwAllocationGranularity ? si.dwAllocationGranularity : 65536;
+#else
+ long ps = sysconf(_SC_PAGESIZE);
+ size_t page = ps > 0 ? (size_t)ps : 4096;
+#endif
+ return ((bytes ? bytes : 1) + page - 1) / page * page;
+}
+/* pages back with the range kept (and unreadable), and back again zeroed */
+static int v41_map_drop(void *p, size_t bytes) {
+#ifdef _WIN32
+ return VirtualFree(p, v41_map_len(bytes), MEM_DECOMMIT) ? 0 : -1;
+#else
+ int flags = MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED;
+#ifdef MAP_NORESERVE
+ flags |= MAP_NORESERVE;
+#endif
+ return mmap(p, v41_map_len(bytes), PROT_NONE, flags, -1, 0) == MAP_FAILED ? -1 : 0;
+#endif
+}
+static int v41_map_restore(void *p, size_t bytes) {
+#ifdef _WIN32
+ return VirtualAlloc(p, v41_map_len(bytes), MEM_COMMIT, PAGE_READWRITE) == p ? 0 : -1;
+#else
+ return mprotect(p, v41_map_len(bytes), PROT_READ | PROT_WRITE);
+#endif
+}
+static void *v41_home_alloc(size_t bytes, const char *name) {
+#ifdef _WIN32
+ void *p = VirtualAlloc(NULL, v41_map_len(bytes), MEM_RESERVE | MEM_COMMIT, PAGE_READWRITE);
+ if (!p) { fprintf(stderr, "OOM mapping %s (%zu bytes)\n", name, bytes); exit(1); }
+#else
+ void *p = mmap(NULL, v41_map_len(bytes), PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
+ if (p == MAP_FAILED) { fprintf(stderr, "OOM mapping %s (%zu bytes)\n", name, bytes); exit(1); }
+#endif
+ if (g_v41_home_n == g_v41_home_cap) {
+ g_v41_home_cap = g_v41_home_cap ? 2 * g_v41_home_cap : 256;
+ g_v41_home = realloc(g_v41_home, g_v41_home_cap * sizeof(*g_v41_home));
+ if (!g_v41_home) { fprintf(stderr, "OOM dense weights table\n"); exit(1); }
+ }
+ V41Home *h = &g_v41_home[g_v41_home_n++];
+ memset(h, 0, sizeof *h);
+ h->base = p; h->bytes = bytes;
+ snprintf(h->name, sizeof h->name, "%s", name);
+ return p;
+}
+static V41Home *v41_home_of(const void *data) {
+ uintptr_t q = (uintptr_t)data;
+ for (size_t i = 0; i < g_v41_home_n; i++)
+ if (q >= (uintptr_t)g_v41_home[i].base && q - (uintptr_t)g_v41_home[i].base < g_v41_home[i].bytes)
+ return &g_v41_home[i];
+ return NULL;
+}
+/* The CPU is about to read the matrix that holds `data`: read back what the device held
+ * alone, at its own address. */
+static void v41_dho_host(const void *data) {
+ if (!g_v41_dho || !data) return;
+ pthread_mutex_lock(&g_v41_home_mx);
+ V41Home *h = v41_home_of(data);
+ if (h && h->gone) {
+ if (v41_map_restore(h->base, h->bytes)) {
+ fprintf(stderr, "[VK] deepseek_v41: %s could not come back into RAM\n", h->name); exit(1);
+ }
+ st_read_raw_cap(g_v41_shards, h->name, h->base, (int64_t)h->bytes, 0);
+ h->gone = 0;
+ coli_vk_dense_host_reloaded(h->bytes);
+ }
+ pthread_mutex_unlock(&g_v41_home_mx);
+}
+#endif
+static void *v41_weight_alloc(size_t bytes, const char *name) {
+#ifdef COLI_VULKAN
+ if (g_v41_map) return v41_home_alloc(bytes, name);
+#endif
+ return xmalloc(bytes, name);
+}
+
static void w8_load(shards *S, W8 *w, const char *name, int O, int I) {
char scale_name[512];
snprintf(scale_name, sizeof(scale_name), "%.*s.scale",
@@ -333,10 +438,13 @@ static void w8_load(shards *S, W8 *w, const char *name, int O, int I) {
name, (long long)t->nbytes, (long long)O * I, O, I); exit(1); }
int tiles = ((O + FP8_TILE - 1) / FP8_TILE) * ((I + FP8_TILE - 1) / FP8_TILE);
w->O = O; w->I = I;
- w->q = xmalloc((size_t)O * I, name);
+ w->q = v41_weight_alloc((size_t)O * I, name);
w->s = xmalloc((size_t)tiles, scale_name);
st_read_raw_cap(S, name, w->q, (int64_t)O * I, 0);
st_read_raw_cap(S, scale_name, w->s, tiles, 0);
+#ifdef COLI_VULKAN
+ if (g_v41_map) { V41Home *h = &g_v41_home[g_v41_home_n - 1]; h->kind = 8; h->O = O; h->I = I; h->tiles = w->s; }
+#endif
}
static void wb_load(shards *S, WB *w, const char *name, int O, int I) {
@@ -346,8 +454,11 @@ static void wb_load(shards *S, WB *w, const char *name, int O, int I) {
fprintf(stderr, "%s: %lld bytes, expected %lld for [%d, %d] bf16\n",
name, (long long)t->nbytes, (long long)O * I * 2, O, I); exit(1); }
w->O = O; w->I = I;
- w->w = xmalloc((size_t)O * I * 2, name);
+ w->w = v41_weight_alloc((size_t)O * I * 2, name);
st_read_raw_cap(S, name, w->w, (int64_t)O * I * 2, 0);
+#ifdef COLI_VULKAN
+ if (g_v41_map) { V41Home *h = &g_v41_home[g_v41_home_n - 1]; h->kind = 16; h->O = O; h->I = I; }
+#endif
}
static void wf_load(shards *S, WF *w, const char *name, int64_t n) {
@@ -386,7 +497,7 @@ static inline float ue8m0(uint8_t byte) {
* lives as long as the model, so a pointer names the same bytes for the whole run.
* One command buffer: calls come from the thread that opened the device and never
* from inside an OpenMP region; anything else stays on the CPU. */
-typedef struct { const void *data; int fmt, O, I, refused; ColiVkTensor *t; } VkEntry;
+typedef struct { const void *data; int fmt, O, I, refused; ColiVkTensor *t; int dho; } VkEntry; /* dho: placed, the device holds it alone */
static VkEntry *g_vk_map;
static size_t g_vk_cap, g_vk_used;
static pthread_t g_vk_thread;
@@ -414,7 +525,7 @@ static VkEntry *vk_entry(const void *data, int fmt, int O, int I) {
for (;; at = (at + 1) & (g_vk_cap - 1)) {
VkEntry *e = &g_vk_map[at];
if (!e->data) {
- *e = (VkEntry){data, fmt, O, I, 0, NULL};
+ *e = (VkEntry){data, fmt, O, I, 0, NULL, 0};
g_vk_used++;
return e;
}
@@ -422,17 +533,30 @@ static VkEntry *vk_entry(const void *data, int fmt, int O, int I) {
}
}
+/* The entry of a matrix, NULL when it has none (no insertion). */
+static VkEntry *vk_find(const void *data, int fmt, int O, int I) {
+ if (!g_vk_cap) return NULL;
+ for (size_t at = vk_hash(data, g_vk_cap);; at = (at + 1) & (g_vk_cap - 1)) {
+ VkEntry *e = &g_vk_map[at];
+ if (!e->data) return NULL;
+ if (e->data == data && e->fmt == fmt && e->O == O && e->I == I) return e;
+ }
+}
+
/* y = W x for `rows` positions on the device; 0 sends the caller to its CPU kernel.
* x and y may be strided (wo_a's blocks read and write inside wider rows): the
* device wants them packed, so they are packed here. */
static int vk_mul(int fmt, const void *data, const uint8_t *tiles, int O, int I,
float *y, int ystride, const float *x, int xstride, int rows) {
- if (!g_vk_ready || !coli_vk_dense() || rows < 1 || !pthread_equal(pthread_self(), g_vk_thread)) return 0;
+ if (!g_vk_ready || rows < 1 || !pthread_equal(pthread_self(), g_vk_thread)) return 0;
+ /* COLI_VK_DENSE=0: the CPU, except for a matrix the device holds alone */
+ if (!coli_vk_dense()) { VkEntry *f = vk_find(data, fmt, O, I); if (!f || !f->dho) return 0; }
#ifdef _OPENMP
if (omp_in_parallel()) return 0;
#endif
VkEntry *e = vk_entry(data, fmt, O, I);
if (!e || e->refused) return 0;
+ if (!e->t) v41_dho_host(data); /* a view the placement did not make: its rows back first */
float *scales = NULL;
if (!e->t && fmt == 12) {
int groups = (I + FP8_TILE - 1) / FP8_TILE;
@@ -471,6 +595,7 @@ static int vk_mul(int fmt, const void *data, const uint8_t *tiles, int O, int I,
static void mv8(float *y, const W8 *w, const float *x) {
#ifdef COLI_VULKAN
if (vk_mul(12, w->q, w->s, w->O, w->I, y, w->O, x, w->I, 1)) return;
+ v41_dho_host(w->q);
#endif
int I = w->I, tiles_i = (I + FP8_TILE - 1) / FP8_TILE;
#pragma omp parallel for schedule(static)
@@ -522,6 +647,7 @@ static int mv_block_rows(int I) {
static void mv8_rows(float *y, int ystride, const W8 *w, const float *x, int xstride, int rows) {
#ifdef COLI_VULKAN
if (vk_mul(12, w->q, w->s, w->O, w->I, y, ystride, x, xstride, rows)) return;
+ v41_dho_host(w->q);
#endif
int I = w->I, tiles_i = (I + FP8_TILE - 1) / FP8_TILE;
int block = mv_block_rows(I);
@@ -587,6 +713,7 @@ static void mvb_cpu(float *y, const WB *w, const float *x) {
static void mvb(float *y, const WB *w, const float *x) {
#ifdef COLI_VULKAN
if (vk_mul(11, w->w, NULL, w->O, w->I, y, w->O, x, w->I, 1)) return;
+ v41_dho_host(w->w);
#endif
mvb_cpu(y, w, x);
}
@@ -596,6 +723,7 @@ static void mvb(float *y, const WB *w, const float *x) {
static void mvb_rows(float *y, int ystride, const WB *w, const float *x, int xstride, int rows) {
#ifdef COLI_VULKAN
if (vk_mul(11, w->w, NULL, w->O, w->I, y, ystride, x, xstride, rows)) return;
+ v41_dho_host(w->w);
#endif
for (int r = 0; r < rows; r++) mvb_cpu(y + (size_t)r * ystride, w, x + (size_t)r * xstride);
}
@@ -1603,6 +1731,94 @@ static void cache_init(Model *m, LCache *cache, int ecap) {
static void attn_project_check(const Cfg *c);
+#ifdef COLI_VULKAN
+static int v41c_decide(Model *m); /* deepseek_v41_chain.h: COLI_VK_CHAIN, decided once */
+/* What the device would hold alone: the trunk matrices model_load reads into mappings. */
+static size_t v41_dho_bytes(Model *m) {
+ const Cfg *c = &m->c;
+ char name[256];
+ size_t total = 0;
+ #define B(...) do { snprintf(name, sizeof name, __VA_ARGS__); st_tensor *t = st_find(&m->S, name); \
+ if (t && t->nbytes > 0) total += (size_t)t->nbytes; } while (0)
+ for (int i = 0; i < c->n_layers; i++) {
+ B("layers.%d.attn.wq_a.weight", i); B("layers.%d.attn.wq_b.weight", i); B("layers.%d.attn.wkv.weight", i);
+ B("layers.%d.attn.wo_a.weight", i); B("layers.%d.attn.wo_b.weight", i);
+ B("layers.%d.ffn.shared_experts.w1.weight", i); B("layers.%d.ffn.shared_experts.w3.weight", i);
+ B("layers.%d.ffn.shared_experts.w2.weight", i);
+ if (c->kv_source[i]) {
+ B("layers.%d.attn.compressor.wkv.weight", i);
+ if (c->compress_ratio[i] > 1) B("layers.%d.attn.compressor.wgate.weight", i);
+ B("layers.%d.attn.indexer.wk.weight", i);
+ }
+ if (c->index_source[i]) { B("layers.%d.attn.indexer.wq_b.weight", i); B("layers.%d.attn.indexer.weights_proj.weight", i); }
+ }
+ for (int t = 0; m->engram.active && t < m->engram.n_layers; t++) B("layers.%d.engram.wkv.weight", m->engram.layer_of[t]);
+ #undef B
+ return total;
+}
+/* Before the layers are read: the device, the chain's decision, and whether the trunk
+ * lives on the device only. COLI_VK_DENSE_HOST=1 (kept) opens the device after the
+ * weights, as before. */
+static void v41_dho_open(Model *m) {
+ const char *on = getenv("COLI_VULKAN"), *keep = getenv("COLI_VK_DENSE_HOST");
+ if (!on || !atoi(on) || (keep && *keep && atoi(keep) != 0)) return;
+ g_vk_opened = 1;
+ g_vk_thread = pthread_self();
+ g_vk_ready = coli_vk_init_env_tier("deepseek_v41", vkt_wanted() && m->c.n_routed > 0);
+ if (!g_vk_ready) return;
+ int chain = v41c_decide(m);
+ g_v41_dho = coli_vk_dense_host_decide("deepseek_v41", chain != 0 || coli_vk_dense(), v41_dho_bytes(m));
+ g_v41_shards = &m->S;
+}
+/* One matrix up as a lookup will ask for it, marked as held by the device alone. */
+static int v41_dho_up(const void *q, const uint8_t *tiles, int fmt, int O, int I) {
+ VkEntry *e = vk_entry(q, fmt, O, I);
+ if (!e || e->refused) return 0;
+ if (!e->t) {
+ float *scales = NULL;
+ if (fmt == 12) {
+ int groups = (I + FP8_TILE - 1) / FP8_TILE;
+ if (!(scales = malloc((size_t)O * groups * sizeof(float)))) return 0;
+ for (int o = 0; o < O; o++)
+ for (int g = 0; g < groups; g++) scales[(size_t)o * groups + g] = ue8m0(tiles[(size_t)(o / FP8_TILE) * groups + g]);
+ }
+ int ok = coli_vk_tensor_ensure(&e->t, q, scales, fmt, I, O, fmt == 12 ? FP8_TILE : 0);
+ free(scales);
+ if (!ok) return 0;
+ }
+ e->dho = 1;
+ return 1;
+}
+/* The matrices read since the last call: up, then their pages back. wo_a goes up whole
+ * for the chain and per output group for the per-matrix path (attention_project's
+ * block views), each only where that path runs. */
+static unsigned g_v41_dho_kept;
+static void v41_dho_place(Model *m) {
+ const Cfg *c = &m->c;
+ int chain = v41c_decide(m);
+ for (size_t k = 0; k < g_v41_home_n; k++) {
+ V41Home *h = &g_v41_home[k];
+ if (h->placed) continue;
+ h->placed = 1;
+ int ok;
+ size_t n = strlen(h->name), tail = strlen(".attn.wo_a.weight");
+ if (h->kind == 8 && n > tail && !strcmp(h->name + n - tail, ".attn.wo_a.weight")) {
+ ok = 1;
+ if (chain) ok = v41_dho_up(h->base, h->tiles, 12, h->O, h->I);
+ int tiles_i = (h->I + FP8_TILE - 1) / FP8_TILE;
+ for (int g = 0; ok && chain != COLI_VK_CHAIN_ON && g < c->o_groups; g++)
+ ok = v41_dho_up(h->base + (size_t)g * c->o_lora * h->I,
+ h->tiles + (size_t)(g * c->o_lora / FP8_TILE) * tiles_i, 12, c->o_lora, h->I);
+ } else ok = v41_dho_up(h->base, h->tiles, h->kind == 8 ? 12 : 11, h->O, h->I);
+ if (ok && !v41_map_drop(h->base, h->bytes)) { h->gone = 1; coli_vk_dense_host_dropped(h->bytes); }
+ else g_v41_dho_kept++;
+ }
+}
+#define V41_DHO(call) do { g_v41_map = g_v41_dho; call; g_v41_map = 0; } while (0)
+#else
+#define V41_DHO(call) call
+#endif
+
static void model_load(Model *m, const char *snap, int ecap, int engram_cache_rows) {
Cfg *c = &m->c;
cfg_load(c, snap);
@@ -1625,6 +1841,9 @@ static void model_load(Model *m, const char *snap, int ecap, int engram_cache_ro
* by the time the replicas exist. */
mirror_setup(&m->S, snap, c->n_routed);
engram_load_sidecar(&m->engram, snap);
+#ifdef COLI_VULKAN
+ v41_dho_open(m); /* COLI_VK_DENSE_HOST: the trunk to the device as it is read, its host pages back */
+#endif
int dim = c->dim, hd = c->head_dim, nh = c->n_heads, hc = c->hc_mult;
m->L = xmalloc((size_t)c->n_layers * sizeof(Layer), "layers");
@@ -1638,12 +1857,12 @@ static void model_load(Model *m, const char *snap, int ecap, int engram_cache_ro
for (int i = 0; i < c->n_layers; i++) {
Layer *l = &m->L[i];
l->engram_index = -1;
- w8_load(&m->S, &l->wq_a, NAME("layers.%d.attn.wq_a.weight", i), c->q_lora, dim);
- w8_load(&m->S, &l->wq_b, NAME("layers.%d.attn.wq_b.weight", i), nh * hd, c->q_lora);
- w8_load(&m->S, &l->wkv, NAME("layers.%d.attn.wkv.weight", i), hd, dim);
- w8_load(&m->S, &l->wo_a, NAME("layers.%d.attn.wo_a.weight", i),
- c->o_groups * c->o_lora, nh * hd / c->o_groups);
- w8_load(&m->S, &l->wo_b, NAME("layers.%d.attn.wo_b.weight", i), dim, c->o_groups * c->o_lora);
+ V41_DHO(w8_load(&m->S, &l->wq_a, NAME("layers.%d.attn.wq_a.weight", i), c->q_lora, dim));
+ V41_DHO(w8_load(&m->S, &l->wq_b, NAME("layers.%d.attn.wq_b.weight", i), nh * hd, c->q_lora));
+ V41_DHO(w8_load(&m->S, &l->wkv, NAME("layers.%d.attn.wkv.weight", i), hd, dim));
+ V41_DHO(w8_load(&m->S, &l->wo_a, NAME("layers.%d.attn.wo_a.weight", i),
+ c->o_groups * c->o_lora, nh * hd / c->o_groups));
+ V41_DHO(w8_load(&m->S, &l->wo_b, NAME("layers.%d.attn.wo_b.weight", i), dim, c->o_groups * c->o_lora));
wf_load(&m->S, &l->q_norm, NAME("layers.%d.attn.q_norm.weight", i), c->q_lora);
wf_load(&m->S, &l->kv_norm, NAME("layers.%d.attn.kv_norm.weight", i), hd);
wf_load(&m->S, &l->attn_sink,NAME("layers.%d.attn.attn_sink", i), nh);
@@ -1658,9 +1877,9 @@ static void model_load(Model *m, const char *snap, int ecap, int engram_cache_ro
wf_load(&m->S, &l->hc_ffn_scale, NAME("layers.%d.hc_ffn_scale", i), 3);
wb_load(&m->S, &l->gate_w, NAME("layers.%d.ffn.gate.weight", i), c->n_routed, dim);
wf_load(&m->S, &l->gate_bias, NAME("layers.%d.ffn.gate.bias", i), c->n_routed);
- w8_load(&m->S, &l->sh_w1, NAME("layers.%d.ffn.shared_experts.w1.weight", i), c->moe_inter, dim);
- w8_load(&m->S, &l->sh_w3, NAME("layers.%d.ffn.shared_experts.w3.weight", i), c->moe_inter, dim);
- w8_load(&m->S, &l->sh_w2, NAME("layers.%d.ffn.shared_experts.w2.weight", i), dim, c->moe_inter);
+ V41_DHO(w8_load(&m->S, &l->sh_w1, NAME("layers.%d.ffn.shared_experts.w1.weight", i), c->moe_inter, dim));
+ V41_DHO(w8_load(&m->S, &l->sh_w3, NAME("layers.%d.ffn.shared_experts.w3.weight", i), c->moe_inter, dim));
+ V41_DHO(w8_load(&m->S, &l->sh_w2, NAME("layers.%d.ffn.shared_experts.w2.weight", i), dim, c->moe_inter));
l->window = xmalloc((size_t)c->window * hd * sizeof(float), "window ring");
memset(l->window, 0, (size_t)c->window * hd * sizeof(float));
@@ -1678,10 +1897,10 @@ static void model_load(Model *m, const char *snap, int ecap, int engram_cache_ro
l->ring_save_pos = xmalloc((size_t)rows * sizeof(int), "displaced ring positions");
int ratio = c->compress_ratio[i];
if (c->kv_source[i]) {
- wb_load(&m->S, &l->comp_wkv, NAME("layers.%d.attn.compressor.wkv.weight", i), hd, dim);
+ V41_DHO(wb_load(&m->S, &l->comp_wkv, NAME("layers.%d.attn.compressor.wkv.weight", i), hd, dim));
wf_load(&m->S, &l->comp_norm, NAME("layers.%d.attn.compressor.norm.weight", i), hd);
if (ratio > 1) {
- wb_load(&m->S, &l->comp_wgate, NAME("layers.%d.attn.compressor.wgate.weight", i), hd, dim);
+ V41_DHO(wb_load(&m->S, &l->comp_wgate, NAME("layers.%d.attn.compressor.wgate.weight", i), hd, dim));
l->cstate_kv = xmalloc((size_t)ratio * hd * sizeof(float), "compressor group");
l->cstate_score = xmalloc((size_t)ratio * hd * sizeof(float), "compressor scores");
l->cstate_save_kv = xmalloc((size_t)rows * hd * sizeof(float), "displaced group");
@@ -1695,17 +1914,20 @@ static void model_load(Model *m, const char *snap, int ecap, int engram_cache_ro
memset(l->ckv, 0, (size_t)slots * hd * sizeof(float));
l->ikey = xmalloc((size_t)slots * c->index_head_dim * sizeof(float), "index keys");
memset(l->ikey, 0, (size_t)slots * c->index_head_dim * sizeof(float));
- wb_load(&m->S, &l->idx_wk, NAME("layers.%d.attn.indexer.wk.weight", i),
- c->index_head_dim, hd);
+ V41_DHO(wb_load(&m->S, &l->idx_wk, NAME("layers.%d.attn.indexer.wk.weight", i),
+ c->index_head_dim, hd));
wf_load(&m->S, &l->idx_knorm, NAME("layers.%d.attn.indexer.k_norm.weight", i),
c->index_head_dim);
}
if (c->index_source[i]) {
- w8_load(&m->S, &l->idx_wq_b, NAME("layers.%d.attn.indexer.wq_b.weight", i),
- c->index_n_heads * c->index_head_dim, c->q_lora);
- wb_load(&m->S, &l->idx_wproj, NAME("layers.%d.attn.indexer.weights_proj.weight", i),
- c->index_n_heads, dim);
+ V41_DHO(w8_load(&m->S, &l->idx_wq_b, NAME("layers.%d.attn.indexer.wq_b.weight", i),
+ c->index_n_heads * c->index_head_dim, c->q_lora));
+ V41_DHO(wb_load(&m->S, &l->idx_wproj, NAME("layers.%d.attn.indexer.weights_proj.weight", i),
+ c->index_n_heads, dim));
}
+#ifdef COLI_VULKAN
+ if (g_v41_dho) v41_dho_place(m); /* one layer in RAM at a time */
+#endif
}
if (m->engram.active) {
Engram *e = &m->engram;
@@ -1715,8 +1937,11 @@ static void model_load(Model *m, const char *snap, int ecap, int engram_cache_ro
fprintf(stderr, "[engram] table %d names layer %d\n", t, layer); exit(1); }
m->L[layer].engram_index = t;
engram_table_open(&e->table[t], &m->S, layer, e->head_dim, engram_cache_rows);
- w8_load(&m->S, &m->L[layer].eng_wkv, NAME("layers.%d.engram.wkv.weight", layer),
- dim * (hc + 1), e->cols * e->head_dim);
+ V41_DHO(w8_load(&m->S, &m->L[layer].eng_wkv, NAME("layers.%d.engram.wkv.weight", layer),
+ dim * (hc + 1), e->cols * e->head_dim));
+#ifdef COLI_VULKAN
+ if (g_v41_dho) v41_dho_place(m);
+#endif
wf_load(&m->S, &m->L[layer].eng_q, NAME("layers.%d.engram.q_weight", layer),
(int64_t)hc * dim);
wf_load(&m->S, &m->L[layer].eng_k, NAME("layers.%d.engram.k_weight", layer),
@@ -2402,9 +2627,9 @@ static void moe_vk_cpu(Model *m, LCache *cache, int layer, int topk, const float
const int *chosen, const float *weights, const uint8_t *want, float *contrib) {
Cfg *c = &m->c;
int dim = c->dim, draws = rows * topk;
- int uniq[MOE_ROW_CHUNK * MOE_TOPK_MAX], head[MOE_ROW_CHUNK * MOE_TOPK_MAX];
- int tail[MOE_ROW_CHUNK * MOE_TOPK_MAX], count[MOE_ROW_CHUNK * MOE_TOPK_MAX];
- int next[MOE_ROW_CHUNK * MOE_TOPK_MAX];
+ /* a block of up to MOE_ROW_CHUNK rows, or a streamed prompt chunk (vkt_step_rows) */
+ int *uniq = xmalloc((size_t)draws * 5 * sizeof(int), "expert union");
+ int *head = uniq + draws, *tail = head + draws, *count = tail + draws, *next = count + draws;
int n_uniq = 0;
for (int d = 0; d < draws; d++) {
if (!want[d]) continue;
@@ -2416,7 +2641,7 @@ static void moe_vk_cpu(Model *m, LCache *cache, int layer, int topk, const float
tail[at] = d;
count[at]++;
}
- if (!n_uniq) return;
+ if (!n_uniq) { free(uniq); return; }
float *gathered = xmalloc((size_t)rows * dim * sizeof(float), "expert inputs");
float *down = xmalloc((size_t)rows * dim * sizeof(float), "expert outputs");
int step = cache->cap < MOE_ROW_CHUNK ? cache->cap : MOE_ROW_CHUNK;
@@ -2442,14 +2667,14 @@ static void moe_vk_cpu(Model *m, LCache *cache, int layer, int topk, const float
vkt_note(layer, uniq[u0 + u], &src);
}
}
- free(down); free(gathered);
+ free(down); free(gathered); free(uniq);
}
static void moe_vk_block(Model *m, Layer *l, LCache *cache, int layer, int topk, const float *xc,
int rows, float *outc, const int *chosen, const float *weights, int with_shared) {
int dim = m->c.dim, draws = rows * topk;
- uint8_t taken[MOE_ROW_CHUNK * MOE_TOPK_MAX], want[MOE_ROW_CHUNK * MOE_TOPK_MAX];
- const float *dev[MOE_ROW_CHUNK * MOE_TOPK_MAX];
+ uint8_t *taken = xmalloc((size_t)draws * 2, "device draws"), *want = taken + draws;
+ const float **dev = xmalloc((size_t)draws * sizeof(*dev), "device rows");
for (int r = 0; r < rows; r++) rt_count(layer, chosen + r * topk, topk);
int ndev = vkt_issue(layer, xc, rows, topk, chosen, taken);
for (int d = 0; d < draws; d++) { want[d] = !taken[d]; if (taken[d]) ehit_mark(m, layer, chosen[d]); }
@@ -2478,8 +2703,26 @@ static void moe_vk_block(Model *m, Layer *l, LCache *cache, int layer, int topk,
const float *sh = shared + (size_t)r * dim;
if (with_shared) for (int i = 0; i < dim; i++) o[i] += sh[i];
}
- free(shared); free(contrib);
+ free(shared); free(contrib); free(taken); free(dev);
+}
+/* The tier's streaming (a big prompt chunk's cold experts on the device): a group of a
+ * backbone layer's experts through its cache as moe_vk_cpu reads them (expert_slots_at:
+ * the misses fetched together), up to the cache's capacity, valid until the next read. */
+static int vk_load_batch(void *ctx, int layer, const int *e, int n, VktExpertSrc *srcs, void **h) {
+ Model *m = ctx;
+ LCache *cache = &m->cache[layer];
+ if (n > cache->cap) n = cache->cap;
+ if (n > 64) n = 64;
+ if (n < 1) return 0;
+ Slot *slot[64];
+ expert_slots_at(m, cache, "layers", layer, e, n, slot);
+ for (int u = 0; u < n; u++) { srcs[u] = vk_slot_src(slot[u]); h[u] = slot[u]; }
+ return n;
}
+static int vk_load(void *ctx, int layer, int e, VktExpertSrc *src, void **h) {
+ return vk_load_batch(ctx, layer, &e, 1, src, h) == 1;
+}
+static void vk_unhold(void *ctx, void *h) { (void)ctx; (void)h; }
#endif
/* The MoE for a block of positions, expert-major.
@@ -2507,6 +2750,22 @@ static void moe_run_at(Model *m, Layer *l, LCache *cache, const char *kind, int
int dim = c->dim;
if (topk > MOE_TOPK_MAX) {
fprintf(stderr, "[moe] n_activated %d exceeds the scratch\n", topk); exit(1); }
+#ifdef COLI_VULKAN
+ /* the tier streams a prompt chunk's cold experts: the whole step in one block, so
+ * each expert's rows meet in one GEMM (moe_vk_block's arrays follow its size) */
+ if (vkt_ready() && !strcmp(kind, "layers") && vkt_step_rows(n, MOE_ROW_CHUNK) > MOE_ROW_CHUNK) {
+ int *chosen = xmalloc((size_t)n * topk * sizeof(int), "routing");
+ float *weights = xmalloc((size_t)n * topk * sizeof(float), "routing weights");
+ float *scores = xmalloc((size_t)n * E * sizeof(float), "gate scores");
+ mvb_rows(scores, E, &l->gate_w, x, dim, n);
+ for (int r = 0; r < n; r++)
+ moe_gate(m, l, E, topk, scores + (size_t)r * E, chosen + r * topk, weights + r * topk);
+ free(scores);
+ moe_vk_block(m, l, cache, layer, topk, x, n, out, chosen, weights, with_shared);
+ free(chosen); free(weights);
+ return;
+ }
+#endif
for (int r0 = 0; r0 < n; r0 += MOE_ROW_CHUNK) {
int rows = n - r0 < MOE_ROW_CHUNK ? n - r0 : MOE_ROW_CHUNK;
@@ -4225,10 +4484,16 @@ static void vk_report(void) {
* device); the activation is the clamped SwiGLU. The history (.coli_usage, beside
* the container or COLI_USAGE) is kept only while the tier is on; it gives the warm
* start, read straight from the container in parallel. */
+static int vk_placed(const void *data) { /* on the device already (the dense weights on the device only) */
+ if (!g_v41_dho) return 0;
+ V41Home *h = v41_home_of(data);
+ return h && h->gone;
+}
static size_t vk_w8_bytes(const W8 *w) { /* as the device holds it: fmt 12, an f32 per 32 inputs */
+ if (vk_placed(w->q)) return 0;
return w->q ? (size_t)w->O * w->I + (size_t)w->O * ((w->I + FP8_TILE - 1) / FP8_TILE) * 4 : 0;
}
-static size_t vk_wb_bytes(const WB *w) { return w->w ? (size_t)w->O * w->I * 2 : 0; }
+static size_t vk_wb_bytes(const WB *w) { return w->w && !vk_placed(w->w) ? (size_t)w->O * w->I * 2 : 0; }
/* The trunk the dense hook would put on the device, for the tier's budget. */
static size_t vk_dense_bytes(Model *m) {
if (!coli_vk_dense()) return 0;
@@ -4262,7 +4527,8 @@ static void vk_tier_start(Model *m, const char *snap, int cap) {
.gate_up = f, .down = f, .act = VKT_ACT_SWIGLU, .act_limit = c->swiglu_limit,
.max_rows = MOE_ROW_CHUNK * c->n_activated,
.ram_reserve = expert * (size_t)cap * (size_t)c->n_layers,
- .dense_bytes = vk_dense_bytes(m), .in_ram = vk_in_ram, .ram_ctx = m};
+ .dense_bytes = vk_dense_bytes(m), .in_ram = vk_in_ram, .ram_ctx = m,
+ .load = vk_load, .release = vk_unhold, .load_ctx = m, .load_batch = vk_load_batch};
rt_init("deepseek_v41", c->n_layers, c->n_routed);
rt_drop_row(c->n_layers); /* no MTP row: the DSpark stages stay on the CPU */
const char *up = getenv("COLI_USAGE");
@@ -4337,8 +4603,17 @@ int main(int argc, char **argv) {
#ifdef COLI_VULKAN
/* After the weights, as in glm53: the device is an option, never a requirement.
* The matrices go up on their first multiply (see vk_mul), from this thread. */
- g_vk_thread = pthread_self();
- g_vk_ready = coli_vk_init_env_tier("deepseek_v41", vkt_wanted() && c->n_routed > 0);
+ if (!g_vk_opened) { /* opened while the weights loaded when they may live on the device only */
+ g_vk_thread = pthread_self();
+ g_vk_ready = coli_vk_init_env_tier("deepseek_v41", vkt_wanted() && c->n_routed > 0);
+ const char *keep = getenv("COLI_VK_DENSE_HOST");
+ if (g_vk_ready && keep && *keep && atoi(keep) != 0)
+ coli_vk_dense_host_decide("deepseek_v41", v41c_decide(&m) != 0 || coli_vk_dense(), v41_dho_bytes(&m));
+ }
+ if (g_v41_dho)
+ coli_vk_dense_host_placed("deepseek_v41", g_v41_dho_kept
+ ? "the embedding, the head, the router, norms and mixers, DSpark's stages, the vision tower; and matrices the device refused"
+ : "the embedding, the head, the router, norms and mixers, DSpark's stages, the vision tower");
v41c_start(&m); /* COLI_VK_CHAIN: the trunk on the device, before the tier sizes itself */
vk_tier_start(&m, snap, cap);
v41c_atexit(); /* after the tier's: the chain goes before the device */
diff --git a/c/deepseek_v41_chain.h b/c/deepseek_v41_chain.h
index f9662f6c..b1d322b4 100644
--- a/c/deepseek_v41_chain.h
+++ b/c/deepseek_v41_chain.h
@@ -40,6 +40,17 @@
* fakes it): the forward runs again on the CPU from its input, which the chain leaves
* untouched until every chunk is through, and the CPU runs from there.
*
+ * Past the device's budget (vk_kvsplit.h) each kv_source layer's compressed rows keep
+ * only some blocks on the device: a window over the newest rows and the blocks the
+ * indexer picks most; the rest stay in the host's ckv, which holds every row. A layer's
+ * sparse attention stages only selected missing rows in bounded device scratch and
+ * reads them alongside resident rows in the original list order (vkc_kv_ds), preserving
+ * the complete device result regardless of residency. New compressed rows go through a
+ * scratch into their slots, and come back into the host's ckv as soon as their layer's
+ * frame is through, so the host part of a later chunk finds every row it reads there.
+ * The window ring and the index keys stay whole on the device (every row reads them
+ * every step). Below the budget the mirrors are whole, as above.
+ *
* The chain declines (the CPU runs the forward, the watermarks follow) under V41_TRACE,
* for prompts only when COLI_VK_CHAIN=2 and the forward has two rows or fewer, and for a
* model its shaders or this file do not take: head_dim above 1024, a window plus top-k
@@ -48,6 +59,7 @@
* of an earlier index source with another ratio (or of none), a candidate-mask consumer
* with another ratio than the candidate source. */
#include "vk_chain.h"
+#include "vk_kvsplit.h"
typedef struct {
int ok, failed;
@@ -66,18 +78,28 @@ typedef struct {
float *host_routed;
unsigned long long forwards;
double host_ms;
+ VkcKvSplit ks; /* the compressed rows split past the device's budget (ks.on) */
+ int *sli, nsl, rmin, planned; /* per layer its table (-1 none), tables, the least ratio */
+ VkcBuf *cnew, *cdn; /* a chunk's new compressed rows, and their copy for the host */
} V41Chain;
static V41Chain *g_v41c;
static int g_vk_chain = 0;
static int g_v41c_inited = 0;
+/* The window ring's extra rows at setup: COLI_VK_CHAIN_ROWS when set, else 512; a forward
+ * whose chunk is bigger grows the rings first (v41c_grow_win). */
static int v41c_rows(void) {
const char *e = getenv("COLI_VK_CHAIN_ROWS");
- int v = e && *e ? atoi(e) : 512;
+ int v = e && *e && strcmp(e, "auto") ? atoi(e) : 512;
return v < 1 ? 1 : v > 65535 ? 65535 : v;
}
-static int v41c_res(VkcBuf **b, size_t floats, int kind) { return vkc_reserve(b, (floats ? floats : 1) * sizeof(float), kind); }
+/* v41c_res counts instead of reserving while g_v41c_count >= 0 (the chunk's sizing) */
+static long long g_v41c_count = -1;
+static int v41c_res(VkcBuf **b, size_t floats, int kind) {
+ if (g_v41c_count >= 0) { g_v41c_count += (long long)(floats ? floats : 1) * (long long)sizeof(float); return 1; }
+ return vkc_reserve(b, (floats ? floats : 1) * sizeof(float), kind);
+}
/* The device copies the per-matrix path keeps (vk_entry), uploaded here if it has not:
* fp8 as fmt 12 with each 32x32 tile's scale written out for its rows, bf16 as fmt 11. */
@@ -86,6 +108,7 @@ static ColiVkTensor *v41c_w8(const W8 *w) {
VkEntry *e = vk_entry(w->q, 12, w->O, w->I);
if (!e || e->refused) return NULL;
if (e->t) return e->t;
+ v41_dho_host(w->q); /* not placed this way (device only): its rows back first */
int groups = (w->I + FP8_TILE - 1) / FP8_TILE;
float *scales = malloc((size_t)w->O * groups * sizeof(float));
if (!scales) return NULL;
@@ -100,6 +123,7 @@ static ColiVkTensor *v41c_wb(const WB *w) {
VkEntry *e = vk_entry(w->w, 11, w->O, w->I);
if (!e || e->refused) return NULL;
if (e->t) return e->t;
+ v41_dho_host(w->w);
return coli_vk_tensor_ensure(&e->t, w->w, NULL, 11, w->I, w->O, 0) ? e->t : NULL;
}
@@ -112,6 +136,7 @@ static void v41c_lower(Model *m, int pos) {
for (int i = 0; i < m->c.n_layers; i++) {
int r = m->c.compress_ratio[i];
if (m->c.kv_source[i] && r > 0 && ch->kv_valid[i] > pos / r) ch->kv_valid[i] = pos / r;
+ if (m->c.kv_source[i] && r > 0 && ch->sli[i] >= 0) vkc_kv_lower(&ch->ks, ch->sli[i], pos / r);
}
}
static void v41c_cpu_step(Model *m, int start_pos) { v41c_lower(m, start_pos); } /* a CPU forward from start_pos */
@@ -167,8 +192,12 @@ static int v41c_setup(Model *m) {
ch->fna = calloc(L, sizeof(void *)); ch->fnf = calloc(L, sizeof(void *));
ch->win = calloc(L, sizeof(void *)); ch->ckv = calloc(L, sizeof(void *));
ch->ikey = calloc(L, sizeof(void *)); ch->ring = calloc(L, sizeof(void *));
- ch->kv_valid = calloc(L, sizeof(int)); ch->ccap = calloc(L, sizeof(int));
- if (!ch->fna || !ch->fnf || !ch->win || !ch->ckv || !ch->ikey || !ch->ring || !ch->kv_valid || !ch->ccap) return 0;
+ ch->kv_valid = calloc(L, sizeof(int)); ch->ccap = calloc(L, sizeof(int)); ch->sli = malloc(L * sizeof(int));
+ if (!ch->fna || !ch->fnf || !ch->win || !ch->ckv || !ch->ikey || !ch->ring || !ch->kv_valid || !ch->ccap || !ch->sli) return 0;
+ for (int i = 0; i < L; i++) { /* the split's tables: one per layer that makes compressed rows */
+ ch->sli[i] = c->kv_source[i] && c->compress_ratio[i] > 0 ? ch->nsl++ : -1;
+ if (ch->sli[i] >= 0 && (ch->rmin == 0 || c->compress_ratio[i] < ch->rmin)) ch->rmin = c->compress_ratio[i];
+ }
g_v41c = ch;
ch->W = c->window; ch->K = c->index_topk > 0 ? c->index_topk : 0; ch->rmax = 1;
for (int i = 0; i < L; i++) if (c->kv_source[i] && c->compress_ratio[i] > ch->rmax) ch->rmax = c->compress_ratio[i];
@@ -232,6 +261,34 @@ static int v41c_setup(Model *m) {
return 1;
}
+/* The compressed rows of the kv_source layers, whole or split: planned when the whole
+ * mirrors would grow (and at the first forward); once split, the split stays. ctx: the
+ * positions the forward reaches. 0 = the device refused the split's rows. */
+static int v41c_plan(V41Chain *ch, Model *m, int ctx) {
+ Cfg *c = &m->c;
+ if (!ch->nsl || ch->ks.on) return 1;
+ int L = c->n_layers, hd = c->head_dim, tcap = 1, grow = !ch->planned;
+ size_t need = 0, held = 0;
+ for (int i = 0; i < L; i++) {
+ if (ch->sli[i] < 0) continue;
+ int r = c->compress_ratio[i], host = c->max_positions / r, want = ctx / r > 0 ? ctx / r : 1, cap = ch->ccap[i];
+ if (host > tcap) tcap = host;
+ if (cap < want) { cap = 64; while (cap < want) cap *= 2; if (cap > host) cap = host; grow = 1; }
+ need += (size_t)cap * hd * sizeof(float); held += (size_t)ch->ccap[i] * hd * sizeof(float);
+ }
+ if (!grow) return 1;
+ ch->planned = 1;
+ /* a window of half the slots, the rest pinned by the indexer's picks; the forward's
+ * chunks shrink to what the window takes (v41c_forward) */
+ if (vkc_kv_plan_need(&ch->ks, "deepseek_v41", ch->nsl, (size_t)hd * sizeof(float), tcap, 1, 1, held, need) != 2) return 1;
+ for (int i = 0; i < L; i++) {
+ if (ch->sli[i] < 0) continue;
+ vkc_free(ch->ckv[i]);
+ if (!(ch->ckv[i] = vkc_buf((size_t)ch->ks.rows * hd * sizeof(float), VKC_DEV))) return 0;
+ }
+ return 1;
+}
+
/* the compressed rows and index keys of a kv_source layer: room for `rows` rows */
static int v41c_cache(V41Chain *ch, Model *m, int i, int rows) {
Cfg *c = &m->c;
@@ -240,8 +297,11 @@ static int v41c_cache(V41Chain *ch, Model *m, int i, int rows) {
int cap = 64; while (cap < rows) cap *= 2;
if (cap > host) cap = host;
if (cap < rows) return 0;
- vkc_free(ch->ckv[i]); vkc_free(ch->ikey[i]);
- ch->ckv[i] = vkc_buf((size_t)cap * c->head_dim * sizeof(float), VKC_DEV);
+ if (!ch->ks.on) { /* under the split the compressed rows keep their ks.rows */
+ vkc_free(ch->ckv[i]);
+ ch->ckv[i] = vkc_buf((size_t)cap * c->head_dim * sizeof(float), VKC_DEV);
+ }
+ vkc_free(ch->ikey[i]);
ch->ikey[i] = vkc_buf((size_t)cap * c->index_head_dim * sizeof(float), VKC_DEV);
ch->kv_valid[i] = 0; ch->ccap[i] = 0;
if (!ch->ckv[i] || !ch->ikey[i]) return 0;
@@ -249,6 +309,38 @@ static int v41c_cache(V41Chain *ch, Model *m, int i, int rows) {
return 1;
}
+static int v41c_scratch(V41Chain *ch, Model *m, int rows, int ctx);
+/* Prompt rows per chunk (vkc_chunk_rows, decided at the first forward, after the tier
+ * filled): the chain's scratch a row at the model's context (the indexer's scores and
+ * mask) at the forward's context, the window ring's row, and the routed experts' outputs. */
+static int v41c_chunk_rows(V41Chain *ch, Model *m, int ctx) {
+ if (!vkc_chunk_auto()) return v41c_rows();
+ size_t wcap = ch->wcap;
+ g_v41c_count = 0; v41c_scratch(ch, m, 1, ctx); long long b1 = g_v41c_count;
+ g_v41c_count = 0; v41c_scratch(ch, m, 2, ctx); long long b2 = g_v41c_count;
+ g_v41c_count = -1; ch->wcap = wcap;
+ const Cfg *c = &m->c;
+ size_t row = (size_t)(b2 - b1) + (size_t)c->n_layers * c->head_dim * sizeof(float) +
+ (size_t)(2 * c->n_activated + 1) * c->dim * sizeof(float);
+ /* The CPU expert path gathers one expert's rows and keeps its gate/up and
+ * output beside the rank-order contribution buffer. */
+ row += 2 * ((size_t)c->dim + c->moe_inter) * sizeof(float);
+ if (ch->ks.on) row += (size_t)(ch->W + ch->K) * sizeof(int);
+ return vkc_chunk_rows("deepseek_v41", row);
+}
+/* The window rings for chunks of `rows`: W + rows rows each (a chunk never overwrites a
+ * row one of its earlier queries reads); a ring that grows is mirrored again. */
+static int v41c_grow_win(V41Chain *ch, Model *m, int rows) {
+ if (ch->W + rows <= ch->Wd) return 1;
+ const Cfg *c = &m->c;
+ int Wd = ch->W + rows;
+ for (int i = 0; i < c->n_layers; i++) {
+ vkc_free(ch->win[i]);
+ if (!(ch->win[i] = vkc_buf((size_t)Wd * c->head_dim * sizeof(float), VKC_DEV))) return 0;
+ }
+ ch->Wd = Wd; ch->win_valid = 0;
+ return 1;
+}
static int v41c_scratch(V41Chain *ch, Model *m, int rows, int ctx) {
Cfg *c = &m->c;
int D = c->dim, H = c->hc_mult, HD = H * D, nm = (2 + H) * H, hr = 2 * H + H * H, nh = c->n_heads, hd = c->head_dim;
@@ -270,7 +362,13 @@ static int v41c_scratch(V41Chain *ch, Model *m, int rows, int ctx) {
v41c_res(&ch->us, r * c->moe_inter, VKC_DEV) && v41c_res(&ch->hs, r * c->moe_inter, VKC_DEV) &&
v41c_res(&ch->ds, r * D, VKC_DEV) && v41c_res(&ch->h2d, r * D, VKC_DOWN) && v41c_res(&ch->mh, r * T * D, VKC_DOWN) &&
v41c_res(&ch->xd, r * HD, VKC_DOWN) && v41c_res(&ch->hpd, r * hr, VKC_DOWN) && v41c_res(&ch->routed, r * D, VKC_UP);
- if (!ok) return 0;
+ if (ok && ch->ks.on) { /* the split: a chunk's new compressed rows, the two parts of an attention */
+ size_t nc = r / (size_t)ch->rmin + 2;
+ if (g_v41c_count >= 0)
+ g_v41c_count += (long long)((2 * r * hd * sizeof(float) + ch->rmin - 1) / ch->rmin);
+ else ok = v41c_res(&ch->cnew, nc * hd, VKC_DEV) && v41c_res(&ch->cdn, nc * hd, VKC_DOWN);
+ }
+ if (!ok || g_v41c_count >= 0) return ok; /* counting: the buffers only */
if (ch->rows < rows) {
float *hrt = realloc(ch->host_routed, r * D * sizeof(float));
if (!hrt) return 0;
@@ -337,24 +435,32 @@ static int v41c_attention(V41Chain *ch, Model *m, int i, const V41Fwd *f) {
ok = vkc_copy_regions(ch->win[i], ch->kv, rg, nr);
free(rg);
if (ok && r > 0 && c->kv_source[i]) { /* the compressor, its latents, the index keys */
- int g0 = pb / r, np = (pb + nr) / r - g0, ihd = c->index_head_dim;
+ int g0 = pb / r, np = (pb + nr) / r - g0, ihd = c->index_head_dim, sp = ch->ks.on;
+ VkcBuf *cb = sp ? ch->cnew : ch->ckv[i]; /* the split: the new rows through a scratch */
+ size_t co = sp ? 0 : (size_t)g0 * hd;
ok = vkc_matmul(v41c_wb(&l->comp_wkv), ch->nrm, 0, ch->craw, 0, nr);
- if (ok && r == 1) ok = v41c_norm(ch->craw, 0, hd, ch->o_cn[i], ch->ckv[i], (size_t)g0 * hd, hd, nr, hd, c->norm_eps);
+ if (ok && r == 1) ok = v41c_norm(ch->craw, 0, hd, ch->o_cn[i], cb, co, hd, nr, hd, c->norm_eps);
else if (ok) {
- VkcDsComp cp = {nr, pb, r, hd, hd, 0, 0, hd, 0, hd, -1, 0, hd, 0};
+ VkcDsComp cp = {nr, pb, r, hd, hd, 0, 0, hd, 0, hd, -1, sp ? -g0 * hd : 0, hd, 0};
ok = vkc_matmul(v41c_wb(&l->comp_wgate), ch->nrm, 0, ch->cscr, 0, nr) &&
- vkc_dsv4_compress(ch->craw, ch->cscr, ch->ring[i], NULL, ch->ckv[i], &cp) &&
- (np == 0 || v41c_norm(ch->ckv[i], (size_t)g0 * hd, hd, ch->o_cn[i], ch->ckv[i], (size_t)g0 * hd, hd, np, hd, c->norm_eps));
+ vkc_dsv4_compress(ch->craw, ch->cscr, ch->ring[i], NULL, cb, &cp) &&
+ (np == 0 || v41c_norm(cb, co, hd, ch->o_cn[i], cb, co, hd, np, hd, c->norm_eps));
if (ok && f->spec && f->praw) /* a verify's raw rows, for its undo rows */
ok = vkc_copy(ch->pull, f->praw[i], ch->craw, 0, (size_t)nr * hd) &&
vkc_copy(ch->pull, f->praw[i] + (size_t)nr * hd, ch->cscr, 0, (size_t)nr * hd);
}
if (ok && np > 0) {
int bc = f->csB + (g0 * r - (pb - ch->rmax + 1)) * rd; /* a latent rotates at its group's first position */
- ok = vkc_matmul(v41c_wb(&l->idx_wk), ch->ckv[i], (size_t)g0 * hd, ch->ikey[i], (size_t)g0 * ihd, np) &&
+ ok = vkc_matmul(v41c_wb(&l->idx_wk), cb, co, ch->ikey[i], (size_t)g0 * ihd, np) &&
v41c_norm(ch->ikey[i], (size_t)g0 * ihd, ihd, ch->o_ikn[i], ch->ikey[i], (size_t)g0 * ihd, ihd, np, ihd, c->norm_eps) &&
v41c_rope(ch, m, ch->ikey[i], np, 1, g0 * ihd + ihd - rd, ihd, 0, bc, r * rd, 0) &&
- v41c_rope(ch, m, ch->ckv[i], np, 1, g0 * hd + hd - rd, hd, 0, bc, r * rd, 0);
+ v41c_rope(ch, m, cb, np, 1, (int)co + hd - rd, hd, 0, bc, r * rd, 0);
+ if (ok && sp) { /* into their slots, and down for the host's ckv once the frame is through */
+ VkcKvPart pt = {1, hd, l->ckv, 0, ch->ckv[i], 0};
+ ok = vkc_kv_store(&ch->ks, ch->sli[i], &pt, cb, 0, (size_t)hd, 0, g0, np) &&
+ vkc_copy(ch->cdn, 0, cb, 0, (size_t)np * hd);
+ vkc_kv_done(&ch->ks, ch->sli[i], g0 + np);
+ }
}
}
if (ok && r > 0 && c->index_source[i]) { /* the indexer: this layer's selection */
@@ -384,7 +490,15 @@ static int v41c_attention(V41Chain *ch, Model *m, int i, const V41Fwd *f) {
1.0f / sqrtf((float)hd)};
int og = c->o_groups, ol = c->o_lora, pg = qrow / og;
VkcHgemv g = {0, nr, og, ol, ol, 0, 0, qrow, pg, 0, og * ol, ol, 0, 0, 0};
- return vkc_dsv4_attn(ch->q, ch->win[i], r > 0 ? ch->ckv[c->index_owner[i]] : NULL, ch->list, ch->prm, ch->heads, &a) &&
+ if (ch->ks.on && r > 0) { /* stage selected cold rows, preserving the original sparse attention order */
+ int own = c->index_owner[i], li = ch->sli[own];
+ VkcKvPart pt = {1, hd, m->L[own].ckv, 0, ch->ckv[own], 0};
+ VkcKvDs d = {&ch->ks, li, ch->q, ch->win[i], ch->ckv[own], ch->prm, ch->list, ch->heads, nr, nh, hd, a.cnt, 0, LR,
+ ch->Wd, 0, 0, qrow, (int)ch->o_sink[i], 0, 0, qrow, (pb + nr) / r, a.scale, m->L[own].ckv};
+ /* the blocks pinned since the chunk began, and the table, before this read */
+ ok = vkc_kv_push(&ch->ks, li, &pt, 1, ch->ks.t[li].valid) && vkc_kv_ds(&d);
+ } else ok = vkc_dsv4_attn(ch->q, ch->win[i], r > 0 ? ch->ckv[c->index_owner[i]] : NULL, ch->list, ch->prm, ch->heads, &a);
+ return ok &&
v41c_rope(ch, m, ch->heads, nr * nh, nh, hd - rd, qrow, hd, tcs, rd, 1) &&
vkc_mla_hgemv(v41c_w8(&l->wo_a), ch->heads, ch->grp, NULL, &g) &&
vkc_matmul(v41c_w8(&l->wo_b), ch->grp, 0, ch->br, 0, nr);
@@ -428,10 +542,24 @@ static int v41c_forward(Model *m, float *h, float *pre_mix, int n, int start, in
int L = c->n_layers, D = c->dim, H = c->hc_mult, HD = H * D, hr = 2 * H + H * H, hd = c->head_dim, rd = c->rope_dim;
int W = ch->W, LR = W + ch->K, T = m->spec.active ? c->n_spec_targets : 0, E = start + n;
if (spec && n > W) return 0; /* the undo rows: one window at most */
- int CH = v41c_rows(), rows;
+ if (!v41c_plan(ch, m, E)) {
+ fprintf(stderr, "[VK] deepseek_v41 chain: device memory for the split's compressed rows refused; the CPU runs the layers\n");
+ ch->failed = 1; g_vk_chain = 0;
+ return 0;
+ }
+ int CH = v41c_chunk_rows(ch, m, E), rows;
{ int64_t lim = ((int64_t)32 << 20) / (E > 1 ? E : 1); if (lim < CH) CH = lim < 1 ? 1 : (int)lim; }
+ if (ch->ks.on) { /* a chunk's new compressed rows fit the split's window */
+ int mx = ch->rmin == 1 ? ch->ks.chunk : (ch->ks.chunk - 1) * ch->rmin;
+ if (CH > mx) CH = mx < 1 ? 1 : mx;
+ }
if (spec && n > CH) return 0; /* a verify is one chunk */
rows = n < CH ? n : CH;
+ if (!v41c_grow_win(ch, m, rows)) {
+ fprintf(stderr, "[VK] deepseek_v41 chain: device memory for window rings of %d rows refused; the CPU runs the layers\n", rows);
+ ch->failed = 1; g_vk_chain = 0;
+ return 0;
+ }
V41Fwd f = {start, n, 0, 0, spec, rows * rd, NULL, NULL};
/* whose index keys each index source reads, per row (indexer_run, published_owner) */
f.keyl = malloc((size_t)L * n * sizeof(int));
@@ -486,7 +614,7 @@ static int v41c_forward(Model *m, float *h, float *pre_mix, int n, int start, in
pc[i] = pn; pn += pw;
int r = c->compress_ratio[i];
if (!c->kv_source[i]) continue;
- pn += (size_t)(E / r - start / r) * (hd + c->index_head_dim);
+ pn += (size_t)(E / r - start / r) * ((ch->ks.on ? 0 : hd) + c->index_head_dim); /* split: the rows came back per chunk */
if (r > 1) { pn += (size_t)2 * r * hd; if (spec) { f.praw[i] = pn; pn += (size_t)2 * n * hd; } }
}
ok = ok && pc && v41c_scratch(ch, m, rows, E) && v41c_res(&ch->pull, pn, VKC_DOWN) && (!T || m->main_hidden);
@@ -546,13 +674,21 @@ static int v41c_forward(Model *m, float *h, float *pre_mix, int n, int start, in
if (!c->kv_source[i]) continue;
const Layer *l = &m->L[i];
int r = c->compress_ratio[i], ihd = c->index_head_dim, t0 = ch->kv_valid[i], t1 = need[i];
- if (t1 > t0) ok = vkc_write(ch->ckv[i], (size_t)t0 * hd, l->ckv + (size_t)t0 * hd, (size_t)(t1 - t0) * hd * sizeof(float)) &&
+ if (t1 > t0) ok = (ch->ks.on || vkc_write(ch->ckv[i], (size_t)t0 * hd, l->ckv + (size_t)t0 * hd, (size_t)(t1 - t0) * hd * sizeof(float))) &&
vkc_write(ch->ikey[i], (size_t)t0 * ihd, l->ikey + (size_t)t0 * ihd, (size_t)(t1 - t0) * ihd * sizeof(float));
if (ok && r > 1) ok = vkc_write(ch->ring[i], 0, l->cstate_kv, (size_t)r * hd * sizeof(float)) &&
vkc_write(ch->ring[i], (size_t)r * hd, l->cstate_score, (size_t)r * hd * sizeof(float));
}
if (!ok) goto lost;
}
+ for (int i = 0; ch->ks.on && i < L && ok; i++) { /* the split: each table's window over this chunk's new rows */
+ if (ch->sli[i] < 0) continue;
+ int r = c->compress_ratio[i], g0 = pb / r;
+ VkcKvPart pt = {1, hd, m->L[i].ckv, 0, ch->ckv[i], 0};
+ vkc_kv_place(&ch->ks, ch->sli[i], g0, (pb + nr) / r - g0);
+ ok = vkc_kv_push(&ch->ks, ch->sli[i], &pt, 1, g0);
+ }
+ if (!ok) goto lost;
int pending = 0;
for (int i = 0; i < L && ok; i++) {
Layer *l = &m->L[i];
@@ -571,9 +707,17 @@ static int v41c_forward(Model *m, float *h, float *pre_mix, int n, int start, in
ok = ok && v41c_pre(ch, m, ch->fna[i], ch->o_hca[i], ch->hpa, ch->hpf, ch->o_an[i], nr) &&
v41c_attention(ch, m, i, &f) && v41c_post(ch, m, ch->hpa, nr) &&
v41c_pre(ch, m, ch->fnf[i], ch->o_hcf[i], ch->hpf, ch->hpa, ch->o_fn[i], nr) &&
- vkc_copy(ch->h2d, 0, ch->nrm, 0, (size_t)nr * D) && vkc_submit(1); /* A1 */
+ vkc_copy(ch->h2d, 0, ch->nrm, 0, (size_t)nr * D) && vkc_submit(0); /* A1 */
+ /* while it runs, the tier loads the experts this layer will likely stream (a
+ * big prompt chunk only) */
+ if (ok) vkt_stream_prefetch(i, nr);
+ ok = ok && vkc_finish();
m->t_attn += now_s() - ta;
if (!ok) break;
+ if (ch->ks.on && ch->sli[i] >= 0) { /* this chunk's new compressed rows into the host's ckv */
+ int r = c->compress_ratio[i], g0 = pb / r, np = (pb + nr) / r - g0;
+ if (np > 0) memcpy(l->ckv + (size_t)g0 * hd, vkc_ptr(ch->cdn), (size_t)np * hd * sizeof(float));
+ }
/* A2: the shared expert, while the host computes the routed experts */
ok = vkc_begin() && v41c_shared(ch, m, l, nr) && vkc_submit(0);
double t1 = now_s();
@@ -598,11 +742,11 @@ static int v41c_forward(Model *m, float *h, float *pre_mix, int n, int start, in
ok = vkc_copy_regions(ch->pull, ch->win[i], rg, nq);
free(rg);
if (!ok || !c->kv_source[i]) continue;
- int r = c->compress_ratio[i], g0 = start / r, np = E / r - g0, ihd = c->index_head_dim;
+ int r = c->compress_ratio[i], g0 = start / r, np = E / r - g0, ihd = c->index_head_dim, cw = ch->ks.on ? 0 : hd;
size_t at = pc[i] + pw;
- ok = vkc_copy(ch->pull, at, ch->ckv[i], (size_t)g0 * hd, (size_t)np * hd) &&
- vkc_copy(ch->pull, at + (size_t)np * hd, ch->ikey[i], (size_t)g0 * ihd, (size_t)np * ihd) &&
- (r <= 1 || vkc_copy(ch->pull, at + (size_t)np * (hd + ihd), ch->ring[i], 0, (size_t)2 * r * hd));
+ ok = (cw == 0 || vkc_copy(ch->pull, at, ch->ckv[i], (size_t)g0 * hd, (size_t)np * hd)) &&
+ vkc_copy(ch->pull, at + (size_t)np * cw, ch->ikey[i], (size_t)g0 * ihd, (size_t)np * ihd) &&
+ (r <= 1 || vkc_copy(ch->pull, at + (size_t)np * (cw + ihd), ch->ring[i], 0, (size_t)2 * r * hd));
}
}
ok = ok && vkc_submit(1);
@@ -634,10 +778,10 @@ static int v41c_forward(Model *m, float *h, float *pre_mix, int n, int start, in
l->window_pos[slot] = p;
}
if (!c->kv_source[i]) continue;
- int r = c->compress_ratio[i], g0 = start / r, np = E / r - g0, ihd = c->index_head_dim;
+ int r = c->compress_ratio[i], g0 = start / r, np = E / r - g0, ihd = c->index_head_dim, cw = ch->ks.on ? 0 : hd;
const float *src = pl + pc[i] + pw;
- memcpy(l->ckv + (size_t)g0 * hd, src, (size_t)np * hd * sizeof(float));
- memcpy(l->ikey + (size_t)g0 * ihd, src + (size_t)np * hd, (size_t)np * ihd * sizeof(float));
+ if (cw) memcpy(l->ckv + (size_t)g0 * hd, src, (size_t)np * hd * sizeof(float)); /* split: back per chunk */
+ memcpy(l->ikey + (size_t)g0 * ihd, src + (size_t)np * cw, (size_t)np * ihd * sizeof(float));
if (r <= 1) continue;
if (save) { /* the groups' undo rows, as compressor_run's decode path writes them */
const float *raw = pl + f.praw[i];
@@ -650,7 +794,7 @@ static int v41c_forward(Model *m, float *h, float *pre_mix, int n, int start, in
memcpy(l->cstate_score + (size_t)slot * hd, raw + (size_t)(n + t) * hd, (size_t)hd * sizeof(float));
}
}
- const float *ring = src + (size_t)np * (hd + ihd);
+ const float *ring = src + (size_t)np * (cw + ihd);
memcpy(l->cstate_kv, ring, (size_t)r * hd * sizeof(float));
memcpy(l->cstate_score, ring + (size_t)r * hd, (size_t)r * hd * sizeof(float));
}
@@ -679,15 +823,22 @@ static void v41c_report(void) {
fprintf(stderr, "[VK] deepseek_v41 chain: %llu forwards, %llu frames (%llu ops, %llu matmuls, %llu tiled GEMM), "
"%.1f ms waiting for the device, %.1f ms of routed experts on the host, %.1f MiB on the device\n",
ch->forwards, st.frames, st.ops, st.matmuls, st.gemms, st.wait_ms, ch->host_ms, st.dev_bytes / 1048576.0);
+ vkc_kv_report(&ch->ks);
vkc_prof_print();
}
/* COLI_VK_CHAIN at startup, before the tier sizes itself (the trunk's device copies count
* as used): the decision, the pipelines, the tensors. */
+static int g_v41c_decision = -1;
+static int v41c_decide(Model *m) { /* once: early when the trunk may live on the device only */
+ if (g_v41c_decision >= 0) return g_v41c_decision;
+ if (!g_vk_ready) return 0;
+ int tier_on = vkt_wanted() && m->c.n_routed > 0;
+ return g_v41c_decision = coli_vk_chain_decide("deepseek_v41", tier_on, COLI_VK_CHAIN_UNMEASURED);
+}
static void v41c_start(Model *m) {
if (!g_vk_ready) return;
- int tier_on = vkt_wanted() && m->c.n_routed > 0;
- int on = coli_vk_chain_decide("deepseek_v41", tier_on, COLI_VK_CHAIN_UNMEASURED);
+ int on = v41c_decide(m);
const char *no = NULL;
if (on && !(g_v41c_inited = vkc_init())) no = "the chain's pipelines did not come up";
if (on && !no && !(vkc_mla_ready() && vkc_mhc_ready() && vkc_dsv4_ready()))
diff --git a/c/deepseek_v4_chain.h b/c/deepseek_v4_chain.h
index e59e0ced..2686e943 100644
--- a/c/deepseek_v4_chain.h
+++ b/c/deepseek_v4_chain.h
@@ -58,6 +58,18 @@
* fakes it): the forward runs again on the CPU from its input, which the chain leaves
* untouched until every chunk is through, and the CPU runs from there.
*
+ * Past the device's budget (vk_kvsplit.h) each compressed layer keeps only some blocks of
+ * its compressed rows on the device: a window over the newest rows and the blocks its
+ * list reads most; the rest stay in the host's compressed rows, which hold every row. The
+ * sparse attention stages only selected missing rows in bounded device scratch and
+ * reads them alongside resident rows in the original list order (vkc_kv_ds). This
+ * preserves one global score maximum and V4's bf16 weights and output exactly. A
+ * chunk's new compressed rows go through a scratch into their slots and come back into
+ * the host's rows (reserved for the forward at its start) as soon as their layer's frame
+ * is through, so the host part of a later chunk finds every row it reads; the count is
+ * set at the end as before. The window ring and the indexer's keys stay whole on the
+ * device. Below the budget the mirrors are whole, as above.
+ *
* The chain declines (the CPU runs, the watermarks follow) when the dense layers are not
* resident (a low-memory plan reloads them per forward; the --oracle path's own copies
* too), under the CUDA tier, for prompts only when COLI_VK_CHAIN=2 and the forward has
@@ -67,6 +79,7 @@
* (the Hadamard transform) or above 4096 floats, a top-k above 4096, more than 8
* streams. */
#include "vk_chain.h"
+#include "vk_kvsplit.h"
typedef struct {
int ratio, idx, kind; /* compress ratio (0 none), an indexer, its list */
@@ -97,26 +110,37 @@ typedef struct {
float *host_routed;
unsigned long long forwards;
double host_ms;
+ VkcKvSplit ks; /* the compressed rows split past the device's budget (ks.on) */
+ int *sli, nsl, rmin, planned; /* per layer its table (-1 none), tables, the least ratio */
+ VkcBuf *cnew, *cdn; /* a chunk's new compressed rows, and their copy for the host */
} V4Chain;
static V4Chain *g_v4c;
static int g_v4c_mode = 0; /* COLI_VK_CHAIN as decided */
static int g_v4c_inited = 0;
-/* Rows per chunk: the CPU's prefill chunk (V4_PREFILL_CHUNK, 128 at most), so the host's
- * MoE sees the batches the CPU block would; COLI_VK_CHAIN_ROWS lowers it. */
+/* Rows per chunk with COLI_VK_CHAIN_ROWS set (or V4_PREFILL_CHUNK): the CPU's prefill
+ * chunk (V4_PREFILL_CHUNK, 128 at most), so the host's MoE sees the batches the CPU block
+ * would; COLI_VK_CHAIN_ROWS lowers it. Unset (the chunk from the budget, vkc_chunk_auto):
+ * up to 8192 rows (v4c_chunk_rows), the MoE step taking the whole chunk; this is then the
+ * window rings' first size, grown by the forward that needs more (v4c_grow_win). */
static int v4c_rows(void) {
static int rows;
if (!rows) {
const char *c = getenv("V4_PREFILL_CHUNK"), *e = getenv("COLI_VK_CHAIN_ROWS");
int cw = c ? atoi(c) : 128;
if (cw < 1 || cw > 128) cw = 128;
- int v = e && *e ? atoi(e) : cw;
+ int v = e && *e && strcmp(e, "auto") ? atoi(e) : cw;
rows = v < 1 ? 1 : v > cw ? cw : v;
}
return rows;
}
-static int v4c_res(VkcBuf **b, size_t floats, int kind) { return vkc_reserve(b, (floats ? floats : 1) * sizeof(float), kind); }
+/* v4c_res counts instead of reserving while g_v4c_count >= 0 (the chunk's sizing) */
+static long long g_v4c_count = -1;
+static int v4c_res(VkcBuf **b, size_t floats, int kind) {
+ if (g_v4c_count >= 0) { g_v4c_count += (long long)(floats ? floats : 1) * (long long)sizeof(float); return 1; }
+ return vkc_reserve(b, (floats ? floats : 1) * sizeof(float), kind);
+}
static const void *v4c_val(const ColiDeepSeekV4LayerWeights *w, const char *suffix) {
char name[COLI_V4_MAX_TENSOR_NAME];
snprintf(name, sizeof name, "layers.%d.%s", w->plan.layer, suffix);
@@ -135,6 +159,7 @@ static ColiVkTensor *v4c_tensor(int fmt, const void *data, const float *scales,
if (e && e->refused) return NULL;
size_t bytes = (size_t)rows * columns * (fmt == 10 ? 4 : fmt == 11 ? 2 : 1);
if (!v4_vk_resident(data, bytes)) return NULL;
+ coli_v4_vk_host_ensure(data); /* a view the placement did not make (device only): its rows back first */
if (!e && !(e = v4_vk_insert(data, fmt, rows, columns))) return NULL;
const unsigned char *weights = data;
unsigned char *unpacked = NULL;
@@ -194,6 +219,7 @@ static void v4c_lower(int pos) {
for (int i = 0; i < ch->L; i++) {
V4cLayer *ly = &ch->ly[i];
if (ly->ratio > 0 && ly->kv_valid > pos / ly->ratio) ly->kv_valid = pos / ly->ratio;
+ if (ly->ratio > 0 && ch->sli[i] >= 0) vkc_kv_lower(&ch->ks, ch->sli[i], pos / ly->ratio);
if (ly->idx && ly->ik_valid > pos / 4) ly->ik_valid = pos / 4;
ly->ring_ok = ly->iring_ok = 0;
}
@@ -238,7 +264,10 @@ static int v4c_setup(ColiV4Engine *engine) {
}
}
V4Chain *ch = calloc(1, sizeof *ch);
- if (!ch || !(ch->ly = calloc((size_t)L, sizeof *ch->ly))) { free(ch); return 0; }
+ if (!ch || !(ch->ly = calloc((size_t)L, sizeof *ch->ly)) || !(ch->sli = malloc((size_t)L * sizeof(int)))) {
+ if (ch) free(ch->ly);
+ free(ch); return 0;
+ }
g_v4c = ch;
ch->engine = engine; ch->L = L; ch->D = c->hidden_size; ch->H = c->hc_mult; ch->HD = ch->H * ch->D;
ch->nm = (2 + ch->H) * ch->H; ch->hr = 2 * ch->H + ch->H * ch->H;
@@ -255,6 +284,8 @@ static int v4c_setup(ColiV4Engine *engine) {
ly->ratio = c->compress_ratios[i]; ly->idx = ly->ratio == 4;
if (ly->ratio > ch->rmax) ch->rmax = ly->ratio;
ly->crows = ly->ratio == 4 ? 8 : ly->ratio; ly->cproj = (ly->ratio == 4 ? 2 : 1) * ch->hd;
+ ch->sli[i] = ly->ratio > 0 ? ch->nsl++ : -1; /* the split's tables: one per compressed layer */
+ if (ly->ratio > 0 && (ch->rmin == 0 || ly->ratio < ch->rmin)) ch->rmin = ly->ratio;
if (ly->ratio > 0 && !ly->idx) {
int k = 1;
while (k < ch->nkind && ch->kratio[k] != ly->ratio) k++;
@@ -342,7 +373,7 @@ static int v4c_setup(ColiV4Engine *engine) {
/* room on the device for `rows` compressed rows (and keys) of layer i; growing drops them */
static int v4c_cache(V4Chain *ch, V4cLayer *ly, int rows) {
- if (ly->ratio > 0 && ly->ccap < rows) {
+ if (ly->ratio > 0 && ly->ccap < rows && !ch->ks.on) { /* under the split the rows keep their ks.rows */
int cap = 64; while (cap < rows) cap *= 2;
vkc_free(ly->ckv);
ly->ckv = vkc_buf((size_t)cap * ch->hd * sizeof(float), VKC_DEV);
@@ -359,6 +390,69 @@ static int v4c_cache(V4Chain *ch, V4cLayer *ly, int rows) {
return 1;
}
+static int v4c_scratch(V4Chain *ch, int rows, int E, int LR);
+/* Prompt rows per chunk (vkc_chunk_rows, decided at the first forward, after the tier
+ * filled; V4_PREFILL_CHUNK or COLI_VK_CHAIN_ROWS set: v4c_rows): the chain's scratch a
+ * row at the forward's context, the window rings' row and the routed experts' outputs. */
+static int v4c_chunk_rows(V4Chain *ch, int E, int LR) {
+ if (!vkc_chunk_auto() || getenv("V4_PREFILL_CHUNK")) return v4c_rows();
+ size_t wcap = ch->wcap;
+ g_v4c_count = 0; v4c_scratch(ch, 1, E, LR); long long b1 = g_v4c_count;
+ g_v4c_count = 0; v4c_scratch(ch, 2, E, LR); long long b2 = g_v4c_count;
+ g_v4c_count = -1; ch->wcap = wcap;
+ const ColiDeepSeekV4Config *c = &ch->engine->config;
+ size_t row = (size_t)(b2 - b1) + (size_t)ch->L * ch->hd * sizeof(float) +
+ (size_t)(2 * c->num_experts_per_tok + 1) * ch->D * sizeof(float);
+ if (ch->ks.on) row += (size_t)LR * sizeof(int);
+ return vkc_chunk_rows("deepseek_v4", row);
+}
+/* The window rings for chunks of `rows`: W + rows rows each; rings that grow are
+ * mirrored again (every slot stale). */
+static int v4c_grow_win(V4Chain *ch, int rows) {
+ if (ch->W + rows <= ch->Wd) return 1;
+ int Wd = ch->W + rows;
+ int *sp = realloc(ch->slot_pos, (size_t)Wd * sizeof(int));
+ if (!sp) return 0;
+ ch->slot_pos = sp;
+ for (int k = 0; k < Wd; k++) sp[k] = -1;
+ for (int i = 0; i < ch->L; i++) {
+ vkc_free(ch->ly[i].win);
+ if (!(ch->ly[i].win = vkc_buf((size_t)Wd * ch->hd * sizeof(float), VKC_DEV))) return 0;
+ }
+ ch->Wd = Wd; ch->win_valid = 0;
+ return 1;
+}
+/* The compressed rows, whole or split: planned when the whole mirrors would grow (and at
+ * the first forward); once split, the split stays. E: the positions the forward reaches.
+ * 0 = the device refused the split's rows. */
+static int v4c_plan(V4Chain *ch, int E) {
+ if (!ch->nsl || ch->ks.on) return 1;
+ int grow = !ch->planned, tcap = 1, hd = ch->hd;
+ size_t need = 0, held = 0;
+ for (int i = 0; i < ch->L; i++) {
+ V4cLayer *ly = &ch->ly[i];
+ if (ly->ratio <= 0) continue;
+ int want = E / ly->ratio + 1, cap = ly->ccap, host = ch->engine->config.max_position_embeddings / ly->ratio + 1;
+ if (host < want) host = want;
+ if (host > tcap) tcap = host;
+ if (cap < want) { cap = 64; while (cap < want) cap *= 2; grow = 1; }
+ need += (size_t)cap * hd * sizeof(float); held += (size_t)ly->ccap * hd * sizeof(float);
+ }
+ if (!grow) return 1;
+ ch->planned = 1;
+ /* a window over recent slots, with optional read-based pins; the forward's
+ * chunks shrink to what the window takes (v4c_forward) */
+ if (vkc_kv_plan_need(&ch->ks, "deepseek_v4", ch->nsl, (size_t)hd * sizeof(float), tcap, 1, 1, held, need) != 2) return 1;
+ for (int i = 0; i < ch->L; i++) {
+ V4cLayer *ly = &ch->ly[i];
+ if (ly->ratio <= 0) continue;
+ vkc_free(ly->ckv);
+ ly->ccap = 0; ly->kv_valid = 0;
+ if (!(ly->ckv = vkc_buf((size_t)ch->ks.rows * hd * sizeof(float), VKC_DEV))) return 0;
+ }
+ return 1;
+}
+
static int v4c_scratch(V4Chain *ch, int rows, int E, int LR) {
size_t r = (size_t)rows;
int cp = 2 * ch->hd, taps = coli_v4_full_dspark_wanted ? 3 : 1;
@@ -380,7 +474,13 @@ static int v4c_scratch(V4Chain *ch, int rows, int E, int LR) {
v4c_res(&ch->hs, r * ch->I, VKC_DEV) && v4c_res(&ch->ds, r * ch->D, VKC_DEV) &&
v4c_res(&ch->h2d, r * ch->D, VKC_DOWN) && v4c_res(&ch->routed, r * ch->D, VKC_UP) &&
v4c_res(&ch->xd, r * ch->HD, VKC_DOWN) && v4c_res(&ch->taps, (size_t)taps * r * ch->HD, VKC_DOWN);
- if (!ok) return 0;
+ if (ok && ch->ks.on) { /* the split: a chunk's new compressed rows and their canonical host copy */
+ size_t nc = r / (size_t)ch->rmin + 2;
+ if (g_v4c_count >= 0)
+ g_v4c_count += (long long)((2 * r * ch->hd * sizeof(float) + ch->rmin - 1) / ch->rmin);
+ else ok = v4c_res(&ch->cnew, nc * ch->hd, VKC_DEV) && v4c_res(&ch->cdn, nc * ch->hd, VKC_DOWN);
+ }
+ if (!ok || g_v4c_count >= 0) return ok; /* counting: the buffers only */
if (ch->rows < rows) {
float *hr = realloc(ch->host_routed, r * ch->D * sizeof(float));
if (!hr) return 0;
@@ -440,14 +540,16 @@ typedef struct { int start, n, pb, nr, csW, csC, LR, s_ident; } V4cFwd;
/* the compressor's ring for nr rows into out (rows of D floats), then the produced rows'
* pooled-latent pipeline: bf16, norm, bf16, RoPE at the group's first position, and the
* attention's E4M3 per 64 on the no-position part (or the indexer's Hadamard and E2M1) */
+/* out's row 0 is compressed row obase (0: out holds every row; the split's scratch: the
+ * chunk's first new row) */
static int v4c_compress(V4Chain *ch, const V4cFwd *f, ColiVkTensor *wkv, ColiVkTensor *wg, VkcBuf *kvb, VkcBuf *scb,
- VkcBuf *ring, size_t ape, VkcBuf *out, int r, int D, size_t norm, int fp4) {
+ VkcBuf *ring, size_t ape, VkcBuf *out, int obase, int r, int D, size_t norm, int fp4) {
int nr = f->nr, pb = f->pb, P = r == 4 ? 2 * D : D, g0 = pb / r, np = (pb + nr) / r - g0, rd = ch->rd;
- VkcDsComp cp = {nr, pb, r, P, D, r == 4, 0, P, 0, P, (int)ape, 0, D, 0};
+ VkcDsComp cp = {nr, pb, r, P, D, r == 4, 0, P, 0, P, (int)ape, -obase * D, D, 0};
if (!vkc_matmul(wkv, ch->nrm, 0, kvb, 0, nr) || !vkc_matmul(wg, ch->nrm, 0, scb, 0, nr) ||
!vkc_dsv4_compress(kvb, scb, ring, ch->prm, out, &cp)) return 0;
if (np <= 0) return 1;
- int o = g0 * D, cs = f->csC + (g0 * r - (pb - ch->rmax + 1)) * rd;
+ int o = (g0 - obase) * D, cs = f->csC + (g0 * r - (pb - ch->rmax + 1)) * rd;
if (!v4c_bf16(out, np, D, o, D) || !v4c_norm(ch, out, o, D, norm, out, o, D, np, D, 0) || !v4c_bf16(out, np, D, o, D) ||
!v4c_rope(ch, out, np, 1, o + D - rd, D, 0, cs, r * rd, 0)) return 0;
if (!fp4) return v4c_round(out, out, VKC_DS_E4M3, np, 1, D - rd, 64, 1, o, D, 0, o, D, 0);
@@ -465,11 +567,20 @@ static int v4c_attention(V4Chain *ch, int i, const V4cFwd *f) {
vkc_matmul(ly->wq_a, ch->nq, 0, ch->qa, 0, nr) && v4c_bf16(ch->qa, nr, QL, 0, QL) &&
v4c_norm(ch, ch->qa, 0, QL, ly->o_qn, ch->qr, 0, QL, nr, QL, 0) && v4c_bf16(ch->qr, nr, QL, 0, QL) &&
v4c_e4m3(ch->qr, ch->qrq, nr, QL);
- if (ok && ly->ratio > 0)
- ok = v4c_compress(ch, f, ly->cwkv, ly->cwg, ch->craw, ch->cscr, ly->ring, ly->o_ape, ly->ckv, ly->ratio, hd, ly->o_cn, 0);
+ int sp = ch->ks.on && ly->ratio > 0, li = ch->sli[i];
+ if (ok && ly->ratio > 0) {
+ int r = ly->ratio, g0 = pb / r, np = (pb + nr) / r - g0;
+ ok = v4c_compress(ch, f, ly->cwkv, ly->cwg, ch->craw, ch->cscr, ly->ring, ly->o_ape, sp ? ch->cnew : ly->ckv,
+ sp ? g0 : 0, r, hd, ly->o_cn, 0);
+ if (ok && sp && np > 0) { /* the split: into their slots, and down for the host's rows once the frame is through */
+ VkcKvPart pt = {1, hd, NULL, 0, ly->ckv, 0};
+ ok = vkc_kv_store(&ch->ks, li, &pt, ch->cnew, 0, (size_t)hd, 0, g0, np) && vkc_copy(ch->cdn, 0, ch->cnew, 0, (size_t)np * hd);
+ vkc_kv_done(&ch->ks, li, g0 + np);
+ }
+ }
if (ok && ly->idx) { /* the indexer: keys, queries, scores, top-k */
int IH = ch->IH, ID = ch->ID, width = (pb + nr) / 4, K = c->index_topk;
- ok = v4c_compress(ch, f, ly->iwkv, ly->iwg, ch->icraw, ch->icscr, ly->iring, ly->o_iape, ly->ikey, 4, ID, ly->o_icn, 1) &&
+ ok = v4c_compress(ch, f, ly->iwkv, ly->iwg, ch->icraw, ch->icscr, ly->iring, ly->o_iape, ly->ikey, 0, 4, ID, ly->o_icn, 1) &&
vkc_matmul(ly->iwq, ch->qrq, 0, ch->iq, 0, nr) && v4c_bf16(ch->iq, nr, IH * ID, 0, IH * ID) &&
v4c_rope(ch, ch->iq, nr * IH, IH, ID - rd, IH * ID, ID, tcs, rd, 0) &&
v4c_round(ch->iq, ch->iq, VKC_DS_HADAMARD, nr * IH, IH, ID, 0, 0, 0, IH * ID, ID, 0, IH * ID, ID) &&
@@ -508,8 +619,14 @@ static int v4c_attention(V4Chain *ch, int i, const V4cFwd *f) {
int kind = ly->ratio > 0 && !ly->idx ? ly->kind : 0;
VkcDsAttn a = {nr, nh, hd, cnt, kind * nr * f->LR, f->LR, ch->Wd, 0, 0, 0, qrow, 0, qrow, (int)ly->o_sink, 1,
1.0f / sqrtf((float)hd)};
- ok = ok && vkc_dsv4_attn(ch->q, ly->win, ly->ratio > 0 ? ly->ckv : NULL, ch->list, ch->prm, ch->heads, &a) &&
- v4c_rope(ch, ch->heads, nr * nh, nh, hd - rd, qrow, hd, tcs, rd, 1) &&
+ if (ok && sp) { /* stage selected cold rows, preserving V4's complete softmax and bf16 rounding */
+ ColiV4AttentionView v;
+ ok = !coli_v4_attention_view(ch->attn[i], &v);
+ VkcKvDs d = {&ch->ks, li, ch->q, ly->win, ly->ckv, ch->prm, ch->list, ch->heads, nr, nh, hd, cnt, a.l_off, a.l_row,
+ ch->Wd, 0, 0, qrow, (int)ly->o_sink, 1, 0, qrow, (pb + nr) / ly->ratio, a.scale, ok ? v.compressed : NULL};
+ ok = ok && vkc_kv_ds(&d);
+ } else ok = ok && vkc_dsv4_attn(ch->q, ly->win, ly->ratio > 0 ? ly->ckv : NULL, ch->list, ch->prm, ch->heads, &a);
+ ok = ok && v4c_rope(ch, ch->heads, nr * nh, nh, hd - rd, qrow, hd, tcs, rd, 1) &&
v4c_round(ch->heads, ch->heads, VKC_DS_E4M3, nr, 1, qrow, 128, 0, 0, qrow, 0, 0, qrow, 0);
int og = ch->og, ol = ch->ol, gw = ch->gw;
if (ok && og == 1) ok = vkc_matmul(ly->wo_a[0], ch->heads, 0, ch->grp, 0, nr);
@@ -590,7 +707,24 @@ static int v4c_forward(ColiV4Engine *engine, float **state_ptr, float **next_ptr
return 0;
}
}
- int rows = n < v4c_rows() ? n : v4c_rows(), LR = W + (K > 0 ? K : 1);
+ if (!v4c_plan(ch, E)) {
+ fprintf(stderr, "[VK] deepseek_v4 chain: device memory for the split's compressed rows refused; the CPU runs the layers\n");
+ ch->failed = 1; g_v4c_mode = 0;
+ return 0;
+ }
+ for (int i = 0; ch->ks.on && i < L; i++) /* the split: the host's rows take this forward's new ones as it goes */
+ if (ch->ly[i].ratio > 0 && coli_v4_attention_reserve(attention[i], E / ch->ly[i].ratio)) return 0;
+ int LR = W + (K > 0 ? K : 1), CH = v4c_chunk_rows(ch, E, LR);
+ if (ch->ks.on) { /* a chunk's new compressed rows fit the split's window */
+ int mx = ch->rmin == 1 ? ch->ks.chunk : (ch->ks.chunk - 1) * ch->rmin;
+ if (CH > mx) CH = mx < 1 ? 1 : mx;
+ }
+ int rows = n < CH ? n : CH;
+ if (!v4c_grow_win(ch, rows)) {
+ fprintf(stderr, "[VK] deepseek_v4 chain: device memory for window rings of %d rows refused; the CPU runs the layers\n", rows);
+ ch->failed = 1; g_v4c_mode = 0;
+ return 0;
+ }
int ok = v4c_scratch(ch, rows, E, LR);
for (int i = 0; ok && i < L; i++) ok = v4c_cache(ch, &ch->ly[i], E / (ch->ly[i].ratio > 0 ? ch->ly[i].ratio : 1) + 1);
/* what comes back at the end: the window rows, each layer's new compressed rows and
@@ -602,7 +736,7 @@ static int v4c_forward(ColiV4Engine *engine, float **state_ptr, float **next_ptr
ly->pw = pn; pn += (size_t)nq * hd;
if (ly->ratio <= 0) continue;
int r = ly->ratio;
- ly->pc = pn; pn += (size_t)(E / r - start / r) * hd;
+ ly->pc = pn; pn += ch->ks.on ? 0 : (size_t)(E / r - start / r) * hd; /* split: back per chunk */
ly->q1 = E - r > start ? E - r : start; /* without the overlap only the slots written change */
ly->ns = r == 4 ? ly->crows : E - ly->q1;
ly->pr = pn; pn += (size_t)2 * ly->ns * ly->cproj;
@@ -674,7 +808,7 @@ static int v4c_forward(ColiV4Engine *engine, float **state_ptr, float **next_ptr
ok = vkc_write(ly->win, (size_t)(q % ch->Wd) * hd, v.kv + (size_t)(q % W) * hd, (size_t)hd * sizeof(float));
if (ly->ratio <= 0) continue;
int t0 = ly->kv_valid, t1 = start / ly->ratio;
- if (ok && t1 > t0) ok = vkc_write(ly->ckv, (size_t)t0 * hd, v.compressed + (size_t)t0 * hd, (size_t)(t1 - t0) * hd * sizeof(float));
+ if (ok && t1 > t0 && !ch->ks.on) ok = vkc_write(ly->ckv, (size_t)t0 * hd, v.compressed + (size_t)t0 * hd, (size_t)(t1 - t0) * hd * sizeof(float));
ColiV4CompressorView cv;
size_t rf = (size_t)ly->crows * ly->cproj;
if (ok && !ly->ring_ok)
@@ -693,6 +827,17 @@ static int v4c_forward(ColiV4Engine *engine, float **state_ptr, float **next_ptr
if (!ok) goto lost;
for (int q = q_lo; q < start; q++) ch->slot_pos[q % ch->Wd] = q;
}
+ for (int i = 0; ch->ks.on && ok && i < L; i++) { /* the split: each table's window over this chunk's new rows */
+ V4cLayer *ly = &ch->ly[i];
+ if (ly->ratio <= 0) continue;
+ ColiV4AttentionView v;
+ if (coli_v4_attention_view(attention[i], &v)) goto lost;
+ int r = ly->ratio, g0 = pb / r;
+ VkcKvPart pt = {1, hd, v.compressed, 0, ly->ckv, 0};
+ vkc_kv_place(&ch->ks, ch->sli[i], g0, (pb + nr) / r - g0);
+ ok = vkc_kv_push(&ch->ks, ch->sli[i], &pt, 1, g0);
+ }
+ if (!ok) goto lost;
for (int s = 0; s < nr; s++) ch->slot_pos[(pb + s) % ch->Wd] = pb + s; /* this chunk's rows go in below */
int pending = 0;
for (int i = 0; i < L && ok; i++) {
@@ -704,8 +849,18 @@ static int v4c_forward(ColiV4Engine *engine, float **state_ptr, float **next_ptr
}
ok = ok && v4c_pre(ch, ly->fna, ly->o_hca, ch->hpa, ly->o_an, nr) && v4c_attention(ch, i, &f) &&
v4c_post(ch, ch->hpa, nr) && v4c_pre(ch, ly->fnf, ly->o_hcf, ch->hpf, ly->o_fn, nr) &&
- vkc_copy(ch->h2d, 0, ch->nrm, 0, (size_t)nr * D) && vkc_submit(1); /* A1 */
+ vkc_copy(ch->h2d, 0, ch->nrm, 0, (size_t)nr * D) && vkc_submit(0); /* A1 */
+ /* while it runs, the tier loads the experts this layer will likely stream (a
+ * big prompt chunk only) */
+ if (ok) vkt_stream_prefetch(i, nr);
+ ok = ok && vkc_finish();
if (!ok) break;
+ if (ch->ks.on && ly->ratio > 0) { /* this chunk's new compressed rows into the host's */
+ int r = ly->ratio, g0 = pb / r, np = (pb + nr) / r - g0;
+ ColiV4AttentionView v;
+ if (np > 0 && !coli_v4_attention_view(attention[i], &v))
+ memcpy(v.compressed + (size_t)g0 * hd, vkc_ptr(ch->cdn), (size_t)np * hd * sizeof(float));
+ }
/* A2: the shared expert, while the host computes the routed experts */
ok = vkc_begin() && v4c_shared(ch, ly, nr) && vkc_submit(0);
if (!ok) break;
@@ -741,7 +896,7 @@ static int v4c_forward(ColiV4Engine *engine, float **state_ptr, float **next_ptr
if (!ok || ly->ratio <= 0) continue;
int r = ly->ratio, g0 = start / r, np = E / r - g0;
size_t rf = (size_t)ly->crows * ly->cproj, cp = (size_t)ly->cproj;
- ok = vkc_copy(ch->pull, ly->pc, ly->ckv, (size_t)g0 * hd, (size_t)np * hd);
+ ok = ch->ks.on || vkc_copy(ch->pull, ly->pc, ly->ckv, (size_t)g0 * hd, (size_t)np * hd);
if (ok && r == 4) ok = vkc_copy(ch->pull, ly->pr, ly->ring, 0, 2 * rf);
else if (ok) {
VkcRegion *rr = malloc((size_t)2 * ly->ns * sizeof *rr);
@@ -783,7 +938,7 @@ static int v4c_forward(ColiV4Engine *engine, float **state_ptr, float **next_ptr
int r = ly->ratio, g0 = start / r, np = E / r - g0;
if (coli_v4_attention_reserve(attention[i], E / r)) goto host_oom;
coli_v4_attention_view(attention[i], &v);
- memcpy(v.compressed + (size_t)g0 * hd, pl + ly->pc, (size_t)np * hd * sizeof(float));
+ if (!ch->ks.on) memcpy(v.compressed + (size_t)g0 * hd, pl + ly->pc, (size_t)np * hd * sizeof(float));
coli_v4_attention_set_count(attention[i], E / r);
ColiV4CompressorView cv;
coli_v4_compressor_view(v.compressor, &cv);
@@ -842,16 +997,25 @@ static void v4c_report(void) {
fprintf(stderr, "[VK] deepseek_v4 chain: %llu forwards, %llu frames (%llu ops, %llu matmuls, %llu tiled GEMM), "
"%.1f ms waiting for the device, %.1f ms of routed experts on the host, %.1f MiB on the device\n",
ch->forwards, st.frames, st.ops, st.matmuls, st.gemms, st.wait_ms, ch->host_ms, st.dev_bytes / 1048576.0);
+ vkc_kv_report(&ch->ks);
vkc_prof_print();
}
/* COLI_VK_CHAIN at startup, before the tier sizes itself (the trunk's device copies count
* as used): the decision, the pipelines, the resident layers and their tensors. */
+/* The decision, made once: by the RAM plan when it asks whether the dense layers may live
+ * on the device only (the expert store is not open then), else at v4c_start. */
+static int g_v4c_decision = -1;
+static int v4c_decide(const ColiV4Engine *engine) {
+ if (g_v4c_decision >= 0) return g_v4c_decision;
+ if (!g_v4_vk_ready || !engine) return 0;
+ int tier_on = vkt_wanted() && !(engine->experts && engine->experts->gpu) && engine->config.n_routed_experts > 0;
+ return g_v4c_decision = coli_vk_chain_decide("deepseek_v4", tier_on, COLI_VK_CHAIN_UNMEASURED);
+}
static void v4c_start(const ColiV4Engine *cengine) {
ColiV4Engine *engine = (ColiV4Engine *)cengine;
if (!g_v4_vk_ready || !engine) return;
- int tier_on = vkt_wanted() && engine->experts && !engine->experts->gpu && engine->config.n_routed_experts > 0;
- int on = coli_vk_chain_decide("deepseek_v4", tier_on, COLI_VK_CHAIN_UNMEASURED);
+ int on = v4c_decide(engine);
const char *no = NULL;
if (on && !(g_v4c_inited = vkc_init())) no = "the chain's pipelines did not come up";
if (on && !no && !(vkc_mla_ready() && vkc_mhc_ready() && vkc_dsv4_ready()))
diff --git a/c/deepseek_v4_internal.h b/c/deepseek_v4_internal.h
index bea0f7f2..945dccd9 100644
--- a/c/deepseek_v4_internal.h
+++ b/c/deepseek_v4_internal.h
@@ -232,8 +232,40 @@ typedef struct {
/* Optional per-tensor backend-resident mirrors (Dsv4CudaTensor* on the CUDA
* tier). Aligned 1:1 with plan.tensors[]; owned by the engine's GPU tier. */
void *gpu[COLI_V4_MAX_LAYER_TENSORS];
+ /* Dense weights on the Vulkan device only (COLI_VK_DENSE_HOST): mapped = the tensor
+ * sits in an anonymous mapping of its own, whose pages can be given back while the
+ * address stays reserved (the device copies are looked up by it); host_gone = they
+ * were given back, the device holds the tensor alone. */
+ unsigned char mapped[COLI_V4_MAX_LAYER_TENSORS];
+ unsigned char host_gone[COLI_V4_MAX_LAYER_TENSORS];
} ColiDeepSeekV4LayerWeights;
+/* The resident tensors a Vulkan device may hold alone (docs/vulkan.md, "Dense weights on
+ * the device only"): every fp8 matrix, which the CPU reads only through the native_quant
+ * entries, and the compressors' bf16 projections, which it reads only in
+ * coli_v4_compressor_step; both read a dropped tensor back first
+ * (coli_v4_layer_host_restore). Everything else stays in RAM: norms, block scales, sinks,
+ * the compressors' ape, the mHC mixes, the router and the hash router's table, the
+ * indexer's weights_proj (multiplied inline on the CPU). */
+static inline int coli_v4_dense_device_only_tensor(const ColiDeepSeekV4TensorSpec *spec) {
+ if (!spec || spec->rank != 2) return 0;
+ if (spec->dtype == COLI_ST_F8_E4M3) return 1;
+ if (spec->dtype != COLI_ST_BF16) return 0;
+ size_t n = strlen(spec->name);
+ static const char *const tail[] = {"compressor.wkv.weight", "compressor.wgate.weight"};
+ for (int k = 0; k < 2; k++) {
+ size_t t = strlen(tail[k]);
+ if (n >= t && !strcmp(spec->name + n - t, tail[k])) return 1;
+ }
+ return 0;
+}
+/* Its bytes in RAM (fp8 one a value, bf16 two). */
+static inline uint64_t coli_v4_dense_tensor_bytes(const ColiDeepSeekV4TensorSpec *spec) {
+ uint64_t count = 1;
+ for (int axis = 0; axis < spec->rank; axis++) count *= (uint64_t)spec->shape[axis];
+ return spec->dtype == COLI_ST_BF16 ? count * 2 : count;
+}
+
int coli_v4_layer_plan(ColiDeepSeekV4LayerPlan *plan,
const ColiDeepSeekV4Config *config, int layer,
char *error, size_t error_size);
@@ -252,6 +284,28 @@ const void *coli_v4_layer_data(const ColiDeepSeekV4LayerWeights *weights,
const char *name,
const ColiDeepSeekV4TensorSpec **spec);
+/* Dense weights on the Vulkan device only. The engine binary built with VK=1 sets these
+ * pointers; every other link sees NULL and keeps its dense layers in RAM.
+ * coli_v4_dense_device_decide: asked by the RAM plan (EXPERT_STORE_AUTO) once it has
+ * measured the memory, with the bytes the device would hold and the RAM it would give
+ * back; 1 = the dense layers live on the device, their droppable tensors out of RAM.
+ * coli_v4_dense_place: after a resident layer is read, its tensors to the device and
+ * the droppable ones' pages given back (coli_v4_layer_host_drop).
+ * coli_v4_dense_device_lost: 1 once the device is gone; resident layers are then read
+ * per forward as a low-memory plan reads them. */
+typedef int (*ColiV4DenseDeviceDecide)(ColiV4Engine *engine, uint64_t device_bytes,
+ uint64_t host_bytes);
+extern ColiV4DenseDeviceDecide coli_v4_dense_device_decide;
+extern int (*coli_v4_dense_place)(ColiV4Engine *engine, int layer);
+extern int (*coli_v4_dense_device_lost)(void);
+/* A droppable tensor read again from disk per forward after the device was lost (bytes). */
+extern void (*coli_v4_dense_reread)(uint64_t bytes);
+/* Gives back the pages of resident tensor i of a layer (mapped, droppable); 0 on success. */
+int coli_v4_layer_host_drop(ColiV4Engine *engine, int layer, size_t i);
+/* Reads back from disk the dropped tensor that holds `data`, at its own address: 1 and
+ * its bytes when it did, 0 when `data` is in no dropped tensor, -1 on a read failure. */
+int coli_v4_layer_host_restore(ColiV4Engine *engine, const void *data, uint64_t *bytes);
+
/* Backend-mirror accessors: the gpu handle attached to the tensor named
* "layers.<N>.<suffix>.weight", or NULL when the tier did not upload it. */
void *coli_v4_layer_gpu(const ColiDeepSeekV4LayerWeights *weights,
@@ -977,6 +1031,10 @@ struct ColiV4Engine {
unsigned char ready[COLI_V4_RESIDENT_MAX_LAYERS];
const ColiSafetensorsIndex *index;
uint64_t total_bytes;
+ /* COLI_VK_DENSE_HOST: the layers' droppable tensors on the Vulkan device only
+ * (the plan left them out of RAM); what the device holds and what RAM gave back */
+ int device_only;
+ uint64_t device_bytes, dropped_bytes;
} dense_resident;
/* Optional CUDA tier (compiled in only when the engine build defines
* COLI_V4_GPU_TIER on Windows). enabled is 1 only after the loader resolved
diff --git a/c/gliner_decide.c b/c/gliner_decide.c
index 76929afd..590da7da 100644
--- a/c/gliner_decide.c
+++ b/c/gliner_decide.c
@@ -946,6 +946,19 @@ static int log_bucket(int rel, int bucket_size, int max_position)
static inline float gelu(float x) { return 0.5f * x * (1.0f + erff(x * 0.70710678118654752f)); }
+#ifdef COLI_VULKAN
+#include "decide_vk.h" /* COLI_VULKAN=1: the forward, or its matrices, on a Vulkan device */
+#endif
+
+/* Y = X W^T + bias: the encoder's and the classifier's matrices */
+static void gemm(float *Y, const float *X, int M, const QiMat *W, const float *bias)
+{
+#ifdef COLI_VULKAN
+ if (dvk_gemm(Y, X, M, W, bias)) return; /* matrix by matrix (COLI_VK_CHAIN=0) */
+#endif
+ qi_gemm(Y, X, M, W, bias);
+}
+
static void add_rows(float *x, const float *y, size_t n)
{
#pragma omp parallel for schedule(static)
@@ -1051,6 +1064,19 @@ static void attention(const Gl *M, const GlLayer *E, const float *qkv, float *ou
}
}
+/* The classifier (Linear -> ReLU -> Linear(., 1)) on the nm marker rows m: logits[nm]. */
+static void classify(const Gl *M, float *m, int nm, double *logits)
+{
+ float *m1 = (float *)xmalloc((size_t)(nm > 0 ? nm : 1) * M->cls_hidden * sizeof(float));
+ if (nm > 0) gemm(m1, m, nm, &M->cls1, M->cls1_b);
+ for (int k = 0; k < nm; k++) {
+ float *row = m1 + (size_t)k * M->cls_hidden;
+ for (int j = 0; j < M->cls_hidden; j++) if (row[j] < 0.f) row[j] = 0.f;
+ logits[k] = (double)(dot_f32_lanes(row, M->cls2_w, M->cls_hidden) + M->cls2_b);
+ }
+ free(m1);
+}
+
/* DebertaV2Model.forward on ids[0..T), then the classifier at the rows in
* marks[0..nm): logits[nm]. */
static void forward(const Gl *M, const int *ids, int T, const int *marks, int nm, double *logits)
@@ -1072,32 +1098,195 @@ static void forward(const Gl *M, const int *ids, int T, const int *marks, int nm
layernorm(x, h, M->emb_ln_w, M->emb_ln_b, T, d, M->eps);
for (int l = 0; l < M->layers; l++) {
const GlLayer *E = &M->L[l];
- qi_gemm(big, x, T, &E->wqkv, E->bqkv);
+ gemm(big, x, T, &E->wqkv, E->bqkv);
attention(M, E, big, att, T, &A);
- qi_gemm(h, att, T, &E->wo, E->bo);
+ gemm(h, att, T, &E->wo, E->bo);
add_rows(h, x, (size_t)T * d);
layernorm(x, h, E->ln1_w, E->ln1_b, T, d, M->eps); /* x = attention_output */
- qi_gemm(big, x, T, &E->wi, E->bi);
+ gemm(big, x, T, &E->wi, E->bi);
#pragma omp parallel for schedule(static)
for (size_t i = 0; i < (size_t)T * I; i++) big[i] = gelu(big[i]);
- qi_gemm(h, big, T, &E->wo2, E->bo2);
+ gemm(h, big, T, &E->wo2, E->bo2);
add_rows(h, x, (size_t)T * d);
layernorm(x, h, E->ln2_w, E->ln2_b, T, d, M->eps);
}
- /* classifier: Linear -> ReLU -> Linear(., 1) at each [L] */
float *m = (float *)xmalloc((size_t)(nm > 0 ? nm : 1) * d * sizeof(float));
- float *m1 = (float *)xmalloc((size_t)(nm > 0 ? nm : 1) * M->cls_hidden * sizeof(float));
for (int k = 0; k < nm; k++) memcpy(m + (size_t)k * d, x + (size_t)marks[k] * d, (size_t)d * sizeof(float));
- if (nm > 0) qi_gemm(m1, m, nm, &M->cls1, M->cls1_b);
- for (int k = 0; k < nm; k++) {
- float *row = m1 + (size_t)k * M->cls_hidden;
- for (int j = 0; j < M->cls_hidden; j++) if (row[j] < 0.f) row[j] = 0.f;
- logits[k] = (double)(dot_f32_lanes(row, M->cls2_w, M->cls_hidden) + M->cls2_b);
- }
- free(m); free(m1); free(x); free(h); free(att); free(big);
+ classify(M, m, nm, logits);
+ free(m); free(x); free(h); free(att); free(big);
free(A.p2c); free(A.vt); free(A.ridx);
}
+#ifdef COLI_VULKAN
+/* ------------------------------------------- the forward on the device */
+
+/* forward() as frames of the dense chain (decide_vk.h): the embedding rows go up, the
+ * 24 layers run on the device in forward()'s order with its arithmetic (f32; the
+ * disentangled attention with both relative terms, chain_enc.comp mode 5), and the [L]
+ * rows come back for classify(). The relative embeddings' projections (pos_q, pos_k),
+ * the norms' weights and the biases are uploaded once. */
+typedef struct {
+ int ok;
+ VkcBuf *prm, *pos;
+ VkcBuf *x, *h, *big, *att, *rng, *ridx, *ipos, *down, *c2p, *p2c;
+ int emb_w, emb_b;
+ int *bqkv, *bo, *ln1w, *ln1b, *bi, *bo2, *ln2w, *ln2b, *pq, *pk;
+} GlVk;
+static GlVk g_gvk;
+
+static void gl_vk_setup(Gl *M)
+{
+ if (!g_dvk.ready || (!g_dvk.chain && !g_dvk.dense)) return; /* nothing of it on the device */
+ int n = 0, want = 0;
+ for (int l = 0; l < M->layers; l++) {
+ GlLayer *E = &M->L[l];
+ const QiMat *w[4] = {&E->wqkv, &E->wo, &E->wi, &E->wo2};
+ for (int k = 0; k < 4; k++, want++) n += dvk_tensor(w[k]) != NULL;
+ }
+ want++;
+ n += dvk_tensor(&M->cls1) != NULL;
+ size_t bytes = 0, tensors = 0;
+ coli_vk_mem_info(&bytes, &tensors);
+ fprintf(stderr, "[VK] gliner_decide: %d of %d matrices on the device (%.1f MiB)\n", n, want, bytes / 1048576.0);
+ if (!g_dvk.chain) return;
+ if (n != want || M->hd > 128 || M->d > 4096) {
+ fprintf(stderr, "[VK] gliner_decide: the forward stays on the CPU (%s)\n",
+ n != want ? "a matrix did not reach the device" : "a geometry past the chain's shaders");
+ g_dvk.chain = 0;
+ g_dvk.dense = coli_vk_dense();
+ return;
+ }
+ GlVk *V = &g_gvk;
+ int L = M->layers, d = M->d, R = 2 * M->span;
+ int **lv[10] = {&V->bqkv, &V->bo, &V->ln1w, &V->ln1b, &V->bi, &V->bo2, &V->ln2w, &V->ln2b, &V->pq, &V->pk};
+ for (int k = 0; k < 10; k++) *lv[k] = (int *)xcalloc((size_t)L, sizeof(int));
+ DvkPrm P = {0}, Q = {0};
+ V->emb_w = dvk_prm_add(&P, M->emb_ln_w, d);
+ V->emb_b = dvk_prm_add(&P, M->emb_ln_b, d);
+ for (int l = 0; l < L; l++) {
+ GlLayer *E = &M->L[l];
+ V->bqkv[l] = dvk_prm_add(&P, E->bqkv, (size_t)3 * d);
+ V->bo[l] = dvk_prm_add(&P, E->bo, d);
+ V->ln1w[l] = dvk_prm_add(&P, E->ln1_w, d);
+ V->ln1b[l] = dvk_prm_add(&P, E->ln1_b, d);
+ V->bi[l] = dvk_prm_add(&P, E->bi, M->inter);
+ V->bo2[l] = dvk_prm_add(&P, E->bo2, d);
+ V->ln2w[l] = dvk_prm_add(&P, E->ln2_w, d);
+ V->ln2b[l] = dvk_prm_add(&P, E->ln2_b, d);
+ V->pq[l] = dvk_prm_add(&Q, E->pos_q, (size_t)R * d);
+ V->pk[l] = dvk_prm_add(&Q, E->pos_k, (size_t)R * d);
+ }
+ V->prm = dvk_prm_upload(&P);
+ V->pos = dvk_prm_upload(&Q);
+ free(P.v); free(Q.v);
+ if (!V->prm || !V->pos) {
+ fprintf(stderr, "[VK] gliner_decide: the forward stays on the CPU (no device memory for its parameters)\n");
+ g_dvk.chain = 0;
+ g_dvk.dense = coli_vk_dense();
+ return;
+ }
+ V->ok = 1;
+}
+
+static int gvk_lin(const QiMat *W, VkcBuf *x, VkcBuf *y, int T, int bias, int act)
+{
+ ColiVkTensor *t = dvk_tensor(W);
+ if (!t || !vkc_matmul(t, x, 0, y, 0, T)) return 0;
+ VkcEncBias p = {0, T * W->N, W->N, 0, W->N, bias, act};
+ return vkc_enc_bias(y, g_gvk.prm, &p);
+}
+
+/* forward() on the device; 0 = the caller runs forward(). */
+static int gl_vk_forward(const Gl *M, const int *ids, int T, const int *marks, int nm, double *logits)
+{
+ GlVk *V = &g_gvk;
+ if (!g_dvk.chain || !V->ok || !vkc_ready() || T < 1 || T > 65535) return 0;
+ double t0 = dvk_now_ms();
+ int d = M->d, I = M->inter, R = 2 * M->span;
+ int wide = 3 * d > I ? 3 * d : I, nread = nm > 0 ? nm : 1;
+ size_t fd = sizeof(float), id = sizeof(int);
+ if (!vkc_reserve(&V->x, (size_t)T * d * fd, VKC_DEV) || !vkc_reserve(&V->h, (size_t)T * d * fd, VKC_DEV) ||
+ !vkc_reserve(&V->att, (size_t)T * d * fd, VKC_DEV) || !vkc_reserve(&V->big, (size_t)T * wide * fd, VKC_DEV) ||
+ !vkc_reserve(&V->rng, (size_t)2 * T * id, VKC_DEV) || !vkc_reserve(&V->ridx, (size_t)2 * T * id, VKC_DEV) ||
+ !vkc_reserve(&V->ipos, (size_t)T * id, VKC_DEV) || !vkc_reserve(&V->down, (size_t)nread * d * fd, VKC_DOWN))
+ return 0;
+ /* the buckets a step can reach: ridx of delta -(T-1) .. T-1, which only grows */
+ int rlo_ = 0, rhi_ = R - 1;
+ {
+ int b = M->bucket_size > 0 && M->max_position > 0 ? log_bucket(-(T - 1), M->bucket_size, M->max_position) : -(T - 1);
+ int e = M->bucket_size > 0 && M->max_position > 0 ? log_bucket(T - 1, M->bucket_size, M->max_position) : T - 1;
+ rlo_ = b + M->span < 0 ? 0 : b + M->span > R - 1 ? R - 1 : b + M->span;
+ rhi_ = e + M->span < 0 ? 0 : e + M->span > R - 1 ? R - 1 : e + M->span;
+ }
+ int nr = rhi_ - rlo_ + 1;
+ if ((M->c2p && !vkc_reserve(&V->c2p, (size_t)M->heads * T * nr * fd, VKC_DEV)) ||
+ (M->p2c && !vkc_reserve(&V->p2c, (size_t)M->heads * T * nr * fd, VKC_DEV)))
+ return 0;
+ /* the host's part: the embedding rows; every row sees the whole sequence; the
+ * bucketed relative position of every delta, attention()'s ridx */
+ float *emb = (float *)xmalloc((size_t)T * d * fd);
+ int *rng = (int *)xmalloc((size_t)2 * T * id), *ridx = (int *)xmalloc((size_t)2 * T * id);
+ int *pos = (int *)xmalloc((size_t)T * id);
+ for (int r = 0; r < T; r++) {
+ int t = ids[r];
+ if (t < 0 || t >= M->vocab) t = 0;
+ memcpy(emb + (size_t)r * d, M->tok_emb + (size_t)t * d, (size_t)d * fd);
+ rng[2 * r] = 0; rng[2 * r + 1] = T - 1; pos[r] = r;
+ }
+ for (int delta = -(T - 1); delta <= T - 1; delta++) {
+ int b = M->bucket_size > 0 && M->max_position > 0 ? log_bucket(delta, M->bucket_size, M->max_position) : delta;
+ int r = b + M->span;
+ ridx[delta + T - 1] = r < 0 ? 0 : r > R - 1 ? R - 1 : r;
+ }
+ ridx[2 * T - 1] = 0;
+ VkcRegion *reg = (VkcRegion *)xmalloc((size_t)nread * sizeof(VkcRegion));
+ for (int k = 0; k < nm; k++) reg[k] = (VkcRegion){(size_t)k * d, (size_t)marks[k] * d, (size_t)d};
+ VkcBuf *x = V->x, *h = V->h, *big = V->big, *att = V->att;
+ int ok = vkc_begin() && vkc_write(h, 0, emb, (size_t)T * d * fd) && vkc_write(V->rng, 0, rng, (size_t)2 * T * id) &&
+ vkc_write(V->ridx, 0, ridx, (size_t)2 * T * id) && vkc_write(V->ipos, 0, pos, (size_t)T * id);
+ free(emb); free(rng); free(ridx); free(pos);
+ VkcEncNorm np = {0, T, d, 0, d, 0, d, 0, d, V->emb_w, V->emb_b, 0, M->eps};
+ ok = ok && vkc_enc_norm(h, NULL, V->prm, x, &np);
+ int sf = 1 + M->c2p + M->p2c;
+ float inv = 1.0f / sqrtf((float)(M->hd * sf));
+ int flags = (M->c2p ? VKC_ENC_C2P : 0) | (M->p2c ? VKC_ENC_P2C : 0);
+ for (int l = 0; ok && l < M->layers; l++) {
+ const GlLayer *E = &M->L[l];
+ ok = gvk_lin(&E->wqkv, x, big, T, V->bqkv[l], VKC_ENC_ACT_NONE);
+ /* the relative terms' products first, attention()'s c2p (q against the keys of
+ * the relative embeddings) and p2c (k against their queries), for the buckets
+ * this step reaches */
+ VkcEncRel rc = {0, T, M->heads, M->hd, 0, 3 * d, V->pk[l], d, 0, nr, rlo_};
+ VkcEncRel rp = {0, T, M->heads, M->hd, d, 3 * d, V->pq[l], d, 0, nr, rlo_};
+ if (M->c2p) ok = ok && vkc_enc_rel(big, V->pos, V->c2p, &rc);
+ if (M->p2c) ok = ok && vkc_enc_rel(big, V->pos, V->p2c, &rp);
+ VkcEncAttn ap = {0, T, M->heads, M->hd, 0, d, 2 * d, 3 * d, 0, d, flags, 0, 0, nr, T - 1, inv, rlo_};
+ ok = ok && vkc_enc_attn(big, V->c2p, V->p2c, att, V->rng, V->ridx, V->ipos, &ap);
+ ok = ok && gvk_lin(&E->wo, att, h, T, V->bo[l], VKC_ENC_ACT_NONE);
+ VkcEncNorm n1 = {0, T, d, 0, d, 0, d, 0, d, V->ln1w[l], V->ln1b[l], VKC_ENC_ADD, M->eps}; /* x = LN(h + x) */
+ ok = ok && vkc_enc_norm(h, x, V->prm, x, &n1);
+ ok = ok && gvk_lin(&E->wi, x, big, T, V->bi[l], VKC_ENC_ACT_GELU);
+ ok = ok && gvk_lin(&E->wo2, big, h, T, V->bo2[l], VKC_ENC_ACT_NONE);
+ VkcEncNorm n2 = {0, T, d, 0, d, 0, d, 0, d, V->ln2w[l], V->ln2b[l], VKC_ENC_ADD, M->eps};
+ ok = ok && vkc_enc_norm(h, x, V->prm, x, &n2);
+ }
+ ok = ok && (nm == 0 || vkc_copy_regions(V->down, x, reg, nm)) && vkc_submit(1);
+ free(reg);
+ if (!ok) {
+ if (vkc_lost()) fprintf(stderr, "[VK] gliner_decide: the device was lost: the forward runs on the CPU\n");
+ else vkc_finish();
+ return 0;
+ }
+ float *m = (float *)xmalloc((size_t)nread * d * fd);
+ memcpy(m, vkc_ptr(V->down), (size_t)nm * d * fd);
+ classify(M, m, nm, logits);
+ free(m);
+ g_dvk.fwd_dev++; g_dvk.rows_dev += (unsigned long long)T;
+ g_dvk.dev_ms += dvk_now_ms() - t0;
+ return 1;
+}
+#endif
+
/* ------------------------------------------------------- the rendering */
typedef struct {
@@ -1231,7 +1420,15 @@ static int gl_decide(void *opaque, const DecideRecord *rec, DecideAnswer *answer
Rendered r;
if (!render(M, rec, &r, err, cap)) { rendered_free(&r); return 0; }
double *logits = (double *)xcalloc((size_t)(r.n_marks > 0 ? r.n_marks : 1), sizeof(double));
+#ifdef COLI_VULKAN
+ if (!gl_vk_forward(M, r.ids.v, r.ids.n, r.label_marks, r.n_marks, logits)) {
+ if (g_dvk.chain) g_dvk.fwd_cpu++;
+ forward(M, r.ids.v, r.ids.n, r.label_marks, r.n_marks, logits);
+ }
+ dvk_report();
+#else
forward(M, r.ids.v, r.ids.n, r.label_marks, r.n_marks, logits);
+#endif
for (int q = 0; q < rec->n_questions; q++) {
const DecideQuestion *Q = &rec->questions[q];
DecideAnswer *a = &answers[q];
@@ -1430,6 +1627,10 @@ int main(int argc, char **argv)
"max_len %d; loaded in %.1f s\n", M->model_name, M->layers, M->d, M->heads, M->span,
M->max_len, (now_ms() - t0) / 1e3);
if (tokenize) return run_tokenize(M, tokenize);
+#ifdef COLI_VULKAN
+ dvk_init("gliner_decide"); /* COLI_VULKAN=1: the device */
+ gl_vk_setup(M);
+#endif
if (records) return run_records(M, records, dump_ids);
if (!serving) die("nothing to do: set SERVE=1, or pass --records / --tokenize / --split");
int slots = getenv("KV_SLOTS") ? atoi(getenv("KV_SLOTS")) : 1;
diff --git a/c/glm53.c b/c/glm53.c
index 05b4cf05..8a8a23c1 100644
--- a/c/glm53.c
+++ b/c/glm53.c
@@ -658,6 +658,11 @@ typedef struct {
void *vk; /* ColiVkTensor*, caricata alla prima uso */
int resident; /* eligible for persistent accelerator wrapping */
void *metal; /* ColiMetalTensor*, created lazily */
+ /* the dense weights on the device only (COLI_VK_DENSE_HOST): the tensor it came from
+ * (kind 0 load_mat, 1 and 2 the halves absorb_kvb makes of kv_b_proj), to read it back
+ * (g53_dho_reload); vk_gone = 1 while the device holds it alone */
+ char *vk_name;
+ int vk_kind, vk_gone;
} Mat;
typedef struct {
@@ -888,6 +893,11 @@ static void absorb_kvb(GModel *m, GLayer *l, const char *name) {
free(whole);
l->kvb_kt = quantize_loaded(kt, H * L, QK);
l->kvb_v = quantize_loaded(vv, H * V, L);
+#ifdef COLI_VULKAN
+ l->kvb_kt.vk_name = strdup(name); l->kvb_kt.vk_kind = 1;
+ l->kvb_v.vk_name = strdup(name); l->kvb_v.vk_kind = 2;
+ if (!l->kvb_kt.vk_name || !l->kvb_v.vk_name) { fprintf(stderr, "OOM on %s\n", name); exit(1); }
+#endif
}
/* rows x columns e' la forma con cui il forward usa la matrice, contata dalla
@@ -953,6 +963,9 @@ static Mat load_mat(GModel *m, int64_t rows, int64_t columns, const char *fmt, .
st_read_raw(&m->S, name, packed, 1);
st_read_f32_cap(&m->S, scales, step, qs->numel, 1);
mat.fmt = 4; mat.q4 = packed; mat.s = step; mat.gs = 64; mat.resident = 1;
+#ifdef COLI_VULKAN
+ if (!(mat.vk_name = strdup(name))) { fprintf(stderr, "OOM on %s\n", name); exit(1); }
+#endif
return mat;
}
@@ -970,12 +983,72 @@ static Mat load_mat(GModel *m, int64_t rows, int64_t columns, const char *fmt, .
mat.columns = (int)columns;
mat = quantize_loaded(buffer, mat.rows, mat.columns);
+#ifdef COLI_VULKAN
+ if (!(mat.vk_name = strdup(name))) { fprintf(stderr, "OOM on %s\n", name); exit(1); }
+#endif
return mat;
}
+#ifdef COLI_VULKAN
+/* ---- the dense weights on the device only (COLI_VK_DENSE_HOST; docs/vulkan.md) -----
+ * With the dense part on the device (the chain, or the per-matrix path), every resident
+ * matrix is uploaded once the device is open and its host copy is given back, before the
+ * expert cache sizes itself from the free memory (expert_cache_init), so that RAM goes to
+ * the experts. mv and mm multiply a matrix the device holds alone by that copy (f32 as
+ * the backend's fmt 10, as the chain uploads it); anything else that reads it (mv_rows,
+ * mm_rows, a parallel region, a lost device) reads it back from the checkpoint first:
+ * load_mat, or absorb_kvb for kv_b's two halves, with the GLM53_BITS of the load, so the
+ * same bytes; the copy stays from there on. */
+static GModel *g_g53_dho_model;
+static pthread_mutex_t g_g53_dho_mx = PTHREAD_MUTEX_INITIALIZER;
+static int g53_vk_fmt(const Mat *w) { return w->fmt == 0 ? 10 : w->fmt; }
+static size_t g53_host_bytes(const Mat *w) {
+ const size_t r = (size_t)w->rows, cl = (size_t)w->columns;
+ if (w->fmt == 0) return r * cl * 4;
+ if (w->fmt == 1) return r * cl + r * 4;
+ return r * ((cl + 1) / 2) + r * ((cl + w->gs - 1) / w->gs) * 4;
+}
+static void g53_dho_reload(Mat *w) {
+ pthread_mutex_lock(&g_g53_dho_mx);
+ if (__atomic_load_n(&w->vk_gone, __ATOMIC_ACQUIRE)) {
+ GModel *m = g_g53_dho_model;
+ if (!m || !w->vk_name) { fprintf(stderr, "[VK] glm53: a dense matrix the device held alone cannot be read back\n"); exit(1); }
+ Mat n;
+ if (w->vk_kind == 0) n = load_mat(m, w->rows, w->columns, "%s", w->vk_name);
+ else {
+ GLayer tmp; memset(&tmp, 0, sizeof tmp);
+ absorb_kvb(m, &tmp, w->vk_name);
+ Mat *other = w->vk_kind == 1 ? &tmp.kvb_v : &tmp.kvb_kt;
+ n = w->vk_kind == 1 ? tmp.kvb_kt : tmp.kvb_v;
+ free((void *)other->f); free((void *)other->q8); free((void *)other->q4); free((void *)other->s);
+ free(other->vk_name);
+ }
+ if (n.fmt != w->fmt || n.rows != w->rows || n.columns != w->columns || n.gs != w->gs) {
+ fprintf(stderr, "[VK] glm53: %s came back from disk in another form (fmt %d, was %d)\n", w->vk_name, n.fmt, w->fmt);
+ exit(1);
+ }
+ w->f = n.f; w->q8 = n.q8; w->q4 = n.q4; w->s = n.s;
+ free(n.vk_name);
+ __atomic_store_n(&w->vk_gone, 0, __ATOMIC_RELEASE);
+ coli_vk_dense_host_reloaded(g53_host_bytes(w));
+ }
+ pthread_mutex_unlock(&g_g53_dho_mx);
+}
+static int g53_dho_gone(const Mat *w) { return __atomic_load_n(&w->vk_gone, __ATOMIC_ACQUIRE); }
+/* The device's copy of a matrix it holds alone: 0 = the CPU computes (after a read-back). */
+static int g53_dho_matmul(const Mat *w, float *out, const float *x, int S) {
+ if (omp_in_parallel() || !w->vk) return 0;
+ return coli_vk_matmul((ColiVkTensor **)&((Mat *)w)->vk, out, x, NULL, NULL, g53_vk_fmt(w), S, w->columns,
+ w->rows, w->fmt == 4 ? w->gs : 0);
+}
+#endif
+
/* Come mv ma su un blocco di righe contigue: serve alle matrici che tengono
* una testa dopo l'altra in un unico tensore. */
static void mv_rows(float *out, const Mat *w, const float *x, int row0, int rows) {
+#ifdef COLI_VULKAN
+ if (g53_dho_gone(w)) g53_dho_reload((Mat *)w);
+#endif
switch (w->fmt) {
case 4: {
const int packed = (w->columns + 1) / 2, groups = w->columns / w->gs;
@@ -1014,6 +1087,10 @@ static void mv(float *out, const Mat *w, const float *x) {
}
#endif
#ifdef COLI_VULKAN
+ if (g53_dho_gone(w)) {
+ if (g53_dho_matmul(w, out, x, 1)) return;
+ g53_dho_reload((Mat *)w);
+ }
if (g_vk_dense && w->resident && (w->fmt == 1 || w->fmt == 4)) {
Mat *mutable_w = (Mat *)w;
if (coli_vk_matmul((ColiVkTensor **)&mutable_w->vk, out, x,
@@ -1042,6 +1119,11 @@ static void mm(float *out, const Mat *w, const float *x, int S) {
gpu |= g_metal_ready && w->resident && (w->fmt == 1 || w->fmt == 4);
#endif
#ifdef COLI_VULKAN
+ if (S > 1 && g53_dho_gone(w)) { /* the device holds it alone: S rows there, else row by row (mv reads it back) */
+ if (g53_dho_matmul(w, out, x, S)) return;
+ for (int t = 0; t < S; t++) mv(out + (size_t)t * w->rows, w, x + (size_t)t * w->columns);
+ return;
+ }
if (S > 1 && g_vk_dense && w->resident && (w->fmt == 1 || w->fmt == 4)) {
Mat *mutable_w = (Mat *)w;
if (coli_vk_matmul((ColiVkTensor **)&mutable_w->vk, out, x,
@@ -1066,6 +1148,9 @@ static void mm(float *out, const Mat *w, const float *x, int S) {
/* mv_rows for S rows: out[S, rows] from x[S, columns] and W's rows
* [row0, row0 + rows). */
static void mm_rows(float *out, const Mat *w, const float *x, int S, int row0, int rows) {
+#ifdef COLI_VULKAN
+ if (g53_dho_gone(w)) g53_dho_reload((Mat *)w);
+#endif
switch (w->fmt) {
case 4: {
const int packed = (w->columns + 1) / 2, groups = w->columns / w->gs;
@@ -1897,11 +1982,11 @@ static void expert_mats(const GModel *m, const Slot *slot, Mat *gate, Mat *up, M
const int hidden = m->c.hidden, inter = m->c.moe_inter;
const Mat shape[3] = {
{ 4, NULL, NULL, slot->piece[0], (const float *)slot->piece[1],
- inter, hidden, 64, NULL, 0, NULL },
+ inter, hidden, 64, NULL, 0, NULL, NULL, 0, 0 },
{ 4, NULL, NULL, slot->piece[2], (const float *)slot->piece[3],
- inter, hidden, 64, NULL, 0, NULL },
+ inter, hidden, 64, NULL, 0, NULL, NULL, 0, 0 },
{ 4, NULL, NULL, slot->piece[4], (const float *)slot->piece[5],
- hidden, inter, 64, NULL, 0, NULL },
+ hidden, inter, 64, NULL, 0, NULL, NULL, 0, 0 },
};
*gate = shape[0]; *up = shape[1]; *down = shape[2];
}
@@ -1962,6 +2047,27 @@ static VktExpertSrc glm53_slot_src(const Slot *slot) {
return (VktExpertSrc){slot->piece[0], slot->piece[2], slot->piece[4],
slot->piece[1], slot->piece[3], slot->piece[5]};
}
+/* The tier's streaming (a big prompt chunk's cold experts on the device): a group of
+ * experts read as ffn_vk_cpu reads them (expert_block_read: the layer cache, the misses
+ * in parallel), up to the cache's capacity, each valid until the next read. */
+static int glm53_vk_load_batch(void *ctx, int index, const int *e, int n, VktExpertSrc *srcs, void **h) {
+ GModel *m = ctx;
+ LCache *cache = &m->ecache[index];
+ if (n > cache->cap) n = cache->cap;
+ if (n < 1 || n > 64) return 0;
+ int slot_of[64], to_read[64];
+ expert_block_read(m, index, e, n, slot_of, to_read);
+ for (int u = 0; u < n; u++) {
+ Slot *slot = &cache->s[slot_of[u]];
+ slot->used = ++m->clock;
+ srcs[u] = glm53_slot_src(slot); h[u] = slot;
+ }
+ return n;
+}
+static int glm53_vk_load(void *ctx, int index, int e, VktExpertSrc *src, void **h) {
+ return glm53_vk_load_batch(ctx, index, &e, 1, src, h) == 1;
+}
+static void glm53_vk_unhold(void *ctx, void *h) { (void)ctx; (void)h; }
/* Pairs i of one block that want[i] marks, on the CPU: their experts in first-seen
* order, in blocks of the layer's cache slots as ffn_layer reads them, each computed
* once for all its rows into ctb[i]. note: offer each to the tier. */
@@ -2006,7 +2112,7 @@ static void ffn_vk_cpu(GModel *m, int index, const float *x, int rows, int K, co
static void ffn_layer_vk(GModel *m, int index, const float *x, int tokens, const int *chosen,
const float *weight, float *out, float *sg, float *su, float *tmp, float *xg) {
const Cfg *c = &m->c;
- const int D = c->hidden, K = c->topk, B = tokens < GLM53_VK_ROWS ? tokens : GLM53_VK_ROWS, nmax = B * K;
+ const int D = c->hidden, K = c->topk, B = vkt_step_rows(tokens, GLM53_VK_ROWS), nmax = B * K; /* a whole prompt chunk when the tier streams */
uint8_t *taken = calloc((size_t)nmax, 1), *want = calloc((size_t)nmax, 1);
const float **dev = malloc((size_t)nmax * sizeof(*dev));
float *ctb = malloc((size_t)nmax * D * sizeof(float));
@@ -2287,6 +2393,62 @@ static void expert_geometry(GModel *m);
static void expert_table_init(GModel *m);
static void expert_cache_init(GModel *m);
+#ifdef COLI_VULKAN
+/* Upload one matrix as glm53_chain.h's g53c_tensor would (the same format and geometry,
+ * so the chain and mv/mm find this copy) and give its host copy back. */
+static int g53_dho_drop(Mat *w, int keep, int drop, size_t *bytes) {
+ if (keep || !w->resident || !w->vk_name || w->vk_gone || w->rows < 1 || w->columns < 1) return 0;
+ const void *p = w->fmt == 0 ? (const void *)w->f : w->fmt == 1 ? (const void *)w->q8 : (const void *)w->q4;
+ if (!p || (w->fmt != 0 && w->fmt != 1 && w->fmt != 4) || (w->fmt == 4 && (w->gs < 8 || w->gs % 8))) return 0;
+ if (!drop) { *bytes += g53_host_bytes(w); return 1; }
+ if (!coli_vk_tensor_ensure((ColiVkTensor **)&w->vk, p, w->fmt ? w->s : NULL, g53_vk_fmt(w), w->columns, w->rows,
+ w->fmt == 4 ? w->gs : 0)) return 0;
+ const size_t b = g53_host_bytes(w);
+ free((void *)w->f); free((void *)w->q8); free((void *)w->q4); free((void *)w->s);
+ w->f = NULL; w->q8 = NULL; w->q4 = NULL; w->s = NULL;
+ __atomic_store_n(&w->vk_gone, 1, __ATOMIC_RELEASE);
+ coli_vk_dense_host_dropped(b);
+ *bytes += b;
+ return 1;
+}
+/* Every resident matrix but the routed experts: the head (unless tied to the embedding),
+ * each layer's attention (KDA or MLA with its indexer), dense MLP and shared expert. The
+ * absorbed kv_b halves stay when the chain will not run every forward: outside it the
+ * CPU's attention reads them by row (mm_rows, mv_rows). */
+static int g53_dho_pass(GModel *m, int keep_kvb, int drop, size_t *bytes) {
+ int n = g53_dho_drop(&m->head, 0, drop, bytes);
+ for (int i = m->layer_begin; i < m->layer_end; i++) {
+ GLayer *l = &m->layer[i];
+ Mat *all[] = {&l->kq, &l->kk, &l->kv, &l->ko, &l->kga, &l->kgb, &l->kfa, &l->kfb, &l->kb,
+ &l->qa, &l->qb, &l->kva, &l->o, &l->iwq, &l->iwk, &l->iwp, &l->ikpg,
+ &l->dg, &l->du, &l->dd, &l->rg, &l->ru, &l->rd};
+ for (size_t k = 0; k < sizeof(all) / sizeof(all[0]); k++) n += g53_dho_drop(all[k], 0, drop, bytes);
+ n += g53_dho_drop(&l->kvb_kt, keep_kvb, drop, bytes);
+ n += g53_dho_drop(&l->kvb_v, keep_kvb, drop, bytes);
+ }
+ return n;
+}
+/* Once the device is open, before expert_cache_init reads the free memory: the decision
+ * (the chain's own, made silently here and printed by g53c_start), the uploads and the
+ * RAM given back. */
+static void g53_dho_start(GModel *m, int tier) {
+ if (!g_vk_ready || !m->has_io || m->layer_begin != 0 || m->layer_end != m->c.n_layers) return;
+ const int chain = coli_vk_chain_decide(NULL, tier, COLI_VK_CHAIN_UNMEASURED);
+ const int keep_kvb = chain != COLI_VK_CHAIN_ON;
+ size_t bytes = 0;
+ g53_dho_pass(m, keep_kvb, 0, &bytes);
+ if (!coli_vk_dense_host_decide("glm53", chain != COLI_VK_CHAIN_OFF || g_vk_dense, bytes)) return;
+ g_g53_dho_model = m;
+ g_vk_dense = 1; /* the steps the chain does not run take the device too: the CPU has no copy */
+ bytes = 0;
+ g53_dho_pass(m, keep_kvb, 1, &bytes);
+ coli_vk_dense_host_placed("glm53", keep_kvb
+ ? "the embedding (and a head tied to it), the routers, the mHC mixes and norms, the vision tower, "
+ "the absorbed kv_b (the chain will not run every forward: the CPU's attention reads it)"
+ : "the embedding (and a head tied to it), the routers, the mHC mixes and norms, the vision tower");
+}
+#endif
+
static void model_load_range(GModel *m, const char *dir, int layer_begin,
int layer_end, int load_io) {
load_cfg(&m->c, dir);
@@ -2468,6 +2630,7 @@ static void model_load_range(GModel *m, const char *dir, int layer_begin,
if (g_vk_ready) coli_vk_set_swiglu_limit(m->c.swiglu_limit);
if (!g_vk_ready) fprintf(stderr, "Vulkan: no usable device, falling back to CPU\n");
else if (g_vk_dense) fprintf(stderr, "Vulkan: active for resident matrices\n");
+ g53_dho_start(m, tier); /* COLI_VK_DENSE_HOST: the trunk on the device only, before the expert cache sizes itself */
}
#endif
/* La cache si dimensiona qui, non prima: quanto si puo' spendere dipende
@@ -2716,6 +2879,7 @@ static float *run_layers(GModel *m, GSession *s, float *streams, float *next,
* una perdita per richiesta. Ogni allocazione fatta dal caricamento ha qui il
* suo rilascio, l'indice dei tensori compreso. */
static void mat_release(Mat *mat) {
+ free(mat->vk_name);
#ifdef COLI_METAL
if (mat->metal) coli_metal_tensor_free((ColiMetalTensor *)mat->metal);
#endif
@@ -2955,6 +3119,11 @@ static float *forward_prefill(GModel *m, GSession *s, const int *tokens, int n,
const Cfg *c = &m->c;
const char *setting = getenv("GLM53_PREFILL_CHUNK");
int chunk = setting ? atoi(setting) : 128;
+#ifdef COLI_VULKAN
+ /* the dense chain with a chunk from the budget: blocks of its chunk, so a long
+ * prompt's MoE steps are big ones the tier can stream */
+ if (!setting && !(g_echo_k > 0 && g_echo_id)) { int r = g53c_prefill_rows(m, s, n); if (r > 0) chunk = r; }
+#endif
if (chunk < 1) chunk = 1;
if (chunk > n) chunk = n;
@@ -4095,8 +4264,10 @@ static void glm53_vk_tier_start(GModel *m) {
.act = VKT_ACT_SWIGLU, .act_limit = c->swiglu_limit,
.max_rows = GLM53_VK_ROWS * c->topk,
.ram_reserve = (size_t)cap * (size_t)sparse * (size_t)m->e_slot,
- .dense_bytes = g_vk_dense ? glm53_dense_dev_bytes(m) : 0,
- .in_ram = glm53_in_ram, .ram_ctx = m};
+ .dense_bytes = g_vk_dense && !coli_vk_dense_device_only() ? glm53_dense_dev_bytes(m) : 0,
+ .in_ram = glm53_in_ram, .ram_ctx = m,
+ .load = glm53_vk_load, .release = glm53_vk_unhold, .load_ctx = m,
+ .load_batch = glm53_vk_load_batch};
atexit(coli_vk_shutdown); /* before vkt_init, which makes the expert batch's pipelines and can still refuse (no room): the device goes at exit either way, after the tier's teardown */
if (vkt_init(&vc, rt_counts_all())) {
atexit(vkt_shutdown);
diff --git a/c/glm53_chain.h b/c/glm53_chain.h
index e58a7ebb..c6162bce 100644
--- a/c/glm53_chain.h
+++ b/c/glm53_chain.h
@@ -32,6 +32,12 @@
* is brought back before anything reads it there (a pin, a state capture, a CPU
* forward of the session, another session taking the device) and pushed up after
* anything writes it there (a restored pin or state).
+ * Past the device's budget (vk_kvsplit.h) each MLA layer keeps only some blocks of its
+ * latent rows on the device: the window of the newest, and the blocks the k-pooled
+ * selection reads most. The attention core runs over those on the device and over the
+ * rest on the CPU (from the host's canonical cache) at once, merged through the softmax
+ * statistics; the index keys, pool gates and pooled keys, which every row scores, stay
+ * whole.
* A device lost while the device holds the newest KDA state: the state is rebuilt on the
* CPU from the input rows of the positions since the host's copy was last current (the
* chain keeps them: the embedding rows, or the vision tower's), the forward runs again on
@@ -44,6 +50,7 @@
* a KDA head above 256 key or 128 value floats, an indexer above 64 heads, 4096 query
* floats or k-pooling below 2). */
#include "vk_chain.h"
+#include "vk_kvsplit.h"
#define G53C_HOST 0
#define G53C_DEV 1
@@ -59,6 +66,9 @@ typedef struct {
VkcMla *mla;
VkcMlaCache *kv; VkcBuf **ik, **ig, **pk, **win, **st;
int *kv_valid, *pool_valid;
+ int *mla_ord, n_mla, dev_rows; /* the MLA layers' order; device latent rows a layer */
+ VkcKvSplit ks; /* the split past the device's budget (ks.on) */
+ VkcMlaCache kvtmp; /* the split: a step's new rows before their slots */
VkcMlaScratch sc;
VkcBuf *xs, *xn, *col, *nrm, *br, *mix, *hp, *gs, *us, *hs, *ds, *qkv3, *cm, *kf, *kb, *kg, *klow, *ky;
VkcBuf *ikd, *iq, *ihw, *igd, *isc, *sel;
@@ -75,11 +85,6 @@ static G53Chain *g_g53c;
static int g_vk_chain = 0;
static int g_g53c_inited = 0;
-static int g53c_rows(void) {
- const char *e = getenv("COLI_VK_CHAIN_ROWS");
- int v = e && *e ? atoi(e) : 512;
- return v < 1 ? 1 : v > 65535 ? 65535 : v;
-}
/* A Mat's device copy (the per-matrix path's own, where it made one); f32 as fmt 10. */
static ColiVkTensor *g53c_tensor(const Mat *wc) {
@@ -136,6 +141,7 @@ static void g53c_cpu_step(const GModel *m, const GSession *s, int start) {
for (int i = 0; i < m->c.n_layers; i++) {
if (ch->kv_valid[i] > start) ch->kv_valid[i] = start;
if (ch->pool_valid[i] > start / m->c.index_kpool) ch->pool_valid[i] = start / m->c.index_kpool;
+ if (m->c.is_full[i]) vkc_kv_lower(&ch->ks, ch->mla_ord[i], start);
}
}
@@ -160,9 +166,10 @@ static int g53c_setup(GModel *m) {
ch->mla = calloc(L, sizeof(VkcMla)); ch->kv = calloc(L, sizeof(VkcMlaCache));
ch->ik = calloc(L, sizeof(void *)); ch->ig = calloc(L, sizeof(void *)); ch->pk = calloc(L, sizeof(void *));
ch->win = calloc(L, sizeof(void *)); ch->st = calloc(L, sizeof(void *));
- ch->kv_valid = calloc(L, sizeof(int)); ch->pool_valid = calloc(L, sizeof(int));
+ ch->kv_valid = calloc(L, sizeof(int)); ch->pool_valid = calloc(L, sizeof(int)); ch->mla_ord = calloc(L, sizeof(int));
if (!ch->fna || !ch->fnf || !ch->mla || !ch->kv || !ch->ik || !ch->ig || !ch->pk || !ch->win || !ch->st ||
- !ch->kv_valid || !ch->pool_valid) return 0;
+ !ch->kv_valid || !ch->pool_valid || !ch->mla_ord) return 0;
+ for (int i = 0; i < L; i++) if (c->is_full[i]) ch->mla_ord[i] = ch->n_mla++;
g_g53c = ch;
/* the parameter arena */
size_t n = 0;
@@ -235,7 +242,38 @@ static int g53c_setup(GModel *m) {
return 1;
}
-static int g53c_res(VkcBuf **b, size_t floats, int kind) { return vkc_reserve(b, (floats ? floats : 1) * sizeof(float), kind); }
+/* g53c_res counts instead of reserving while g_g53c_count >= 0 (the chunk's sizing) */
+static long long g_g53c_count = -1;
+static int g53c_res(VkcBuf **b, size_t floats, int kind) {
+ if (g_g53c_count >= 0) { g_g53c_count += (long long)(floats ? floats : 1) * (long long)sizeof(float); return 1; }
+ return vkc_reserve(b, (floats ? floats : 1) * sizeof(float), kind);
+}
+static int g53c_scratch(G53Chain *ch, const GModel *m, int rows, int ctx);
+/* Prompt rows per chunk (vkc_chunk_rows): the chain's scratch a row, counted from the
+ * reservations (the MLA scratch as vkc_mla_scratch sizes it, the indexer's scores at
+ * the session's capacity), and the routed experts' outputs (the tier's rows, the
+ * host's sum) for it. */
+static int g53c_rows(G53Chain *ch, const GModel *m, int ctx) {
+ if (!vkc_chunk_auto()) return vkc_chunk_rows("glm53", 0);
+ const Cfg *c = &m->c;
+ size_t mla = 0;
+ for (int i = 0; i < c->n_layers; i++) if (c->is_full[i]) {
+ const VkcMla *a = &ch->mla[i];
+ mla = (size_t)(a->q_lora > 0 ? a->q_lora : 1) + (size_t)a->H * (a->Q + a->R) + (size_t)(a->K + a->R) +
+ 2 * (size_t)a->H * a->K + (size_t)a->H * a->V;
+ break;
+ }
+ size_t ksz = ch->kvd_layer;
+ g_g53c_count = 0; g53c_scratch(ch, m, 1, ctx); long long b1 = g_g53c_count;
+ g_g53c_count = 0; g53c_scratch(ch, m, 2, ctx); long long b2 = g_g53c_count;
+ g_g53c_count = -1; ch->kvd_layer = ksz;
+ size_t row = (size_t)(b2 - b1) + mla * sizeof(float) + (size_t)(2 * c->topk + 2) * c->hidden * sizeof(float);
+ /* ffn_layer_ex allocates both activation matrices and its gather/output
+ * buffers even when the tier takes every routed assignment. */
+ size_t wide = c->dense_inter > c->moe_inter ? c->dense_inter : c->moe_inter;
+ row += 2 * (wide + c->hidden) * sizeof(float);
+ return vkc_chunk_rows("glm53", row);
+}
static int g53c_scratch(G53Chain *ch, const GModel *m, int rows, int ctx) {
const Cfg *c = &m->c;
@@ -245,7 +283,7 @@ static int g53c_scratch(G53Chain *ch, const GModel *m, int rows, int ctx) {
int mfull = -1; for (int i = 0; i < L; i++) if (c->is_full[i]) { mfull = i; break; }
size_t r = (size_t)rows;
ch->kvd_layer = r * (c->kv_lora + 2 * ID);
- int ok = (mfull < 0 || vkc_mla_scratch(&ch->sc, &ch->mla[mfull], rows)) &&
+ int ok = (mfull < 0 || g_g53c_count >= 0 || vkc_mla_scratch(&ch->sc, &ch->mla[mfull], rows)) &&
g53c_res(&ch->xs, r * HD, VKC_DEV) && g53c_res(&ch->xn, r * HD, VKC_DEV) && g53c_res(&ch->col, r * D, VKC_DEV) &&
g53c_res(&ch->nrm, r * D, VKC_DEV) && g53c_res(&ch->br, r * D, VKC_DEV) && g53c_res(&ch->mix, r * nm, VKC_DEV) &&
g53c_res(&ch->hp, r * (2 * H + H * H), VKC_DEV) && g53c_res(&ch->gs, r * MI, VKC_DEV) &&
@@ -260,6 +298,16 @@ static int g53c_scratch(G53Chain *ch, const GModel *m, int rows, int ctx) {
ok = g53c_res(&ch->ikd, r * ID, VKC_DEV) && g53c_res(&ch->iq, r * IH * ID, VKC_DEV) && g53c_res(&ch->ihw, r * IH, VKC_DEV) &&
g53c_res(&ch->igd, r * ID, VKC_DEV) && g53c_res(&ch->isc, r * (size_t)(ctx / c->index_kpool + 1), VKC_DEV) &&
g53c_res(&ch->sel, r * (size_t)(1 + width), VKC_DEV);
+ if (g_g53c_count >= 0) { /* count the split scratch without allocating or changing capacities */
+ if (ch->ks.on && mfull >= 0)
+ g_g53c_count += (long long)r * (3LL * c->n_heads * (c->kv_lora + 2) +
+ (long long)c->n_heads * c->kv_lora + c->kv_lora + 1 +
+ coli_sparse_index_width(c->index_topk, c->index_kpool, c->index_kpool_tail)) * sizeof(float);
+ return ok;
+ }
+ if (ok && ch->ks.on && mfull >= 0) /* the split: the partial results, a step's new rows before their slots */
+ ok = vkc_kv_parts(&ch->ks, r * c->n_heads * (c->kv_lora + 2)) &&
+ (ch->kvtmp.cap >= rows || (g53c_res(&ch->kvtmp.lat, r * c->kv_lora, VKC_DEV) && (ch->kvtmp.cap = rows)));
if (!ok) return 0;
if (ch->rows < rows) {
float *hr = realloc(ch->host_routed, r * D * sizeof(float));
@@ -269,34 +317,41 @@ static int g53c_scratch(G53Chain *ch, const GModel *m, int rows, int ctx) {
return 1;
}
-/* The MLA mirror: room for `need` positions (the owner's), filled again when it grows. */
-static int g53c_mirror(G53Chain *ch, const GModel *m, int need, int limit) {
+/* The MLA mirror: room for `need` positions (the owner's), filled again when it grows,
+ * and planned again then (vkc_kv_plan): whole when it fits the device's budget, else the
+ * split (the latent rows; the index keys, pool gates and pooled keys stay whole), for
+ * steps of `rows` rows. */
+static int g53c_mirror(G53Chain *ch, const GModel *m, int need, int limit, int rows) {
const Cfg *c = &m->c; int L = c->n_layers, ID = c->index_hd, pool = c->index_kpool;
if (ch->cap >= need) return 1;
int cap = 256; while (cap < need) cap *= 2;
if (cap > limit) cap = limit;
if (cap < need) return 0;
+ size_t row = (size_t)c->kv_lora * sizeof(float);
+ if (ch->n_mla && !vkc_kv_plan(&ch->ks, "glm53", ch->n_mla, row, cap, rows, 1, (size_t)ch->n_mla * ch->dev_rows * row)) return 0;
+ int dr = ch->ks.on ? ch->ks.rows : cap;
for (int i = 0; i < L; i++) {
if (!c->is_full[i]) continue;
vkc_free(ch->kv[i].lat); vkc_free(ch->ik[i]); vkc_free(ch->ig[i]); vkc_free(ch->pk[i]);
ch->kv[i] = (VkcMlaCache){NULL, NULL, 0}; ch->ik[i] = ch->ig[i] = ch->pk[i] = NULL;
ch->kv_valid[i] = ch->pool_valid[i] = 0;
}
- ch->cap = 0;
+ ch->cap = 0; ch->dev_rows = 0;
for (int i = 0; i < L; i++) {
if (!c->is_full[i]) continue;
- ch->kv[i].lat = vkc_buf((size_t)cap * c->kv_lora * sizeof(float), VKC_DEV); ch->kv[i].cap = cap;
+ ch->kv[i].lat = vkc_buf((size_t)dr * c->kv_lora * sizeof(float), VKC_DEV); ch->kv[i].cap = dr;
ch->ik[i] = vkc_buf((size_t)cap * ID * sizeof(float), VKC_DEV);
ch->ig[i] = vkc_buf((size_t)cap * ID * sizeof(float), VKC_DEV);
ch->pk[i] = vkc_buf((size_t)(cap / pool + 1) * ID * sizeof(float), VKC_DEV);
if (!ch->kv[i].lat || !ch->ik[i] || !ch->ig[i] || !ch->pk[i]) return 0;
}
- ch->cap = cap;
+ ch->cap = cap; ch->dev_rows = dr;
return 1;
}
-/* Record the uploads that make the device the owner's below `start`. */
-static int g53c_push(G53Chain *ch, const GModel *m, const GSession *s, int start) {
+/* Record the uploads that make the device the owner's below `start`, for a step of
+ * n_rows rows (the split: its window placed first). */
+static int g53c_push(G53Chain *ch, const GModel *m, const GSession *s, int start, int n_rows) {
const Cfg *c = &m->c; int ok = 1, K = c->kv_lora, ID = c->index_hd;
if (ch->where == G53C_HOST) {
size_t ns = (size_t)c->kda_heads * c->kda_hd * c->kda_hd, nw = (size_t)3 * c->kda_proj * c->conv_k;
@@ -311,8 +366,13 @@ static int g53c_push(G53Chain *ch, const GModel *m, const GSession *s, int start
const GLayerState *ls = &s->layer[i];
int t0 = ch->kv_valid[i], n = start - t0;
if (ch->pool_valid[i] > start / c->index_kpool) ch->pool_valid[i] = start / c->index_kpool; /* rows from start change */
- if (n <= 0) continue;
- ok = vkc_write(ch->kv[i].lat, (size_t)t0 * K, ls->latent + (size_t)t0 * K, (size_t)n * K * sizeof(float)) &&
+ if (ch->ks.on) { /* the split: the window placed, its latent rows below start uploaded */
+ VkcKvPart pl = {1, K, ls->latent, 0, ch->kv[i].lat, 0};
+ vkc_kv_place(&ch->ks, ch->mla_ord[i], start, n_rows);
+ ok = vkc_kv_push(&ch->ks, ch->mla_ord[i], &pl, 1, start);
+ }
+ if (n <= 0 || !ok) continue;
+ ok = (ch->ks.on || vkc_write(ch->kv[i].lat, (size_t)t0 * K, ls->latent + (size_t)t0 * K, (size_t)n * K * sizeof(float))) &&
vkc_write(ch->ik[i], (size_t)t0 * ID, ls->ikeys + (size_t)t0 * ID, (size_t)n * ID * sizeof(float)) &&
vkc_write(ch->ig[i], (size_t)t0 * ID, ls->igates + (size_t)t0 * ID, (size_t)n * ID * sizeof(float));
ch->kv_valid[i] = start;
@@ -370,7 +430,7 @@ static int g53c_kda(G53Chain *ch, const GModel *m, const GLayer *l, int i, int n
vkc_kda_rec(KD, ch->cm, ch->kf, ch->kb, ch->kg, ch->prm, ch->st[i], ch->ky, &rp) &&
vkc_matmul(g53c_tensor(&l->ko), ch->ky, 0, ch->br, 0, n);
}
-static int g53c_mla(G53Chain *ch, const GModel *m, const GLayer *l, int i, int n, int pb) {
+static int g53c_mla(G53Chain *ch, const GModel *m, const GSession *s, const GLayer *l, int i, int n, int pb) {
const Cfg *c = &m->c; int IH = c->index_nh, ID = c->index_hd, pool = c->index_kpool, K = c->kv_lora;
int width = coli_sparse_index_width(c->index_topk, pool, c->index_kpool_tail);
size_t ko = (size_t)i * ch->kvd_layer;
@@ -379,7 +439,9 @@ static int g53c_mla(G53Chain *ch, const GModel *m, const GLayer *l, int i, int n
VkcDsaPool kp = {done > p0 ? done - p0 : 0, pool, p0, ID, 0, ID, (int)ch->o_ape[i], 0, 0, ID};
VkcDsaPick sp = {n, pb, IH, ID, c->index_topk, pool, 0, IH * ID, 0, IH, 0, c->index_kpool_tail, (pb + n) / pool + 1,
1 + width, sqrtf((float)IH), 1.f / sqrtf((float)ID)};
- int ok = vkc_mla_qkv(&ch->mla[i], &ch->sc, ch->nrm, 0, n, pb, NULL, &ch->kv[i], ch->kvd, ko) &&
+ int ok = (ch->ks.on ? vkc_kv_mla_qkv(&ch->ks, ch->mla_ord[i], &ch->mla[i], &ch->sc, ch->nrm, 0, n, pb, NULL, &ch->kv[i],
+ &ch->kvtmp, ch->kvd, ko)
+ : vkc_mla_qkv(&ch->mla[i], &ch->sc, ch->nrm, 0, n, pb, NULL, &ch->kv[i], ch->kvd, ko)) &&
vkc_matmul(g53c_tensor(&l->iwq), ch->sc.qa, 0, ch->iq, 0, n) &&
vkc_matmul(g53c_tensor(&l->iwk), ch->nrm, 0, ch->ikd, 0, n) &&
vkc_mla_lnorm(ch->ikd, ch->prm, ch->ik[i], &ln) &&
@@ -390,7 +452,9 @@ static int g53c_mla(G53Chain *ch, const GModel *m, const GLayer *l, int i, int n
vkc_copy(ch->kvd, ko + (size_t)n * (K + ID), ch->igd, 0, (size_t)n * ID) &&
vkc_dsa_pool_keys(ch->ik[i], ch->ig[i], ch->prm, ch->pk[i], &kp) &&
vkc_dsa_pool_select(ch->iq, ch->ihw, ch->pk[i], ch->isc, ch->sel, &sp) &&
- vkc_mla_attn(&ch->mla[i], &ch->sc, n, pb, 0, &ch->kv[i], ch->sel, 0, 1 + width, NULL, 0, ch->br, 0);
+ (ch->ks.on ? vkc_kv_mla_attn(&ch->ks, ch->mla_ord[i], &ch->mla[i], &ch->sc, n, pb, 0, &ch->kv[i], ch->sel, 0, 1 + width,
+ NULL, 0, ch->br, 0, s->layer[i].latent, NULL)
+ : vkc_mla_attn(&ch->mla[i], &ch->sc, n, pb, 0, &ch->kv[i], ch->sel, 0, 1 + width, NULL, 0, ch->br, 0));
if (ok && done > ch->pool_valid[i]) ch->pool_valid[i] = done;
return ok;
}
@@ -427,6 +491,15 @@ static void g53c_recover(GModel *m, GSession *s, int start, int dev) {
free(xs); free(xn);
}
+/* The prompt block forward_prefill hands the chain: its chunk, when the chain runs with
+ * a chunk from the budget (vkc_chunk_auto); 0 = forward_prefill's own (GLM53_PREFILL_CHUNK). */
+static int g53c_prefill_rows(GModel *m, GSession *s, int n) {
+ G53Chain *ch = g_g53c;
+ if (!g_vk_chain || !ch || !ch->ok || ch->failed || !vkc_chunk_auto()) return 0;
+ if (!g53c_mirror(ch, m, s->filled + n, s->cap, 1)) return 0;
+ int rows = g53c_rows(ch, m, s->cap);
+ return ch->ks.on && rows > ch->ks.chunk ? ch->ks.chunk : rows;
+}
/* Every layer for n rows of `streams` (the session's positions start..), the final
* streams back into it. 0 = not taken: the CPU runs the layers (streams untouched). */
static int g53c_forward(GModel *m, GSession *s, float *streams, int n, int start) {
@@ -438,10 +511,14 @@ static int g53c_forward(GModel *m, GSession *s, float *streams, int n, int start
if (ch->owner) g53c_sync_host(m, ch->owner);
ch->owner = s; ch->where = G53C_HOST;
for (int i = 0; i < L; i++) ch->kv_valid[i] = ch->pool_valid[i] = 0;
+ vkc_kv_reset(&ch->ks);
}
- int CH = g53c_rows(), rows = n < CH ? n : CH, dev0 = ch->where == G53C_DEV;
+ int dev0 = ch->where == G53C_DEV;
if (vkc_lost()) { g53c_recover(m, s, start, dev0); return 0; }
- if (!g53c_mirror(ch, m, start + n, s->cap) || !g53c_scratch(ch, m, rows, start + n)) {
+ int mirror_ok = g53c_mirror(ch, m, start + n, s->cap, 1);
+ int CH = mirror_ok ? g53c_rows(ch, m, s->cap) : 1, rows = n < CH ? n : CH;
+ if (ch->ks.on && rows > ch->ks.chunk) rows = ch->ks.chunk; /* a step's rows fit the split's window */
+ if (!mirror_ok || !g53c_scratch(ch, m, rows, start + n)) {
fprintf(stderr, "[VK] glm53 chain: device memory for %d rows at %d positions refused; per-matrix path\n", rows, start + n);
g53c_cpu_step(m, s, start);
ch->failed = 1; g_vk_chain = 0;
@@ -472,7 +549,7 @@ static int g53c_forward(GModel *m, GSession *s, float *streams, int n, int start
if (!outs) return 0;
for (int c0 = 0; c0 < n; c0 += rows) {
int nr = n - c0 < rows ? n - c0 : rows, pb = start + c0;
- if (!vkc_begin() || !g53c_push(ch, m, s, pb) || !vkc_write(ch->xs, 0, streams + (size_t)c0 * HD, (size_t)nr * HD * sizeof(float)))
+ if (!vkc_begin() || !g53c_push(ch, m, s, pb, nr) || !vkc_write(ch->xs, 0, streams + (size_t)c0 * HD, (size_t)nr * HD * sizeof(float)))
goto lost;
int ok = 1, pending = 0, pulled = 0;
for (int i = 0; i < L && ok; i++) {
@@ -484,7 +561,7 @@ static int g53c_forward(GModel *m, GSession *s, float *streams, int n, int start
}
double ta = now_s();
ok = ok && g53c_pre(ch, m, ch->fna[i], ch->o_hca[i], ch->o_in[i], nr) &&
- (c->is_full[i] ? g53c_mla(ch, m, l, i, nr, pb) : g53c_kda(ch, m, l, i, nr)) &&
+ (c->is_full[i] ? g53c_mla(ch, m, s, l, i, nr, pb) : g53c_kda(ch, m, l, i, nr)) &&
g53c_post(ch, m, nr) &&
g53c_pre(ch, m, ch->fnf[i], ch->o_hcf[i], ch->o_post[i], nr);
if (!ok) break;
@@ -493,7 +570,11 @@ static int g53c_forward(GModel *m, GSession *s, float *streams, int n, int start
m->t_attn += now_s() - ta;
continue;
}
- ok = vkc_copy(ch->h2d, 0, ch->nrm, 0, (size_t)nr * D) && vkc_submit(1); /* A1 */
+ /* A1; while it runs, the tier loads the experts this layer will likely
+ * stream (a big prompt chunk only) */
+ ok = vkc_copy(ch->h2d, 0, ch->nrm, 0, (size_t)nr * D) && vkc_submit(0);
+ if (ok) vkt_stream_prefetch(i, nr);
+ ok = ok && vkc_finish();
m->t_attn += now_s() - ta;
if (!ok) break;
g53c_pull(ch, m, s, pulled, i + 1, pb, nr); pulled = i + 1;
@@ -514,7 +595,7 @@ static int g53c_forward(GModel *m, GSession *s, float *streams, int n, int start
if (!ok) goto lost;
g53c_pull(ch, m, s, pulled, L, pb, nr);
memcpy(outs + (size_t)c0 * HD, vkc_ptr(ch->xd), (size_t)nr * HD * sizeof(float));
- for (int i = 0; i < L; i++) if (c->is_full[i]) ch->kv_valid[i] = pb + nr;
+ for (int i = 0; i < L; i++) if (c->is_full[i]) { ch->kv_valid[i] = pb + nr; vkc_kv_done(&ch->ks, ch->mla_ord[i], pb + nr); }
ch->where = G53C_DEV;
}
memcpy(streams, outs, (size_t)n * HD * sizeof(float));
@@ -535,6 +616,7 @@ static void g53c_report(void) {
fprintf(stderr, "[VK] glm53 chain: %llu forwards, %llu frames (%llu ops, %llu matmuls, %llu tiled GEMM), "
"%.1f ms waiting for the device, %.1f ms of routed experts on the host, %.1f MiB on the device\n",
ch->forwards, st.frames, st.ops, st.matmuls, st.gemms, st.wait_ms, ch->host_ms, st.dev_bytes / 1048576.0);
+ vkc_kv_report(&ch->ks);
vkc_prof_print();
}
diff --git a/c/glm_chain.h b/c/glm_chain.h
index 4838a27b..57ae7349 100644
--- a/c/glm_chain.h
+++ b/c/glm_chain.h
@@ -35,6 +35,11 @@
* (glmc_kv_reset), a slot adopting another's rows (glmc_host_rows), a different KV
* state bound (the owner check). The mirror grows with the context (powers of two
* up to the host's max_t), and a grown mirror is filled again from the host.
+ * Past the device's budget (vk_kvsplit.h) each layer keeps only some blocks of its
+ * latent and rope rows on the device: the window of the newest, and the blocks the DSA
+ * selection reads most. The attention core then runs over those on the device and over
+ * the rest on the CPU (from the host's canonical cache) at once, merged through the
+ * softmax statistics; the index keys, which every row scores every step, stay whole.
* MLA has no recurrent state: a rejected draft is rows the next step rewrites, prefix
* reuse is rows below the watermark, and a device lost mid-forward leaves nothing to
* rebuild: the forward runs again on the CPU from its input rows (still on the host)
@@ -50,11 +55,15 @@
* above 128, qk_nope above 1024; an indexer above 64 heads or 4096 query floats). The
* MTP head runs on the CPU, as before: its own KV row is not mirrored. */
#include "vk_chain.h"
+#include "vk_kvsplit.h"
typedef struct {
int ok, failed;
int rows; /* scratch capacity in rows */
- int cap; /* device KV rows a layer */
+ int cap; /* host positions the mirror covers */
+ int dev_rows; /* device latent/rope rows a layer: cap, or the split's */
+ VkcKvSplit ks; /* the split past the device's budget (ks.on) */
+ VkcMlaCache kvtmp; /* the split: a step's new rows before their slots */
KVState *owner; /* the KV state the mirror holds */
VkcBuf *prm; /* norm weights */
size_t *o_in, *o_post, *o_ixw, *o_ixb;
@@ -71,11 +80,6 @@ typedef struct {
static int g_vk_chain = 0; /* COLI_VK_CHAIN decided on, and the chain's pipelines are up */
static int g_glmc_inited = 0; /* vkc_init ran: vkc_shutdown at exit, whatever came after */
-static int glmc_chunk_rows(void) {
- const char *e = getenv("COLI_VK_CHAIN_ROWS");
- int v = e && *e ? atoi(e) : 512;
- return v < 1 ? 1 : v > 65535 ? 65535 : v;
-}
/* The device's copy of a resident QT, uploaded once (the per-matrix path's own copy
* when COLI_VK_DENSE put it there); f32 goes up as the backend's fmt 10. NULL: no
@@ -104,19 +108,24 @@ static void glmc_cpu_rows(Model *m, int layer, KVState *const *kvs, const int *p
KVState *ks = kvs ? kvs[s] : m->kv;
int pos = positions ? positions[s] : pos_base + s;
if (ks == ch->owner && ch->kv_valid[layer] > pos) ch->kv_valid[layer] = pos;
+ if (ks == ch->owner) vkc_kv_lower(&ch->ks, layer, pos);
}
}
/* Rows from `from` of every layer of k were written on the host. */
static void glmc_host_rows(Model *m, KVState *k, int from) {
GlmChain *ch = (GlmChain *)m->vkchain;
if (!ch || !ch->kv_valid || k != ch->owner) return;
- for (int i = 0; i < m->c.n_layers; i++) if (ch->kv_valid[i] > from) ch->kv_valid[i] = from < 0 ? 0 : from;
+ for (int i = 0; i < m->c.n_layers; i++) {
+ if (ch->kv_valid[i] > from) ch->kv_valid[i] = from < 0 ? 0 : from;
+ vkc_kv_lower(&ch->ks, i, from < 0 ? 0 : from);
+ }
}
/* kv_alloc gave the bound state new arrays: nothing on the device is theirs. */
static void glmc_kv_reset(Model *m) {
GlmChain *ch = (GlmChain *)m->vkchain;
if (!ch || !ch->kv_valid) return;
for (int i = 0; i < m->c.n_layers; i++) ch->kv_valid[i] = 0;
+ vkc_kv_reset(&ch->ks);
ch->owner = NULL;
}
@@ -185,7 +194,33 @@ static int glmc_setup(Model *m) {
return 1;
}
-static int glmc_res(VkcBuf **b, size_t floats, int kind) { return vkc_reserve(b, (floats ? floats : 1) * sizeof(float), kind); }
+/* glmc_res counts instead of reserving while g_glmc_count >= 0 (the chunk's sizing) */
+static long long g_glmc_count = -1;
+static int glmc_res(VkcBuf **b, size_t floats, int kind) {
+ if (g_glmc_count >= 0) { g_glmc_count += (long long)(floats ? floats : 1) * (long long)sizeof(float); return 1; }
+ return vkc_reserve(b, (floats ? floats : 1) * sizeof(float), kind);
+}
+static int glmc_scratch(GlmChain *ch, Model *m, int rows, int ctx);
+/* Prompt rows per chunk (vkc_chunk_rows): the chain's scratch a row, counted from the
+ * reservations (the MLA scratch as vkc_mla_scratch sizes it; the DSA scores at the
+ * model's context), and the routed experts' outputs (the tier's rows, the CPU's
+ * contributions, and the host's sum). */
+static int glmc_chunk_rows(GlmChain *ch, Model *m) {
+ if (!vkc_chunk_auto()) return vkc_chunk_rows("colibri", 0);
+ const VkcMla *a = &ch->mla[0];
+ size_t mla = (size_t)(a->q_lora > 0 ? a->q_lora : 1) + (size_t)a->H * (a->Q + a->R) + (size_t)(a->K + a->R) +
+ (size_t)a->H * a->K + (size_t)a->H * a->K + (size_t)a->H * a->V;
+ size_t ksz = (size_t)ch->kvd_layer;
+ g_glmc_count = 0; glmc_scratch(ch, m, 1, m->max_t); long long b1 = g_glmc_count;
+ g_glmc_count = 0; glmc_scratch(ch, m, 2, m->max_t); long long b2 = g_glmc_count;
+ g_glmc_count = -1; ch->kvd_layer = ksz;
+ size_t row = (size_t)(b2 - b1) + mla * sizeof(float) + (size_t)(2 * m->c.topk + 2) * m->c.hidden * sizeof(float);
+ /* moe_vk's CPU expert gather/gate/up/output workspace, plus the optional
+ * second device's packed inputs and outputs. */
+ row += 2 * ((size_t)m->c.hidden + m->c.moe_inter) * sizeof(float);
+ if (g_vk_reg_n2 > 0) row += 2 * (size_t)m->c.topk * m->c.hidden * sizeof(float);
+ return vkc_chunk_rows("colibri", row);
+}
/* scratch for `rows` rows at contexts up to `ctx` positions */
static int glmc_scratch(GlmChain *ch, Model *m, int rows, int ctx) {
@@ -193,18 +228,27 @@ static int glmc_scratch(GlmChain *ch, Model *m, int rows, int ctx) {
int SI = c->moe_inter * c->n_shared, DI = c->dense_inter, MI = SI > DI ? SI : DI;
size_t r = (size_t)rows;
ch->kvd_layer = r * (c->kv_lora + c->qk_rope + (m->has_dsa ? ID : 0));
- int ok = vkc_mla_scratch(&ch->sc, &ch->mla[0], rows) &&
+ int ok = (g_glmc_count >= 0 || vkc_mla_scratch(&ch->sc, &ch->mla[0], rows)) &&
glmc_res(&ch->x, r * D, VKC_DEV) && glmc_res(&ch->nrm, r * D, VKC_DEV) && glmc_res(&ch->tmp, r * D, VKC_DEV) &&
glmc_res(&ch->h2, r * D, VKC_DEV) && glmc_res(&ch->gs, r * MI, VKC_DEV) && glmc_res(&ch->us, r * MI, VKC_DEV) &&
glmc_res(&ch->hs, r * MI, VKC_DEV) && glmc_res(&ch->ds, r * D, VKC_DEV) &&
glmc_res(&ch->h2d, r * D, VKC_DOWN) && glmc_res(&ch->kvd, (size_t)L * ch->kvd_layer, VKC_DOWN) &&
glmc_res(&ch->xd, r * D, VKC_DOWN) && glmc_res(&ch->routed, r * D, VKC_UP) &&
glmc_res(&ch->cs, r * (c->qk_rope > 0 ? c->qk_rope : 2), VKC_UP);
+ if (ok && ch->ks.on && g_glmc_count >= 0)
+ g_glmc_count += (long long)r * (3LL * c->n_heads * (c->kv_lora + 2) +
+ (long long)c->n_heads * (c->kv_lora + c->qk_rope) + c->kv_lora + c->qk_rope +
+ (m->has_dsa ? 1 + c->index_topk : 0)) * sizeof(float);
+ else if (ok && ch->ks.on) /* the split: the partial results, a step's new rows before their slots */
+ ok = vkc_kv_parts(&ch->ks, r * c->n_heads * (c->kv_lora + 2)) &&
+ (ch->kvtmp.cap >= rows || (glmc_res(&ch->kvtmp.lat, r * c->kv_lora, VKC_DEV) &&
+ (c->qk_rope == 0 || glmc_res(&ch->kvtmp.rope, r * c->qk_rope, VKC_DEV)) &&
+ (ch->kvtmp.cap = rows)));
if (ok && m->has_dsa)
ok = glmc_res(&ch->ikd, r * ID, VKC_DEV) && glmc_res(&ch->iq, r * IH * ID, VKC_DEV) &&
glmc_res(&ch->ihw, r * IH, VKC_DEV) && glmc_res(&ch->isc, r * (size_t)ctx, VKC_DEV) &&
glmc_res(&ch->sel, r * (1 + (size_t)c->index_topk), VKC_DEV);
- if (!ok) return 0;
+ if (!ok || g_glmc_count >= 0) return ok; /* counting: the buffers only */
if (ch->rows < rows) {
float *hr = realloc(ch->host_routed, r * D * sizeof(float));
if (!hr) return 0;
@@ -214,34 +258,50 @@ static int glmc_scratch(GlmChain *ch, Model *m, int rows, int ctx) {
}
/* The device's KV mirror: room for `need` positions of the bound state, and the
- * watermarks reset when the state is not the one mirrored. */
-static int glmc_mirror(GlmChain *ch, Model *m, int need) {
+ * watermarks reset when the state is not the one mirrored. A mirror that grows is
+ * planned again (vkc_kv_plan): whole when it fits the device's budget, else the split
+ * (latent and rope rows; the index keys stay whole), for steps of `rows` rows. */
+static int glmc_mirror(GlmChain *ch, Model *m, int need, int rows) {
Cfg *c = &m->c; int L = c->n_layers;
- if (ch->owner != m->kv) { for (int i = 0; i < L; i++) ch->kv_valid[i] = 0; ch->owner = m->kv; }
+ if (ch->owner != m->kv) { for (int i = 0; i < L; i++) ch->kv_valid[i] = 0; ch->owner = m->kv; vkc_kv_reset(&ch->ks); }
if (ch->cap >= need) return 1;
int cap = 256; while (cap < need) cap *= 2;
if (cap > m->max_t) cap = m->max_t;
if (cap < need) return 0;
+ size_t row = (size_t)(c->kv_lora + c->qk_rope) * sizeof(float);
+ if (!vkc_kv_plan(&ch->ks, "colibri", L, row, cap, rows, m->has_dsa, (size_t)L * ch->dev_rows * row)) return 0;
+ int dr = ch->ks.on ? ch->ks.rows : cap;
for (int i = 0; i < L; i++) {
vkc_free(ch->kv[i].lat); vkc_free(ch->kv[i].rope); vkc_free(ch->ik[i]);
ch->kv[i] = (VkcMlaCache){NULL, NULL, 0}; ch->ik[i] = NULL; ch->kv_valid[i] = 0;
}
- ch->cap = 0;
+ ch->cap = 0; ch->dev_rows = 0;
for (int i = 0; i < L; i++) {
- ch->kv[i].lat = vkc_buf((size_t)cap * c->kv_lora * sizeof(float), VKC_DEV);
- ch->kv[i].rope = c->qk_rope > 0 ? vkc_buf((size_t)cap * c->qk_rope * sizeof(float), VKC_DEV) : NULL;
- ch->kv[i].cap = cap;
+ ch->kv[i].lat = vkc_buf((size_t)dr * c->kv_lora * sizeof(float), VKC_DEV);
+ ch->kv[i].rope = c->qk_rope > 0 ? vkc_buf((size_t)dr * c->qk_rope * sizeof(float), VKC_DEV) : NULL;
+ ch->kv[i].cap = dr;
if (glmc_full(m, i)) ch->ik[i] = vkc_buf((size_t)cap * c->index_hd * sizeof(float), VKC_DEV);
if (!ch->kv[i].lat || (c->qk_rope > 0 && !ch->kv[i].rope) || (glmc_full(m, i) && !ch->ik[i])) return 0;
}
- ch->cap = cap;
+ ch->cap = cap; ch->dev_rows = dr;
return 1;
}
-/* Record the uploads that make the mirror the host's below pos_base. */
-static int glmc_push_kv(GlmChain *ch, Model *m, int pos_base) {
+/* Record the uploads that make the mirror the host's below pos_base, for a step of
+ * n_rows rows (the split: its window placed first). */
+static int glmc_push_kv(GlmChain *ch, Model *m, int pos_base, int n_rows) {
Cfg *c = &m->c; int ok = 1, K = c->kv_lora, R = c->qk_rope, ID = c->index_hd;
for (int i = 0; i < c->n_layers && ok; i++) {
+ if (ch->ks.on) { /* the split: the window placed, its rows below pos_base uploaded */
+ VkcKvPart pt[2] = {{1, K, m->Lc[i], 0, ch->kv[i].lat, 0}, {1, R, m->Rc[i], 0, ch->kv[i].rope, 0}};
+ vkc_kv_place(&ch->ks, i, pos_base, n_rows);
+ ok = vkc_kv_push(&ch->ks, i, pt, R ? 2 : 1, pos_base);
+ int t0 = ch->kv_valid[i], n = pos_base - t0; /* the index keys stay whole */
+ if (ok && n > 0 && glmc_full(m, i))
+ ok = vkc_write(ch->ik[i], (size_t)t0 * ID, coli_kv_row(m->Ic[i], t0, ID), (size_t)n * ID * sizeof(float));
+ ch->kv_valid[i] = pos_base;
+ continue;
+ }
int t0 = ch->kv_valid[i], n = pos_base - t0;
if (n <= 0) continue;
ok = vkc_write(ch->kv[i].lat, (size_t)t0 * K, coli_kv_row(m->Lc[i], t0, K), (size_t)n * K * sizeof(float)) &&
@@ -317,13 +377,15 @@ static int glmc_forward(Model *m, float *xh, int S, int pos_base) {
Cfg *c = &m->c; int D = c->hidden, L = c->n_layers, R = c->qk_rope;
if (!m->Lc || !m->Rc || (m->has_dsa && !m->Ic)) return 0;
for (int i = 0; i < L; i++) if (m->kv_start[i] != 0 || !m->Lc[i]) return 0;
- int CH = glmc_chunk_rows();
+ int mirror_ok = glmc_mirror(ch, m, pos_base + S, 1);
+ int CH = mirror_ok ? glmc_chunk_rows(ch, m) : 1;
if (m->has_dsa && (pos_base + S > c->index_topk || g_dsa_force)) { /* the scores' scratch: rows x context */
int64_t lim = ((int64_t)32 << 20) / (pos_base + S);
if (lim < CH) CH = lim < 1 ? 1 : (int)lim;
}
int rows = S < CH ? S : CH;
- if (!glmc_mirror(ch, m, pos_base + S) || !glmc_scratch(ch, m, rows, pos_base + S)) {
+ if (ch->ks.on && rows > ch->ks.chunk) rows = ch->ks.chunk; /* a step's rows fit the split's window */
+ if (!mirror_ok || !glmc_scratch(ch, m, rows, pos_base + S)) {
fprintf(stderr, "[VK] colibri chain: device memory for %d rows at %d positions refused; per-matrix path\n",
rows, pos_base + S);
ch->failed = 1;
@@ -345,7 +407,7 @@ static int glmc_forward(Model *m, float *xh, int S, int pos_base) {
cs[0] = cosf(ang); cs[1] = sinf(ang);
}
if (!vkc_begin() || !vkc_write(ch->x, 0, xh + (size_t)c0 * D, (size_t)n * D * sizeof(float)) ||
- !glmc_push_kv(ch, m, pb)) goto lost;
+ !glmc_push_kv(ch, m, pb, n)) goto lost;
int ok = 1, pending = 0, pulled = 0, sel_on = 0;
for (int i = 0; i < L && ok; i++) {
Layer *l = &m->L[i];
@@ -353,10 +415,16 @@ static int glmc_forward(Model *m, float *xh, int S, int pos_base) {
for (int z = 0; z < m->enr[i]; z++) if (!(n <= 4 && vk_reg_served(i, m->eroute[i][z]))) expert_prefetch(m, i, m->eroute[i][z]);
if (pending) { ok = glmc_combine(ch, m, n); pending = 0; }
ok = ok && glmc_norm(ch->x, ch->prm, ch->o_in[i], ch->nrm, n, D, c->eps) &&
- vkc_mla_qkv(&ch->mla[i], &ch->sc, ch->nrm, 0, n, pb, ch->cs, &ch->kv[i], ch->kvd, (size_t)i * ch->kvd_layer);
+ (ch->ks.on ? vkc_kv_mla_qkv(&ch->ks, i, &ch->mla[i], &ch->sc, ch->nrm, 0, n, pb, ch->cs, &ch->kv[i], &ch->kvtmp,
+ ch->kvd, (size_t)i * ch->kvd_layer)
+ : vkc_mla_qkv(&ch->mla[i], &ch->sc, ch->nrm, 0, n, pb, ch->cs, &ch->kv[i], ch->kvd,
+ (size_t)i * ch->kvd_layer));
if (ok && glmc_full(m, i)) ok = glmc_dsa(ch, m, i, n, pb, &sel_on);
- ok = ok && vkc_mla_attn(&ch->mla[i], &ch->sc, n, pb, 0, &ch->kv[i], sel_on ? ch->sel : NULL, 0,
- 1 + c->index_topk, NULL, 0, ch->tmp, 0);
+ if (ok && ch->ks.on) /* the split: the core over the device's blocks and the host's rows */
+ ok = vkc_kv_mla_attn(&ch->ks, i, &ch->mla[i], &ch->sc, n, pb, 0, &ch->kv[i], sel_on ? ch->sel : NULL, 0,
+ 1 + c->index_topk, NULL, 0, ch->tmp, 0, m->Lc[i], m->Rc[i]);
+ else ok = ok && vkc_mla_attn(&ch->mla[i], &ch->sc, n, pb, 0, &ch->kv[i], sel_on ? ch->sel : NULL, 0,
+ 1 + c->index_topk, NULL, 0, ch->tmp, 0);
VkcEw add = {VKC_EW_ADD, n * D, D, 1, 0, 1, 0, 0, 0, 0, 0, 1.f};
ok = ok && vkc_ew(ch->x, ch->x, ch->tmp, NULL, NULL, &add) &&
glmc_norm(ch->x, ch->prm, ch->o_post[i], ch->h2, n, D, c->eps);
@@ -366,7 +434,11 @@ static int glmc_forward(Model *m, float *xh, int S, int pos_base) {
vkc_ew(ch->x, ch->x, ch->tmp, NULL, NULL, &add);
continue;
}
- ok = vkc_copy(ch->h2d, 0, ch->h2, 0, (size_t)n * D) && vkc_submit(1); /* A1 */
+ /* A1; while it runs, the tier loads the experts this layer will likely
+ * stream (a big prompt chunk only) */
+ ok = vkc_copy(ch->h2d, 0, ch->h2, 0, (size_t)n * D) && vkc_submit(0);
+ if (ok) vkt_stream_prefetch(i, n);
+ ok = ok && vkc_finish();
if (!ok) break;
glmc_pull_kv(ch, m, pulled, i + 1, pb, n); pulled = i + 1;
/* A2: the shared expert, while the host computes the routed experts */
@@ -383,7 +455,7 @@ static int glmc_forward(Model *m, float *xh, int S, int pos_base) {
if (!ok) goto lost;
glmc_pull_kv(ch, m, pulled, L, pb, n);
memcpy(outs + (size_t)c0 * D, vkc_ptr(ch->xd), (size_t)n * D * sizeof(float));
- for (int i = 0; i < L; i++) ch->kv_valid[i] = pb + n;
+ for (int i = 0; i < L; i++) { ch->kv_valid[i] = pb + n; vkc_kv_done(&ch->ks, i, pb + n); }
}
memcpy(xh, outs, (size_t)S * D * sizeof(float));
free(inv); free(outs);
@@ -405,6 +477,7 @@ static void glmc_report(Model *m) {
fprintf(stderr, "[VK] colibri chain: %llu forwards, %llu frames (%llu ops, %llu matmuls, %llu tiled GEMM), "
"%.1f ms waiting for the device, %.1f ms of routed experts on the host, %.1f MiB on the device\n",
ch->forwards, st.frames, st.ops, st.matmuls, st.gemms, st.wait_ms, ch->host_ms, st.dev_bytes / 1048576.0);
+ vkc_kv_report(&ch->ks);
vkc_prof_print();
}
static Model *g_glmc_model;
diff --git a/c/inkling.c b/c/inkling.c
index 81a9de31..0b89b021 100644
--- a/c/inkling.c
+++ b/c/inkling.c
@@ -80,6 +80,7 @@ static int g_metal = 0;
static int g_vk_ready = 0;
static int g_vk_tier_try = 0; /* the device opened with the expert tier to be tried */
static int g_vk_chain = 0; /* COLI_VK_CHAIN decided on, and the chain's pipelines are up (inkling_chain.h) */
+static int g_ink_dho = 0; /* COLI_VK_DENSE_HOST: the dense matrices on the device only (ink_dho_place) */
#endif
#if defined(__APPLE__)
#include <mach/mach.h>
@@ -128,6 +129,9 @@ typedef struct { float *f; uint16_t *h; void *dev;
#ifdef COLI_VULKAN
void *vk; /* InkVk of the loaded tensor, shared by its wt_off_i
* views (NULL = CPU only) */
+ int64_t vk_view; /* this view's first element in the tensor (wt_off_i):
+ * the key of its device copy, whatever the host holds */
+ const char *vk_name;/* the tensor's name, to read it back (ink_dho_reload) */
#endif
} Wt;
@@ -428,8 +432,10 @@ static void matmul_i8r(float *y, const float *x, const int8_t *q, const float *s
* cache cannot live in the copy matmul_w receives: it lives in a table the
* loaded Wt points to and every view copies, one entry per view (keyed by the
* view's first weight). Released with the tensor it belongs to. */
-typedef struct { const void *q; ColiVkTensor *t; int dead; } InkVkView;
-typedef struct { int n; InkVkView e[]; } InkVk;
+typedef struct { int64_t view; int used; ColiVkTensor *t; int dead; } InkVkView;
+/* gone: the device holds the tensor alone (COLI_VK_DENSE_HOST): fmt and gs are the ones
+ * its views went up in, home the loaded Wt the CPU reads it back into. */
+typedef struct { int n, fmt, gs, gone; Wt *home; InkVkView e[]; } InkVk;
/* The shader format that reads a weight exactly as the CPU kernel does, 0 for none.
* int8 per row (matmul_i8r) is fmt 1; int4 group-scaled with +8 nibbles and the low
@@ -450,7 +456,7 @@ static int ink_vk_attach(Wt *w, int views) {
if (!fmt || w->vk || views < 1) return 0;
InkVk *v = calloc(1, sizeof(*v) + (size_t)views * sizeof(v->e[0]));
if (!v) return 0;
- v->n = views;
+ v->n = views; v->fmt = fmt; v->gs = w->q4 ? w->gs : 0; v->home = w;
w->vk = v;
return fmt;
}
@@ -463,28 +469,49 @@ static void ink_vk_release(Wt *w) {
}
/* 1 when y was computed on the device. One command buffer in the backend:
* main thread only, never from inside a parallel region. */
+/* The entry of a view (keyed by its first element in the tensor), claimed on first use;
+ * NULL when every entry belongs to another view. */
+static InkVkView *ink_vk_entry(InkVk *v, int64_t view) {
+ for (int k = 0; k < v->n; k++) {
+ InkVkView *e = &v->e[k];
+ if (e->used && e->view != view) continue;
+ e->used = 1; e->view = view;
+ return e;
+ }
+ return NULL;
+}
static int ink_vk_matmul(float *y, const float *x, const Wt *W, int S, int I, int O) {
InkVk *v = W->vk;
if (!g_vk_ready || !v) return 0;
#ifdef _OPENMP
if (omp_in_parallel()) return 0;
#endif
- int fmt = ink_vk_fmt(W);
+ int fmt = v->gone ? v->fmt : ink_vk_fmt(W);
const void *data = W->q4 ? (const void *)W->q4 : W->f ? (const void *)W->f : (const void *)W->h;
- if (!fmt || !data) return 0;
- for (int k = 0; k < v->n; k++) {
- InkVkView *e = &v->e[k];
- if (e->q && e->q != data) continue;
- if (e->dead) return 0;
- e->q = data;
- if (coli_vk_matmul(&e->t, y, x, data, W->q4 ? W->qs : NULL, fmt, S, I, O,
- W->q4 ? W->gs : 0))
- return 1;
- if (!e->t) e->dead = 1; /* refused at upload: this view stays on the CPU */
- return 0;
- }
+ if (!fmt || (!data && !v->gone)) return 0;
+ InkVkView *e = ink_vk_entry(v, W->vk_view);
+ if (!e || e->dead || (v->gone && !e->t)) return 0; /* gone: only a copy made before the drop serves */
+ if (coli_vk_matmul(&e->t, y, x, data, W->q4 ? W->qs : NULL, fmt, S, I, O, v->gs))
+ return 1;
+ if (!e->t) e->dead = 1; /* refused at upload: this view stays on the CPU */
return 0;
}
+/* A view's device copy, made now if it is not there yet (the chain's tensors and the
+ * eager upload of COLI_VK_DENSE_HOST); NULL when the view has none. */
+static ColiVkTensor *ink_vk_view_tensor(const Wt *W, int I, int O) {
+ InkVk *v = W->vk;
+ if (!v) return NULL;
+ int fmt = v->gone ? v->fmt : ink_vk_fmt(W);
+ const void *data = W->q4 ? (const void *)W->q4 : W->f ? (const void *)W->f : (const void *)W->h;
+ if (!fmt || (!data && !v->gone)) return NULL;
+ InkVkView *e = ink_vk_entry(v, W->vk_view);
+ if (!e || e->dead) return NULL;
+ if (!e->t && (v->gone || !coli_vk_tensor_ensure(&e->t, data, W->q4 ? W->qs : NULL, fmt, I, O, v->gs))) {
+ e->dead = 1; return NULL;
+ }
+ return e->t;
+}
+static Wt ink_dho_reload(Wt W, int I); /* below, beside load_w */
static void ink_vk_report(void) {
if (g_vk_ready)
fprintf(stderr, "[VK] inkling: %llu matmuls on the GPU\n", coli_vk_matmul_calls());
@@ -498,6 +525,12 @@ static void matmul_w(float *y, const float *x, Wt W, int S, int I, int O) {
#ifdef COLI_INKLING_SHARED_BATCH_TEST
g_matmul_w_calls++;
#endif
+#ifdef COLI_VULKAN
+ if (W.vk && ((InkVk *)W.vk)->gone) { /* the device holds it alone */
+ if (ink_vk_matmul(y, x, &W, S, I, O)) return;
+ W = ink_dho_reload(W, I); /* the CPU needs it after all (a lost device): from disk */
+ }
+#endif
#ifdef COLI_CUDA
if (W.dev) {
if (ink_cuda_matmul_bf16(y, x, W.dev, S, I, O) == 0) return;
@@ -888,11 +921,24 @@ static int load_w_quant(Model *m, const char *name, int64_t orig_numel, Wt *out)
return 1;
}
+#ifdef COLI_VULKAN
+/* the names of loaded tensors, kept for the life of the process (a Wt travels by value) */
+static const char *ink_name_keep(const char *name) {
+ char *n = strdup(name);
+ if (!n) { fprintf(stderr, "OOM weight name\n"); exit(1); }
+ return n;
+}
+#endif
static Wt load_w(Model *m, const char *name, int gpu_ok) {
Wt w = {0};
st_tensor *t = st_find(&m->S, name);
if (!t) { fprintf(stderr, "missing %s\n", name); exit(1); }
+#ifdef COLI_VULKAN
+ if (load_w_quant(m, name, t->numel, &w)) { w.vk_name = ink_name_keep(name); return w; }
+ w.vk_name = ink_name_keep(name);
+#else
if (load_w_quant(m, name, t->numel, &w)) return w;
+#endif
if (t->dtype == 0) {
w.h = malloc(t->nbytes); if (!w.h) { fprintf(stderr,"OOM %s\n",name); exit(1); }
pread_all(t->fd, w.h, t->nbytes, t->off);
@@ -916,6 +962,9 @@ static Wt load_w(Model *m, const char *name, int gpu_ok) {
* st_tensor non porta le shape e non voglio indovinarle. */
static Wt wt_off_i(Wt w, int64_t off, int I) {
Wt r = w;
+#ifdef COLI_VULKAN
+ r.vk_view = w.vk_view + off;
+#endif
r.f = w.f ? w.f + off : NULL;
r.h = w.h ? w.h + off : NULL;
r.dev = w.dev ? (char*)w.dev + off*2 : NULL; /* dev is always bf16 */
@@ -1023,6 +1072,108 @@ static void ink_vk_mark_dense(Model *m, int layer_begin, int layer_end) {
* of which wins. The dense matrices then go where coli_vk_dense() says: on a device
* sharing the CPU's RAM with the tier on, they stay on the CPU unless COLI_VK_DENSE=1;
* if the tier does not start after all, ink_vk_tier_start marks them then. */
+/* ---- the dense matrices on the device only (COLI_VK_DENSE_HOST) ----------------
+ * Decided when the device opens (ink_vk_init_model), before the expert cache is sized:
+ * every resident matrix the shaders read goes up now, view by view (each shared expert
+ * of the fused tensors is a view), into the table the per-matrix path and the chain
+ * share, and its host copy is given back. The per-matrix path then answers on the device
+ * whatever coli_vk_dense() says, the chain finds the same copies, and the CPU reads a
+ * matrix back from disk only when it needs it after all (ink_dho_reload: a lost device).
+ * Kept on the host: the embedding and the audio table (row lookups), the routers (the
+ * chain holds its own copy, the host step reads them off the chain), norms, convolution
+ * taps and the bias banks, and bf16 matrices on a CPU whose bf16 dot rounds activations. */
+#ifndef INKLING_CHAIN_IGPU /* inkling_chain.h's default, needed here first */
+#define INKLING_CHAIN_IGPU COLI_VK_CHAIN_UNMEASURED
+#endif
+static Model *g_ink_dho_model;
+static pthread_mutex_t g_ink_dho_mx = PTHREAD_MUTEX_INITIALIZER;
+/* Host bytes of a loaded tensor of `n` elements, in the form RAM holds it. */
+static size_t ink_wt_bytes(const Wt *w, int64_t n, int I) {
+ if (w->q4) return (size_t)w->qn + (size_t)(w->qbits == 8 ? n / I : (n / I) * ((I + w->gs - 1) / w->gs)) * 4;
+ if (w->f) return (size_t)n * 4;
+ if (w->h) return (size_t)n * 2;
+ return 0;
+}
+/* The CPU needs the view W of a tensor the device holds alone: read the tensor back
+ * into its home Wt (the bytes load_w read the first time), and hand the view of it. */
+static Wt ink_dho_reload(Wt W, int I) {
+ InkVk *v = W.vk;
+ pthread_mutex_lock(&g_ink_dho_mx);
+ if (v->gone) {
+ Model *m = g_ink_dho_model;
+ Wt *h = v->home;
+ if (!m || !h->vk_name) { fprintf(stderr, "[VK] inkling: a dense matrix the device held alone cannot be read back\n"); exit(1); }
+ int q_loaded = m->q_loaded; int64_t q_bytes = m->q_bytes;
+ Wt t = load_w(m, h->vk_name, 0);
+ m->q_loaded = q_loaded; m->q_bytes = q_bytes;
+ if (!!t.q4 != (v->fmt == 1 || v->fmt == 4) || (t.q4 && t.qbits != h->qbits)) {
+ fprintf(stderr, "[VK] inkling: %s came back in another form\n", h->vk_name); exit(1);
+ }
+ size_t b = (size_t)t.qn;
+ h->f = t.f; h->h = t.h; h->q4 = t.q4; h->qs = t.qs; h->qn = t.qn;
+ free((void *)t.vk_name);
+ __atomic_store_n(&v->gone, 0, __ATOMIC_RELEASE);
+ coli_vk_dense_host_reloaded(b ? b : 1);
+ }
+ pthread_mutex_unlock(&g_ink_dho_mx);
+ return wt_off_i(*v->home, W.vk_view, I);
+}
+/* One loaded tensor up, view by view (`views` views `stride` elements apart, each
+ * [O, I]), then its host copy given back: all views or none. */
+static int ink_dho_drop(Wt *w, int views, int64_t stride, int I, int O, int *n) {
+ InkVk *v = w->vk;
+ if (!v || v->gone || !w->vk_name || (!w->q4 && !w->f && !w->h)) return 0;
+ for (int j = 0; j < views; j++) {
+ Wt view = wt_off_i(*w, (int64_t)j * stride, I);
+ if (!ink_vk_view_tensor(&view, I, O)) return 0;
+ }
+ size_t b = ink_wt_bytes(w, (int64_t)views * I * O, I);
+ free(w->f); free(w->h); free(w->q4); free(w->qs);
+ w->f = NULL; w->h = NULL; w->q4 = NULL; w->qs = NULL; /* qbits, gs and qn stay: the form it went up in */
+ v->gone = 1;
+ coli_vk_dense_host_dropped(b);
+ (*n)++;
+ return 1;
+}
+static size_t ink_dho_count(const Wt *w, int64_t n, int I) { return ink_vk_fmt(w) ? ink_wt_bytes(w, n, I) : 0; }
+static size_t ink_dho_bytes(const Model *m, int layer_begin, int layer_end) {
+ const Cfg *c = &m->c; int64_t D = c->hidden;
+ size_t b = ink_dho_count(&m->lm_head, (int64_t)c->unpad_vocab * D, (int)D);
+ for (int i = layer_begin; i < layer_end; i++) {
+ const Layer *l = &m->L[i];
+ int64_t q = (int64_t)L_HEADS(c, i) * L_HD(c, i), kv = (int64_t)L_KV(c, i) * L_HD(c, i), I = c->moe_inter;
+ b += ink_dho_count(&l->q, D * q, (int)D) + ink_dho_count(&l->k, D * kv, (int)D) + ink_dho_count(&l->v, D * kv, (int)D) +
+ ink_dho_count(&l->r, D * L_HEADS(c, i) * c->d_rel, (int)D) + ink_dho_count(&l->o, q * D, (int)q) +
+ ink_dho_count(&l->dg, D * c->dense_inter, (int)D) + ink_dho_count(&l->du, D * c->dense_inter, (int)D) +
+ ink_dho_count(&l->dd, D * c->dense_inter, c->dense_inter) +
+ ink_dho_count(&l->sh_g, c->n_shared * I * D, (int)D) + ink_dho_count(&l->sh_u, c->n_shared * I * D, (int)D) +
+ ink_dho_count(&l->sh_d, c->n_shared * I * D, (int)I);
+ }
+ return b;
+}
+static void ink_dho_place(Model *m, int layer_begin, int layer_end) {
+ const Cfg *c = &m->c; int D = c->hidden, n = 0;
+ ink_dho_drop(&m->lm_head, 1, 0, D, c->unpad_vocab, &n);
+ for (int i = layer_begin; i < layer_end; i++) {
+ Layer *l = &m->L[i];
+ int q = L_HEADS(c, i) * L_HD(c, i), kv = L_KV(c, i) * L_HD(c, i), I = c->moe_inter, ns = c->n_shared;
+ ink_dho_drop(&l->q, 1, 0, D, q, &n); ink_dho_drop(&l->k, 1, 0, D, kv, &n); ink_dho_drop(&l->v, 1, 0, D, kv, &n);
+ ink_dho_drop(&l->r, 1, 0, D, L_HEADS(c, i) * c->d_rel, &n); ink_dho_drop(&l->o, 1, 0, q, D, &n);
+ if (!c->sparse[i]) {
+ ink_dho_drop(&l->dg, 1, 0, D, c->dense_inter, &n); ink_dho_drop(&l->du, 1, 0, D, c->dense_inter, &n);
+ ink_dho_drop(&l->dd, 1, 0, c->dense_inter, D, &n);
+ } else if (ns > 0) {
+ ink_dho_drop(&l->sh_g, ns, (int64_t)I * D, D, I, &n); ink_dho_drop(&l->sh_u, ns, (int64_t)I * D, D, I, &n);
+ ink_dho_drop(&l->sh_d, ns, (int64_t)D * I, I, D, &n);
+ }
+ }
+ coli_vk_dense_host_placed("inkling", "the embedding and audio tables (row lookups), the routers, norms, convolution taps, bias banks"
+#ifdef HAVE_BF16_DOT
+ ", bf16 matrices (this CPU's bf16 dot rounds activations, the device would not)"
+#endif
+ );
+}
+
static void ink_vk_init_model(Model *m, int layer_begin, int layer_end) {
int sparse = 0, other_gpu = 0;
for (int i = layer_begin; i < layer_end; i++) sparse += m->c.sparse[i];
@@ -1036,6 +1187,17 @@ static void ink_vk_init_model(Model *m, int layer_begin, int layer_end) {
layer_begin == 0 && layer_end == m->c.n_layers;
if (!g_vk_ready) g_vk_ready = coli_vk_init_env_tier("inkling", g_vk_tier_try);
if (!g_vk_ready) { g_vk_tier_try = 0; return; }
+ /* COLI_VK_DENSE_HOST: with the dense part on the device (the per-matrix path, or the
+ * chain: asked here silently, inkc_start says it after the tier), every matrix the
+ * shaders read is marked and goes up now, its host copy given back */
+ int chain = !other_gpu && layer_begin == 0 && layer_end == m->c.n_layers &&
+ coli_vk_chain_decide(NULL, g_vk_tier_try, INKLING_CHAIN_IGPU) != COLI_VK_CHAIN_OFF;
+ if (coli_vk_dense_host_decide("inkling", (coli_vk_dense() || chain) && !other_gpu, ink_dho_bytes(m, layer_begin, layer_end))) {
+ g_ink_dho = 1; g_ink_dho_model = m;
+ ink_vk_mark_dense(m, layer_begin, layer_end);
+ ink_dho_place(m, layer_begin, layer_end);
+ return;
+ }
if (!coli_vk_dense()) return; /* nothing marked: the CPU computes the dense matrices */
ink_vk_mark_dense(m, layer_begin, layer_end);
}
@@ -1206,13 +1368,28 @@ static void model_init_range(Model *m, const char *snap, int cap, int bits,
int nsp = 0; for (int i = layer_begin; i < layer_end; i++) nsp += c->sparse[i];
int64_t slotb = m->xq ? m->rb13*2*I + m->rb2*D + (2*I+D)*4
: m->quant_bits ? 3*I*D + (2*I+D)*4 : 3*I*D*4;
+ double avail_auto = cap <= 0 ? mem_avail_bytes() : 0;
+#ifdef COLI_VULKAN
+ /* the device opens here, before the expert cache is sized: with the dense matrices on
+ * the device only (COLI_VK_DENSE_HOST) their host copies are gone before the auto cap
+ * measures what RAM is left (on a GPU that shares the RAM the device copy holds it) */
+ double t_vk = now_s();
+ ink_vk_init_model(m, layer_begin, layer_end);
+ t0 += now_s() - t_vk; /* the device's start is not the weights' load (dense_load_s), as before */
+ if (cap <= 0 && g_ink_dho) avail_auto = mem_avail_bytes();
+#endif
if (cap <= 0) { /* auto: fit the LRU in available RAM, 20% + 4 GB headroom */
- double avail = mem_avail_bytes();
+ double avail = avail_auto;
cap = avail > 0 ? (int)((avail*0.80 - 4e9) / ((double)slotb * (nsp ? nsp : 1))) : 16;
if (cap < 4) cap = 4;
if (cap > c->n_experts) cap = c->n_experts;
fprintf(stderr, "[cap auto] %d experts/layer (%.1f GB cache budget)\n",
cap, (double)cap*slotb*nsp/1e9);
+#ifdef COLI_VULKAN
+ if (g_ink_dho)
+ fprintf(stderr, "[cap auto] dense weights on the device only: RAM measured after %.3f GB of host copies "
+ "went (%.3f GB available)\n", coli_vk_dense_host_dropped_bytes() / 1e9, avail / 1e9);
+#endif
}
m->cache = calloc(c->n_layers, sizeof(LCache));
for (int i = layer_begin; i < layer_end; i++) {
@@ -1260,9 +1437,6 @@ static void model_init_range(Model *m, const char *snap, int cap, int bits,
m->eusage = rt_counts_all(); /* alias: the bump sites stay as they are */
}
m->dense_load_s = now_s() - t0;
-#ifdef COLI_VULKAN
- ink_vk_init_model(m, layer_begin, layer_end);
-#endif
}
static void model_init(Model *m, const char *snap, int cap, int bits) {
@@ -1688,6 +1862,7 @@ static int ink_vk_in_ram(void *ctx, int layer, int e) {
static size_t ink_vk_dense_bytes(const Model *m) {
const Cfg *c = &m->c;
double b = 0;
+ if (g_ink_dho) return 0; /* placed already (ink_dho_place): the free memory the tier reads counts them */
#define INK_VKB(w, n) do { const Wt *w_ = (w); int f_ = w_->vk ? ink_vk_fmt(w_) : 0; \
b += (double)(n) * (f_ == 1 ? 1.0 : f_ == 4 ? 0.5625 : f_ == 10 ? 4.0 : f_ == 11 ? 2.0 : 0.0); } while (0)
INK_VKB(&m->lm_head, (double)c->vocab * c->hidden);
@@ -1721,6 +1896,9 @@ static void ink_vk_tmp_free(Slot *s) {
/* COLI_VULKAN=1: describe the experts to the tier, after the weights and the history
* (pins_load reads it in the generate and serve modes; the oracle reads none), and
* warm it from that history: the hottest experts first, read from disk in parallel. */
+static int ink_vk_load(void *ctx, int layer, int e, VktExpertSrc *src, void **h);
+static void ink_vk_unhold(void *ctx, void *h);
+static int ink_vk_load_batch(void *ctx, int layer, const int *e, int n, VktExpertSrc *srcs, void **h);
static void ink_vk_tier_start(Model *m) {
Cfg *c = &m->c;
if (!g_vk_ready) return;
@@ -1752,7 +1930,8 @@ static void ink_vk_tier_start(Model *m) {
.max_rows = INK_VK_ROWS * c->topk,
.ram_reserve = (size_t)slotb * (size_t)m->cache[0].cap * (size_t)nsp,
.dense_bytes = ink_vk_dense_bytes(m),
- .in_ram = ink_vk_in_ram, .ram_ctx = m};
+ .in_ram = ink_vk_in_ram, .ram_ctx = m,
+ .load = ink_vk_load, .release = ink_vk_unhold, .load_ctx = m, .load_batch = ink_vk_load_batch};
atexit(coli_vk_shutdown); /* before vkt_init, which makes the expert batch's pipelines and can still refuse (no room): the device goes at exit either way, after the tier's teardown */
if (vkt_init(&vc, m->eusage)) {
atexit(vkt_shutdown);
@@ -1784,6 +1963,45 @@ static void ink_vk_tier_start(Model *m) {
}
g_vk_tier_try = vkt_ready();
}
+/* The tier's streaming (a big prompt chunk's cold experts on the device): an expert's
+ * bytes through the layer cache as moe()'s rounds get them; a group at once, filled in
+ * parallel, up to the cache's capacity (a slot stays the expert's until the next
+ * acquire). Hits and misses count as the CPU path counts them. */
+static Slot *ink_vk_get(Model *m, int layer, int e, Slot **fill, int *nfill) {
+ Slot *s = slot_find(m, layer, e);
+ if (s) m->hits++;
+ else { m->miss++; s = slot_acquire(m, layer, e); fill[(*nfill)++] = s; }
+ return s;
+}
+static int ink_vk_load_batch(void *ctx, int layer, const int *e, int n, VktExpertSrc *srcs, void **h) {
+ Model *m = ctx;
+ int cap = m->cache[layer].cap;
+ if (n > cap) n = cap;
+ if (n < 1 || n > 64) return 0;
+ Slot *use[64], *fill[64]; int nfill = 0;
+ for (int i = 0; i < n; i++) use[i] = ink_vk_get(m, layer, e[i], fill, &nfill);
+ if (nfill) {
+ double tf = now_s();
+ #pragma omp parallel for schedule(dynamic,1)
+ for (int j = 0; j < nfill; j++) slot_fill(m, layer, fill[j]);
+ m->t_fill += now_s() - tf;
+ }
+ for (int i = 0; i < n; i++) {
+ if (use[i]->eid != e[i]) return i; /* the cache served another: the rest one by one */
+ srcs[i] = ink_vk_slot_src(m, use[i]); h[i] = use[i];
+ }
+ return n;
+}
+static int ink_vk_load(void *ctx, int layer, int e, VktExpertSrc *src, void **h) {
+ Model *m = ctx;
+ Slot *fill[1]; int nfill = 0;
+ Slot *s = ink_vk_get(m, layer, e, fill, &nfill);
+ if (nfill) { double tf = now_s(); slot_fill(m, layer, s); m->t_fill += now_s() - tf; }
+ if (s->eid != e) return 0;
+ *src = ink_vk_slot_src(m, s); *h = s;
+ return 1;
+}
+static void ink_vk_unhold(void *ctx, void *h) { (void)ctx; (void)h; }
static void ink_vk_tier_report(const Model *m, const char *scope) {
vkt_report(scope, m->hits, m->miss); /* nothing when the tier never started */
}
@@ -1857,7 +2075,7 @@ static void ink_vk_moe(Model *m, Layer *l, int layer, const float *x, int S, flo
const int *idx, const int *keff, const float *wgt, int routed_only) {
Cfg *c = &m->c;
int D = c->hidden, K = c->topk, I = c->moe_inter, ns = c->n_shared;
- int B = S < INK_VK_ROWS ? S : INK_VK_ROWS;
+ int B = vkt_step_rows(S, INK_VK_ROWS); /* a whole prompt chunk when the tier streams */
float *ctb = falloc((int64_t)B * K * D), *shb = ns > 0 && !routed_only ? falloc((int64_t)B * D) : NULL;
float *g = falloc(2*I), *u = g + I, *hh = falloc(D);
int *ib = malloc((size_t)B * K * sizeof(int));
@@ -3337,6 +3555,7 @@ static void inkling_segment_wt_destroy(Wt *weight) {
if (!weight) return;
#ifdef COLI_VULKAN
ink_vk_release(weight);
+ free((void *)weight->vk_name);
#endif
free(weight->f); free(weight->h);
free(weight->q4); free(weight->qs);
@@ -3722,6 +3941,7 @@ static void inkling_edge_wt_destroy(Wt *weight) {
if (!weight) return;
#ifdef COLI_VULKAN
ink_vk_release(weight);
+ free((void *)weight->vk_name);
#endif
free(weight->f); free(weight->h); free(weight->q4); free(weight->qs);
memset(weight, 0, sizeof(*weight));
diff --git a/c/inkling_chain.h b/c/inkling_chain.h
index 9548e2d9..cd074e06 100644
--- a/c/inkling_chain.h
+++ b/c/inkling_chain.h
@@ -50,8 +50,17 @@
* on the CPU from the prefix record (the ids the state was built from), a prefill's
* worth of CPU work, and runs on the CPU from there. Only a state the ids do not
* describe (audio) cannot be rebuilt: that stops the engine with a message.
- * COLI_VK_CHAIN_FAULT=n (vk_chain.c) fakes the loss at the n-th frame, for tests. */
+ * COLI_VK_CHAIN_FAULT=n (vk_chain.c) fakes the loss at the n-th frame, for tests.
+ *
+ * Past the device's budget (vk_kvsplit.h) each global layer keeps only a window of
+ * blocks of its K/V on the device (the sliding layers' rings are small and stay whole):
+ * the step's new rows go into their slots before the attention, which runs over the
+ * device's rows (chain_kvs.comp's form of chain_relattn) and on the CPU over the older
+ * positions from the host's canonical cache at once, the two merged through their
+ * softmax statistics. A split layer's mirror follows a watermark (vkc_kv_lower on a CPU
+ * step) instead of the [dlo, dhi) ranges. Below the budget the mirrors are whole. */
#include "vk_chain.h"
+#include "vk_kvsplit.h"
/* COLI_VK_CHAIN unset on an integrated GPU with the expert tier (coli_vk_chain_decide):
* no Inkling checkpoint has been timed on one */
@@ -70,6 +79,9 @@ typedef struct {
ColiVkTensor *t_lm, **t_q, **t_k, **t_v, **t_r, **t_o, **t_router, **t_dg, **t_du, **t_dd, **t_sg, **t_su, **t_sd;
VkcBuf **kc, **vc, **ring[4];
int *cap, *dlo, *dhi; /* per layer: ring rows; positions the host wrote alone */
+ VkcKvSplit ks; /* the global layers' split past the device's budget (ks.on) */
+ int *kli, nglob; /* per layer: its split table (-1: a sliding layer); global layers */
+ int *drows; /* per layer: the device mirror's rows (cap, or ks.rows when split) */
int *slot, nslot; /* per layer: its K/V region in kvd, within its frame */
int kvo_max, qo_max, ro_max, mi_max;
size_t vs_off; /* the V rows' offset in kvs */
@@ -82,11 +94,6 @@ typedef struct {
double host_ms;
} InkChain;
-static int inkc_chunk_rows(void) {
- const char *e = getenv("COLI_VK_CHAIN_ROWS");
- int v = e && *e ? atoi(e) : 512;
- return v < 1 ? 1 : v > 65535 ? 65535 : v;
-}
static void inkc_fatal(const char *what) {
fprintf(stderr, "[VK] inkling chain: %s -- stopping (COLI_VK_CHAIN=0 keeps the state on the CPU)\n", what);
exit(1);
@@ -97,23 +104,15 @@ static int64_t inkc_cs_cells(const Cfg *c, int bank, int i) {
/* The device copy of a resident matrix or of a view of one (wt_off_i: a shared expert
* of the fused [ns][R][I] tensors): the per-matrix path's copy when it keeps a table
- * for the tensor (ink_vk_matmul, keyed by the view's first weight), else the chain's
- * own. NULL: not a form the shaders read as this CPU does (ink_vk_fmt). */
+ * for the tensor (ink_vk_view_tensor, keyed by the view's first element in the tensor),
+ * else the chain's own. NULL: not a form the shaders read as this CPU does (ink_vk_fmt). */
static ColiVkTensor *inkc_tensor(Wt view, int I, int O) {
+ if (view.vk) return ink_vk_view_tensor(&view, I, O); /* the per-matrix path's table, keyed by the view */
int fmt = ink_vk_fmt(&view);
const void *data = view.q4 ? (const void *)view.q4 : view.f ? (const void *)view.f : (const void *)view.h;
const float *sc = view.q4 ? view.qs : NULL;
int gs = view.q4 ? view.gs : 0;
if (!fmt || !data) return NULL;
- InkVk *v = view.vk;
- if (v) for (int k = 0; k < v->n; k++) {
- InkVkView *e = &v->e[k];
- if (e->q && e->q != data) continue;
- if (e->dead) return NULL;
- e->q = data;
- if (!e->t && !coli_vk_tensor_ensure(&e->t, data, sc, fmt, I, O, gs)) { e->dead = 1; return NULL; }
- return e->t;
- }
ColiVkTensor *t = NULL;
return coli_vk_tensor_ensure(&t, data, sc, fmt, I, O, gs) ? t : NULL;
}
@@ -146,7 +145,8 @@ static InkChain *inkc_setup(Model *m) {
!INKC_ARR(t_r, void *) || !INKC_ARR(t_o, void *) || !INKC_ARR(t_router, void *) || !INKC_ARR(t_dg, void *) ||
!INKC_ARR(t_du, void *) || !INKC_ARR(t_dd, void *) || !INKC_ARR(kc, void *) || !INKC_ARR(vc, void *) ||
!INKC_ARR(ring[0], void *) || !INKC_ARR(ring[1], void *) || !INKC_ARR(ring[2], void *) || !INKC_ARR(ring[3], void *) ||
- !INKC_ARR(cap, int) || !INKC_ARR(dlo, int) || !INKC_ARR(dhi, int) || !INKC_ARR(slot, int)) return NULL;
+ !INKC_ARR(cap, int) || !INKC_ARR(dlo, int) || !INKC_ARR(dhi, int) || !INKC_ARR(slot, int) ||
+ !INKC_ARR(kli, int) || !INKC_ARR(drows, int)) return NULL;
#undef INKC_ARR
int nsl = ns > 0 ? ns : 1;
ch->t_sg = calloc((size_t)L * nsl, sizeof(void *)); ch->t_su = calloc((size_t)L * nsl, sizeof(void *));
@@ -228,6 +228,7 @@ static InkChain *inkc_setup(Model *m) {
return NULL;
}
}
+ for (int i = 0; i < L; i++) ch->kli[i] = c->local[i] ? -1 : ch->nglob++;
ch->cs_where = INKC_HOST; ch->host_zero = 0;
ch->ok = 1;
int nsp = 0; for (int i = 0; i < L; i++) nsp += c->sparse[i];
@@ -236,14 +237,38 @@ static InkChain *inkc_setup(Model *m) {
return ch;
}
-static int inkc_res(VkcBuf **b, size_t floats, int kind) { return vkc_reserve(b, (floats ? floats : 1) * sizeof(float), kind); }
-
-/* scratch for `rows` rows, and the K/V mirrors at the host's layout */
-static int inkc_scratch(InkChain *ch, Model *m, int rows) {
+/* inkc_res counts instead of reserving while g_inkc_count >= 0 (the chunk's sizing) */
+static long long g_inkc_count = -1;
+static int inkc_res(VkcBuf **b, size_t floats, int kind) {
+ if (g_inkc_count >= 0) { g_inkc_count += (long long)(floats ? floats : 1) * (long long)sizeof(float); return 1; }
+ return vkc_reserve(b, (floats ? floats : 1) * sizeof(float), kind);
+}
+static int inkc_bufs(InkChain *ch, Model *m, int rows);
+/* Prompt rows per chunk (vkc_chunk_rows): the chain's scratch a row, counted from the
+ * reservations, and the routed experts' outputs (the tier's rows, the CPU's
+ * contributions, and the host's sum). */
+static int inkc_chunk_rows(InkChain *ch, Model *m) {
+ size_t vs = ch->vs_off;
+ g_inkc_count = 0; inkc_bufs(ch, m, 1); long long b1 = g_inkc_count;
+ g_inkc_count = 0; inkc_bufs(ch, m, 2); long long b2 = g_inkc_count;
+ g_inkc_count = -1; ch->vs_off = vs;
+ size_t row = (size_t)(b2 - b1) + (size_t)(2 * m->c.topk + 2) * m->c.hidden * sizeof(float);
+ if (ch->ks.on) {
+ int heads = 1, dim = 1;
+ for (int i = 0; i < m->c.n_layers; i++) if (ch->kli[i] >= 0) {
+ if (L_HEADS(&m->c, i) > heads) heads = L_HEADS(&m->c, i);
+ if (L_HD(&m->c, i) > dim) dim = L_HD(&m->c, i);
+ }
+ row += (3 * (size_t)heads * (dim + 2) + ch->qo_max + ch->ro_max) * sizeof(float);
+ }
+ return vkc_chunk_rows("inkling", row);
+}
+/* the scratch buffers for `rows` rows */
+static int inkc_bufs(InkChain *ch, Model *m, int rows) {
Cfg *c = &m->c; int D = c->hidden, ET = c->n_experts + c->n_shared, ns = c->n_shared > 0 ? c->n_shared : 1;
size_t r = (size_t)rows;
ch->vs_off = (r * ch->kvo_max + 63) & ~(size_t)63;
- int ok = inkc_res(&ch->x, r * D, VKC_DEV) && inkc_res(&ch->nrm, r * D, VKC_DEV) && inkc_res(&ch->tmp, r * D, VKC_DEV) &&
+ return inkc_res(&ch->x, r * D, VKC_DEV) && inkc_res(&ch->nrm, r * D, VKC_DEV) && inkc_res(&ch->tmp, r * D, VKC_DEV) &&
inkc_res(&ch->q, r * ch->qo_max, VKC_DEV) && inkc_res(&ch->kvs, ch->vs_off + r * ch->kvo_max, VKC_DEV) &&
inkc_res(&ch->r, r * ch->ro_max, VKC_DEV) && inkc_res(&ch->ctx, r * ch->qo_max, VKC_DEV) &&
inkc_res(&ch->h2, r * D, VKC_DEV) && inkc_res(&ch->lg, r * ET, VKC_DEV) &&
@@ -253,22 +278,32 @@ static int inkc_scratch(InkChain *ch, Model *m, int rows) {
inkc_res(&ch->kvd, (size_t)ch->nslot * 2 * r * ch->kvo_max, VKC_DOWN) && inkc_res(&ch->csd, ch->csd_n, VKC_DOWN) &&
inkc_res(&ch->outd, c->unpad_vocab, VKC_DOWN) && inkc_res(&ch->routed, r * D, VKC_UP) &&
inkc_res(&ch->shw, (size_t)ns * r, VKC_UP) && inkc_res(&ch->tau, 2 * r, VKC_UP);
- if (!ok) return 0;
- if (ch->rows < rows) {
- float *hr = realloc(ch->host_routed, r * D * sizeof(float)), *hw = hr ? realloc(ch->host_w, (size_t)ns * r * sizeof(float)) : NULL;
- if (hr) ch->host_routed = hr;
- if (!hr || !hw) return 0;
- ch->host_w = hw; ch->rows = rows;
- }
+}
+/* Plan and allocate the KV mirror before sizing a prompt chunk. */
+static int inkc_mirror(InkChain *ch, Model *m) {
+ Cfg *c = &m->c;
if (ch->max_t != m->max_t || ch->hostK != m->K) { /* a cache allocated anew: mirror it again */
+ /* the global layers' mirrors: whole, or split past the device's budget */
+ size_t row = 0, held = 0;
+ for (int i = 0; i < c->n_layers; i++) {
+ if (ch->kli[i] < 0) continue;
+ size_t b = (size_t)2 * L_KV(c, i) * L_HD(c, i) * sizeof(float);
+ if (b > row) row = b;
+ held += b * (size_t)ch->drows[i];
+ }
+ int plan = ch->nglob ? vkc_kv_plan(&ch->ks, "inkling", ch->nglob, row, m->max_t, 1, 0, held) : 1;
+ if (!plan) { ch->max_t = 0; ch->hostK = NULL; return 0; }
for (int i = 0; i < c->n_layers; i++) {
vkc_free(ch->kc[i]); vkc_free(ch->vc[i]); ch->kc[i] = ch->vc[i] = NULL;
int cap = kv_ring_rows(c, i, m->max_t), hd = L_HD(c, i), KV = L_KV(c, i);
+ int split = ch->ks.on && ch->kli[i] >= 0;
ch->cap[i] = cap;
- /* a ring: every row, whatever positions it holds; a whole cache: the positions held */
- ch->dlo[i] = 0; ch->dhi[i] = cap < m->max_t ? m->max_t : m->kv_len;
- ch->kc[i] = vkc_buf((size_t)KV * cap * hd * sizeof(float), VKC_DEV);
- ch->vc[i] = vkc_buf((size_t)KV * cap * hd * sizeof(float), VKC_DEV);
+ ch->drows[i] = split ? ch->ks.rows : cap;
+ /* a ring: every row, whatever positions it holds; a whole cache: the positions held;
+ * a split layer: its watermark (vkc_kv_plan reset it) */
+ ch->dlo[i] = 0; ch->dhi[i] = split ? 0 : cap < m->max_t ? m->max_t : m->kv_len;
+ ch->kc[i] = vkc_buf((size_t)KV * ch->drows[i] * hd * sizeof(float), VKC_DEV);
+ ch->vc[i] = vkc_buf((size_t)KV * ch->drows[i] * hd * sizeof(float), VKC_DEV);
if (!ch->kc[i] || !ch->vc[i]) { ch->max_t = 0; ch->hostK = NULL; return 0; }
}
ch->max_t = m->max_t; ch->hostK = m->K;
@@ -276,6 +311,28 @@ static int inkc_scratch(InkChain *ch, Model *m, int rows) {
return 1;
}
+/* scratch for `rows` rows, and the K/V mirrors at the host's layout */
+static int inkc_scratch(InkChain *ch, Model *m, int rows) {
+ Cfg *c = &m->c; int D = c->hidden, ns = c->n_shared > 0 ? c->n_shared : 1;
+ size_t r = (size_t)rows;
+ if (!inkc_bufs(ch, m, rows)) return 0;
+ if (ch->rows < rows) {
+ float *hr = realloc(ch->host_routed, r * D * sizeof(float)), *hw = hr ? realloc(ch->host_w, (size_t)ns * r * sizeof(float)) : NULL;
+ if (hr) ch->host_routed = hr;
+ if (!hr || !hw) return 0;
+ ch->host_w = hw; ch->rows = rows;
+ }
+ if (ch->ks.on) {
+ int hmax = 1, hdm = 1;
+ for (int i = 0; i < c->n_layers; i++) if (ch->kli[i] >= 0) {
+ if (L_HEADS(c, i) > hmax) hmax = L_HEADS(c, i);
+ if (L_HD(c, i) > hdm) hdm = L_HD(c, i);
+ }
+ if (!vkc_kv_parts(&ch->ks, r * hmax * (hdm + 2))) return 0;
+ }
+ return 1;
+}
+
/* ---- state between the host and the device ----------------------------------- */
/* The host wrote its convolution states (zero: state_reset's zeros). */
static void inkc_host_wrote(Model *m, int zero) {
@@ -289,14 +346,17 @@ static void inkc_cpu_step(Model *m, int pos_base, int S) {
if (!ch || !ch->ok) return;
ch->cs_where = INKC_HOST; ch->host_zero = 0;
for (int i = 0; i < m->c.n_layers; i++) {
+ if (ch->ks.on && ch->kli[i] >= 0) { vkc_kv_lower(&ch->ks, ch->kli[i], pos_base); continue; }
if (ch->dlo[i] >= ch->dhi[i]) { ch->dlo[i] = pos_base; ch->dhi[i] = pos_base + S; continue; }
if (pos_base < ch->dlo[i]) ch->dlo[i] = pos_base;
if (pos_base + S > ch->dhi[i]) ch->dhi[i] = pos_base + S;
}
}
/* Record the uploads that make the device state the host's: the convolution states if
- * the host's are newer, and the rows of the last `cap` positions the host wrote alone. */
-static int inkc_push_state(InkChain *ch, Model *m) {
+ * the host's are newer, and the rows of the last `cap` positions the host wrote alone;
+ * a split layer: its window placed for the n rows from pos_base, its resident rows below
+ * pos_base uploaded. */
+static int inkc_push_state(InkChain *ch, Model *m, int pos_base, int n) {
Cfg *c = &m->c; int ok = 1;
if (ch->cs_where == INKC_HOST) {
for (int i = 0; i < c->n_layers && ok; i++) for (int b = 0; b < 4 && ok; b++) {
@@ -306,6 +366,14 @@ static int inkc_push_state(InkChain *ch, Model *m) {
ch->cs_where = INKC_BOTH;
}
for (int i = 0; i < c->n_layers && ok; i++) {
+ if (ch->ks.on && ch->kli[i] >= 0) {
+ int KV = L_KV(c, i), hd = L_HD(c, i);
+ VkcKvPart pt[2] = {{KV, hd, m->K[i], (size_t)ch->cap[i] * hd, ch->kc[i], 0},
+ {KV, hd, m->V[i], (size_t)ch->cap[i] * hd, ch->vc[i], 0}};
+ vkc_kv_place(&ch->ks, ch->kli[i], pos_base, n);
+ ok = vkc_kv_push(&ch->ks, ch->kli[i], pt, 2, pos_base);
+ continue;
+ }
if (ch->dlo[i] >= ch->dhi[i]) continue;
int cap = ch->cap[i], hd = L_HD(c, i), KV = L_KV(c, i);
int t0 = ch->dhi[i] - cap > ch->dlo[i] ? ch->dhi[i] - cap : ch->dlo[i];
@@ -373,6 +441,22 @@ static int inkc_attention(InkChain *ch, Model *m, int i, int n, int pb) {
VkcNorm qn = {n * H, hd, H, 0, H * hd, hd, 0, H * hd, hd, (int)ch->o_qn[i], 0, 0, c->eps, 1.f};
VkcNorm kn = {n * KV, hd, KV, 0, kvo, hd, 0, kvo, hd, (int)ch->o_kn[i], 0, 0, c->eps, 1.f};
ok = ok && vkc_norm(ch->q, ch->prm, ch->q, &qn) && vkc_norm(ch->kvs, ch->prm, ch->kvs, &kn);
+ if (ok && ch->ks.on && ch->kli[i] >= 0) {
+ /* the split: the step's rows into their window slots first (the window holds every
+ * row of a step), then the attention over the device's rows and the host's */
+ int li = ch->kli[i];
+ VkcKvPart pk = {KV, hd, m->K[i], (size_t)cap * hd, ch->kc[i], 0}, pv = {KV, hd, m->V[i], (size_t)cap * hd, ch->vc[i], 0};
+ size_t ko = (size_t)ch->slot[i] * 2 * ch->rows * ch->kvo_max;
+ VkcKvRel a = {&ch->ks, li, ch->q, ch->kc[i], ch->vc[i], ch->r, ch->tau, ch->prm, ch->ctx, n, H, KV, hd, pb,
+ L_EXT(c, i), dr, H * hd, H * dr, (int)ch->o_relp[i], 0, H * hd, 1.f / (float)hd,
+ m->K[i], m->V[i], (size_t)cap * hd, (size_t)hd, (size_t)cap * hd, (size_t)hd,
+ (const float *)vkc_ptr(ch->tau), m->L[i].relp, 0};
+ return vkc_kv_store(&ch->ks, li, &pk, ch->kvs, 0, (size_t)kvo, (size_t)hd, pb, n) &&
+ vkc_kv_store(&ch->ks, li, &pv, ch->kvs, vs, (size_t)kvo, (size_t)hd, pb, n) &&
+ vkc_copy(ch->kvd, ko, ch->kvs, 0, (size_t)n * kvo) &&
+ vkc_copy(ch->kvd, ko + (size_t)ch->rows * ch->kvo_max, ch->kvs, vs, (size_t)n * kvo) &&
+ vkc_kv_rel(&a) && vkc_matmul(ch->t_o[i], ch->ctx, 0, ch->tmp, 0, n);
+ }
VkcRelAttn a = {n, H, KV, hd, pb, cap, local ? c->window : 0, L_EXT(c, i), dr,
0, H * hd, 0, H * hd, 0, 0, 0, (int)vs, kvo, 0, H * dr, (int)ch->o_relp[i], local ? ch->rows : 0,
1.f / (float)hd};
@@ -453,8 +537,10 @@ static int inkc_forward(Model *m, float *xh, int S, int pos_base, int want_x, fl
InkChain *ch = inkc_setup(m);
if (!ch || ch->failed) return 0;
Cfg *c = &m->c; int D = c->hidden, L = c->n_layers, ET = c->n_experts + c->n_shared, ns = c->n_shared;
- int CH = inkc_chunk_rows(), rows = S < CH ? S : CH;
- if (pos_base + S > m->max_t || !inkc_scratch(ch, m, rows) || (want_x && !inkc_res(&ch->xd, (size_t)rows * D, VKC_DOWN))) {
+ int mirror_ok = inkc_mirror(ch, m);
+ int CH = mirror_ok ? inkc_chunk_rows(ch, m) : 1, rows = S < CH ? S : CH;
+ if (ch->ks.on && rows > ch->ks.chunk) rows = ch->ks.chunk; /* a step's rows fit the split's window */
+ if (!mirror_ok || pos_base + S > m->max_t || !inkc_scratch(ch, m, rows) || (want_x && !inkc_res(&ch->xd, (size_t)rows * D, VKC_DOWN))) {
fprintf(stderr, "[VK] inkling chain: device memory for %d rows refused; per-matrix path\n", rows);
ch->failed = 1;
return 0;
@@ -465,7 +551,7 @@ static int inkc_forward(Model *m, float *xh, int S, int pos_base, int want_x, fl
for (int c0 = 0; c0 < S; c0 += rows) {
int n = S - c0 < rows ? S - c0 : rows, pb = pos_base + c0;
if (!vkc_begin() || !vkc_write(ch->x, 0, xh + (size_t)c0 * D, (size_t)n * D * sizeof(float)) ||
- !inkc_push_state(ch, m)) goto lost;
+ !inkc_push_state(ch, m, pb, n)) goto lost;
float *tau = (float *)vkc_ptr(ch->tau); /* attention()'s tau: global rows, then 1 for sliding ones */
for (int s = 0; s < n; s++) {
float t = 1.f;
@@ -486,7 +572,11 @@ static int inkc_forward(Model *m, float *xh, int S, int pos_base, int want_x, fl
ok = ok && vkc_matmul(ch->t_router[i], ch->h2, 0, ch->lg, 0, n) &&
vkc_copy(ch->lgd, 0, ch->lg, 0, (size_t)n * ET) && vkc_copy(ch->h2d, 0, ch->h2, 0, (size_t)n * D);
double t0 = now_s();
- ok = ok && vkc_submit(1); /* A1 */
+ /* A1; while it runs, the tier loads the experts this layer will likely
+ * stream (a big prompt chunk only) */
+ ok = ok && vkc_submit(0);
+ if (ok) vkt_stream_prefetch(i, n);
+ ok = ok && vkc_finish();
m->t_attn += now_s() - t0;
if (!ok) break;
inkc_kv_down(ch, m, f0, i, n, pb);
@@ -521,6 +611,7 @@ static int inkc_forward(Model *m, float *xh, int S, int pos_base, int want_x, fl
for (int i = 0; i < L; i++) for (int b = 0; b < 4; b++)
memcpy(m->cs[b][i], csd + ch->o_csd[(size_t)b * L + i], (size_t)inkc_cs_cells(c, b, i) * sizeof(float));
ch->cs_where = INKC_BOTH; ch->host_zero = 0;
+ for (int i = 0; i < L; i++) if (ch->kli[i] >= 0) vkc_kv_done(&ch->ks, ch->kli[i], pb + n);
if (want_x) memcpy(xfin + (size_t)c0 * D, vkc_ptr(ch->xd), (size_t)n * D * sizeof(float));
if (last) memcpy(logit, vkc_ptr(ch->outd), (size_t)c->unpad_vocab * sizeof(float));
}
@@ -541,6 +632,7 @@ static void inkc_report(Model *m) {
fprintf(stderr, "[VK] inkling chain: %llu forwards, %llu frames (%llu ops, %llu matmuls, %llu tiled GEMM), "
"%.1f ms waiting for the device, %.1f ms of routed experts on the host, %.1f MiB on the device\n",
ch->forwards, st.frames, st.ops, st.matmuls, st.gemms, st.wait_ms, ch->host_ms, st.dev_bytes / 1048576.0);
+ vkc_kv_report(&ch->ks);
vkc_prof_print();
}
diff --git a/c/kimi_k3.c b/c/kimi_k3.c
index dc424221..ccb1cc34 100644
--- a/c/kimi_k3.c
+++ b/c/kimi_k3.c
@@ -159,7 +159,10 @@ typedef struct { int fmt; float *f; int8_t *q8; uint8_t *q4; float *s; int O, I,
void *vk; /* ColiVkTensor* once uploaded (K3_VK); NULL = CPU only */
void *metal; /* ColiMetalTensor* once registered (K3_METAL); NULL = CPU only */
st_mapped_raw data_map, scale_map;
- int mapped; } W;
+ int mapped;
+ /* COLI_VULKAN: how to read it back from disk (k3_w_reload), and 1 while the
+ * device holds it alone (COLI_VK_DENSE_HOST, the host copy dropped) */
+ char *vk_name; int vk_bits, vk_gone; } W;
typedef struct { /* KDA layer */
W q, k, v, o, g;
@@ -168,6 +171,7 @@ typedef struct { /* KDA layer */
float *bp; /* beta proj f32 [heads,hidden] */
float *dt, *A, *onw; /* dt_bias[proj], exp(A_log)[heads], o_norm[hd] */
void *metal_fa, *metal_fb, *metal_bp; /* Metal tensor wrappers for f32 weights */
+ void *vk_fa, *vk_fb, *vk_bp; /* COLI_VK_DENSE_HOST: the chain's device copies, for the per-matrix path */
} Kda;
typedef struct { /* gated MLA layer */
@@ -181,6 +185,7 @@ typedef struct { /* gated MLA layer */
typedef struct { /* LatentMoE */
float *router, *rbias, *lat_norm; /* [E,hidden] f32, [E], [latent] */
+ void *vk_router; /* COLI_VK_DENSE_HOST: the chain's device copy, for the per-matrix path */
W lat_down, lat_up, sh_gate, sh_up, sh_down;
} Moe;
@@ -405,6 +410,17 @@ static int dsa_score_single(int *sel, int maxsel, const float *qi, const float *
#ifdef COLI_VULKAN
#include "backend_vulkan.h"
static int g_k3_vk=0; /* the device is open */
+/* COLI_VK_DENSE_HOST: the dense matrices on the device only (k3_dho_*): every W the
+ * chain does not run goes to the device through w_matmul, prompts included */
+static int g_k3_dho=0;
+static void k3_w_reload(W *w); /* below, beside w_load */
+static int k3_in_par(void){
+#ifdef _OPENMP
+ return omp_in_parallel();
+#else
+ return 0;
+#endif
+}
/* Did the run ask for the device? K3_VK, when set, decides (0: never, any other
* number: yes, as COLI_VULKAN=1); unset, COLI_VULKAN does, as for every engine. */
static int k3_vk_requested(void){
@@ -491,9 +507,14 @@ static void w_matmul(float *y, const float *x, const W *w, int S){
}
#endif
#ifdef COLI_VULKAN
- if(g_k3_vk&&S==1&&w->vk&&
+ if(g_k3_vk&&S==1&&w->vk&&!g_k3_dho&&
coli_vk_matmul((ColiVkTensor**)&((W*)w)->vk,y,x,NULL,NULL,k3_vk_fmt(w),1,w->I,w->O,w->gs))
return;
+ /* device only: every row count, from the main thread (one command buffer) */
+ if(g_k3_dho&&g_k3_vk&&w->vk&&S>=1&&S<=65535&&!k3_in_par()&&
+ coli_vk_matmul((ColiVkTensor**)&((W*)w)->vk,y,x,NULL,NULL,k3_vk_fmt(w),S,w->I,w->O,w->gs))
+ return;
+ if(w->vk_gone) k3_w_reload((W*)w); /* the CPU needs a matrix the device held alone (a lost device) */
#endif
if(w->fmt==0) matmul(y,x,w->f,S,w->I,w->O);
else if(w->fmt==1) matmul_q(y,x,w->q8,w->s,S,w->I,w->O);
@@ -591,6 +612,9 @@ static void w_release_host(W *w){
if(!w) return;
if(w->mapped){ st_unmap_raw(&w->scale_map); st_unmap_raw(&w->data_map); }
else { free(w->f); free(w->q8); free(w->q4); free(w->s); }
+#ifdef COLI_VULKAN
+ free(w->vk_name);
+#endif
memset(w,0,sizeof(*w));
}
@@ -599,6 +623,10 @@ static void w_load(Model *m, W *w, const char *name, int O, int I, int bits){
st_tensor *t=st_find(&m->S,nm);
if(!t) st_die_missing(&m->S,nm);
memset(w,0,sizeof(*w)); w->O=O; w->I=I;
+#ifdef COLI_VULKAN
+ w->vk_name=strdup(name); w->vk_bits=bits; /* to read it back as it is read now (k3_w_reload) */
+ if(!w->vk_name){fprintf(stderr,"OOM weight name\n");exit(1);}
+#endif
if(t->dtype==3){
/* repacked container (tools/k3_repack.py): pre-quantized U8 + .qs f32
* scales — no load-time quantization, K3_BITS is ignored for these
@@ -706,6 +734,102 @@ static float *f32_load(Model *m, const char *name, int64_t want){
float *p=falloc(t->numel); st_read_f32(&m->S,nm,p,0); return p;
}
+#ifdef COLI_VULKAN
+/* ---- the dense matrices on the device only (COLI_VK_DENSE_HOST) -----------------
+ * Decided once the device is open and before the expert cache is sized (k3_dho_decide),
+ * so the cache counts the dense weights out of RAM; placed in k3_vk_tier_start: every W
+ * goes up into its own vk (the chain adopts those copies), then its host copy is
+ * dropped, and the f32 matrices only the chain multiplies on the device (the routers,
+ * the KDA decay and beta projections) are dropped once the chain holds them. The CPU
+ * reads a matrix back from disk when it needs it after all: w_matmul after a lost
+ * device, the CPU MLA path's kv_b rows, the CPU KDA and router matmuls (k3_f32_need). */
+static Model *g_k3_dho_model;
+static size_t g_k3_dho_bytes; /* what will be dropped: RAM the expert cache may count on */
+static pthread_mutex_t g_k3_dho_mx=PTHREAD_MUTEX_INITIALIZER;
+static size_t k3_w_host_bytes(const W *w){
+ if(w->fmt==0) return w->f?(size_t)w->O*w->I*4:0;
+ if(w->fmt==1) return w->q8?(size_t)w->O*w->I+(size_t)w->O*4:0;
+ if(w->fmt==4) return w->q4?(size_t)w->O*((w->I+1)/2)+(size_t)w->O*((w->I+w->gs-1)/w->gs)*4:0;
+ return 0;
+}
+static void k3_w_reload(W *w){
+ pthread_mutex_lock(&g_k3_dho_mx);
+ if(w->vk_gone){
+ Model *m=g_k3_dho_model;
+ if(!m||!w->vk_name){fprintf(stderr,"[VK] kimi_k3: a dense matrix the device held alone cannot be read back\n");exit(1);}
+ W t; w_load(m,&t,w->vk_name,w->O,w->I,w->vk_bits);
+ if(t.fmt!=w->fmt||t.gs!=w->gs){fprintf(stderr,"[VK] kimi_k3: %s came back in another form\n",w->vk_name);exit(1);}
+ w->f=t.f; w->q8=t.q8; w->q4=t.q4; w->s=t.s;
+ free(t.vk_name);
+ w->vk_gone=0;
+ coli_vk_dense_host_reloaded(k3_w_host_bytes(w));
+ }
+ pthread_mutex_unlock(&g_k3_dho_mx);
+}
+/* The f32 matrices the chain holds on the device: where each lives, and its name. */
+typedef struct { float **p; char name[160]; int64_t n; } K3F32Home;
+static K3F32Home *g_k3_f32; static int g_k3_f32_n;
+/* A dropped f32 matrix the CPU needs: read it back (NULL is "dropped": loaded ones never are). */
+static void k3_f32_need(float **p){
+ if(*p) return;
+ pthread_mutex_lock(&g_k3_dho_mx);
+ if(!*p){
+ for(int i=0;i<g_k3_f32_n;i++) if(g_k3_f32[i].p==p){
+ *p=f32_load(g_k3_dho_model,g_k3_f32[i].name,g_k3_f32[i].n);
+ coli_vk_dense_host_reloaded((size_t)g_k3_f32[i].n*4);
+ break;
+ }
+ if(!*p){fprintf(stderr,"[VK] kimi_k3: an f32 matrix the device held alone cannot be read back\n");exit(1);}
+ }
+ pthread_mutex_unlock(&g_k3_dho_mx);
+}
+#define K3_F32_NEED(p) k3_f32_need(&(p))
+/* y = x W^T on the device with the chain's copy of an f32 matrix (the device only: the CPU
+ * may hold none); 0 = the CPU multiplies, reading the matrix back first if it must. */
+static int k3_f32_dev(void *t, float *y, const float *x, int S, int I, int O){
+ return g_k3_dho&&g_k3_vk&&t&&S>=1&&S<=65535&&!k3_in_par()&&
+ coli_vk_matmul((ColiVkTensor**)&t,y,x,NULL,NULL,10,S,I,O,0);
+}
+#define K3_F32_DEV(t,y,x,S,I,O) k3_f32_dev(t,y,x,S,I,O)
+/* Every W of the model, in a fixed order. */
+static void k3_dho_each(Model *m, void (*f)(W *, size_t *, int *), size_t *b, int *n){
+ for(int i=0;i<m->c.n_layers;i++){
+ Layer *l=&m->L[i];
+ if(l->kda){ Kda *a=&l->a; W *ws[]={&a->q,&a->k,&a->v,&a->g,&a->o}; for(size_t k=0;k<5;k++) f(ws[k],b,n); }
+ else { Mla *a=&l->m; W *ws[]={&a->qa,&a->qb,&a->kva,&a->kvb,&a->o,&a->g,&a->wk,&a->wq,&a->wp}; for(size_t k=0;k<9;k++) f(ws[k],b,n); }
+ if(l->sparse){ Moe *o=&l->moe; W *ws[]={&o->lat_down,&o->lat_up,&o->sh_gate,&o->sh_up,&o->sh_down}; for(size_t k=0;k<5;k++) f(ws[k],b,n); }
+ else { W *ws[]={&l->d_gate,&l->d_up,&l->d_down}; for(size_t k=0;k<3;k++) f(ws[k],b,n); }
+ }
+ if(m->has_head) f(&m->lm_head,b,n);
+}
+static void k3_dho_count(W *w, size_t *b, int *n){ size_t x=k3_w_host_bytes(w); if(x&&!w->mapped&&k3_vk_fmt(w)>0){ *b+=x; (*n)++; } }
+/* The f32 matrices the chain multiplies on the device: the routers, the KDA decay pair and beta. */
+static size_t k3_dho_f32_bytes(const Model *m){
+ const Cfg *c=&m->c; size_t b=0;
+ for(int i=0;i<c->n_layers;i++){
+ const Layer *l=&m->L[i];
+ if(l->sparse&&l->moe.router) b+=(size_t)c->n_experts*c->hidden*4;
+ if(l->kda&&l->a.fa) b+=((size_t)c->kda_hd*c->hidden+(size_t)c->kda_proj*c->kda_hd+(size_t)c->kda_heads*c->hidden)*4;
+ }
+ return b;
+}
+/* Once the device is open and before the expert cache is sized: the decision, with the
+ * chain's own decision asked silently (k3c_start prints it later). The cache counts the
+ * W matrices out of RAM; the f32 ones only leave once the chain has them, so it does not
+ * count on those. */
+static void k3_dho_decide(Model *m, int tier_on, int cuda_on){
+ if(!g_k3_vk) return;
+ int chain=!cuda_on&&!k3_dsa_indexer_on()&&coli_vk_chain_decide(NULL,tier_on,COLI_VK_CHAIN_UNMEASURED)!=COLI_VK_CHAIN_OFF;
+ size_t b=0; int n=0;
+ k3_dho_each(m,k3_dho_count,&b,&n);
+ if(!coli_vk_dense_host_decide("kimi_k3",coli_vk_dense()||chain,b+(chain?k3_dho_f32_bytes(m):0))) return;
+ g_k3_dho=1; g_k3_dho_model=m; g_k3_dho_bytes=b;
+}
+#else
+#define K3_F32_NEED(p) ((void)0)
+#define K3_F32_DEV(t,y,x,S,I,O) 0
+#endif
+
/* ---------- config ---------- */
static double req_num(jval *r, const char *k){
jval *v=json_get(r,k);
@@ -1014,6 +1138,7 @@ static void model_init_range(Model *m, const char *snap, int layer_begin,
cuda_on=g_k3_cuda;
#endif
g_k3_vk=coli_vk_init_env_tier("kimi_k3",vkt_wanted()&&any_sparse&&!cuda_on);
+ k3_dho_decide(m,vkt_wanted()&&any_sparse&&!cuda_on,cuda_on); /* COLI_VK_DENSE_HOST, before the cache is sized */
}
#endif
#ifdef COLI_METAL
@@ -1068,6 +1193,16 @@ static void model_init_range(Model *m, const char *snap, int layer_begin,
{
double resident = rss_gb();
double avail_now = k3_mem_avail();
+#ifdef COLI_VULKAN
+ double resident_host = resident, avail_host = avail_now; /* with the host copies in */
+ /* the dense weights on the device only: their host copies go once the device has
+ * them (k3_vk_tier_start); on a discrete GPU that RAM comes back to the machine,
+ * on one that shares it the device copy takes it */
+ if(g_k3_dho){
+ resident -= g_k3_dho_bytes/1e9;
+ if(!coli_vk_device_shares_ram()) avail_now += g_k3_dho_bytes/1e9;
+ }
+#endif
double ram_env = getenv("RAM_GB") ? atof(getenv("RAM_GB")) : 0.0;
/* Explicit --ram is a ceiling on the WHOLE process. Absent, take 88% of
* what the OS still offers and add what we already hold -- the same
@@ -1095,6 +1230,16 @@ static void model_init_range(Model *m, const char *snap, int layer_begin,
double slot_gb = (double)m->e_slot/1e9;
int cap_fit = k3_cap_for_ram(budget, resident, reserve, slot_gb,
nmoe, cap, c->n_experts, &for_experts);
+#ifdef COLI_VULKAN
+ if(g_k3_dho){ /* the same plan with the host copies kept, for the line */
+ double fe_host = 0.0, budget_host = ram_env > 0 ? ram_env : resident_host + avail_host*0.88;
+ int fit_host = k3_cap_for_ram(budget_host, resident_host, reserve, slot_gb, nmoe, cap, c->n_experts, &fe_host);
+ int cap_dev = cap_fit < cap ? (cap_fit > 0 ? cap_fit : 1) : cap, cap_host = fit_host < cap ? (fit_host > 0 ? fit_host : 1) : cap;
+ fprintf(stderr,"[K3] dense weights on the device only: %.4f GB of host copies out of the resident %.4f GB "
+ "(reserve %.4f GB): %.4f GB for experts instead of %.4f, expert cache %d/layer instead of %d\n",
+ g_k3_dho_bytes/1e9, resident_host, reserve, for_experts, fe_host, cap_dev, cap_host);
+ }
+#endif
if(cap_fit < cap){
/* Name every term. The user's number is not being ignored, it is
@@ -1248,9 +1393,12 @@ static void kda_forward(Model *m, Layer *l, int li, const float *x, int C, float
fflush(stderr); } }
w_matmul(q,x,&a->q,C); w_matmul(k,x,&a->k,C); w_matmul(v,x,&a->v,C);
w_matmul(gp,x,&a->g,C);
- k3_matmul_f32((void*)&a->metal_fa, t1, x, a->fa, C, c->hidden, c->kda_hd);
- k3_matmul_f32((void*)&a->metal_fb, graw, t1, a->fb, C, c->kda_hd, P);
- k3_matmul_f32((void*)&a->metal_bp, braw, x, a->bp, C, c->hidden, H);
+ if(!K3_F32_DEV(a->vk_fa,t1,x,C,c->hidden,c->kda_hd)){
+ K3_F32_NEED(a->fa); k3_matmul_f32((void*)&a->metal_fa, t1, x, a->fa, C, c->hidden, c->kda_hd); }
+ if(!K3_F32_DEV(a->vk_fb,graw,t1,C,c->kda_hd,P)){
+ K3_F32_NEED(a->fb); k3_matmul_f32((void*)&a->metal_fb, graw, t1, a->fb, C, c->kda_hd, P); }
+ if(!K3_F32_DEV(a->vk_bp,braw,x,C,c->hidden,H)){
+ K3_F32_NEED(a->bp); k3_matmul_f32((void*)&a->metal_bp, braw, x, a->bp, C, c->hidden, H); }
float qscale=1.f/sqrtf((float)hd);
for(int t=0;t<C;t++){
float *qt=q+(int64_t)t*P, *kt=k+(int64_t)t*P, *tv=v+(int64_t)t*P;
@@ -1460,6 +1608,9 @@ static void kda_forward(Model *m, Layer *l, int li, const float *x, int C, float
}
}
w_matmul(gv,x,&a->g,C);
+#ifdef COLI_VULKAN
+ if(a->kvb.vk_gone) k3_w_reload(&a->kvb); /* its rows are read on the CPU below */
+#endif
for(int tt=0;tt<C;tt++){
int nt=pos0+tt+1;
const float *qvt=qv+(int64_t)tt*H*qh, *gvt=gv+(int64_t)tt*H*vh;
@@ -1926,6 +2077,9 @@ static void k3c_adopt_shared(Model *m);
static void k3c_report(void);
static int g_k3c_on;
static int k3_vk_in_ram(void *ctx, int li, int e){ return slot_indexed((Model*)ctx,li,e)!=NULL; }
+static int k3_vk_load(void *ctx,int li,int e,VktExpertSrc *src,void **h);
+static int k3_vk_load_batch(void *ctx,int li,const int *e,int n,VktExpertSrc *srcs,void **h);
+static void k3_vk_unhold(void *ctx,void *h);
static size_t k3_w_bytes(const W *w){
int fmt=k3_vk_fmt(w);
if(fmt==10) return (size_t)w->O*w->I*4;
@@ -1949,6 +2103,56 @@ static int k3_expert_on_disk(const Model *m, int li, int e){
for(int k=0;k<6;k++) if(er->fd[k]<0) return 0;
return 1;
}
+static void k3_dho_upload(W *w, size_t *b, int *n){ (void)b; if(k3_w_host_bytes(w)&&!w->mapped&&w_vk_upload(w)) (*n)++; }
+static void k3_dho_drop(W *w, size_t *b, int *n){
+ if(!w->vk||w->vk_gone||!w->vk_name||w->mapped) return;
+ size_t x=k3_w_host_bytes(w);
+ if(!x) return;
+ free(w->f); free(w->q8); free(w->q4); free(w->s);
+ w->f=NULL; w->q8=NULL; w->q4=NULL; w->s=NULL; w->vk_gone=1;
+ coli_vk_dense_host_dropped(x);
+ *b+=x; (*n)++;
+}
+static ColiVkTensor *k3c_f32_tensor(int li, int which); /* kimi_k3_chain.h */
+static void k3_dho_f32_drop(Model *m, float **p, const char *name, int64_t n){
+ if(!*p) return;
+ g_k3_f32=realloc(g_k3_f32,(size_t)(g_k3_f32_n+1)*sizeof(*g_k3_f32));
+ if(!g_k3_f32){fprintf(stderr,"OOM f32 home\n");exit(1);}
+ K3F32Home *h=&g_k3_f32[g_k3_f32_n++];
+ h->p=p; h->n=n; snprintf(h->name,sizeof h->name,"%s",name);
+ free(*p); *p=NULL;
+ coli_vk_dense_host_dropped((size_t)n*4);
+ (void)m;
+}
+/* The W matrices up first, into their own vk, so the chain adopts the same copies; the
+ * host copies go once the chain is set up, and the f32 ones the chain holds with them. */
+static void k3_dho_place(Model *m){
+ size_t b=0; int n=0;
+ k3_dho_each(m,k3_dho_drop,&b,&n);
+ if(g_k3c_on){
+ const Cfg *c=&m->c; char nm[200];
+ for(int i=0;i<c->n_layers;i++){
+ Layer *l=&m->L[i];
+ if(l->sparse){
+ snprintf(nm,sizeof nm,"model.layers.%d.block_sparse_moe.gate.weight",i);
+ l->moe.vk_router=k3c_f32_tensor(i,0);
+ k3_dho_f32_drop(m,&l->moe.router,nm,(int64_t)c->n_experts*c->hidden);
+ }
+ if(l->kda){
+ l->a.vk_fa=k3c_f32_tensor(i,1); l->a.vk_fb=k3c_f32_tensor(i,2); l->a.vk_bp=k3c_f32_tensor(i,3);
+ snprintf(nm,sizeof nm,"model.layers.%d.self_attn.f_a_proj.weight",i);
+ k3_dho_f32_drop(m,&l->a.fa,nm,(int64_t)c->kda_hd*c->hidden);
+ snprintf(nm,sizeof nm,"model.layers.%d.self_attn.f_b_proj.weight",i);
+ k3_dho_f32_drop(m,&l->a.fb,nm,(int64_t)c->kda_proj*c->kda_hd);
+ snprintf(nm,sizeof nm,"model.layers.%d.self_attn.b_proj.weight",i);
+ k3_dho_f32_drop(m,&l->a.bp,nm,(int64_t)c->kda_heads*c->hidden);
+ }
+ }
+ }
+ coli_vk_dense_host_placed("kimi_k3",g_k3c_on
+ ? "the embedding (read from disk per token), norms, vectors and the AttnRes weights"
+ : "the embedding (read from disk per token), norms, vectors, the AttnRes weights, the routers and the KDA decay and beta projections (f32: without the chain the CPU computes them)");
+}
static void k3_vk_tier_start(Model *m){
Cfg *c=&m->c;
if(!g_k3_vk) return;
@@ -1957,7 +2161,9 @@ static void k3_vk_tier_start(Model *m){
#ifdef COLI_CUDA
cuda_on=g_k3_cuda;
#endif
+ if(g_k3_dho){ size_t b=0; int n=0; k3_dho_each(m,k3_dho_upload,&b,&n); } /* COLI_VK_DENSE_HOST: every W into its own vk first */
k3c_start(m,nmoe&&vkt_wanted()&&!cuda_on); /* COLI_VK_CHAIN: the trunk on the device, before the tier sizes itself */
+ if(g_k3_dho) k3_dho_place(m); /* then the host copies go, before the tier sizes its budget */
if(cuda_on) fprintf(stderr,"[VK] tier kimi_k3: the CUDA expert tier is on and wins; the Vulkan tier stays off\n");
else if(nmoe&&vkt_wanted()){
VktConfig vc={.engine="kimi_k3",.layers=c->n_layers,.experts=c->n_experts,
@@ -1966,8 +2172,9 @@ static void k3_vk_tier_start(Model *m){
.act=VKT_ACT_SITU,.act_a=c->situ_b1,.act_b=c->situ_b2,
.max_rows=K3_VK_ROWS*c->topk,
.ram_reserve=(size_t)(m->e_slot+8192)*(size_t)cap*(size_t)nmoe,
- .dense_bytes=coli_vk_dense()&&!g_k3c_on?k3_vk_dense_bytes(m):0, /* the chain's are there already */
- .in_ram=k3_vk_in_ram,.ram_ctx=m};
+ .dense_bytes=coli_vk_dense()&&!g_k3c_on&&!g_k3_dho?k3_vk_dense_bytes(m):0, /* the chain's are there already */
+ .in_ram=k3_vk_in_ram,.ram_ctx=m,
+ .load=k3_vk_load,.release=k3_vk_unhold,.load_ctx=m,.load_batch=k3_vk_load_batch};
atexit(coli_vk_shutdown); /* before vkt_init, which makes the expert batch's pipelines and can still refuse (no room): the device goes at exit either way, after the tier's teardown */
if(vkt_init(&vc,rt_counts_all())){
atexit(vkt_shutdown);
@@ -2027,6 +2234,60 @@ static float *k3_shared(Model *m, Moe *o, const float *x, int C){
}
#ifdef COLI_VULKAN
+/* The tier's streaming (a big prompt chunk's cold experts on the device): a group of
+ * experts as experts_apply_union resolves them -- the layer cache, the misses read into
+ * the working-set slots in parallel -- each valid until the tier gives it back; the
+ * misses then join the LRU as experts_apply_union's do, when the group's last one is
+ * given back (before anything else can reuse the working set). */
+static int g_k3s_layer=-1, g_k3s_nmiss=0, g_k3s_held=0;
+static void k3_vk_promote(Model *m){
+ int li=g_k3s_layer, nmiss=g_k3s_nmiss;
+ g_k3s_layer=-1; g_k3s_nmiss=0; g_k3s_held=0;
+ if(li<0 || nmiss<=0) return;
+ LCache *lc=&m->ecache[li];
+ int promo = nmiss<lc->cap ? nmiss : lc->cap;
+ for(int a=0;a<promo;a++){
+ int q=nmiss-1-a; Slot *dst;
+ if(lc->n<lc->cap) dst=&lc->s[lc->n++];
+ else {
+ int lru=-1;
+ for(int i=0;i<lc->n;i++){
+ if(lc->s[i].pinned) continue;
+ if(lru<0 || lc->s[i].used<lc->s[lru].used) lru=i;
+ }
+ if(lru<0) break;
+ dst=&lc->s[lru];
+ }
+ cache_promote(m,li,dst,&m->ws[q]);
+ }
+}
+static int k3_vk_load_batch(void *ctx,int li,const int *e,int n,VktExpertSrc *srcs,void **h){
+ Model *m=ctx;
+ k3_vk_promote(m);
+ if(n<1) return 0;
+ if(n>LP_MAX) n=LP_MAX;
+ if(n>64) n=64;
+ Slot *use[64]; int miss[64], nmiss=0;
+ for(int j=0;j<n;j++){
+ ehit_mark(m,li,e[j]);
+ use[j]=slot_find(m,li,e[j]);
+ if(!use[j]){ m->miss++; use[j]=&m->ws[nmiss]; miss[nmiss++]=j; }
+ }
+ if(nmiss){
+ double t0=now_s();
+ #pragma omp parallel for schedule(dynamic,1)
+ for(int q=0;q<nmiss;q++) expert_read(m,li,e[miss[q]],&m->ws[q]);
+ m->t_eload+=now_s()-t0;
+ m->ebytes+=(uint64_t)nmiss*(uint64_t)m->e_slot;
+ }
+ for(int j=0;j<n;j++){ srcs[j]=k3_vk_src(m,use[j]->buf); h[j]=m; }
+ g_k3s_layer=li; g_k3s_nmiss=nmiss; g_k3s_held=n;
+ return n;
+}
+static int k3_vk_load(void *ctx,int li,int e,VktExpertSrc *src,void **h){
+ return k3_vk_load_batch(ctx,li,&e,1,src,h)==1;
+}
+static void k3_vk_unhold(void *ctx,void *h){ (void)h; if(g_k3s_held>0 && --g_k3s_held==0) k3_vk_promote((Model *)ctx); }
/* ---- the shared Vulkan routed-expert tier, one MoE layer step -------------
* k3_vk_issue hands the step's (position, rank) pairs whose expert is resident
* on the device to one asynchronous batch (vkt_issue) as soon as z is known.
@@ -2244,7 +2505,7 @@ static void moe_forward(Model *m, Layer *l, int li, const float *x, int C, float
Cfg *c=&m->c; Moe *o=&l->moe;
int E=c->n_experts, K=c->topk, LT=c->latent;
float *sco=falloc((int64_t)C*E);
- matmul(sco,x,o->router,C,c->hidden,E);
+ if(!K3_F32_DEV(o->vk_router,sco,x,C,c->hidden,E)){ K3_F32_NEED(o->router); matmul(sco,x,o->router,C,c->hidden,E); }
int *idxs=malloc((size_t)C*K*sizeof(int)); float *wsels=falloc((int64_t)C*K);
int *keff=malloc((size_t)C*sizeof(int));
if(!idxs||!keff){fprintf(stderr,"OOM moe sel\n");exit(1);}
@@ -3361,6 +3622,9 @@ static int serve_one(Model *m, Tok *T, ServeReq *q){
}
int chunk=getenv("K3_CHUNK")?atoi(getenv("K3_CHUNK")):32;
if(chunk<1) chunk=1; if(chunk>512) chunk=512;
+#ifdef COLI_VULKAN
+ { int r=k3c_prefill_rows(m); if(r>0 && !g_lfp) chunk=r; } /* the chain's chunk from the budget: big MoE steps */
+#endif
double t0=now_s(), a0=m->t_attn, e0=m->t_moe, d0=m->t_eload, h0=m->t_head;
uint64_t hit0=m->hits, miss0=m->miss;
float *lo=NULL;
@@ -3617,6 +3881,9 @@ int main(int argc, char **argv){
int chunk=getenv("K3_CHUNK")?atoi(getenv("K3_CHUNK")):32;
if(chunk<1) chunk=1;
if(chunk>512) chunk=512;
+#ifdef COLI_VULKAN
+ { int r=k3c_prefill_rows(&m); if(r>0 && !g_lfp) chunk=r; } /* the chain's chunk from the budget: big MoE steps */
+#endif
if(m.trace && chunk>1){
chunk=1; /* trace rows are token-major by contract */
fprintf(stderr,"[K3] K3_TRACE set: prefill chunk forced to 1\n");
diff --git a/c/kimi_k3_chain.h b/c/kimi_k3_chain.h
index 50d57c70..cd7a38fa 100644
--- a/c/kimi_k3_chain.h
+++ b/c/kimi_k3_chain.h
@@ -47,6 +47,11 @@
* Prompt-cache and prefix reuse need nothing more: a reused prefix is rows below the
* watermark and a KDA state that already sits where the next step expects it.
*
+ * Past the device's budget (vk_kvsplit.h) each MLA layer keeps only a window of the
+ * newest blocks of its latent and rope rows on the device; the attention core runs
+ * over those on the device and over the older positions on the CPU (from the host's
+ * canonical cache) at once, merged through the softmax statistics.
+ *
* A device lost while it holds the newest KDA state: the state is rebuilt on the CPU
* from the host's copy (current at host_pos) and the prefix record's ids up to where
* the device was (a prefill's worth of CPU work), the forward runs again on the CPU
@@ -59,6 +64,7 @@
* K3_DEBUG_OUT), without a head (K3_LAYERS), and for a geometry past the ops' limits
* (a KDA head above 128 floats, kv_lora above 1024, qk_rope above 128 or odd). */
#include "vk_chain.h"
+#include "vk_kvsplit.h"
#define K3C_HOST 0
#define K3C_DEV 1
@@ -75,6 +81,9 @@ typedef struct {
K3cKda *kda; K3cFfn *ffn; VkcMla *mla; ColiVkTensor **mg, *head;
VkcMlaCache *kv; VkcBuf **win, **st;
int *kv_valid, *mla_ord, n_mla;
+ int dev_rows; /* device latent/rope rows a layer: cap, or the split's */
+ VkcKvSplit ks; /* the split past the device's budget (ks.on) */
+ VkcMlaCache kvtmp; /* the split: a step's new rows before their slots */
int where, host_zero, dev_pos, host_pos; /* the KDA state: who holds the newest, where each side is */
VkcMlaScratch sc;
VkcBuf *x, *bres, *hm, *nrm, *att, *qkv3, *cm, *t1, *kf, *kb, *kg, *ky, *gate, *lg, *z;
@@ -89,13 +98,48 @@ typedef struct {
static K3Chain *g_k3c;
static int g_k3c_on; /* COLI_VK_CHAIN as decided (on, or prompts only), the chain set up */
static int g_k3c_inited; /* vkc_init ran: vkc_shutdown goes at exit */
+/* The chain's own f32 tensors (the router, the KDA decay pair, beta): with the dense
+ * weights on the device only (COLI_VK_DENSE_HOST) the per-matrix path multiplies by them
+ * too, and the CPU drops its copies. which: 0 the router, 1 f_a, 2 f_b, 3 b_proj. */
+static ColiVkTensor *k3c_f32_tensor(int li, int which) {
+ K3Chain *ch = g_k3c;
+ if (!ch || !ch->ok || li < 0 || li >= ch->m->c.n_layers) return NULL;
+ switch (which) {
+ case 0: return ch->ffn[li].router;
+ case 1: return ch->kda[li].fa;
+ case 2: return ch->kda[li].fb;
+ case 3: return ch->kda[li].bp;
+ }
+ return NULL;
+}
-static int k3c_chunk_rows(void) {
- const char *e = getenv("COLI_VK_CHAIN_ROWS");
- int v = e && *e ? atoi(e) : 512;
- return v < 1 ? 1 : v > 65535 ? 65535 : v;
+/* k3c_res counts instead of reserving while g_k3c_count >= 0 (the chunk's sizing) */
+static long long g_k3c_count = -1;
+static int k3c_res(VkcBuf **b, size_t floats, int kind) {
+ if (g_k3c_count >= 0) { g_k3c_count += (long long)(floats ? floats : 1) * (long long)sizeof(float); return 1; }
+ return vkc_reserve(b, (floats ? floats : 1) * sizeof(float), kind);
+}
+static int k3c_scratch(K3Chain *ch, const Model *m, int rows);
+/* Prompt rows per chunk (vkc_chunk_rows): the chain's scratch a row, counted from the
+ * reservations (the MLA scratch as vkc_mla_scratch sizes it), and the routed experts'
+ * outputs (the tier's rows, the CPU's, the host's sum, in the latent space) for it. */
+static int k3c_chunk_rows(K3Chain *ch, const Model *m) {
+ if (!vkc_chunk_auto()) return vkc_chunk_rows("kimi_k3", 0);
+ const Cfg *c = &m->c;
+ size_t mla = 0;
+ for (int i = 0; i < c->n_layers; i++) if (!m->L[i].kda) {
+ const VkcMla *a = &ch->mla[i];
+ mla = (size_t)(a->q_lora > 0 ? a->q_lora : 1) + (size_t)a->H * (a->Q + a->R) + (size_t)(a->K + a->R) +
+ 2 * (size_t)a->H * a->K + (size_t)a->H * a->V;
+ break;
+ }
+ size_t ksz = ch->kvd_layer;
+ g_k3c_count = 0; k3c_scratch(ch, m, 1); long long b1 = g_k3c_count;
+ g_k3c_count = 0; k3c_scratch(ch, m, 2); long long b2 = g_k3c_count;
+ g_k3c_count = -1; ch->kvd_layer = ksz;
+ size_t row = (size_t)(b2 - b1) + mla * sizeof(float) + (size_t)(2 * c->topk + 1) * c->latent * sizeof(float);
+ return vkc_chunk_rows("kimi_k3", row);
}
-static int k3c_res(VkcBuf **b, size_t floats, int kind) { return vkc_reserve(b, (floats ? floats : 1) * sizeof(float), kind); }
/* A W's device copy into *t: the per-matrix path's own where it made one (the shared
* experts under COLI_VK_DENSE), else one of the chain's, which w_matmul never sees (so
@@ -147,7 +191,7 @@ static void k3c_host_wrote(Model *m, int zero) {
K3Chain *ch = g_k3c;
if (!ch || !ch->ok || ch->m != m) return;
ch->where = K3C_HOST; ch->host_zero = zero;
- if (zero) for (int i = 0; i < m->c.n_layers; i++) ch->kv_valid[i] = 0;
+ if (zero) { for (int i = 0; i < m->c.n_layers; i++) ch->kv_valid[i] = 0; vkc_kv_reset(&ch->ks); }
}
/* A CPU forward from pos0: the host's state current before it, the device's stale after
* (and its MLA rows from pos0 on). */
@@ -156,7 +200,10 @@ static void k3c_cpu_step(Model *m, int pos0) {
if (!ch || !ch->ok || ch->m != m) return;
k3c_sync_host(m);
ch->where = K3C_HOST; ch->host_zero = 0;
- for (int i = 0; i < m->c.n_layers; i++) if (ch->kv_valid[i] > pos0) ch->kv_valid[i] = pos0;
+ for (int i = 0; i < m->c.n_layers; i++) {
+ if (ch->kv_valid[i] > pos0) ch->kv_valid[i] = pos0;
+ if (!m->L[i].kda) vkc_kv_lower(&ch->ks, ch->mla_ord[i], pos0);
+ }
}
/* The device was lost. If it held the newest KDA state (dev), the state is rebuilt on
@@ -314,7 +361,7 @@ static int k3c_scratch(K3Chain *ch, const Model *m, int rows) {
int mfull = -1; for (int i = 0; i < c->n_layers; i++) if (!m->L[i].kda) { mfull = i; break; }
size_t r = (size_t)rows;
ch->kvd_layer = r * (c->kv_lora + R);
- int ok = (mfull < 0 || vkc_mla_scratch(&ch->sc, &ch->mla[mfull], rows)) &&
+ int ok = (mfull < 0 || g_k3c_count >= 0 || vkc_mla_scratch(&ch->sc, &ch->mla[mfull], rows)) &&
k3c_res(&ch->x, r * D, VKC_DEV) && k3c_res(&ch->bres, r * ch->nbmax * D, VKC_DEV) &&
k3c_res(&ch->hm, r * D, VKC_DEV) && k3c_res(&ch->nrm, r * D, VKC_DEV) && k3c_res(&ch->att, r * D, VKC_DEV) &&
k3c_res(&ch->qkv3, 3 * r * P, VKC_DEV) && k3c_res(&ch->cm, r * 3 * P, VKC_DEV) &&
@@ -328,6 +375,12 @@ static int k3c_scratch(K3Chain *ch, const Model *m, int rows) {
k3c_res(&ch->kvd, (size_t)(ch->n_mla ? ch->n_mla : 1) * ch->kvd_layer, VKC_DOWN) &&
k3c_res(&ch->find, r * D, VKC_DOWN) && k3c_res(&ch->outd, c->vocab, VKC_DOWN) &&
k3c_res(&ch->routed, r * LT, VKC_UP);
+ if (g_k3c_count >= 0) { /* count the split without allocating or changing capacities */
+ if (ch->ks.on && mfull >= 0)
+ g_k3c_count += (long long)r * (3LL * c->n_heads * (c->kv_lora + 2) +
+ (long long)c->n_heads * (c->kv_lora + R) + c->kv_lora + R) * sizeof(float);
+ return ok && (R <= 0 || k3c_res(&ch->cs, r * R, VKC_UP));
+ }
if (ok && R > 0) { /* NoPE: the rope parts pass through a rotation by angle 0 */
size_t had = vkc_bytes(ch->cs) / sizeof(float);
ok = k3c_res(&ch->cs, r * R, VKC_UP);
@@ -337,6 +390,10 @@ static int k3c_scratch(K3Chain *ch, const Model *m, int rows) {
for (size_t i = 0; i + 1 < nf; i += 2) { cs[i] = 1.f; cs[i + 1] = 0.f; }
}
}
+ if (ok && ch->ks.on && mfull >= 0) /* the split: the partial results, a step's new rows before their slots */
+ ok = vkc_kv_parts(&ch->ks, r * c->n_heads * (c->kv_lora + 2)) &&
+ (ch->kvtmp.cap >= rows || (k3c_res(&ch->kvtmp.lat, r * c->kv_lora, VKC_DEV) &&
+ (R == 0 || k3c_res(&ch->kvtmp.rope, r * R, VKC_DEV)) && (ch->kvtmp.cap = rows)));
if (!ok) return 0;
if (ch->rows < rows) {
float *hu = realloc(ch->host_u, r * LT * sizeof(float)), *hs = hu ? realloc(ch->host_sco, r * E * sizeof(float)) : NULL;
@@ -348,30 +405,37 @@ static int k3c_scratch(K3Chain *ch, const Model *m, int rows) {
return 1;
}
-/* The MLA mirror at the host's capacity (m->max_t), filled again when that grows. */
-static int k3c_mirror(K3Chain *ch, const Model *m) {
+/* The MLA mirror at the host's capacity (m->max_t), filled again when that grows, and
+ * planned again then (vkc_kv_plan): whole when it fits the device's budget, else the
+ * split, for steps of `rows` rows. */
+static int k3c_mirror(K3Chain *ch, const Model *m, int rows) {
const Cfg *c = &m->c;
if (ch->cap == m->max_t) return 1;
+ size_t row = (size_t)(c->kv_lora + c->qk_rope) * sizeof(float);
+ if (ch->n_mla && !vkc_kv_plan(&ch->ks, "kimi_k3", ch->n_mla, row, m->max_t, rows, 0, (size_t)ch->n_mla * ch->dev_rows * row))
+ return 0;
+ int dr = ch->ks.on ? ch->ks.rows : m->max_t;
for (int i = 0; i < c->n_layers; i++) {
if (m->L[i].kda) continue;
vkc_free(ch->kv[i].lat); vkc_free(ch->kv[i].rope);
ch->kv[i] = (VkcMlaCache){NULL, NULL, 0};
ch->kv_valid[i] = 0;
}
- ch->cap = 0;
+ ch->cap = 0; ch->dev_rows = 0;
for (int i = 0; i < c->n_layers; i++) {
if (m->L[i].kda) continue;
- ch->kv[i].lat = vkc_buf((size_t)m->max_t * c->kv_lora * sizeof(float), VKC_DEV);
- if (c->qk_rope > 0) ch->kv[i].rope = vkc_buf((size_t)m->max_t * c->qk_rope * sizeof(float), VKC_DEV);
- ch->kv[i].cap = m->max_t;
+ ch->kv[i].lat = vkc_buf((size_t)dr * c->kv_lora * sizeof(float), VKC_DEV);
+ if (c->qk_rope > 0) ch->kv[i].rope = vkc_buf((size_t)dr * c->qk_rope * sizeof(float), VKC_DEV);
+ ch->kv[i].cap = dr;
if (!ch->kv[i].lat || (c->qk_rope > 0 && !ch->kv[i].rope)) return 0;
}
- ch->cap = m->max_t;
+ ch->cap = m->max_t; ch->dev_rows = dr;
return 1;
}
-/* Record the uploads that make the device the host's below pb. */
-static int k3c_push(K3Chain *ch, const Model *m, int pb) {
+/* Record the uploads that make the device the host's below pb, for a step of n_rows
+ * rows (the split: its window placed first). */
+static int k3c_push(K3Chain *ch, const Model *m, int pb, int n_rows) {
const Cfg *c = &m->c; int ok = 1, K = c->kv_lora, R = c->qk_rope;
if (ch->where == K3C_HOST) {
size_t ns = (size_t)c->kda_heads * c->kda_hd * c->kda_hd, nw = (size_t)c->kda_proj * c->conv_k;
@@ -386,6 +450,13 @@ static int k3c_push(K3Chain *ch, const Model *m, int pb) {
ch->where = K3C_BOTH; ch->host_pos = pb;
}
for (int i = 0; i < c->n_layers && ok; i++) {
+ if (!m->L[i].kda && ch->ks.on) { /* the split: the window placed, its rows below pb uploaded */
+ VkcKvPart pt[2] = {{1, K, m->Lc[i], 0, ch->kv[i].lat, 0}, {1, R, m->Rc[i], 0, ch->kv[i].rope, 0}};
+ vkc_kv_place(&ch->ks, ch->mla_ord[i], pb, n_rows);
+ ok = vkc_kv_push(&ch->ks, ch->mla_ord[i], pt, R ? 2 : 1, pb);
+ ch->kv_valid[i] = pb;
+ continue;
+ }
if (m->L[i].kda || ch->kv_valid[i] >= pb) continue;
int t0 = ch->kv_valid[i], n = pb - t0;
ok = vkc_write(ch->kv[i].lat, (size_t)t0 * K, m->Lc[i] + (size_t)t0 * K, (size_t)n * K * sizeof(float)) &&
@@ -429,6 +500,12 @@ static int k3c_kda(K3Chain *ch, const Model *m, int i, int n) {
}
static int k3c_mla(K3Chain *ch, const Model *m, int i, int n, int pb) {
size_t ko = (size_t)ch->mla_ord[i] * ch->kvd_layer;
+ if (ch->ks.on) /* the split: the new rows into their slots, the core in two parts */
+ return vkc_kv_mla_qkv(&ch->ks, ch->mla_ord[i], &ch->mla[i], &ch->sc, ch->nrm, 0, n, pb, m->c.qk_rope > 0 ? ch->cs : NULL,
+ &ch->kv[i], &ch->kvtmp, ch->kvd, ko) &&
+ vkc_matmul(ch->mg[i], ch->nrm, 0, ch->gate, 0, n) &&
+ vkc_kv_mla_attn(&ch->ks, ch->mla_ord[i], &ch->mla[i], &ch->sc, n, pb, 0, &ch->kv[i], NULL, 0, 0, ch->gate, 0,
+ ch->att, 0, m->Lc[i], m->Rc[i]);
return vkc_mla_qkv(&ch->mla[i], &ch->sc, ch->nrm, 0, n, pb, m->c.qk_rope > 0 ? ch->cs : NULL, &ch->kv[i], ch->kvd, ko) &&
vkc_matmul(ch->mg[i], ch->nrm, 0, ch->gate, 0, n) &&
vkc_mla_attn(&ch->mla[i], &ch->sc, n, pb, 0, &ch->kv[i], NULL, 0, 0, ch->gate, 0, ch->att, 0);
@@ -467,6 +544,16 @@ static void k3c_moe_host(K3Chain *ch, Model *m, int li, int n) {
* final norm: the rows into *fin (when need_rows), the last row's logits into *last.
* 0: not taken (the CPU runs the layers; nothing the caller holds changed). On a
* cancel (poll), 1 with *cancelled set: the state is the caller's to reset. */
+/* The prompt block the engine hands the chain: its chunk, when the chain runs with a
+ * chunk from the budget (vkc_chunk_auto) and K3_CHUNK is not set; 0 = K3_CHUNK's (32). */
+static int k3c_prefill_rows(const Model *m) {
+ K3Chain *ch = g_k3c;
+ if (!g_k3c_on || !ch || !ch->ok || ch->failed || ch->m != m || getenv("K3_CHUNK") || !vkc_chunk_auto()) return 0;
+ if (m->trace || g_k3_val_layer >= 0 || g_k3_dfp) return 0;
+ if (!k3c_mirror(ch, m, 1)) return 0;
+ int rows = k3c_chunk_rows(ch, m);
+ return ch->ks.on && rows > ch->ks.chunk ? ch->ks.chunk : rows;
+}
static int k3c_forward(Model *m, const float *hidden, int pos0, int C, int need_rows, float **fin_out,
float **last_out, K3CancelPoll poll, void *pctx, int *cancelled) {
K3Chain *ch = g_k3c;
@@ -478,10 +565,12 @@ static int k3c_forward(Model *m, const float *hidden, int pos0, int C, int need_
return 0;
}
const Cfg *c = &m->c; int L = c->n_layers, D = c->hidden, E = c->n_experts, LT = c->latent;
- int CH = k3c_chunk_rows(), rows = C < CH ? C : CH;
int dev0 = ch->where == K3C_DEV, dev_start = ch->dev_pos;
if (vkc_lost()) { k3c_recover(m, dev0, dev_start); return 0; }
- if (!k3c_mirror(ch, m) || !k3c_scratch(ch, m, rows)) {
+ int mirror_ok = k3c_mirror(ch, m, 1);
+ int CH = mirror_ok ? k3c_chunk_rows(ch, m) : 1, rows = C < CH ? C : CH;
+ if (ch->ks.on && rows > ch->ks.chunk) rows = ch->ks.chunk; /* a step's rows fit the split's window */
+ if (!mirror_ok || !k3c_scratch(ch, m, rows)) {
if (vkc_lost()) { k3c_recover(m, dev0, dev_start); return 0; } /* lost while its buffers were made */
fprintf(stderr, "[VK] kimi_k3 chain: device memory for %d rows at %d positions refused; the CPU runs the layers\n",
rows, m->max_t);
@@ -493,7 +582,7 @@ static int k3c_forward(Model *m, const float *hidden, int pos0, int C, int need_
vkc_gemm_rows(-1);
for (int c0 = 0; c0 < C; c0 += rows) {
int n = C - c0 < rows ? C - c0 : rows, pb = pos0 + c0, end = c0 + n == C;
- if (!vkc_begin() || !k3c_push(ch, m, pb) || !vkc_write(ch->x, 0, hidden + (size_t)c0 * D, (size_t)n * D * sizeof(float)))
+ if (!vkc_begin() || !k3c_push(ch, m, pb, n) || !vkc_write(ch->x, 0, hidden + (size_t)c0 * D, (size_t)n * D * sizeof(float)))
goto lost;
int ok = 1, pending = 0, pulled = 0, nb = 0;
for (int i = 0; i < L && ok; i++) {
@@ -534,7 +623,11 @@ static int k3c_forward(Model *m, const float *hidden, int pos0, int C, int need_
}
ok = vkc_matmul(ch->ffn[i].router, ch->nrm, 0, ch->lg, 0, n) && vkc_matmul(ch->ffn[i].down, ch->nrm, 0, ch->z, 0, n) &&
vkc_copy(ch->lgd, 0, ch->lg, 0, (size_t)n * E) && vkc_copy(ch->zd, 0, ch->z, 0, (size_t)n * LT) &&
- vkc_submit(1); /* A1 */
+ vkc_submit(0); /* A1 */
+ /* while it runs, the tier loads the experts this layer will likely stream (a
+ * big prompt chunk only) */
+ if (ok) vkt_stream_prefetch(i, n);
+ ok = ok && vkc_finish();
m->t_attn += now_s() - ta;
if (!ok) break;
k3c_pull(ch, m, pulled, i + 1, pb, n); pulled = i + 1;
@@ -563,7 +656,7 @@ static int k3c_forward(Model *m, const float *hidden, int pos0, int C, int need_
if (!ok) goto lost;
k3c_pull(ch, m, pulled, L, pb, n);
if (need_rows) memcpy(fin + (size_t)c0 * D, vkc_ptr(ch->find), (size_t)n * D * sizeof(float));
- for (int i = 0; i < L; i++) if (!m->L[i].kda) ch->kv_valid[i] = pb + n;
+ for (int i = 0; i < L; i++) if (!m->L[i].kda) { ch->kv_valid[i] = pb + n; vkc_kv_done(&ch->ks, ch->mla_ord[i], pb + n); }
ch->where = K3C_DEV;
if (end) memcpy(last, vkc_ptr(ch->outd), (size_t)c->vocab * sizeof(float));
}
@@ -586,6 +679,7 @@ static void k3c_report(void) {
fprintf(stderr, "[VK] kimi_k3 chain: %llu forwards, %llu frames (%llu ops, %llu matmuls, %llu tiled GEMM), "
"%.1f ms waiting for the device, %.1f ms of routed experts on the host, %.1f MiB on the device\n",
ch->forwards, st.frames, st.ops, st.matmuls, st.gemms, st.wait_ms, ch->host_ms, st.dev_bytes / 1048576.0);
+ vkc_kv_report(&ch->ks);
vkc_prof_print();
}
diff --git a/c/laya.c b/c/laya.c
index 5cad196c..17df262e 100644
--- a/c/laya.c
+++ b/c/laya.c
@@ -673,8 +673,15 @@ static void add_rows(float *x, const float *y, size_t n)
for (size_t i = 0; i < n; i++) x[i] += y[i];
}
+#ifdef COLI_VULKAN
+#include "decide_vk.h" /* COLI_VULKAN=1: the forward, or its matrices, on a Vulkan device */
+#endif
+
static void gemm(float *Y, const float *X, int M, const QiMat *W, const float *bias)
{
+#ifdef COLI_VULKAN
+ if (dvk_gemm(Y, X, M, W, bias)) return; /* matrix by matrix (COLI_VK_CHAIN=0) */
+#endif
qi_gemm(Y, X, M, W, bias);
}
@@ -754,6 +761,9 @@ typedef struct {
double actions[DECIDE_MAX_ACTIONS];
} SeqOut;
+static void forward_tail(Laya *M, Seq *seqs, int S, const float *x, const int *first_row, const int *mark_row,
+ SeqOut *outs);
+
/* DecisionModel.forward over a batch of sequences. */
static void forward(Laya *M, Seq *seqs, int S, SeqOut *outs)
{
@@ -830,15 +840,31 @@ static void forward(Laya *M, Seq *seqs, int S, SeqOut *outs)
add_rows(x, h, (size_t)T * d);
}
+ int nm = 0;
+ for (int s = 0; s < S; s++) nm += seqs[s].n_markers;
+ int *mark_row = (int *)xmalloc((size_t)(nm > 0 ? nm : 1) * sizeof(int));
+ for (int s = 0, k = 0; s < S; s++)
+ for (int i = 0; i < seqs[s].n_markers; i++, k++) mark_row[k] = seq_off[s] + seqs[s].markers[i];
+ forward_tail(M, seqs, S, x, seq_off, mark_row, outs);
+ free(mark_row);
+ free(x); free(h); free(att); free(big); free(mid);
+ free(seq_off); free(row_seq); free(row_pos);
+}
+
+/* The scorer at the markers and the act head, on the final rows x: sequence s's first
+ * row is x[first_row[s]], its markers (all sequences' in order) x[mark_row[k]]. */
+static void forward_tail(Laya *M, Seq *seqs, int S, const float *x, const int *first_row, const int *mark_row,
+ SeqOut *outs)
+{
+ int d = M->d;
/* scorer at the markers: LayerNorm -> Linear -> GELU -> Linear(d, 1) */
int nm = 0;
for (int s = 0; s < S; s++) nm += seqs[s].n_markers;
float *m = (float *)xmalloc((size_t)(nm > 0 ? nm : 1) * d * sizeof(float));
float *mn = (float *)xmalloc((size_t)(nm > 0 ? nm : 1) * d * sizeof(float));
float *m1 = (float *)xmalloc((size_t)(nm > 0 ? nm : 1) * d * sizeof(float));
- for (int s = 0, k = 0; s < S; s++)
- for (int i = 0; i < seqs[s].n_markers; i++, k++)
- memcpy(m + (size_t)k * d, x + (size_t)(seq_off[s] + seqs[s].markers[i]) * d, (size_t)d * sizeof(float));
+ for (int k = 0; k < nm; k++)
+ memcpy(m + (size_t)k * d, x + (size_t)mark_row[k] * d, (size_t)d * sizeof(float));
layernorm(mn, m, M->sc_ln_w, M->sc_ln_b, nm, d, 1e-5f);
gemm(m1, mn, nm, &M->sc1, M->sc1_b);
for (int s = 0, k = 0; s < S; s++) {
@@ -858,7 +884,7 @@ static void forward(Laya *M, Seq *seqs, int S, SeqOut *outs)
for (int s = 0; s < S; s++) {
int K = seqs[s].n_markers;
float *f = feat + (size_t)s * (d + 4);
- memcpy(f, x + (size_t)seq_off[s] * d, (size_t)d * sizeof(float));
+ memcpy(f, x + (size_t)first_row[s] * d, (size_t)d * sizeof(float));
float mx = -INFINITY;
for (int i = 0; i < K; i++) if ((float)outs[s].logits[i] > mx) mx = (float)outs[s].logits[i];
float *p = (float *)xmalloc((size_t)(K > 0 ? K : 1) * sizeof(float));
@@ -888,10 +914,242 @@ static void forward(Laya *M, Seq *seqs, int S, SeqOut *outs)
}
free(feat); free(hid); free(m); free(mn); free(m1);
- free(x); free(h); free(att); free(big); free(mid);
- free(seq_off); free(row_seq); free(row_pos);
}
+#ifdef COLI_VULKAN
+/* ------------------------------------------- the forward on the device */
+
+/* forward() as frames of the dense chain (decide_vk.h): the embedding rows go up, the
+ * encoder and the head layers run on the device in forward()'s order with its
+ * arithmetic (f32 throughout), and the rows forward_tail() reads come back. The
+ * norms' weights, the biases, the type embedding and the RoPE tables are uploaded once. */
+typedef struct {
+ int ok;
+ VkcBuf *prm, *tab;
+ VkcBuf *x, *h, *big, *mid, *att, *ipos, *rg, *rl, *tid, *down;
+ int emb_w, emb_b, fin_w, fin_b, type_emb;
+ int *an_w, *an_b, *mn_w, *mn_b, *bqkv, *bo, *bi, *bo2; /* per encoder layer */
+ int *n1w, *n1b, *n2w, *n2b, *inb, *outb, *l1b, *l2b; /* per head layer */
+ int cos_g, sin_g, cos_l, sin_l;
+} LayaVk;
+static LayaVk g_lvk;
+
+/* Every matrix's device copy now (the first request then pays nothing), and with the
+ * forward on the device, its parameters and tables. */
+static void laya_vk_setup(Laya *M)
+{
+ if (!g_dvk.ready || (!g_dvk.chain && !g_dvk.dense)) return; /* nothing of it on the device */
+ g_dvk.f16 = 1; /* the release stores F16: the device reads those values in half the bytes */
+ int n = 0, want = 0;
+ for (int l = 0; l < M->layers; l++) {
+ EncLayer *E = &M->L[l];
+ const QiMat *w[4] = {&E->wqkv, &E->wo, &E->wi, &E->wo2};
+ for (int k = 0; k < 4; k++, want++) n += dvk_tensor(w[k]) != NULL;
+ }
+ for (int l = 0; l < M->head_layers; l++) {
+ HeadLayer *H = &M->H[l];
+ const QiMat *w[4] = {&H->in_proj, &H->out_proj, &H->lin1, &H->lin2};
+ for (int k = 0; k < 4; k++, want++) n += dvk_tensor(w[k]) != NULL;
+ }
+ want += 2;
+ n += dvk_tensor(&M->sc1) != NULL;
+ n += dvk_tensor(&M->act1) != NULL;
+ size_t bytes = 0, tensors = 0;
+ coli_vk_mem_info(&bytes, &tensors);
+ fprintf(stderr, "[VK] laya: %d of %d matrices on the device (%.1f MiB; %u in f16, every value the checkpoint's, "
+ "%u in f32)\n", n, want, bytes / 1048576.0, g_dvk.n_f16, g_dvk.n_f32);
+ if (!g_dvk.chain) return;
+ if (n != want || M->hd > 128 || M->d / M->head_heads > 128 || M->d > 4096) {
+ fprintf(stderr, "[VK] laya: the forward stays on the CPU (%s)\n",
+ n != want ? "a matrix did not reach the device" : "a geometry past the chain's shaders");
+ g_dvk.chain = 0;
+ g_dvk.dense = coli_vk_dense();
+ return;
+ }
+ LayaVk *V = &g_lvk;
+ DvkPrm P = {0};
+ int L = M->layers, HL = M->head_layers > 0 ? M->head_layers : 1, d = M->d;
+ int **lv[8] = {&V->an_w, &V->an_b, &V->mn_w, &V->mn_b, &V->bqkv, &V->bo, &V->bi, &V->bo2};
+ int **hv[8] = {&V->n1w, &V->n1b, &V->n2w, &V->n2b, &V->inb, &V->outb, &V->l1b, &V->l2b};
+ for (int k = 0; k < 8; k++) { *lv[k] = (int *)xcalloc((size_t)L, sizeof(int)); *hv[k] = (int *)xcalloc((size_t)HL, sizeof(int)); }
+ V->emb_w = dvk_prm_add(&P, M->emb_norm_w, d);
+ V->emb_b = dvk_prm_add(&P, M->emb_norm_b, d);
+ V->fin_w = dvk_prm_add(&P, M->final_norm_w, d);
+ V->fin_b = dvk_prm_add(&P, M->final_norm_b, d);
+ V->type_emb = dvk_prm_add(&P, M->type_emb, (size_t)3 * d);
+ for (int l = 0; l < L; l++) {
+ EncLayer *E = &M->L[l];
+ V->an_w[l] = dvk_prm_add(&P, E->attn_norm_w, d);
+ V->an_b[l] = dvk_prm_add(&P, E->attn_norm_b, d);
+ V->mn_w[l] = dvk_prm_add(&P, E->mlp_norm_w, d);
+ V->mn_b[l] = dvk_prm_add(&P, E->mlp_norm_b, d);
+ V->bqkv[l] = dvk_prm_add(&P, E->bqkv, (size_t)3 * d);
+ V->bo[l] = dvk_prm_add(&P, E->bo, d);
+ V->bi[l] = dvk_prm_add(&P, E->bi, (size_t)2 * M->inter);
+ V->bo2[l] = dvk_prm_add(&P, E->bo2, d);
+ }
+ for (int l = 0; l < M->head_layers; l++) {
+ HeadLayer *H = &M->H[l];
+ V->n1w[l] = dvk_prm_add(&P, H->n1w, d);
+ V->n1b[l] = dvk_prm_add(&P, H->n1b, d);
+ V->n2w[l] = dvk_prm_add(&P, H->n2w, d);
+ V->n2b[l] = dvk_prm_add(&P, H->n2b, d);
+ V->inb[l] = dvk_prm_add(&P, H->in_proj_b, (size_t)3 * d);
+ V->outb[l] = dvk_prm_add(&P, H->out_proj_b, d);
+ V->l1b[l] = dvk_prm_add(&P, H->lin1_b, (size_t)4 * d);
+ V->l2b[l] = dvk_prm_add(&P, H->lin2_b, d);
+ }
+ V->prm = dvk_prm_upload(&P);
+ free(P.v);
+ DvkPrm T = {0};
+ size_t tn = (size_t)M->max_pos * (M->hd / 2);
+ V->cos_g = dvk_prm_add(&T, M->rope_g, tn);
+ V->sin_g = dvk_prm_add(&T, M->rope_g + tn, tn);
+ V->cos_l = dvk_prm_add(&T, M->rope_l, tn);
+ V->sin_l = dvk_prm_add(&T, M->rope_l + tn, tn);
+ V->tab = dvk_prm_upload(&T);
+ free(T.v);
+ if (!V->prm || !V->tab) {
+ fprintf(stderr, "[VK] laya: the forward stays on the CPU (no device memory for its parameters)\n");
+ g_dvk.chain = 0;
+ g_dvk.dense = coli_vk_dense();
+ return;
+ }
+ V->ok = 1;
+}
+
+/* x = LN(x + h) with the sum kept in x, or LN(x) alone; into y */
+static int lvk_norm(VkcBuf *x, VkcBuf *h, VkcBuf *y, int T, int d, int w, int b, int add, float eps)
+{
+ VkcEncNorm p = {0, T, d, 0, d, 0, d, 0, d, w, b, add ? VKC_ENC_ADD | VKC_ENC_SUM : 0, eps};
+ return vkc_enc_norm(x, h, g_lvk.prm, y, &p);
+}
+static int lvk_lin(const QiMat *W, VkcBuf *x, VkcBuf *y, int T, int bias, int act)
+{
+ ColiVkTensor *t = dvk_tensor(W);
+ if (!t || !vkc_matmul(t, x, 0, y, 0, T)) return 0;
+ if (bias < 0 && act == VKC_ENC_ACT_NONE) return 1;
+ VkcEncBias p = {0, T * W->N, W->N, 0, W->N, bias, act};
+ return vkc_enc_bias(y, g_lvk.prm, &p);
+}
+
+/* forward() for the S sequences on the device; 0 = the caller runs forward(). */
+static int laya_vk_forward(Laya *M, Seq *seqs, int S, SeqOut *outs)
+{
+ LayaVk *V = &g_lvk;
+ if (!g_dvk.chain || !V->ok || !vkc_ready()) return 0;
+ double t0 = dvk_now_ms();
+ int d = M->d, I = M->inter, T = 0, nm = 0;
+ for (int s = 0; s < S; s++) { T += seqs[s].n; nm += seqs[s].n_markers; }
+ if (T < 1 || T > 65535) return 0;
+ int wide = 3 * d;
+ if (2 * I > wide) wide = 2 * I;
+ if (4 * d > wide) wide = 4 * d;
+ int nread = S + nm;
+ size_t fd = sizeof(float), id = sizeof(int);
+ if (!vkc_reserve(&V->x, (size_t)T * d * fd, VKC_DEV) || !vkc_reserve(&V->h, (size_t)T * d * fd, VKC_DEV) ||
+ !vkc_reserve(&V->att, (size_t)T * d * fd, VKC_DEV) || !vkc_reserve(&V->big, (size_t)T * wide * fd, VKC_DEV) ||
+ !vkc_reserve(&V->mid, (size_t)T * (I > d ? I : d) * fd, VKC_DEV) ||
+ !vkc_reserve(&V->ipos, (size_t)T * id, VKC_DEV) || !vkc_reserve(&V->rg, (size_t)2 * T * id, VKC_DEV) ||
+ !vkc_reserve(&V->rl, (size_t)2 * T * id, VKC_DEV) || !vkc_reserve(&V->tid, (size_t)T * id, VKC_DEV) ||
+ !vkc_reserve(&V->down, (size_t)nread * d * fd, VKC_DOWN))
+ return 0;
+ /* the host's part: the embedding rows, and per row its position, its question type
+ * and the rows its attention sees (the sequence; the window inside it) */
+ float *emb = (float *)xmalloc((size_t)T * d * fd);
+ int *pos = (int *)xmalloc((size_t)T * id), *tid = (int *)xmalloc((size_t)T * id);
+ int *rg = (int *)xmalloc((size_t)2 * T * id), *rl = (int *)xmalloc((size_t)2 * T * id);
+ VkcRegion *reg = (VkcRegion *)xmalloc((size_t)nread * sizeof(VkcRegion));
+ for (int s = 0, r = 0, k = S; s < S; s++) {
+ int start = r, L = seqs[s].n;
+ reg[s] = (VkcRegion){(size_t)s * d, (size_t)start * d, (size_t)d};
+ for (int i = 0; i < seqs[s].n_markers; i++, k++)
+ reg[k] = (VkcRegion){(size_t)k * d, (size_t)(start + seqs[s].markers[i]) * d, (size_t)d};
+ for (int i = 0; i < L; i++, r++) {
+ int id_ = seqs[s].ids[i];
+ if (id_ < 0 || id_ >= M->vocab) id_ = 0;
+ memcpy(emb + (size_t)r * d, M->tok_emb + (size_t)id_ * d, (size_t)d * fd);
+ pos[r] = i; tid[r] = seqs[s].qtype;
+ rg[2 * r] = start; rg[2 * r + 1] = start + L - 1;
+ int lo = i - M->window < 0 ? 0 : i - M->window, hi = i + M->window > L - 1 ? L - 1 : i + M->window;
+ rl[2 * r] = start + lo; rl[2 * r + 1] = start + hi;
+ }
+ }
+ VkcBuf *x = V->x, *h = V->h, *big = V->big, *mid = V->mid, *att = V->att;
+ int ok = vkc_begin() && vkc_write(h, 0, emb, (size_t)T * d * fd) && vkc_write(V->ipos, 0, pos, (size_t)T * id) &&
+ vkc_write(V->tid, 0, tid, (size_t)T * id) && vkc_write(V->rg, 0, rg, (size_t)2 * T * id) &&
+ vkc_write(V->rl, 0, rl, (size_t)2 * T * id);
+ free(emb); free(pos); free(tid); free(rg); free(rl);
+ float eps = M->eps;
+ ok = ok && lvk_norm(h, NULL, x, T, d, V->emb_w, V->emb_b, 0, eps);
+ float scale = 1.0f / sqrtf((float)M->hd);
+ for (int l = 0; ok && l < M->layers; l++) {
+ EncLayer *E = &M->L[l];
+ /* layer 0 has no attention norm; the others' norm joined the previous add below */
+ VkcBuf *in = E->attn_norm_w ? h : x;
+ if (l == 0 && E->attn_norm_w) ok = lvk_norm(x, NULL, h, T, d, V->an_w[l], V->an_b[l], 0, eps);
+ ok = ok && lvk_lin(&E->wqkv, in, big, T, V->bqkv[l], VKC_ENC_ACT_NONE);
+ VkcEncRope rp = {0, T * M->heads * M->hd, M->heads, M->hd, 0, 3 * d, d,
+ E->global ? V->cos_g : V->cos_l, E->global ? V->sin_g : V->sin_l};
+ ok = ok && vkc_enc_rope(big, V->tab, V->ipos, &rp);
+ VkcEncAttn ap = {0, T, M->heads, M->hd, 0, d, 2 * d, 3 * d, 0, d, 0, 0, 0, 0, 0, scale, 0};
+ ok = ok && vkc_enc_attn(big, NULL, NULL, att, E->global ? V->rg : V->rl, NULL, NULL, &ap);
+ ok = ok && lvk_lin(&E->wo, att, h, T, V->bo[l], VKC_ENC_ACT_NONE);
+ ok = ok && lvk_norm(x, h, h, T, d, V->mn_w[l], V->mn_b[l], 1, eps); /* x += h; h = mlp_norm(x) */
+ ok = ok && lvk_lin(&E->wi, h, big, T, -1, VKC_ENC_ACT_NONE);
+ VkcEncGeglu gp = {0, T * I, I, 0, 2 * I, 0, I, V->bi[l],
+ M->gelu_tanh ? VKC_ENC_ACT_GELU_TANH : VKC_ENC_ACT_GELU};
+ ok = ok && vkc_enc_geglu(big, V->prm, mid, &gp);
+ ok = ok && lvk_lin(&E->wo2, mid, h, T, V->bo2[l], VKC_ENC_ACT_NONE);
+ if (l + 1 < M->layers && M->L[l + 1].attn_norm_w) /* x += h; h = attn_norm(x) */
+ ok = ok && lvk_norm(x, h, h, T, d, V->an_w[l + 1], V->an_b[l + 1], 1, eps);
+ else if (l + 1 < M->layers) {
+ VkcEw ep = {VKC_EW_ADD, T * d, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.f};
+ ok = ok && vkc_ew(x, x, h, NULL, NULL, &ep);
+ }
+ }
+ /* x += h; the final norm into h, which carries the stream from here (forward()'s swap) */
+ ok = ok && lvk_norm(x, h, h, T, d, V->fin_w, V->fin_b, 1, eps);
+ { VkcBuf *t = x; x = h; h = t; }
+ VkcEncAddRow tp = {0, T * d, d, 0, d, V->type_emb};
+ ok = ok && vkc_enc_addrow(x, V->prm, V->tid, &tp);
+ int hh = M->head_heads, hhd = d / hh;
+ float hscale = 1.0f / sqrtf((float)hhd);
+ for (int l = 0; ok && l < M->head_layers; l++) {
+ HeadLayer *H = &M->H[l];
+ if (l == 0) ok = lvk_norm(x, NULL, h, T, d, V->n1w[l], V->n1b[l], 0, 1e-5f);
+ ok = ok && lvk_lin(&H->in_proj, h, big, T, V->inb[l], VKC_ENC_ACT_NONE);
+ VkcEncAttn ap = {0, T, hh, hhd, 0, d, 2 * d, 3 * d, 0, d, 0, 0, 0, 0, 0, hscale, 0};
+ ok = ok && vkc_enc_attn(big, NULL, NULL, att, V->rg, NULL, NULL, &ap);
+ ok = ok && lvk_lin(&H->out_proj, att, h, T, V->outb[l], VKC_ENC_ACT_NONE);
+ ok = ok && lvk_norm(x, h, h, T, d, V->n2w[l], V->n2b[l], 1, 1e-5f); /* x += h; h = norm2(x) */
+ ok = ok && lvk_lin(&H->lin1, h, big, T, V->l1b[l], VKC_ENC_ACT_RELU);
+ ok = ok && lvk_lin(&H->lin2, big, h, T, V->l2b[l], VKC_ENC_ACT_NONE);
+ if (l + 1 < M->head_layers) /* x += h; h = norm1(x) */
+ ok = ok && lvk_norm(x, h, h, T, d, V->n1w[l + 1], V->n1b[l + 1], 1, 1e-5f);
+ else {
+ VkcEw ep = {VKC_EW_ADD, T * d, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.f};
+ ok = ok && vkc_ew(x, x, h, NULL, NULL, &ep);
+ }
+ }
+ ok = ok && vkc_copy_regions(V->down, x, reg, nread) && vkc_submit(1);
+ free(reg);
+ if (!ok) {
+ if (vkc_lost()) fprintf(stderr, "[VK] laya: the device was lost: the forward runs on the CPU\n");
+ else vkc_finish();
+ return 0;
+ }
+ int *first = (int *)xmalloc((size_t)S * id), *mark = (int *)xmalloc((size_t)(nm > 0 ? nm : 1) * id);
+ for (int s = 0; s < S; s++) first[s] = s;
+ for (int k = 0; k < nm; k++) mark[k] = S + k;
+ forward_tail(M, seqs, S, (const float *)vkc_ptr(V->down), first, mark, outs);
+ free(first); free(mark);
+ g_dvk.fwd_dev++; g_dvk.rows_dev += (unsigned long long)T;
+ g_dvk.dev_ms += dvk_now_ms() - t0;
+ return 1;
+}
+#endif
+
/* ------------------------------------------------------------ calibration */
/* common.temp_bucket */
@@ -945,9 +1203,16 @@ static int laya_decide(void *opaque, const DecideRecord *rec, DecideAnswer *answ
for (int start = 0; start < Q;) {
int end = start, rows = 0;
while (end < Q && (end == start || rows + seqs[end].n <= LAYA_MAX_BATCH_ROWS)) rows += seqs[end++].n;
+#ifdef COLI_VULKAN
+ if (laya_vk_forward(M, seqs + start, end - start, outs + start)) { start = end; continue; }
+ if (g_dvk.chain) g_dvk.fwd_cpu++;
+#endif
forward(M, seqs + start, end - start, outs + start);
start = end;
}
+#ifdef COLI_VULKAN
+ dvk_report();
+#endif
int total = 0;
for (int q = 0; q < Q; q++) {
DecideAnswer *a = &answers[q];
@@ -1110,6 +1375,9 @@ int main(int argc, char **argv)
fprintf(stderr, "[laya] %s: ModernBERT %d layers x %d, %d heads; head %d layers; max_len %d "
"(head %d); loaded in %.1f s\n", M->model_name, M->layers, M->d, M->heads,
M->head_layers, M->max_len, M->head_max_len, (now_ms() - t0) / 1e3);
+#ifdef COLI_VULKAN
+ if (!tokenize) { dvk_init("laya"); laya_vk_setup(M); } /* COLI_VULKAN=1: the device */
+#endif
if (tokenize) return run_tokenize(M, tokenize);
if (records) return run_records(M, records, dump_ids);
if (!serving) die("nothing to do: set SERVE=1, or pass --records / --tokenize");
diff --git a/c/mimo.c b/c/mimo.c
index 6ac06050..a3f6d773 100644
--- a/c/mimo.c
+++ b/c/mimo.c
@@ -74,7 +74,11 @@
* matrix whose upload fails, stays on the CPU path below. The backend's dense
* matmuls share one command buffer: main thread only. */
#include "backend_vulkan.h"
+#include <pthread.h>
static int g_vk_ready;
+/* COLI_VK_DENSE_HOST: the dense matrices on the device only (mimo_dho_start); the
+ * per-matrix path then runs on the device whatever coli_vk_dense() says */
+static int g_mimo_dho;
#endif
#define MIMO_MAX_LAYERS 128
@@ -263,6 +267,10 @@ typedef struct {
float *s; /* FP8: [O][nblk]; I8: [O] */
void *vk; /* COLI_VULKAN: the device copy, uploaded at start-up */
int vk_off; /* its upload failed: this matrix stays on the CPU */
+ /* COLI_VULKAN: how to read it back from disk (dw_reload), and 1 while the device
+ * holds it alone (COLI_VK_DENSE_HOST, the host copy dropped) */
+ char *vk_name;
+ int vk_kind, vk_bits, vk_li, vk_gone;
} DW;
/* 0 (default): the checkpoint's own FP8 and BF16 bytes, exact.
@@ -355,18 +363,20 @@ static int dw_upload(DW *d) {
}
static int dw_matmul_vk(float *y, const float *x, int S, const DW *d) {
- if (!g_vk_ready || !coli_vk_dense() || d->vk_off || !vk_main_thread()) return 0;
+ if (!g_vk_ready || !(coli_vk_dense() || g_mimo_dho) || d->vk_off || !vk_main_thread()) return 0;
DW *dev = (DW *)d; /* the device copy is a cache inside a read-only matrix */
if (!dw_upload(dev)) return 0;
int gs; const float *sc;
int fmt = dw_vk_fmt(d, &gs, &sc);
return coli_vk_matmul((ColiVkTensor **)&dev->vk, y, x, d->w, sc, fmt, S, d->I, d->O, gs);
}
+static void dw_reload(DW *d); /* below, beside the loaders it re-runs */
#endif
static void dw_matmul(float *y, const float *x, int S, const DW *d) {
#ifdef COLI_VULKAN
if (dw_matmul_vk(y, x, S, d)) return;
+ if (d->vk_gone) dw_reload((DW *)d); /* the CPU needs a matrix the device held alone (a lost device) */
#endif
switch (d->fmt) {
case DW_F32: matmul(y, x, (const float *)d->w, S, d->I, d->O); break;
@@ -417,8 +427,24 @@ static void dw_from_f32(DW *d, float *w, int O, int I) {
}
}
+#ifdef COLI_VULKAN
+enum { DWK_NONE = 0, DWK_BF16, DWK_FP8, DWK_QKV };
+/* What dw_reload needs to read the matrix back exactly as it was loaded: its name,
+ * its loader, the MIMO_DENSE_BITS it was loaded under (the tower forces its own). */
+static void dw_note(DW *d, const char *name, int kind) {
+ free(d->vk_name);
+ d->vk_name = strdup(name);
+ if (!d->vk_name) { fprintf(stderr, "OOM weight name\n"); exit(1); }
+ d->vk_kind = kind; d->vk_bits = g_dense_bits; d->vk_li = -1;
+}
+#define DW_NOTE(d, name, kind) dw_note(d, name, kind)
+#else
+#define DW_NOTE(d, name, kind) ((void)0)
+#endif
+
static void dw_load_bf16(DW *d, shards *S, const char *name, int O, int I) {
st_tensor *t = need(S, name, 0, O, I);
+ DW_NOTE(d, name, DWK_BF16);
if (g_dense_bits == 0) {
d->fmt = DW_BF16; d->O = O; d->I = I; d->s = NULL;
d->w = xmalloc((size_t)t->nbytes, name);
@@ -441,6 +467,7 @@ static void dw_load_fp8(DW *d, shards *S, const char *name, int O, int I,
if (scale_rows <= 0) scale_rows = (O + 127) / 128;
st_tensor *t = need(S, name, 4, O, I);
need(S, sname, 2, scale_rows, nblk);
+ DW_NOTE(d, name, DWK_FP8);
uint8_t *raw = xmalloc((size_t)t->nbytes, name);
st_read_raw_cap(S, name, raw, t->nbytes, 1);
float *grid = xmalloc((size_t)scale_rows * nblk * sizeof(float), sname);
@@ -509,7 +536,42 @@ static void load_qkv(DW *d, shards *S, const Cfg *c, int li) {
for (int ch = 0; ch < T; ch++) for (int r = 0; r < v_rows; r++) order[w++] = ch * rpc + q_rows + k_rows + r;
dw_load_fp8(d, S, name, O, c->hidden, row_block, scale_rows, order);
free(row_block); free(order);
+#ifdef COLI_VULKAN
+ d->vk_kind = DWK_QKV; d->vk_li = li; /* read back through this function: the de-interleave */
+#endif
+}
+
+#ifdef COLI_VULKAN
+/* A matrix whose host copy was dropped (COLI_VK_DENSE_HOST) and that the CPU needs
+ * after all: read it back from the checkpoint with the loader and the MIMO_DENSE_BITS
+ * it was first read with, the same bytes, and keep it from there on. */
+static size_t dw_bytes(const DW *d);
+static pthread_mutex_t g_dw_dho_mx = PTHREAD_MUTEX_INITIALIZER;
+static shards *g_dw_dho_S;
+static const Cfg *g_dw_dho_c;
+static void dw_reload(DW *d) {
+ pthread_mutex_lock(&g_dw_dho_mx);
+ if (d->vk_gone) {
+ DW t = {0};
+ int saved = g_dense_bits;
+ g_dense_bits = d->vk_bits;
+ if (!g_dw_dho_S || !d->vk_name) { fprintf(stderr, "[VK] mimo: a dense matrix the device held alone cannot be read back\n"); exit(1); }
+ switch (d->vk_kind) {
+ case DWK_BF16: dw_load_bf16(&t, g_dw_dho_S, d->vk_name, d->O, d->I); break;
+ case DWK_FP8: dw_load_fp8(&t, g_dw_dho_S, d->vk_name, d->O, d->I, NULL, 0, NULL); break;
+ case DWK_QKV: load_qkv(&t, g_dw_dho_S, g_dw_dho_c, d->vk_li); break;
+ default: fprintf(stderr, "[VK] mimo: %s has no loader to read it back\n", d->vk_name); exit(1);
+ }
+ g_dense_bits = saved;
+ if (t.fmt != d->fmt || t.O != d->O || t.I != d->I) { fprintf(stderr, "[VK] mimo: %s came back in another form\n", d->vk_name); exit(1); }
+ d->w = t.w; d->s = t.s; d->nblk = t.nblk;
+ free(t.vk_name);
+ d->vk_gone = 0;
+ coli_vk_dense_host_reloaded(dw_bytes(d));
+ }
+ pthread_mutex_unlock(&g_dw_dho_mx);
}
+#endif
/* ------------------------------------------------------------------ model ---- */
@@ -902,6 +964,23 @@ static VktExpertSrc mimo_vk_src(const Model *m, const uint8_t *buf) {
*uq = gs + m->part[3], *us = uq + m->part[4];
return (VktExpertSrc){gq, uq, dq, gs, us, ds};
}
+/* The tier's streaming (a big prompt chunk's cold experts on the device): the experts'
+ * bytes through the layer cache as moe_cpu gets them (experts_ensure: a group at once,
+ * read in parallel, up to the cache's capacity), valid until the next ensure. */
+static int mimo_vk_load_batch(void *ctx, int li, const int *e, int n, VktExpertSrc *srcs, void **h) {
+ Model *m = ctx;
+ int cap = m->cache[li].cap;
+ if (n > cap) n = cap;
+ if (n < 1 || n > 64) return 0;
+ Slot *sl[64];
+ experts_ensure(m, li, e, n, sl);
+ for (int i = 0; i < n; i++) { srcs[i] = mimo_vk_src(m, sl[i]->buf); h[i] = sl[i]; }
+ return n;
+}
+static int mimo_vk_load(void *ctx, int li, int e, VktExpertSrc *src, void **h) {
+ return mimo_vk_load_batch(ctx, li, &e, 1, src, h) == 1;
+}
+static void mimo_vk_release(void *ctx, void *h) { (void)ctx; (void)h; }
/* The CPU's experts of one MoE step: every (row, choice) pair i of sel -- or,
* with a mask, those whose mask[i] is mval -- runs its expert, and its output
@@ -1217,6 +1296,15 @@ static void echo_frame(const Echo *e, int pos, int token, const float *lo, int v
static void prefill(Model *m, const int *ids, int n, float *logits, float *all_logits,
const ImageRows *img, const Echo *echo) {
int chunk = env_int("MIMO_CHUNK", 64);
+#ifdef COLI_VULKAN
+ /* the dense chain with a chunk from the budget (vkc_chunk_auto): blocks of its chunk,
+ * so a long prompt's MoE steps are big ones the tier can stream (not with a read-out,
+ * which holds a block's logits) */
+ if (!getenv("MIMO_CHUNK") && g_vk_chain && vkc_chunk_auto() && !echo) {
+ MimoChain *ch = mc_setup(m);
+ if (ch && !ch->failed) chunk = mc_chunk_rows(ch, m);
+ }
+#endif
if (chunk < 1) chunk = 1;
int used = 0, V = m->c.vocab;
float *rows = NULL;
@@ -1301,6 +1389,50 @@ static size_t vk_dense_bytes(const Model *m, int vision) {
return b;
}
+/* ---- the dense matrices on the device only (COLI_VK_DENSE_HOST) ----------------
+ * With the trunk on the device (the chain, or COLI_VK_DENSE), every trunk matrix goes
+ * up now, before the expert tier sizes its budget, and its host copy is dropped; the
+ * tower too when the per-matrix path is on (COLI_VK_DENSE), else it stays on the CPU
+ * with its copy. The per-matrix path then answers on the device whatever
+ * coli_vk_dense() says (the CPU holds no copy), and the CPU reads a matrix back from
+ * disk only when it needs it after all (dw_reload: a lost device). Kept on the host:
+ * the embedding (its rows are gathered on the CPU), the routers (the host computes
+ * them), norms and vectors, the patch embedding. */
+static void dw_drop(DW *d, int *n) {
+ if (!d->w || d->vk_off || d->vk_gone || !d->vk_name || !dw_upload(d)) return;
+ size_t b = dw_bytes(d);
+ free(d->w); free(d->s);
+ d->w = NULL; d->s = NULL; d->vk_gone = 1;
+ coli_vk_dense_host_dropped(b);
+ (*n)++;
+}
+static void mimo_dho_start(Model *m) {
+ if (!g_vk_ready) return;
+ int vision = coli_vk_dense();
+ if (!coli_vk_dense_host_decide("mimo", coli_vk_dense() || g_vk_chain, vk_dense_bytes(m, vision))) return;
+ g_mimo_dho = 1;
+ g_dw_dho_S = &m->S; g_dw_dho_c = &m->c;
+ int n = 0;
+ for (int li = 0; li < m->c.n_layers; li++) {
+ Layer *l = &m->L[li];
+ dw_drop(&l->qkv, &n); dw_drop(&l->o, &n);
+ if (!m->c.moe[li]) { dw_drop(&l->gate, &n); dw_drop(&l->up, &n); dw_drop(&l->down, &n); }
+ }
+ dw_drop(&m->head, &n);
+ if (g_vision) {
+ Vision *v = g_vision;
+ DW *t[3] = {&v->embed, &v->fc1, &v->fc2};
+ for (int k = 0; k < 3; k++) { if (vision) dw_drop(t[k], &n); else t[k]->vk_off = 1; }
+ for (int i = 0; i < v->depth; i++) {
+ DW *b[5] = {&v->b[i].qkv, &v->b[i].proj, &v->b[i].gate, &v->b[i].up, &v->b[i].down};
+ for (int k = 0; k < 5; k++) { if (vision) dw_drop(b[k], &n); else b[k]->vk_off = 1; }
+ }
+ }
+ coli_vk_dense_host_placed("mimo", vision || !g_vision
+ ? "the embedding (its rows are gathered on the CPU), the routers, norms, the patch embedding"
+ : "the embedding (its rows are gathered on the CPU), the routers, norms, the vision tower (on the CPU: COLI_VK_DENSE=1 puts it on the device)");
+}
+
/* Is the expert in this layer's RAM cache now (the tier's balance asks)? */
static int vk_in_ram(void *ctx, int li, int e) {
const Model *m = ctx;
@@ -1333,9 +1465,11 @@ static void vk_tier_start(Model *m) {
.gate_up = f, .down = f, .act = VKT_ACT_SWIGLU,
.max_rows = MIMO_VK_ROWS * c->topk,
.ram_reserve = (size_t)(m->e_bytes + 8192) * (size_t)cap * (size_t)nmoe,
- .dense_bytes = (coli_vk_dense() ? vk_dense_bytes(m, 1) : g_vk_chain ? vk_dense_bytes(m, 0) : 0) +
+ .dense_bytes = (coli_vk_dense_device_only() ? 0 : /* placed already (mimo_dho_start) */
+ coli_vk_dense() ? vk_dense_bytes(m, 1) : g_vk_chain ? vk_dense_bytes(m, 0) : 0) +
(g_vk_chain ? mc_kv_bytes(m) : 0), /* the chain's KV caches */
- .in_ram = vk_in_ram, .ram_ctx = m};
+ .in_ram = vk_in_ram, .ram_ctx = m,
+ .load = mimo_vk_load, .release = mimo_vk_release, .load_ctx = m, .load_batch = mimo_vk_load_batch};
atexit(coli_vk_shutdown); /* before vkt_init, which makes the expert batch's pipelines and can still refuse (no room): the device goes at exit either way, after the tier's teardown */
if (vkt_init(&vc, NULL)) atexit(vkt_shutdown);
}
@@ -1852,6 +1986,7 @@ int main(int argc, char **argv) {
g_vk_chain = coli_vk_chain_decide("mimo", tier, COLI_VK_CHAIN_UNMEASURED);
if (g_vk_chain && !vkc_init()) g_vk_chain = 0;
}
+ mimo_dho_start(m); /* COLI_VK_DENSE_HOST: the dense matrices on the device only, before the tier sizes its budget */
if (g_vk_ready && tier) vk_tier_start(m);
if (g_vk_ready && !vkt_ready() && !coli_vk_dense()) coli_vk_dense_decide("mimo", 0, 1); /* no tier after all */
if (g_vk_chain) { /* the chain's teardown before the device's (the tier registered the device's) */
diff --git a/c/mimo_chain.h b/c/mimo_chain.h
index 801bf034..7c1ef687 100644
--- a/c/mimo_chain.h
+++ b/c/mimo_chain.h
@@ -45,10 +45,18 @@
* the position the host's caches end at (no rebuild, a turn with a picture included).
* COLI_VK_CHAIN_FAULT=n (vk_chain.c) fakes the loss at the n-th frame, for tests.
*
+ * Past the device's budget (vk_kvsplit.h) each full-attention layer keeps a window of
+ * the newest blocks on the device (the windowed layers keep their rings, which are
+ * small): the step's attention runs there, sink included, and on the CPU over the older
+ * positions of the host's cache at once, merged through the softmax statistics. The
+ * host's caches hold every position before the step (each chunk commits its rows), so
+ * the host's part always reads them there.
+ *
* The chain declines (the per-matrix path or the CPU runs) when a dense matrix did not
* reach the device, for geometry outside chain_attn (a head dim above 256) and under
* MIMO_TRACE (the CPU's per-layer dump). */
#include "vk_chain.h"
+#include "vk_kvsplit.h"
typedef struct {
int ok, failed;
@@ -58,6 +66,8 @@ typedef struct {
size_t *o_ln1, *o_ln2, *o_sink, o_final;
VkcBuf **kc, **vc; /* the device's caches, per layer */
int *kv_valid;
+ int *kvi; /* a full layer's index in the split (ks), -1 windowed */
+ VkcKvSplit ks; /* the full layers' split past the device's budget (ks.on) */
size_t *o_kvd; /* a layer's new K rows in kvd (V after cap rows) */
VkcBuf *x, *nrm, *tmp, *qkv, *kn, *vn, *wk, *wv, *ctx, *g, *u, *fin;
VkcBuf *h2d, *kvd, *outd, *routed, *cs;
@@ -68,11 +78,6 @@ typedef struct {
static int g_vk_chain = 0; /* COLI_VK_CHAIN decided on, and the chain's pipelines are up */
-static int mc_chunk_rows(void) {
- const char *e = getenv("COLI_VK_CHAIN_ROWS");
- int v = e && *e ? atoi(e) : 512;
- return v < 1 ? 1 : v > 65535 ? 65535 : v;
-}
/* the geometry of layer li's attention */
typedef struct { int k, nh, kvh, hd, vd, qd, kd, vdd, rw, half, rows, swa; } McGeo;
@@ -89,10 +94,16 @@ static McGeo mc_geo(const Model *m, int li) {
static int mc_ctensor(DW *d) { return dw_upload(d) && d->vk; }
-/* the device's KV caches, in bytes (the tier's budget leaves them room) */
+/* the device's KV caches, in bytes (the tier's budget leaves them room); a full layer
+ * holds COLI_VK_KV_DEVICE_ROWS rows when that asks for fewer than the context */
static size_t mc_kv_bytes(const Model *m) {
size_t b = 0;
- for (int i = 0; i < m->c.n_layers; i++) { McGeo g = mc_geo(m, i); b += (size_t)g.rows * (g.kd + g.vdd) * sizeof(float); }
+ long forced = vkc_kv_env("COLI_VK_KV_DEVICE_ROWS", 0);
+ for (int i = 0; i < m->c.n_layers; i++) {
+ McGeo g = mc_geo(m, i);
+ size_t rows = !g.swa && forced > 0 && forced < g.rows ? (size_t)forced : (size_t)g.rows;
+ b += rows * (g.kd + g.vdd) * sizeof(float);
+ }
return b;
}
@@ -112,9 +123,9 @@ static MimoChain *mc_setup(Model *m) {
return NULL;
}
ch->o_ln1 = calloc(L, sizeof(size_t)); ch->o_ln2 = calloc(L, sizeof(size_t)); ch->o_sink = calloc(L, sizeof(size_t));
- ch->o_kvd = calloc(L, sizeof(size_t)); ch->kv_valid = calloc(L, sizeof(int));
+ ch->o_kvd = calloc(L, sizeof(size_t)); ch->kv_valid = calloc(L, sizeof(int)); ch->kvi = calloc(L, sizeof(int));
ch->kc = calloc(L, sizeof(void *)); ch->vc = calloc(L, sizeof(void *));
- if (!ch->o_ln1 || !ch->o_ln2 || !ch->o_sink || !ch->o_kvd || !ch->kv_valid || !ch->kc || !ch->vc) return NULL;
+ if (!ch->o_ln1 || !ch->o_ln2 || !ch->o_sink || !ch->o_kvd || !ch->kv_valid || !ch->kvi || !ch->kc || !ch->vc) return NULL;
size_t n = 0;
for (int i = 0; i < L; i++) {
ch->o_ln1[i] = n; n += H; ch->o_ln2[i] = n; n += H;
@@ -142,10 +153,25 @@ static MimoChain *mc_setup(Model *m) {
}
ok = ok && mc_ctensor(&m->head);
if (!ok) { fprintf(stderr, "[VK] mimo chain: a dense matrix did not reach the device; per-matrix path\n"); return NULL; }
- /* the caches, in the host's layout and at its size */
+ /* the caches, in the host's layout and at its size; the full layers' split past the
+ * device's budget (a window of blocks a layer, the rest in the host's cache) */
+ int nfull = 0, frows = 0;
+ size_t frow = 0;
+ for (int i = 0; i < L; i++) {
+ McGeo g = mc_geo(m, i);
+ ch->kvi[i] = g.swa ? -1 : nfull++;
+ if (!g.swa) { frows = g.rows; frow = (size_t)(g.kd + g.vdd) * sizeof(float); }
+ }
+ /* KV placement is row-anchored; its legacy chunk argument is not used. The
+ * actual prompt chunk is sized after setup and clamped to ks.chunk. */
+ if (nfull && !vkc_kv_plan(&ch->ks, "mimo", nfull, frow, frows, 1, 0, 0)) {
+ fprintf(stderr, "[VK] mimo chain: the KV split's tables refused; per-matrix path\n");
+ return NULL;
+ }
size_t kv = 0;
for (int i = 0; i < L; i++) {
McGeo g = mc_geo(m, i);
+ if (!g.swa && ch->ks.on) g.rows = ch->ks.rows;
ch->kc[i] = vkc_buf((size_t)g.rows * g.kd * sizeof(float), VKC_DEV);
ch->vc[i] = vkc_buf((size_t)g.rows * g.vdd * sizeof(float), VKC_DEV);
if (!ch->kc[i] || !ch->vc[i]) {
@@ -162,7 +188,25 @@ static MimoChain *mc_setup(Model *m) {
return ch;
}
-static int mc_res(VkcBuf **b, size_t floats, int kind) { return vkc_reserve(b, (floats ? floats : 1) * sizeof(float), kind); }
+/* mc_res counts instead of reserving while g_mc_count >= 0 (the chunk's sizing) */
+static long long g_mc_count = -1;
+static int mc_res(VkcBuf **b, size_t floats, int kind) {
+ if (g_mc_count >= 0) { g_mc_count += (long long)(floats ? floats : 1) * (long long)sizeof(float); return 1; }
+ return vkc_reserve(b, (floats ? floats : 1) * sizeof(float), kind);
+}
+static int mc_scratch(MimoChain *ch, Model *m, int rows, int out);
+/* Prompt rows per chunk (vkc_chunk_rows): the chain's scratch a row, counted from the
+ * reservations (mc_scratch stops at its first host allocation in this mode), and the
+ * routed experts' outputs (the tier's rows, the CPU's, the host's sum) for it. */
+static int mc_chunk_rows(MimoChain *ch, Model *m) {
+ g_mc_count = 0; mc_scratch(ch, m, 1, 1); long long b1 = g_mc_count;
+ g_mc_count = 0; mc_scratch(ch, m, 2, 1); long long b2 = g_mc_count;
+ g_mc_count = -1;
+ size_t row = (size_t)(b2 - b1) + (size_t)(2 * m->c.topk + 1) * m->c.hidden * sizeof(float);
+ /* moe_cpu retains a whole chunk's gathered inputs and gate/up outputs. */
+ row += (size_t)m->c.topk * ((size_t)m->c.hidden + 2 * (size_t)m->c.moe_inter) * sizeof(float);
+ return vkc_chunk_rows("mimo", row);
+}
/* scratch for `rows` rows, `out` rows of logits */
static int mc_scratch(MimoChain *ch, Model *m, int rows, int out) {
@@ -185,6 +229,12 @@ static int mc_scratch(MimoChain *ch, Model *m, int rows, int out) {
dense |= !c->moe[i];
}
ch->rd_off[0] = 0; ch->rd_off[1] = rows * 2 * (c->rope_dim[0] / 2);
+ if (ch->ks.on) { /* the split's partial results: a full layer's heads */
+ size_t part = 0;
+ for (int i = 0; i < L; i++) { McGeo g = mc_geo(m, i); if (!g.swa && (size_t)g.nh * (g.vd + 2) > part) part = (size_t)g.nh * (g.vd + 2); }
+ if (g_mc_count >= 0) g_mc_count += (long long)r * (3 * part + rw) * sizeof(float);
+ else if (!vkc_kv_parts(&ch->ks, r * part)) return 0;
+ }
size_t cs = r * 2 * (size_t)(c->rope_dim[0] / 2 + c->rope_dim[1] / 2);
int ok = mc_res(&ch->x, r * H, VKC_DEV) && mc_res(&ch->nrm, r * H, VKC_DEV) && mc_res(&ch->tmp, r * H, VKC_DEV) &&
mc_res(&ch->qkv, r * rw, VKC_DEV) && mc_res(&ch->kn, r * kd, VKC_DEV) && mc_res(&ch->vn, r * vdd, VKC_DEV) &&
@@ -194,7 +244,7 @@ static int mc_scratch(MimoChain *ch, Model *m, int rows, int out) {
mc_res(&ch->cs, cs, VKC_UP);
if (ok && wkd) ok = mc_res(&ch->wk, ((size_t)win + r) * wkd, VKC_DEV) && mc_res(&ch->wv, ((size_t)win + r) * wvd, VKC_DEV);
if (ok && dense) ok = mc_res(&ch->g, r * c->dense_inter, VKC_DEV) && mc_res(&ch->u, r * c->dense_inter, VKC_DEV);
- if (!ok) return 0;
+ if (!ok || g_mc_count >= 0) return ok; /* counting: the buffers only */
if (ch->rows < rows) {
float *ho = realloc(ch->host_out, r * H * sizeof(float));
if (!ho) return 0;
@@ -209,7 +259,10 @@ static int mc_scratch(MimoChain *ch, Model *m, int rows, int out) {
static void mc_cpu_step(Model *m, int pos_base) {
MimoChain *ch = (MimoChain *)m->vkchain;
if (!ch || !ch->ok) return;
- for (int i = 0; i < m->c.n_layers; i++) if (ch->kv_valid[i] > pos_base) ch->kv_valid[i] = pos_base;
+ for (int i = 0; i < m->c.n_layers; i++) {
+ if (ch->kv_valid[i] > pos_base) ch->kv_valid[i] = pos_base;
+ if (ch->kvi[i] >= 0) vkc_kv_lower(&ch->ks, ch->kvi[i], pos_base);
+ }
}
/* The host's rings were replaced (a photo restored): the device's no longer match. */
static void mc_rings_rewritten(Model *m) {
@@ -239,15 +292,20 @@ static int mc_ring_copy(VkcBuf *ring, VkcBuf *lin, int R, int base, int t0, int
}
return ok;
}
-/* Record the uploads that make the device's caches the host's, as the step at pos_base
- * reads them. */
-static int mc_push_state(MimoChain *ch, Model *m, int pos_base) {
+/* Record the uploads that make the device's caches the host's, as the step of n rows at
+ * pos_base reads them. */
+static int mc_push_state(MimoChain *ch, Model *m, int pos_base, int n_rows) {
Cfg *c = &m->c; int ok = 1;
for (int i = 0; i < c->n_layers && ok; i++) {
McGeo g = mc_geo(m, i);
Layer *l = &m->L[i];
if (ch->kv_valid[i] > pos_base) ch->kv_valid[i] = pos_base;
- if (g.swa) {
+ if (!g.swa && ch->ks.on) { /* the split: its window placed, the rows below pos_base uploaded */
+ VkcKvPart pt[2] = {{1, g.kd, l->K, 0, ch->kc[i], 0}, {1, g.vdd, l->V, 0, ch->vc[i], 0}};
+ vkc_kv_lower(&ch->ks, ch->kvi[i], pos_base);
+ vkc_kv_place(&ch->ks, ch->kvi[i], pos_base, n_rows);
+ ok = vkc_kv_push(&ch->ks, ch->kvi[i], pt, 2, pos_base);
+ } else if (g.swa) {
/* the positions the window still sees must be in the host's ring, as the
* CPU's attention() demands */
int lo = pos_base - (c->window - 1) > 0 ? pos_base - (c->window - 1) : 0;
@@ -305,6 +363,18 @@ static int mc_attention(MimoChain *ch, Model *m, int i, int n, int pb) {
a.vd = g.vd; a.kv_pm = 1;
a.sink = l->sink != NULL; a.sink_off = (int)ch->o_sink[i];
VkcBuf *kc = ch->kc[i], *vc = ch->vc[i];
+ if (!g.swa && ch->ks.on) { /* the split: the rows into their window slots, the attention in two parts */
+ VkcKvPart pk = {1, g.kd, l->K, 0, kc, 0}, pv = {1, g.vdd, l->V, 0, vc, 0};
+ VkcKvGqa ga = {&ch->ks, ch->kvi[i], ch->qkv, kc, vc, NULL, NULL, l->sink ? ch->prm : NULL, ch->ctx,
+ n, g.nh, g.kvh, g.hd, g.vd, pb, 0, 1, l->sink != NULL, (int)ch->o_sink[i],
+ g.rw, g.hd, 0, 0, 0, 0, g.nh * g.vd, 0, 0, 1.0f / sqrtf((float)g.hd),
+ l->K, l->V, (size_t)g.hd, (size_t)g.kd, (size_t)g.vd, (size_t)g.vdd};
+ ok = vkc_kv_store(&ch->ks, ch->kvi[i], &pk, ch->kn, 0, (size_t)g.kd, 0, pb, n) &&
+ vkc_kv_store(&ch->ks, ch->kvi[i], &pv, ch->vn, 0, (size_t)g.vdd, 0, pb, n) &&
+ vkc_copy(ch->kvd, ch->o_kvd[i], ch->kn, 0, (size_t)n * g.kd) &&
+ vkc_copy(ch->kvd, ch->o_kvd[i] + (size_t)ch->rows * g.kd, ch->vn, 0, (size_t)n * g.vdd);
+ return ok && vkc_kv_gqa(&ga) && vkc_matmul((ColiVkTensor *)l->o.vk, ch->ctx, 0, ch->tmp, 0, n);
+ }
if (!g.swa) { /* full attention: the rows in at pb, then every position */
ok = vkc_copy(kc, (size_t)pb * g.kd, ch->kn, 0, (size_t)n * g.kd) &&
vkc_copy(vc, (size_t)pb * g.vdd, ch->vn, 0, (size_t)n * g.vdd);
@@ -358,6 +428,7 @@ static void mc_commit(MimoChain *ch, Model *m, int n, int pb) {
if (g.swa) l->ring_pos[slot] = p;
}
ch->kv_valid[i] = pb + n;
+ if (ch->kvi[i] >= 0) vkc_kv_done(&ch->ks, ch->kvi[i], pb + n);
}
}
@@ -372,7 +443,8 @@ static int mc_forward(Model *m, const float *h, int S, int pos_base, float *logi
MimoChain *ch = mc_setup(m);
if (!ch || ch->failed) return 0;
Cfg *c = &m->c; int H = c->hidden, L = c->n_layers, V = c->vocab;
- int CH = mc_chunk_rows(), rows = S < CH ? S : CH;
+ int CH = mc_chunk_rows(ch, m), rows = S < CH ? S : CH;
+ if (ch->ks.on && rows > ch->ks.chunk) rows = ch->ks.chunk; /* a step's rows fit the split's window */
int out = logits && all_rows ? rows : 1;
if (!mc_scratch(ch, m, rows, out)) {
fprintf(stderr, "[VK] mimo chain: device memory for %d rows refused; per-matrix path\n", rows);
@@ -384,7 +456,7 @@ static int mc_forward(Model *m, const float *h, int S, int pos_base, float *logi
for (int c0 = 0; c0 < S; c0 += rows) {
int n = S - c0 < rows ? S - c0 : rows, pb = pos_base + c0, last = c0 + n == S;
if (!vkc_begin() || !vkc_write(ch->x, 0, h + (size_t)c0 * H, (size_t)n * H * sizeof(float)) ||
- !mc_push_state(ch, m, pb)) goto lost;
+ !mc_push_state(ch, m, pb, n)) goto lost;
float *cs = (float *)vkc_ptr(ch->cs); /* the CPU's angles, cosines and sines (rope()) */
for (int k = 0; k < 2; k++) {
int rd = c->rope_dim[k], half = rd / 2;
@@ -406,7 +478,11 @@ static int mc_forward(Model *m, const float *h, int S, int pos_base, float *logi
else if (ok) {
ok = vkc_copy(ch->h2d, 0, ch->nrm, 0, (size_t)n * H);
double t0 = now_s();
- ok = ok && vkc_submit(1);
+ /* while the frame runs, the tier loads the experts this layer will likely
+ * stream (a big prompt chunk only) */
+ ok = ok && vkc_submit(0);
+ if (ok) vkt_stream_prefetch(i, n);
+ ok = ok && vkc_finish();
ch->frames++; ch->wait_ms += (now_s() - t0) * 1e3;
if (!ok) break;
double t1 = now_s();
@@ -448,5 +524,6 @@ static void mc_report(Model *m, const char *what) {
"%.1f ms waiting for the device, %.1f ms of routed experts on the host, %.1f MiB on the device (%s)\n",
ch->forwards, st.frames, st.ops, st.matmuls, st.gemms, ch->wait_ms, ch->host_ms,
st.dev_bytes / 1048576.0, what);
+ vkc_kv_report(&ch->ks);
vkc_prof_print();
}
diff --git a/c/native_quant.h b/c/native_quant.h
index 7cda2cd0..61a815d0 100644
--- a/c/native_quant.h
+++ b/c/native_quant.h
@@ -97,6 +97,16 @@ typedef int (*ColiV4VkMatmul)(int fmt, const void *data, const float *scales,
const float *input, int batch);
extern ColiV4VkMatmul coli_v4_vk_matmul;
+/* With the dense weights on the device only (COLI_VK_DENSE_HOST), a matrix may have no
+ * host copy: the CPU paths call this before they read one, and it is read back from disk
+ * at its own address (deepseek_v4_internal.h, coli_v4_layer_host_restore). NULL when no
+ * device holds a matrix alone. */
+typedef int (*ColiV4VkHost)(const void *data);
+extern ColiV4VkHost coli_v4_vk_host;
+static inline void coli_v4_vk_host_ensure(const void *data) {
+ if (coli_v4_vk_host && data) coli_v4_vk_host(data);
+}
+
/* An fp8 view on the device, fed the activation the CPU kernel would read
* (already rounded to E4M3 per 128 by coli_fp8_activation_qdq_ref). */
static inline int coli_v4_vk_fp8(float *output, const ColiTensorView *weight,
diff --git a/c/olmoe.c b/c/olmoe.c
index ebd5a6f8..95374050 100644
--- a/c/olmoe.c
+++ b/c/olmoe.c
@@ -324,15 +324,18 @@ static float g_nuc = 0.95f; /* NUCLEUS env overrides */
* COLI_VULKAN it is the plain matmul call it replaces. */
#ifdef COLI_VULKAN
static char g_vk_refused; /* *vk == &g_vk_refused: stays on the CPU */
+static float *olm_dho_reload(void **vk); /* below: a dropped host copy read back from disk */
static void matmul_res(float *y, const float *x, const float *W, void **vk, int S, int I, int O) {
int serial = 1;
#ifdef _OPENMP
serial = !omp_in_parallel();
#endif
- if (g_vk_ready && serial && *vk != (void *)&g_vk_refused && coli_vk_dense()) {
+ /* with the dense weights on the device only the device is the matrix's one home */
+ if (g_vk_ready && serial && *vk != (void *)&g_vk_refused && (coli_vk_dense() || coli_vk_dense_device_only())) {
if (coli_vk_matmul((ColiVkTensor **)vk, y, x, W, NULL, 10, S, I, O, 0)) return;
if (!*vk) *vk = &g_vk_refused;
}
+ if (!W) W = olm_dho_reload(vk); /* the CPU needs a matrix the device holds alone (a lost device) */
matmul(y, x, W, S, I, O);
}
static void vk_res_free(void **vk) {
@@ -535,6 +538,11 @@ static void load_cfg(Cfg *c, const char *snap) {
free(buf); free(arena);
}
+/* The parameters of an automatic cache size (cap <= 0), for olm_dho_grow_cap: with the
+ * dense weights on the device only, the RAM they held goes to the experts. */
+typedef struct { int on; double ram_arg, resident, kv_gb, slot_gb; int layers; } OlmAutoCap;
+static OlmAutoCap g_olm_auto_cap;
+
static float *load_t(Model *m, const char *name) {
int64_t n = st_numel(&m->S, name);
if (n < 0) { fprintf(stderr, "missing %s\n", name); exit(1); }
@@ -624,6 +632,9 @@ static void model_init_range(Model *m, const char *snap, int cap, int bits,
derived = 1;
}
if (derived > c->n_experts) derived = c->n_experts;
+ if (load_boundaries) { /* the standalone engine: kept for olm_dho_grow_cap */
+ g_olm_auto_cap = (OlmAutoCap){1, ram_arg, resident, kv_gb, slot_gb, layers};
+ }
fprintf(stderr, "[cache] %d slots/layer of %d experts: %.1f GB budget "
"(%s), %.1f GB dense resident, %.1f GB projected KV, "
"%.0f MB per expert\n",
@@ -722,6 +733,7 @@ static void model_init_range(Model *m, const char *snap, int cap, int bits,
#ifdef COLI_VULKAN
static void olmoe_vk_tier_start(Model *m);
static void olc_start(Model *m);
+static void olm_dho_start(Model *m);
#endif
static void model_init(Model *m, const char *snap, int cap, int bits) {
model_init_range(m, snap, cap, bits, 0, 0, 1, 1);
@@ -732,6 +744,7 @@ static void model_init(Model *m, const char *snap, int cap, int bits) {
* the expert tier will be tried (vk_tier.h step 0): on a device sharing the
* CPU's RAM the dense matrices then stay on the CPU unless COLI_VK_DENSE=1. */
if (!g_vk_ready) g_vk_ready = coli_vk_init_env_tier("olmoe", vkt_wanted() && m->c.n_experts > 0);
+ if (g_vk_ready) olm_dho_start(m); /* COLI_VK_DENSE_HOST: the dense matrices on the device only, before the tier sizes its budget */
if (g_vk_ready) olmoe_vk_tier_start(m); /* after the history: COLI_USAGE, read above */
if (g_vk_ready && !vkt_ready() && !coli_vk_dense()) coli_vk_dense_decide("olmoe", 0, 1); /* no tier after all */
if (g_vk_ready && coli_vk_dense())
@@ -818,19 +831,32 @@ static int olmoe_vk_in_ram(void *ctx, int layer, int e) {
* row, gate, up and down as the merged container holds them: fmt 1), after the
* weights and the history (COLI_USAGE, read by model_init_range), and warm it from
* that history: the hottest experts first, read from disk in parallel. */
+/* The tier's streaming (a big prompt chunk's cold experts on the device): an expert's
+ * bytes through the layer cache as the CPU path gets them, held until copied. */
+static void expert_get(Model *m, int layer, int eid, Slot **out);
+static void expert_put(Slot *s);
+static int olmoe_vk_load(void *ctx, int layer, int e, VktExpertSrc *src, void **h) {
+ Slot *s = NULL; expert_get((Model *)ctx, layer, e, &s);
+ if (!s) return 0;
+ *src = olmoe_vk_src(s); *h = s;
+ return 1;
+}
+static void olmoe_vk_release(void *ctx, void *h) { (void)ctx; expert_put((Slot *)h); }
static void olmoe_vk_tier_start(Model *m) {
Cfg *c = &m->c;
if (!vkt_wanted() || c->n_experts < 1) return;
int64_t D = c->hidden, I = c->inter;
size_t slotb = (size_t)(3 * I * D + (2 * I + D) * 4);
- size_t dense = coli_vk_dense() ? (size_t)c->n_layers * (size_t)(4 * D * D + (int64_t)c->n_experts * D) * 4 +
- (size_t)c->vocab * (size_t)D * 4 : 0;
+ size_t dense = coli_vk_dense() && !coli_vk_dense_device_only() /* device only: placed already */
+ ? (size_t)c->n_layers * (size_t)(4 * D * D + (int64_t)c->n_experts * D) * 4 +
+ (size_t)c->vocab * (size_t)D * 4 : 0;
VktConfig vc = {.engine = "olmoe", .layers = c->n_layers, .experts = c->n_experts,
.hidden = c->hidden, .inter = c->inter, .topk = c->topk,
.gate_up = {VKT_SRC_I8_ROW, 0}, .down = {VKT_SRC_I8_ROW, 0},
.act = VKT_ACT_SWIGLU, .max_rows = OLMOE_VK_ROWS * c->topk,
.ram_reserve = slotb * (size_t)m->cache[0].cap * (size_t)c->n_layers,
- .dense_bytes = dense, .in_ram = olmoe_vk_in_ram, .ram_ctx = m};
+ .dense_bytes = dense, .in_ram = olmoe_vk_in_ram, .ram_ctx = m,
+ .load = olmoe_vk_load, .release = olmoe_vk_release, .load_ctx = m};
atexit(coli_vk_shutdown); /* before vkt_init, which makes the expert batch's pipelines and can still refuse (no room): the device goes at exit either way, after the tier's teardown */
if (!vkt_init(&vc, m->freq)) return;
atexit(vkt_shutdown);
@@ -1195,7 +1221,7 @@ static void moe_vk_run(Model *m, int layer, const float *x, int S, float *logits
Cfg *c = &m->c; int D = c->hidden, E = c->n_experts, K = c->topk, I = c->inter;
int *idx = malloc((size_t)S * K * sizeof(int));
float *val = malloc((size_t)S * K * sizeof(float));
- int B = S < OLMOE_VK_ROWS ? S : OLMOE_VK_ROWS;
+ int B = vkt_step_rows(S, OLMOE_VK_ROWS); /* a whole prompt chunk when the tier streams */
float *ctb = falloc((int64_t)B * K * D), *g = falloc(I), *u = falloc(I);
uint8_t *taken = malloc((size_t)B * K), *want = malloc((size_t)B * K);
const float **dev = malloc((size_t)B * K * sizeof(*dev));
@@ -1339,6 +1365,102 @@ static void olmoe_echo(const char *id, int pos, int token, const float *lo, int
#ifdef COLI_VULKAN
#include "olmoe_chain.h" /* COLI_VK_CHAIN: every layer's dense chain on the device */
+
+/* ---- dense weights on the device only (COLI_VK_DENSE_HOST) ------------------------
+ * With the dense part on the device (the chain, or COLI_VK_DENSE), the attention's
+ * q/k/v/o, the router and lm_head go up at start and their f32 host copies are given
+ * back; matmul_res then runs them on the device whatever COLI_VK_DENSE says. The CPU
+ * multiplies by one only after a lost device: matmul_res first reads it back here
+ * (load_t: the same f32 bytes) and keeps it from there on. What keeps its host copy:
+ * the embedding (its rows are gathered on the CPU) and the norms. The chain is decided
+ * after the tier (olc_start); its decision is taken here silently first, from the same
+ * inputs, so the matrices are placed before the tier sizes its budget. */
+typedef struct { void **vk; float **field; int64_t n; char name[96]; } OlmDho;
+static OlmDho *g_olm_dho; static int g_olm_dho_n, g_olm_dho_cap;
+static Model *g_olm_dho_model;
+static pthread_mutex_t g_olm_dho_mx = PTHREAD_MUTEX_INITIALIZER;
+static float *olm_dho_reload(void **vk) {
+ pthread_mutex_lock(&g_olm_dho_mx);
+ OlmDho *e = NULL;
+ for (int i = 0; i < g_olm_dho_n && !e; i++) if (g_olm_dho[i].vk == vk) e = &g_olm_dho[i];
+ if (!e || !g_olm_dho_model) { fprintf(stderr, "[VK] olmoe: a dense matrix the device held alone cannot be read back\n"); exit(1); }
+ if (!*e->field) {
+ *e->field = load_t(g_olm_dho_model, e->name);
+ coli_vk_dense_host_reloaded((size_t)e->n * sizeof(float));
+ }
+ float *w = *e->field;
+ pthread_mutex_unlock(&g_olm_dho_mx);
+ return w;
+}
+static void olm_dho_drop(Model *m, float **field, void **vk, const char *name, int I, int O, size_t *bytes) {
+ if (!*field || *vk == (void *)&g_vk_refused) return;
+ if (!*vk && !coli_vk_tensor_ensure((ColiVkTensor **)vk, *field, NULL, 10, I, O, 0)) { *vk = &g_vk_refused; return; }
+ if (g_olm_dho_n == g_olm_dho_cap) {
+ g_olm_dho_cap = g_olm_dho_cap ? 2 * g_olm_dho_cap : 64;
+ g_olm_dho = realloc(g_olm_dho, (size_t)g_olm_dho_cap * sizeof *g_olm_dho);
+ if (!g_olm_dho) { fprintf(stderr, "OOM dense matrix table\n"); exit(1); }
+ }
+ OlmDho *e = &g_olm_dho[g_olm_dho_n++];
+ e->vk = vk; e->field = field; e->n = (int64_t)I * O;
+ snprintf(e->name, sizeof e->name, "%s", name);
+ free(*field); *field = NULL;
+ size_t b = (size_t)I * O * sizeof(float);
+ coli_vk_dense_host_dropped(b);
+ *bytes += b;
+ (void)m;
+}
+/* An automatic cache (cap <= 0) sized with the dense weights in RAM: give the experts
+ * what they held. With RAM_GB the process's own resident set shrinks by what was
+ * dropped; without it, what the OS offers now is read again (on a discrete GPU it grew
+ * by that much; on a device sharing the RAM the device copy took it back). Slots fill
+ * lazily, so a layer's cache grows by zeroed slots. */
+static void olm_dho_grow_cap(Model *m, size_t dropped) {
+ OlmAutoCap *a = &g_olm_auto_cap;
+ Cfg *c = &m->c;
+ if (!a->on || !m->cache || a->slot_gb <= 0.0) return;
+ double resident = a->resident - dropped / 1e9;
+ double budget = a->ram_arg > 0.0 ? a->ram_arg : resident + mem_available_gb() * 0.88;
+ double room = budget - resident - a->kv_gb - 0.5;
+ int derived = room > 0.0 ? (int)(room / a->slot_gb / (double)a->layers) : 1;
+ if (derived < 1) derived = 1;
+ if (derived > c->n_experts) derived = c->n_experts;
+ int was = m->cache[0].cap;
+ if (derived <= was) return;
+ pthread_mutex_lock(&g_pilot_mx);
+ for (int i = 0; i < c->n_layers; i++) {
+ LCache *lc = &m->cache[i];
+ if (!lc->slots) continue;
+ Slot *s = realloc(lc->slots, (size_t)derived * sizeof(Slot));
+ if (!s) { fprintf(stderr, "OOM growing the expert cache\n"); exit(1); }
+ memset(s + lc->cap, 0, (size_t)(derived - lc->cap) * sizeof(Slot));
+ lc->slots = s; lc->cap = derived;
+ }
+ pthread_mutex_unlock(&g_pilot_mx);
+ fprintf(stderr, "[cache] %d slots/layer of %d experts (was %d): the dense weights on the device only gave "
+ "%.2f GB back, %.2f GB dense resident now\n", derived, c->n_experts, was, dropped / 1e9, resident);
+}
+static void olm_dho_start(Model *m) {
+ Cfg *c = &m->c;
+ int D = c->hidden, E = c->n_experts;
+ size_t bytes = (size_t)c->n_layers * (size_t)(4 * (int64_t)D * D + (int64_t)E * D) * sizeof(float) +
+ (size_t)c->vocab * (size_t)D * sizeof(float);
+ int chain = coli_vk_chain_decide(NULL, vkt_wanted() && c->n_experts > 0, OLMOE_CHAIN_IGPU);
+ if (!coli_vk_dense_host_decide("olmoe", (chain || coli_vk_dense()) && m->lm_head, bytes)) return;
+ g_olm_dho_model = m;
+ size_t dropped = 0;
+ char nm[96];
+ for (int i = 0; i < c->n_layers; i++) {
+ Layer *l = &m->L[i];
+ snprintf(nm, sizeof nm, "model.layers.%d.self_attn.q_proj.weight", i); olm_dho_drop(m, &l->q, &l->vk_q, nm, D, D, &dropped);
+ snprintf(nm, sizeof nm, "model.layers.%d.self_attn.k_proj.weight", i); olm_dho_drop(m, &l->k, &l->vk_k, nm, D, D, &dropped);
+ snprintf(nm, sizeof nm, "model.layers.%d.self_attn.v_proj.weight", i); olm_dho_drop(m, &l->v, &l->vk_v, nm, D, D, &dropped);
+ snprintf(nm, sizeof nm, "model.layers.%d.self_attn.o_proj.weight", i); olm_dho_drop(m, &l->o, &l->vk_o, nm, D, D, &dropped);
+ snprintf(nm, sizeof nm, "model.layers.%d.mlp.gate.weight", i); olm_dho_drop(m, &l->gate, &l->vk_gate, nm, D, E, &dropped);
+ }
+ olm_dho_drop(m, &m->lm_head, &m->vk_lm_head, "lm_head.weight", D, c->vocab, &dropped);
+ coli_vk_dense_host_placed("olmoe", "the embedding (its rows are gathered on the CPU), norms");
+ olm_dho_grow_cap(m, dropped);
+}
#endif
static float *step(Model *m, const int *ids, int S, int pos_base) {
diff --git a/c/olmoe_chain.h b/c/olmoe_chain.h
index f39b9914..b5d26fb4 100644
--- a/c/olmoe_chain.h
+++ b/c/olmoe_chain.h
@@ -28,6 +28,9 @@
* [kv_valid, pos_base) first; a CPU step lowers the watermark to its pos_base; a host
* cache allocated anew (generate, a larger max_t) is mirrored again. Prompt-cache and
* prefix reuse need nothing more: a reused prefix is rows below the watermark.
+ * Past the device's budget (vk_kvsplit.h) each layer keeps a window of the newest
+ * blocks on the device; the attention runs there and on the CPU over the older
+ * positions (the host's cache) at once, merged through the softmax statistics.
*
* The chain declines (the per-matrix path runs, the watermark lowered) for a geometry
* outside chain_attn.comp (head dim above 256 or odd). A device lost mid-step costs
@@ -35,6 +38,7 @@
* step from the embedding rows (which the chain never overwrites) and runs from there.
* COLI_VK_CHAIN_FAULT=n (vk_chain.c) fakes the loss at the n-th frame, for tests. */
#include "vk_chain.h"
+#include "vk_kvsplit.h"
/* COLI_VK_CHAIN unset on an integrated GPU with the expert tier (coli_vk_chain_decide):
* on. Measured on a Radeon 780M with OLMoE-1B-7B (docs/vulkan.md, "OLMoE and Inkling"):
@@ -47,7 +51,9 @@
typedef struct {
int ok, failed;
int rows; /* scratch capacity in rows */
- int cap; /* device KV rows (the host's max_t) */
+ int cap; /* the host's max_t the mirrors follow */
+ int dev_rows; /* device KV rows a layer: cap, or the split's */
+ VkcKvSplit ks; /* the split past the device's budget (ks.on) */
float **hostK; /* the host cache the mirror copies */
VkcBuf *prm; /* every norm weight */
size_t *o_in, *o_post, *o_qn, *o_kn, o_final;
@@ -60,17 +66,13 @@ typedef struct {
double host_ms;
} OlmChain;
-static int olc_chunk_rows(void) {
- const char *e = getenv("COLI_VK_CHAIN_ROWS");
- int v = e && *e ? atoi(e) : 512;
- return v < 1 ? 1 : v > 65535 ? 65535 : v;
-}
/* A resident f32 matrix [O x I] on the device: the copy the per-matrix path uses
* (matmul_res), uploaded here if it has not been yet. NULL: refused (it stays so). */
static ColiVkTensor *olc_tensor(void **vk, const float *w, int I, int O) {
- if (*vk == (void *)&g_vk_refused || !w) return NULL;
- if (*vk) return (ColiVkTensor *)*vk;
+ if (*vk == (void *)&g_vk_refused) return NULL;
+ if (*vk) return (ColiVkTensor *)*vk; /* with the dense weights on the device only, w is NULL */
+ if (!w) return NULL;
if (!coli_vk_tensor_ensure((ColiVkTensor **)vk, w, NULL, 10, I, O, 0)) { *vk = &g_vk_refused; return NULL; }
return (ColiVkTensor *)*vk;
}
@@ -128,31 +130,48 @@ static OlmChain *olc_setup(Model *m) {
return ch;
}
-static int olc_res(VkcBuf **b, size_t floats, int kind) { return vkc_reserve(b, (floats ? floats : 1) * sizeof(float), kind); }
-
-/* scratch for `rows` rows, and the KV mirrors at the host's capacity */
-static int olc_scratch(OlmChain *ch, Model *m, int rows) {
- Cfg *c = &m->c; int D = c->hidden, E = c->n_experts, H = c->n_heads, hd = c->head_dim;
+/* olc_res counts instead of reserving while g_olc_count >= 0 (the chunk's sizing) */
+static long long g_olc_count = -1;
+static int olc_res(VkcBuf **b, size_t floats, int kind) {
+ if (g_olc_count >= 0) { g_olc_count += (long long)(floats ? floats : 1) * (long long)sizeof(float); return 1; }
+ return vkc_reserve(b, (floats ? floats : 1) * sizeof(float), kind);
+}
+static int olc_bufs(OlmChain *ch, Model *m, int rows);
+/* Prompt rows per chunk (vkc_chunk_rows): the chain's scratch a row, counted from the
+ * reservations, and the routed experts' outputs (the tier's rows, the CPU's
+ * contributions, and the host's sum). */
+static int olc_chunk_rows(OlmChain *ch, Model *m) {
+ g_olc_count = 0; olc_bufs(ch, m, 1); long long b1 = g_olc_count;
+ g_olc_count = 0; olc_bufs(ch, m, 2); long long b2 = g_olc_count;
+ g_olc_count = -1;
+ size_t row = (size_t)(b2 - b1) + (size_t)(2 * m->c.topk + 2) * m->c.hidden * sizeof(float);
+ if (ch->ks.on) row += (long long)m->c.n_heads * (4 * m->c.head_dim + 6) * sizeof(float);
+ return vkc_chunk_rows("olmoe", row);
+}
+/* the scratch buffers for `rows` rows */
+static int olc_bufs(OlmChain *ch, Model *m, int rows) {
+ Cfg *c = &m->c; int D = c->hidden, E = c->n_experts, hd = c->head_dim;
size_t r = (size_t)rows;
- int ok = olc_res(&ch->x, r * D, VKC_DEV) && olc_res(&ch->nrm, r * D, VKC_DEV) && olc_res(&ch->tmp, r * D, VKC_DEV) &&
+ return olc_res(&ch->x, r * D, VKC_DEV) && olc_res(&ch->nrm, r * D, VKC_DEV) && olc_res(&ch->tmp, r * D, VKC_DEV) &&
olc_res(&ch->q, r * D, VKC_DEV) && olc_res(&ch->k, r * D, VKC_DEV) && olc_res(&ch->v, r * D, VKC_DEV) &&
olc_res(&ch->ctx, r * D, VKC_DEV) && olc_res(&ch->h2, r * D, VKC_DEV) && olc_res(&ch->lg, r * E, VKC_DEV) &&
olc_res(&ch->fin, D, VKC_DEV) &&
olc_res(&ch->h2d, r * D, VKC_DOWN) && olc_res(&ch->lgd, r * E, VKC_DOWN) && olc_res(&ch->kvd, 2 * r * D, VKC_DOWN) &&
olc_res(&ch->outd, c->vocab, VKC_DOWN) && olc_res(&ch->routed, r * D, VKC_UP) && olc_res(&ch->cs, r * hd, VKC_UP);
- if (!ok) return 0;
- if (ch->rows < rows) {
- float *hr = realloc(ch->host_routed, r * D * sizeof(float)), *xp = hr ? realloc(ch->xpost, r * D * sizeof(float)) : NULL;
- if (hr) ch->host_routed = hr;
- if (!hr || !xp) return 0;
- ch->xpost = xp; ch->rows = rows;
- }
- if (ch->cap != m->max_t || ch->hostK != m->K) { /* the host cache is new: mirror it again */
+}
+/* Plan and allocate the KV mirror before sizing a prompt chunk. */
+static int olc_mirror(OlmChain *ch, Model *m) {
+ Cfg *c = &m->c; int H = c->n_heads, hd = c->head_dim;
+ if (ch->cap != m->max_t || ch->hostK != m->K) { /* the host cache is new: mirror it again (whole, or split) */
+ size_t row = (size_t)H * hd * 2 * sizeof(float);
+ int plan = vkc_kv_plan(&ch->ks, "olmoe", c->n_layers, row, m->max_t, 1, 0, (size_t)c->n_layers * ch->dev_rows * row);
+ if (!plan) { ch->cap = 0; ch->hostK = NULL; return 0; }
+ ch->dev_rows = ch->ks.on ? ch->ks.rows : m->max_t;
for (int i = 0; i < c->n_layers; i++) {
vkc_free(ch->kc[i]); vkc_free(ch->vc[i]); ch->kc[i] = ch->vc[i] = NULL;
ch->kv_valid[i] = 0;
- ch->kc[i] = vkc_buf((size_t)H * m->max_t * hd * sizeof(float), VKC_DEV);
- ch->vc[i] = vkc_buf((size_t)H * m->max_t * hd * sizeof(float), VKC_DEV);
+ ch->kc[i] = vkc_buf((size_t)H * ch->dev_rows * hd * sizeof(float), VKC_DEV);
+ ch->vc[i] = vkc_buf((size_t)H * ch->dev_rows * hd * sizeof(float), VKC_DEV);
if (!ch->kc[i] || !ch->vc[i]) { ch->cap = 0; ch->hostK = NULL; return 0; }
}
ch->cap = m->max_t; ch->hostK = m->K;
@@ -160,16 +179,43 @@ static int olc_scratch(OlmChain *ch, Model *m, int rows) {
return 1;
}
+/* scratch for `rows` rows, and the KV mirrors at the host's capacity */
+static int olc_scratch(OlmChain *ch, Model *m, int rows) {
+ Cfg *c = &m->c; int D = c->hidden, H = c->n_heads, hd = c->head_dim;
+ size_t r = (size_t)rows;
+ if (!olc_bufs(ch, m, rows)) return 0;
+ if (ch->rows < rows) {
+ float *hr = realloc(ch->host_routed, r * D * sizeof(float)), *xp = hr ? realloc(ch->xpost, r * D * sizeof(float)) : NULL;
+ if (hr) ch->host_routed = hr;
+ if (!hr || !xp) return 0;
+ ch->xpost = xp; ch->rows = rows;
+ }
+ if (ch->ks.on && !vkc_kv_parts(&ch->ks, r * H * (hd + 2))) return 0;
+ return 1;
+}
+
/* A CPU step from pos_base: the rows it writes into the host cache are not the device's. */
static void olc_cpu_step(Model *m, int pos_base) {
OlmChain *ch = (OlmChain *)m->vkchain;
if (!ch || !ch->ok) return;
- for (int i = 0; i < m->c.n_layers; i++) if (ch->kv_valid[i] > pos_base) ch->kv_valid[i] = pos_base;
+ for (int i = 0; i < m->c.n_layers; i++) {
+ if (ch->kv_valid[i] > pos_base) ch->kv_valid[i] = pos_base;
+ vkc_kv_lower(&ch->ks, i, pos_base);
+ }
}
-/* Record the uploads that make the device cache the host's below pos_base. */
-static int olc_push_kv(OlmChain *ch, Model *m, int pos_base) {
+/* Record the uploads that make the device cache the host's below pos_base, for a step
+ * of n rows (the split places its window). */
+static int olc_push_kv(OlmChain *ch, Model *m, int pos_base, int n_rows) {
Cfg *c = &m->c; int hd = c->head_dim, ok = 1;
for (int i = 0; i < c->n_layers && ok; i++) {
+ if (ch->ks.on) {
+ VkcKvPart pt[2] = {{c->n_heads, hd, m->K[i], (size_t)m->max_t * hd, ch->kc[i], 0},
+ {c->n_heads, hd, m->V[i], (size_t)m->max_t * hd, ch->vc[i], 0}};
+ vkc_kv_place(&ch->ks, i, pos_base, n_rows);
+ ok = vkc_kv_push(&ch->ks, i, pt, 2, pos_base);
+ ch->kv_valid[i] = pos_base;
+ continue;
+ }
if (ch->kv_valid[i] >= pos_base) continue;
int t0 = ch->kv_valid[i], n = pos_base - t0;
for (int h = 0; h < c->n_heads && ok; h++) {
@@ -196,15 +242,25 @@ static int olc_attention(OlmChain *ch, Model *m, Layer *l, int i, int n, int pb)
ok = ok && vkc_norm(ch->q, ch->prm, ch->q, &qn) && vkc_norm(ch->k, ch->prm, ch->k, &kn) &&
vkc_rope(ch->q, ch->cs, &rp) && vkc_rope(ch->k, ch->cs, &rp);
if (!ok) return 0;
+ if (ch->ks.on) { /* the split: the rows into their window slots, the attention in two parts */
+ VkcKvPart pk = {H, hd, m->K[i], (size_t)m->max_t * hd, ch->kc[i], 0}, pv = {H, hd, m->V[i], (size_t)m->max_t * hd, ch->vc[i], 0};
+ VkcKvGqa a = {&ch->ks, i, ch->q, ch->kc[i], ch->vc[i], NULL, NULL, NULL, ch->ctx, n, H, H, hd, hd, pb, 0, 0, 0, 0,
+ D, hd, 0, 0, 0, 0, D, 0, 0, 1.f / sqrtf((float)hd),
+ m->K[i], m->V[i], (size_t)m->max_t * hd, (size_t)hd, (size_t)m->max_t * hd, (size_t)hd};
+ return vkc_kv_store(&ch->ks, i, &pk, ch->k, 0, (size_t)D, (size_t)hd, pb, n) &&
+ vkc_kv_store(&ch->ks, i, &pv, ch->v, 0, (size_t)D, (size_t)hd, pb, n) &&
+ vkc_copy(ch->kvd, 0, ch->k, 0, (size_t)n * D) && vkc_copy(ch->kvd, (size_t)ch->rows * D, ch->v, 0, (size_t)n * D) &&
+ vkc_kv_gqa(&a) && vkc_matmul((ColiVkTensor *)l->vk_o, ch->ctx, 0, ch->tmp, 0, n);
+ }
/* the new rows into the device cache, and down for the host's */
VkcRegion *rg = malloc(sizeof *rg * (size_t)n * H);
if (!rg) return 0;
for (int s = 0; s < n; s++) for (int h = 0; h < H; h++)
- rg[s * H + h] = (VkcRegion){((size_t)h * ch->cap + pb + s) * hd, (size_t)s * D + (size_t)h * hd, (size_t)hd};
+ rg[s * H + h] = (VkcRegion){((size_t)h * ch->dev_rows + pb + s) * hd, (size_t)s * D + (size_t)h * hd, (size_t)hd};
ok = vkc_copy_regions(ch->kc[i], ch->k, rg, n * H) && vkc_copy_regions(ch->vc[i], ch->v, rg, n * H) &&
vkc_copy(ch->kvd, 0, ch->k, 0, (size_t)n * D) && vkc_copy(ch->kvd, (size_t)ch->rows * D, ch->v, 0, (size_t)n * D);
free(rg);
- VkcAttn a = {n, H, H, hd, pb, ch->cap, 0, D, hd, 0, 0, 0, 0, 0, D, 0, 0, 1.f / sqrtf((float)hd), 0, 0};
+ VkcAttn a = {n, H, H, hd, pb, ch->dev_rows, 0, D, hd, 0, 0, 0, 0, 0, D, 0, 0, 1.f / sqrtf((float)hd), 0, 0};
return ok && vkc_attn(ch->q, ch->kc[i], ch->vc[i], ch->ctx, NULL, NULL, &a) &&
vkc_matmul((ColiVkTensor *)l->vk_o, ch->ctx, 0, ch->tmp, 0, n);
}
@@ -231,9 +287,11 @@ static int olc_forward(Model *m, float *xh, int S, int pos_base, int want_x, flo
OlmChain *ch = olc_setup(m);
if (!ch || ch->failed) return 0;
Cfg *c = &m->c; int D = c->hidden, L = c->n_layers, E = c->n_experts, H = c->n_heads, hd = c->head_dim;
- int CH = olc_chunk_rows(), rows = S < CH ? S : CH, half = hd / 2;
+ int mirror_ok = olc_mirror(ch, m);
+ int CH = mirror_ok ? olc_chunk_rows(ch, m) : 1, rows = S < CH ? S : CH, half = hd / 2;
+ if (ch->ks.on && rows > ch->ks.chunk) rows = ch->ks.chunk; /* a step's rows fit the split's window */
int pilot = g_pilot >= 1 && S <= 8; /* layers_forward_range's PILOT condition */
- if (pos_base + S > m->max_t || !olc_scratch(ch, m, rows) ||
+ if (!mirror_ok || pos_base + S > m->max_t || !olc_scratch(ch, m, rows) ||
(want_x && !olc_res(&ch->xd, (size_t)rows * D, VKC_DOWN)) ||
(pilot && !olc_res(&ch->xpd, (size_t)rows * D, VKC_DOWN))) {
fprintf(stderr, "[VK] olmoe chain: device memory for %d rows refused; per-matrix path\n", rows);
@@ -246,7 +304,7 @@ static int olc_forward(Model *m, float *xh, int S, int pos_base, int want_x, flo
for (int c0 = 0; c0 < S; c0 += rows) {
int n = S - c0 < rows ? S - c0 : rows, pb = pos_base + c0;
if (!vkc_begin() || !vkc_write(ch->x, 0, xh + (size_t)c0 * D, (size_t)n * D * sizeof(float)) ||
- !olc_push_kv(ch, m, pb)) goto lost;
+ !olc_push_kv(ch, m, pb, n)) goto lost;
float *cs = (float *)vkc_ptr(ch->cs); /* rope_head's angles, cosines and sines */
for (int s = 0; s < n; s++)
for (int j = 0; j < half; j++) {
@@ -269,7 +327,11 @@ static int olc_forward(Model *m, float *xh, int S, int pos_base, int want_x, flo
vkc_matmul((ColiVkTensor *)l->vk_gate, ch->h2, 0, ch->lg, 0, n) &&
vkc_copy(ch->lgd, 0, ch->lg, 0, (size_t)n * E) && vkc_copy(ch->h2d, 0, ch->h2, 0, (size_t)n * D);
double t0 = now_s();
- ok = ok && vkc_submit(1);
+ /* while the frame runs, the tier loads the experts this layer will likely
+ * stream (a big prompt chunk only) */
+ ok = ok && vkc_submit(0);
+ if (ok) vkt_stream_prefetch(i, n);
+ ok = ok && vkc_finish();
g_prof_attn_s += now_s() - t0;
if (!ok) break;
const float *kv = (const float *)vkc_ptr(ch->kvd); /* the rows into the host's cache */
@@ -303,7 +365,7 @@ static int olc_forward(Model *m, float *xh, int S, int pos_base, int want_x, flo
g_prof_head_s += now_s() - t0;
if (!ok) goto lost;
if (want_x) memcpy(xfin + (size_t)c0 * D, vkc_ptr(ch->xd), (size_t)n * D * sizeof(float));
- for (int i = 0; i < L; i++) ch->kv_valid[i] = pb + n;
+ for (int i = 0; i < L; i++) { ch->kv_valid[i] = pb + n; vkc_kv_done(&ch->ks, i, pb + n); }
if (last) memcpy(logit, vkc_ptr(ch->outd), (size_t)c->vocab * sizeof(float));
}
if (want_x) { memcpy(xh, xfin, (size_t)S * D * sizeof(float)); free(xfin); }
@@ -323,6 +385,7 @@ static void olc_report(Model *m) {
fprintf(stderr, "[VK] olmoe chain: %llu forwards, %llu frames (%llu ops, %llu matmuls, %llu tiled GEMM), "
"%.1f ms waiting for the device, %.1f ms of routed experts on the host, %.1f MiB on the device\n",
ch->forwards, st.frames, st.ops, st.matmuls, st.gemms, st.wait_ms, ch->host_ms, st.dev_bytes / 1048576.0);
+ vkc_kv_report(&ch->ks);
vkc_prof_print();
}
diff --git a/c/qwen36.c b/c/qwen36.c
index f6072fd5..9f60e7a4 100644
--- a/c/qwen36.c
+++ b/c/qwen36.c
@@ -44,6 +44,11 @@
* -- which is why Q36_MAXT still defaults far below this. */
#define QWEN36_ATTN_MAX_CTX 262144
#define QWEN36_DEFAULT_MAX_CTX 8192
+/* A speculative verify's rows: the picked token and up to five prompt-lookup drafts.
+ * A rejection after row k restores the state the verify copied after that row. */
+#define Q36_SPEC_ROWS 6
+#define Q36_SPEC_SNAPS (Q36_SPEC_ROWS - 1)
+static int g_q36_rowwise; /* set for a speculative verify: its rows must get a decode step's bits (q36_spec_step) */
/* Effective ceiling: Q36_MAXT if set and sane, the conservative default
* otherwise; never above the hard limit. */
@@ -66,6 +71,7 @@ static int qwen36_max_ctx(void) {
#include "st.h"
#include "omp_tune.h"
#include "kv_prefix.h"
+#include "spec_draft.h" /* prompt-lookup drafts and the gate that decides when they pay (COLI_LOOKUP=1) */
#include "pin_pool.h" /* riuso del prefisso tra turni (shared) */
#include "decode_batch.h" /* ColiSubmit + coli_submit_ext: le chiavi key=value di SUBMIT */
#include "json.h" /* tokenizer.json parsing (reuse minimal parser) */
@@ -86,7 +92,37 @@ static int qwen36_max_ctx(void) {
#include "backend_vulkan.h" /* COLI_VULKAN=1: the resident dense trunk on a Vulkan device */
static int g_vk_ready = 0;
static int g_vk_dense = 1; /* COLI_VK_DENSE=0: the dense trunk stays on the CPU, the expert tier alone uses the device */
+static int g_vk_import = 0; /* COLI_VK_IMPORT: the device reads the int8, f16 and f32 rows where the host keeps them */
#endif
+/* The dense rows' host allocation. With COLI_VULKAN set (a VK=1 build), page-aligned and
+ * a whole number of pages, so a device that shares the CPU's RAM can read them in place
+ * (coli_vk_tensor_import) instead of holding a second copy; otherwise malloc. The
+ * values, and what the CPU computes from them, are the same either way. */
+#define Q36_PAGE 4096
+static int q36_waligned(void){
+#ifdef COLI_VULKAN
+ static int on = -1;
+ if (on < 0) { const char *e = getenv("COLI_VULKAN"); on = e && atoi(e) != 0; }
+ return on;
+#else
+ return 0;
+#endif
+}
+static void *q36_walloc(size_t bytes){
+ if (q36_waligned()) {
+ void *p = NULL;
+ size_t sz = (bytes + Q36_PAGE - 1) / Q36_PAGE * Q36_PAGE;
+ return posix_memalign(&p, Q36_PAGE, sz ? sz : Q36_PAGE) ? NULL : p;
+ }
+ return malloc(bytes);
+}
+static void q36_wfree(void *p){
+ /* Windows' posix_memalign is _aligned_malloc. Every q/h allocation uses
+ * the same cached choice as q36_walloc, including an int8 copy discarded
+ * after int4 packing, so its matching release must follow that choice. */
+ if (q36_waligned()) compat_aligned_free(p);
+ else free(p);
+}
#ifdef COLI_SEGMENT_ADAPTER
#include "segment_runtime.h"
#include "segment_adapters.h"
@@ -778,14 +814,19 @@ typedef struct {
* matmul_d reads, uploaded at the first matmul_d and kept; vk_off = the upload
* failed once, this matrix stays on the CPU. Both stay zero without VK=1. */
/* h: the matrix as f16 (COLI_DENSE_BITS=16), the only copy then. */
+/* vk_name, vk_tag, vk_quant: what load_tq read it with, to read it back from disk
+ * (q36_dho_reload); vk_fmt, vk_gone: with the dense weights on the device only
+ * (COLI_VK_DENSE_HOST), the device copy's format and 1 while no host copy is kept. */
typedef struct { const float *w; int8_t *q; float *sc; int I, O; uint8_t *q4; float *sg; int ng;
- void *vk; int vk_off; uint16_t *h; } QW;
+ void *vk; int vk_off; uint16_t *h;
+ char *vk_name; const char *vk_tag; int vk_quant, vk_fmt, vk_gone, vk_imported; } QW;
static void qw_free(QW *w) {
#ifdef COLI_VULKAN
if (w->vk) coli_vk_tensor_free((ColiVkTensor *)w->vk);
- w->vk = NULL; w->vk_off = 0;
+ w->vk = NULL; w->vk_off = 0; w->vk_imported = 0;
+ free(w->vk_name); w->vk_name = NULL; w->vk_gone = 0;
#endif
- free((void*)w->w); free(w->q); free(w->sc); free(w->q4); free(w->sg); free(w->h);
+ free((void*)w->w); q36_wfree(w->q); free(w->sc); free(w->q4); free(w->sg); q36_wfree(w->h);
w->w = NULL; w->q = NULL; w->sc = NULL; w->q4 = NULL; w->sg = NULL; w->ng = 0; w->h = NULL;
}
@@ -882,6 +923,12 @@ typedef struct {
#ifdef COLI_VULKAN
void *vkchain; /* the dense chain's device state (qwen36_chain.h), NULL until it runs */
#endif
+ /* A speculative verify (prompt lookup, q36_spec_step) copies the DeltaNet state
+ * after each of its first snap_rows rows, row r into slot r (0 = no copy), so a
+ * draft rejected after row r+1 rolls back by swapping slot r in. snap_slots: the
+ * slots allocated, the first time a verify that deep runs. */
+ int snap_rows, snap_slots;
+ float **snap_rec[Q36_SPEC_SNAPS], **snap_conv[Q36_SPEC_SNAPS];
} Model;
static pthread_mutex_t g_pilot_mx = PTHREAD_MUTEX_INITIALIZER;
@@ -1342,7 +1389,7 @@ static int dense_keep_i8(void){
return v;
}
static void qw_quantize(const float *W, int I, int O, const char *tag, QW *out) {
- int8_t *q = malloc((size_t)O*I); float *sc = malloc((size_t)O*sizeof(float));
+ int8_t *q = q36_walloc((size_t)O*I); float *sc = malloc((size_t)O*sizeof(float));
if (!q || !sc) { fprintf(stderr, "OOM qw_quantize\n"); exit(1); }
#pragma omp parallel for schedule(static)
for (int o = 0; o < O; o++) {
@@ -1359,7 +1406,7 @@ static void qw_quantize(const float *W, int I, int O, const char *tag, QW *out)
if (q4 && sg) {
pack_int4_g64_planar(W, q4, sg, O, I);
out->q4 = q4; out->sg = sg; out->ng = I/64;
- if (!dense_keep_i8()) { free(out->q); free(out->sc); out->q = NULL; out->sc = NULL; }
+ if (!dense_keep_i8()) { q36_wfree(out->q); free(out->sc); out->q = NULL; out->sc = NULL; }
} else { free(q4); free(sg); }
}
}
@@ -1391,21 +1438,48 @@ static void vk_q4_planar_to_fmt4(const QW *w, uint8_t *dst) {
}
}
}
-static unsigned g_vk_placed[3]; /* uploads by format: int8 rows, int4, f32 */
+static unsigned g_vk_placed[4]; /* uploads by format: int8 rows, int4, f32, f16 */
+/* The format a matrix goes to the device in: the copy matmul_d reads (int4, int8 rows,
+ * f16 with COLI_DENSE_BITS=16, else f32), its rows and its scales. */
+static int vk_qw_fmt(const QW *w, const void **wq, const float **sc) {
+ if (w->vk_gone) { *wq = NULL; *sc = NULL; return w->vk_fmt; }
+ if (w->q4) { *wq = w->q4; *sc = w->sg; return 4; }
+ if (w->q) { *wq = w->q; *sc = w->sc; return 1; }
+ if (w->h) { *wq = w->h; *sc = NULL; return 14; }
+ *wq = w->w; *sc = NULL; /* fmt 10: no scales */
+ return 10;
+}
+static void q36_dho_reload(QW *w);
/* w's device copy, uploaded on the first call (the int4 copy repacked); NULL when it
* cannot be (vk_off then keeps the matrix on the CPU). The dense chain reads the same
- * tensor, so the per-matrix path and the chain never hold a matrix twice. */
+ * tensor, so the per-matrix path and the chain never hold a matrix twice. With
+ * COLI_VK_IMPORT and host copies retained (or an integrated GPU), the int8 and f16
+ * rows are not copied: the device reads q36_walloc's pages in place, and a matrix
+ * the import refuses is copied as before. */
static ColiVkTensor *vk_qw_tensor(const QW *w) {
if (w->vk_off) return NULL;
QW *mw = (QW *)w; /* vk is a cache in a matrix the forward pass treats as read-only */
ColiVkTensor **t = (ColiVkTensor **)&mw->vk;
if (*t) return *t;
+ if (w->vk_gone) return NULL;
int I = w->I, O = w->O;
- int fmt = w->q4 ? 4 : w->q ? 1 : 10, gs = w->q4 ? 64 : 0;
- const void *wq = w->q4 ? (const void *)w->q4 : w->q ? (const void *)w->q : (const void *)w->w;
- const float *sc = w->q4 ? w->sg : w->q ? w->sc : NULL; /* fmt 10: no scales */
+ const void *wq; const float *sc;
+ int fmt = vk_qw_fmt(w, &wq, &sc), gs = fmt == 4 ? 64 : 0;
if (!wq) return NULL;
int ok;
+ /* On an integrated device these are already one shared allocation. Retain
+ * it instead of moving the same physical bytes into a smaller logical
+ * device-local heap. q36_dho_drop keeps imported pages alive. */
+ if (g_vk_import && (coli_vk_device_integrated() || !coli_vk_dense_device_only()) &&
+ fmt != 4 && w->w != wq) { /* q36_walloc'd rows (f32 rows are falloc's) */
+ size_t rb = fmt == 1 ? (size_t)I : (size_t)I * 2;
+ size_t alloc = (rb * (size_t)O + Q36_PAGE - 1) / Q36_PAGE * Q36_PAGE;
+ if (coli_vk_tensor_import(t, wq, alloc, sc, fmt, I, O, gs)) {
+ mw->vk_imported = 1;
+ g_vk_placed[fmt == 1 ? 0 : 3]++;
+ return *t;
+ }
+ }
if (w->q4) {
uint8_t *packed = malloc((size_t)O * (I / 2));
if (packed) vk_q4_planar_to_fmt4(w, packed);
@@ -1413,7 +1487,7 @@ static ColiVkTensor *vk_qw_tensor(const QW *w) {
free(packed);
} else ok = coli_vk_tensor_ensure(t, wq, sc, fmt, I, O, gs);
if (!ok) { mw->vk_off = 1; return NULL; }
- g_vk_placed[fmt == 1 ? 0 : fmt == 4 ? 1 : 2]++;
+ g_vk_placed[fmt == 1 ? 0 : fmt == 4 ? 1 : fmt == 14 ? 3 : 2]++;
return *t;
}
static int vk_dense_matmul(float *y, const float *x, const QW *w, int S, int I, int O) {
@@ -1421,12 +1495,17 @@ static int vk_dense_matmul(float *y, const float *x, const QW *w, int S, int I,
if (omp_in_parallel()) return 0;
#endif
if (w->vk_off || w->I != I || w->O != O || S < 1 || S > 65535) return 0;
- int fmt = w->q4 ? 4 : w->q ? 1 : 10, gs = w->q4 ? 64 : 0;
- const void *wq = w->q4 ? (const void *)w->q4 : w->q ? (const void *)w->q : (const void *)w->w;
- const float *sc = w->q4 ? w->sg : w->q ? w->sc : NULL; /* fmt 10: no scales */
- if (!wq) return 0;
+ const void *wq; const float *sc;
+ int fmt = vk_qw_fmt(w, &wq, &sc), gs = fmt == 4 ? 64 : 0;
+ if (!wq && !w->vk_gone) return 0;
ColiVkTensor *t = vk_qw_tensor(w);
if (!t) return 0;
+ /* a verify's rows (g_q36_rowwise) one at a time: a decode step's GEMV, not the batch's GEMM */
+ if (g_q36_rowwise && S > 1) {
+ for (int s = 0; s < S; s++)
+ if (!coli_vk_matmul(&t, y + (int64_t)s * O, x + (int64_t)s * I, wq, sc, fmt, 1, I, O, gs)) return 0;
+ return 1;
+ }
return coli_vk_matmul(&t, y, x, wq, sc, fmt, S, I, O, gs);
}
/* One line at the end of a run or a serve turn: how many matmuls the device
@@ -1442,9 +1521,13 @@ static void vk_report(void) {
if (!g_vk_ready) return;
size_t bytes = 0, tensors = 0;
coli_vk_mem_info(&bytes, &tensors);
- fprintf(stderr, "[VK] qwen36: %llu matmuls on the GPU (%zu matrices resident, %.1f MiB; placed int8 %u, int4 %u, f32 %u)\n",
+ char more[96] = "";
+ size_t imp = coli_vk_imported_bytes();
+ if (g_vk_placed[3] || imp)
+ snprintf(more, sizeof more, ", f16 %u; %.1f MiB read in place", g_vk_placed[3], imp / 1048576.0);
+ fprintf(stderr, "[VK] qwen36: %llu matmuls on the GPU (%zu matrices resident, %.1f MiB; placed int8 %u, int4 %u, f32 %u%s)\n",
coli_vk_matmul_calls(), tensors, bytes / 1048576.0,
- g_vk_placed[0], g_vk_placed[1], g_vk_placed[2]);
+ g_vk_placed[0], g_vk_placed[1], g_vk_placed[2], more);
}
#endif
/* f32 -> f16, round to nearest even: exact for a value that came from f16 (the
@@ -1522,7 +1605,9 @@ static void matmul_d(float *y, const float *x, const QW *w, int S, int I, int O)
g_qwen_matmul_d_calls++;
#endif
#ifdef COLI_VULKAN
- if (g_vk_ready && g_vk_dense && (w->q || w->q4 || w->w) && vk_dense_matmul(y, x, w, S, I, O)) return;
+ if (g_vk_ready && g_vk_dense && (w->q || w->q4 || w->w || w->h || w->vk_gone) && vk_dense_matmul(y, x, w, S, I, O)) return;
+ /* the CPU needs a matrix the device holds alone (a lost device): read it back */
+ if (w->vk_gone) q36_dho_reload((QW *)w);
#endif
if (w->h) { matmul_h(y, x, w->h, S, I, O); return; }
if (w->q || w->q4) {
@@ -1963,7 +2048,7 @@ static void load_tq(Model *m, const char *name, int I, int O, int quantize, cons
out->vk = NULL; out->vk_off = 0; out->h = NULL;
if (!quantize || !dense_i8_on()) return;
if (dense_bits() == 16) {
- uint16_t *h = malloc((size_t)I * O * sizeof(uint16_t));
+ uint16_t *h = q36_walloc((size_t)I * O * sizeof(uint16_t));
if (!h) { fprintf(stderr, "OOM keeping %s in f16\n", name); exit(1); }
#pragma omp parallel for schedule(static)
for (int64_t k = 0; k < (int64_t)I * O; k++) h[k] = f32_to_f16_bits(p[k]);
@@ -1974,6 +2059,76 @@ static void load_tq(Model *m, const char *name, int I, int O, int quantize, cons
qw_quantize(p, I, O, tag, out);
if (getenv("COLI_KEEP_F32")) out->w = p; else { free(p); out->w = NULL; }
}
+/* load_tq, and under COLI_VULKAN a note of what it read (name, tag, whether it was
+ * quantized) for q36_dho_reload. Every resident dense matrix of the model loads here. */
+static void load_tq_noted(Model *m, const char *name, int I, int O, int quantize, const char *tag, QW *out) {
+ load_tq(m, name, I, O, quantize, tag, out);
+#ifdef COLI_VULKAN
+ out->vk_name = strdup(name); out->vk_tag = tag; out->vk_quant = quantize;
+ out->vk_fmt = 0; out->vk_gone = 0;
+ if (!out->vk_name) { fprintf(stderr, "OOM dense matrix name\n"); exit(1); }
+#endif
+}
+#ifdef COLI_VULKAN
+/* ---- dense weights on the device only (COLI_VK_DENSE_HOST) ------------------------
+ * With the dense part on the device (the chain, or COLI_VK_DENSE), every resident
+ * dense matrix goes up at start and its host copy (int8 rows, the int4 copy, f32 or f16) is
+ * given back. The CPU then never multiplies by it, except after a lost device:
+ * matmul_d first reads it back here, through load_tq with what it was loaded with
+ * (the same quantization, so the same bytes), and keeps it from there on. What keeps
+ * its host copy: the embedding (its rows are gathered on the CPU), the DeltaNet's
+ * dn_a/dn_b and the shared expert's gate vector (the CPU's DeltaNet and shared expert
+ * read them, and the chain uploads its own copies), norms, the vision tower, a matrix
+ * the device refused, or rows it already imported in place. */
+static Model *g_q36_dho_model;
+static pthread_mutex_t g_q36_dho_mx = PTHREAD_MUTEX_INITIALIZER;
+static size_t q36_dho_host_bytes(const QW *w) {
+ size_t b = 0;
+ if (w->q4) b += (size_t)w->O * (w->I / 2) + (size_t)w->O * (w->I / 64) * sizeof(float);
+ if (w->q) b += (size_t)w->O * w->I + (size_t)w->O * sizeof(float);
+ if (w->w) b += (size_t)w->O * w->I * sizeof(float);
+ if (w->h) b += (size_t)w->O * w->I * sizeof(uint16_t);
+ return b;
+}
+static void q36_dho_reload(QW *w) {
+ pthread_mutex_lock(&g_q36_dho_mx);
+ if (w->vk_gone) {
+ Model *m = g_q36_dho_model;
+ if (!m || !w->vk_name) { fprintf(stderr, "[VK] qwen36: a dense matrix the device held alone cannot be read back\n"); exit(1); }
+ QW t; memset(&t, 0, sizeof t);
+ load_tq(m, w->vk_name, w->I, w->O, w->vk_quant, w->vk_tag, &t);
+ w->w = t.w; w->q = t.q; w->sc = t.sc; w->q4 = t.q4; w->sg = t.sg; w->ng = t.ng; w->h = t.h;
+ w->vk_gone = 0;
+ coli_vk_dense_host_reloaded(q36_dho_host_bytes(w));
+ }
+ pthread_mutex_unlock(&g_q36_dho_mx);
+}
+static void q36_dho_drop(QW *w, size_t *bytes, int *n) {
+ if (!w->vk_name || w->vk_gone || w->vk_off || w->vk_imported || !(w->q || w->q4 || w->w || w->h)) return;
+ const void *wq; const float *sc;
+ int fmt = vk_qw_fmt(w, &wq, &sc);
+ if (!vk_qw_tensor(w) || w->vk_imported) return; /* refused or still read in place: keep its host pages */
+ size_t b = q36_dho_host_bytes(w);
+ free((void *)w->w); q36_wfree(w->q); free(w->sc); free(w->q4); free(w->sg); q36_wfree(w->h);
+ w->h = NULL; w->w = NULL; w->q = NULL; w->sc = NULL; w->q4 = NULL; w->sg = NULL;
+ w->vk_fmt = fmt; w->vk_gone = 1;
+ coli_vk_dense_host_dropped(b);
+ *bytes += b; (*n)++;
+}
+static void q36_dho_count(QW *w, size_t *bytes, int *n) {
+ if (w->vk_name && !w->vk_off && !w->vk_imported && (w->q || w->q4 || w->w || w->h)) { *bytes += q36_dho_host_bytes(w); (*n)++; }
+}
+/* Every resident dense matrix: the head, then each layer's. */
+static void q36_dho_each(Model *m, void (*f)(QW *, size_t *, int *), size_t *bytes, int *n) {
+ f(&m->lm_head, bytes, n);
+ for (int i = 0; i < m->c.n_layers; i++) {
+ Layer *l = &m->L[i];
+ QW *ws[] = {&l->q, &l->k, &l->v, &l->o, &l->gate, &l->sh_g, &l->sh_u, &l->sh_d,
+ &l->dn_qkv, &l->dn_z, &l->dn_out};
+ for (size_t k = 0; k < sizeof ws / sizeof ws[0]; k++) f(ws[k], bytes, n);
+ }
+}
+#endif
/* ---------- vision (#1757) ----------
* The weights are the checkpoint's own model.visual.*, copied by the converter,
@@ -2148,7 +2303,7 @@ static void model_init_range(Model *m, const char *snap, int cap, int bits,
free(m->embed); m->embed = NULL;
}
#endif
- load_tq(m, "lm_head.weight", c->hidden, c->vocab, quantize_dense, "lmhead", &m->lm_head);
+ load_tq_noted(m, "lm_head.weight", c->hidden, c->vocab, quantize_dense, "lmhead", &m->lm_head);
if (m->lm_head.q || m->lm_head.q4 || m->lm_head.h) { qcount++; qfreed += (double)c->hidden * c->vocab * sizeof(float); }
m->final_norm = load_norm_n(m, "model.norm.weight", c->hidden);
q36_load_vision(m);
@@ -2173,7 +2328,7 @@ static void model_init_range(Model *m, const char *snap, int cap, int bits,
#undef LD
if (c->n_experts) {
snprintf(nm,sizeof(nm),"model.layers.%d.mlp.gate.weight", ai);
- load_tq(m, nm, c->hidden, c->n_experts, quantize_dense, "router", &l->gate);
+ load_tq_noted(m, nm, c->hidden, c->n_experts, quantize_dense, "router", &l->gate);
QCOUNT(l->gate);
} else l->gate = (QW){0};
/* q/k norms are per-head [head_dim]; only on attention layers, load if present */
@@ -2192,11 +2347,11 @@ static void model_init_range(Model *m, const char *snap, int cap, int bits,
const char *shp = c->n_experts ? "mlp.shared_expert." : "mlp.";
if (c->shared_inter > 0) { /* Qwen3 MoE has none: its sh_* stay empty */
snprintf(nm,sizeof(nm),"model.layers.%d.%sgate_proj.weight", ai, shp);
- load_tq(m, nm, c->hidden, c->shared_inter, quantize_dense, "shexp", &l->sh_g); QCOUNT(l->sh_g);
+ load_tq_noted(m, nm, c->hidden, c->shared_inter, quantize_dense, "shexp", &l->sh_g); QCOUNT(l->sh_g);
snprintf(nm,sizeof(nm),"model.layers.%d.%sup_proj.weight", ai, shp);
- load_tq(m, nm, c->hidden, c->shared_inter, quantize_dense, "shexp", &l->sh_u); QCOUNT(l->sh_u);
+ load_tq_noted(m, nm, c->hidden, c->shared_inter, quantize_dense, "shexp", &l->sh_u); QCOUNT(l->sh_u);
snprintf(nm,sizeof(nm),"model.layers.%d.%sdown_proj.weight", ai, shp);
- load_tq(m, nm, c->shared_inter, c->hidden, quantize_dense, "shexp", &l->sh_d); QCOUNT(l->sh_d);
+ load_tq_noted(m, nm, c->shared_inter, c->hidden, quantize_dense, "shexp", &l->sh_d); QCOUNT(l->sh_d);
}
/* shared_expert_gate: Linear(hidden -> 1), sigmoid-gated shared expert */
snprintf(nm,sizeof(nm),"model.layers.%d.mlp.shared_expert_gate.weight", ai);
@@ -2204,13 +2359,13 @@ static void model_init_range(Model *m, const char *snap, int cap, int bits,
if (c->is_attn[i]) {
/* Gated Attention (full_attention) layer, dense projections int8-during-load */
snprintf(nm,sizeof(nm),"model.layers.%d.self_attn.q_proj.weight", ai);
- load_tq(m, nm, c->hidden, q_out, quantize_dense, "attn", &l->q); QCOUNT(l->q);
+ load_tq_noted(m, nm, c->hidden, q_out, quantize_dense, "attn", &l->q); QCOUNT(l->q);
snprintf(nm,sizeof(nm),"model.layers.%d.self_attn.k_proj.weight", ai);
- load_tq(m, nm, c->hidden, kv_out, quantize_dense, "attn", &l->k); QCOUNT(l->k);
+ load_tq_noted(m, nm, c->hidden, kv_out, quantize_dense, "attn", &l->k); QCOUNT(l->k);
snprintf(nm,sizeof(nm),"model.layers.%d.self_attn.v_proj.weight", ai);
- load_tq(m, nm, c->hidden, kv_out, quantize_dense, "attn", &l->v); QCOUNT(l->v);
+ load_tq_noted(m, nm, c->hidden, kv_out, quantize_dense, "attn", &l->v); QCOUNT(l->v);
snprintf(nm,sizeof(nm),"model.layers.%d.self_attn.o_proj.weight", ai);
- load_tq(m, nm, c->o_in, c->hidden, quantize_dense, "attn", &l->o); QCOUNT(l->o);
+ load_tq_noted(m, nm, c->o_in, c->hidden, quantize_dense, "attn", &l->o); QCOUNT(l->o);
l->dn_qkv=l->dn_z=(QW){0}; l->dn_b=l->dn_a=l->dn_conv=NULL;
l->dn_dtbias=l->dn_alog=l->dn_norm=NULL; l->dn_out=(QW){0};
} else {
@@ -2219,9 +2374,9 @@ static void model_init_range(Model *m, const char *snap, int cap, int bits,
#define LD4(field, suffix, want) snprintf(nm,sizeof(nm),"model.layers.%d.linear_attn." suffix,ai); l->field = load_t_n(m,nm,(want))
int64_t vdim_tot = (int64_t)c->dn_vheads * c->dn_vdim;
snprintf(nm,sizeof(nm),"model.layers.%d.linear_attn.in_proj_qkv.weight", ai);
- load_tq(m, nm, c->hidden, c->dn_conv_dim, quantize_dense, "dnproj", &l->dn_qkv); QCOUNT(l->dn_qkv);
+ load_tq_noted(m, nm, c->hidden, c->dn_conv_dim, quantize_dense, "dnproj", &l->dn_qkv); QCOUNT(l->dn_qkv);
snprintf(nm,sizeof(nm),"model.layers.%d.linear_attn.in_proj_z.weight", ai);
- load_tq(m, nm, c->hidden, (int)vdim_tot, quantize_dense, "dnproj", &l->dn_z); QCOUNT(l->dn_z);
+ load_tq_noted(m, nm, c->hidden, (int)vdim_tot, quantize_dense, "dnproj", &l->dn_z); QCOUNT(l->dn_z);
LD4(dn_b, "in_proj_b.weight", (int64_t)c->dn_vheads * c->hidden);
LD4(dn_a, "in_proj_a.weight", (int64_t)c->dn_vheads * c->hidden);
LD4(dn_conv,"conv1d.weight", (int64_t)c->dn_conv_dim * c->dn_convk);
@@ -2230,7 +2385,7 @@ static void model_init_range(Model *m, const char *snap, int cap, int bits,
LD4(dn_norm, "norm.weight", c->dn_vdim);
#undef LD4
snprintf(nm,sizeof(nm),"model.layers.%d.linear_attn.out_proj.weight", ai);
- load_tq(m, nm, (int)vdim_tot, c->hidden, quantize_dense, "dnout", &l->dn_out); QCOUNT(l->dn_out);
+ load_tq_noted(m, nm, (int)vdim_tot, c->hidden, quantize_dense, "dnout", &l->dn_out); QCOUNT(l->dn_out);
}
}
#undef QCOUNT
@@ -3079,7 +3234,7 @@ static void moe_vk_i8_cpu(Model *m, int layer, const float *x, int S, const int
static void moe_vk_run(Model *m, Layer *l, int layer, const float *x, int S, float *out,
const int *idx, const float *val, int routed_only) {
Cfg *c = &m->c; int D = c->hidden, K = c->topk, I = c->inter, SI = routed_only ? 0 : c->shared_inter;
- int xf = xf_mode(m), B = S < QWEN_VK_ROWS ? S : QWEN_VK_ROWS;
+ int xf = xf_mode(m), B = vkt_step_rows(S, QWEN_VK_ROWS); /* a whole prompt chunk when the tier streams */
float *ctb = falloc((int64_t)B * K * D), *shb = SI > 0 ? falloc((int64_t)B * D) : NULL;
float *g = falloc(I > SI ? I : SI), *u = falloc(I > SI ? I : SI), *hh = falloc(D);
uint8_t *taken = malloc((size_t)B * K), *want = malloc((size_t)B * K);
@@ -3547,6 +3702,12 @@ static void deltanet(Model *m, Layer *l, int layer, float *x, int S, int pos_bas
}
}
if (tm_on() && S==1){ double t=tm_now(); g_dn_sub[2]+=t-_d0; _d0=t; }
+ /* a verify: the state this row leaves, for the rollback of a draft rejected
+ * after it (q36_spec_rollback) */
+ if (base + r < m->snap_rows) {
+ memcpy(m->snap_rec[base + r][layer], rec, (size_t)vh * kdim * vdim * sizeof(float));
+ memcpy(m->snap_conv[base + r][layer], ring, (size_t)conv_dim * (convk - 1) * sizeof(float));
+ }
/* per-head Gated RMSNorm (plain weight, r=1/sqrt(mean+eps)) then silu(z) gate; the
* out_proj runs over the block below.
* HF Qwen3_5MoeRMSNormGated: out = (o*r)*weight * silu(z) = (o*r)*weight * z/(1+e^-z).
@@ -3796,7 +3957,21 @@ static float *g_hidden_sink = NULL;
#include "qwen36_chain.h" /* COLI_VK_CHAIN: every layer's dense chain on the device */
#endif
-static float *step(Model *m, const int *ids, int S, int pos_base) {
+static void q36_embed_row(Model *m, int id, int pos, float *row) {
+ int D = m->c.hidden;
+ int vrow = (m->vis_map && pos < m->vis_map_len) ? m->vis_map[pos] : -1;
+ if (vrow >= 0 && vrow < m->vis_rows_n)
+ memcpy(row, m->vis_rows + (int64_t)vrow*D, D*sizeof(float));
+ else if (m->embed_h)
+ f16_to_f32_bulk(m->embed_h + (int64_t)id*D, row, D);
+ else
+ memcpy(row, m->embed + (int64_t)id*D, D*sizeof(float));
+}
+
+/* The forward: ids[0..S) at pos_base through every layer, the final norm and
+ * lm_head on the last `nlogits` rows (one for a step, every row of a speculative
+ * verify), each row's logits computed as a decode step computes them. */
+static float *step_ex(Model *m, const int *ids, int S, int pos_base, int nlogits) {
Cfg *c = &m->c; int D = c->hidden;
if (m->resident_mode && m->first_step) m->resident_collecting = 1;
/* Per-layer residual dump (last token) for torch-free cosine debugging.
@@ -3819,13 +3994,7 @@ static float *step(Model *m, const int *ids, int S, int pos_base) {
ids[s], c->vocab - 1);
exit(1);
}
- int vrow = (m->vis_map && pos_base + s < m->vis_map_len) ? m->vis_map[pos_base + s] : -1;
- if (vrow >= 0 && vrow < m->vis_rows_n) /* an image placeholder: the tower's row */
- memcpy(x + (int64_t)s*D, m->vis_rows + (int64_t)vrow*D, D*sizeof(float));
- else if (m->embed_h)
- f16_to_f32_bulk(m->embed_h + (int64_t)ids[s]*D, x + (int64_t)s*D, D);
- else
- memcpy(x + (int64_t)s*D, m->embed + (int64_t)ids[s]*D, D*sizeof(float));
+ q36_embed_row(m, ids[s], pos_base + s, x + (int64_t)s*D);
}
#ifdef COLI_VULKAN
/* COLI_VK_CHAIN: the layers, the final norm and lm_head on the device; x comes
@@ -3837,10 +4006,14 @@ static float *step(Model *m, const int *ids, int S, int pos_base) {
echo = g_echo_k > 0 && g_echo_id && S > 1;
#endif
if (g_hidden_sink) echo = 1; /* the head reads every row */
- chain_logit = falloc(c->vocab);
- if (!q36c_forward(m, x, S, pos_base, lf, echo, chain_logit)) {
+ chain_logit = falloc((int64_t)nlogits * c->vocab);
+ if (!q36c_forward(m, x, S, pos_base, lf, echo, nlogits, chain_logit)) {
free(chain_logit); chain_logit = NULL;
q36c_cpu_step(m, pos_base);
+ /* Earlier chunks may have returned their final rows for Clef or
+ * prompt logprobs. A failed forward replays every row on the CPU. */
+ for (int s = 0; s < S; s++)
+ q36_embed_row(m, ids[s], pos_base + s, x + (int64_t)s*D);
}
}
if (!chain_logit)
@@ -3889,16 +4062,17 @@ static float *step(Model *m, const int *ids, int S, int pos_base) {
if (chain_logit) logit = chain_logit; /* the chain ran the final norm and lm_head */
else
#endif
- {
- rmsnorm_row(last, x + (int64_t)(S-1)*D, m->final_norm, D, c->eps);
- logit = falloc(c->vocab);
- }
+ logit = falloc((int64_t)nlogits * c->vocab);
double _th = tm_now();
#ifdef COLI_VULKAN
if (!chain_logit)
#endif
- if (!qt_lmhead_matmul(logit, last, D, c->vocab))
- matmul_d(logit, last, &m->lm_head, 1, D, c->vocab);
+ for (int r = 0; r < nlogits; r++) {
+ float *lr = logit + (int64_t)r * c->vocab;
+ rmsnorm_row(last, x + (int64_t)(S - nlogits + r)*D, m->final_norm, D, c->eps);
+ if (!qt_lmhead_matmul(lr, last, D, c->vocab))
+ matmul_d(lr, last, &m->lm_head, 1, D, c->vocab);
+ }
if (tm_on()) { tm_add(S, 5, tm_now()-_th); if (S==1) g_tm_dec_tokens++; else g_tm_pre_tokens += S; }
free(x); free(last);
if (lf) fclose(lf);
@@ -3914,6 +4088,10 @@ static float *step(Model *m, const int *ids, int S, int pos_base) {
return logit;
}
+static float *step(Model *m, const int *ids, int S, int pos_base) {
+ return step_ex(m, ids, S, pos_base, 1);
+}
+
static void pilot_realload(Model *m, int layer, int eid) {
LCache *lc = &m->cache[layer]; Cfg *c = &m->c;
pthread_mutex_lock(&g_pilot_mx);
@@ -4212,6 +4390,226 @@ static void ensure_kv(Model *m){
}
}
+/* ---- speculative decoding with prompt lookup (COLI_LOOKUP=1) ----------------
+ * The caller asks for the logits that follow the token it just picked
+ * (q36_spec_step). When the recent tokens repeat an n-gram of the context, the
+ * tokens that followed it there are proposed (spec_draft.h, up to
+ * COLI_LOOKUP_DRAFTS of them), and one forward over the token and its k drafts
+ * (the verify, S = k+1 rows) returns the first row's logits as step() would and
+ * keeps the others. The caller's next picks settle the drafts one at a time: a
+ * pick equal to the next draft is answered with that draft's row and no
+ * forward; the first pick that differs undoes the rows after the ones that
+ * stood. The DeltaNet state and conv rings go back to the copy the verify took
+ * after its last standing row (deltanet() copies them after each of the first
+ * snap_rows rows, row r into slot r; the rollback swaps pointers), kv_len and
+ * the kv_prefix record go back to it, and the attention rows past it are a
+ * stale tail the next forward overwrites.
+ *
+ * Every verify row is computed as a decode step computes it: the CPU kernels
+ * give a row the same bits whatever S is, a Vulkan matrix runs a verify's rows
+ * one at a time (g_q36_rowwise, vk_dense_matmul and the chain), and lm_head
+ * reads each row on its own (step_ex). So the logits the caller sees, and its
+ * picks, greedy or sampled, are those of plain decoding. Drafting stays off
+ * where that does not hold: the CUDA tier (its float order follows residency),
+ * CACHE_ROUTE (it routes by residency) and a qpack container.
+ *
+ * Whether a proposal is drafted is the gate's call (spec_draft.h): the measured
+ * acceptance by draft position and the measured cost of a verify by its rows;
+ * COLI_SPEC_GATE=0 drafts every proposal in full (tests). */
+typedef struct {
+ int lookup, lookup_max; /* COLI_LOOKUP: on; COLI_LOOKUP_DRAFTS: drafts per verify, 1..5 */
+ int force, force_row; /* COLI_LOOKUP_FORCE (tests): the oracle's tokens as the proposal */
+ int *hist, hist_n, hist_cap; /* the tokens fed so far and the one about to be: what lookup searches */
+ SpecGate gate;
+ int ahead_n, ahead_i, ahead_pos; /* a verify's rows at ahead_pos.., ahead_i the next to hand out */
+ int ids[Q36_SPEC_ROWS];
+ float *ahead_logit;
+ uint64_t drafts, accepted, verifies, forwards, tokens;
+ int ended_ahead; /* the generation ended with a verify's rows unconsumed (undone) */
+} Q36Spec;
+static const int *g_q36_oracle; static int g_q36_oracle_n; /* ref.json's full ids (COLI_LOOKUP_FORCE) */
+
+static int q36_spec_force_mode(const char *v, int *row) {
+ *row = 0;
+ if (!v || !*v) return 0;
+ if (!strcmp(v, "accept")) return 'a';
+ if (!strcmp(v, "mixed")) return 'm';
+ if (!strcmp(v, "cycle")) return 'c';
+ if (!strncmp(v, "row", 3) && v[3] >= '1' && v[3] <= '0' + Q36_SPEC_SNAPS && !v[4]) { *row = v[3] - '0'; return 'w'; }
+ fprintf(stderr, "COLI_LOOKUP_FORCE must be accept, mixed, cycle or row1..row%d\n", Q36_SPEC_SNAPS);
+ exit(1);
+}
+
+static void q36_spec_begin(Model *m, Q36Spec *sp, const int *prompt, int np) {
+ memset(sp, 0, sizeof *sp);
+ const char *e = getenv("COLI_LOOKUP");
+ sp->lookup = e && *e == '1';
+ e = getenv("COLI_LOOKUP_DRAFTS");
+ sp->lookup_max = e && *e ? atoi(e) : Q36_SPEC_SNAPS;
+ if (sp->lookup_max < 1 || sp->lookup_max > Q36_SPEC_SNAPS) {
+ fprintf(stderr, "COLI_LOOKUP_DRAFTS must be an integer in 1..%d\n", Q36_SPEC_SNAPS); exit(1);
+ }
+ sp->force = q36_spec_force_mode(getenv("COLI_LOOKUP_FORCE"), &sp->force_row);
+ e = getenv("COLI_SPEC_GATE");
+ spec_gate_init(&sp->gate, e && *e == '0');
+ if (sp->lookup && (qt_ready() || g_cache_route || qq_active() || m->dn_dev)) {
+ static int said;
+ if (!said) {
+ said = 1;
+ fprintf(stderr, "[qwen36] COLI_LOOKUP=1: no drafts under %s (its results depend on what is "
+ "resident, so a verify would not reproduce plain decoding)\n",
+ qt_ready() || m->dn_dev ? "the CUDA tier" : g_cache_route ? "CACHE_ROUTE" : "a qpack container");
+ }
+ sp->lookup = 0;
+ }
+ if (sp->lookup && prompt && np > 0) {
+ sp->hist_cap = np + 256;
+ sp->hist = (int *)malloc((size_t)sp->hist_cap * sizeof(int));
+ if (sp->hist) { memcpy(sp->hist, prompt, (size_t)np * sizeof(int)); sp->hist_n = np; }
+ else { sp->lookup = 0; sp->hist_cap = 0; }
+ }
+}
+
+static void q36_spec_free(Q36Spec *sp) { free(sp->hist); sp->hist = NULL; sp->hist_n = sp->hist_cap = 0; }
+
+/* Copy slots for a verify of up to rows+1 rows, allocated the first time they are
+ * needed. 0 = out of memory (the caller takes a plain step). */
+static int q36_spec_alloc(Model *m, int rows) {
+ Cfg *c = &m->c;
+ if (rows > Q36_SPEC_SNAPS) rows = Q36_SPEC_SNAPS;
+ size_t nr = (size_t)c->dn_vheads * c->dn_kdim * c->dn_vdim, nc = (size_t)c->dn_conv_dim * (c->dn_convk - 1);
+ for (int sl = m->snap_slots; sl < rows; sl++) {
+ if (!m->snap_rec[sl] && !(m->snap_rec[sl] = (float **)calloc((size_t)c->n_layers, sizeof(float *)))) return 0;
+ if (!m->snap_conv[sl] && !(m->snap_conv[sl] = (float **)calloc((size_t)c->n_layers, sizeof(float *)))) return 0;
+ for (int i = 0; i < c->n_layers; i++) {
+ if (c->is_attn[i]) continue;
+ if (!m->snap_rec[sl][i] && !(m->snap_rec[sl][i] = (float *)malloc(nr * sizeof(float)))) return 0;
+ if (!m->snap_conv[sl][i] && !(m->snap_conv[sl][i] = (float *)malloc(nc * sizeof(float)))) return 0;
+ }
+ m->snap_slots = sl + 1;
+ }
+ return 1;
+}
+
+/* Back to the state after the verify's first `keep` rows, `len` positions fed. */
+static void q36_spec_rollback(Model *m, int len, int keep) {
+ Cfg *c = &m->c; int slot = keep - 1;
+#ifdef COLI_VULKAN
+ q36c_rollback(m, slot, len); /* the dense chain's own copies: its device buffers swap too */
+#endif
+ for (int i = 0; i < c->n_layers; i++) {
+ if (c->is_attn[i]) continue;
+ float *t = m->DN_rec[i]; m->DN_rec[i] = m->snap_rec[slot][i]; m->snap_rec[slot][i] = t;
+ t = m->DN_conv[i]; m->DN_conv[i] = m->snap_conv[slot][i]; m->snap_conv[slot][i] = t;
+ }
+ m->kv_len = len;
+ if (m->kvp.len > len) m->kvp.len = len;
+}
+
+static void q36_spec_settle(Model *m, Q36Spec *sp, int keep, int rejected) {
+ if (keep < sp->ahead_n) q36_spec_rollback(m, sp->ahead_pos + keep, keep);
+ spec_gate_result(&sp->gate, SPEC_SRC_LOOKUP, keep - 1 + (rejected ? 1 : 0), keep - 1);
+ free(sp->ahead_logit); sp->ahead_logit = NULL; sp->ahead_n = sp->ahead_i = 0;
+}
+
+/* The oracle's token at draft row j (1-based) of a verify at pos, or a wrong one at
+ * the row the forced mode rejects; -1 past the oracle. */
+static int q36_spec_forced(const Q36Spec *sp, int k, int j, int pos, int V) {
+ int truth = pos + j < g_q36_oracle_n ? g_q36_oracle[pos + j] : -1;
+ if (truth < 0) return -1;
+ int v = (int)sp->verifies, wrong = sp->force == 'm' ? (v & 1) : sp->force == 'w' ? sp->force_row :
+ sp->force == 'c' ? v % (k + 1) + 1 : 0;
+ return j == wrong ? (truth + 1) % V : truth;
+}
+
+static void q36_spec_hist_push(Q36Spec *sp, int tok, int pos) {
+ if (!sp->lookup || pos < 0) return;
+ if (pos >= sp->hist_cap) {
+ int cap = sp->hist_cap ? sp->hist_cap : 256; while (cap <= pos) cap *= 2;
+ int *h = (int *)realloc(sp->hist, (size_t)cap * sizeof(int));
+ if (!h) { sp->lookup = 0; return; }
+ sp->hist = h; sp->hist_cap = cap;
+ }
+ if (pos > sp->hist_n) { sp->lookup = 0; return; } /* a gap: the caller's history is not ours */
+ sp->hist[pos] = tok; sp->hist_n = pos + 1;
+}
+
+/* The logits that follow `tok`, fed at `pos`, exactly as step(m,&tok,1,pos) gives
+ * them; `more` is how many tokens the caller may still want after `tok`. The
+ * returned buffer is the caller's (it may hold more rows past the first vocab). */
+static float *q36_spec_step(Model *m, Q36Spec *sp, int tok, int pos, int more) {
+ Cfg *c = &m->c; int V = c->vocab;
+ sp->tokens++;
+ q36_spec_hist_push(sp, tok, pos);
+ if (sp->ahead_n) {
+ int i = sp->ahead_i;
+ if (tok == sp->ids[i] && pos == sp->ahead_pos + i) {
+ float *logit = falloc(V);
+ memcpy(logit, sp->ahead_logit + (int64_t)(i - 1) * V, (size_t)V * sizeof(float));
+ sp->accepted++;
+ if (++sp->ahead_i == sp->ahead_n) q36_spec_settle(m, sp, sp->ahead_n, 0);
+ return logit;
+ }
+ q36_spec_settle(m, sp, i, 1);
+ }
+ int room = more - 1;
+ if (room > sp->lookup_max) room = sp->lookup_max;
+ if (pos + room >= m->kv_cap) room = m->kv_cap - 1 - pos;
+ int d[Q36_SPEC_ROWS], k = 0;
+ if (sp->lookup && room > 0) {
+ int n = 0;
+ if (sp->force)
+ while (n < room) { int t = q36_spec_forced(sp, room, n + 1, pos, V); if (t < 0) break; d[n++] = t; }
+ else n = spec_lookup(sp->hist, sp->hist_n, 2, 4, room, d);
+ if (n > 0) k = spec_gate_pick(&sp->gate, SPEC_SRC_LOOKUP, n, NULL);
+ if (k > 0 && !q36_spec_alloc(m, k)) k = 0;
+ }
+ if (!k) {
+ sp->forwards++;
+ double t0 = now_s();
+ float *logit = step(m, &tok, 1, pos);
+ spec_gate_forward(&sp->gate, 1, now_s() - t0);
+ return logit;
+ }
+ int S = k + 1;
+ sp->ids[0] = tok; memcpy(sp->ids + 1, d, (size_t)k * sizeof(int));
+ m->snap_rows = k; g_q36_rowwise = 1;
+ double t0 = now_s();
+ float *logit = step_ex(m, sp->ids, S, pos, S);
+ spec_gate_forward(&sp->gate, S, now_s() - t0);
+ m->snap_rows = 0; g_q36_rowwise = 0;
+ sp->forwards++; sp->drafts += (uint64_t)k; sp->verifies++;
+ sp->ahead_n = S; sp->ahead_i = 1; sp->ahead_pos = pos;
+ sp->ahead_logit = falloc((int64_t)k * V);
+ memcpy(sp->ahead_logit, logit + V, (size_t)k * V * sizeof(float));
+ return logit;
+}
+
+/* End of a generation: verify rows the caller never consumed are undone, so the
+ * state is the one plain decoding leaves after the same tokens. */
+static void q36_spec_end(Model *m, Q36Spec *sp) {
+ if (sp->ahead_n) { sp->ended_ahead = 1; q36_spec_settle(m, sp, sp->ahead_i, 0); }
+}
+
+static void q36_spec_report(const Q36Spec *sp, const char *scope) {
+ if (!sp->lookup && !sp->verifies) return;
+ if (!sp->gate.off) {
+ char g[384];
+ spec_gate_describe(&sp->gate, SPEC_SRC_LOOKUP, g, sizeof g);
+ fprintf(stderr, "[qwen36 spec gate] %s: %s\n", scope, g);
+ }
+ fprintf(stderr, "[qwen36 lookup] %s: %.2f tokens/forward (%llu forwards per %llu tokens) | "
+ "acceptance %.1f%% (%llu/%llu drafts in %llu verifies) | gate %s, %llu declined, %llu probes%s\n",
+ scope, sp->forwards ? (double)sp->tokens / sp->forwards : 0.0,
+ (unsigned long long)sp->forwards, (unsigned long long)sp->tokens,
+ sp->drafts ? 100.0 * sp->accepted / sp->drafts : 0.0,
+ (unsigned long long)sp->accepted, (unsigned long long)sp->drafts, (unsigned long long)sp->verifies,
+ sp->gate.off ? "off" : "on", (unsigned long long)sp->gate.declined[SPEC_SRC_LOOKUP],
+ (unsigned long long)sp->gate.probes[SPEC_SRC_LOOKUP], sp->ended_ahead ? " | ended mid-verify" : "");
+}
+
+static Q36Spec g_q36_run_spec; /* generate()'s, reported by main() */
+
static void generate(Model *m, const int *prompt, int np, int n_new, int *out) {
Cfg *c = &m->c;
/* Same ceiling serve_one() enforces. Past max_position_embeddings the RoPE
@@ -4229,6 +4627,8 @@ static void generate(Model *m, const int *prompt, int np, int n_new, int *out) {
for (int i = 0; i < np; i++) out[i] = prompt[i];
float *logit = step(m, prompt, np, 0);
int len = np;
+ Q36Spec *sp = &g_q36_run_spec;
+ q36_spec_begin(m, sp, prompt, np); /* COLI_LOOKUP=1: drafts, every token still the argmax below */
for (int s = 0; s < n_new; s++) {
int best = 0; float bv = logit[0];
for (int i = 1; i < c->vocab; i++) if (logit[i] > bv) { bv = logit[i]; best = i; }
@@ -4242,11 +4642,11 @@ static void generate(Model *m, const int *prompt, int np, int n_new, int *out) {
free(logit); out[len++] = best; break;
}
free(logit); out[len++] = best;
- int one = best;
{ extern double g_tm_step; double _s0 = tm_now();
- logit = step(m, &one, 1, len - 1);
+ logit = q36_spec_step(m, sp, best, len - 1, n_new - 1 - s);
if (tm_on()) g_tm_step += tm_now()-_s0; }
}
+ q36_spec_end(m, sp);
}
static int tf_nll(Model *m, const int *full, int nfull, int np, double *nll_out) {
@@ -4658,6 +5058,7 @@ static void serve_one(Model *m, ServeReq *q){
double t0=now_s();
unsigned char sbuf[16]; int sbn=0;
g_echo_k = 0; g_echo_id = NULL; /* la lettura riguarda il prefill, non la decodifica */
+ Q36Spec spec; q36_spec_begin(m, &spec, ids, np); /* COLI_LOOKUP=1: drafts, every token still sampled from the exact logits */
for(int s=0;s<q->max_tok;s++){
int tk = serve_sample(lo, m->c.vocab, q->temp, q->top_p);
/* La coda si calcola PRIMA della free: dopo, lo non c'e piu. */
@@ -4679,6 +5080,7 @@ static void serve_one(Model *m, ServeReq *q){
* turno che nessuno vuole piu'. */
if(serve_cancel_pending(q->id)){
free(lo); lo=NULL;
+ q36_spec_end(m, &spec); q36_spec_free(&spec); /* the state of the tokens fed, drafts undone */
if(sbn>0) serve_data(q->id,(char*)sbuf,sbn);
printf("ERROR %s CANCELLED\n",q->id); fflush(stdout);
free(ids);
@@ -4688,8 +5090,10 @@ static void serve_one(Model *m, ServeReq *q){
* serve_one() resets the recurrent/KV state for every request, so
* stepping here would only run a full discarded decode pass. */
if(s == q->max_tok - 1) break;
- lo = step(m, &tk, 1, np+s); forwards++;
+ lo = q36_spec_step(m, &spec, tk, np+s, q->max_tok-1-s);
}
+ q36_spec_end(m, &spec);
+ forwards += (int)spec.forwards;
if(sbn>0) serve_data(q->id,(char*)sbuf,sbn); /* flush trailing partial UTF-8 */
free(lo); free(ids);
double dt=now_s()-t0;
@@ -4707,6 +5111,8 @@ static void serve_one(Model *m, ServeReq *q){
printf("DONE %s STAT %d %.3f %.1f %.2f %d %d\n",q->id,gen,
dt>0?gen/dt:0.0,0.0,rss_gb(),np,limited);
fflush(stdout);
+ {char scope[96]; snprintf(scope, sizeof scope, "turn %s", q->id); q36_spec_report(&spec, scope);}
+ q36_spec_free(&spec);
#ifdef COLI_VULKAN
vk_report(); /* stderr: the wire protocol on stdout is untouched */
q36c_report(m);
@@ -4866,10 +5272,20 @@ static int q36_clef_decide(Model *m, const DecideRecord *rec, DecideAnswer *answ
if (m->seen) memset(m->seen, 0, (size_t)c->n_layers * c->n_experts);
if (m->momentum_logits) memset(m->momentum_logits, 0, (size_t)c->n_layers * c->n_experts * sizeof(float));
g_hidden_sink = hidden;
+#ifdef COLI_VULKAN
+ double tb = decide_now_ms();
+#endif
float *lo = step(m, in.ids, in.n, 0);
g_hidden_sink = NULL;
free(lo);
+#ifdef COLI_VULKAN
+ double th = decide_now_ms();
+#endif
ok = clef_head_forward(&g_clef_head, hidden, &in, q36_clef_lm_row, m, logits, err, cap);
+#ifdef COLI_VULKAN
+ /* where a decision's time goes, for the device's measurements (docs/vulkan.md) */
+ fprintf(stderr, "[clef] %d tokens: backbone %.1f ms, head %.1f ms\n", in.n, th - tb, decide_now_ms() - th);
+#endif
}
for (int q = 0; ok && q < in.n_q; q++) {
ok = clef_answer(&in.q[q], logits[q], &answers[q]);
@@ -4915,6 +5331,10 @@ static void clef_serve_one(Model *m, ServeReq *q){
}
}
fprintf(stderr, "[clef] DECIDE %s: %d question(s), %d tokens, %.1f ms\n", q->id, rec.n_questions, tokens, elapsed);
+#ifdef COLI_VULKAN
+ vk_report(); /* stderr: the device's share of the decision */
+ q36c_report(m);
+#endif
decide_answers_free(answers, answers ? rec.n_questions : 0);
free(answers);
decide_record_free(&rec);
@@ -5006,6 +5426,10 @@ static int q36_clef_test_modes(Model *m){
printf("%s\n", out.data ? out.data : "{\"error\":\"out of memory\"}");
fflush(stdout);
free(out.data);
+#ifdef COLI_VULKAN
+ vk_report();
+ q36c_report(m);
+#endif
clef_input_free(&in);
decide_answers_free(answers, answers ? rec.n_questions : 0);
free(answers);
@@ -5131,6 +5555,7 @@ static size_t vk_qw_bytes(const QW *w) {
}
static size_t vk_dense_bytes(Model *m) {
if (!g_vk_dense && !g_vk_chain) return 0;
+ if (coli_vk_dense_device_only()) return 0; /* placed already (q36_dho_start): the free memory the tier reads counts them */
size_t b = vk_qw_bytes(&m->lm_head);
for (int i = 0; i < m->c.n_layers; i++) {
Layer *L = &m->L[i];
@@ -5148,6 +5573,39 @@ static int vk_in_ram(void *ctx, int layer, int e) {
pthread_mutex_unlock(&g_pilot_mx);
return r;
}
+/* After the device, the chain, a qpack container and the CUDA tier are decided, before
+ * the expert tier sizes its budget: the dense matrices go up now, so the tier sees them
+ * placed. The CUDA tier and a qpack container keep the host copies (the dense part
+ * stays theirs or the CPU's). */
+static void q36_dho_start(Model *m) {
+ if (!g_vk_ready) return;
+ size_t bytes = 0; int n = 0;
+ q36_dho_each(m, q36_dho_count, &bytes, &n);
+ if (!coli_vk_dense_host_decide("qwen36", (g_vk_chain || g_vk_dense) && n > 0 && !qt_ready() && !qq_active(), bytes))
+ return;
+ g_q36_dho_model = m;
+ g_vk_dense = 1; /* the steps the chain declines run their matrices on the device too: the CPU has none */
+ bytes = 0; n = 0;
+ q36_dho_each(m, q36_dho_drop, &bytes, &n);
+ if (coli_vk_device_integrated() && coli_vk_imported_bytes())
+ fprintf(stderr, "[VK] qwen36: dense rows shared with the integrated GPU, %.1f MiB read in place; "
+ "the imported pages stay alive as the only weight copy\n", coli_vk_imported_bytes() / 1048576.0);
+ coli_vk_dense_host_placed("qwen36", "the embedding (its rows are gathered on the CPU), the DeltaNet's a/b rows and "
+ "the shared expert's gate vector, norms, the vision tower, any imported pages");
+}
+
+/* The tier's streaming (a big prompt chunk's cold experts on the device): an expert's
+ * bytes as the CPU path would have them, its RAM cache or the disk, held until the tier
+ * has copied them. */
+static int vk_load(void *ctx, int layer, int e, VktExpertSrc *src, void **h) {
+ Model *m = ctx;
+ Slot *s = expert_hold(m, layer, e);
+ if (!s) return 0;
+ if (!xf_mode(m)) slot_ensure_int8(m, s);
+ *src = vk_slot_src(m, s); *h = s;
+ return 1;
+}
+static void vk_release(void *ctx, void *h) { (void)ctx; slot_release((Slot *)h); }
static void vk_tier_start(Model *m, const char *snap, int cap, int expert_is_int4, int expert_mixed) {
Cfg *c = &m->c;
if (!g_vk_ready || c->n_experts == 0 || qq_active()) return;
@@ -5174,7 +5632,8 @@ static void vk_tier_start(Model *m, const char *snap, int cap, int expert_is_int
.max_rows = QWEN_VK_ROWS * c->topk,
.ram_reserve = slot * (size_t)cap * (size_t)c->n_layers,
.dense_bytes = vk_dense_bytes(m),
- .in_ram = vk_in_ram, .ram_ctx = m};
+ .in_ram = vk_in_ram, .ram_ctx = m,
+ .load = vk_load, .release = vk_release, .load_ctx = m};
rt_init("qwen36", c->n_layers, c->n_experts);
rt_drop_row(c->n_layers); /* no MTP row */
const char *up = getenv("COLI_USAGE");
@@ -5323,6 +5782,22 @@ int main(int argc, char **argv) {
/* measured on a Radeon 780M: decode and prefill both faster (docs/vulkan.md) */
if (g_vk_ready && !qq_active())
g_vk_chain = coli_vk_chain_decide("qwen36", vkt_wanted() && m.c.n_experts > 0, COLI_VK_CHAIN_ON);
+ if (g_vk_ready) {
+ /* COLI_VK_IMPORT: 1 the device reads the dense rows in place, 0 it copies them.
+ * Unset: in place for a model without routed experts (Qwen3.8-27B, Clef) on a
+ * device that shares the CPU's RAM, whose trunk is the whole model and would be
+ * held twice (54 GB in f16: past a 61 GB box; measured on a Radeon 780M, a
+ * prompt's GEMM within 4% and a decode GEMV within 9% of the copy's); a copy
+ * elsewhere, as before. */
+ const char *e = getenv("COLI_VK_IMPORT");
+ int want = e && *e ? atoi(e) != 0 : m.c.n_experts == 0 && coli_vk_device_shares_ram();
+ g_vk_import = want && coli_vk_import_alignment() > 0 && coli_vk_import_alignment() <= Q36_PAGE;
+ if (want || (e && *e))
+ fprintf(stderr, "[VK] qwen36: int8 and f16 dense rows %s (%s)\n", g_vk_import ? "read in place, not copied" : "copied",
+ !want ? "COLI_VK_IMPORT=0" : g_vk_import ? (e && *e ? "COLI_VK_IMPORT=1" :
+ "a dense model on a device sharing the CPU's RAM; COLI_VK_IMPORT=0 copies them")
+ : "this device cannot import host memory");
+ }
if (g_vk_chain && !vkc_init()) g_vk_chain = 0;
#endif
if (ref_image && ref_image->t == J_OBJ) {
@@ -5527,6 +6002,7 @@ int main(int argc, char **argv) {
}
#ifdef COLI_VULKAN
+ q36_dho_start(&m); /* COLI_VK_DENSE_HOST: the dense matrices on the device only, before the tier sizes its budget */
vk_tier_start(&m, snap, cap, expert_is_int4, expert_mixed); /* COLI_VULKAN=1: hot routed experts on the device */
if (g_vk_ready && !vkt_ready() && !g_vk_dense) g_vk_dense = coli_vk_dense_decide("qwen36", 0, 1); /* no tier after all */
if (g_vk_chain && qt_ready()) { /* the CUDA tier keeps its priority */
@@ -5590,6 +6066,7 @@ int main(int argc, char **argv) {
}
}
double t = now_s();
+ if (is_ref) { g_q36_oracle = full; g_q36_oracle_n = nfull; } /* COLI_LOOKUP_FORCE */
generate(&m, prompt, np, n_new, out);
double dt = now_s() - t;
@@ -5624,6 +6101,8 @@ int main(int argc, char **argv) {
}
double tot = m.hits + m.miss;
if (g_ttft >= 0) fprintf(stderr, "TTFT: %.2f s (time to first token)\n", g_ttft);
+ q36_spec_report(&g_q36_run_spec, "run");
+ q36_spec_free(&g_q36_run_spec);
tm_report();
qt_stats();
/* qpack dispatch evidence, mirroring what the parity gates assert: which
diff --git a/c/qwen36_chain.h b/c/qwen36_chain.h
index de8327bb..07ddc85b 100644
--- a/c/qwen36_chain.h
+++ b/c/qwen36_chain.h
@@ -34,6 +34,18 @@
* (reset_recurrent: a fill with zeros on the device, pin_restore: an upload).
* Prompt-cache and prefix reuse need nothing more: a reused prefix is rows below the
* watermark and a DeltaNet state that already sits where the next step expects it.
+ * A speculative verify (prompt lookup, q36_spec_step) runs here too: its matrices take
+ * the per-row GEMV (a decode step's bits), lm_head reads each of its rows, and the
+ * DeltaNet state and conv rings are copied on the device after each of its rows but the
+ * last, one slot per row (the recurrent ops split into one dispatch per copy, each
+ * ending on its row: the same bits), so a draft rejected after row r rolls back by
+ * swapping slot r's buffers in (q36c_rollback).
+ *
+ * Past the device's budget (vk_kvsplit.h) each attention layer keeps only a window of
+ * the newest positions on the device; the step's attention runs over that window on
+ * the device and over the older positions on the CPU (from the host's canonical cache)
+ * at once, the two merged through their softmax statistics. Below the budget the
+ * mirrors are whole, as above.
*
* The chain declines (the per-matrix path runs, state synced first) under the CUDA
* expert tier, a qpack container, PILOT prefetch, or geometry outside its shaders
@@ -44,6 +56,7 @@
* the ids do not describe (an image) cannot be rebuilt: that stops the engine.
* COLI_VK_CHAIN_FAULT=n (vk_chain.c) fakes the loss at the n-th frame, for tests. */
#include "vk_chain.h"
+#include "vk_kvsplit.h"
#define Q36C_HOST 0
#define Q36C_DEV 1
@@ -52,11 +65,15 @@
typedef struct {
int ok, failed;
int rows; /* scratch capacity in rows */
- int cap; /* device KV rows (the host's kv_cap) */
+ int cap; /* the host's kv_cap the mirrors follow */
+ int dev_rows; /* device KV rows a layer: cap, or the split's */
+ VkcKvSplit ks; /* the split past the device's budget (ks.on) */
VkcBuf *prm; /* every norm / conv / DeltaNet parameter */
size_t *o_in, *o_post, *o_qn, *o_kn, *o_conv, *o_dn, o_final;
ColiVkTensor **t_ab, **t_sg; /* DeltaNet b|a rows, shared gate row (f32) */
VkcBuf **rec, **ring, **kc, **vc; /* per layer state */
+ VkcBuf **rec_snap[Q36_SPEC_SNAPS], **ring_snap[Q36_SPEC_SNAPS]; /* a verify's copies, slot r after row r */
+ int snap_slots, snap_valid; /* allocated; written by the last verify */
int *kv_valid, *attn_ord, n_attn;
int dn_where, host_zero;
VkcBuf *x, *nrm, *tmp, *q, *k, *v, *ctx, *qkv, *z, *ab, *cv, *dny, *h2, *lg, *gs, *us, *hs, *ds, *sgd, *fin;
@@ -67,11 +84,6 @@ typedef struct {
} Q36Chain;
static int g_vk_chain = 0; /* COLI_VK_CHAIN decided on, and the chain's pipelines are up */
-static int q36c_chunk_rows(void) {
- const char *e = getenv("COLI_VK_CHAIN_ROWS");
- int v = e && *e ? atoi(e) : 512;
- return v < 1 ? 1 : v > 65535 ? 65535 : v;
-}
static void q36c_fatal(const char *what) {
fprintf(stderr, "[VK] qwen36 chain: %s -- stopping (COLI_VK_CHAIN=0 keeps the state on the CPU)\n", what);
@@ -84,7 +96,7 @@ static void q36c_recover(Model *m, int upto) {
Q36Chain *ch = (Q36Chain *)m->vkchain;
Cfg *c = &m->c; int D = c->hidden;
g_vk_chain = 0;
- if (ch) { ch->failed = 1; ch->dn_where = Q36C_HOST; ch->host_zero = 0; }
+ if (ch) { ch->failed = 1; ch->dn_where = Q36C_HOST; ch->host_zero = 0; ch->snap_valid = 0; }
for (int i = 0; i < c->n_layers; i++) {
if (c->is_attn[i]) continue;
memset(m->DN_rec[i], 0, (size_t)c->dn_vheads * c->dn_kdim * c->dn_vdim * sizeof(float));
@@ -100,7 +112,11 @@ static void q36c_recover(Model *m, int upto) {
if (!ids) { fprintf(stderr, "OOM rebuilding the state\n"); exit(1); }
memcpy(ids, m->kvp.fed, (size_t)upto * sizeof(int));
for (int s = 0; s < upto; s++) memcpy(x + (int64_t)s * D, m->embed + (int64_t)ids[s] * D, D * sizeof(float));
+ /* a rebuild, not the verify being run: no copies, the batch's own matmuls */
+ int snap = m->snap_rows, rowwise = g_q36_rowwise;
+ m->snap_rows = 0; g_q36_rowwise = 0;
layers_forward_range(m, x, upto, 0, 0, c->n_layers, 0, NULL);
+ m->snap_rows = snap; g_q36_rowwise = rowwise;
free(ids); free(x);
}
@@ -213,10 +229,27 @@ static Q36Chain *q36c_setup(Model *m) {
return ch;
}
-static int q36c_res(VkcBuf **b, size_t floats, int kind) { return vkc_reserve(b, (floats ? floats : 1) * sizeof(float), kind); }
+/* q36c_res counts instead of reserving while g_q36c_count >= 0 (the chunk's sizing) */
+static long long g_q36c_count = -1;
+static int q36c_res(VkcBuf **b, size_t floats, int kind) {
+ if (g_q36c_count >= 0) { g_q36c_count += (long long)(floats ? floats : 1) * (long long)sizeof(float); return 1; }
+ return vkc_reserve(b, (floats ? floats : 1) * sizeof(float), kind);
+}
+static int q36c_bufs(Q36Chain *ch, Model *m, int rows);
+/* Prompt rows per chunk (vkc_chunk_rows): the chain's scratch a row, counted from the
+ * reservations themselves, and the routed experts' outputs (the tier's rows, the
+ * CPU's contributions, and the host's sum) for it. */
+static int q36c_chunk_rows(Q36Chain *ch, Model *m) {
+ g_q36c_count = 0; q36c_bufs(ch, m, 1); long long b1 = g_q36c_count;
+ g_q36c_count = 0; q36c_bufs(ch, m, 2); long long b2 = g_q36c_count;
+ g_q36c_count = -1;
+ size_t row = (size_t)(b2 - b1) + (size_t)(m->c.n_experts > 0 ? 2 * m->c.topk + 1 : 0) * m->c.hidden * sizeof(float);
+ if (ch->ks.on) row += (long long)m->c.q_heads * (3 * (m->c.head_dim + 2) + m->c.q_head_dim) * sizeof(float);
+ return vkc_chunk_rows("qwen36", row);
+}
-/* scratch for `rows` rows, and the KV mirrors at the host's capacity */
-static int q36c_scratch(Q36Chain *ch, Model *m, int rows) {
+/* the scratch buffers for `rows` rows */
+static int q36c_bufs(Q36Chain *ch, Model *m, int rows) {
Cfg *c = &m->c; int D = c->hidden, H = c->q_heads, KV = c->kv_heads, hd = c->head_dim;
int qo = H * c->q_head_dim, kvo = KV * c->k_head_dim, vd = c->dn_vheads * c->dn_vdim;
int E = c->n_experts > 0 ? c->n_experts : 1, SI = c->shared_inter > 0 ? c->shared_inter : 1;
@@ -233,19 +266,23 @@ static int q36c_scratch(Q36Chain *ch, Model *m, int rows) {
q36c_res(&ch->kvd, (size_t)(ch->n_attn ? ch->n_attn : 1) * 2 * r * kvo, VKC_DOWN) &&
q36c_res(&ch->outd, c->vocab, VKC_DOWN) && q36c_res(&ch->routed, r * D, VKC_UP) &&
q36c_res(&ch->cs, r * (c->rotary_dim > 0 ? c->rotary_dim : 2), VKC_UP);
- if (!ok) return 0;
- if (ch->rows < rows) {
- float *hr = realloc(ch->host_routed, r * D * sizeof(float));
- if (!hr) return 0;
- ch->host_routed = hr; ch->rows = rows;
- }
- if (ch->cap != m->kv_cap) { /* the host cache grew: mirror it again */
+ return ok;
+}
+/* Plan and allocate the KV mirror before sizing a prompt chunk. */
+static int q36c_mirror(Q36Chain *ch, Model *m) {
+ Cfg *c = &m->c; int KV = c->kv_heads;
+ if (ch->cap != m->kv_cap) { /* the host cache grew: mirror it again (whole, or split past the budget) */
+ size_t row = (size_t)KV * c->k_head_dim * 2 * sizeof(float);
+ int plan = ch->n_attn ? vkc_kv_plan(&ch->ks, "qwen36", ch->n_attn, row, m->kv_cap, 1, 0,
+ (size_t)ch->n_attn * ch->dev_rows * row) : 1;
+ if (!plan) { ch->cap = 0; return 0; }
+ ch->dev_rows = ch->ks.on ? ch->ks.rows : m->kv_cap;
for (int i = 0; i < c->n_layers; i++) {
if (!c->is_attn[i]) continue;
vkc_free(ch->kc[i]); vkc_free(ch->vc[i]); ch->kc[i] = ch->vc[i] = NULL;
ch->kv_valid[i] = 0;
- ch->kc[i] = vkc_buf((size_t)KV * m->kv_cap * c->k_head_dim * sizeof(float), VKC_DEV);
- ch->vc[i] = vkc_buf((size_t)KV * m->kv_cap * c->k_head_dim * sizeof(float), VKC_DEV);
+ ch->kc[i] = vkc_buf((size_t)KV * ch->dev_rows * c->k_head_dim * sizeof(float), VKC_DEV);
+ ch->vc[i] = vkc_buf((size_t)KV * ch->dev_rows * c->k_head_dim * sizeof(float), VKC_DEV);
if (!ch->kc[i] || !ch->vc[i]) { ch->cap = 0; return 0; }
}
ch->cap = m->kv_cap;
@@ -253,6 +290,20 @@ static int q36c_scratch(Q36Chain *ch, Model *m, int rows) {
return 1;
}
+/* scratch for `rows` rows, and the KV mirrors at the host's capacity */
+static int q36c_scratch(Q36Chain *ch, Model *m, int rows) {
+ Cfg *c = &m->c; int D = c->hidden;
+ size_t r = (size_t)rows;
+ if (!q36c_bufs(ch, m, rows)) return 0;
+ if (ch->rows < rows) {
+ float *hr = realloc(ch->host_routed, r * D * sizeof(float));
+ if (!hr) return 0;
+ ch->host_routed = hr; ch->rows = rows;
+ }
+ if (ch->ks.on && !vkc_kv_parts(&ch->ks, r * c->q_heads * (c->head_dim + 2))) return 0;
+ return 1;
+}
+
/* ---- state between the host and the device ----------------------------------- */
/* The host's DeltaNet state brought up to date (before anything reads it there). */
static void q36c_sync_host(Model *m) {
@@ -271,18 +322,57 @@ static void q36c_sync_host(Model *m) {
static void q36c_host_wrote(Model *m, int zero) {
Q36Chain *ch = (Q36Chain *)m->vkchain;
if (!ch || !ch->ok) return;
- ch->dn_where = Q36C_HOST; ch->host_zero = zero;
+ ch->dn_where = Q36C_HOST; ch->host_zero = zero; ch->snap_valid = 0;
}
/* A CPU step from pos_base: the host state current before it, the device's stale after. */
static void q36c_cpu_step(Model *m, int pos_base) {
Q36Chain *ch = (Q36Chain *)m->vkchain;
if (!ch || !ch->ok) return;
q36c_sync_host(m);
- ch->dn_where = Q36C_HOST; ch->host_zero = 0;
- for (int i = 0; i < m->c.n_layers; i++) if (ch->kv_valid[i] > pos_base) ch->kv_valid[i] = pos_base;
+ ch->dn_where = Q36C_HOST; ch->host_zero = 0; ch->snap_valid = 0;
+ for (int i = 0; i < m->c.n_layers; i++) {
+ if (ch->kv_valid[i] > pos_base) ch->kv_valid[i] = pos_base;
+ if (m->c.is_attn[i]) vkc_kv_lower(&ch->ks, ch->attn_ord[i], pos_base);
+ }
+}
+/* A rejected draft: the state after the verify's row `slot` (its first slot+1 rows) is
+ * the device's copy in that slot; `len` positions stand, and the KV watermark comes
+ * down to them (the rows past them are the rejected drafts'). */
+static void q36c_rollback(Model *m, int slot, int len) {
+ Q36Chain *ch = (Q36Chain *)m->vkchain;
+ if (!ch || !ch->ok) return;
+ for (int i = 0; i < m->c.n_layers; i++) {
+ if (ch->kv_valid[i] > len) ch->kv_valid[i] = len;
+ if (m->c.is_attn[i]) vkc_kv_lower(&ch->ks, ch->attn_ord[i], len);
+ }
+ if (slot < 0 || slot >= ch->snap_valid || ch->dn_where != Q36C_DEV) { ch->snap_valid = 0; return; }
+ for (int i = 0; i < m->c.n_layers; i++) {
+ if (m->c.is_attn[i]) continue;
+ VkcBuf *t = ch->rec[i]; ch->rec[i] = ch->rec_snap[slot][i]; ch->rec_snap[slot][i] = t;
+ t = ch->ring[i]; ch->ring[i] = ch->ring_snap[slot][i]; ch->ring_snap[slot][i] = t;
+ }
+ ch->snap_valid = 0;
+}
+/* The device's copy slots for a verify that copies `rows` rows, allocated the first
+ * time a verify that deep runs. */
+static int q36c_spec_slots(Q36Chain *ch, Model *m, int rows) {
+ Cfg *c = &m->c; int L = c->n_layers;
+ if (rows > Q36_SPEC_SNAPS) rows = Q36_SPEC_SNAPS;
+ size_t nr = (size_t)c->dn_vheads * c->dn_kdim * c->dn_vdim, nc = (size_t)c->dn_conv_dim * (c->dn_convk - 1);
+ for (int sl = ch->snap_slots; sl < rows; sl++) {
+ if (!ch->rec_snap[sl] && !(ch->rec_snap[sl] = (VkcBuf **)calloc((size_t)L, sizeof(VkcBuf *)))) return 0;
+ if (!ch->ring_snap[sl] && !(ch->ring_snap[sl] = (VkcBuf **)calloc((size_t)L, sizeof(VkcBuf *)))) return 0;
+ for (int i = 0; i < L; i++) {
+ if (c->is_attn[i]) continue;
+ if (!ch->rec_snap[sl][i] && !(ch->rec_snap[sl][i] = vkc_buf(nr * sizeof(float), VKC_DEV))) return 0;
+ if (!ch->ring_snap[sl][i] && !(ch->ring_snap[sl][i] = vkc_buf(nc * sizeof(float), VKC_DEV))) return 0;
+ }
+ ch->snap_slots = sl + 1;
+ }
+ return 1;
}
/* Record the uploads that make the device state the host's, as the step at pos_base needs it. */
-static int q36c_push_state(Q36Chain *ch, Model *m, int pos_base) {
+static int q36c_push_state(Q36Chain *ch, Model *m, int pos_base, int n_rows) {
Cfg *c = &m->c; int ok = 1;
if (ch->dn_where == Q36C_HOST) {
size_t nr = (size_t)c->dn_vheads * c->dn_kdim * c->dn_vdim, nc = (size_t)c->dn_conv_dim * (c->dn_convk - 1);
@@ -296,10 +386,18 @@ static int q36c_push_state(Q36Chain *ch, Model *m, int pos_base) {
}
int kvd = c->k_head_dim, KV = c->kv_heads;
for (int i = 0; i < c->n_layers && ok; i++) {
+ if (c->is_attn[i] && ch->ks.on) { /* the split: the window placed, its rows below pos_base uploaded */
+ VkcKvPart pt[2] = {{KV, kvd, m->K[i], (size_t)m->max_t * kvd, ch->kc[i], 0},
+ {KV, kvd, m->V[i], (size_t)m->max_t * kvd, ch->vc[i], 0}};
+ vkc_kv_place(&ch->ks, ch->attn_ord[i], pos_base, n_rows);
+ ok = vkc_kv_push(&ch->ks, ch->attn_ord[i], pt, 2, pos_base);
+ ch->kv_valid[i] = pos_base;
+ continue;
+ }
if (!c->is_attn[i] || ch->kv_valid[i] >= pos_base) continue;
int t0 = ch->kv_valid[i], n = pos_base - t0;
for (int h = 0; h < KV && ok; h++) {
- size_t src = ((size_t)h * m->max_t + t0) * kvd, dst = ((size_t)h * ch->cap + t0) * kvd;
+ size_t src = ((size_t)h * m->max_t + t0) * kvd, dst = ((size_t)h * ch->dev_rows + t0) * kvd;
ok = vkc_write(ch->kc[i], dst, m->K[i] + src, (size_t)n * kvd * sizeof(float)) &&
vkc_write(ch->vc[i], dst, m->V[i] + src, (size_t)n * kvd * sizeof(float));
}
@@ -327,32 +425,53 @@ static int q36c_attention(Q36Chain *ch, Model *m, Layer *l, int i, int n, int pb
ok = vkc_norm(ch->k, ch->prm, ch->k, &p); }
if (ok && half) { VkcRope p = {n * KV, KV, 0, kvo, kvd, half, 0, 2 * half}; ok = vkc_rope(ch->k, ch->cs, &p); }
if (!ok) return 0;
+ size_t ko = (size_t)ch->attn_ord[i] * 2 * ch->rows * kvo;
+ if (ch->ks.on) { /* the split: the rows into their window slots, the attention in two parts */
+ VkcKvPart pk = {KV, kvd, m->K[i], (size_t)m->max_t * kvd, ch->kc[i], 0}, pv = {KV, kvd, m->V[i], (size_t)m->max_t * kvd, ch->vc[i], 0};
+ VkcKvGqa a = {&ch->ks, ch->attn_ord[i], ch->q, ch->kc[i], ch->vc[i], ch->q, NULL, NULL, ch->ctx, n, H, KV, hd, kvd, pb, 0, 0, 0, 0,
+ qo, qdim, gate_dim > 0, hd, qo, qdim, H * hd, 0, 0, 1.f / sqrtf((float)hd),
+ m->K[i], m->V[i], (size_t)m->max_t * kvd, (size_t)kvd, (size_t)m->max_t * kvd, (size_t)kvd};
+ return vkc_kv_store(&ch->ks, ch->attn_ord[i], &pk, ch->k, 0, (size_t)kvo, (size_t)kvd, pb, n) &&
+ vkc_kv_store(&ch->ks, ch->attn_ord[i], &pv, ch->v, 0, (size_t)kvo, (size_t)kvd, pb, n) &&
+ vkc_copy(ch->kvd, ko, ch->k, 0, (size_t)n * kvo) && vkc_copy(ch->kvd, ko + (size_t)ch->rows * kvo, ch->v, 0, (size_t)n * kvo) &&
+ vkc_kv_gqa(&a) && vkc_matmul(vk_qw_tensor(&l->o), ch->ctx, 0, ch->tmp, 0, n);
+ }
/* the new rows into the device cache, and to the host's */
VkcRegion *rg = malloc(sizeof *rg * (size_t)n * KV);
if (!rg) return 0;
for (int s = 0; s < n; s++) for (int h = 0; h < KV; h++)
- rg[s * KV + h] = (VkcRegion){((size_t)h * ch->cap + pb + s) * kvd, (size_t)s * kvo + (size_t)h * kvd, (size_t)kvd};
- size_t ko = (size_t)ch->attn_ord[i] * 2 * ch->rows * kvo;
+ rg[s * KV + h] = (VkcRegion){((size_t)h * ch->dev_rows + pb + s) * kvd, (size_t)s * kvo + (size_t)h * kvd, (size_t)kvd};
ok = vkc_copy_regions(ch->kc[i], ch->k, rg, n * KV) && vkc_copy_regions(ch->vc[i], ch->v, rg, n * KV) &&
vkc_copy(ch->kvd, ko, ch->k, 0, (size_t)n * kvo) && vkc_copy(ch->kvd, ko + (size_t)ch->rows * kvo, ch->v, 0, (size_t)n * kvo);
free(rg);
- VkcAttn a = {n, H, KV, hd, pb, ch->cap, 0, qo, qdim, hd, qo, qdim, gate_dim > 0, 0, H * hd, 0, 0,
+ VkcAttn a = {n, H, KV, hd, pb, ch->dev_rows, 0, qo, qdim, hd, qo, qdim, gate_dim > 0, 0, H * hd, 0, 0,
1.f / sqrtf((float)hd), 0, 0};
return ok && vkc_attn(ch->q, ch->kc[i], ch->vc[i], ch->ctx, ch->q, NULL, &a) &&
vkc_matmul(vk_qw_tensor(&l->o), ch->ctx, 0, ch->tmp, 0, n);
}
-static int q36c_deltanet(Q36Chain *ch, Model *m, Layer *l, int i, int n) {
+/* ns: the chunk's rows a verify copies the state after (rows 0..ns-1, slots c0..). One
+ * copy rides on the one dispatch as the shaders pass its row; more split the
+ * convolution and the recurrence into one dispatch per copy, each ending on its row
+ * (the last takes the chunk's remaining rows): the state crosses between them through
+ * memory in f32, so every row gets the single dispatch's bits. */
+static int q36c_deltanet(Q36Chain *ch, Model *m, Layer *l, int i, int n, int c0, int ns) {
Cfg *c = &m->c;
int vh = c->dn_vheads, CD = c->dn_conv_dim, vd = vh * c->dn_vdim;
int ok = vkc_matmul(vk_qw_tensor(&l->dn_qkv), ch->nrm, 0, ch->qkv, 0, n) &&
vkc_matmul(vk_qw_tensor(&l->dn_z), ch->nrm, 0, ch->z, 0, n) &&
vkc_matmul(ch->t_ab[i], ch->nrm, 0, ch->ab, 0, n);
- VkcDnConv cp = {n, CD, c->dn_convk, 0, CD, 0, CD, -1, 0, (int)ch->o_conv[i], 0, 0};
- VkcDnRec rp = {n, vh, c->dn_kheads, c->dn_vdim, c->dn_kheads * c->dn_kdim, 0, CD, 0, 2 * vh, vh, 2 * vh,
- 0, vd, 0, vd, -1, 0, c->eps, 1.f / sqrtf((float)c->dn_kdim), 0, 0, (int)ch->o_dn[i]};
- return ok && vkc_dnconv(ch->qkv, ch->prm, ch->ring[i], ch->cv, NULL, &cp) &&
- vkc_dnrec(c->dn_kdim, ch->cv, ch->ab, ch->z, ch->rec[i], ch->prm, ch->dny, NULL, &rp) &&
- vkc_matmul(vk_qw_tensor(&l->dn_out), ch->dny, 0, ch->tmp, 0, n);
+ int segs = ns > 1 ? ns : 1;
+ for (int r = 0; r < segs && ok; r++) {
+ int s0 = ns > 1 ? r : 0, len = ns > 1 && r < ns - 1 ? 1 : n - s0, snap_row = ns ? 0 : -1;
+ VkcBuf *rs = ns ? ch->rec_snap[c0 + r][i] : NULL, *cs = ns ? ch->ring_snap[c0 + r][i] : NULL;
+ VkcDnConv cp = {len, CD, c->dn_convk, s0 * CD, CD, s0 * CD, CD, snap_row, 0, (int)ch->o_conv[i], 0, 0};
+ VkcDnRec rp = {len, vh, c->dn_kheads, c->dn_vdim, c->dn_kheads * c->dn_kdim, s0 * CD, CD, s0 * 2 * vh, 2 * vh,
+ vh + s0 * 2 * vh, 2 * vh, s0 * vd, vd, s0 * vd, vd, snap_row, 0, c->eps,
+ 1.f / sqrtf((float)c->dn_kdim), 0, 0, (int)ch->o_dn[i]};
+ ok = vkc_dnconv(ch->qkv, ch->prm, ch->ring[i], ch->cv, cs, &cp) &&
+ vkc_dnrec(c->dn_kdim, ch->cv, ch->ab, ch->z, ch->rec[i], ch->prm, ch->dny, rs, &rp);
+ }
+ return ok && vkc_matmul(vk_qw_tensor(&l->dn_out), ch->dny, 0, ch->tmp, 0, n);
}
static int q36c_shared(Q36Chain *ch, Model *m, Layer *l, int i, int n) {
Cfg *c = &m->c; int ok = 1, SI = c->shared_inter;
@@ -376,25 +495,39 @@ static int q36c_combine(Q36Chain *ch, Model *m, int i, int n) {
/* Every layer for S rows from host rows xh, the last row's logits into `logit`; xh gets
* the final rows back when want_x. 0 = not taken (nothing on the device changed). */
-static int q36c_forward(Model *m, float *xh, int S, int pos_base, FILE *lf, int want_x, float *logit) {
+static int q36c_forward(Model *m, float *xh, int S, int pos_base, FILE *lf, int want_x, int nlogits, float *logit) {
if (!g_vk_chain || qt_ready() || qq_active() || g_pilot) return 0;
- if (g_vk_chain == COLI_VK_CHAIN_PREFILL && S <= 2) return 0; /* prompts only: decode on the CPU */
+ /* prompts only: decode and verifies on the CPU */
+ if (g_vk_chain == COLI_VK_CHAIN_PREFILL && (S <= 2 || g_q36_rowwise)) return 0;
Q36Chain *ch = q36c_setup(m);
if (!ch || ch->failed) return 0;
Cfg *c = &m->c; int D = c->hidden, L = c->n_layers, E = c->n_experts;
- int CH = q36c_chunk_rows(), rows = S < CH ? S : CH;
+ int mirror_ok = q36c_mirror(ch, m);
+ int CH = mirror_ok ? q36c_chunk_rows(ch, m) : 1, rows = S < CH ? S : CH;
+ if (ch->ks.on && rows > ch->ks.chunk) rows = ch->ks.chunk; /* a step's rows fit the split's window */
int kvo = c->kv_heads * c->k_head_dim, half = c->rotary_dim / 2;
- if (!q36c_scratch(ch, m, rows) || (lf && !q36c_res(&ch->lfd, (size_t)L * 3 * D, VKC_DOWN)) ||
- (want_x && !q36c_res(&ch->xd, (size_t)rows * D, VKC_DOWN))) {
+ if (m->snap_rows > 0 && !q36c_spec_slots(ch, m, m->snap_rows)) {
+ fprintf(stderr, "[VK] qwen36 chain: device memory for a verify's %d copies refused; this verify on the CPU\n",
+ m->snap_rows);
+ return 0;
+ }
+ if (!mirror_ok || !q36c_scratch(ch, m, rows) || (lf && !q36c_res(&ch->lfd, (size_t)L * 3 * D, VKC_DOWN)) ||
+ (want_x && !q36c_res(&ch->xd, (size_t)rows * D, VKC_DOWN)) ||
+ (nlogits > 1 && (!q36c_res(&ch->fin, (size_t)nlogits * D, VKC_DEV) ||
+ !q36c_res(&ch->outd, (size_t)nlogits * c->vocab, VKC_DOWN)))) {
fprintf(stderr, "[VK] qwen36 chain: device memory for %d rows refused; per-matrix path\n", rows);
ch->failed = 1;
return 0;
}
- vkc_gemm_rows(-1);
+ vkc_gemm_rows(g_q36_rowwise ? 0 : -1); /* a verify's rows get a decode step's bits */
+ int snapped = 0;
for (int c0 = 0; c0 < S; c0 += rows) {
int n = S - c0 < rows ? S - c0 : rows, pb = pos_base + c0;
+ int ns = m->snap_rows - c0 < n ? m->snap_rows - c0 : n; /* the rows of this chunk a verify copies */
+ if (ns < 0) ns = 0;
+ if (ns) snapped = c0 + ns;
if (!vkc_begin() || !vkc_write(ch->x, 0, xh + (size_t)c0 * D, (size_t)n * D * sizeof(float)) ||
- !q36c_push_state(ch, m, pb)) goto lost;
+ !q36c_push_state(ch, m, pb, n)) goto lost;
if (half) { /* the CPU's own angles, cosines and sines (rope_head_partial / _mrope) */
float *cs = (float *)vkc_ptr(ch->cs);
for (int s = 0; s < n; s++) {
@@ -416,7 +549,7 @@ static int q36c_forward(Model *m, float *xh, int S, int pos_base, FILE *lf, int
if (ok && lf) ok = vkc_copy(ch->lfd, ((size_t)(i - 1) * 3 + 2) * D, ch->x, (size_t)(n - 1) * D, D);
}
ok = ok && q36c_norm(ch->x, 0, ch->prm, ch->o_in[i], ch->nrm, 0, n, D, c->eps);
- ok = ok && (c->is_attn[i] ? q36c_attention(ch, m, l, i, n, pb) : q36c_deltanet(ch, m, l, i, n));
+ ok = ok && (c->is_attn[i] ? q36c_attention(ch, m, l, i, n, pb) : q36c_deltanet(ch, m, l, i, n, c0, ns));
if (ok && lf) ok = vkc_copy(ch->lfd, (size_t)i * 3 * D, ch->tmp, (size_t)(n - 1) * D, D);
VkcEw add = {VKC_EW_ADD, n * D, D, 1, 0, 1, 0, 0, 0, 0, 0, 1.f};
ok = ok && vkc_ew(ch->x, ch->x, ch->tmp, NULL, NULL, &add);
@@ -426,7 +559,11 @@ static int q36c_forward(Model *m, float *xh, int S, int pos_base, FILE *lf, int
ok = vkc_matmul(vk_qw_tensor(&l->gate), ch->h2, 0, ch->lg, 0, n) &&
vkc_copy(ch->lgd, 0, ch->lg, 0, (size_t)n * E) && vkc_copy(ch->h2d, 0, ch->h2, 0, (size_t)n * D);
double t0 = tm_now();
- ok = ok && vkc_submit(1); /* A1 */
+ /* A1; while it runs, the tier loads the experts this layer will likely
+ * stream (a big prompt chunk only) */
+ ok = ok && vkc_submit(0);
+ if (ok) vkt_stream_prefetch(i, n);
+ ok = ok && vkc_finish();
ch->frames++; ch->wait_ms += tm_now() - t0;
if (!ok) break;
if (c->is_attn[i]) { /* the rows into the host's cache */
@@ -446,15 +583,27 @@ static int q36c_forward(Model *m, float *xh, int S, int pos_base, FILE *lf, int
double t2 = tm_now();
tm_add(n, 2, t2 - t1); ch->host_ms += t2 - t1;
ok = ok && vkc_begin();
- } else if (ok) ok = q36c_shared(ch, m, l, i, n); /* a dense model: no host step */
+ } else if (ok) {
+ ok = q36c_shared(ch, m, l, i, n);
+ /* A dense backbone has no expert readback to end the frame.
+ * Keep each layer in its own submission: a full 27B prompt in
+ * one submission can exceed the driver's watchdog. The next
+ * frame's barrier orders its residual add after this MLP. */
+ if (ok && i + 1 < L) ok = vkc_submit(0) && vkc_begin();
+ }
pending = 1;
}
if (ok) ok = q36c_combine(ch, m, L - 1, n);
if (ok && lf) ok = vkc_copy(ch->lfd, ((size_t)(L - 1) * 3 + 2) * D, ch->x, (size_t)(n - 1) * D, D);
if (ok && want_x) ok = vkc_copy(ch->xd, 0, ch->x, 0, (size_t)n * D);
int last = c0 + n == S;
- if (ok && last) ok = q36c_norm(ch->x, (size_t)(n - 1) * D, ch->prm, ch->o_final, ch->fin, 0, 1, D, c->eps) &&
- vkc_matmul(vk_qw_tensor(&m->lm_head), ch->fin, 0, ch->outd, 0, 1);
+ /* the final norm and lm_head on this chunk's share of the last nlogits rows */
+ int lo0 = S - nlogits > c0 ? S - nlogits - c0 : 0, lo_n = n - lo0;
+ if (ok && lo_n > 0) {
+ int dst = c0 + lo0 - (S - nlogits);
+ ok = q36c_norm(ch->x, (size_t)lo0 * D, ch->prm, ch->o_final, ch->fin, (size_t)dst * D, lo_n, D, c->eps) &&
+ vkc_matmul(vk_qw_tensor(&m->lm_head), ch->fin, (size_t)dst * D, ch->outd, (size_t)dst * c->vocab, lo_n);
+ }
double t0 = tm_now();
ok = ok && vkc_submit(1);
ch->frames++; ch->wait_ms += tm_now() - t0;
@@ -469,14 +618,17 @@ static int q36c_forward(Model *m, float *xh, int S, int pos_base, FILE *lf, int
}
}
if (want_x) memcpy(xh + (size_t)c0 * D, vkc_ptr(ch->xd), (size_t)n * D * sizeof(float));
- for (int i = 0; i < L; i++) if (c->is_attn[i]) ch->kv_valid[i] = pb + n;
+ for (int i = 0; i < L; i++) if (c->is_attn[i]) { ch->kv_valid[i] = pb + n; vkc_kv_done(&ch->ks, ch->attn_ord[i], pb + n); }
ch->dn_where = Q36C_DEV; ch->host_zero = 0;
- if (last) memcpy(logit, vkc_ptr(ch->outd), (size_t)c->vocab * sizeof(float));
+ if (last) memcpy(logit, vkc_ptr(ch->outd), (size_t)nlogits * c->vocab * sizeof(float));
}
if (lf) fwrite(vkc_ptr(ch->lfd), sizeof(float), (size_t)L * 3 * D, lf);
+ ch->snap_valid = snapped;
+ vkc_gemm_rows(-1);
ch->forwards++;
return 1;
lost: /* a frame failed: the device is gone (or would not take a command); the CPU takes over */
+ vkc_gemm_rows(-1);
if (!vkc_lost()) { vkc_finish(); coli_vk_mark_lost(); }
q36c_recover(m, pos_base);
return 0;
@@ -489,5 +641,6 @@ static void q36c_report(Model *m) {
fprintf(stderr, "[VK] qwen36 chain: %llu forwards, %llu frames (%llu ops, %llu matmuls, %llu tiled GEMM), "
"%.1f ms waiting for the device, %.1f ms of routed experts on the host, %.1f MiB on the device\n",
ch->forwards, st.frames, st.ops, st.matmuls, st.gemms, st.wait_ms, ch->host_ms, st.dev_bytes / 1048576.0);
+ vkc_kv_report(&ch->ks);
vkc_prof_print();
}
diff --git a/c/qwen38.c b/c/qwen38.c
index 92baab5c..29229130 100644
--- a/c/qwen38.c
+++ b/c/qwen38.c
@@ -971,34 +971,94 @@ static void q38_validate_ids(const Cfg *c, const int *ids, int count,
}
}
-/* A decode run's speculation: drafting when the MTP head is attached, in the
- * mode Q38_MTP_FORCE names (tests: reject, accept, mixed; qwen38_core.h). */
-static Q38Spec q38_spec_begin(const Model *m) {
+/* A decode run's speculation (qwen38_core.h, "speculative decoding"): the MTP
+ * head's drafts when it is attached, Q38_MTP_DRAFTS of them per verify (1..3,
+ * default 2, the depth measured fastest on a Ryzen 8700GE, CPU and Vulkan chain
+ * alike, docs/speculative.md; 1 is the verify before it; 0 or auto lets the gate
+ * pick); prompt-lookup drafts under
+ * COLI_LOOKUP=1, up to COLI_LOOKUP_DRAFTS (1..5, default 5); the forced modes
+ * the tests use (Q38_MTP_FORCE, COLI_LOOKUP_FORCE), and COLI_SPEC_GATE=0,
+ * which drafts every proposal in full. */
+static int q38_spec_force_mode(const char *name, const char *value, int max_row, int lookup, int *row) {
+ *row = 0;
+ if (!value || !*value) return 0;
+ if (!lookup && !strcmp(value, "reject")) return 'r';
+ if (!strcmp(value, "accept")) return 'a';
+ if (!strcmp(value, "mixed")) return 'm';
+ if (!strcmp(value, "cycle")) return 'c';
+ if (!strncmp(value, "row", 3) && value[3] >= '1' && value[3] <= '0' + max_row && !value[4]) {
+ *row = value[3] - '0'; return 'w';
+ }
+ fprintf(stderr, "%s must be %saccept, mixed, cycle or row1..row%d\n", name, lookup ? "" : "reject, ", max_row);
+ exit(1);
+}
+static Q38Spec q38_spec_begin(Model *m, const int *prompt, int np) {
Q38Spec sp; memset(&sp, 0, sizeof sp);
sp.on = m->mtp;
- const char *force = getenv("Q38_MTP_FORCE");
- if (force && *force) {
- if (!strcmp(force, "reject")) sp.force = 'r';
- else if (!strcmp(force, "accept")) sp.force = 'a';
- else if (!strcmp(force, "mixed")) sp.force = 'm';
- else { fprintf(stderr, "Q38_MTP_FORCE must be reject, accept or mixed\n"); exit(1); }
+ sp.force = q38_spec_force_mode("Q38_MTP_FORCE", getenv("Q38_MTP_FORCE"), 3, 0, &sp.force_row);
+ const char *depth = getenv("Q38_MTP_DRAFTS");
+ sp.depth = 2;
+ if (depth && *depth) {
+ if (!strcmp(depth, "auto") || !strcmp(depth, "0")) sp.depth = 0;
+ else if (depth[0] >= '1' && depth[0] <= '3' && !depth[1]) sp.depth = depth[0] - '0';
+ else { fprintf(stderr, "Q38_MTP_DRAFTS must be 1, 2, 3, or 0 (auto: the gate picks)\n"); exit(1); }
+ }
+ sp.lookup = q38_env_bool("COLI_LOOKUP", 0);
+ sp.lookup_max = q38_env_positive_int("COLI_LOOKUP_DRAFTS", Q38_SPEC_ROWS - 1, Q38_SPEC_ROWS - 1);
+ sp.lk_force = q38_spec_force_mode("COLI_LOOKUP_FORCE", getenv("COLI_LOOKUP_FORCE"), Q38_SPEC_ROWS - 1, 1,
+ &sp.lk_force_row);
+ spec_gate_init(&sp.gate, !q38_env_bool("COLI_SPEC_GATE", 1));
+ if (sp.lookup && prompt && np > 0) {
+ sp.hist_cap = np + 256;
+ sp.hist = (int *)malloc((size_t)sp.hist_cap * sizeof(int));
+ if (sp.hist) { memcpy(sp.hist, prompt, (size_t)np * sizeof(int)); sp.hist_n = np; }
+ else { sp.lookup = 0; sp.hist_cap = 0; }
}
return sp;
}
/* Acceptance and tokens per forward, colibri.c's speculation line: a run's
* at the end, a serve turn's after its DONE (stderr, never the wire). Tokens
- * are the decode tokens fed after the prompt, forwards the ones that fed them. */
+ * are the decode tokens fed after the prompt, forwards the ones that fed them;
+ * the MTP head's acceptance follows by draft position when it drafts deeper
+ * than one, and prompt lookup has a line of its own. */
static void q38_spec_report(const Model *m, const Q38Spec *sp, const char *scope) {
- if (!m->mtp) return;
- fprintf(stderr, "[qwen38 MTP] %s: %.2f tokens/forward (%llu forwards per %llu tokens) | "
- "acceptance %.1f%% (%llu/%llu drafts) | wiring %c%s%s\n", scope,
- sp->forwards ? (double)sp->tokens / sp->forwards : 0.0,
- (unsigned long long)sp->forwards, (unsigned long long)sp->tokens,
- sp->drafts ? 100.0 * sp->accepted / sp->drafts : 0.0,
- (unsigned long long)sp->accepted, (unsigned long long)sp->drafts, m->mtp_wiring,
- sp->force ? ", Q38_MTP_FORCE=" : "",
- sp->force == 'r' ? "reject" : sp->force == 'a' ? "accept" : sp->force == 'm' ? "mixed" : "");
+ if (m->mtp) {
+ char depth[160] = "";
+ if (sp->depth != 1) {
+ int at = snprintf(depth, sizeof depth, " | %s, by position:", sp->depth ? "fixed depth" : "depth by the gate");
+ for (int j = 0; j < 3 && at < (int)sizeof depth; j++)
+ at += snprintf(depth + at, sizeof depth - (size_t)at, " %d %.1f%% (%llu/%llu)", j + 1,
+ sp->depth_prop[j] ? 100.0 * sp->depth_hit[j] / sp->depth_prop[j] : 0.0,
+ (unsigned long long)sp->depth_hit[j], (unsigned long long)sp->depth_prop[j]);
+ }
+ fprintf(stderr, "[qwen38 MTP] %s: %.2f tokens/forward (%llu forwards per %llu tokens) | "
+ "acceptance %.1f%% (%llu/%llu drafts) | wiring %c%s%s%s%s\n", scope,
+ sp->forwards ? (double)sp->tokens / sp->forwards : 0.0,
+ (unsigned long long)sp->forwards, (unsigned long long)sp->tokens,
+ sp->drafts ? 100.0 * sp->accepted / sp->drafts : 0.0,
+ (unsigned long long)sp->accepted, (unsigned long long)sp->drafts, m->mtp_wiring,
+ sp->force ? ", Q38_MTP_FORCE=" : "",
+ sp->force == 'r' ? "reject" : sp->force == 'a' ? "accept" : sp->force == 'm' ? "mixed" :
+ sp->force == 'c' ? "cycle" : sp->force == 'w' ? "row" : "", depth,
+ sp->ended_ahead ? " | ended mid-verify" : "");
+ }
+ if (!sp->gate.off && ((m->mtp && sp->depth == 0) || sp->lookup)) {
+ char g[384];
+ spec_gate_describe(&sp->gate, m->mtp && sp->depth == 0 ? SPEC_SRC_MTP : SPEC_SRC_LOOKUP, g, sizeof g);
+ fprintf(stderr, "[qwen38 spec gate] %s: %s\n", scope, g);
+ }
+ if (sp->lookup || sp->lk_verifies)
+ fprintf(stderr, "[qwen38 lookup] %s: %.2f tokens/forward (%llu forwards per %llu tokens) | "
+ "acceptance %.1f%% (%llu/%llu drafts in %llu verifies) | gate %s, %llu declined, %llu probes%s\n",
+ scope, sp->forwards ? (double)sp->tokens / sp->forwards : 0.0,
+ (unsigned long long)sp->forwards, (unsigned long long)sp->tokens,
+ sp->lk_drafts ? 100.0 * sp->lk_accepted / sp->lk_drafts : 0.0,
+ (unsigned long long)sp->lk_accepted, (unsigned long long)sp->lk_drafts,
+ (unsigned long long)sp->lk_verifies, sp->gate.off ? "off" : "on",
+ (unsigned long long)sp->gate.declined[SPEC_SRC_LOOKUP],
+ (unsigned long long)sp->gate.probes[SPEC_SRC_LOOKUP],
+ sp->ended_ahead && !m->mtp ? " | ended mid-verify" : "");
}
static Q38Spec g_q38_run_spec; /* generate()'s, reported by main() */
@@ -1017,7 +1077,7 @@ static void generate(Model *m, const int *prompt, int np, int n_new, int *out) {
float *logit = step(m, prompt, np, 0);
q38_tm_snapshot_prefill(m); /* COLI_TIMERS: decode bank starts here */
Q38Spec *sp = &g_q38_run_spec;
- *sp = q38_spec_begin(m); /* Q38_MTP=1: the head drafts, every token still the argmax below */
+ *sp = q38_spec_begin(m, prompt, np); /* Q38_MTP=1, COLI_LOOKUP=1: drafts, every token still the argmax below */
int len = np;
for (int s = 0; s < n_new; s++) {
int best = 0; float bv = logit[0];
@@ -1344,7 +1404,7 @@ typedef struct {
/* the MTP head where the prompt left it: its settled rows and the
* streams still waiting for their next token (Model.mtp_*) */
float *mtp_pend;
- int mtp_len, mtp_pend_n, mtp_pend_tok;
+ int mtp_len, mtp_pend_n, mtp_pend_tok[Q38_SPEC_ROWS];
} Q38PrefixCache;
static Q38PrefixCache g_q38_prefix;
@@ -1360,7 +1420,7 @@ static int q38_prefix_geometry(const Model *m,size_t *rec_cells,
size_t *conv_cells,size_t *ple_cells);
static ColiPinPool g_q38_pins;
typedef struct { float **rec, **conv, *ple; int64_t ple_hist[2]; int ple_len; int n_layers;
- float *mtp_pend; int mtp_len, mtp_pend_n, mtp_pend_tok; } Q38PinState;
+ float *mtp_pend; int mtp_len, mtp_pend_n, mtp_pend_tok[Q38_SPEC_ROWS]; } Q38PinState;
/* The MTP head's place in the sequence travels with every snapshot of the
* recurrent state: its KV rows below mtp_len stay valid wherever the
@@ -1370,13 +1430,14 @@ typedef struct { float **rec, **conv, *ple; int64_t ple_hist[2]; int ple_len; in
static void q38_mtp_state_copy(Model *m, float *pend, int *len, int *pend_n,
int *pend_tok, int to_state){
if(!m->mtp||!pend) return;
- size_t bytes=(size_t)2*m->c.hc_width*sizeof(float);
+ size_t bytes=(size_t)Q38_SPEC_ROWS*m->c.hc_width*sizeof(float);
+ size_t toks=(size_t)Q38_SPEC_ROWS*sizeof(int);
if(to_state){
memcpy(pend,m->mtp_pend,bytes);
- *len=m->mtp_len; *pend_n=m->mtp_pend_n; *pend_tok=m->mtp_pend_tok;
+ *len=m->mtp_len; *pend_n=m->mtp_pend_n; memcpy(pend_tok,m->mtp_pend_tok,toks);
} else {
memcpy(m->mtp_pend,pend,bytes);
- m->mtp_len=*len; m->mtp_pend_n=*pend_n; m->mtp_pend_tok=*pend_tok;
+ m->mtp_len=*len; m->mtp_pend_n=*pend_n; memcpy(m->mtp_pend_tok,pend_tok,toks);
}
}
@@ -1419,7 +1480,7 @@ static int q38_pin_state_copy(Model *m, Q38PinState **slot, int to_state){
if (!st->ple){ q38_pin_state_free(st); return 0; }
}
if (m->mtp){
- st->mtp_pend = (float*)malloc((size_t)2 * c->hc_width * sizeof(float));
+ st->mtp_pend = (float*)malloc((size_t)Q38_SPEC_ROWS * c->hc_width * sizeof(float));
if (!st->mtp_pend){ q38_pin_state_free(st); return 0; }
}
*slot = st;
@@ -1445,7 +1506,7 @@ static int q38_pin_state_copy(Model *m, Q38PinState **slot, int to_state){
memcpy(m->ple_history, st->ple_hist, sizeof(st->ple_hist));
}
}
- q38_mtp_state_copy(m, st->mtp_pend, &st->mtp_len, &st->mtp_pend_n, &st->mtp_pend_tok, to_state);
+ q38_mtp_state_copy(m, st->mtp_pend, &st->mtp_len, &st->mtp_pend_n, st->mtp_pend_tok, to_state);
#ifdef COLI_VULKAN
if (!to_state) q38c_host_wrote(m, 0); /* up to the device before the next chain step */
#endif
@@ -1503,7 +1564,7 @@ static int q38_prefix_cache_layout(Model *m){
g_q38_prefix.ple_cells=ple;
}
if(m->mtp&&!g_q38_prefix.mtp_pend){
- g_q38_prefix.mtp_pend=(float*)malloc((size_t)2*m->c.hc_width*sizeof(float));
+ g_q38_prefix.mtp_pend=(float*)malloc((size_t)Q38_SPEC_ROWS*m->c.hc_width*sizeof(float));
if(!g_q38_prefix.mtp_pend){q38_prefix_cache_dispose(&g_q38_prefix);return 0;}
}
if(g_q38_prefix.dn_rec&&g_q38_prefix.dn_conv&&g_q38_prefix.logits&&
@@ -1574,7 +1635,7 @@ static void q38_prefix_copy_state(Model *m,int to_cache){
}
}
q38_mtp_state_copy(m,g_q38_prefix.mtp_pend,&g_q38_prefix.mtp_len,&g_q38_prefix.mtp_pend_n,
- &g_q38_prefix.mtp_pend_tok,to_cache);
+ g_q38_prefix.mtp_pend_tok,to_cache);
}
static int q38_prefix_cache_save(Model *m,const int *ids,int len,const float *logits,
@@ -1806,7 +1867,7 @@ static int serve_one(Model *m, ServeReq *q){
int eos_ids[4];int n_eos=serve_eos_ids(eos_ids,4,m->c.eos_id,m->c.vocab);
/* Q38_MTP=1: the head drafts and a verify forward checks it; every token
* is still sampled from the exact logits below, greedy or not. */
- Q38Spec spec=q38_spec_begin(m);
+ Q38Spec spec=q38_spec_begin(m,ids,np);
double first_token_at=0.0,last_token_at=0.0;
unsigned char sbuf[16]; int sbn=0;
for(int s=0;s<q->max_tok;s++){
@@ -1883,6 +1944,7 @@ static int serve_one(Model *m, ServeReq *q){
serve_hits(m);
q38_tm_report_bank(&timers,"request");
{char scope[96];snprintf(scope,sizeof scope,"turn %s",q->id);q38_spec_report(m,&spec,scope);}
+ q38_spec_free(&spec);
return input_eof?-1:0;
}
@@ -1930,6 +1992,22 @@ static void q38_expert_report(Model *m, int cap); /* below, beside the expert
static void q38_vk_tier_start(Model *m, int cap); /* below: the Vulkan routed-expert tier */
#endif
+#ifdef COLI_VULKAN
+/* After the device and the chain are decided, before the expert tier sizes its budget:
+ * the trunk goes up now, so the tier sees it placed. */
+static void q38_dho_start(Model *m) {
+ if(!g_vk_ready)return;
+ size_t bytes=0; int n=0;
+ q38_dho_each(m,q38_dho_count,&bytes,&n);
+ if(!coli_vk_dense_host_decide("qwen38",(g_vk_chain||g_vk_dense)&&!qt_ready(),bytes))return;
+ g_q38_dho_model=m;
+ g_vk_dense=1; /* the steps the chain declines run their matrices on the device too: the CPU has none */
+ bytes=0; n=0;
+ q38_dho_each(m,q38_dho_drop,&bytes,&n);
+ coli_vk_dense_host_placed("qwen38","the embedding (its rows are gathered on the CPU), the vision tower, norms");
+}
+#endif
+
#ifndef QWEN38_TEST_SERVE
int main(int argc, char **argv) {
/* Physical-core team sizing, as colibri/inkling/kimi_k3/olmoe/deepseek-v41
@@ -2062,6 +2140,7 @@ int main(int argc, char **argv) {
if(g_vk_ready&&!qt_ready())
g_vk_chain=coli_vk_chain_decide("qwen38",vkt_wanted()&&m.c.experts>0,COLI_VK_CHAIN_OFF);
if(g_vk_chain&&!vkc_init())g_vk_chain=0;
+ q38_dho_start(&m); /* COLI_VK_DENSE_HOST: the trunk on the device only, before the tier sizes its budget */
#endif
if(is_ref)ref_logits=read_reference_logits(ref_root,m.c.vocab);
g_capture_last_logit=ref_logits!=NULL||getenv("DUMP")!=NULL;
@@ -2178,6 +2257,7 @@ int main(int argc, char **argv) {
double tot = m.hits + m.miss;
if (g_ttft >= 0) fprintf(stderr, "TTFT: %.2f s (time to first token)\n", g_ttft);
q38_spec_report(&m, &g_q38_run_spec, "run");
+ q38_spec_free(&g_q38_run_spec);
if (g_q38_mtp_dump) { fclose(g_q38_mtp_dump); g_q38_mtp_dump = NULL; }
tm_report(&m);
fprintf(stderr, "\nPEAK RSS: %.2f GB\n", rss_gb());
@@ -2427,6 +2507,7 @@ static void q38_vk_dense_add(const Q38Weight *w,size_t *bytes){
static size_t q38_vk_dense_bytes(Model *m){
size_t b=0;
if(!g_vk_dense&&!g_vk_chain)return 0;
+ if(coli_vk_dense_device_only())return 0; /* placed already (q38_dho_start): the free memory the tier reads counts them */
q38_vk_dense_add(&m->lm_head,&b);
const GatedResidual *f=&m->final_gr;
q38_vk_dense_add(&f->down,&b);q38_vk_dense_add(&f->up,&b);q38_vk_dense_add(&f->inject,&b);
@@ -2476,6 +2557,22 @@ static int q38_vk_in_ram(void *ctx,int layer,int e){
int si=lc->by_expert?lc->by_expert[e]:-1;
return si>=0&&si<lc->n&&lc->slots[si].eid==e;
}
+/* The tier's streaming (a big prompt chunk's cold experts on the device): an expert's
+ * bytes as the CPU path has them, through the layer's RAM cache (a slot stays valid
+ * until the next load); several at once through the parallel reads. */
+static int q38_vk_load(void *ctx,int layer,int e,VktExpertSrc *src,void **h){
+ Model *m=(Model*)ctx;Slot *s=q38_expert_get(m,layer,e);
+ if(!s)return 0;
+ *src=q38_vk_src(s);*h=s;return 1;
+}
+static void q38_vk_release(void *ctx,void *h){(void)ctx;(void)h;}
+static int q38_vk_load_batch(void *ctx,int layer,const int *e,int n,VktExpertSrc *srcs,void **h){
+ Model *m=(Model*)ctx;Slot *sl[64];
+ if(n>m->cache[layer].cap)n=m->cache[layer].cap;
+ if(n<2||n>64||!q38_expert_get_batch(m,layer,e,n,sl))return 0;
+ for(int i=0;i<n;i++){srcs[i]=q38_vk_src(sl[i]);h[i]=sl[i];}
+ return n;
+}
static void q38_vk_tier_start(Model *m,int cap){
if(!g_vk_ready||qt_ready()){
if(g_vk_ready&&qt_ready())
@@ -2505,7 +2602,8 @@ static void q38_vk_tier_start(Model *m,int cap){
.max_rows=q38_moe_prefill_rows(c,q38_prefill_batch_rows())*c->topk,
.ram_reserve=ram_expert*(size_t)cap*(size_t)c->layers,
.dense_bytes=q38_vk_dense_bytes(m),
- .in_ram=q38_vk_in_ram,.ram_ctx=m};
+ .in_ram=q38_vk_in_ram,.ram_ctx=m,
+ .load=q38_vk_load,.release=q38_vk_release,.load_ctx=m,.load_batch=q38_vk_load_batch};
if(vc.max_rows<64)vc.max_rows=64;
uint32_t **heat=rt_counts_all();
atexit(coli_vk_shutdown); /* before vkt_init, which makes the expert batch's pipelines and can still refuse (no room): the device goes at exit either way, after the tier's teardown */
diff --git a/c/qwen38_chain.h b/c/qwen38_chain.h
index 25296aa9..8d9783d1 100644
--- a/c/qwen38_chain.h
+++ b/c/qwen38_chain.h
@@ -14,11 +14,15 @@
* when a step completes the block, and the indexer's top-k selection per query
* row on the device (chain_qsa.comp), feeding the attention's selection list;
* - the MTP head stays on the CPU (its experts are FP8 beside the int4 sidecar,
- * and it reads two rows per draft): the chain hands it every row's four streams
- * when the model has one; a verify forward (S = 2) runs here row-wise (GEMV for
- * every matrix, so its rows get a decode step's bits) and snapshots the DeltaNet
- * state, the conv rings and the PLE ring after its first row on the device, so a
- * rejected draft rolls back by swapping device buffers (q38c_rollback).
+ * and it reads a few rows per draft): the chain hands it every row's four streams
+ * when the model has one; a verify forward (S = 2 to Q38_SPEC_ROWS: the token and
+ * its MTP or prompt-lookup drafts) runs here row-wise (GEMV for every matrix, so its
+ * rows get a decode step's bits) and snapshots the DeltaNet state, the conv rings
+ * and the PLE ring after each of its rows but the last on the device, one slot per
+ * row, so a draft rejected after row r rolls back by swapping slot r's device
+ * buffers in (q38c_rollback). One snapshot is written by the shaders as they pass
+ * its row; more split the recurrent ops into one dispatch per snapshot, each ending
+ * on its row (q38c_deltanet, q38c_ple), which gives the same bits.
* Per layer the host gets the MoE input rows and the router logits, and the K/V and
* index-key rows of an attention layer; it sends the routed sum back. The shared
* expert runs on the device (frame A2) while the host computes the routed experts.
@@ -29,8 +33,18 @@
* brought back before the host reads them (prefix cache, pins) and pushed up after
* the host writes them (reset, restore). The CUDA tier keeps its priority: with it
* on, the chain stays off. A device lost while it holds that state: the engine rebuilds
- * it on the CPU from the prefix record and continues there (q38c_recover). */
+ * it on the CPU from the prefix record and continues there (q38c_recover).
+ *
+ * Past the device's budget (vk_kvsplit.h) each attention layer's K/V keep only some
+ * blocks on the device: a window of the newest, and the blocks QSA's selections read
+ * most (pinned by their counts). The selection list goes to both parts: the device
+ * attends over its listed positions, the CPU over the rest from the host's cache, and
+ * the two merge through their softmax statistics. The index keys and the pooled block
+ * keys stay whole on the device: every row scores every pooled key every step, and an
+ * index-key row is read once, to pool its block (ID floats a position beside K/V's
+ * 2*KVH*D). */
#include "vk_chain.h"
+#include "vk_kvsplit.h"
#define Q38C_HOST 0
#define Q38C_DEV 1
@@ -42,26 +56,26 @@ typedef struct {
size_t *o_an, *o_mn, *o_qn, *o_kn, *o_iqn, *o_ikn, *o_conv, *o_dn;
size_t o_fn, o_ple, o_pconv;
ColiVkTensor **t_sg;
- VkcBuf **rec, **ring, **rec_snap, **ring_snap, **kc, **vc, **ik, **pk;
- VkcBuf *ple_ring; /* [2][W][SL]: current at ple_cur, the verify's snapshot at the other */
- int ple_cur;
+ VkcBuf **rec, **ring, **kc, **vc, **ik, **pk;
+ VkcBuf **rec_snap[Q38_SPEC_SNAPS], **ring_snap[Q38_SPEC_SNAPS]; /* a verify's copies, slot r after row r */
+ int snap_slots; /* slots allocated (1 at setup, more when a deeper verify first runs) */
+ VkcBuf *ple_ring; /* [ple_nslots][W][SL]: current at ple_slot[0], the copy after row r at ple_slot[1+r] */
+ int ple_slot[Q38_SPEC_ROWS], ple_nslots;
int *kv_valid, *pk_valid, *attn_ord, n_attn;
+ int dev_rows; /* device K/V rows a layer: cap, or the split's */
+ VkcKvSplit ks; /* the K/V split past the device's budget (ks.on) */
int dn_where, host_zero, snap_valid;
VkcBuf *hyper, *hn, *low, *mix, *mixed, *inj_a, *inj_m, *blk, *q, *k, *v, *ip, *ctx, *qsc, *sel;
VkcBuf *qkv, *z, *ab, *cv, *dny, *lg, *gs, *us, *hs, *ds, *sgd, *fin;
VkcBuf *emb, *keys, *val, *gated, *normv;
VkcBuf *mixd, *lgd, *kvd, *outd, *hypd, *find, *routed, *cs, *pcs;
+ VkcBuf *gseg, *nseg; /* a deep verify's PLE rows past its first copy (q38c_ple) */
float *host_routed, *host_emb;
unsigned long long forwards;
double host_ms;
} Q38Chain;
static int g_vk_chain = 0;
-static int q38c_chunk_rows(void) {
- const char *e = getenv("COLI_VK_CHAIN_ROWS");
- int v = e && *e ? atoi(e) : 512;
- return v < 1 ? 1 : v > 65535 ? 65535 : v;
-}
static void q38c_fatal(const char *what) {
fprintf(stderr, "[VK] qwen38 chain: %s -- stopping (COLI_VK_CHAIN=0 keeps the state on the CPU)\n", what);
exit(1);
@@ -91,8 +105,9 @@ static Q38Chain *q38c_setup(Model *m) {
#define Q38C_ARR(f, t) if (!(ch->f = (t *)calloc((size_t)L, sizeof(t)))) return NULL
Q38C_ARR(o_an, size_t); Q38C_ARR(o_mn, size_t); Q38C_ARR(o_qn, size_t); Q38C_ARR(o_kn, size_t);
Q38C_ARR(o_iqn, size_t); Q38C_ARR(o_ikn, size_t); Q38C_ARR(o_conv, size_t); Q38C_ARR(o_dn, size_t);
- Q38C_ARR(t_sg, ColiVkTensor *); Q38C_ARR(rec, VkcBuf *); Q38C_ARR(ring, VkcBuf *); Q38C_ARR(rec_snap, VkcBuf *);
- Q38C_ARR(ring_snap, VkcBuf *); Q38C_ARR(kc, VkcBuf *); Q38C_ARR(vc, VkcBuf *); Q38C_ARR(ik, VkcBuf *); Q38C_ARR(pk, VkcBuf *);
+ Q38C_ARR(t_sg, ColiVkTensor *); Q38C_ARR(rec, VkcBuf *); Q38C_ARR(ring, VkcBuf *); Q38C_ARR(rec_snap[0], VkcBuf *);
+ Q38C_ARR(ring_snap[0], VkcBuf *); Q38C_ARR(kc, VkcBuf *); Q38C_ARR(vc, VkcBuf *); Q38C_ARR(ik, VkcBuf *); Q38C_ARR(pk, VkcBuf *);
+ ch->snap_slots = 1;
Q38C_ARR(kv_valid, int); Q38C_ARR(pk_valid, int); Q38C_ARR(attn_ord, int);
#undef Q38C_ARR
int VH = c->dn_vheads, CD = c->dn_conv_dim, CK = c->dn_convk;
@@ -150,7 +165,7 @@ static Q38Chain *q38c_setup(Model *m) {
else if (ok) {
ok = q38c_t(&l->dn_qkv) && q38c_t(&l->dn_z) && q38c_t(&l->dn_b) && q38c_t(&l->dn_a) && q38c_t(&l->dn_out) &&
(ch->rec[i] = vkc_buf(nr * 4, VKC_DEV)) && (ch->ring[i] = vkc_buf(nc * 4, VKC_DEV)) &&
- (ch->rec_snap[i] = vkc_buf(nr * 4, VKC_DEV)) && (ch->ring_snap[i] = vkc_buf(nc * 4, VKC_DEV));
+ (ch->rec_snap[0][i] = vkc_buf(nr * 4, VKC_DEV)) && (ch->ring_snap[0][i] = vkc_buf(nc * 4, VKC_DEV));
}
}
ok = ok && q38c_t(&m->final_gr.down) && q38c_t(&m->final_gr.up) && q38c_t(&m->lm_head);
@@ -158,6 +173,7 @@ static Q38Chain *q38c_setup(Model *m) {
Layer *pl = &m->L[c->ple_layer];
ok = q38c_t(&pl->ple_key) && q38c_t(&pl->ple_value) &&
(ch->ple_ring = vkc_buf((size_t)2 * W * (c->ple_convk - 1) * c->ngram_size * 4, VKC_DEV));
+ ch->ple_nslots = 2; ch->ple_slot[0] = 0; ch->ple_slot[1] = 1;
}
if (!ok) { fprintf(stderr, "[VK] qwen38 chain: a matrix did not reach the device; per-matrix path\n"); return NULL; }
ch->dn_where = Q38C_HOST;
@@ -166,19 +182,24 @@ static Q38Chain *q38c_setup(Model *m) {
return ch;
}
-static int q38c_res(VkcBuf **b, size_t floats, int kind) { return vkc_reserve(b, (floats ? floats : 1) * sizeof(float), kind); }
+/* q38c_res counts instead of reserving while g_q38c_count >= 0 (the chunk's sizing) */
+static long long g_q38c_count = -1;
+static int q38c_res(VkcBuf **b, size_t floats, int kind) {
+ if (g_q38c_count >= 0) { g_q38c_count += (long long)(floats ? floats : 1) * (long long)sizeof(float); return 1; }
+ return vkc_reserve(b, (floats ? floats : 1) * sizeof(float), kind);
+}
+static int q38c_bufs(Q38Chain *ch, Model *m, int rows);
static int q38c_selrow(const Cfg *c) { return 1 + c->idx_budget + c->idx_ratio - 1; }
static size_t q38c_kv_stride(Q38Chain *ch, const Cfg *c) { /* per attention layer in kvd: K, V, IK rows */
return (size_t)ch->rows * (2 * (size_t)c->kv_heads * c->head_dim + c->idx_dim);
}
-static int q38c_scratch(Q38Chain *ch, Model *m, int rows) {
+/* the scratch buffers for `rows` rows (kvd's stride is ch->rows') */
+static int q38c_bufs(Q38Chain *ch, Model *m, int rows) {
Cfg *c = &m->c; int H = c->hidden, W = c->hc_width, C = c->hc_count, R = c->hc_rank;
int QH = c->q_heads, D = c->head_dim, kvo = c->kv_heads * D, ipw = (c->idx_qheads + 1) * c->idx_dim;
int V = c->dn_vheads * c->dn_vdim, E = c->experts, SI = c->shared_inter;
int Ep = c->ngram_heads * c->ngram_head_dim; if (Ep < 1) Ep = 1;
size_t r = (size_t)rows;
- int need_rows = ch->rows < rows;
- if (need_rows) ch->rows = rows; /* kvd's stride follows */
int nbmax = m->kv_cap / (c->idx_ratio > 0 ? c->idx_ratio : 1) + 1;
int ok = q38c_res(&ch->hyper, r * W, VKC_DEV) && q38c_res(&ch->hn, r * W, VKC_DEV) && q38c_res(&ch->low, r * R, VKC_DEV) &&
q38c_res(&ch->mix, r * W, VKC_DEV) && q38c_res(&ch->mixed, r * H, VKC_DEV) && q38c_res(&ch->inj_a, r * C, VKC_DEV) &&
@@ -197,22 +218,43 @@ static int q38c_scratch(Q38Chain *ch, Model *m, int rows) {
q38c_res(&ch->outd, 2 * (size_t)c->vocab, VKC_DOWN) && q38c_res(&ch->routed, r * H, VKC_UP) &&
q38c_res(&ch->emb, r * Ep, VKC_UP) && q38c_res(&ch->cs, r * (c->rotary_dim > 0 ? c->rotary_dim : 2), VKC_UP) &&
q38c_res(&ch->pcs, (size_t)nbmax * (c->rotary_dim > 0 ? c->rotary_dim : 2), VKC_UP);
- if (!ok) return 0;
- if (need_rows) {
- float *hr = realloc(ch->host_routed, r * H * sizeof(float)), *he;
- if (!hr) return 0;
- ch->host_routed = hr;
- he = realloc(ch->host_emb, r * Ep * sizeof(float));
- if (!he) return 0;
- ch->host_emb = he;
- }
- if (ch->cap != m->kv_cap) {
+ return ok;
+}
+/* Prompt rows per chunk (vkc_chunk_rows): the chain's scratch a row, counted from the
+ * reservations themselves (at the cache's capacity: QSA's scores grow with it), and the
+ * routed experts' outputs (the tier's rows and the host's sum) for it. */
+static int q38c_chunk_rows(Q38Chain *ch, Model *m) {
+ int keep = ch->rows;
+ ch->rows = 1; g_q38c_count = 0; q38c_bufs(ch, m, 1); long long b1 = g_q38c_count;
+ ch->rows = 2; g_q38c_count = 0; q38c_bufs(ch, m, 2); long long b2 = g_q38c_count;
+ g_q38c_count = -1; ch->rows = keep;
+ const Cfg *c = &m->c;
+ size_t row = (size_t)(b2 - b1) + (size_t)(c->topk + 1) * c->hidden * sizeof(float);
+ if (ch->ks.on) row += ((long long)m->c.q_heads * (5 * m->c.head_dim + 6) + q38c_selrow(&m->c)) * sizeof(float);
+ /* q38_moe_prefill keeps CPU input, gate, up and routed output buffers even
+ * when the tier takes every expert: all must fit beside the device scratch. */
+ size_t assignment = 2 * ((size_t)c->hidden + c->inter) * sizeof(float)
+ + sizeof(Q38RouteAssignment) + 4 * sizeof(int) + sizeof(uint8_t) + sizeof(float *);
+ row += (size_t)c->topk * assignment
+ + ((size_t)c->experts + 3 * (size_t)c->shared_inter + c->hidden + 1) * sizeof(float);
+ return vkc_chunk_rows("qwen38", row);
+}
+/* Plan and allocate the KV mirror before sizing a prompt chunk. */
+static int q38c_mirror(Q38Chain *ch, Model *m) {
+ Cfg *c = &m->c; int kvo = c->kv_heads * c->head_dim;
+ int nbmax = m->kv_cap / (c->idx_ratio > 0 ? c->idx_ratio : 1) + 1;
+ if (ch->cap != m->kv_cap) { /* the host cache is new: mirror it again (K/V whole, or split past the budget) */
+ size_t row = (size_t)kvo * 2 * 4;
+ int plan = ch->n_attn ? vkc_kv_plan(&ch->ks, "qwen38", ch->n_attn, row, m->kv_cap, 1, 1,
+ (size_t)ch->n_attn * ch->dev_rows * row) : 1;
+ if (!plan) { ch->cap = 0; return 0; }
+ ch->dev_rows = ch->ks.on ? ch->ks.rows : m->kv_cap;
for (int i = 0; i < c->layers; i++) {
if (!c->is_attn[i]) continue;
vkc_free(ch->kc[i]); vkc_free(ch->vc[i]); vkc_free(ch->ik[i]); vkc_free(ch->pk[i]);
ch->kv_valid[i] = ch->pk_valid[i] = 0;
- ch->kc[i] = vkc_buf((size_t)kvo * m->kv_cap * 4, VKC_DEV);
- ch->vc[i] = vkc_buf((size_t)kvo * m->kv_cap * 4, VKC_DEV);
+ ch->kc[i] = vkc_buf((size_t)kvo * ch->dev_rows * 4, VKC_DEV);
+ ch->vc[i] = vkc_buf((size_t)kvo * ch->dev_rows * 4, VKC_DEV);
ch->ik[i] = vkc_buf((size_t)c->idx_dim * m->kv_cap * 4, VKC_DEV);
ch->pk[i] = vkc_buf((size_t)c->idx_dim * nbmax * 4, VKC_DEV);
if (!ch->kc[i] || !ch->vc[i] || !ch->ik[i] || !ch->pk[i]) { ch->cap = 0; return 0; }
@@ -222,6 +264,25 @@ static int q38c_scratch(Q38Chain *ch, Model *m, int rows) {
return 1;
}
+static int q38c_scratch(Q38Chain *ch, Model *m, int rows) {
+ Cfg *c = &m->c; int H = c->hidden;
+ int Ep = c->ngram_heads * c->ngram_head_dim; if (Ep < 1) Ep = 1;
+ size_t r = (size_t)rows;
+ int need_rows = ch->rows < rows;
+ if (need_rows) ch->rows = rows; /* kvd's stride follows */
+ if (!q38c_bufs(ch, m, rows)) return 0;
+ if (need_rows) {
+ float *hr = realloc(ch->host_routed, r * H * sizeof(float)), *he;
+ if (!hr) return 0;
+ ch->host_routed = hr;
+ he = realloc(ch->host_emb, r * Ep * sizeof(float));
+ if (!he) return 0;
+ ch->host_emb = he;
+ }
+ if (ch->ks.on && !vkc_kv_parts(&ch->ks, r * c->q_heads * (c->head_dim + 2))) return 0;
+ return 1;
+}
+
/* ---- state between the host and the device ------------------------------------ */
static size_t q38c_ple_cells(const Cfg *c) { return (size_t)c->hc_width * (c->ple_convk - 1) * c->ngram_size; }
static void q38_layer_forward(Model *m,int i,float *hyper,const int *ids,int S,int pos_base,
@@ -259,11 +320,11 @@ static void q38c_recover(Model *m, int upto) {
for (int b = 1; b < C; b++) memcpy(e + (int64_t)b * H, e, (size_t)H * sizeof(float));
}
/* the step being run keeps its prefetched n-gram rows and its snapshot request */
- float *pref = m->ple_pref; int pref_rows = m->ple_pref_rows, snap = m->snap_after, rowwise = g_q38_rowwise;
- m->ple_pref = NULL; m->ple_pref_rows = 0; m->snap_after = 0; g_q38_rowwise = 0;
+ float *pref = m->ple_pref; int pref_rows = m->ple_pref_rows, snap = m->snap_rows, rowwise = g_q38_rowwise;
+ m->ple_pref = NULL; m->ple_pref_rows = 0; m->snap_rows = 0; g_q38_rowwise = 0;
for (int i = 0; i < c->layers; i++) q38_layer_forward(m, i, hyper, ids, upto, 0, mixed, inject, block);
free(m->ple_pref);
- m->ple_pref = pref; m->ple_pref_rows = pref_rows; m->snap_after = snap; g_q38_rowwise = rowwise;
+ m->ple_pref = pref; m->ple_pref_rows = pref_rows; m->snap_rows = snap; g_q38_rowwise = rowwise;
free(ids); free(hyper); free(mixed); free(inject); free(block);
}
static void q38c_sync_host(Model *m) {
@@ -278,7 +339,7 @@ static void q38c_sync_host(Model *m) {
}
}
if (ch->ple_ring && m->PLE_conv_state &&
- !vkc_read(ch->ple_ring, (size_t)ch->ple_cur * q38c_ple_cells(c), m->PLE_conv_state, q38c_ple_cells(c) * 4)) {
+ !vkc_read(ch->ple_ring, (size_t)ch->ple_slot[0] * q38c_ple_cells(c), m->PLE_conv_state, q38c_ple_cells(c) * 4)) {
q38c_recover(m, m->kv_len); return;
}
ch->dn_where = Q38C_BOTH;
@@ -295,23 +356,64 @@ static void q38c_cpu_step(Model *m, int pos_base) {
ch->dn_where = Q38C_HOST; ch->host_zero = 0; ch->snap_valid = 0;
for (int i = 0; i < m->c.layers; i++) {
if (ch->kv_valid[i] > pos_base) ch->kv_valid[i] = pos_base;
+ if (m->c.is_attn[i]) vkc_kv_lower(&ch->ks, ch->attn_ord[i], pos_base);
int R = m->c.idx_ratio > 0 ? m->c.idx_ratio : 1;
if (ch->pk_valid[i] > pos_base / R) ch->pk_valid[i] = pos_base / R;
}
}
-/* A rejected draft: the state after the verify's first row is the device's snapshot. */
-static void q38c_rollback(Model *m) {
+/* A rejected draft: the state after the verify's row `slot` (its first slot+1 rows) is
+ * the device's copy in that slot; `len` positions stand. The KV and pooled-key
+ * watermarks come down to them: the rows past them are the rejected drafts'. */
+static void q38c_rollback(Model *m, int slot, int len) {
Q38Chain *ch = (Q38Chain *)m->vkchain;
- if (!ch || !ch->ok || !ch->snap_valid || ch->dn_where != Q38C_DEV) return;
+ if (!ch || !ch->ok) return;
+ int R = m->c.idx_ratio > 0 ? m->c.idx_ratio : 1;
+ for (int i = 0; i < m->c.layers; i++) {
+ if (ch->kv_valid[i] > len) ch->kv_valid[i] = len;
+ if (m->c.is_attn[i]) vkc_kv_lower(&ch->ks, ch->attn_ord[i], len);
+ if (ch->pk_valid[i] > len / R) ch->pk_valid[i] = len / R;
+ }
+ if (slot < 0 || slot >= ch->snap_valid || ch->dn_where != Q38C_DEV) { ch->snap_valid = 0; return; }
for (int i = 0; i < m->c.layers; i++) {
if (m->c.is_attn[i]) continue;
- VkcBuf *t = ch->rec[i]; ch->rec[i] = ch->rec_snap[i]; ch->rec_snap[i] = t;
- t = ch->ring[i]; ch->ring[i] = ch->ring_snap[i]; ch->ring_snap[i] = t;
+ VkcBuf *t = ch->rec[i]; ch->rec[i] = ch->rec_snap[slot][i]; ch->rec_snap[slot][i] = t;
+ t = ch->ring[i]; ch->ring[i] = ch->ring_snap[slot][i]; ch->ring_snap[slot][i] = t;
}
- if (ch->ple_ring) ch->ple_cur ^= 1;
+ if (ch->ple_ring) { int t = ch->ple_slot[0]; ch->ple_slot[0] = ch->ple_slot[1 + slot]; ch->ple_slot[1 + slot] = t; }
ch->snap_valid = 0;
}
-static int q38c_push_state(Q38Chain *ch, Model *m, int pos_base) {
+/* Copy slots on the device for a verify that snapshots `rows` rows: the DeltaNet states
+ * and conv rings (slot 0 since setup), and the PLE ring's slots, all in one buffer since
+ * the shader writes the copy beside the ring (it grows once, keeping the current ring). */
+static int q38c_spec_slots(Q38Chain *ch, Model *m, int rows) {
+ Cfg *c = &m->c; int L = c->layers;
+ if (rows > Q38_SPEC_SNAPS) rows = Q38_SPEC_SNAPS;
+ size_t nr = (size_t)c->dn_vheads * c->dn_kdim * c->dn_vdim, nc = (size_t)c->dn_conv_dim * (c->dn_convk - 1);
+ for (int sl = ch->snap_slots; sl < rows; sl++) {
+ if (!ch->rec_snap[sl] && !(ch->rec_snap[sl] = (VkcBuf **)calloc((size_t)L, sizeof(VkcBuf *)))) return 0;
+ if (!ch->ring_snap[sl] && !(ch->ring_snap[sl] = (VkcBuf **)calloc((size_t)L, sizeof(VkcBuf *)))) return 0;
+ for (int i = 0; i < L; i++) {
+ if (c->is_attn[i]) continue;
+ if (!ch->rec_snap[sl][i] && !(ch->rec_snap[sl][i] = vkc_buf(nr * 4, VKC_DEV))) return 0;
+ if (!ch->ring_snap[sl][i] && !(ch->ring_snap[sl][i] = vkc_buf(nc * 4, VKC_DEV))) return 0;
+ }
+ ch->snap_slots = sl + 1;
+ }
+ if (ch->ple_ring && ch->ple_nslots < rows + 1) {
+ size_t pc = q38c_ple_cells(c);
+ VkcBuf *nb = vkc_buf((size_t)Q38_SPEC_ROWS * pc * 4, VKC_DEV);
+ if (!nb) return 0;
+ if (!vkc_begin() || !vkc_copy(nb, 0, ch->ple_ring, (size_t)ch->ple_slot[0] * pc, pc) || !vkc_submit(1)) {
+ vkc_free(nb); return 0;
+ }
+ vkc_free(ch->ple_ring);
+ ch->ple_ring = nb; ch->ple_nslots = Q38_SPEC_ROWS;
+ for (int r = 0; r < Q38_SPEC_ROWS; r++) ch->ple_slot[r] = r;
+ ch->snap_valid = 0;
+ }
+ return 1;
+}
+static int q38c_push_state(Q38Chain *ch, Model *m, int pos_base, int n_rows) {
Cfg *c = &m->c; int ok = 1;
if (ch->dn_where == Q38C_HOST) {
size_t nr = (size_t)c->dn_vheads * c->dn_kdim * c->dn_vdim, nc = (size_t)c->dn_conv_dim * (c->dn_convk - 1);
@@ -321,7 +423,7 @@ static int q38c_push_state(Q38Chain *ch, Model *m, int pos_base) {
else ok = vkc_write(ch->rec[i], 0, m->DN_rec[i], nr * 4) && vkc_write(ch->ring[i], 0, m->DN_conv[i], nc * 4);
}
if (ok && ch->ple_ring && m->PLE_conv_state) {
- size_t pc = q38c_ple_cells(c), off = (size_t)ch->ple_cur * pc;
+ size_t pc = q38c_ple_cells(c), off = (size_t)ch->ple_slot[0] * pc;
ok = ch->host_zero ? vkc_zero(ch->ple_ring, off, pc) : vkc_write(ch->ple_ring, off, m->PLE_conv_state, pc * 4);
}
ch->dn_where = Q38C_BOTH;
@@ -330,10 +432,16 @@ static int q38c_push_state(Q38Chain *ch, Model *m, int pos_base) {
for (int i = 0; i < c->layers && ok; i++) {
if (!c->is_attn[i]) continue;
if (ch->pk_valid[i] > pos_base / R) ch->pk_valid[i] = pos_base / R; /* blocks this step rewrites */
+ if (ch->ks.on) { /* the split: the window placed, the K/V rows below pos_base of its blocks uploaded */
+ VkcKvPart pt[2] = {{KVH, D, m->K[i], (size_t)m->kv_cap * D, ch->kc[i], 0},
+ {KVH, D, m->V[i], (size_t)m->kv_cap * D, ch->vc[i], 0}};
+ vkc_kv_place(&ch->ks, ch->attn_ord[i], pos_base, n_rows);
+ ok = vkc_kv_push(&ch->ks, ch->attn_ord[i], pt, 2, pos_base);
+ }
if (ch->kv_valid[i] >= pos_base) continue;
int t0 = ch->kv_valid[i], n = pos_base - t0;
- for (int h = 0; h < KVH && ok; h++) {
- size_t src = ((size_t)h * m->kv_cap + t0) * D, dst = ((size_t)h * ch->cap + t0) * D;
+ for (int h = 0; h < KVH && ok && !ch->ks.on; h++) {
+ size_t src = ((size_t)h * m->kv_cap + t0) * D, dst = ((size_t)h * ch->dev_rows + t0) * D;
ok = vkc_write(ch->kc[i], dst, m->K[i] + src, (size_t)n * D * 4) && vkc_write(ch->vc[i], dst, m->V[i] + src, (size_t)n * D * 4);
}
ok = ok && vkc_write(ch->ik[i], (size_t)t0 * ID, m->IK[i] + (size_t)t0 * ID, (size_t)n * ID * 4);
@@ -386,11 +494,14 @@ static int q38c_attention(Q38Chain *ch, Model *m, Layer *l, int i, int n, int pb
if (!rg || !ri) { free(rg); free(ri); return 0; }
for (int s = 0; s < n; s++) {
for (int h = 0; h < KVH; h++)
- rg[s * KVH + h] = (VkcRegion){((size_t)h * ch->cap + pb + s) * D, (size_t)s * kvo + (size_t)h * D, (size_t)D};
+ rg[s * KVH + h] = (VkcRegion){((size_t)h * ch->dev_rows + pb + s) * D, (size_t)s * kvo + (size_t)h * D, (size_t)D};
ri[s] = (VkcRegion){(size_t)(pb + s) * ID, (size_t)s * ipw + (size_t)IQ * ID, (size_t)ID};
}
size_t ko = (size_t)ch->attn_ord[i] * q38c_kv_stride(ch, c);
- ok = vkc_copy_regions(ch->kc[i], ch->k, rg, n * KVH) && vkc_copy_regions(ch->vc[i], ch->v, rg, n * KVH) &&
+ VkcKvPart pk_ = {KVH, D, m->K[i], (size_t)m->kv_cap * D, ch->kc[i], 0}, pv_ = {KVH, D, m->V[i], (size_t)m->kv_cap * D, ch->vc[i], 0};
+ ok = (ch->ks.on ? vkc_kv_store(&ch->ks, ch->attn_ord[i], &pk_, ch->k, 0, (size_t)kvo, (size_t)D, pb, n) &&
+ vkc_kv_store(&ch->ks, ch->attn_ord[i], &pv_, ch->v, 0, (size_t)kvo, (size_t)D, pb, n)
+ : vkc_copy_regions(ch->kc[i], ch->k, rg, n * KVH) && vkc_copy_regions(ch->vc[i], ch->v, rg, n * KVH)) &&
vkc_copy_regions(ch->ik[i], ch->ip, ri, n) &&
vkc_copy(ch->kvd, ko, ch->k, 0, (size_t)n * kvo) &&
vkc_copy(ch->kvd, ko + (size_t)ch->rows * kvo, ch->v, 0, (size_t)n * kvo);
@@ -411,24 +522,41 @@ static int q38c_attention(Q38Chain *ch, Model *m, Layer *l, int i, int n, int pb
}
int nbmax = (int)(vkc_bytes(ch->qsc) / 4 / (2 * (size_t)ch->rows));
VkcQsa ps = {1, ID, R, 0, 0, n, pb, c->idx_budget, IQ, 0, ipw, nbmax, q38c_selrow(c), c->eps, 0, 0, 0, 0};
- VkcAttn at = {n, QH, KVH, D, pb, ch->cap, 0, qo, 2 * D, D, qo, 2 * D, 1, 0, QH * D, 0, q38c_selrow(c),
+ if (ch->ks.on) { /* the split: the selection's listed positions in two parts */
+ VkcKvGqa a = {&ch->ks, ch->attn_ord[i], ch->q, ch->kc[i], ch->vc[i], ch->q, ch->sel, NULL, ch->ctx, n, QH, KVH, D, D, pb,
+ 0, 0, 0, 0, qo, 2 * D, 1, D, qo, 2 * D, QH * D, 0, q38c_selrow(c), 1.f / sqrtf((float)D),
+ m->K[i], m->V[i], (size_t)m->kv_cap * D, (size_t)D, (size_t)m->kv_cap * D, (size_t)D};
+ return ok && vkc_qsa(ch->ip, NULL, ch->pk[i], NULL, ch->qsc, ch->sel, &ps) && vkc_kv_gqa(&a) &&
+ vkc_matmul(q38c_t(&l->o), ch->ctx, 0, ch->blk, 0, n);
+ }
+ VkcAttn at = {n, QH, KVH, D, pb, ch->dev_rows, 0, qo, 2 * D, D, qo, 2 * D, 1, 0, QH * D, 0, q38c_selrow(c),
1.f / sqrtf((float)D), 0, 0};
return ok && vkc_qsa(ch->ip, NULL, ch->pk[i], NULL, ch->qsc, ch->sel, &ps) &&
vkc_attn(ch->q, ch->kc[i], ch->vc[i], ch->ctx, ch->q, ch->sel, &at) &&
vkc_matmul(q38c_t(&l->o), ch->ctx, 0, ch->blk, 0, n);
}
-static int q38c_deltanet(Q38Chain *ch, Model *m, Layer *l, int i, int n, int snap_row) {
+/* ns: the chunk's rows whose state a verify copies (rows 0..ns-1, into slots c0..c0+ns-1).
+ * One copy rides on the one dispatch, as the shaders pass its row; more split the
+ * convolution and the recurrence into one dispatch per copy, each ending on its row
+ * (the last one takes the chunk's remaining rows): the state crosses between them
+ * through memory in f32, so every row gets the bits of the single dispatch. */
+static int q38c_deltanet(Q38Chain *ch, Model *m, Layer *l, int i, int n, int c0, int ns) {
Cfg *c = &m->c;
int VH = c->dn_vheads, CD = c->dn_conv_dim, V = VH * c->dn_vdim;
int ok = vkc_matmul(q38c_t(&l->dn_qkv), ch->mixed, 0, ch->qkv, 0, n) && vkc_matmul(q38c_t(&l->dn_z), ch->mixed, 0, ch->z, 0, n) &&
vkc_matmul(q38c_t(&l->dn_b), ch->mixed, 0, ch->ab, 0, n) &&
vkc_matmul(q38c_t(&l->dn_a), ch->mixed, 0, ch->ab, (size_t)ch->rows * VH, n);
- VkcDnConv cp = {n, CD, c->dn_convk, 0, CD, 0, CD, snap_row, 1, (int)ch->o_conv[i], 0, 0};
- VkcDnRec rp = {n, VH, c->dn_kheads, c->dn_vdim, c->dn_kheads * c->dn_kdim, 0, CD, 0, VH, ch->rows * VH, VH,
- 0, V, 0, V, snap_row, 1, c->eps, 1.f / sqrtf((float)c->dn_kdim), 0, 0, (int)ch->o_dn[i]};
- return ok && vkc_dnconv(ch->qkv, ch->prm, ch->ring[i], ch->cv, ch->ring_snap[i], &cp) &&
- vkc_dnrec(c->dn_kdim, ch->cv, ch->ab, ch->z, ch->rec[i], ch->prm, ch->dny, ch->rec_snap[i], &rp) &&
- vkc_matmul(q38c_t(&l->dn_out), ch->dny, 0, ch->blk, 0, n);
+ int segs = ns > 1 ? ns : 1;
+ for (int r = 0; r < segs && ok; r++) {
+ int s0 = ns > 1 ? r : 0, len = ns > 1 && r < ns - 1 ? 1 : n - s0, snap_row = ns ? 0 : -1, slot = ns ? c0 + r : 0;
+ VkcDnConv cp = {len, CD, c->dn_convk, s0 * CD, CD, s0 * CD, CD, snap_row, 1, (int)ch->o_conv[i], 0, 0};
+ VkcDnRec rp = {len, VH, c->dn_kheads, c->dn_vdim, c->dn_kheads * c->dn_kdim, s0 * CD, CD, s0 * VH, VH,
+ ch->rows * VH + s0 * VH, VH, s0 * V, V, s0 * V, V, snap_row, 1, c->eps,
+ 1.f / sqrtf((float)c->dn_kdim), 0, 0, (int)ch->o_dn[i]};
+ ok = vkc_dnconv(ch->qkv, ch->prm, ch->ring[i], ch->cv, ch->ring_snap[slot][i], &cp) &&
+ vkc_dnrec(c->dn_kdim, ch->cv, ch->ab, ch->z, ch->rec[i], ch->prm, ch->dny, ch->rec_snap[slot][i], &rp);
+ }
+ return ok && vkc_matmul(q38c_t(&l->dn_out), ch->dny, 0, ch->blk, 0, n);
}
/* the n-gram rows of n tokens, in order, as q38_ple fills them (history and the verify's snapshot) */
static void q38c_ple_rows(Model *m, const int *ids, int c0, int n, float *emb) {
@@ -447,13 +575,17 @@ static void q38c_ple_rows(Model *m, const int *ids, int c0, int n, float *emb) {
else if (m->ple_history_len == 0) { m->ple_history[0] = ids[s]; m->ple_history_len = 1; }
else if (m->ple_history_len == 1) { m->ple_history[1] = ids[s]; m->ple_history_len = 2; }
else { m->ple_history[0] = m->ple_history[1]; m->ple_history[1] = ids[s]; }
- if (s + 1 == m->snap_after) {
- memcpy(m->snap_ple_history, m->ple_history, sizeof(m->snap_ple_history));
- m->snap_ple_history_len = m->ple_history_len;
+ if (s < m->snap_rows) {
+ memcpy(m->snap_ple_history[s], m->ple_history, sizeof(m->snap_ple_history[s]));
+ m->snap_ple_history_len[s] = m->ple_history_len;
}
}
}
-static int q38c_ple(Q38Chain *ch, Model *m, int n, int snap_row) {
+/* ns as q38c_deltanet's: the copies of the PLE ring after rows 0..ns-1 go to the ring
+ * buffer's slots ple_slot[1+c0+r]. The convolution reads a row's gate and norm outputs
+ * from the start of their buffers, so with more than one copy each dispatch past the
+ * first reads its rows from a copy of them at the start of the scratch (gseg, nseg). */
+static int q38c_ple(Q38Chain *ch, Model *m, int n, int c0, int ns) {
Cfg *c = &m->c; Layer *l = &m->L[c->ple_layer];
int W = c->hc_width, H = c->hidden, E = c->ngram_heads * c->ngram_head_dim;
size_t pc = q38c_ple_cells(c);
@@ -461,11 +593,28 @@ static int q38c_ple(Q38Chain *ch, Model *m, int n, int snap_row) {
vkc_matmul(q38c_t(&l->ple_key), ch->emb, 0, ch->keys, 0, n) &&
vkc_matmul(q38c_t(&l->ple_value), ch->emb, 0, ch->val, 0, n);
VkcPle g = {0, n, c->hc_count, H, c->ple_convk, c->ngram_size, 0, 0, 0, -1, 0, c->eps, (int)ch->o_ple, 0, 0};
- VkcPle cv = {1, n, c->hc_count, H, c->ple_convk, c->ngram_size, 0, 0, 0, snap_row,
- (int)((size_t)(ch->ple_cur ^ 1) * pc), c->eps, 0, (int)ch->o_pconv, (int)((size_t)ch->ple_cur * pc)};
- (void)W;
- return ok && vkc_ple(ch->keys, ch->hyper, ch->val, ch->prm, ch->gated, ch->normv, NULL, NULL, &g) &&
- vkc_ple(NULL, ch->hyper, NULL, NULL, ch->gated, ch->normv, ch->prm, ch->ple_ring, &cv);
+ ok = ok && vkc_ple(ch->keys, ch->hyper, ch->val, ch->prm, ch->gated, ch->normv, NULL, NULL, &g);
+ if (ns <= 1) {
+ int snap = ns ? ch->ple_slot[1 + c0] : ch->ple_slot[1];
+ VkcPle cv = {1, n, c->hc_count, H, c->ple_convk, c->ngram_size, 0, 0, 0, ns ? 0 : -1,
+ (int)((size_t)snap * pc), c->eps, 0, (int)ch->o_pconv, (int)((size_t)ch->ple_slot[0] * pc)};
+ return ok && vkc_ple(NULL, ch->hyper, NULL, NULL, ch->gated, ch->normv, ch->prm, ch->ple_ring, &cv);
+ }
+ ok = ok && q38c_res(&ch->gseg, (size_t)n * W, VKC_DEV) && q38c_res(&ch->nseg, (size_t)n * W, VKC_DEV);
+ for (int r = 0; r < ns && ok; r++) {
+ int len = r < ns - 1 ? 1 : n - r;
+ VkcBuf *gb = ch->gated, *nb = ch->normv;
+ if (r) {
+ ok = vkc_copy(ch->gseg, 0, ch->gated, (size_t)r * W, (size_t)len * W) &&
+ vkc_copy(ch->nseg, 0, ch->normv, (size_t)r * W, (size_t)len * W);
+ gb = ch->gseg; nb = ch->nseg;
+ }
+ VkcPle cv = {1, len, c->hc_count, H, c->ple_convk, c->ngram_size, 0, r * W, 0, 0,
+ (int)((size_t)ch->ple_slot[1 + c0 + r] * pc), c->eps, 0, (int)ch->o_pconv,
+ (int)((size_t)ch->ple_slot[0] * pc)};
+ ok = ok && vkc_ple(NULL, ch->hyper, NULL, NULL, gb, nb, ch->prm, ch->ple_ring, &cv);
+ }
+ return ok;
}
static int q38c_shared(Q38Chain *ch, Model *m, Layer *l, int i, int n) {
Cfg *c = &m->c; int SI = c->shared_inter;
@@ -487,12 +636,21 @@ static int q38c_moe_apply(Q38Chain *ch, Model *m, int n) {
static int q38c_forward(Model *m, const int *ids, int S, int pos_base, int nlogits, float *hyper_h,
int want_streams, float *mixed_h, float *logit) {
if (!g_vk_chain || qt_ready()) return 0;
- if (g_vk_chain == COLI_VK_CHAIN_PREFILL && S <= 2) return 0; /* prompts only: decode and verify on the CPU */
+ /* prompts only: decode and verifies on the CPU */
+ if (g_vk_chain == COLI_VK_CHAIN_PREFILL && (S <= 2 || g_q38_rowwise)) return 0;
Q38Chain *ch = q38c_setup(m);
if (!ch || ch->failed) return 0;
Cfg *c = &m->c; int H = c->hidden, W = c->hc_width, L = c->layers, E = c->experts, V = c->vocab;
- int rows = S < q38c_chunk_rows() ? S : q38c_chunk_rows();
- if (!q38c_scratch(ch, m, rows) || (want_streams && !q38c_res(&ch->hypd, (size_t)rows * W, VKC_DOWN)) ||
+ int mirror_ok = q38c_mirror(ch, m);
+ int CH = mirror_ok ? q38c_chunk_rows(ch, m) : 1, rows = S < CH ? S : CH;
+ if (ch->ks.on && rows > ch->ks.chunk) rows = ch->ks.chunk; /* a step's rows fit the split's window */
+ if (m->snap_rows > 1 && !q38c_spec_slots(ch, m, m->snap_rows)) {
+ fprintf(stderr, "[VK] qwen38 chain: device memory for a verify's %d copies refused; this verify on the CPU\n",
+ m->snap_rows);
+ return 0;
+ }
+ if (!mirror_ok || !q38c_scratch(ch, m, rows) || (want_streams && !q38c_res(&ch->hypd, (size_t)rows * W, VKC_DOWN)) ||
+ (nlogits > 2 && !q38c_res(&ch->outd, (size_t)nlogits * V, VKC_DOWN)) ||
(mixed_h && !q38c_res(&ch->find, (size_t)rows * H, VKC_DOWN))) {
fprintf(stderr, "[VK] qwen38 chain: device memory for %d rows refused; per-matrix path\n", rows);
ch->failed = 1;
@@ -503,10 +661,11 @@ static int q38c_forward(Model *m, const int *ids, int S, int pos_base, int nlogi
int snapped = 0;
for (int c0 = 0; c0 < S; c0 += rows) {
int n = S - c0 < rows ? S - c0 : rows, pb = pos_base + c0;
- int snap_row = m->snap_after > c0 && m->snap_after <= c0 + n ? m->snap_after - 1 - c0 : -1;
- if (snap_row >= 0) snapped = 1;
+ int ns = m->snap_rows - c0 < n ? m->snap_rows - c0 : n; /* the rows of this chunk a verify copies */
+ if (ns < 0) ns = 0;
+ if (ns) snapped = c0 + ns;
if (!vkc_begin() || !vkc_write(ch->hyper, 0, hyper_h + (size_t)c0 * W, (size_t)n * W * 4) ||
- !q38c_push_state(ch, m, pb)) goto lost;
+ !q38c_push_state(ch, m, pb, n)) goto lost;
if (half) {
float *cs = (float *)vkc_ptr(ch->cs);
for (int s = 0; s < n; s++) for (int j = 0; j < half; j++) {
@@ -519,15 +678,19 @@ static int q38c_forward(Model *m, const int *ids, int S, int pos_base, int nlogi
for (int i = 0; i < L && ok; i++) {
Layer *l = &m->L[i];
if (pending) ok = q38c_moe_apply(ch, m, n);
- if (ok && i == c->ple_layer) ok = q38c_ple(ch, m, n, snap_row);
+ if (ok && i == c->ple_layer) ok = q38c_ple(ch, m, n, c0, ns);
ok = ok && q38c_gr_read(ch, m, &l->attn_gr, ch->o_an[i], n, ch->inj_a);
- ok = ok && (c->is_attn[i] ? q38c_attention(ch, m, l, i, n, pb) : q38c_deltanet(ch, m, l, i, n, snap_row));
+ ok = ok && (c->is_attn[i] ? q38c_attention(ch, m, l, i, n, pb) : q38c_deltanet(ch, m, l, i, n, c0, ns));
ok = ok && q38c_gr_apply(ch, m, ch->inj_a, n);
ok = ok && q38c_gr_read(ch, m, &l->mlp_gr, ch->o_mn[i], n, ch->inj_m);
ok = ok && vkc_matmul(q38c_t(&l->router), ch->mixed, 0, ch->lg, 0, n) &&
vkc_copy(ch->lgd, 0, ch->lg, 0, (size_t)n * E) && vkc_copy(ch->mixd, 0, ch->mixed, 0, (size_t)n * H);
double t0 = now_s();
- ok = ok && vkc_submit(1); /* A1 */
+ /* A1; while it runs, the tier loads the experts this layer will likely
+ * stream (a big prompt chunk only) */
+ ok = ok && vkc_submit(0);
+ if (ok) vkt_stream_prefetch(i, n);
+ ok = ok && vkc_finish();
m->timers.seconds[Q38_TM_DENSE_MATMUL] += now_s() - t0;
if (!ok) break;
if (c->is_attn[i]) {
@@ -567,7 +730,7 @@ static int q38c_forward(Model *m, const int *ids, int S, int pos_base, int nlogi
if (!ok) goto lost;
if (want_streams) memcpy(hyper_h + (size_t)c0 * W, vkc_ptr(ch->hypd), (size_t)n * W * 4);
if (mixed_h) memcpy(mixed_h + (size_t)c0 * H, vkc_ptr(ch->find), (size_t)n * H * 4);
- for (int i = 0; i < L; i++) if (c->is_attn[i]) ch->kv_valid[i] = pb + n;
+ for (int i = 0; i < L; i++) if (c->is_attn[i]) { ch->kv_valid[i] = pb + n; vkc_kv_done(&ch->ks, ch->attn_ord[i], pb + n); }
ch->dn_where = Q38C_DEV; ch->host_zero = 0;
}
memcpy(logit, vkc_ptr(ch->outd), (size_t)nlogits * V * sizeof(float));
@@ -589,5 +752,6 @@ static void q38c_report(Model *m) {
fprintf(stderr, "[VK] qwen38 chain: %llu forwards, %llu frames (%llu ops, %llu matmuls, %llu tiled GEMM), "
"%.1f ms waiting for the device, %.1f ms of routed experts on the host, %.1f MiB on the device\n",
ch->forwards, st.frames, st.ops, st.matmuls, st.gemms, st.wait_ms, ch->host_ms, st.dev_bytes / 1048576.0);
+ vkc_kv_report(&ch->ks);
vkc_prof_print();
}
diff --git a/c/qwen38_core.h b/c/qwen38_core.h
index bf1e0fc7..6943489d 100644
--- a/c/qwen38_core.h
+++ b/c/qwen38_core.h
@@ -12,6 +12,7 @@
#ifndef COLI_QWEN38_CORE_H
#define COLI_QWEN38_CORE_H
#include "kv_prefix.h"
+#include "spec_draft.h" /* prompt-lookup drafts and the gate that decides when drafting pays */
#include "expert_ffn.h" /* the int4-g64 routed experts (experts-int4g64/ sidecar) run through its f32 kernel */
#include <pthread.h> /* q38_ehit_mark publishes the lazy HITS table under a lock */
#ifdef COLI_VULKAN
@@ -25,6 +26,11 @@ static int g_vk_dense = 1; /* COLI_VK_DENSE=0: the trunk stays on the CPU, the
#define Q38_MAX_TOPK 256
#define Q38_MAX_PLE_PARTS 512
#define Q38_PREFILL_BATCH_ROWS 32
+/* A speculative verify's rows: the picked token and up to five drafts (three from the
+ * MTP head, five from prompt lookup). A rejection after row k restores the state the
+ * verify copied after that row: one copy slot per row but the last. */
+#define Q38_SPEC_ROWS 6
+#define Q38_SPEC_SNAPS (Q38_SPEC_ROWS - 1)
#define Q38_PREFILL_WORKSPACE_BYTES (64u << 20)
typedef struct {
@@ -72,6 +78,8 @@ typedef struct {
int8_t *q8; float *q8sc; /* the trunk's int8 rows on the CPU (default; Q38_TRUNK_CPU_INT8=0 keeps BF16): the same rows the GPU holds, met by an int8 activation in idot.h */
void *vk; int vk_off; /* COLI_VULKAN=1: device copy of q8, or of the BF16/F32 rows (uploaded at the first matmul), vk_off = upload failed, stays on the CPU */
int vk_res; /* COLI_VULKAN=1: a resident matrix (q38_load_weight), never an expert slot that is refilled in place */
+ char *vk_name; /* COLI_VULKAN=1: its tensor's name, to read it back from disk (q38_dho_reload) */
+ int vk_fmt, vk_gone; /* dense weights on the device only (COLI_VK_DENSE_HOST): the device copy's format, and 1 while no host copy is kept */
} Q38Weight;
typedef struct { float *norm; Q38Weight down, up, inject; } GatedResidual;
@@ -203,23 +211,25 @@ typedef struct {
* expert_scales[], K/V/IK[] and the route-trace row. mtp_len counts the
* head's KV rows that hold a verified pair (the streams at position p
* with the token at p+1); mtp_pend keeps the streams of the rows after
- * them whose next token has not reached the head yet (one, or two after
- * an accepted draft, the first one's next token in mtp_pend_tok). */
+ * them whose next token has not reached the head yet (one, or the rows of
+ * a verify that stood: up to Q38_SPEC_ROWS, every one but the last with
+ * its next token in mtp_pend_tok). */
int mtp, mtp_wiring;
char mtp_prefix[32];
float *mtp_norm_emb, *mtp_norm_hid;
Q38Weight mtp_fc_emb, mtp_fc_hid;
GatedResidual mtp_mixer;
- int mtp_len, mtp_pend_n, mtp_pend_tok;
- float *mtp_pend;
- /* A verify forward (S=2) copies the DeltaNet and PLE state as it stands
- * once its first snap_after rows are in (0 = no copy), so a rejected
- * draft rolls back by swapping these in instead of running the forward
- * again. */
- int snap_after;
- float **snap_rec, **snap_conv, *snap_ple;
- int64_t snap_ple_history[2];
- int snap_ple_history_len;
+ int mtp_len, mtp_pend_n, mtp_pend_tok[Q38_SPEC_ROWS];
+ float *mtp_pend; /* Q38_SPEC_ROWS rows of hc_width */
+ /* A verify forward copies the DeltaNet and PLE state as it stands after
+ * each of its first snap_rows rows, row r into slot r (0 = no copy), so a
+ * draft rejected after row r+1 rolls back by swapping slot r in instead
+ * of running the forward again. snap_slots: the slots allocated
+ * (q38_spec_alloc). */
+ int snap_rows, snap_slots;
+ float **snap_rec[Q38_SPEC_SNAPS], **snap_conv[Q38_SPEC_SNAPS], *snap_ple[Q38_SPEC_SNAPS];
+ int64_t snap_ple_history[Q38_SPEC_SNAPS][2];
+ int snap_ple_history_len[Q38_SPEC_SNAPS];
#ifdef COLI_VULKAN
void *vkchain; /* the dense chain's device state (qwen38_chain.h), NULL until it runs */
#endif
@@ -291,6 +301,7 @@ static void q38_weight_free(Q38Weight *weight) {
free(weight->q8); free(weight->q8sc);
#ifdef COLI_VULKAN
if(weight->vk)coli_vk_tensor_free((ColiVkTensor*)weight->vk);
+ free(weight->vk_name);
#endif
memset(weight,0,sizeof(*weight));
}
@@ -463,10 +474,16 @@ static void q38_matmul_int4g64(float *y,const float *x,const uint8_t *codes,
* first call; a failed upload sets vk_off and the matrix stays on the CPU.
* The backend has one command buffer: never from a parallel region. */
static int q38_vk_eligible(const Q38Weight *w) {
- return w->q8 || (w->vk_res && w->data &&
+ return w->q8 || w->vk_gone || (w->vk_res && w->data &&
(w->kind==Q38_WEIGHT_BF16 || w->kind==Q38_WEIGHT_F32));
}
+/* The format of the weight's device copy: the int8 rows, else the rows as loaded; with
+ * the host copy dropped (vk_gone), the one it was uploaded in. */
+static int q38_vk_fmt(const Q38Weight *w) {
+ return w->vk_gone?w->vk_fmt:w->q8?1:w->kind==Q38_WEIGHT_BF16?11:10;
+}
static unsigned g_q38_vk_placed[3]; /* uploads by format: int8 rows, bf16, f32 */
+static void q38_dho_reload(Q38Weight *w); /* below, beside the trunk's int8 rows */
/* The weight's device copy, uploaded on the first call; NULL when it is not eligible
* or the upload failed (vk_off). The dense chain (qwen38_chain.h) reads the same one. */
static ColiVkTensor *q38_vk_tensor(const Q38Weight *weight) {
@@ -474,7 +491,8 @@ static ColiVkTensor *q38_vk_tensor(const Q38Weight *weight) {
Q38Weight *w=(Q38Weight*)weight; /* vk is a cache in a weight the forward pass treats as read-only */
ColiVkTensor **t=(ColiVkTensor**)&w->vk;
if(*t)return *t;
- int fmt=w->q8?1:w->kind==Q38_WEIGHT_BF16?11:10;
+ if(w->vk_gone)return NULL;
+ int fmt=q38_vk_fmt(w);
const void *wq=w->q8?(const void*)w->q8:(const void*)w->data;
const float *sc=w->q8?w->q8sc:NULL; /* fmt 10/11: no scales */
if(!coli_vk_tensor_ensure(t,wq,sc,fmt,w->cols,w->rows,0)){w->vk_off=1;return NULL;}
@@ -487,7 +505,7 @@ static int q38_vk_matmul(float *y,const float *x,const Q38Weight *weight,int S,i
#endif
if(weight->vk_off||S<1||S>65535)return 0;
Q38Weight *w=(Q38Weight*)weight;
- int fmt=w->q8?1:w->kind==Q38_WEIGHT_BF16?11:10;
+ int fmt=q38_vk_fmt(w);
const void *wq=w->q8?(const void*)w->q8:(const void*)w->data;
const float *sc=w->q8?w->q8sc:NULL; /* fmt 10/11: no scales */
ColiVkTensor *t=q38_vk_tensor(w);
@@ -529,6 +547,8 @@ static void q38_weight_matmul(float *y,const float *x,const Q38Weight *weight,
#ifdef COLI_VULKAN
if(g_vk_ready&&g_vk_dense&&weight&&weight->rows==O&&weight->cols==I&&q38_vk_eligible(weight)&&
q38_vk_matmul(y,x,weight,S,I,O))return;
+ /* the CPU needs a matrix the device holds alone (a lost device): read it back */
+ if(weight&&weight->vk_gone)q38_dho_reload((Q38Weight*)weight);
#endif
if(weight&&weight->q8&&weight->rows==O&&weight->cols==I){
/* the trunk's int8 rows (the same the GPU holds) meet an int8
@@ -914,6 +934,8 @@ static Q38Weight q38_load_weight(Model *m,const char *name,int rows,int cols) {
m->resident_weight_bytes+=q38_weight_bytes(&weight);
#ifdef COLI_VULKAN
weight.vk_res=1; /* resident for the life of the model: its rows may live on the device */
+ weight.vk_name=strdup(name); /* to read it back if the device ends up holding it alone */
+ if(!weight.vk_name){fprintf(stderr,"OOM weight name\n");exit(1);}
#endif
return weight;
}
@@ -2090,9 +2112,9 @@ static void q38_ple(Model *m,const int *ids,int S,const float *hyper,float *out)
else if(m->ple_history_len==0){m->ple_history[0]=ids[s];m->ple_history_len=1;}
else if(m->ple_history_len==1){m->ple_history[1]=ids[s];m->ple_history_len=2;}
else {m->ple_history[0]=m->ple_history[1];m->ple_history[1]=ids[s];}
- if(s+1==m->snap_after){ /* MTP verify: the history after this row (the ring below) */
- memcpy(m->snap_ple_history,m->ple_history,sizeof(m->snap_ple_history));
- m->snap_ple_history_len=m->ple_history_len;
+ if(s<m->snap_rows){ /* a verify: the history after this row (the ring below) */
+ memcpy(m->snap_ple_history[s],m->ple_history,sizeof(m->snap_ple_history[s]));
+ m->snap_ple_history_len[s]=m->ple_history_len;
}
}
q38_dense_matmul(m,keysb,embs,&l->ple_key,rows,E,W);q38_dense_matmul(m,valueb,embs,&l->ple_value,rows,E,H);
@@ -2112,8 +2134,8 @@ static void q38_ple(Model *m,const int *ids,int S,const float *hyper,float *out)
out[(int64_t)s*W+d]=gated[d]+q38_silu(a);
float *rg=ring+(int64_t)d*state_len;for(int k=0;k<state_len-1;k++)rg[k]=rg[k+1];rg[state_len-1]=norm[d];
}
- if(s+1==m->snap_after) /* MTP verify: as q38_deltanet */
- memcpy(m->snap_ple,ring,(size_t)c->hc_width*state_len*sizeof(float));
+ if(s<m->snap_rows) /* a verify: as q38_deltanet */
+ memcpy(m->snap_ple[s],ring,(size_t)c->hc_width*state_len*sizeof(float));
}
}
free(embs);free(keysb);free(valueb);free(kn);free(qn);free(gated);free(norm);
@@ -2275,11 +2297,11 @@ static void q38_deltanet(Model *m,Layer *l,int layer,const float *x,int S,
for(int h=0;h<VH;h++)
q38_rmsg(norm_row+(int64_t)h*VD,core+(int64_t)h*VD,
z_row+(int64_t)h*VD,l->dn_norm,VD,c->eps,1);
- /* MTP verify: the state the token before the draft leaves, for
- * the rollback of a rejected draft (q38_spec_rollback) */
- if(base+s+1==m->snap_after){
- memcpy(m->snap_rec[layer],rec,(size_t)VH*KD*VD*sizeof(float));
- memcpy(m->snap_conv[layer],ring,(size_t)CD*(CK-1)*sizeof(float));
+ /* a verify: the state this row leaves, for the rollback of a
+ * draft rejected after it (q38_spec_rollback) */
+ if(base+s<m->snap_rows){
+ memcpy(m->snap_rec[base+s][layer],rec,(size_t)VH*KD*VD*sizeof(float));
+ memcpy(m->snap_conv[base+s][layer],ring,(size_t)CD*(CK-1)*sizeof(float));
}
}
q38_dense_matmul(m,out+(int64_t)base*H,norm,&l->dn_out,
@@ -2526,6 +2548,69 @@ static void q38_trunk_cpu_int8(Model *m) {
fprintf(stderr,"[qwen38] trunk: %d matrices int8 on the CPU (%.2f GiB, %.2f GiB of BF16 released) in %.1fs; Q38_TRUNK_CPU_INT8=0 keeps BF16\n",
n,bytes/1073741824.0,released/1073741824.0,now_s()-t0);
}
+#ifdef COLI_VULKAN
+/* ---- dense weights on the device only (COLI_VK_DENSE_HOST) ----------------------
+ * With the trunk on the device (the chain, or COLI_VK_DENSE), every resident matrix is
+ * uploaded at start and its host copy (the int8 rows, or the BF16/F32 rows of a matrix
+ * below Q38_TRUNK_MIN_KB) is given back. The CPU then never multiplies by it, except
+ * after a lost device: q38_weight_matmul reads it back from the checkpoint first
+ * (q38_load_weight, then the int8 rows exactly as q38_trunk_cpu_int8 made them) and the
+ * copy stays from there on. What keeps its host copy: the embedding (its rows are
+ * gathered on the CPU), the vision tower (CPU only), norms and vectors. */
+static Model *g_q38_dho_model;
+static pthread_mutex_t g_q38_dho_mx=PTHREAD_MUTEX_INITIALIZER;
+static size_t q38_dho_host_bytes(const Q38Weight *w) {
+ if(w->q8)return (size_t)w->rows*w->cols+(size_t)w->rows*sizeof(float);
+ return w->data?(size_t)q38_weight_bytes(w):0;
+}
+static void q38_dho_reload(Q38Weight *w) {
+ pthread_mutex_lock(&g_q38_dho_mx);
+ if(w->vk_gone){
+ Model *m=g_q38_dho_model;
+ if(!m||!w->vk_name){fprintf(stderr,"[VK] qwen38: a dense matrix the device held alone cannot be read back\n");exit(1);}
+ size_t keep=m->resident_weight_bytes;
+ Q38Weight nw=q38_load_weight(m,w->vk_name,w->rows,w->cols);
+ m->resident_weight_bytes=keep;
+ if(w->vk_fmt==1){
+ q38_trunk_quantize(&nw,&w->q8,&w->q8sc);
+ free(nw.data);
+ } else { w->data=nw.data; w->owns_data=1; w->kind=nw.kind; w->elements=nw.elements; }
+ free(nw.vk_name);
+ w->vk_gone=0;
+ coli_vk_dense_host_reloaded(q38_dho_host_bytes(w));
+ }
+ pthread_mutex_unlock(&g_q38_dho_mx);
+}
+static void q38_dho_drop(Q38Weight *w,size_t *bytes,int *n) {
+ if(!w->vk_res||w->vk_gone||!q38_vk_eligible(w)||!w->vk_name||!q38_vk_tensor(w))return;
+ size_t b=q38_dho_host_bytes(w);
+ w->vk_fmt=q38_vk_fmt(w);
+ if(w->q8){free(w->q8);free(w->q8sc);w->q8=NULL;w->q8sc=NULL;}
+ if(w->owns_data&&w->data){free(w->data);w->data=NULL;w->owns_data=0;}
+ w->vk_gone=1;
+ coli_vk_dense_host_dropped(b);
+ *bytes+=b; (*n)++;
+}
+/* Every resident matrix the device can take, in a fixed order: the head, each layer's
+ * (the MTP head's layer included), the PLE projections, the MTP head's own. */
+static void q38_dho_each(Model *m,void (*f)(Q38Weight *,size_t *,int *),size_t *bytes,int *n) {
+ Cfg *c=&m->c;
+ f(&m->lm_head,bytes,n); f(&m->final_gr.down,bytes,n); f(&m->final_gr.up,bytes,n);
+ for(int i=0;i<=c->layers;i++){
+ if(i==c->layers&&!m->mtp)break;
+ Layer *l=&m->L[i];
+ Q38Weight *ws[]={&l->attn_gr.down,&l->attn_gr.up,&l->attn_gr.inject,&l->mlp_gr.down,&l->mlp_gr.up,
+ &l->mlp_gr.inject,&l->router,&l->sh_g,&l->sh_u,&l->sh_d,&l->q,&l->k,&l->v,&l->o,
+ &l->idx_qk,&l->dn_qkv,&l->dn_z,&l->dn_b,&l->dn_a,&l->dn_out,&l->ple_key,&l->ple_value};
+ for(size_t k=0;k<sizeof ws/sizeof ws[0];k++)f(ws[k],bytes,n);
+ }
+ if(m->mtp){f(&m->mtp_fc_emb,bytes,n);f(&m->mtp_fc_hid,bytes,n);f(&m->mtp_mixer.down,bytes,n);f(&m->mtp_mixer.up,bytes,n);}
+}
+static void q38_dho_count(Q38Weight *w,size_t *bytes,int *n) {
+ if(w->vk_res&&!w->vk_off&&q38_vk_eligible(w)){*bytes+=q38_dho_host_bytes(w);(*n)++;}
+}
+#endif
+
/* after qt_init: quantize and upload what the placer accepted */
static void q38_trunk_place_all(Model *m) {
(void)m;
@@ -2748,6 +2833,9 @@ static void q38_moe_prefill(Model *m,Layer *l,int layer,const float *x,
Cfg *c=&m->c;
int H=c->hidden,E=c->experts,K=c->topk,I=c->inter,SI=c->shared_inter;
int rows_capacity=q38_moe_prefill_rows(c,S);
+ /* the Vulkan tier streams a prompt chunk's cold experts (vk_tier.c): it takes the
+ * whole chunk in one step, so each expert's rows meet in one GEMM */
+ if(vkt_ready()&&layer<c->layers)rows_capacity=vkt_step_rows(S,rows_capacity);
int64_t max_assign=(int64_t)rows_capacity*K;
Q38RouteAssignment *routes=(Q38RouteAssignment*)malloc(
@@ -3125,6 +3213,14 @@ static float *q38_forward(Model *m,const int *ids,int S,int pos_base,int nlogits
} else {
free(chain_logit); chain_logit=NULL;
q38c_cpu_step(m,pos_base);
+ /* A completed chunk may already have returned its final streams in
+ * hyper. Replay the whole forward from embeddings after a later
+ * chunk fails, including rows the device had finished. */
+ for(int s=0;s<S;s++){
+ float *e=hyper+(int64_t)s*W;
+ q38_embed_row(m,ids[s],pos_base+s,e);
+ for(int b=1;b<C;b++)memcpy(e+(int64_t)b*H,e,(size_t)H*sizeof(float));
+ }
}
}
if(!chain_logit){
@@ -3253,10 +3349,13 @@ static void q38_mtp_input(Model *m,const float *streams,const int *next,int S,in
/* The head over S pairs at rows pos_base..pos_base+S-1: their K/V/indexer
* rows enter its attention and, with `logit`, the last row's logits come
- * back through the head's mixer and lm_head. Rows go in bounded chunks so a
- * long prompt's pairs do not need a prompt-sized workspace; the head's
- * attention is causal, so chunking changes no result. */
-static void q38_mtp_rows(Model *m,const float *streams,const int *next,int S,int pos_base,float *logit) {
+ * back through the head's mixer and lm_head; with `hidden`, the last row's
+ * four streams as the head's layer leaves them (what a deeper draft reads in
+ * place of the model's). Rows go in bounded chunks so a long prompt's pairs
+ * do not need a prompt-sized workspace; the head's attention is causal, so
+ * chunking changes no result. */
+static void q38_mtp_rows(Model *m,const float *streams,const int *next,int S,int pos_base,float *logit,
+ float *hidden) {
Cfg *c=&m->c; int H=c->hidden,W=c->hc_width,C=c->hc_count;
int cap=q38_bounded_prefill_rows(S,0,(uint64_t)(3*W+2*H+C)*sizeof(float));
float *hyper=falloc((int64_t)cap*W),*mixed=falloc((int64_t)cap*H);
@@ -3266,6 +3365,7 @@ static void q38_mtp_rows(Model *m,const float *streams,const int *next,int S,int
q38_mtp_input(m,streams+(int64_t)base*W,next+base,rows,pos_base+base,hyper);
q38_layer_forward(m,c->layers,hyper,next+base,rows,pos_base+base,mixed,inject,block);
m->mtp_len=pos_base+base+rows;
+ if(hidden&&base+rows==S)memcpy(hidden,hyper+(int64_t)(rows-1)*W,(size_t)W*sizeof(float));
if(logit&&base+rows==S){
double started=now_s();
q38_gr_read(m,&m->mtp_mixer,hyper+(int64_t)(rows-1)*W,1,mixed,NULL);
@@ -3276,11 +3376,13 @@ static void q38_mtp_rows(Model *m,const float *streams,const int *next,int S,int
free(hyper);free(mixed);free(inject);free(block);
}
-/* The pending rows meet their next token: the head reads them (the first of
- * two with the token already known in mtp_pend_tok, the last with `tok`). */
-static void q38_mtp_take_pending(Model *m,int tok,float *logit) {
- int next[2]={m->mtp_pend_tok,tok},n=m->mtp_pend_n;
- q38_mtp_rows(m,m->mtp_pend,next+2-n,n,m->mtp_len,logit);
+/* The pending rows meet their next token: the head reads them (each but the
+ * last with the token already known in mtp_pend_tok, the last with `tok`). */
+static void q38_mtp_take_pending(Model *m,int tok,float *logit,float *hidden) {
+ int next[Q38_SPEC_ROWS],n=m->mtp_pend_n;
+ for(int j=0;j+1<n;j++)next[j]=m->mtp_pend_tok[j];
+ next[n-1]=tok;
+ q38_mtp_rows(m,m->mtp_pend,next,n,m->mtp_len,logit,hidden);
m->mtp_pend_n=0;
}
@@ -3296,9 +3398,9 @@ static void q38_mtp_feed(Model *m,const int *ids,int S,int pos_base,const float
m->mtp_pend_n=0;
if(m->mtp_len>pos_base-1)m->mtp_len=pos_base>0?pos_base-1:0;
}
- if(m->mtp_pend_n&&m->mtp_len+m->mtp_pend_n==pos_base)q38_mtp_take_pending(m,ids[0],NULL);
+ if(m->mtp_pend_n&&m->mtp_len+m->mtp_pend_n==pos_base)q38_mtp_take_pending(m,ids[0],NULL,NULL);
if(m->mtp_len!=pos_base){m->mtp_pend_n=0;return;}
- if(S>1)q38_mtp_rows(m,streams,ids+1,S-1,pos_base,NULL);
+ if(S>1)q38_mtp_rows(m,streams,ids+1,S-1,pos_base,NULL,NULL);
memcpy(m->mtp_pend,streams+(int64_t)(S-1)*W,(size_t)W*sizeof(float));
m->mtp_pend_n=1;
}
@@ -3311,11 +3413,12 @@ static int q38_argmax(const float *lo,int V) {
/* The head's draft for the token after `tok`, which is about to be fed at
* `pos`: the pending rows take `tok`, and the last one's logits name the
- * token at pos+1. -1 when the head has not read up to pos. */
-static int q38_mtp_draft(Model *m,int tok,int pos) {
+ * token at pos+1; `hidden` gets that row's streams. -1 when the head has not
+ * read up to pos. */
+static int q38_mtp_draft(Model *m,int tok,int pos,float *hidden) {
if(!m->mtp_pend_n||m->mtp_len+m->mtp_pend_n!=pos)return -1;
float *logit=falloc(m->c.vocab);
- q38_mtp_take_pending(m,tok,logit);
+ q38_mtp_take_pending(m,tok,logit,hidden);
int draft=q38_argmax(logit,m->c.vocab);
if(g_q38_mtp_dump){
int32_t head[2]={pos-1,tok};
@@ -3328,6 +3431,28 @@ static int q38_mtp_draft(Model *m,int tok,int pos) {
return draft;
}
+/* A deeper draft: the head runs one more row past its settled ones, on its own
+ * streams of the row before (`hidden`, standing in for the model's, which only
+ * the verify computes) with the draft just proposed, at head row `row`. Its
+ * K/V/indexer row there is a tail: the settled pair the head reads at that row
+ * once the verify has run overwrites it before anything reads past it.
+ * mtp_len stays where the settled rows end; `hidden` becomes this row's
+ * streams, for the draft after. */
+static int q38_mtp_draft_more(Model *m,float *hidden,int tok,int row) {
+ Cfg *c=&m->c; int H=c->hidden,W=c->hc_width,C=c->hc_count;
+ float *hyper=falloc(W),*mixed=falloc(H),*inject=falloc(C),*block=falloc(H),*logit=falloc(c->vocab);
+ q38_mtp_input(m,hidden,&tok,1,row,hyper);
+ q38_layer_forward(m,c->layers,hyper,&tok,1,row,mixed,inject,block);
+ memcpy(hidden,hyper,(size_t)W*sizeof(float));
+ double started=now_s();
+ q38_gr_read(m,&m->mtp_mixer,hyper,1,mixed,NULL);
+ q38_weight_matmul(logit,mixed,&m->lm_head,1,H,c->vocab);
+ q38_tm_add(m,Q38_TM_LM_HEAD,started);
+ int draft=q38_argmax(logit,c->vocab);
+ free(hyper);free(mixed);free(inject);free(block);free(logit);
+ return draft;
+}
+
static float *step(Model *m,const int *ids,int S,int pos_base) {
float *streams=NULL;
float *logit=q38_forward(m,ids,S,pos_base,1,m->mtp?&streams:NULL);
@@ -3335,108 +3460,267 @@ static float *step(Model *m,const int *ids,int S,int pos_base) {
return logit;
}
-/* ---- speculative decoding with the MTP head -------------------------------
- * colibri.c's DRAFT loop on a model with recurrent state. The caller asks
- * for the logits that follow the token it just picked; q38_spec_step has
- * the head draft the token after that one and feeds both in one forward
- * (S=2, the verify), returns the first row's logits as step() would and
- * keeps the second row's. The caller's next pick settles the draft. Equal:
- * the kept logits answer with no forward. Not equal: the second row is
- * undone -- the DeltaNet and PLE state go back to the copy the verify took
- * after its first row (q38_deltanet and q38_ple take it when snap_after
- * says so; the rollback swaps pointers), kv_len and the kv_prefix record go
- * back one row, and the attention and indexer rows past it are a stale tail
- * the next forward overwrites, as every rewind here treats them. The head's
- * own rows never need undoing: it reads settled rows only.
+/* ---- speculative decoding: MTP drafts and prompt lookup -------------------
+ * colibri.c's DRAFT loop on a model with recurrent state. The caller asks for
+ * the logits that follow the token it just picked (q38_spec_step); a draft
+ * source proposes up to five tokens after it -- the MTP head (Q38_MTP=1) up to
+ * three, the first from the model's streams and each next one from the head's
+ * own streams of the row before (q38_mtp_draft_more), or prompt lookup
+ * (COLI_LOOKUP=1) the tokens that followed the last n-gram where it occurred
+ * before (spec_draft.h) -- and one forward over the token and its k drafts
+ * (the verify, S = k+1 rows) returns the first row's logits as step() would
+ * and keeps the others. The caller's next picks settle the drafts one at a
+ * time: a pick equal to the next draft is answered with that draft's row and
+ * no forward; the first pick that differs undoes the rows after the ones that
+ * stood. The DeltaNet and PLE state go back to the copy the verify took after
+ * its last standing row (q38_deltanet and q38_ple copy it after each of the
+ * first snap_rows rows, row r into slot r; the rollback swaps pointers),
+ * kv_len and the kv_prefix record go back to it, and the attention and
+ * indexer rows past it are a stale tail the next forward overwrites, as every
+ * rewind here treats them. The head's own rows never need undoing: it reads
+ * settled rows only, and the rows its deeper drafts write past them are such
+ * a tail too.
*
* Every verify row is computed the way a decode step computes it (the CPU
* kernels give a row the same bits whatever S is; device matrices run row by
- * row, see q38_weight_matmul), so the logits the caller sees, and its pick,
- * are those of plain decoding, greedy or sampled: the draft never reaches
- * the sampler, and a sample that equals it is simply an accepted draft. The
- * one exception is the CUDA expert tier, whose float order already depends
- * on which experts are resident when, with drafts or without. */
+ * row, see q38_weight_matmul), so the logits the caller sees, and its picks,
+ * are those of plain decoding, greedy or sampled: no draft reaches the
+ * sampler, and a sample that equals one is simply an accepted draft. The one
+ * exception is the CUDA expert tier, whose float order already depends on
+ * which experts are resident when, with drafts or without.
+ *
+ * How many drafts: Q38_MTP_DRAFTS (1..3, default 2) fixes the head's; 0 lets
+ * the gate pick 0..3 from the measured acceptance per draft position and the
+ * measured cost of a verify by its rows (spec_draft.h). Lookup drafts are
+ * always gated (COLI_SPEC_GATE=0 drafts every proposal in full: tests). With
+ * both sources a verify carries the proposal whose expected tokens per unit of
+ * time is higher, the longer one on a tie. */
typedef struct {
- int on, force; /* drafting; Q38_MTP_FORCE's mode letter or 0 */
- int ahead, draft, ahead_pos; /* a verify whose second row (the draft at ahead_pos) awaits the caller's pick */
+ int on, force, force_row; /* MTP drafting; Q38_MTP_FORCE's mode letter (or 0) and its row */
+ int depth; /* Q38_MTP_DRAFTS: drafts per verify, 0 = the gate picks 1..3 */
+ int lookup, lookup_max; /* COLI_LOOKUP: prompt-lookup drafts, up to COLI_LOOKUP_DRAFTS */
+ int lk_force, lk_force_row; /* COLI_LOOKUP_FORCE (tests): the oracle's tokens as the proposal */
+ int *hist, hist_n, hist_cap; /* the tokens fed so far and the one about to be: what lookup searches */
+ SpecGate gate;
+ /* a verify whose rows past the first await the caller's picks: rows
+ * 0..ahead_n-1 at ahead_pos.., ids[] their tokens, ahead_i the next row to
+ * hand out (its draft ids[ahead_i] must be the caller's pick) */
+ int ahead_n, ahead_i, ahead_pos, ahead_src, head_ok;
+ int ids[Q38_SPEC_ROWS];
float *ahead_logit, *ahead_streams;
- uint64_t drafts, accepted, forwards, tokens;
+ uint64_t drafts, accepted, forwards, tokens; /* the MTP head's drafts (the report's counts) */
+ uint64_t verifies; /* the MTP head's verifies */
+ uint64_t lk_drafts, lk_accepted, lk_verifies; /* prompt lookup's */
+ uint64_t depth_prop[Q38_SPEC_ROWS], depth_hit[Q38_SPEC_ROWS]; /* MTP drafts by position */
+ int ended_ahead; /* the generation ended with a verify's rows unconsumed (undone) */
} Q38Spec;
-/* Back to the state after the verify's first row, `len` positions fed. */
-static void q38_spec_rollback(Model *m,int len) {
+/* Copy slots for a verify of up to rows+1 rows: allocated the first time they are
+ * needed, kept for the model's life. 0 = out of memory (the caller drafts less). */
+static int q38_spec_alloc(Model *m,int rows) {
Cfg *c=&m->c;
+ if(rows>Q38_SPEC_SNAPS)rows=Q38_SPEC_SNAPS;
+ for(int s=m->snap_slots;s<rows;s++){
+ m->snap_rec[s]=(float**)calloc((size_t)c->layers,sizeof(float*));
+ m->snap_conv[s]=(float**)calloc((size_t)c->layers,sizeof(float*));
+ if(!m->snap_rec[s]||!m->snap_conv[s])goto refused;
+ for(int i=0;i<c->layers;i++)if(!c->is_attn[i]){
+ m->snap_rec[s][i]=(float*)malloc((size_t)c->dn_vheads*c->dn_kdim*c->dn_vdim*sizeof(float));
+ m->snap_conv[s][i]=(float*)malloc((size_t)c->dn_conv_dim*(c->dn_convk-1)*sizeof(float));
+ if(!m->snap_rec[s][i]||!m->snap_conv[s][i])goto refused;
+ }
+ if(m->PLE_conv_state){
+ m->snap_ple[s]=(float*)malloc((size_t)c->hc_width*(c->ple_convk-1)*c->ngram_size*sizeof(float));
+ if(!m->snap_ple[s])goto refused;
+ }
+ m->snap_slots=s+1;
+ continue;
+refused:
+ /* The next token may try again. Keep complete slots, but release the
+ * unfinished one so retries neither lose its pointers nor consume
+ * the memory the plain decode step needs. */
+ for(int i=0;i<c->layers;i++){
+ free(m->snap_rec[s]?m->snap_rec[s][i]:NULL);
+ free(m->snap_conv[s]?m->snap_conv[s][i]:NULL);
+ }
+ free(m->snap_rec[s]);free(m->snap_conv[s]);free(m->snap_ple[s]);
+ m->snap_rec[s]=m->snap_conv[s]=NULL;m->snap_ple[s]=NULL;
+ return 0;
+ }
+ return 1;
+}
+
+/* Back to the state after the verify's first `keep` rows, `len` positions fed. */
+static void q38_spec_rollback(Model *m,int len,int keep) {
+ Cfg *c=&m->c; int slot=keep-1;
#ifdef COLI_VULKAN
- q38c_rollback(m); /* the dense chain's own copies: its device buffers swap too */
+ q38c_rollback(m,slot,len); /* the dense chain's own copies: its device buffers swap too */
#endif
for(int i=0;i<c->layers;i++)if(!c->is_attn[i]){
- float *t=m->DN_rec[i];m->DN_rec[i]=m->snap_rec[i];m->snap_rec[i]=t;
- t=m->DN_conv[i];m->DN_conv[i]=m->snap_conv[i];m->snap_conv[i]=t;
+ float *t=m->DN_rec[i];m->DN_rec[i]=m->snap_rec[slot][i];m->snap_rec[slot][i]=t;
+ t=m->DN_conv[i];m->DN_conv[i]=m->snap_conv[slot][i];m->snap_conv[slot][i]=t;
}
- if(m->snap_ple){
- float *t=m->PLE_conv_state;m->PLE_conv_state=m->snap_ple;m->snap_ple=t;
- memcpy(m->ple_history,m->snap_ple_history,sizeof(m->snap_ple_history));
- m->ple_history_len=m->snap_ple_history_len;
+ if(m->snap_ple[slot]){
+ float *t=m->PLE_conv_state;m->PLE_conv_state=m->snap_ple[slot];m->snap_ple[slot]=t;
+ memcpy(m->ple_history,m->snap_ple_history[slot],sizeof(m->snap_ple_history[slot]));
+ m->ple_history_len=m->snap_ple_history_len[slot];
}
m->kv_len=len;
if(m->kvp.len>len)m->kvp.len=len;
}
-/* The verify's second row did not stand: undo it, and its first row's
- * streams wait for the token the caller picked instead. */
-static void q38_spec_drop_ahead(Model *m,Q38Spec *sp) {
- q38_spec_rollback(m,sp->ahead_pos);
- memcpy(m->mtp_pend,sp->ahead_streams,(size_t)m->c.hc_width*sizeof(float));
- m->mtp_pend_n=1;
+/* The verify's first `keep` rows stand (rejected: the next one did not): undo
+ * the rest, and hand the standing rows' streams to the head, each but the last
+ * with the token that followed it. */
+static void q38_spec_settle(Model *m,Q38Spec *sp,int keep,int rejected) {
+ int n=sp->ahead_n,W=m->c.hc_width;
+ if(keep<n)q38_spec_rollback(m,sp->ahead_pos+keep,keep);
+ if(sp->head_ok){
+ memcpy(m->mtp_pend,sp->ahead_streams,(size_t)keep*W*sizeof(float));
+ m->mtp_pend_n=keep;
+ for(int j=0;j+1<keep;j++)m->mtp_pend_tok[j]=sp->ids[j+1];
+ }
+ spec_gate_result(&sp->gate,sp->ahead_src,keep-1+(rejected?1:0),keep-1);
free(sp->ahead_logit);free(sp->ahead_streams);
- sp->ahead_logit=sp->ahead_streams=NULL;sp->ahead=0;
+ sp->ahead_logit=sp->ahead_streams=NULL;sp->ahead_n=sp->ahead_i=0;
+}
+
+/* The token the forced modes put at draft row j (1-based) of a verify at pos, or
+ * -1 to keep the source's: the oracle's token there (ref.json), or a wrong one at
+ * the row the mode rejects. */
+static int q38_spec_forced(int mode,int row_mode,int verify,int k,int j,int pos,int V) {
+ int truth=pos+j<g_q38_mtp_oracle_n?g_q38_mtp_oracle[pos+j]:-1;
+ if(truth<0||!mode||mode=='r')return -1;
+ int wrong=0;
+ if(mode=='m')wrong=(verify&1)?1:0; /* alternate verifies: the first draft wrong */
+ else if(mode=='w')wrong=row_mode; /* row N: drafts before it right, N wrong */
+ else if(mode=='c')wrong=verify%(k+1)+1; /* cycle the rejected row, k+1 = none */
+ return j==wrong?(truth+1)%V:truth;
+}
+
+static void q38_spec_hist_push(Q38Spec *sp,int tok,int pos) {
+ if(!sp->lookup||pos<0)return;
+ if(pos>=sp->hist_cap){
+ int cap=sp->hist_cap?sp->hist_cap:256; while(cap<=pos)cap*=2;
+ int *h=(int*)realloc(sp->hist,(size_t)cap*sizeof(int));
+ if(!h){sp->lookup=0;return;}
+ sp->hist=h;sp->hist_cap=cap;
+ }
+ if(pos>sp->hist_n){sp->lookup=0;return;} /* a gap: the caller's history is not ours */
+ sp->hist[pos]=tok;sp->hist_n=pos+1;
}
/* The logits that follow `tok`, fed at `pos`, exactly as step(m,&tok,1,pos)
* gives them. `more` is how many tokens the caller may still want after
- * `tok`: a draft saves a forward only when there is a second one. The
- * returned buffer is the caller's (it may hold a second row past the first
+ * `tok`: a verify of k drafts is worth its rows only when k+1 more are wanted.
+ * The returned buffer is the caller's (it may hold more rows past the first
* vocab floats). */
static float *q38_spec_step(Model *m,Q38Spec *sp,int tok,int pos,int more) {
- Cfg *c=&m->c; int V=c->vocab,W=c->hc_width;
+ Cfg *c=&m->c; int V=c->vocab;
sp->tokens++;
- if(sp->ahead){
- if(tok==sp->draft&&pos==sp->ahead_pos&&sp->force!='r'){
- /* both verify rows stand: the first one's next token is the
- * draft, the second waits for the caller's next pick */
- float *logit=sp->ahead_logit;
- memcpy(m->mtp_pend,sp->ahead_streams,(size_t)2*W*sizeof(float));
- m->mtp_pend_n=2;m->mtp_pend_tok=tok;
- free(sp->ahead_streams);
- sp->ahead_logit=sp->ahead_streams=NULL;sp->ahead=0;sp->accepted++;
+ q38_spec_hist_push(sp,tok,pos);
+ if(sp->ahead_n){
+ int i=sp->ahead_i;
+ if(tok==sp->ids[i]&&pos==sp->ahead_pos+i&&sp->force!='r'){
+ /* draft i stands: its row's logits answer, with no forward */
+ float *logit=falloc(V);
+ memcpy(logit,sp->ahead_logit+(int64_t)(i-1)*V,(size_t)V*sizeof(float));
+ if(sp->ahead_src==SPEC_SRC_MTP){sp->accepted++;sp->depth_hit[i-1]++;}
+ else sp->lk_accepted++;
+ if(++sp->ahead_i==sp->ahead_n)q38_spec_settle(m,sp,sp->ahead_n,0);
return logit;
}
- q38_spec_drop_ahead(m,sp);
- }
- int draft=-1;
- if(sp->on&&more>=2&&pos+1<m->kv_cap){
- draft=q38_mtp_draft(m,tok,pos);
- if(draft>=0&&(sp->force=='a'||sp->force=='m')){
- int truth=pos+1<g_q38_mtp_oracle_n?g_q38_mtp_oracle[pos+1]:-1;
- if(truth>=0)draft=sp->force=='m'&&(sp->drafts&1)?(truth+1)%V:truth;
+ q38_spec_settle(m,sp,i,1);
+ }
+ /* how many drafts this verify can carry: rows the caller still wants, room in the cache */
+ int room=more-1;
+ if(room>Q38_SPEC_ROWS-1)room=Q38_SPEC_ROWS-1;
+ if(pos+room>=m->kv_cap)room=m->kv_cap-1-pos;
+ int d[Q38_SPEC_ROWS],k=0,src=SPEC_SRC_MTP;
+ double t_verify=0.0; /* when the verify's cost began (0: at its forward) */
+ int head_ready=sp->on&&m->mtp_pend_n&&m->mtp_len+m->mtp_pend_n==pos;
+ if(room>0&&(head_ready||sp->lookup)){
+ /* the lookup's proposal, gated; the forced mode proposes the oracle's tokens */
+ int kl=0,dl[Q38_SPEC_ROWS];double vl=0.0;
+ if(sp->lookup){
+ int want=room<sp->lookup_max?room:sp->lookup_max,n=0;
+ if(sp->lk_force)
+ while(n<want){
+ int t=q38_spec_forced(sp->lk_force,sp->lk_force_row,(int)sp->lk_verifies,want,n+1,pos,V);
+ if(t<0)break;
+ dl[n++]=t;
+ }
+ else n=spec_lookup(sp->hist,sp->hist_n,2,4,want,dl);
+ if(n>0)kl=spec_gate_pick(&sp->gate,SPEC_SRC_LOOKUP,n,&vl);
+ }
+ /* the head's depth: fixed, or the gate's pick */
+ int km=0;double vm=0.0;
+ if(head_ready){
+ int maxm=room<3?room:3;
+ if(sp->depth){km=sp->depth<maxm?sp->depth:maxm;vm=spec_gate_value(&sp->gate,SPEC_SRC_MTP,km,NULL);}
+ else km=spec_gate_pick(&sp->gate,SPEC_SRC_MTP,maxm,&vm);
+ }
+ if(kl>0&&(km==0||sp->lk_force||vl>vm||(vl==vm&&kl>km))){k=kl;src=SPEC_SRC_LOOKUP;memcpy(d,dl,(size_t)k*sizeof(int));}
+ else if(km>0)k=km;
+ if(k>0&&!q38_spec_alloc(m,k))k=0; /* no memory for the copies: a plain step */
+ if(k>0&&src==SPEC_SRC_MTP){
+ float *hidden=falloc(c->hc_width);
+ t_verify=now_s(); /* the head's first draft is the head pass a plain step makes too */
+ d[0]=q38_mtp_draft(m,tok,pos,hidden);
+ double t_first=now_s();
+ for(int j=1;j<k;j++){
+ double t0=now_s();
+ d[j]=q38_mtp_draft_more(m,hidden,d[j-1],pos+j-1);
+ spec_gate_draft_cost(&sp->gate,SPEC_SRC_MTP,now_s()-t0);
+ }
+ free(hidden);
+ t_verify+=now_s()-t_first; /* the deeper drafts are the gate's d, not the verify's */
+ if(sp->force)for(int j=1;j<=k;j++){
+ int t=q38_spec_forced(sp->force,sp->force_row,(int)sp->verifies,k,j,pos,V);
+ if(t>=0)d[j-1]=t;
+ }
}
}
- if(draft<0){sp->forwards++;return step(m,&tok,1,pos);}
- int ids[2]={tok,draft};float *streams=NULL;
- m->snap_after=1;g_q38_rowwise=1;
- float *logit=q38_forward(m,ids,2,pos,2,&streams);
- m->snap_after=0;g_q38_rowwise=0;
- sp->drafts++;sp->forwards++;
- sp->ahead=1;sp->draft=draft;sp->ahead_pos=pos+1;
- sp->ahead_logit=falloc(V);memcpy(sp->ahead_logit,logit+V,(size_t)V*sizeof(float));
+ if(!k){
+ sp->forwards++;
+ double t0=now_s();
+ float *logit=step(m,&tok,1,pos);
+ spec_gate_forward(&sp->gate,1,now_s()-t0);
+ return logit;
+ }
+ /* the head reads its pending rows before the verify's: the MTP draft did, a
+ * lookup verify has them take `tok` here */
+ if(!t_verify)t_verify=now_s();
+ if(m->mtp&&src!=SPEC_SRC_MTP&&m->mtp_pend_n&&m->mtp_len+m->mtp_pend_n==pos)
+ q38_mtp_take_pending(m,tok,NULL,NULL);
+ int S=k+1;
+ sp->ids[0]=tok;memcpy(sp->ids+1,d,(size_t)k*sizeof(int));
+ float *streams=NULL;
+ m->snap_rows=k;g_q38_rowwise=1;
+ float *logit=q38_forward(m,sp->ids,S,pos,S,m->mtp?&streams:NULL);
+ spec_gate_forward(&sp->gate,S,now_s()-t_verify);
+ m->snap_rows=0;g_q38_rowwise=0;
+ sp->forwards++;
+ if(src==SPEC_SRC_MTP){
+ sp->drafts+=(uint64_t)k;sp->verifies++;
+ for(int j=0;j<k;j++)sp->depth_prop[j]++;
+ } else {sp->lk_drafts+=(uint64_t)k;sp->lk_verifies++;}
+ sp->ahead_n=S;sp->ahead_i=1;sp->ahead_pos=pos;sp->ahead_src=src;
+ sp->head_ok=m->mtp&&m->mtp_len==pos&&streams;
+ sp->ahead_logit=falloc((int64_t)k*V);
+ memcpy(sp->ahead_logit,logit+V,(size_t)k*V*sizeof(float));
sp->ahead_streams=streams;
return logit;
}
-/* End of a generation: a verify row the caller never consumed is undone, so
+/* End of a generation: verify rows the caller never consumed are undone, so
* the state is the one plain decoding leaves after the same tokens. */
static void q38_spec_end(Model *m,Q38Spec *sp) {
- if(sp->ahead)q38_spec_drop_ahead(m,sp);
+ if(sp->ahead_n){sp->ended_ahead=1;q38_spec_settle(m,sp,sp->ahead_i,0);}
+}
+
+static void q38_spec_free(Q38Spec *sp) {
+ free(sp->hist);sp->hist=NULL;sp->hist_n=sp->hist_cap=0;
}
static float q38_mean(const float *x,int n) {
@@ -3496,16 +3780,9 @@ static void q38_mtp_attach(Model *m,int cap) {
* sized to fill RAM with int4 experts it costs more than a model layer. */
int mtp_cap=q38_env_positive_int("Q38_MTP_CAP",cap,c->experts);
q38_load_layer(m,c->layers,mtp_cap);
- m->mtp_pend=falloc((int64_t)2*W);
- /* the verify's rollback copies (q38_spec_rollback) */
- m->snap_rec=(float**)calloc((size_t)c->layers,sizeof(float*));
- m->snap_conv=(float**)calloc((size_t)c->layers,sizeof(float*));
- if(!m->snap_rec||!m->snap_conv){fprintf(stderr,"OOM MTP rollback state\n");exit(1);}
- for(int i=0;i<c->layers;i++)if(!c->is_attn[i]){
- m->snap_rec[i]=falloc((int64_t)c->dn_vheads*c->dn_kdim*c->dn_vdim);
- m->snap_conv[i]=falloc((int64_t)c->dn_conv_dim*(c->dn_convk-1));
- }
- if(m->PLE_conv_state)m->snap_ple=falloc((int64_t)W*(c->ple_convk-1)*c->ngram_size);
+ m->mtp_pend=falloc((int64_t)Q38_SPEC_ROWS*W);
+ /* the verify's first rollback copy (q38_spec_rollback); deeper verifies add theirs */
+ if(!q38_spec_alloc(m,1)){fprintf(stderr,"OOM MTP rollback state\n");exit(1);}
snprintf(nm,sizeof nm,"%s.layers.0.mlp.experts.0.gate_proj.weight",found);
st_tensor *expert=st_find(&m->S,nm);
double expert_bytes=expert?3.0*(double)expert->nbytes:0.0; /* gate, up and down are the same size */
@@ -3607,10 +3884,13 @@ static void q38_model_free(Model *m) {
}
if(m->expert_scales)free(m->expert_scales[i].values);
if(i<m->c.layers){free(m->DN_rec ? m->DN_rec[i] : NULL); free(m->DN_conv ? m->DN_conv[i] : NULL);}
- if(i<m->c.layers){free(m->snap_rec ? m->snap_rec[i] : NULL); free(m->snap_conv ? m->snap_conv[i] : NULL);}
+ if(i<m->c.layers)for(int sl=0;sl<m->snap_slots;sl++){
+ free(m->snap_rec[sl] ? m->snap_rec[sl][i] : NULL); free(m->snap_conv[sl] ? m->snap_conv[sl][i] : NULL);
+ }
free(m->K ? m->K[i] : NULL); free(m->V ? m->V[i] : NULL); free(m->IK ? m->IK[i] : NULL);
}
- free(m->snap_rec); free(m->snap_conv); free(m->snap_ple); free(m->mtp_pend);
+ for(int sl=0;sl<Q38_SPEC_SNAPS;sl++){ free(m->snap_rec[sl]); free(m->snap_conv[sl]); free(m->snap_ple[sl]); }
+ free(m->mtp_pend);
free(m->mtp_norm_emb); free(m->mtp_norm_hid);
q38_weight_free(&m->mtp_fc_emb); q38_weight_free(&m->mtp_fc_hid);
free(m->mtp_mixer.norm); q38_weight_free(&m->mtp_mixer.down); q38_weight_free(&m->mtp_mixer.up);
diff --git a/c/resource_plan.py b/c/resource_plan.py
index 60ce35b6..df21cc43 100644
--- a/c/resource_plan.py
+++ b/c/resource_plan.py
@@ -75,7 +75,7 @@ class _CgroupMount:
# sidecar and self-invalidate on any change. Best-effort: any read/write failure falls
# straight back to a full recompute (see analyze_model). Sits alongside .coli_usage/.coli_ssd.
_ANALYSIS_CACHE_NAME = ".coli_analysis.json"
-_ANALYSIS_CACHE_VERSION = 8
+_ANALYSIS_CACHE_VERSION = 11
def _dense_in_ram(descriptor, on_disk_bytes):
@@ -84,6 +84,11 @@ def _dense_in_ram(descriptor, on_disk_bytes):
Coincidono per chi li carica come stanno; una famiglia che li riquantizza
al caricamento lo dichiara nel registro. Un errore qui non e' cosmetico:
e' la differenza fra dire a qualcuno che il modello ci sta e dirgli di no."""
+ # GLM53's quantization is per matrix, with F32 vectors/vision and an int8
+ # fallback for narrow rows. A whole-checkpoint ratio cannot represent it;
+ # _glm53_dense_tensors prices its source inventory below, outside the cache.
+ if descriptor.id == "glm53":
+ return on_disk_bytes
ratio = getattr(descriptor, "dense_load_ratio", None)
if ratio is None:
return on_disk_bytes
@@ -93,6 +98,270 @@ def _dense_in_ram(descriptor, on_disk_bytes):
return on_disk_bytes
+# A token embedding table: "embed_tokens", DeepSeek's "embed", "tok_embeddings",
+# "wte"; not a vision tower's patch or position embedding.
+_EMBED_TABLE = re.compile(r"(^|\.)(embed_tokens|embed|tok_embeddings|wte|word_embeddings)\.weight$")
+
+
+def _vk_flag(env, name):
+ value = (env.get(name) or "").strip()
+ if not value:
+ return None
+ try:
+ return int(value)
+ except ValueError:
+ return 1
+
+
+# The families whose dense chain is on by default on an integrated GPU with the
+# expert tier (coli_vk_chain_decide's measured `igpu`, docs/vulkan.md).
+_VK_IGPU_CHAIN_ON = ("qwen36", "olmoe")
+_VK_DENSE_HOST_FAMILIES = frozenset((
+ "qwen36", "qwen38", "glm", "glm53", "inkling", "kimi", "mimo",
+ "olmoe", "deepseek_v4", "deepseek_v41",
+))
+
+
+def _vk_device(env, vulkan):
+ if vulkan is not None or not _vk_flag(env, "COLI_VULKAN"):
+ return vulkan
+ try:
+ from setup_hw import best_vulkan_device, detect_vulkan
+ return best_vulkan_device(detect_vulkan().get("devices", []))
+ except Exception:
+ return None
+
+
+def _vk_dense_active(family_id, env, vulkan):
+ if (family_id not in _VK_DENSE_HOST_FAMILIES or
+ not _vk_flag(env, "COLI_VULKAN") or _vk_flag(env, "COLI_CUDA")):
+ return False
+ kind = (vulkan or {}).get("type")
+ if kind not in ("discrete", "integrated", "cpu", "virtual", "other"):
+ return False
+ tier = _vk_flag(env, "COLI_VK_TIER") != 0
+ chain, dense = _vk_flag(env, "COLI_VK_CHAIN"), _vk_flag(env, "COLI_VK_DENSE")
+ if chain is None:
+ chain = (kind not in ("integrated", "cpu") or
+ kind == "integrated" and tier and family_id in _VK_IGPU_CHAIN_ON)
+ if dense is None:
+ # colibri.c passes default=0 to coli_vk_dense_decide; its chain is
+ # independent, but disabling that chain does not enable per-matrix GEMVs.
+ dense = family_id != "glm" and (kind not in ("integrated", "cpu") or not tier)
+ return bool(chain or dense)
+
+
+def _text_weight_name(name):
+ return name.removeprefix("language_model.").replace("model.language_model.", "model.", 1)
+
+
+_Q38_TRUNK_COMPONENTS = {
+ "self_attn.q_proj": "attnq", "self_attn.k_proj": "attnk",
+ "self_attn.v_proj": "attnv", "self_attn.o_proj": "attno",
+ "self_attn.indexer.index_qk_proj": "qsaidx",
+ "linear_attn.in_proj_qkv": "dnqkv", "linear_attn.in_proj_z": "dnz",
+ "linear_attn.out_proj": "dnout", "mlp.gate": "router",
+ "mlp.shared_expert.gate_proj": "shg", "mlp.shared_expert.up_proj": "shu",
+ "mlp.shared_expert.down_proj": "shd",
+}
+for _block, _tag in (("attn", "hca"), ("mlp", "hcm")):
+ for _part, _suffix in (("input_mix_weight_down", "d"),
+ ("input_mix_weight_up", "u"), ("block_inject_weight", "i")):
+ _Q38_TRUNK_COMPONENTS[f"{_block}_hyper_connection.{_part}"] = _tag + _suffix
+
+
+def _layer_component(name):
+ match = re.fullmatch(r"model\.layers\.\d+\.(.+)\.weight", _text_weight_name(name))
+ return match[1] if match else None
+
+
+def _matrix_shape(tensor):
+ shape = tensor.get("shape")
+ element = {"BF16": 2, "F16": 2, "F32": 4, "F8_E4M3": 1}.get(tensor["dtype"])
+ if (not element or not isinstance(shape, list) or len(shape) != 2 or
+ any(type(n) is not int or n <= 0 for n in shape) or
+ shape[0] * shape[1] * element != tensor["size"]):
+ return None
+ return shape
+
+
+def _q38_cpu_dense_tensors(info, env):
+ """Price q38_trunk_offer_all's CPU rows, independently of CUDA placement.
+
+ The scan cache is environment independent. Selection happens here on each
+ plan: the threshold includes a float scale per output row, the skip list is
+ component based, and F16 checkpoints are loaded as F32 before quantization.
+ """
+ tensors = info.get("dense_tensors", [])
+ if info["resolved_family"].descriptor.id != "qwen38" or env.get("Q38_TRUNK_CPU_INT8") == "0":
+ return info
+ threshold = re.match(r"\s*([+-]?\d+)", env.get("Q38_TRUNK_MIN_KB", "1024"))
+ min_kb = int(threshold[1]) if threshold else 0 # C atol; negative casts to size_t
+ skipped = set(env.get("Q38_TRUNK_SKIP", "").split(","))
+ adjusted, delta = [], 0
+ for tensor in tensors:
+ shape = _matrix_shape(tensor)
+ name = _text_weight_name(tensor["name"])
+ tag = "lmhead" if name == "lm_head.weight" else _Q38_TRUNK_COMPONENTS.get(_layer_component(name))
+ if shape and tag and tag not in skipped and min_kb >= 0:
+ rows, columns = shape
+ quantized = rows * columns + 4 * rows
+ if quantized >= min_kb * 1024:
+ delta += quantized - tensor["resident"]
+ tensor = dict(tensor, resident=quantized)
+ adjusted.append(tensor)
+ return dict(info, dense_bytes=info["dense_bytes"] + delta, dense_tensors=adjusted)
+
+
+_ATTENTION_PROJECTIONS = frozenset("self_attn." + p + "_proj" for p in ("q", "k", "v", "o"))
+_MLP_PROJECTIONS = frozenset("mlp." + p + "_proj" for p in ("gate", "up", "down"))
+_SHARED_PROJECTIONS = frozenset("mlp.shared_experts." + p + "_proj" for p in ("gate", "up", "down"))
+_G53_MATRICES = (_ATTENTION_PROJECTIONS | _MLP_PROJECTIONS | _SHARED_PROJECTIONS |
+ frozenset("self_attn." + p + "_proj" for p in ("q_a", "q_b", "g_a", "g_b", "f_a", "f_b", "b")) |
+ {"self_attn.kv_a_proj_with_mqa", "self_attn.indexer.wq_b",
+ "self_attn.indexer.wk", "self_attn.indexer.weights_proj"})
+
+
+def _glm53_dense_tensors(info, env):
+ """Mirror glm53.c's quantize_loaded for known matrices, reserve F32 otherwise.
+
+ Cached scans contain source geometry. Never cache GLM53_BITS or multiply
+ vectors, embeddings and the vision tower by a quantized-matrix ratio. kv_b's
+ absorbed halves and unrecognized components retain a conservative F32 budget.
+ """
+ if info["resolved_family"].descriptor.id != "glm53":
+ return info
+ setting = re.match(r"\s*([+-]?\d+)", env.get("GLM53_BITS", "4"))
+ bits = int(setting[1]) if setting else 0
+ if bits not in (4, 8, 32):
+ raise ValueError("GLM53_BITS must be 4, 8 or 32")
+ adjusted, total, embedding = [], 0, 0
+ for tensor in info.get("dense_tensors", []):
+ name = _text_weight_name(tensor["name"])
+ size, dtype = tensor["size"], tensor["dtype"]
+ element = {"BF16": 2, "F16": 2, "F32": 4}.get(dtype)
+ resident = (size // element * 4) if element and size % element == 0 else size
+ shape = _matrix_shape(tensor)
+ known = (name == "lm_head.weight" or _layer_component(name) in _G53_MATRICES or
+ re.fullmatch(r"model\.layers\.\d+\.self_attn\.indexer\.index_kpool_compress_gate", name))
+ if element and shape and known and bits != 32:
+ rows, columns = shape
+ if bits == 4 and columns % 64 == 0:
+ resident = rows * (columns // 2) + rows * (columns // 64) * 4
+ else:
+ resident = rows * columns + rows * 4
+ adjusted.append(dict(tensor, resident=resident))
+ total += resident
+ if _EMBED_TABLE.search(name):
+ embedding += resident
+ return dict(info, dense_bytes=total, embed_bytes=embedding, dense_tensors=adjusted)
+
+
+def _vk_released_tensor_bytes(tensor, family, env):
+ """A lower bound on the host allocation an engine's dho pass can free.
+
+ Only audited matrix names and source formats get credit. Keep norms, vision,
+ routers used by the CPU, unknown packed layouts and hardware-dependent BF16
+ kernels in RAM. Upload refusal can still retain a copy at runtime.
+ """
+ shape = _matrix_shape(tensor)
+ if not shape:
+ return 0
+ name, dtype = _text_weight_name(tensor["name"]), tensor["dtype"]
+ part = _layer_component(name)
+ rows, columns = shape
+ count, resident = rows * columns, tensor["resident"]
+ head = name == "lm_head.weight"
+ floating = dtype in ("BF16", "F16", "F32")
+ if family == "qwen38":
+ known = (head or part in _Q38_TRUNK_COMPONENTS or
+ part in ("linear_attn.in_proj_a", "linear_attn.in_proj_b", "ple.key_proj", "ple.value_proj") or
+ name in ("model.hyper_connection_mixer.input_mix_weight_down.weight",
+ "model.hyper_connection_mixer.input_mix_weight_up.weight"))
+ return resident if known and floating else 0
+ if family == "qwen36":
+ known = (head or part in _ATTENTION_PROJECTIONS or part in _MLP_PROJECTIONS or
+ part in ("mlp.gate", "linear_attn.in_proj_qkv", "linear_attn.in_proj_z", "linear_attn.out_proj") or
+ part in {p.replace("shared_experts", "shared_expert") for p in _SHARED_PROJECTIONS})
+ setting = re.match(r"\s*([+-]?\d+)", env.get("COLI_DENSE_BITS", "8"))
+ bits = int(setting[1]) if setting else 8
+ if not known or not floating or bits == 16:
+ return 0
+ # The checkpoint can be F32 while the engine holds int4/int8. Crediting
+ # the checkpoint bytes would also erase RAM that was never duplicated.
+ lower = count // 2 if bits == 4 else count
+ return min(resident, lower)
+ if family == "olmoe":
+ return min(resident, count * 4) if floating and (head or part in _ATTENTION_PROJECTIONS or part == "mlp.gate") else 0
+ if family == "glm53":
+ # kv_b's absorbed halves and a tied head can remain CPU-readable.
+ # Conservatively reserve those; every matrix listed here is in g53_dho_pass.
+ if not floating or part not in _G53_MATRICES:
+ return 0
+ bits = env.get("GLM53_BITS", "4")
+ lower = count * 4 if bits == "32" else count if bits == "8" else count // 2
+ return min(resident, lower)
+ if family == "kimi":
+ if env.get("K3_MMAP") not in (None, "", "0") or not floating:
+ return 0
+ known = (head or part in _ATTENTION_PROJECTIONS or part in _MLP_PROJECTIONS or
+ part in ("self_attn.g_proj", "self_attn.q_a_proj", "self_attn.q_b_proj", "self_attn.kv_a_proj_with_mqa", "self_attn.kv_b_proj") or
+ part in ("block_sparse_moe.routed_expert_down_proj", "block_sparse_moe.routed_expert_up_proj") or
+ part in {p.replace("mlp.", "block_sparse_moe.", 1) for p in _SHARED_PROJECTIONS})
+ return min(resident, count // 2) if known else 0
+ if family == "inkling":
+ known = head or part in _ATTENTION_PROJECTIONS or part == "self_attn.r_proj" or part in _MLP_PROJECTIONS or part in _SHARED_PROJECTIONS
+ # BF16 may use AVX512-BF16 on the CPU and be ineligible for Vulkan;
+ # packed 2/3-bit or unknown group layouts may be ineligible as well.
+ return resident if known and dtype == "F32" else 0
+ if family == "mimo":
+ known = head or part in ("self_attn.qkv_proj", "self_attn.o_proj") or part in _MLP_PROJECTIONS
+ return min(resident, count) if known else 0
+ if family == "deepseek_v4":
+ # coli_v4_dense_device_only_tensor: layer FP8 matrices and the two
+ # BF16 compressor projections. Head, mHC, routers and scales stay.
+ known = re.fullmatch(r"layers\.\d+\.(?:attn|ffn)\..+\.weight", name)
+ droppable = dtype == "F8_E4M3" or (dtype == "BF16" and name.endswith(("compressor.wkv.weight", "compressor.wgate.weight")))
+ return resident if known and droppable else 0
+ if family == "deepseek_v41":
+ # v41_dho_bytes names the mapped trunk. Sidecar block scales stay.
+ known = re.fullmatch(r"layers\.\d+\.(?:attn\.(?:wq_a|wq_b|wkv|wo_a|wo_b|compressor\.(?:wkv|wgate)|indexer\.(?:wk|wq_b|weights_proj))|ffn\.shared_experts\.w[123]|engram\.wkv)\.weight", name)
+ return resident if known and dtype in ("BF16", "F8_E4M3") else 0
+ # GLM's CLI-selected QT format (including unsupported planar/IQ/E8/2-bit
+ # forms) is not described by this scan. Do not guess a releasable allocation.
+ return 0
+
+
+def vk_dense_device_only(dense_bytes, family_id=None, env=None, vulkan=None):
+ """(device_only, why): whether a Vulkan engine keeps its dense weights on the
+ device only, with no host copy (docs/vulkan.md, "Dense weights on the device
+ only"), as the engine's coli_vk_dense_host_decide() decides it: COLI_VULKAN=1,
+ the dense part on the device (the chain or the per-matrix path), and
+ COLI_VK_DENSE_HOST=0, or unset with an integrated GPU or a discrete one whose
+ free memory, less 1 GiB, holds them. `vulkan` is the device setup_hw would
+ report (probed when None and needed)."""
+ env = os.environ if env is None else env
+ host = _vk_flag(env, "COLI_VK_DENSE_HOST")
+ if host is not None and host != 0:
+ return False, None
+ vulkan = _vk_device(env, vulkan)
+ kind = (vulkan or {}).get("type")
+ if not _vk_dense_active(family_id, env, vulkan):
+ return False, None
+ if host == 0:
+ return True, "COLI_VK_DENSE_HOST=0"
+ if kind == "integrated":
+ return True, "an integrated GPU: its memory is the same RAM"
+ if kind == "discrete":
+ used = sum(heap.get("usage", 0) for heap in vulkan.get("heaps", [])
+ if heap.get("device_local"))
+ free = ((vulkan.get("budget_bytes") or vulkan.get("device_local_bytes") or 0)
+ - used - (1 << 30))
+ if dense_bytes <= free:
+ return True, "a discrete GPU with room for the dense weights"
+ return False, None
+
+
def _analysis_signature(shards, config_path):
parts = [f"v{_ANALYSIS_CACHE_VERSION}"]
st = config_path.stat()
@@ -103,7 +372,7 @@ def _analysis_signature(shards, config_path):
return "|".join(parts)
-def _tensor_sizes(path):
+def _tensor_sizes(path, with_shape=False):
file_size = path.stat().st_size
with path.open("rb") as stream:
raw = stream.read(8)
@@ -122,7 +391,8 @@ def _tensor_sizes(path):
dtype = meta.get("dtype")
if not isinstance(dtype, str):
raise ValueError(f"invalid tensor dtype for {name}: {path}")
- yield name, end - start, dtype
+ item = (name, end - start, dtype)
+ yield (*item, meta.get("shape")) if with_shape else item
def analyze_model(model):
@@ -165,6 +435,10 @@ def analyze_model(model):
pass # missing/corrupt/unreadable cache -> recompute
dense_bytes = 0
+ # The embedding tables among them: the engines gather their rows on the CPU, so
+ # they keep a host copy when the rest of the dense weights live on a Vulkan
+ # device only (vk_dense_device_only below).
+ embed_bytes = 0
# Model-owned allocations that are retained once, independently of the
# number of per-layer cache slots. Qwen3.8's native FP8 path uses this for
# its normalized scale bank; fallback accounting remains conservative.
@@ -172,11 +446,14 @@ def analyze_model(model):
# What the engine's GPU trunk offload would put in VRAM (int8), for the
# families that have one; taken out of the VRAM budget before experts.
trunk_int8_bytes = 0
+ # Keep the source geometry, dtype and resident contribution for placement
+ # decisions. These facts can be cached; environment-dependent credits cannot.
+ dense_tensors = []
expert_groups = {}
tensor_names = set()
for shard in shards:
try:
- sizes = list(_tensor_sizes(shard))
+ sizes = list(_tensor_sizes(shard, with_shape=True))
except OSError as error:
# Name the file. An OSError raised by read() on an already-open
# stream carries no filename, so `coli doctor` reported bare
@@ -187,7 +464,7 @@ def analyze_model(model):
# storage, all of them is the mount.
raise OSError(error.errno,
f"{error.strerror or error}: {shard}") from error
- for name, size, dtype in sizes:
+ for name, size, dtype, shape in sizes:
tensor_names.add(name)
contributions = expert_contributions(resolved, name, size, dtype)
if contributions:
@@ -200,8 +477,14 @@ def analyze_model(model):
if fixed_bytes:
expert_fixed_bytes += fixed_bytes
else:
- dense_bytes += resident_contribution(
- resolved, name, size, dtype)
+ resident = resident_contribution(resolved, name, size, dtype)
+ dense_bytes += resident
+ if resident:
+ dense_tensors.append({"name": name, "size": size, "dtype": dtype,
+ "shape": shape,
+ "resident": _dense_in_ram(resolved.descriptor, resident)})
+ if _EMBED_TABLE.search(name):
+ embed_bytes += resident
trunk_int8_bytes += trunk_contribution(
resolved, name, size, dtype)
@@ -245,8 +528,10 @@ def analyze_model(model):
# serve a rispondere "ci sta?", non "quanto pesa il file".
"dense_bytes": _dense_in_ram(resolved.descriptor, dense_bytes),
"dense_disk_bytes": dense_bytes,
+ "embed_bytes": _dense_in_ram(resolved.descriptor, embed_bytes),
"expert_fixed_bytes": expert_fixed_bytes,
"trunk_int8_bytes": trunk_int8_bytes,
+ "dense_tensors": dense_tensors,
"expert_bytes": sum(expert_groups.values()),
"expert_count": len(expert_groups),
"expert_layers": len(per_layer),
@@ -257,6 +542,9 @@ def analyze_model(model):
"config": config,
"resolved_family": resolved,
}
+ # analyze_model has one stable, default-format answer regardless of the
+ # caller's process environment. build_plan reapplies its requested settings.
+ result = _glm53_dense_tensors(result, {})
try: # best-effort write; a read-only model dir must never break planning.
# Atomic write (tmp file + os.replace): a concurrent `coli plan` on the
# same model dir must never observe a half-written cache -- write_text()
@@ -1528,10 +1816,33 @@ def _family_expert_cache_knob(family_id, cache_bytes):
def build_plan(model, ram_gb=0, context=4096, gpu_indices=None, vram_gb=0,
available_memory=None, available_disk=None, gpus=None,
policy="quality", physical_cpus=None, cpu_sockets=None,
- kv_slots=1):
+ kv_slots=1, env=None, vulkan=None):
if policy not in POLICIES:
raise ValueError(f"unknown policy: {policy}")
info = qwen38_int4_sidecar(analyze_model(model))
+ env_now = os.environ if env is None else env
+ info = _q38_cpu_dense_tensors(info, env_now)
+ info = _glm53_dense_tensors(info, env_now)
+ # Only the matrices a family's dho pass can release earn RAM credit. The
+ # embedding, norms, CPU-only components and unrecognized formats stay reserved.
+ vulkan = _vk_device(env_now, vulkan)
+ family_id = info["resolved_family"].descriptor.id
+ kept = min(info["dense_bytes"], info.get("embed_bytes", 0))
+ device_dense_bytes = max(0, info["dense_bytes"] - kept)
+ vk_dense, vk_dense_why = vk_dense_device_only(
+ device_dense_bytes, family_id, env_now, vulkan)
+ dense_on_device = 0
+ if vk_dense:
+ dense_on_device = min(device_dense_bytes, sum(
+ _vk_released_tensor_bytes(tensor, family_id, env_now)
+ for tensor in info.get("dense_tensors", [])))
+ info = dict(info, dense_bytes=info["dense_bytes"] - dense_on_device)
+ # On an integrated/software device the remaining device copy still consumes
+ # physical RAM. Keeping host copies costs two copies; dropping them saves one,
+ # not both. Price the device copy even when COLI_VK_DENSE_HOST=1.
+ shared_dense_bytes = (device_dense_bytes
+ if (vulkan or {}).get("type") in ("integrated", "cpu")
+ and _vk_dense_active(family_id, env_now, vulkan) else 0)
physical_cpus = physical_cpu_count() if physical_cpus is None else physical_cpus
cpu_sockets = cpu_socket_count() if cpu_sockets is None else cpu_sockets
resolved = info["resolved_family"]
@@ -1591,7 +1902,7 @@ def build_plan(model, ram_gb=0, context=4096, gpu_indices=None, vram_gb=0,
placement_unified = any(gpu.get("unified_memory", False)
for gpu in planning_gpus)
- unified = placement_unified or _host_unified_memory()
+ unified = placement_unified or _host_unified_memory() or bool(shared_dense_bytes)
typical = info["typical_expert_bytes"]
max_expert = info["max_expert_bytes"] or typical
kv_bytes = (geometry.context_state_bytes + geometry.fixed_state_bytes) * kv_slots
@@ -1601,7 +1912,7 @@ def build_plan(model, ram_gb=0, context=4096, gpu_indices=None, vram_gb=0,
# every cache slot. Include it in the resident runtime reservation so the
# selected capacity cannot overrun the model's actual allocation.
runtime_bytes = int(1.2 * GB + 2.5 * GB + 64 * max_expert +
- info["expert_fixed_bytes"] + kv_bytes + kv_buffer)
+ info["expert_fixed_bytes"] + kv_bytes + kv_buffer + shared_dense_bytes)
per_cap = info["per_cap_bytes"]
configured_experts = geometry.configured_experts
@@ -1765,6 +2076,9 @@ def build_plan(model, ram_gb=0, context=4096, gpu_indices=None, vram_gb=0,
"available_bytes": available_disk, "cold_expert_bytes": cold_bytes},
"ram": {"role": "resident+warm-experts", "available_bytes": available_memory,
"budget_bytes": ram_budget, "dense_bytes": info["dense_bytes"],
+ "dense_on_device_bytes": dense_on_device,
+ "dense_on_device_reason": vk_dense_why if vk_dense else None,
+ "shared_device_dense_bytes": shared_dense_bytes,
"runtime_bytes": runtime_bytes,
"expert_fixed_bytes": info["expert_fixed_bytes"],
"sequence_state_bytes": geometry.context_state_bytes,
@@ -1884,7 +2198,9 @@ def format_plan(plan):
f"disk {format_bytes(tiers['disk']['cold_expert_bytes'])} cold experts · "
f"{format_bytes(tiers['disk']['available_bytes'])} free",
f"RAM {format_bytes(tiers['ram']['budget_bytes'])} budget · "
- f"{format_bytes(tiers['ram']['dense_bytes'])} dense · "
+ f"{format_bytes(tiers['ram']['dense_bytes'])} dense"
+ + (f" (+{format_bytes(tiers['ram']['dense_on_device_bytes'])} on the Vulkan device only)"
+ if tiers['ram'].get('dense_on_device_bytes') else "") + " · "
f"{format_bytes(tiers['ram']['runtime_bytes'])} runtime · "
f"{format_bytes(tiers['ram']['warm_expert_bytes'])} warm experts · "
f"cap {tiers['ram']['cache_slots_per_layer']}/layer"]
diff --git a/c/shaders/chain_ares.comp b/c/shaders/chain_ares.comp
index 3c6f7464..3279aab0 100644
--- a/c/shaders/chain_ares.comp
+++ b/c/shaders/chain_ares.comp
@@ -80,12 +80,15 @@ void mixrow() {
}
}
+// Preserve small SiTU inputs; tanh(x) rounds to x below 2^-12 in f32.
+float situ_tanh(float x) { return abs(x) < 0.000244140625 ? x : tanh(x); }
+
void situ() {
int i = int(gl_GlobalInvocationID.x + gl_GlobalInvocationID.y * gl_NumWorkGroups.x * 256u);
if (i >= N_) return;
float g = xv[X_OFF + i], u = bv[B_OFF + i];
float sg = 1.0 / (1.0 + exp(-g));
- yv[Y_OFF + i] = B1 * tanh(g / B1) * sg * B2 * tanh(u / B2);
+ yv[Y_OFF + i] = B1 * situ_tanh(g / B1) * sg * B2 * situ_tanh(u / B2);
}
void main() {
diff --git a/c/shaders/chain_ares.spv b/c/shaders/chain_ares.spv
index 8d162a61..58eb5f9e 100644
Binary files a/c/shaders/chain_ares.spv and b/c/shaders/chain_ares.spv differ
diff --git a/c/shaders/chain_attnb.comp b/c/shaders/chain_attnb.comp
new file mode 100644
index 00000000..599ecda9
--- /dev/null
+++ b/c/shaders/chain_attnb.comp
@@ -0,0 +1,163 @@
+#version 450
+// chain_attn.comp's attention for a prompt chunk: the same results (to rounding), the
+// K and V rows read once per block of query rows instead of once per (head, row).
+// chain_attn runs one workgroup per (head, row), and each reads every position the row
+// sees: a chunk of S rows reads its KV cache S * H times (68 GB a layer for 2048 rows of
+// Qwen3.6, 1.2 s on a Radeon 780M). Here one workgroup takes the G = H / KVH query
+// heads of one KV head for BR consecutive rows, P = G * BR (head, row) pairs, at most 32,
+// and walks the positions of its rows in tiles of BC (a specialization constant, 4 to 16
+// as the device's shared memory allows): each tile's K and V rows go to shared memory
+// once and serve every pair. 8 lanes per pair; lane l owns the query dims l, l + 8, ...
+// (at most 32) and the output dims likewise, in registers:
+// - each lane's partial dot products with the tile's keys go to shared memory, summed
+// per pair (score = sum of the 8 partials, times the scale);
+// - a position the pair's row does not see (past it, before its window, past its
+// list) scores nothing; the online softmax (running max and sum, the accumulator
+// rescaled when the max grows) as chain_attn's;
+// - the output, divided by the sum, times sigmoid(gate) when has_gate.
+// Everything chain_attn takes: a sliding window (win), a ring (ring), V's own head dim
+// (vd), position-major rows (kv_pm), a sink logit (sink, sink_off), and a selection list
+// per row (sel_row > 0): then BR = 1, the row's list is the workgroup's.
+// The order of the sums differs from chain_attn's and the CPU's; all agree to rounding.
+// Push constants: chain_attn's, then br (rows per workgroup, br * H / KVH <= 32).
+// Limits: hd, vd <= 256.
+layout(local_size_x = 256) in;
+layout(constant_id = 0) const int BC = 8;
+
+layout(std430, binding = 0) readonly buffer QB { float q[]; };
+layout(std430, binding = 1) readonly buffer KB { float kc[]; };
+layout(std430, binding = 2) readonly buffer VB { float vc[]; };
+layout(std430, binding = 3) buffer OB { float o[]; };
+layout(std430, binding = 4) readonly buffer GB { float gt[]; };
+layout(std430, binding = 5) readonly buffer LB { int sel[]; };
+layout(std430, binding = 6) readonly buffer SB { float snk[]; };
+
+layout(push_constant) uniform P {
+ int S, H, KVH, hd, pos_base, cap;
+ int q_off, q_row, q_seg;
+ int g_off, g_row, g_seg, has_gate;
+ int o_off, o_row;
+ int sel_off, sel_row;
+ float scale;
+ int k_off, v_off;
+ int win, ring, vd, kv_pm, sink, sink_off;
+ int br;
+} p;
+
+shared float ks[BC * 256]; // the tile's keys [j][d]
+shared float vs[BC * 256]; // the tile's values [j][d]
+shared float part[32 * 8 * BC]; // [pair][lane][position]
+shared float scr[32 * BC]; // [pair][position]
+shared int tpos[BC]; // the tile's positions (-1: none)
+
+void main() {
+ int tid = int(gl_LocalInvocationID.x);
+ int pair = tid >> 3, lane = tid & 7;
+ int kvh = int(gl_WorkGroupID.x);
+ int G = p.H / p.KVH;
+ int s0 = int(gl_WorkGroupID.y) * p.br;
+ int r = pair / G, g = pair - r * G;
+ int s = s0 + r;
+ bool live = r < p.br && s < p.S;
+ int h = kvh * G + g;
+ float qv[32], av[32];
+ int qb = p.q_off + s * p.q_row + h * p.q_seg;
+ for (int i = 0; i < 32; i++) {
+ int d = lane + 8 * i;
+ qv[i] = live && d < p.hd ? q[qb + d] : 0.0;
+ av[i] = 0.0;
+ }
+ // the block's positions: the union of its rows' ranges, or the one row's list
+ int slast = min(s0 + p.br, p.S) - 1;
+ int pos = p.pos_base + s;
+ int first = p.win > 0 ? max(0, pos - p.win + 1) : 0; // this pair's row
+ int bfirst = p.win > 0 ? max(0, p.pos_base + s0 - p.win + 1) : 0;
+ int n = p.pos_base + slast + 1 - bfirst;
+ bool list = false;
+ int lb = 0;
+ if (p.sel_row > 0) {
+ lb = p.sel_off + s0 * p.sel_row;
+ int c = sel[lb];
+ if (c >= 0) { n = c; list = true; }
+ }
+ float m = p.sink != 0 && live ? snk[p.sink_off + h] : -3.0e38, l = p.sink != 0 ? 1.0 : 0.0;
+ int kbase = kvh * p.cap;
+ for (int t0 = 0; t0 < n; t0 += BC) {
+ if (tid < BC) {
+ int i = t0 + tid;
+ tpos[tid] = i < n ? (list ? sel[lb + 1 + i] : bfirst + i) : -1;
+ }
+ barrier();
+ // the tile's keys and values into shared memory, once for every pair
+ for (int x = tid; x < BC * p.hd; x += 256) {
+ int j = x / p.hd, d = x - j * p.hd, t = tpos[j];
+ float kv = 0.0;
+ if (t >= 0) {
+ int rr = p.ring > 0 ? t % p.ring : t;
+ kv = kc[p.k_off + (p.kv_pm != 0 ? rr * p.KVH + kvh : kbase + rr) * p.hd + d];
+ }
+ ks[j * 256 + d] = kv;
+ }
+ for (int x = tid; x < BC * p.vd; x += 256) {
+ int j = x / p.vd, d = x - j * p.vd, t = tpos[j];
+ float vv = 0.0;
+ if (t >= 0) {
+ int rr = p.ring > 0 ? t % p.ring : t;
+ vv = vc[p.v_off + (p.kv_pm != 0 ? rr * p.KVH + kvh : kbase + rr) * p.vd + d];
+ }
+ vs[j * 256 + d] = vv;
+ }
+ barrier();
+ // the partial dot products of this lane's query dims with the tile's keys
+ for (int j = 0; j < BC; j++) {
+ float a = 0.0;
+ for (int i = 0; i < 32; i++) {
+ int d = lane + 8 * i;
+ if (d < p.hd) a += qv[i] * ks[j * 256 + d];
+ }
+ part[(pair * 8 + lane) * BC + j] = a;
+ }
+ barrier();
+ // the scores: lane j sums position j's (and j + 8's) partials
+ for (int j = lane; j < BC; j += 8) {
+ int t = tpos[j];
+ float sc = -3.0e38;
+ if (live && t >= 0 && (list || (t >= first && t <= pos))) {
+ float a = 0.0;
+ for (int k = 0; k < 8; k++) a += part[(pair * 8 + k) * BC + j];
+ sc = a * p.scale;
+ }
+ scr[pair * BC + j] = sc;
+ }
+ barrier();
+ // the online softmax over the tile, each lane of the pair alike
+ float mx = m;
+ for (int j = 0; j < BC; j++) mx = max(mx, scr[pair * BC + j]);
+ float corr = exp(m - mx), sum = 0.0;
+ for (int i = 0; i < 32; i++) av[i] *= corr;
+ for (int j = 0; j < BC; j++) {
+ float sc = scr[pair * BC + j];
+ if (sc <= -3.0e38) continue;
+ float w = exp(sc - mx);
+ sum += w;
+ for (int i = 0; i < 32; i++) {
+ int d = lane + 8 * i;
+ if (d < p.vd) av[i] += w * vs[j * 256 + d];
+ }
+ }
+ l = l * corr + sum;
+ m = mx;
+ barrier();
+ }
+ if (!live) return;
+ float inv = l > 0.0 ? 1.0 / l : 0.0;
+ int ob = p.o_off + s * p.o_row + h * p.vd;
+ int gb = p.g_off + s * p.g_row + h * p.g_seg;
+ for (int i = 0; i < 32; i++) {
+ int d = lane + 8 * i;
+ if (d >= p.vd) break;
+ float v = av[i] * inv;
+ if (p.has_gate != 0) v *= 1.0 / (1.0 + exp(-gt[gb + d]));
+ o[ob + d] = v;
+ }
+}
diff --git a/c/shaders/chain_attnb.spv b/c/shaders/chain_attnb.spv
new file mode 100644
index 00000000..697ed5c9
Binary files /dev/null and b/c/shaders/chain_attnb.spv differ
diff --git a/c/shaders/chain_enc.comp b/c/shaders/chain_enc.comp
new file mode 100644
index 00000000..2662086e
--- /dev/null
+++ b/c/shaders/chain_enc.comp
@@ -0,0 +1,309 @@
+#version 450
+// The bidirectional encoders of the decision engines on the dense chain (vk_chain.c,
+// vkc_enc_*): Laya's ModernBERT and its decision head (laya.c), GLiNER2.5-Decide's
+// DeBERTa-v3 (gliner_decide.c). A forward has no cache and no state: every row of every
+// sequence of a request goes through the encoder at once, so the attention reads the
+// step's own K and V, and a row sees a range of rows instead of a causal prefix.
+// The matrices are vkc_matmul's; this shader is everything between them. Six modes
+// (push constants: v[0] the mode, the rest per mode, vk_chain.h's VkcEnc* structs):
+//
+// mode 0, LayerNorm. One workgroup per row r < rows:
+// t = a[a_off + r*a_row + i] (+ b[b_off + r*b_row + i] with flags & 1)
+// with flags & 2, t is written back to a (the residual stream of a pre-norm layer:
+// x += h, then h = LayerNorm(x))
+// y[y_off + r*y_row + i] = (t - mean) * rstd * w[w_off + i] (+ prm[bias_off + i] when
+// bias_off >= 0), rstd = 1 / sqrt(var + eps), the biased variance.
+// The engines sum in double; this is a float tree: they agree to rounding. D <= 4096.
+// mode 1, bias and activation, in place: y[y_off + r*y_row + o] = act(y[...] +
+// prm[b_off + o]) (no bias when b_off < 0), one thread per (r, o < O); act 0 none,
+// 1 GELU (erf), 2 GELU (tanh), 3 ReLU.
+// mode 2, GeGLU: y[y_off + r*y_row + j] = gelu(u_j + bu_j) * (u_{I+j} + bu_{I+j}), u the
+// row a[a_off + r*a_row ...], bu the bias at prm[b_off] (none when b_off < 0); act 1
+// erf, 2 tanh (ModernBERT's MLP: Wi's output split in halves, the first activated).
+// mode 3, RoPE, rotate-half, in place on the query and key heads of each row:
+// v = x[x_off + r*x_row + part*part_off + h*hd], c = tab[cos_off + pos[r]*half + i],
+// s = tab[sin_off + pos[r]*half + i]: v[i] = a*c - b*s, v[i+half] = b*c + a*s,
+// a = v[i], b = v[i+half]: laya.c's apply_rope, the host's own cosf/sinf in the table.
+// mode 4, a table row per row: y[y_off + r*y_row + i] += tab[t_off + idx[r]*D + i]
+// (Laya's type embedding).
+// mode 5, the attention, every head of every row over a range of rows (its own
+// sequence, and a window inside it): one workgroup per (head h, block of 16 rows).
+// Row i sees the rows j in rng[2i] .. rng[2i+1] (inclusive; an empty range gives 0),
+// q, k and v at q_off, k_off, v_off + row*kv_row + h*hd:
+// score(i, j) = (q_i . k_j [+ c2p[h][i][r] with flags & 1] [+ p2c[h][j][r] with
+// flags & 2]) * scale, r = ridx[pos[i] - pos[j] + rc] - r0
+// (DeBERTa's disentangled attention: content to content, content to position,
+// position to content, the relative position bucketed by the host's table; c2p and p2c
+// are mode 6's products, [H][S][nr] at c_off and p_off), out = sum_j softmax(score)_j
+// v_j at o_off + i*o_row + h*hd. The softmax runs online over tiles of 32 rows
+// (running max and sum); the engines normalize first, then sum: the two agree to
+// rounding. hd <= 128.
+// mode 6, the relative terms' products, as the engine's attention() computes them:
+// y[y_off + (h*S + i)*nr + t] = sum_d x[x_off + i*x_row + h*hd + d] * p[p_off + (r0 + t)*p_row
+// + h*hd + d] for every head h, row i < S and t < nr: each row's query against the key
+// projection of the relative embeddings (c2p), or its key against their query
+// projection (p2c), for the nr buckets r0 .. r0 + nr - 1 the step can reach. A
+// workgroup takes 16 rows by 64 buckets of one head, d in steps of 32.
+layout(local_size_x = 256) in;
+
+layout(std430, binding = 0) buffer AB { float a[]; }; // 0: a (and the sum); 2: u; 3: x; 5: qkv
+layout(std430, binding = 1) readonly buffer BB { float b[]; }; // 0: b; 3: the cos/sin table; 5: pk
+layout(std430, binding = 2) readonly buffer PB { float prm[]; }; // 0: w, bias; 1, 2: bias; 4: table; 5: pq
+layout(std430, binding = 3) buffer YB { float y[]; }; // 0, 1, 2, 4: y; 5: out
+layout(std430, binding = 4) readonly buffer IA { int ia[]; }; // 3: pos; 4: idx; 5: rng
+layout(std430, binding = 5) readonly buffer IB { int ib[]; }; // 5: ridx
+layout(std430, binding = 6) readonly buffer IC { int ic[]; }; // 5: pos
+
+layout(push_constant) uniform P { int v[32]; } p;
+#define MODE p.v[0]
+
+shared float red[256];
+#define BQ 16
+#define BK 32
+#define HDM 128
+#define HDP (HDM + 1)
+shared float qs[BQ * HDM];
+shared float kv[BK * HDP];
+shared float sc[BQ * BK];
+shared float mrow[BQ], lrow[BQ], crow[BQ];
+#define BR 64 // mode 6: buckets per workgroup
+#define DC 32 // mode 6: inputs per staged step
+shared int rlo[BQ], rhi[BQ], rpos[BQ];
+shared int klo, khi;
+
+// erfc(x) to a relative error under 1.2e-7 everywhere (Chebyshev fit, Numerical
+// Recipes' erfcc), so 0.5 x erfc(-x / sqrt 2) is GELU to the CPU's erff within float
+// rounding at any magnitude
+float erfc_(float x) {
+ float z = abs(x), t = 1.0 / (1.0 + 0.5 * z);
+ float r = t * exp(-z * z - 1.26551223 + t * (1.00002368 + t * (0.37409196 + t * (0.09678418 +
+ t * (-0.18628806 + t * (0.27886807 + t * (-1.13520398 + t * (1.48851587 +
+ t * (-0.82215223 + t * 0.17087277)))))))));
+ return x >= 0.0 ? r : 2.0 - r;
+}
+float act(float x, int k) {
+ if (k == 1) return 0.5 * x * erfc_(-x * 0.70710678118654752);
+ if (k == 2) return 0.5 * x * (1.0 + tanh(0.7978845608028654 * (x + 0.044715 * x * x * x)));
+ if (k == 3) return max(x, 0.0);
+ return x;
+}
+float sum256(float v, int tid) {
+ red[tid] = v;
+ barrier();
+ for (int off = 128; off > 0; off >>= 1) {
+ if (tid < off) red[tid] += red[tid + off];
+ barrier();
+ }
+ float s = red[0];
+ barrier();
+ return s;
+}
+int gid() { return int(gl_GlobalInvocationID.x + gl_GlobalInvocationID.y * gl_NumWorkGroups.x * 256u); }
+
+void layernorm() {
+ int rows = p.v[1], D = p.v[2], a_off = p.v[3], a_row = p.v[4], b_off = p.v[5], b_row = p.v[6];
+ int y_off = p.v[7], y_row = p.v[8], w_off = p.v[9], bias_off = p.v[10], flags = p.v[11];
+ float eps = intBitsToFloat(p.v[12]);
+ int r = int(gl_WorkGroupID.x + gl_WorkGroupID.y * gl_NumWorkGroups.x), tid = int(gl_LocalInvocationID.x);
+ if (r >= rows) return;
+ float t[16];
+ float s = 0.0;
+ for (int k = 0; k < 16; k++) {
+ int i = tid + 256 * k;
+ t[k] = 0.0;
+ if (i < D) {
+ float v = a[a_off + r * a_row + i];
+ if ((flags & 1) != 0) v = v + b[b_off + r * b_row + i];
+ t[k] = v;
+ s += v;
+ }
+ }
+ float mean = sum256(s, tid) / float(D);
+ float q = 0.0;
+ for (int k = 0; k < 16; k++) { int i = tid + 256 * k; if (i < D) { float c = t[k] - mean; q += c * c; } }
+ float var = sum256(q, tid) / float(D);
+ float rstd = 1.0 / sqrt(var + eps);
+ // every read of the row is done (the sums' barriers): the row may now be rewritten
+ for (int k = 0; k < 16; k++) {
+ int i = tid + 256 * k;
+ if (i >= D) continue;
+ if ((flags & 2) != 0) a[a_off + r * a_row + i] = t[k];
+ float o = (t[k] - mean) * rstd * prm[w_off + i];
+ if (bias_off >= 0) o = o + prm[bias_off + i];
+ y[y_off + r * y_row + i] = o;
+ }
+}
+
+int g_r0 = 0;
+int r0_row(int i) { return g_r0 + i; }
+
+void attention() {
+ int S = p.v[1], hd = p.v[3];
+ int q_off = p.v[4], k_off = p.v[5], v_off = p.v[6], kv_row = p.v[7], o_off = p.v[8], o_row = p.v[9];
+ int flags = p.v[10], c_off = p.v[11], p_off = p.v[12], nr = p.v[13], rc = p.v[14];
+ float scale = intBitsToFloat(p.v[15]);
+ int r0 = p.v[16];
+ int h = int(gl_WorkGroupID.x), b0 = int(gl_WorkGroupID.y) * BQ, tid = int(gl_LocalInvocationID.x);
+ g_r0 = b0;
+ int nrow = min(BQ, S - b0);
+ for (int e = tid; e < nrow * hd; e += 256) {
+ int i = e / hd, d = e - i * hd;
+ qs[i * HDM + d] = a[q_off + (b0 + i) * kv_row + h * hd + d];
+ }
+ if (tid < BQ) {
+ if (tid < nrow) {
+ rlo[tid] = ia[2 * (b0 + tid)]; rhi[tid] = ia[2 * (b0 + tid) + 1];
+ rpos[tid] = (flags & 3) != 0 ? ic[b0 + tid] : 0;
+ } else { rlo[tid] = 1; rhi[tid] = 0; rpos[tid] = 0; }
+ mrow[tid] = -3.0e38; lrow[tid] = 0.0;
+ }
+ barrier();
+ if (tid == 0) {
+ int lo = 0x7fffffff, hi = -1;
+ for (int i = 0; i < nrow; i++) if (rlo[i] <= rhi[i]) { lo = min(lo, rlo[i]); hi = max(hi, rhi[i]); }
+ klo = lo; khi = hi;
+ }
+ barrier();
+ int pi = tid >> 4, pd = tid & 15; // the PV step: row pi, dims pd + 16k
+ float acc[8];
+ for (int k = 0; k < 8; k++) acc[k] = 0.0;
+ for (int j0 = klo; j0 <= khi; j0 += BK) {
+ int nk = min(BK, khi - j0 + 1);
+ for (int e = tid; e < nk * hd; e += 256) {
+ int jj = e / hd, d = e - jj * hd;
+ kv[jj * HDP + d] = a[k_off + (j0 + jj) * kv_row + h * hd + d];
+ }
+ barrier();
+ for (int u = 0; u < 2; u++) {
+ int pr = tid + 256 * u, i = pr / BK, jj = pr - i * BK, j = j0 + jj;
+ float sv = 0.0;
+ if (i < nrow && jj < nk && j >= rlo[i] && j <= rhi[i]) {
+ float s = 0.0;
+ for (int d = 0; d < hd; d++) s += qs[i * HDM + d] * kv[jj * HDP + d];
+ if ((flags & 3) != 0) {
+ int rr = ib[rpos[i] - ic[j] + rc] - r0;
+ if ((flags & 1) != 0) s += b[c_off + (h * S + r0_row(i)) * nr + rr];
+ if ((flags & 2) != 0) s += prm[p_off + (h * S + j) * nr + rr];
+ }
+ sv = s * scale;
+ }
+ sc[i * BK + jj] = sv;
+ }
+ barrier();
+ if (tid < BQ) {
+ int i = tid;
+ float m = mrow[i], mn = m;
+ int jlo = max(rlo[i] - j0, 0), jhi = min(rhi[i] - j0, nk - 1);
+ for (int jj = jlo; jj <= jhi; jj++) mn = max(mn, sc[i * BK + jj]);
+ float l = 0.0;
+ for (int jj = 0; jj < BK; jj++) {
+ float e = jj >= jlo && jj <= jhi ? exp(sc[i * BK + jj] - mn) : 0.0;
+ sc[i * BK + jj] = e;
+ l += e;
+ }
+ float c = exp(m - mn);
+ crow[i] = c;
+ lrow[i] = lrow[i] * c + l;
+ mrow[i] = mn;
+ }
+ barrier();
+ for (int e = tid; e < nk * hd; e += 256) {
+ int jj = e / hd, d = e - jj * hd;
+ kv[jj * HDP + d] = a[v_off + (j0 + jj) * kv_row + h * hd + d];
+ }
+ barrier();
+ if (pi < nrow) {
+ float c = crow[pi];
+ for (int k = 0; k < 8; k++) {
+ int d = pd + 16 * k;
+ if (d >= hd) break;
+ float t = acc[k] * c;
+ for (int jj = 0; jj < nk; jj++) t += sc[pi * BK + jj] * kv[jj * HDP + d];
+ acc[k] = t;
+ }
+ }
+ barrier();
+ }
+ if (pi < nrow) {
+ float l = lrow[pi];
+ float inv = l > 0.0 ? 1.0 / l : 0.0;
+ for (int k = 0; k < 8; k++) {
+ int d = pd + 16 * k;
+ if (d >= hd) break;
+ y[o_off + (b0 + pi) * o_row + h * hd + d] = acc[k] * inv;
+ }
+ }
+}
+
+void relprod() {
+ int S = p.v[1], hd = p.v[3], x_off = p.v[4], x_row = p.v[5], p_off = p.v[6], p_row = p.v[7];
+ int y_off = p.v[8], nr = p.v[9], r0 = p.v[10];
+ int t0 = int(gl_WorkGroupID.x) * BR, i0 = int(gl_WorkGroupID.y) * BQ, h = int(gl_WorkGroupID.z);
+ int tid = int(gl_LocalInvocationID.x), tt = tid & 63, iq = tid >> 6; // bucket tt, rows 4iq .. 4iq+3
+ float acc[4] = float[4](0.0, 0.0, 0.0, 0.0);
+ for (int d0 = 0; d0 < hd; d0 += DC) {
+ int dn = min(DC, hd - d0);
+ for (int e = tid; e < BQ * DC; e += 256) {
+ int i = e / DC, d = e - i * DC;
+ qs[i * (DC + 1) + d] = i0 + i < S && d < dn ? a[x_off + (i0 + i) * x_row + h * hd + d0 + d] : 0.0;
+ }
+ for (int e = tid; e < BR * DC; e += 256) {
+ int t = e / DC, d = e - t * DC;
+ kv[t * (DC + 1) + d] = t0 + t < nr && d < dn ? b[p_off + (r0 + t0 + t) * p_row + h * hd + d0 + d] : 0.0;
+ }
+ barrier();
+ for (int d = 0; d < dn; d++) {
+ float pv = kv[tt * (DC + 1) + d];
+ for (int k = 0; k < 4; k++) acc[k] += qs[(4 * iq + k) * (DC + 1) + d] * pv;
+ }
+ barrier();
+ }
+ if (t0 + tt < nr)
+ for (int k = 0; k < 4; k++) {
+ int i = i0 + 4 * iq + k;
+ if (i < S) y[y_off + (h * S + i) * nr + t0 + tt] = acc[k];
+ }
+}
+
+void main() {
+ int mode = MODE;
+ if (mode == 0) { layernorm(); return; }
+ if (mode == 5) { attention(); return; }
+ if (mode == 6) { relprod(); return; }
+ int i = gid();
+ if (mode == 1) {
+ int n = p.v[1], O = p.v[2], y_off = p.v[3], y_row = p.v[4], b_off = p.v[5], k = p.v[6];
+ if (i >= n) return;
+ int r = i / O, o = i - r * O, at = y_off + r * y_row + o;
+ float v = y[at];
+ if (b_off >= 0) v = v + prm[b_off + o];
+ y[at] = act(v, k);
+ } else if (mode == 2) {
+ int n = p.v[1], I = p.v[2], a_off = p.v[3], a_row = p.v[4], y_off = p.v[5], y_row = p.v[6];
+ int b_off = p.v[7], k = p.v[8];
+ if (i >= n) return;
+ int r = i / I, j = i - r * I, ab = a_off + r * a_row;
+ float u = a[ab + j], g = a[ab + I + j];
+ if (b_off >= 0) { u = u + prm[b_off + j]; g = g + prm[b_off + I + j]; }
+ y[y_off + r * y_row + j] = act(u, k) * g;
+ } else if (mode == 3) {
+ int n = p.v[1], H = p.v[2], hd = p.v[3], x_off = p.v[4], x_row = p.v[5], part_off = p.v[6];
+ int cos_off = p.v[7], sin_off = p.v[8];
+ int half_ = hd / 2;
+ if (i >= n) return;
+ int per_row = 2 * H * half_;
+ int r = i / per_row, rem = i - r * per_row;
+ int part = rem / (H * half_), rem2 = rem - part * H * half_, hh = rem2 / half_, k = rem2 - hh * half_;
+ int vb = x_off + r * x_row + part * part_off + hh * hd;
+ int ps = ia[r] * half_ + k;
+ float c = b[cos_off + ps], s = b[sin_off + ps];
+ float x0 = a[vb + k], x1 = a[vb + k + half_];
+ a[vb + k] = x0 * c - x1 * s;
+ a[vb + k + half_] = x1 * c + x0 * s;
+ } else if (mode == 4) {
+ int n = p.v[1], D = p.v[2], y_off = p.v[3], y_row = p.v[4], t_off = p.v[5];
+ if (i >= n) return;
+ int r = i / D, d = i - r * D;
+ y[y_off + r * y_row + d] += prm[t_off + ia[r] * D + d];
+ }
+}
diff --git a/c/shaders/chain_enc.spv b/c/shaders/chain_enc.spv
new file mode 100644
index 00000000..1f526303
Binary files /dev/null and b/c/shaders/chain_enc.spv differ
diff --git a/c/shaders/chain_gemv.comp b/c/shaders/chain_gemv.comp
index 4febbda9..068bc5af 100644
--- a/c/shaders/chain_gemv.comp
+++ b/c/shaders/chain_gemv.comp
@@ -4,7 +4,7 @@
// layout) for the formats a dense trunk uses, read 16 bytes per lane per step:
// 1 int8 rows, one scale per row (16 weights per step)
// 4 int4 in groups of 64, nibble - 8, low nibble = even column (32 per step)
-// 10 f32 (4 per step), 11 bf16, low half = even column (8 per step)
+// 10 f32 (4 per step), 11 bf16 and 14 f16, low half = even column (8 per step)
// qmatmul.comp reads one 32-bit word per lane per step and checks every element
// against I; on an integrated GPU at its floor clock that leaves the GEMV short of
// loads in flight and long on instructions (measured on a Radeon 780M: int8 at 22 to
@@ -44,7 +44,7 @@ void main() {
int s = int(gl_WorkGroupID.y);
int tid = int(gl_LocalInvocationID.x);
int nq = p.rowWords / 4; // uvec4 steps per row
- int per = p.fmt == 1 ? 16 : p.fmt == 4 ? 32 : p.fmt == 11 ? 8 : 4; // values per step
+ int per = p.fmt == 1 ? 16 : p.fmt == 4 ? 32 : p.fmt == 11 || p.fmt == 14 ? 8 : 4; // values per step
int nx4 = nq * per / 4, i4 = p.I / 4; // padded x length in vec4s
for (int k = tid; k < nx4; k += 256) xs[k] = k < i4 ? x4[s * i4 + k] : vec4(0.0);
barrier();
@@ -82,11 +82,13 @@ void main() {
}
a0 += (t0 + t1 + t2 + t3) * sc;
}
- } else if (p.fmt == 11) {
+ } else if (p.fmt == 11 || p.fmt == 14) {
for (int q = lane; q < nql; q += lpr) {
uvec4 w = w4[rb + q]; int xb = q * 2;
vec4 x0 = xs[xb], x1 = xs[xb + 1];
- vec2 b0 = bf(w.x), b1 = bf(w.y), b2 = bf(w.z), b3 = bf(w.w);
+ vec2 b0, b1, b2, b3;
+ if (p.fmt == 11) { b0 = bf(w.x); b1 = bf(w.y); b2 = bf(w.z); b3 = bf(w.w); }
+ else { b0 = unpackHalf2x16(w.x); b1 = unpackHalf2x16(w.y); b2 = unpackHalf2x16(w.z); b3 = unpackHalf2x16(w.w); }
a0 += x0.x * b0.x + x0.y * b0.y;
a1 += x0.z * b1.x + x0.w * b1.y;
a2 += x1.x * b2.x + x1.y * b2.y;
diff --git a/c/shaders/chain_gemv.spv b/c/shaders/chain_gemv.spv
index 8fc2700c..df0ee5a2 100644
Binary files a/c/shaders/chain_gemv.spv and b/c/shaders/chain_gemv.spv differ
diff --git a/c/shaders/chain_kvs.comp b/c/shaders/chain_kvs.comp
new file mode 100644
index 00000000..bb460029
--- /dev/null
+++ b/c/shaders/chain_kvs.comp
@@ -0,0 +1,482 @@
+#version 450
+// Attention over a KV cache split between the device and the host (vk_chain.c,
+// VKC_P_KVS; vk_chain.h: vkc_kvs_*; the host side is vk_kvsplit.h). Past the device's
+// budget a layer's cache keeps only some blocks of B positions on the device: `ns`
+// slots of B rows each, a block table bt (block -> slot, -1: the block is in the
+// host's RAM only) and its inverse rb (slot -> block, -1: a free slot), both in the
+// int buffer `tab`; a block pinned by a selection's reads has bit 30 set in bt. The
+// partition of a row's positions depends on the row's own position only, so a row
+// gets the same bits however the forward around it is cut into steps: the device
+// takes the positions of the `anchor` + 1 newest blocks up to the row's (blocks
+// pos/B - anchor .. pos/B, every one of them on the device while a step holds the
+// row) and, from a list, the positions in pinned blocks; the host takes the rest.
+// Each part writes a partial result: the unnormalized value sum acc and the softmax
+// statistics (m, the running max, and l, the sum of exp(score - m)); mode 2 merges
+// the two:
+// M = max(m_d, m_c), e_x = exp(m_x - M), out = (e_d acc_d + e_c acc_c) / (e_d l_d + e_c l_c)
+// which is the full attention up to the order of the sums. A part with no position
+// has m = -3e38, l = 0, acc = 0 and weighs nothing. Position t sits in device row
+// slot*B + t % B, slot = bt[t / B] without bit 30.
+//
+// mode 0, grouped-query attention (chain_attn.comp's arithmetic over the device's
+// rows): one workgroup per (head h = WorkGroupID.x, row s = WorkGroupID.y), the
+// online softmax over tiles of 256 positions. Causal: the device's positions of
+// first..pos_base+s (first = max(0, pos - win + 1) with a window, else 0), in
+// order; with a selection list (sel_row > 0, sel[sel_off + s*sel_row] = count >= 0,
+// then the positions; a negative count means causal), the device's listed positions
+// in list order (a negative entry is skipped). K and V as
+// chain_attn keeps them (head-major (kvh*rows + r)*hd, or position-major
+// (r*KVH + kvh)*hd with kv_pm), at k_off / v_off; V's head dim vd. With a sink the
+// running max and sum start from snk[sink_off + h] (MiMo's attention sink: it joins
+// the denominator, so it belongs to exactly one part, the device's).
+// flags 1 (fin): every visible position is on the device; the output is normalized
+// (times sigmoid(gate[g_off + s*g_row + h*g_seg + d]) with flags 2) at o_off +
+// s*o_row + h*vd, as chain_attn writes it. Otherwise acc at o_off + s*o_row + h*vd
+// and (m, l) at st_off + (s*H + h)*2. Limits: hd, vd <= 256.
+// mode 1, multi-head latent attention (chain_mla.comp's core over the device's rows):
+// score_t = (qa . L[t] + qr . Rc[t]) * scale, acc = sum softmax-weighted L[t] (K
+// floats); the device's positions of kv_start..pos_base+s, or of the row's list (a
+// negative entry skipped); the latent at lat_off + r*lat_row, the
+// rope key at rope_off + r*rope_row. flags 1: the normalized clat at o_off + s*o_row
+// + h*o_seg, as chain_mla writes it; else acc there and (m, l) at st_off + (s*H+h)*2.
+// Limits: K <= 1024, R <= 128.
+// mode 2, the merge: segment g = (s, h) of d floats, s = g / H. The device's part at
+// a_off + s*a_row + h*a_seg with its (m, l) at sa_off + 2g (buffer 0), the host's
+// packed at c_off + g*d with (m, l) at sc_off + 2g (buffer 1); the result at o_off +
+// s*o_row + h*o_seg, times sigmoid(gate[g_off + s*g_row + h*g_seg + i]) with flags 2,
+// rounded to bf16 with flags 4 (DeepSeek V4's attention output).
+// mode 3, Inkling's attention (chain_relattn.comp's arithmetic over the device's rows,
+// causal, a global layer): score_t = tau[s] * (q . k_t * scale + (d < ext ? bias(d) :
+// 0)), d = pos - t, bias(d) = sum_j r[s][h][j] * relp[j*ext + d] in j's order; q at
+// q_off + s*q_row + h*hd, r at r_off + s*r_row + h*d_rel (buffer 4), relp at relp_off
+// (buffer 6), tau[s] at tau_off + s of buffer 5 (floats); K/V head-major. Outputs as
+// mode 0's (fin: normalized). Limits: hd <= 256, d_rel <= 64.
+// mode 4, DeepSeek's sparse attention with a sink (chain_dsv4.comp mode 0's arithmetic):
+// row s's list l[l_off + s*l_row + j], j < cnt (buffer 5): e < 0 skipped, e < nwin the
+// window ring's row e (w_off + e*hd, buffer 1), else the compressed row c = e - nwin
+// when its block has a slot (c_off + r*hd, buffer 2), skipped otherwise: the
+// with flags 16, -2-j names cold row j staged in buffer 6 (-1 remains skipped).
+// This complete-list path preserves the arithmetic regardless of residency. Scores (q . k)
+// * scale, each dot in one thread in order; best = the largest; w = exp(score - best),
+// the denominator exp(sink - best) + the w in list order (one thread), the value sum
+// in list order per dim with w rounded to bf16 under flags 1 (DeepSeek V4). fin (flags
+// 8): out = sum / den (bf16 with flags 1) at o_off + s*o_row + h*hd: chain_dsv4's bits. Else
+// the part, its max taken over the sink too (a part with no row still has the sink),
+// acc and (m, l) as mode 0's. The sink, in buffer 4 at sink_off + h, belongs to the
+// device's part. Limits: hd <= 1024.
+#extension GL_KHR_shader_subgroup_arithmetic : require
+layout(local_size_x = 256) in;
+
+layout(std430, binding = 0) readonly buffer B0 { float b0[]; }; // q / qa / the device's part
+layout(std430, binding = 1) readonly buffer B1 { float b1[]; }; // K / qr / the host's part
+layout(std430, binding = 2) readonly buffer B2 { float b2[]; }; // V / latent / gate
+layout(std430, binding = 3) buffer B3 { float b3[]; }; // out (modes 0, 2) / rope keys
+layout(std430, binding = 4) readonly buffer B4 { float b4[]; }; // gate (mode 0)
+layout(std430, binding = 5) readonly buffer L5 { int sel[]; }; // selection lists
+layout(std430, binding = 6) buffer B6 { float b6[]; }; // sink (mode 0) / out (mode 1)
+layout(std430, binding = 7) readonly buffer T7 { int tab[]; }; // bt and rb
+
+layout(push_constant) uniform P { int v[32]; } p;
+#define MODE p.v[0]
+
+shared float qs[1024];
+shared float qrs[128];
+shared float pe[256];
+shared int pr[256];
+shared float red[256];
+
+float bf16r(float f) { // round to nearest even, as coli_bf16_round
+ uint u = floatBitsToUint(f);
+ if ((u & 0x7f800000u) != 0x7f800000u) u += 0x7fffu + ((u >> 16) & 1u);
+ return uintBitsToFloat(u & 0xffff0000u);
+}
+int slot_of(int t, int B, int bt_off, int nblk) {
+ if (t < 0) return -1;
+ int b = t / B;
+ int j = b < nblk ? tab[bt_off + b] : -1;
+ return j < 0 ? -1 : (j & 0x3fffffff);
+}
+// the device row of a listed position, or -1 (the host's): the anchored blocks of the
+// row at pos, and pinned ones
+int dev_row(int t, int pos, int B, int bt_off, int nblk, int anchor) {
+ if (t < 0 || t > pos) return -1;
+ int b = t / B;
+ int j = b < nblk ? tab[bt_off + b] : -1;
+ if (j < 0 || (b < pos / B - anchor && (j & 0x40000000) == 0)) return -1;
+ return (j & 0x3fffffff) * B + t % B;
+}
+
+// ---- mode 0 ----------------------------------------------------------------------
+void gqa() {
+ int S = p.v[1], H = p.v[2], KVH = p.v[3], hd = p.v[4], vd = p.v[5], pos_base = p.v[6], rows = p.v[7];
+ int q_off = p.v[8], q_row = p.v[9], q_seg = p.v[10], sel_off = p.v[11], sel_row = p.v[12];
+ float scale = intBitsToFloat(p.v[13]);
+ int k_off = p.v[14], v_off = p.v[15], win = p.v[16], kv_pm = p.v[17], sink = p.v[18], sink_off = p.v[19];
+ int B = p.v[20], bt_off = p.v[22], nblk = p.v[23], anchor = p.v[24];
+ int o_off = p.v[25], o_row = p.v[26], st_off = p.v[27], flags = p.v[28];
+ int g_off = p.v[29], g_row = p.v[30], g_seg = p.v[31];
+ int h = int(gl_WorkGroupID.x), s = int(gl_WorkGroupID.y), tid = int(gl_LocalInvocationID.x);
+ if (h >= H || s >= S) return;
+ int kvh = h / (H / KVH);
+ int qb = q_off + s * q_row + h * q_seg;
+ for (int d = tid; d < hd; d += 256) qs[d] = b0[qb + d];
+ int pos = pos_base + s;
+ int first = max(win > 0 ? max(0, pos - win + 1) : 0, (pos / B - anchor) * B);
+ bool list = false;
+ int lb = 0, n = pos - first + 1;
+ if (sel_row > 0) {
+ lb = sel_off + s * sel_row;
+ int c = sel[lb];
+ if (c >= 0) { n = c; list = true; }
+ }
+ barrier();
+ float m = sink != 0 ? b6[sink_off + h] : -3.0e38, l = sink != 0 ? 1.0 : 0.0, acc = 0.0;
+ for (int t0 = 0; t0 < n; t0 += 256) {
+ int i = t0 + tid, r = -1;
+ if (i < n) {
+ if (list) r = dev_row(sel[lb + 1 + i], pos, B, bt_off, nblk, anchor);
+ else { int t = first + i, j = slot_of(t, B, bt_off, nblk); r = j >= 0 ? j * B + t % B : -1; }
+ }
+ float sv = -3.0e38;
+ if (r >= 0) {
+ int kb = k_off + (kv_pm != 0 ? r * KVH + kvh : kvh * rows + r) * hd;
+ float a = 0.0;
+ for (int d = 0; d < hd; d++) a += qs[d] * b1[kb + d];
+ sv = a * scale;
+ }
+ pr[tid] = r;
+ red[tid] = sv;
+ barrier();
+ for (int off = 128; off > 0; off >>= 1) {
+ if (tid < off) red[tid] = max(red[tid], red[tid + off]);
+ barrier();
+ }
+ float mt = red[0];
+ barrier();
+ if (mt > -1.0e38) { // a tile with a position on the device
+ float mn = max(m, mt);
+ float e = r >= 0 ? exp(sv - mn) : 0.0;
+ pe[tid] = e;
+ red[tid] = e;
+ barrier();
+ for (int off = 128; off > 0; off >>= 1) {
+ if (tid < off) red[tid] += red[tid + off];
+ barrier();
+ }
+ float corr = m > -1.0e38 ? exp(m - mn) : 0.0;
+ l = l * corr + red[0];
+ acc *= corr;
+ int cnt = min(256, n - t0);
+ if (tid < vd)
+ for (int j = 0; j < cnt; j++) {
+ int rr = pr[j];
+ if (rr < 0) continue;
+ int vb = v_off + (kv_pm != 0 ? rr * KVH + kvh : kvh * rows + rr) * vd;
+ acc += pe[j] * b2[vb + tid];
+ }
+ m = mn;
+ }
+ barrier();
+ }
+ if (tid < vd) {
+ int ob = o_off + s * o_row + h * vd;
+ if ((flags & 1) != 0) {
+ float v = acc * (l > 0.0 ? 1.0 / l : 0.0);
+ if ((flags & 2) != 0) v *= 1.0 / (1.0 + exp(-b4[g_off + s * g_row + h * g_seg + tid]));
+ b3[ob + tid] = v;
+ } else b3[ob + tid] = acc;
+ }
+ if (tid == 0 && (flags & 1) == 0) { b3[st_off + (s * H + h) * 2] = m; b3[st_off + (s * H + h) * 2 + 1] = l; }
+}
+
+// ---- mode 1 ----------------------------------------------------------------------
+void mla() {
+ int S = p.v[1], H = p.v[2], K = p.v[3], R = p.v[4], pos_base = p.v[5], kv_start = p.v[6];
+ int qa_off = p.v[7], qa_row = p.v[8], qa_seg = p.v[9], qr_off = p.v[10], qr_row = p.v[11], qr_seg = p.v[12];
+ int lat_off = p.v[13], lat_row = p.v[14], rope_off = p.v[15], rope_row = p.v[16];
+ int sel_off = p.v[17], sel_row = p.v[18], o_off = p.v[19], o_row = p.v[20], o_seg = p.v[21];
+ float scale = intBitsToFloat(p.v[22]);
+ int B = p.v[23], bt_off = p.v[25], nblk = p.v[26], anchor = p.v[27], flags = p.v[28], st_off = p.v[29];
+ int h = int(gl_WorkGroupID.x), s = int(gl_WorkGroupID.y), tid = int(gl_LocalInvocationID.x);
+ if (h >= H || s >= S) return;
+ int lane = int(gl_SubgroupInvocationID), sgs = int(gl_SubgroupSize);
+ int sg = int(gl_SubgroupID), nsg = 256 / sgs;
+ int qb = qa_off + s * qa_row + h * qa_seg, rbq = qr_off + s * qr_row + h * qr_seg;
+ for (int i = tid; i < K; i += 256) qs[i] = b0[qb + i];
+ for (int i = tid; i < R; i += 256) qrs[i] = b1[rbq + i];
+ int pos = pos_base + s;
+ int first = max(kv_start, (pos / B - anchor) * B);
+ bool list = false;
+ int lb = 0, n = max(0, pos - first + 1);
+ if (sel_row > 0) {
+ lb = sel_off + s * sel_row;
+ int c = sel[lb];
+ if (c >= 0) { n = c; list = true; }
+ }
+ barrier();
+ float m = -3.0e38, l = 0.0;
+ float acc[4] = float[4](0.0, 0.0, 0.0, 0.0);
+ for (int t0 = 0; t0 < n; t0 += 64) {
+ int cnt = min(64, n - t0);
+ if (tid < 64) {
+ int r = -1;
+ if (tid < cnt) {
+ int i = t0 + tid;
+ if (list) r = dev_row(sel[lb + 1 + i], pos, B, bt_off, nblk, anchor);
+ else { int t = first + i, j = slot_of(t, B, bt_off, nblk); r = j >= 0 ? j * B + t % B : -1; }
+ }
+ pr[tid] = r;
+ }
+ barrier();
+ for (int j = sg; j < cnt; j += nsg) {
+ int r = pr[j];
+ float a = 0.0;
+ if (r >= 0) {
+ int lo = lat_off + r * lat_row, ro = rope_off + r * rope_row;
+ for (int i = lane; i < K; i += sgs) a += qs[i] * b2[lo + i];
+ for (int i = lane; i < R; i += sgs) a += qrs[i] * b3[ro + i];
+ }
+ float tot = subgroupAdd(a);
+ if (lane == 0) pe[j] = r >= 0 ? tot * scale : -3.0e38;
+ }
+ barrier();
+ red[tid] = tid < cnt ? pe[tid] : -3.0e38;
+ barrier();
+ for (int off = 128; off > 0; off >>= 1) {
+ if (tid < off) red[tid] = max(red[tid], red[tid + off]);
+ barrier();
+ }
+ float mt = red[0];
+ barrier();
+ if (mt > -1.0e38) {
+ float mn = max(m, mt);
+ float corr = m > -1.0e38 ? exp(m - mn) : 0.0;
+ if (tid < 64) pe[tid] = tid < cnt && pr[tid] >= 0 ? exp(pe[tid] - mn) : 0.0;
+ barrier();
+ red[tid] = tid < 64 ? pe[tid] : 0.0;
+ barrier();
+ for (int off = 128; off > 0; off >>= 1) {
+ if (tid < off) red[tid] += red[tid + off];
+ barrier();
+ }
+ l = l * corr + red[0];
+ for (int k = 0; k < 4; k++) {
+ int i = tid + 256 * k;
+ if (i >= K) break;
+ float a = acc[k] * corr;
+ for (int j = 0; j < cnt; j++) {
+ int r = pr[j];
+ if (r >= 0) a += pe[j] * b2[lat_off + r * lat_row + i];
+ }
+ acc[k] = a;
+ }
+ m = mn;
+ }
+ barrier();
+ }
+ float inv = (flags & 1) != 0 ? (l > 0.0 ? 1.0 / l : 0.0) : 1.0;
+ int ob = o_off + s * o_row + h * o_seg;
+ for (int k = 0; k < 4; k++) {
+ int i = tid + 256 * k;
+ if (i >= K) break;
+ b6[ob + i] = acc[k] * inv;
+ }
+ if (tid == 0 && (flags & 1) == 0) { b6[st_off + (s * H + h) * 2] = m; b6[st_off + (s * H + h) * 2 + 1] = l; }
+}
+
+// ---- mode 2 ----------------------------------------------------------------------
+void merge() {
+ int n = p.v[1], H = p.v[2], d = p.v[3];
+ int a_off = p.v[4], a_row = p.v[5], a_seg = p.v[6], sa_off = p.v[7], c_off = p.v[8], sc_off = p.v[9];
+ int o_off = p.v[10], o_row = p.v[11], o_seg = p.v[12], flags = p.v[13], g_off = p.v[14], g_row = p.v[15], g_seg = p.v[16];
+ int g = int(gl_WorkGroupID.x + gl_WorkGroupID.y * gl_NumWorkGroups.x), tid = int(gl_LocalInvocationID.x);
+ if (g >= n) return;
+ int s = g / H, h = g - s * H;
+ float md = b0[sa_off + 2 * g], ld = b0[sa_off + 2 * g + 1];
+ float mc = b1[sc_off + 2 * g], lc = b1[sc_off + 2 * g + 1];
+ float mx = max(md, mc);
+ float ed = md > -1.0e38 ? exp(md - mx) : 0.0, ec = mc > -1.0e38 ? exp(mc - mx) : 0.0;
+ float den = ed * ld + ec * lc;
+ float inv = den > 0.0 ? 1.0 / den : 0.0;
+ int ab = a_off + s * a_row + h * a_seg, cb = c_off + g * d, ob = o_off + s * o_row + h * o_seg;
+ for (int i = tid; i < d; i += 256) {
+ float v = (ed * b0[ab + i] + ec * b1[cb + i]) * inv;
+ if ((flags & 2) != 0) v *= 1.0 / (1.0 + exp(-b2[g_off + s * g_row + h * g_seg + i]));
+ if ((flags & 4) != 0) v = bf16r(v);
+ b3[ob + i] = v;
+ }
+}
+
+// ---- mode 3 ----------------------------------------------------------------------
+void rel() {
+ int S = p.v[1], H = p.v[2], KVH = p.v[3], hd = p.v[4], pos_base = p.v[5], rows = p.v[6];
+ int q_off = p.v[7], q_row = p.v[8];
+ float scale = intBitsToFloat(p.v[9]);
+ int k_off = p.v[10], v_off = p.v[11], ext = p.v[12], d_rel = p.v[13], r_off = p.v[14], r_row = p.v[15];
+ int relp_off = p.v[16], tau_off = p.v[17], B = p.v[18], bt_off = p.v[19], nblk = p.v[20], anchor = p.v[21];
+ int o_off = p.v[22], o_row = p.v[23], st_off = p.v[24], flags = p.v[25];
+ int h = int(gl_WorkGroupID.x), s = int(gl_WorkGroupID.y), tid = int(gl_LocalInvocationID.x);
+ if (h >= H || s >= S) return;
+ int kvh = h / (H / KVH);
+ int qb = q_off + s * q_row + h * hd;
+ for (int d = tid; d < hd; d += 256) qs[d] = b0[qb + d];
+ for (int j = tid; j < d_rel; j += 256) qrs[j] = b4[r_off + s * r_row + h * d_rel + j];
+ int pos = pos_base + s, first = max(0, (pos / B - anchor) * B), n = pos - first + 1;
+ float ts = intBitsToFloat(sel[tau_off + s]);
+ barrier();
+ float m = -3.0e38, l = 0.0, acc = 0.0;
+ for (int t0 = 0; t0 < n; t0 += 256) {
+ int i = t0 + tid, r = -1, t = first + i;
+ if (i < n) { int j = slot_of(t, B, bt_off, nblk); r = j >= 0 ? j * B + t % B : -1; }
+ float sv = -3.0e38;
+ if (r >= 0) {
+ int kb = k_off + (kvh * rows + r) * hd;
+ float a = 0.0;
+ for (int d = 0; d < hd; d++) a += qs[d] * b1[kb + d];
+ int dist = pos - t;
+ float bias = 0.0;
+ if (dist < ext)
+ for (int j = 0; j < d_rel; j++) bias += qrs[j] * b6[relp_off + j * ext + dist];
+ sv = ts * (a * scale + bias);
+ }
+ pr[tid] = r;
+ red[tid] = sv;
+ barrier();
+ for (int off = 128; off > 0; off >>= 1) {
+ if (tid < off) red[tid] = max(red[tid], red[tid + off]);
+ barrier();
+ }
+ float mt = red[0];
+ barrier();
+ if (mt > -1.0e38) {
+ float mn = max(m, mt);
+ float e = r >= 0 ? exp(sv - mn) : 0.0;
+ pe[tid] = e;
+ red[tid] = e;
+ barrier();
+ for (int off = 128; off > 0; off >>= 1) {
+ if (tid < off) red[tid] += red[tid + off];
+ barrier();
+ }
+ float corr = m > -1.0e38 ? exp(m - mn) : 0.0;
+ l = l * corr + red[0];
+ acc *= corr;
+ int cnt = min(256, n - t0);
+ if (tid < hd)
+ for (int j = 0; j < cnt; j++) {
+ int rr = pr[j];
+ if (rr >= 0) acc += pe[j] * b2[v_off + (kvh * rows + rr) * hd + tid];
+ }
+ m = mn;
+ }
+ barrier();
+ }
+ if (tid < hd) b3[o_off + s * o_row + h * hd + tid] = (flags & 1) != 0 ? acc * (l > 0.0 ? 1.0 / l : 0.0) : acc;
+ if (tid == 0 && (flags & 1) == 0) { b3[st_off + (s * H + h) * 2] = m; b3[st_off + (s * H + h) * 2 + 1] = l; }
+}
+
+// ---- mode 4 ----------------------------------------------------------------------
+void dsa() {
+ int S = p.v[1], H = p.v[2], hd = p.v[3], cnt = p.v[4], l_off = p.v[5], l_row = p.v[6], nwin = p.v[7];
+ int w_off = p.v[8], c_off = p.v[9], q_off = p.v[10], q_row = p.v[11], sink_off = p.v[12], flags = p.v[13];
+ float scale = intBitsToFloat(p.v[14]);
+ int B = p.v[15], ns = p.v[16], bt_off = p.v[17], nblk = p.v[18], o_off = p.v[19], o_row = p.v[20], st_off = p.v[21];
+ int h = int(gl_WorkGroupID.x), s = int(gl_WorkGroupID.y), tid = int(gl_LocalInvocationID.x);
+ if (h >= H || s >= S) return;
+ int lb = l_off + s * l_row;
+ for (int i = tid; i < hd; i += 256) qs[i] = b0[q_off + s * q_row + h * hd + i];
+ float sink = b4[sink_off + h];
+ barrier();
+ // In complete sparse mode, -2-j addresses staged cold row j. Give these
+ // offsets a disjoint range after the resident cache in the value-sum table.
+ int cold_base = c_off + ns * B * hd;
+ // pass 1: the part's largest score (with the sink unless fin)
+ float best = (flags & 8) != 0 ? -1.0e30 : sink;
+ for (int t0 = 0; t0 < cnt; t0 += 256) {
+ int j = t0 + tid;
+ if (j < cnt) {
+ int e = sel[lb + j], kb = -1;
+ if ((flags & 16) != 0 && e < -1) kb = cold_base + (-2 - e) * hd;
+ else if (e >= 0 && e < nwin) kb = w_off + e * hd;
+ else if (e >= nwin) { int sl = slot_of(e - nwin, B, bt_off, nblk); if (sl >= 0) kb = c_off + (sl * B + (e - nwin) % B) * hd; }
+ if (kb >= 0) {
+ float d = 0.0;
+ if (e >= 0 && e < nwin) for (int i = 0; i < hd; i++) d += qs[i] * b1[kb + i];
+ else if (kb < cold_base) for (int i = 0; i < hd; i++) d += qs[i] * b2[kb + i];
+ else for (int i = 0; i < hd; i++) d += qs[i] * b6[kb - cold_base + i];
+ best = max(best, d * scale);
+ }
+ }
+ }
+ red[tid] = best;
+ barrier();
+ for (int off = 128; off > 0; off >>= 1) {
+ if (tid < off && red[tid + off] > red[tid]) red[tid] = red[tid + off];
+ barrier();
+ }
+ best = red[0];
+ barrier();
+ // pass 2: the weights tile by tile, the denominator and the value sums in list order
+ float den = exp(sink - best);
+ float acc[4] = float[4](0.0, 0.0, 0.0, 0.0);
+ for (int t0 = 0; t0 < cnt; t0 += 256) {
+ int j = t0 + tid, kb = -1;
+ float w = 0.0;
+ if (j < cnt) {
+ int e = sel[lb + j];
+ if ((flags & 16) != 0 && e < -1) kb = cold_base + (-2 - e) * hd;
+ else if (e >= 0 && e < nwin) kb = w_off + e * hd;
+ else if (e >= nwin) { int sl = slot_of(e - nwin, B, bt_off, nblk); if (sl >= 0) kb = c_off + (sl * B + (e - nwin) % B) * hd; }
+ if (kb >= 0) {
+ float d = 0.0;
+ if (e >= 0 && e < nwin) for (int i = 0; i < hd; i++) d += qs[i] * b1[kb + i];
+ else if (kb < cold_base) for (int i = 0; i < hd; i++) d += qs[i] * b2[kb + i];
+ else for (int i = 0; i < hd; i++) d += qs[i] * b6[kb - cold_base + i];
+ w = d * scale;
+ kb = e >= 0 && e < nwin ? kb : -2 - kb; // the compressed rows' offsets as -2 - offset
+ }
+ }
+ pe[tid] = w;
+ pr[tid] = kb;
+ barrier();
+ // The whole-cache shader stores each scaled score before subtracting
+ // the maximum. Keep that f32 rounding boundary here too: contracting
+ // d * scale - best changes f32 attention on hardware that fuses it.
+ pe[tid] = pr[tid] != -1 ? exp(pe[tid] - best) : 0.0;
+ barrier();
+ int c = min(256, cnt - t0);
+ if (tid == 0) for (int q = 0; q < c; q++) if (pr[q] != -1) den += pe[q]; // thread 0: in list order
+ for (int k = 0; k < 4; k++) {
+ int i = tid + 256 * k;
+ if (i >= hd) break;
+ float a = acc[k];
+ for (int q = 0; q < c; q++) {
+ int o = pr[q];
+ if (o == -1) continue;
+ float wv = (flags & 1) != 0 ? bf16r(pe[q]) : pe[q];
+ a += wv * (o >= 0 ? b1[o + i] : -2 - o < cold_base ? b2[-2 - o + i] : b6[-2 - o - cold_base + i]);
+ }
+ acc[k] = a;
+ }
+ barrier();
+ }
+ if (tid == 0) red[0] = den;
+ barrier();
+ den = red[0];
+ int ob = o_off + s * o_row + h * hd;
+ for (int k = 0; k < 4; k++) {
+ int i = tid + 256 * k;
+ if (i >= hd) break;
+ if ((flags & 8) != 0) { float o = acc[k] / den; b3[ob + i] = (flags & 1) != 0 ? bf16r(o) : o; }
+ else b3[ob + i] = acc[k];
+ }
+ if (tid == 0 && (flags & 8) == 0) { b3[st_off + (s * H + h) * 2] = best; b3[st_off + (s * H + h) * 2 + 1] = den; }
+}
+
+void main() {
+ if (MODE == 0) gqa();
+ else if (MODE == 1) mla();
+ else if (MODE == 2) merge();
+ else if (MODE == 3) rel();
+ else dsa();
+}
diff --git a/c/shaders/chain_kvs.spv b/c/shaders/chain_kvs.spv
new file mode 100644
index 00000000..96d2a0c2
Binary files /dev/null and b/c/shaders/chain_kvs.spv differ
diff --git a/c/shaders/chain_relattn.comp b/c/shaders/chain_relattn.comp
index 5749774f..c9af0c36 100644
--- a/c/shaders/chain_relattn.comp
+++ b/c/shaders/chain_relattn.comp
@@ -16,6 +16,9 @@
// The softmax runs online over tiles of 128 positions, as chain_attn.comp's does; the
// CPU normalizes first, so the two agree to rounding.
// Limits: hd <= 256 (two dims per thread), d_rel <= 64.
+// s0 (the last push constant, 0 for a whole step): the step's row of this dispatch's
+// first row, when vk_chain.c cuts a prompt chunk's rows over several submissions; q, o,
+// r and tau's offsets then point at that row, pos_base stays the step's first position.
layout(local_size_x = 128) in;
layout(std430, binding = 0) readonly buffer QB { float q[]; };
@@ -33,6 +36,7 @@ layout(push_constant) uniform P {
int k_off, v_off, ks_off, vs_off, kv_row;
int r_off, r_row, relp_off, tau_off;
float scale;
+ int s0;
} p;
shared float qs[256];
@@ -48,7 +52,7 @@ void main() {
int qb = p.q_off + s * p.q_row + h * p.hd;
for (int d = tid; d < p.hd; d += 128) qs[d] = q[qb + d];
for (int j = tid; j < p.d_rel; j += 128) rs[j] = r[p.r_off + s * p.r_row + h * p.d_rel + j];
- int qpos = p.pos_base + s;
+ int qpos = p.pos_base + p.s0 + s;
int t0 = p.window > 0 ? max(0, qpos - p.window + 1) : 0;
int n = qpos - t0 + 1;
float ts = tau[p.tau_off + s];
diff --git a/c/shaders/chain_relattn.spv b/c/shaders/chain_relattn.spv
index da24087e..320f154c 100644
Binary files a/c/shaders/chain_relattn.spv and b/c/shaders/chain_relattn.spv differ
diff --git a/c/shaders/expert_act.comp b/c/shaders/expert_act.comp
index a4669870..cfd2ed89 100644
--- a/c/shaders/expert_act.comp
+++ b/c/shaders/expert_act.comp
@@ -34,12 +34,16 @@ float act_v4(float g, float u, float limit) {
return g * s * u;
}
+// Below 2^-12 the cubic correction is below half an f32 ULP. Some drivers'
+// tanh lowering loses these values through cancellation around exp(0) == 1.
+float situ_tanh(float x) { return abs(x) < 0.000244140625 ? x : tanh(x); }
+
void main() {
int i = int(gl_GlobalInvocationID.x) + int(gl_GlobalInvocationID.y) * int(gl_NumWorkGroups.x) * 256;
if (i >= p.n) return;
float gt = gate[i], ut = up[i], h;
if (p.act == 1) {
- h = p.a * tanh(gt / p.a) * (1.0 / (1.0 + exp(-gt))) * (p.b * tanh(ut / p.b));
+ h = p.a * situ_tanh(gt / p.a) * (1.0 / (1.0 + exp(-gt))) * (p.b * situ_tanh(ut / p.b));
} else if (p.act == 2) {
h = act_v4(gt, ut, p.limit);
} else {
diff --git a/c/shaders/qmatmul.comp b/c/shaders/qmatmul.comp
index 06b5d184..489343ea 100644
--- a/c/shaders/qmatmul.comp
+++ b/c/shaders/qmatmul.comp
@@ -26,7 +26,7 @@ layout(std430, binding = 2) readonly buffer Sc { float scale[]; };
layout(std430, binding = 3) writeonly buffer Y { float y[]; };
layout(push_constant) uniform P {
- int fmt; // 1=int8, 2=int4, 4=grouped int4, 5=int3-g64, 7=MXFP4, 10=f32, 11=bf16, 12=fp8-e4m3 grouped, 13=int8 grouped
+ int fmt; // 1=int8, 2=int4, 4=grouped int4, 5=int3-g64, 7=MXFP4, 10=f32, 11=bf16, 12=fp8-e4m3 grouped, 13=int8 grouped, 14=f16
int S, I, O;
int rowWords; // uint32 words per weight row
int gs; // fmt=4 group size (multiple of 8; host gates otherwise)
@@ -107,6 +107,13 @@ void main() {
if (i0 + 1 < p.I)
sum += (staged ? xsh[i0 + 1] : x[xoff + i0 + 1]) * uintBitsToFloat(pk & 0xffff0000u);
}
+ } else if (p.fmt == 14) { // f16 weights, two per word, LOW half = even column
+ int words = (p.I + 1) / 2;
+ for (int wi = lane; wi < words; wi += sgsize) {
+ vec2 h = unpackHalf2x16(w[rowBase + uint(wi)]); int i0 = wi * 2;
+ sum += (staged ? xsh[i0] : x[xoff + i0]) * h.x;
+ if (i0 + 1 < p.I) sum += (staged ? xsh[i0 + 1] : x[xoff + i0 + 1]) * h.y;
+ }
} else if (p.fmt == 12 || p.fmt == 13) { // fp8 e4m3 / int8, 4 per word, one f32
// scale per gs inputs. gs % 4 == 0 (host-gated): a word never straddles a group.
int words = (p.I + 3) / 4, ng = (p.I + p.gs - 1) / p.gs;
@@ -173,7 +180,7 @@ void main() {
}
}
float tot = subgroupAdd(sum);
- if (lane == 0) y[s * p.O + o] = (p.fmt == 5 || p.fmt == 4 || p.fmt == 7 || p.fmt == 10 || p.fmt == 11 || p.fmt == 12 || p.fmt == 13)
+ if (lane == 0) y[s * p.O + o] = (p.fmt == 5 || p.fmt == 4 || p.fmt == 7 || p.fmt == 10 || p.fmt == 11 || p.fmt == 12 || p.fmt == 13 || p.fmt == 14)
? tot : tot * scale[o];
}
}
diff --git a/c/shaders/qmatmul.spv b/c/shaders/qmatmul.spv
index 2d527865..ddd75fcf 100644
Binary files a/c/shaders/qmatmul.spv and b/c/shaders/qmatmul.spv differ
diff --git a/c/shaders/qmatmul_coop.comp b/c/shaders/qmatmul_coop.comp
index bcae0690..cb244fbe 100644
--- a/c/shaders/qmatmul_coop.comp
+++ b/c/shaders/qmatmul_coop.comp
@@ -29,9 +29,12 @@ layout(constant_id = 3) const int WN = 32; // rows per subgroup, a multiple o
layout(constant_id = 4) const int BK = 32; // inputs per staged step, a multiple of 32
layout(constant_id = 5) const int SG = 64; // subgroup size the pipeline requires
layout(local_size_x_id = 6) in; // (BM/WM) * (BN/WN) * SG threads
+// Expert intermediates stay on the device. Compute their row scales here instead
+// of downloading them for the host to append scales to the activation buffer.
+layout(constant_id = 7) const int DEVICE_SCALE = 0;
layout(std430, binding = 0) readonly buffer X { float x[]; }; // x[S*I], then rs[S]
-layout(std430, binding = 0) readonly buffer X4 { vec4 x4[]; };
+// One view of the dynamic input binding, including under MoltenVK.
layout(std430, binding = 1) readonly buffer W { uint w[]; };
layout(std430, binding = 1) readonly buffer W2 { uvec2 w2[]; };
layout(std430, binding = 2) readonly buffer Sc { float scale[]; };
@@ -54,6 +57,33 @@ shared float16_t xhs[BN * WST]; // [s][i]: its B, column-major; hi part
shared float16_t xls[BN * WST]; // lo part
shared float ssh[BM]; // the current group's scales
shared float esh[NW * 16 * EST]; // per subgroup: one accumulator on its way to y
+shared uint xmag[DEVICE_SCALE != 0 ? NT : 1];
+shared float xscale[DEVICE_SCALE != 0 ? BN : 1];
+
+float row_scale(int s) {
+ return DEVICE_SCALE != 0 ? xscale[s % BN] : x[uint(p.S) * uint(p.I) + uint(s)];
+}
+
+void prepare_scales(int tid, int s0) {
+ const int lanes = max(1, NT / BN), rows = NT / lanes;
+ for (int first = 0; first < BN; first += rows) {
+ int r = first + tid / lanes, lane = tid % lanes, s = s0 + r;
+ uint m = 0u;
+ if (r < BN && s < p.S)
+ for (int i = lane; i < p.I; i += lanes)
+ m = max(m, floatBitsToUint(x[uint(s) * uint(p.I) + uint(i)]) & 0x7fffffffu);
+ xmag[tid] = m;
+ barrier();
+ if (lane == 0 && r < BN) {
+ for (int j = 1; j < lanes; j++) m = max(m, xmag[tid + j]);
+ int e = int(m >> 23) - 126;
+ if (m >= 0x7f800000u) e = 15;
+ e = max(e, -100);
+ xscale[r] = uintBitsToFloat(uint(15 - e + 127) << 23);
+ }
+ barrier();
+ }
+}
#define ACC coopmat<float, gl_ScopeSubgroup, 16, 16, gl_MatrixUseAccumulator>
#define MATA coopmat<float16_t, gl_ScopeSubgroup, 16, 16, gl_MatrixUseA>
@@ -134,14 +164,14 @@ void fetch_step(int tid, int o0, int s0, int k0) {
vec4 v = vec4(0.0);
if (u < BN * (BK / 4) && s < p.S && k < p.I) {
uint b = uint(s) * uint(p.I) + uint(k);
- if ((p.I & 3) == 0) v = x4[b >> 2];
+ if ((p.I & 3) == 0) v = vec4(x[b], x[b + 1u], x[b + 2u], x[b + 3u]);
else {
v.x = x[b];
if (k + 1 < p.I) v.y = x[b + 1u];
if (k + 2 < p.I) v.z = x[b + 2u];
if (k + 3 < p.I) v.w = x[b + 3u];
}
- v *= x[uint(p.S) * uint(p.I) + uint(s)]; // the row's power of two
+ v *= row_scale(s);
}
rx[r] = v;
}
@@ -152,6 +182,7 @@ void main() {
int sg = int(gl_SubgroupID), lane = int(gl_SubgroupInvocationID);
int wm = sg % (BM / WM), wn = sg / (BM / WM); // this subgroup's WM x WN
int o0 = int(gl_WorkGroupID.x) * BM, s0 = int(gl_WorkGroupID.y) * BN;
+ if (DEVICE_SCALE != 0) prepare_scales(tid, s0);
bool grouped = p.fmt != 1 && p.fmt != 2;
int gsz = p.fmt == 5 ? 64 : p.gs; // a multiple of BK
int ng = grouped ? (p.I + gsz - 1) / gsz : 1;
@@ -239,7 +270,7 @@ void main() {
[[unroll]] for (int t = 0; t < 256 / SG; t++) {
int n = lane / 16 + t * (SG / 16), s = s0 + wn * WN + j * 16 + n;
if (o < p.O && s < p.S)
- y[uint(s) * uint(p.O) + uint(o)] = esh[eb + m * EST + n] * sc / x[uint(p.S) * uint(p.I) + uint(s)];
+ y[uint(s) * uint(p.O) + uint(o)] = esh[eb + m * EST + n] * sc / row_scale(s);
}
subgroupBarrier();
}
diff --git a/c/shaders/qmatmul_coop.spv b/c/shaders/qmatmul_coop.spv
index 8685197a..1454a6d1 100644
Binary files a/c/shaders/qmatmul_coop.spv and b/c/shaders/qmatmul_coop.spv differ
diff --git a/c/shaders/qmatmul_gate_up.comp b/c/shaders/qmatmul_gate_up.comp
index 3a0a6b34..e28fadba 100644
--- a/c/shaders/qmatmul_gate_up.comp
+++ b/c/shaders/qmatmul_gate_up.comp
@@ -73,6 +73,9 @@ float act_v4(float g, float u, float limit) {
return g * s * u;
}
+// Preserve small SiTU inputs; tanh(x) rounds to x below 2^-12 in f32.
+float situ_tanh(float x) { return abs(x) < 0.000244140625 ? x : tanh(x); }
+
void main() {
int s = int(gl_WorkGroupID.y);
int tid = int(gl_LocalInvocationID.x);
@@ -180,7 +183,7 @@ void main() {
if (p.fmt == 1 || p.fmt == 2) { gt *= gscale[o]; ut *= uscale[o]; } // per-row formats only
float h;
if (p.act == 1) {
- h = p.a * tanh(gt / p.a) * (1.0 / (1.0 + exp(-gt))) * (p.b * tanh(ut / p.b));
+ h = p.a * situ_tanh(gt / p.a) * (1.0 / (1.0 + exp(-gt))) * (p.b * situ_tanh(ut / p.b));
} else if (p.act == 2) {
h = act_v4(gt, ut, p.limit);
} else {
diff --git a/c/shaders/qmatmul_gemm.comp b/c/shaders/qmatmul_gemm.comp
index 732508d5..97a7c442 100644
--- a/c/shaders/qmatmul_gemm.comp
+++ b/c/shaders/qmatmul_gemm.comp
@@ -38,7 +38,7 @@ layout(std430, binding = 2) readonly buffer Sc { float scale[]; };
layout(std430, binding = 3) writeonly buffer Y { float y[]; };
layout(push_constant) uniform P {
- int fmt; // as qmatmul.comp: 1, 2, 4, 5, 7, 10, 11, 12, 13
+ int fmt; // as qmatmul.comp: 1, 2, 4, 5, 7, 10, 11, 12, 13, 14
int S, I, O;
int rowWords; // uint32 words per weight row
int gs; // group size of fmt 4/7 (multiple of 8) and 12/13 (multiple of 4)
@@ -98,7 +98,7 @@ void fetch8(int o, int k, out uvec4 lo, out uvec4 hi, out vec2 sc) {
if (k + j < p.I) lo[j] = w[rb + uint(k + j)];
if (k + 4 + j < p.I) hi[j] = w[rb + uint(k + 4 + j)];
}
- } else if (p.fmt == 11) { // bf16, two per word
+ } else if (p.fmt == 11 || p.fmt == 14) { // bf16, f16: two per word
if ((p.I & 7) == 0) lo = w4[(rb + uint(k >> 1)) >> 2];
else [[unroll]] for (int j = 0; j < 4; j++)
if (k + 2 * j < p.I) lo[j] = w[rb + uint((k >> 1) + j)];
@@ -132,6 +132,9 @@ void decode8(uvec4 a, uvec4 b, vec2 sc, out vec4 lo, out vec4 hi) {
uintBitsToFloat(a.y << 16), uintBitsToFloat(a.y & 0xffff0000u));
hi = vec4(uintBitsToFloat(a.z << 16), uintBitsToFloat(a.z & 0xffff0000u),
uintBitsToFloat(a.w << 16), uintBitsToFloat(a.w & 0xffff0000u));
+ } else if (p.fmt == 14) { // f16, low half = even column
+ lo = vec4(unpackHalf2x16(a.x), unpackHalf2x16(a.y));
+ hi = vec4(unpackHalf2x16(a.z), unpackHalf2x16(a.w));
} else if (p.fmt == 5) { // value = 2 low-plane bits | high bit << 2, - 4
float v[8];
[[unroll]] for (int t = 0; t < 8; t++)
diff --git a/c/shaders/qmatmul_gemm.spv b/c/shaders/qmatmul_gemm.spv
index 1691e167..362319f9 100644
Binary files a/c/shaders/qmatmul_gemm.spv and b/c/shaders/qmatmul_gemm.spv differ
diff --git a/c/spec_draft.h b/c/spec_draft.h
new file mode 100644
index 00000000..03a3f5bd
--- /dev/null
+++ b/c/spec_draft.h
@@ -0,0 +1,200 @@
+/* spec_draft.h -- what a speculative verify should carry, and whether drafting pays.
+ *
+ * An engine with a verify forward (several rows in one forward, each computed the way a
+ * decode step computes it, so the rows that stand give the logits plain decoding gives)
+ * can take its drafts from any source: its MTP head, or the context itself. This header
+ * holds the parts that do not depend on the engine; it has no engine state and no clock.
+ *
+ * spec_lookup prompt lookup: the longest n-gram (max_n down to min_n tokens) that ends
+ * the token history and also occurs earlier in it, at its most recent earlier
+ * occurrence; the up to k tokens that followed it there are the proposal.
+ * Code edits, quotes and repeated structure give long runs of accepted
+ * drafts; free text gives few proposals, and short ones.
+ *
+ * SpecGate how many drafts a verify should carry. Per source it keeps the running
+ * probability a_j that draft j is accepted when the drafts before it were,
+ * the measured seconds T(n) of a forward of n rows, and the measured seconds
+ * d a source spends on each draft past its first. The depth k it picks
+ * maximizes the expected tokens per second of a verify against a plain step:
+ * value(k) = E(k) / cost(k)
+ * E(k) = 1 + a_1 + a_1 a_2 + ... + a_1 ... a_k (tokens it yields)
+ * cost(k) = (T(k+1) + (k-1) d) / T(1) (in decode steps)
+ * and drafts only when value(k) > 1 (k = 0 is a plain step, value 1). A row
+ * count not measured yet costs T(1) (1 + (n-1) rho), rho fitted from the
+ * multi-row forwards that ran (SPEC_RHO0 until one has). Every SPEC_PROBE
+ * picks the gate tries one draft deeper than its best (so an estimate the
+ * gate stopped feeding can recover; every SPEC_PROBE_NEW while that deeper
+ * verify's cost has fewer than 3 measurements), and after SPEC_REFRESH
+ * forwards with no plain step it takes one, so T(1) stays measured.
+ *
+ * Neither ever changes a token: every proposal goes through the engine's verify, which
+ * keeps exactly what plain decoding produces. They change how many forwards that takes.
+ *
+ * Callers: qwen38_core.h (MTP drafts and prompt lookup, combined) and qwen36.c (prompt
+ * lookup). docs/speculative.md describes how another engine with a verify path wires it. */
+#ifndef COLI_SPEC_DRAFT_H
+#define COLI_SPEC_DRAFT_H
+
+#include <stdlib.h>
+#include <string.h>
+
+#define SPEC_MAX_DRAFTS 8 /* positions the gate tracks (an engine's verify may take fewer) */
+#define SPEC_PROBE 16 /* every this many picks, one draft deeper than the best */
+#define SPEC_PROBE_NEW 4 /* the same while that deeper row count is barely measured */
+#define SPEC_REFRESH 64 /* forwards with no plain step before the gate takes one */
+#define SPEC_RHO0 0.5 /* a row's extra cost, relative to one row, before any is measured */
+#define SPEC_ACC_W 16.0 /* the acceptance's memory, in verifies */
+#define SPEC_T_W 8.0 /* a forward time's memory, in forwards */
+
+enum { SPEC_SRC_MTP = 0, SPEC_SRC_LOOKUP = 1, SPEC_NSRC = 2 };
+
+typedef struct {
+ double acc[SPEC_NSRC][SPEC_MAX_DRAFTS]; /* a_{j+1}: P(draft j+1 accepted | drafts 1..j accepted) */
+ double acc_w[SPEC_NSRC][SPEC_MAX_DRAFTS]; /* weight behind it (a prior's pseudo-count first) */
+ double t[SPEC_MAX_DRAFTS + 2]; /* T(n), seconds of a forward of n rows; t_w 0 = not measured */
+ double t_w[SPEC_MAX_DRAFTS + 2];
+ double rho, rho_w; /* a row's extra cost relative to one row */
+ double d[SPEC_NSRC], d_w[SPEC_NSRC]; /* seconds a source spends per draft past its first */
+ int picks[SPEC_NSRC]; /* picks so far (the probe's clock) */
+ int since_plain; /* forwards since the last plain step */
+ int off; /* 1: every proposal drafted in full (tests, fixed depths) */
+ /* the run's counts, for the engines' report lines */
+ unsigned long long prop[SPEC_NSRC][SPEC_MAX_DRAFTS], hit[SPEC_NSRC][SPEC_MAX_DRAFTS];
+ unsigned long long verifies[SPEC_NSRC], probes[SPEC_NSRC], declined[SPEC_NSRC];
+} SpecGate;
+
+static inline void spec_gate_init(SpecGate *g, int off) {
+ memset(g, 0, sizeof *g);
+ g->off = off;
+ for (int j = 0; j < SPEC_MAX_DRAFTS; j++) {
+ /* priors, worth two verifies each: an MTP head is trained to be right, and each
+ * draft past the first reads the head's own state instead of the model's */
+ g->acc[SPEC_SRC_MTP][j] = j == 0 ? 0.8 : j == 1 ? 0.6 : 0.5;
+ g->acc[SPEC_SRC_LOOKUP][j] = 0.5;
+ g->acc_w[SPEC_SRC_MTP][j] = g->acc_w[SPEC_SRC_LOOKUP][j] = 2.0;
+ }
+ g->rho = SPEC_RHO0; g->rho_w = 1.0;
+}
+
+static inline void spec_ema(double *x, double *w, double v, double memory) {
+ if (*w < memory) *w += 1.0;
+ *x += (v - *x) / *w;
+}
+
+/* A forward of `rows` rows took `seconds` (a plain step, or a verify). */
+static inline void spec_gate_forward(SpecGate *g, int rows, double seconds) {
+ if (rows < 1 || rows > SPEC_MAX_DRAFTS + 1 || !(seconds > 0.0)) return;
+ spec_ema(&g->t[rows], &g->t_w[rows], seconds, SPEC_T_W);
+ if (rows == 1) { g->since_plain = 0; return; }
+ g->since_plain++;
+ if (g->t_w[1] > 0.0 && g->t[1] > 0.0) {
+ double r = (seconds / g->t[1] - 1.0) / (rows - 1);
+ if (r < 0.0) r = 0.0;
+ if (r > 4.0) r = 4.0;
+ spec_ema(&g->rho, &g->rho_w, r, SPEC_T_W);
+ }
+}
+
+/* A source spent `seconds` on one draft past its first (an MTP head's extra step). */
+static inline void spec_gate_draft_cost(SpecGate *g, int src, double seconds) {
+ if (src < 0 || src >= SPEC_NSRC || !(seconds >= 0.0)) return;
+ spec_ema(&g->d[src], &g->d_w[src], seconds, SPEC_T_W);
+}
+
+/* A verify's drafts from `src` were judged: the first `accepted` stood, and `judged` of
+ * them were looked at (accepted + 1 after a rejection; accepted when every draft stood, or
+ * when the generation ended before the rest were reached: those say nothing). */
+static inline void spec_gate_result(SpecGate *g, int src, int judged, int accepted) {
+ if (src < 0 || src >= SPEC_NSRC || judged < 1) return;
+ if (judged > SPEC_MAX_DRAFTS) judged = SPEC_MAX_DRAFTS;
+ if (accepted > judged) accepted = judged;
+ g->verifies[src]++;
+ for (int j = 0; j < judged; j++) {
+ g->prop[src][j]++;
+ if (j < accepted) g->hit[src][j]++;
+ spec_ema(&g->acc[src][j], &g->acc_w[src][j], j < accepted ? 1.0 : 0.0, SPEC_ACC_W);
+ }
+}
+
+/* T(n) in units of T(1): measured, or the fitted line. */
+static inline double spec_gate_rows_cost(const SpecGate *g, int rows) {
+ if (rows <= 1) return 1.0;
+ if (rows <= SPEC_MAX_DRAFTS + 1 && g->t_w[rows] > 0.0 && g->t_w[1] > 0.0 && g->t[1] > 0.0)
+ return g->t[rows] / g->t[1];
+ return 1.0 + (rows - 1) * g->rho;
+}
+
+/* The expected tokens a verify of k drafts from src yields, and that per unit of time
+ * against a plain step (value 1). */
+static inline double spec_gate_value(const SpecGate *g, int src, int k, double *expect) {
+ double e = 1.0, p = 1.0;
+ for (int j = 0; j < k && j < SPEC_MAX_DRAFTS; j++) { p *= g->acc[src][j]; e += p; }
+ double cost = spec_gate_rows_cost(g, k + 1);
+ if (k > 1 && g->d_w[src] > 0.0 && g->t_w[1] > 0.0 && g->t[1] > 0.0) cost += (k - 1) * g->d[src] / g->t[1];
+ if (expect) *expect = e;
+ return k > 0 ? e / cost : 1.0;
+}
+
+/* How many of the up to maxk drafts src can offer the next verify should carry: 0 for a
+ * plain step. *value gets that depth's value (1 for a plain step). */
+static inline int spec_gate_pick(SpecGate *g, int src, int maxk, double *value) {
+ if (value) *value = 1.0;
+ if (maxk <= 0 || src < 0 || src >= SPEC_NSRC) return 0;
+ if (maxk > SPEC_MAX_DRAFTS) maxk = SPEC_MAX_DRAFTS;
+ if (g->off) { if (value) *value = spec_gate_value(g, src, maxk, NULL); return maxk; }
+ if (g->since_plain >= SPEC_REFRESH) { g->declined[src]++; return 0; } /* keep T(1) measured */
+ int best = 0; double bv = 1.0;
+ for (int k = 1; k <= maxk; k++) {
+ double v = spec_gate_value(g, src, k, NULL);
+ if (v > bv) { bv = v; best = k; }
+ }
+ /* explore one deeper: every SPEC_PROBE picks, every SPEC_PROBE_NEW while that row
+ * count has fewer than 3 measurements (its cost is still the fitted line's guess) */
+ int every = best < maxk && g->t_w[best + 2] < 3.0 ? SPEC_PROBE_NEW : SPEC_PROBE;
+ if (++g->picks[src] % every == 0 && best < maxk) {
+ best++; bv = spec_gate_value(g, src, best, NULL); g->probes[src]++;
+ }
+ if (!best) g->declined[src]++;
+ if (value) *value = bv;
+ return best;
+}
+
+/* What the gate measured, for a report line: T(1), the cost of the row counts seen (in
+ * units of T(1)), the fitted rho, a source's cost per extra draft and its acceptance
+ * estimates by position. */
+#include <stdio.h>
+static inline void spec_gate_describe(const SpecGate *g, int src, char *buf, size_t cap) {
+ size_t at = 0;
+ if (!cap) return;
+ buf[0] = 0;
+#define SPEC_PUT(...) do { int w_ = snprintf(buf + at, cap - at, __VA_ARGS__); \
+ if (w_ < 0 || (size_t)w_ >= cap - at) return; at += (size_t)w_; } while (0)
+ SPEC_PUT("T(1) %.1f ms, rows", g->t_w[1] > 0.0 ? 1e3 * g->t[1] : 0.0);
+ for (int n = 2; n <= SPEC_MAX_DRAFTS + 1; n++)
+ if (g->t_w[n] > 0.0) SPEC_PUT(" %d:%.2f", n, spec_gate_rows_cost(g, n));
+ SPEC_PUT(", rho %.2f", g->rho);
+ if (g->d_w[src] > 0.0) SPEC_PUT(", %.1f ms per extra draft", 1e3 * g->d[src]);
+ SPEC_PUT(", acceptance by position");
+ for (int j = 0; j < 5; j++) SPEC_PUT(" %.2f", g->acc[src][j]);
+#undef SPEC_PUT
+}
+
+/* Prompt lookup: up to k tokens proposed for what follows h[0..len), from the most recent
+ * earlier occurrence of the longest suffix of n tokens, max_n >= n >= min_n, that has one
+ * with at least one token after it. Returns how many were written to out. */
+static inline int spec_lookup(const int *h, int len, int min_n, int max_n, int k, int *out) {
+ if (!h || !out || k < 1 || min_n < 1 || len < min_n + 1) return 0;
+ if (max_n > len - 1) max_n = len - 1;
+ for (int n = max_n; n >= min_n; n--) {
+ const int *suf = h + len - n;
+ for (int i = len - n - 1; i >= 0; i--) { /* h[i..i+n) against the suffix, h[i+n] after it */
+ if (h[i + n - 1] != suf[n - 1] || memcmp(h + i, suf, (size_t)n * sizeof(int))) continue;
+ int g = 0;
+ for (int j = i + n; j < len && g < k; j++) out[g++] = h[j];
+ return g;
+ }
+ }
+ return 0;
+}
+
+#endif /* COLI_SPEC_DRAFT_H */
diff --git a/c/tests/qwen38_mtp_harness.py b/c/tests/qwen38_mtp_harness.py
index 6e338940..b5cd5998 100644
--- a/c/tests/qwen38_mtp_harness.py
+++ b/c/tests/qwen38_mtp_harness.py
@@ -39,10 +39,12 @@ HERE = Path(__file__).resolve().parent.parent
sys.path.insert(0, str(HERE / "tools"))
FAILS = []
+# Q38_MTP_DRAFTS=1: this harness gates the one-draft verify (S=2) and counts its loop;
+# deeper verifies and prompt lookup are tests/spec_drafts_harness.py's
BASE_ENV = {"OMP_NUM_THREADS": "2", "COLI_NO_OMP_TUNE": "1", "COLI_CUDA": "0",
- "Q38_TRUNK_GPU": "0", "NOSTREAM": "1", "USAGE_SAVE": "0"}
+ "Q38_TRUNK_GPU": "0", "NOSTREAM": "1", "USAGE_SAVE": "0", "Q38_MTP_DRAFTS": "1"}
MTP_KEYS = ("Q38_MTP", "Q38_MTP_WIRING", "Q38_MTP_FORCE", "Q38_MTP_DUMP", "DUMP",
- "Q38_PREFILL_BATCH", "Q38_TRUNK_MIN_KB", "Q38_EXPERT_INT4")
+ "Q38_PREFILL_BATCH", "Q38_TRUNK_MIN_KB", "Q38_EXPERT_INT4", "Q38_MTP_DRAFTS", "COLI_LOOKUP")
def check(ok, what):
diff --git a/c/tests/spec_drafts_harness.py b/c/tests/spec_drafts_harness.py
new file mode 100644
index 00000000..744eefb9
--- /dev/null
+++ b/c/tests/spec_drafts_harness.py
@@ -0,0 +1,461 @@
+#!/usr/bin/env python3
+"""Speculative decoding with deeper verifies: MTP drafts up to 3 (qwen38) and
+prompt-lookup drafts up to 5 (qwen38 and qwen36), against plain decoding.
+
+The gate is the one docs/speculative.md states: with drafts, the tokens and the
+last logits are byte-identical to the same run without them, whatever the drafts
+were and wherever they were rejected. On a tiny fixture:
+
+ reference the fixture's prompt with a repeating tail (so the context has
+ n-grams to find), decoded plainly for N_NEW tokens; that decode is
+ the oracle every forced mode drafts from.
+ mtp (qwen38, a fixture with an MTP head) Q38_MTP_DRAFTS=1, 2, 3 in the
+ normal mode and forced: every draft rejected (reject), every one
+ right (accept), alternating verifies (mixed), the rejected row
+ cycling over the verifies (cycle) and the rejection at each row
+ (row1..rowD); the run's forwards, tokens and accepted/proposed
+ drafts must be the count the loop gives. Q38_MTP_DRAFTS=0 (the gate
+ picks the depth) in the normal and cycle modes.
+ lookup COLI_LOOKUP=1 with the gate off (COLI_SPEC_GATE=0) and forced
+ proposals of the oracle's tokens: accept, mixed, cycle, row1..row5
+ and a shallower COLI_LOOKUP_DRAFTS, counts checked; natural lookup
+ with the gate on and off (off: the repeating prompt must give
+ verifies, and accepted drafts), and on qwen38 combined with the MTP
+ head at depth 3 and at the gate's depth.
+ serve (fixtures with a tokenizer) a session of turns with drafts against
+ the same session without: a repeating prompt, its continuation by
+ the reply (prefix reuse of the state a speculating turn left, which
+ a turn cut off mid-verify must leave as plain decoding would), a
+ pinned prompt and a turn restoring the pin, a sampled turn and a
+ logprobs turn. Every DATA and ECHO frame must be the same.
+
+usage: spec_drafts_harness.py --engine ./qwen38 --kind qwen38 --fixture qwen38_tiny_mtp [--ref ref.json]
+ [--env K=V ...] [--mtp] [--serve] [--n-new 40] [--cap 2]
+The extra environment applies to both arms (COLI_VULKAN=1 COLI_VK_CHAIN=1 runs the
+whole gate in the dense chain: with drafts against without, on the device).
+"""
+import argparse
+import json
+import os
+import re
+import subprocess
+import sys
+import tempfile
+import threading
+from pathlib import Path
+
+FAILS = []
+SANITIZE = False # --sanitize: every run's stderr is checked for an ASan or UBSan report
+REQUIRE_KV_SPLIT = False
+SAN_RE = re.compile(rb"ERROR: AddressSanitizer|runtime error:")
+BASE_ENV = {"OMP_NUM_THREADS": "2", "COLI_NO_OMP_TUNE": "1", "COLI_CUDA": "0", "Q38_TRUNK_GPU": "0",
+ "NOSTREAM": "1", "USAGE_SAVE": "0"}
+SPEC_KEYS = ("Q38_MTP", "Q38_MTP_DRAFTS", "Q38_MTP_FORCE", "Q38_MTP_DUMP", "COLI_LOOKUP", "COLI_LOOKUP_DRAFTS",
+ "COLI_LOOKUP_FORCE", "COLI_SPEC_GATE", "DUMP", "SERVE", "N_NEW", "TOK")
+MAX_ROWS = 6
+
+
+def sanitizer_check(stderr, what):
+ if SANITIZE and SAN_RE.search(stderr if isinstance(stderr, bytes) else stderr.encode()):
+ text = stderr.decode(errors="replace") if isinstance(stderr, bytes) else stderr
+ check(False, f"{what}: sanitizer report\n" + text[-4000:])
+
+
+def check(ok, what):
+ print((" ok " if ok else " FAIL ") + what, flush=True)
+ if not ok:
+ FAILS.append(what)
+
+
+class Engine:
+ def __init__(self, path, kind, fixture, cap, extra):
+ self.path, self.kind, self.fixture, self.cap, self.extra = Path(path).resolve(), kind, Path(fixture), cap, extra
+
+ def env(self, spec):
+ env = {k: v for k, v in os.environ.items() if k not in SPEC_KEYS}
+ env.update(BASE_ENV)
+ env.update(self.extra)
+ env["SNAP"] = str(self.fixture)
+ env.update(spec)
+ return env
+
+ def run(self, ref, spec, dump):
+ env = self.env(dict(spec, DUMP=str(dump)))
+ r = subprocess.run([str(self.path), str(self.cap), "8", str(ref)], env=env, capture_output=True,
+ timeout=1800)
+ sanitizer_check(r.stderr, " ".join(f"{k}={v}" for k, v in spec.items()) or "plain")
+ if REQUIRE_KV_SPLIT:
+ trace = rb"\[VK\] " + self.kind.encode() + rb" chain: KV split: [0-9]+ layer steps, [1-9][0-9]* with a host part"
+ check(re.search(trace, r.stderr) is not None,
+ "split KV executed host attention: " + (" ".join(f"{k}={v}" for k, v in spec.items()) or "plain"))
+ return r
+
+
+SPEC_LINE = r"\[{kind} {src}\] run: [0-9.]+ tokens/forward \((\d+) forwards per (\d+) tokens\) \| acceptance [0-9.]+% \((\d+)/(\d+) drafts"
+
+
+def counts(kind, src, stderr):
+ """(forwards, tokens, accepted, drafts) of the run's line for a draft source."""
+ m = re.search(SPEC_LINE.format(kind=kind, src=src), stderr.decode(errors="replace"))
+ return tuple(int(m.group(i)) for i in (1, 2, 3, 4)) if m else None
+
+
+def expected(n_new, depth, mode, row=0, room_cap=MAX_ROWS - 1):
+ """What the speculation loop must count for a CLI run of n_new tokens with every
+ verify carrying min(depth, more - 1) drafts, forced: the prompt's forward gives
+ token 1, then one call per token fed; a pick equal to the next draft is answered
+ from the verify, the first that differs ends it."""
+ drafts = accepted = forwards = tokens = verifies = 0
+ queue = []
+ for s in range(n_new - 1):
+ tokens += 1
+ if queue:
+ if queue.pop(0):
+ accepted += 1
+ continue
+ queue = []
+ more = n_new - 1 - s
+ k = min(depth, more - 1, room_cap)
+ if k >= 1:
+ if mode == "reject":
+ wrong = 1
+ elif mode == "accept":
+ wrong = 0
+ elif mode == "mixed":
+ wrong = 1 if verifies % 2 else 0
+ elif mode == "cycle":
+ wrong = verifies % (k + 1) + 1
+ else:
+ wrong = row
+ queue = [wrong == 0 or j < wrong - 1 for j in range(k)]
+ drafts += k
+ verifies += 1
+ forwards += 1
+ return forwards, tokens, accepted, drafts
+
+
+def engine_tokens(stdout):
+ m = re.search(rb"^C engine :([^\n]*?)(?:Text|$)", stdout, re.M)
+ return [int(t) for t in m.group(1).split()] if m else None
+
+
+def check_device_loss(eng, ref, base_out, tmp):
+ """A late failed chunk must replay embeddings, not earlier chunks' streams."""
+ forced = ({"Q38_MTP": "1", "Q38_MTP_DRAFTS": "3", "Q38_MTP_FORCE": "cycle"}
+ if eng.kind == "qwen38" else
+ {"COLI_LOOKUP": "1", "COLI_LOOKUP_DRAFTS": "5", "COLI_SPEC_GATE": "0", "COLI_LOOKUP_FORCE": "cycle"})
+ probe = eng.run(ref, forced, tmp / "loss-probe.f32")
+ frames = re.search(rb"\[VK\] " + eng.kind.encode() + rb" chain: [0-9]+ forwards, ([0-9]+) frames", probe.stderr)
+ if not frames or int(frames[1]) <= 15:
+ check(False, "device-loss probe: no sufficiently long chain run")
+ print(probe.stderr.decode(errors="replace")[-4000:])
+ return
+ check(probe.returncode == 0 and engine_tokens(probe.stdout) == engine_tokens(base_out),
+ "device-loss probe: speculative tokens = plain")
+ fault = int(frames[1]) - 15
+ lost = eng.run(ref, dict(forced, COLI_VK_CHAIN_FAULT=str(fault)), tmp / "loss.f32")
+ trace = lost.stderr.decode(errors="replace")
+ released = re.search(r": ([1-9][0-9]*) dense matrices on the device only", trace)
+ reloaded = re.search(r"\), ([1-9][0-9]*) read back from disk for the CPU", trace)
+ rebuilt = re.search(r"rebuilding the state of ([1-9][0-9]*) positions", trace)
+ check(bool(released and reloaded and rebuilt), "device loss: host matrices released, restored and recurrent state rebuilt")
+ same = lost.returncode == 0 and engine_tokens(lost.stdout) == engine_tokens(base_out)
+ check(same, f"device loss at frame {fault}: every token = plain decoding after CPU replay")
+ if not same:
+ print(lost.stdout.decode(errors="replace"))
+ print(trace[-4000:])
+
+
+def build_reference(eng, base_ref, n_new, tmp):
+ """The fixture's prompt with a repeating tail, decoded plainly: the oracle."""
+ data = json.loads(Path(base_ref).read_text(encoding="utf-8"))
+ vocab = len(data.get("final_logits") or []) or 64
+ prompt = list(data["prompt_ids"])
+ tail = [t % vocab for t in (7, 11, 13, 17)]
+ prompt += tail * 3 + tail[:2]
+ probe = tmp / "probe.json"
+ probe.write_text(json.dumps({"schema_version": 1, "prompt_ids": prompt, "full_ids": prompt + [0] * n_new}))
+ r = eng.run(probe, {}, tmp / "probe.f32")
+ gen = engine_tokens(r.stdout)
+ if not gen or len(gen) != n_new:
+ sys.exit("could not decode the reference:\n" + r.stdout.decode(errors="replace")
+ + r.stderr.decode(errors="replace")[-3000:])
+ ref = tmp / "ref_spec.json"
+ ref.write_text(json.dumps({"schema_version": 1, "prompt_ids": prompt, "full_ids": prompt + gen}))
+ base = eng.run(ref, {}, tmp / "base.f32")
+ if base.returncode:
+ sys.exit("the plain decode does not reproduce itself:\n" + base.stdout.decode(errors="replace"))
+ return ref, base.stdout, (tmp / "base.f32").read_bytes()
+
+
+def gate(eng, ref, base_out, base_logits, tmp, name, spec, src=None, want=None, extra_check=None):
+ dump = tmp / "spec.f32"
+ if dump.exists():
+ dump.unlink()
+ r = eng.run(ref, spec, dump)
+ same = r.returncode == 0 and r.stdout == base_out and dump.exists() and dump.read_bytes() == base_logits
+ got = counts(eng.kind, src, r.stderr) if src else None
+ if want is not None:
+ same &= got == want
+ elif src:
+ same &= got is not None and got[2] <= got[3] and got[0] <= got[1]
+ extra = extra_check(got, r.stderr) if extra_check else ""
+ if extra is None:
+ same = False
+ extra = ""
+ detail = f" {src} forwards/tokens/accepted/drafts {got}" if src else ""
+ check(same, f"{name}: tokens and last logits = plain decoding{detail}{extra}")
+ if not same:
+ print(r.stdout.decode(errors="replace")[-1500:] + r.stderr.decode(errors="replace")[-2500:])
+ if want is not None:
+ print(f" want {want}")
+ return r
+
+
+def check_mtp(eng, ref, base_out, base_logits, n_new, tmp, quick):
+ print(f"mtp: {eng.fixture}")
+ for depth in ((3,) if quick else (1, 2, 3)):
+ modes = ["reject", "cycle", "row2"] if quick else \
+ ["", "reject", "accept", "mixed", "cycle"] + [f"row{r}" for r in range(1, depth + 1)]
+ for mode in modes:
+ spec = {"Q38_MTP": "1", "Q38_MTP_DRAFTS": str(depth), "Q38_MTP_FORCE": mode}
+ want = None
+ if mode:
+ m, row = (mode, 0) if not mode.startswith("row") else ("row", int(mode[3:]))
+ want = expected(n_new, depth, m, row, 3)
+ gate(eng, ref, base_out, base_logits, tmp, f"MTP depth {depth} {mode or 'normal'}", spec, "MTP", want)
+ for mode in (() if quick else ("", "cycle")):
+ gate(eng, ref, base_out, base_logits, tmp, f"MTP depth by the gate {mode or 'normal'}",
+ {"Q38_MTP": "1", "Q38_MTP_DRAFTS": "auto", "Q38_MTP_FORCE": mode}, "MTP")
+
+
+def check_lookup(eng, ref, base_out, base_logits, n_new, tmp, mtp, quick):
+ print(f"lookup: {eng.fixture}")
+ for drafts in ((5,) if quick else (5, 2)):
+ modes = ["cycle", "row3"] if quick else ["accept", "mixed", "cycle"] + [f"row{r}" for r in range(1, drafts + 1)]
+ for mode in modes:
+ m, row = (mode, 0) if not mode.startswith("row") else ("row", int(mode[3:]))
+ spec = {"COLI_LOOKUP": "1", "COLI_SPEC_GATE": "0", "COLI_LOOKUP_FORCE": mode,
+ "COLI_LOOKUP_DRAFTS": str(drafts)}
+ gate(eng, ref, base_out, base_logits, tmp, f"lookup forced {mode}, {drafts} drafts", spec, "lookup",
+ expected(n_new, drafts, m, row, drafts))
+
+ def proposed(got, _err):
+ return f" (proposals on the repeating prompt: {got[3] if got else 0})" if got and got[3] > 0 else None
+
+ gate(eng, ref, base_out, base_logits, tmp, "lookup, gate off", {"COLI_LOOKUP": "1", "COLI_SPEC_GATE": "0"},
+ "lookup", extra_check=proposed)
+ if quick:
+ return
+ gate(eng, ref, base_out, base_logits, tmp, "lookup, gate on", {"COLI_LOOKUP": "1"}, "lookup")
+ if mtp:
+ gate(eng, ref, base_out, base_logits, tmp, "MTP depth 3 + lookup",
+ {"Q38_MTP": "1", "Q38_MTP_DRAFTS": "3", "COLI_LOOKUP": "1"}, "lookup")
+ gate(eng, ref, base_out, base_logits, tmp, "MTP depth by the gate + lookup",
+ {"Q38_MTP": "1", "Q38_MTP_DRAFTS": "auto", "COLI_LOOKUP": "1"}, "lookup")
+ gate(eng, ref, base_out, base_logits, tmp, "MTP depth 2 + lookup, gate off",
+ {"Q38_MTP": "1", "Q38_MTP_DRAFTS": "2", "COLI_LOOKUP": "1", "COLI_SPEC_GATE": "0"}, "lookup")
+
+
+# ---------- serve ----------
+
+READY = b"\x01\x01READY\x01\x01"
+
+
+def serve_session(eng, spec, alphabet):
+ """Turns sent one at a time. Returns (frames, stderr). A turn's prompt may be
+ built from an earlier turn's reply (the transcript a chat client resends), so
+ both arms run the same procedure and get the same prompts when they agree."""
+ env = eng.env(dict(spec, SERVE="1", TOK=str(eng.fixture / "tokenizer.json"), COLI_PREFIX_LOG="1",
+ Q38_PREFIX_LOG="1", Q38_MAXT="128", Q36_MAXT="512"))
+ p = subprocess.Popen([str(eng.path), str(eng.cap), "8"], env=env, stdin=subprocess.PIPE, stdout=subprocess.PIPE,
+ stderr=subprocess.PIPE)
+ err = []
+ threading.Thread(target=lambda: [err.append(x) for x in iter(p.stderr.readline, b"")], daemon=True).start()
+ frames, replies = [], {}
+ a = alphabet
+ rep = (a[:5] * 6) # a prompt built to repeat
+ turns = [
+ ("t1", lambda: rep, 24, ""),
+ ("t2", lambda: rep + replies["t1"] + a[:3], 20, ""), # the transcript, continued
+ ("t3", lambda: rep + replies["t1"] + a[:3] + replies["t2"], 9, "pin=1"),
+ ("t4", lambda: rep + replies["t1"] + a[:3] + replies["t2"] + a[5:8], 16, ""),
+ ("t5", lambda: rep + a[1:4] * 3, 16, "logprobs=2"),
+ ("t6", lambda: rep + a[1:4] * 3 + replies["t5"], 12, ""),
+ ("t7", lambda: rep, 14, "", "0.9", "0.95"),
+ ("t8", lambda: rep + replies["t1"] + a[:3] + replies["t2"] + a[2:4], 18, ""),
+ ("t9", lambda: rep + replies["t1"] + a[:3] + replies["t2"] + a[2:4] + replies["t8"] + a[4:6], 10, ""),
+ ]
+ try:
+ while True:
+ line = p.stdout.readline()
+ if not line:
+ raise RuntimeError("engine exited before READY")
+ if READY in line:
+ break
+ for t in turns:
+ rid, build, max_tok, ext = t[:4]
+ temp, top_p = (t[4], t[5]) if len(t) > 4 else ("0", "1")
+ prompt = build()
+ p.stdin.write(f"SUBMIT {rid} 0 {len(prompt)} {max_tok} {temp} {top_p}{' ' + ext if ext else ''}\n".encode()
+ + prompt + b"\n")
+ p.stdin.flush()
+ reply = []
+ while True:
+ line = p.stdout.readline()
+ if not line:
+ raise RuntimeError("engine died")
+ kind = line.split(b" ", 1)[0]
+ if kind in (b"DATA", b"ECHO"):
+ body = p.stdout.read(int(line.split()[2]))
+ p.stdout.readline()
+ frames.append(line.rstrip() + b"|" + body)
+ if kind == b"DATA":
+ reply.append(body)
+ elif kind == b"DONE":
+ frames.append(b" ".join(line.split()[:4]))
+ break
+ elif kind == b"ERROR":
+ raise RuntimeError(f"engine error: {line!r}")
+ replies[rid] = b"".join(reply)
+ p.stdin.close()
+ p.wait(timeout=120)
+ except Exception as exc: # noqa: BLE001 -- a failed gate, reported
+ p.kill()
+ p.wait()
+ frames.append(f"harness: {exc!r}".encode())
+ text = b"".join(err).decode(errors="replace")
+ sanitizer_check(text, "serve " + (" ".join(f"{k}={v}" for k, v in spec.items()) or "plain"))
+ return frames, text
+
+
+def check_serve(eng, mtp, quick):
+ print(f"serve: {eng.fixture}")
+ sys.path.insert(0, str(Path(__file__).resolve().parent))
+ from prefix_serve_harness import ensure_byte_tokenizer
+ ensure_byte_tokenizer(eng.fixture)
+ alphabet = b"0123456789" if eng.kind == "qwen38" else b"abcdefghij"
+ base, berr = serve_session(eng, {}, alphabet)
+ breused = reuse_count(berr)
+ check(len([f for f in base if f.startswith(b"DONE")]) == 9, f"plain session: 9 turns ({len(base)} frames)")
+ # (name, environment, what the session must have done for the gate to mean something)
+ arms = [("lookup, gate off", {"COLI_LOOKUP": "1", "COLI_SPEC_GATE": "0"}, "verified"),
+ ("lookup, gate on", {"COLI_LOOKUP": "1"}, "")]
+ if mtp:
+ arms += [("MTP depth 3 + lookup, gate off", {"Q38_MTP": "1", "Q38_MTP_DRAFTS": "3", "COLI_LOOKUP": "1",
+ "COLI_SPEC_GATE": "0"}, "cut"),
+ ("MTP depth by the gate", {"Q38_MTP": "1", "Q38_MTP_DRAFTS": "auto"}, ""),
+ ("MTP depth 2, every draft rejected", {"Q38_MTP": "1", "Q38_MTP_DRAFTS": "2",
+ "Q38_MTP_FORCE": "reject"}, "")]
+ if quick:
+ arms = arms[:1] + arms[2:3]
+ if eng.kind == "qwen36":
+ check_serve_eos_mid_verify(eng, base, alphabet)
+ for name, spec, need in arms:
+ frames, err = serve_session(eng, spec, alphabet)
+ verifies = sum(int(x) for x in re.findall(r"acceptance [0-9.]+% \(\d+/\d+ drafts in (\d+) verifies\)", err))
+ mtp_drafts = sum(int(x) for x in re.findall(r"\[qwen38 MTP\] turn \S+: .*?\(\d+/(\d+) drafts\)", err))
+ cut = len(re.findall(r"ended mid-verify", err))
+ reused = reuse_count(err)
+ meant = reused > 0 and reused == breused and (need != "verified" or verifies > 0) and (need != "cut" or cut > 0)
+ check(frames == base and meant,
+ f"{name}: every frame = plain ({verifies} lookup verifies, {mtp_drafts} MTP drafts, {cut} turns "
+ f"ended mid-verify, {reused} turns reused a prefix, plain {breused})")
+ if frames != base:
+ for x, y in zip(base, frames):
+ if x != y:
+ print(f" first difference:\n plain {x[:160]!r}\n spec {y[:160]!r}")
+ break
+ print(err[-3000:])
+
+
+def reuse_count(err):
+ """Turns that reused a prefix: live state (qwen36) or a snapshot (qwen38). A
+ wrong state after a speculating turn can hide behind a reuse that is refused
+ (the record no longer matches, so the turn prefills from scratch and answers
+ right): the sessions with drafts must reuse as often as the one without."""
+ return len(re.findall(r"\[PREFIX\] reusing|\[qwen38 prefix\] request=\S+ reused=[1-9]", err))
+
+
+def check_serve_eos_mid_verify(eng, base_frames, alphabet):
+ """A turn that ends inside a verify (the sampled token is an end of sequence while
+ drafts are pending) must leave the state of the tokens fed, which the next turn
+ reuses live (qwen36 keeps the state and reuses it when the transcript extends it).
+ The end-of-sequence id is picked from the first turn's reply: the first of its
+ tokens that, made the end of sequence (Q36_EOS), ends a turn mid-verify."""
+ reply = b"".join(f.split(b"|", 1)[1] for f in base_frames if f.startswith(b"DATA "))
+ spec = {"COLI_LOOKUP": "1", "COLI_SPEC_GATE": "0"}
+ tried = []
+ for b in dict.fromkeys(reply):
+ if b >= 128 or len(tried) >= 16:
+ continue
+ tried.append(b)
+ frames, err = serve_session(eng, dict(spec, Q36_EOS=str(b)), alphabet)
+ # a turn that ended mid-verify, and the next turn reusing its live state
+ lines = [x for x in err.splitlines() if x.startswith("[PREFIX]") or "lookup] turn" in x]
+ if not any("ended mid-verify" in x and nxt.startswith("[PREFIX] reusing")
+ for x, nxt in zip(lines, lines[1:])):
+ continue
+ plain, perr = serve_session(eng, {"Q36_EOS": str(b)}, alphabet)
+ reused, preused = reuse_count(err), reuse_count(perr)
+ check(frames == plain and reused > 0 and reused == preused,
+ f"lookup, end of sequence {b} inside a verify, the next turn reusing the state it left: every "
+ f"frame = plain, {reused} turns reused the live state (plain: {preused})")
+ if frames != plain:
+ for x, y in zip(plain, frames):
+ if x != y:
+ print(f" first difference:\n plain {x[:160]!r}\n spec {y[:160]!r}")
+ break
+ return
+ check(False, f"no end-of-sequence id among {tried} ends a turn inside a verify before a turn that reuses it")
+
+
+def main():
+ ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
+ ap.add_argument("--engine", required=True)
+ ap.add_argument("--kind", required=True, choices=("qwen38", "qwen36"))
+ ap.add_argument("--fixture", required=True)
+ ap.add_argument("--ref", help="the fixture's reference (default: <fixture>/ref.json, ref_full.json for qwen36)")
+ ap.add_argument("--cap", type=int, default=2)
+ ap.add_argument("--n-new", type=int, default=40)
+ ap.add_argument("--env", action="append", default=[], help="KEY=VALUE for both arms")
+ ap.add_argument("--mtp", action="store_true", help="the fixture has an MTP head (qwen38)")
+ ap.add_argument("--serve", action="store_true", help="the serve session too (needs a tokenizer)")
+ ap.add_argument("--sanitize", action="store_true", help="fail on an ASan or UBSan report in any run's stderr")
+ ap.add_argument("--require-kv-split", action="store_true", help="require every CLI run to execute attention over host KV rows")
+ ap.add_argument("--device-loss", action="store_true", help="only the late device-loss gate with host copies released")
+ ap.add_argument("--quick", action="store_true",
+ help="a subset (sanitizer runs): depth 3 rejected at row 1, at row 2 and cycling; lookup cycling "
+ "and at row 3; one serve arm per source")
+ a = ap.parse_args()
+ global SANITIZE, REQUIRE_KV_SPLIT
+ SANITIZE = a.sanitize
+ REQUIRE_KV_SPLIT = a.require_kv_split
+ extra = dict(kv.split("=", 1) for kv in a.env)
+ fixture = Path(a.fixture)
+ ref = a.ref or str(fixture / ("ref_full.json" if a.kind == "qwen36" else "ref.json"))
+ eng = Engine(a.engine, a.kind, fixture, a.cap, extra)
+ with tempfile.TemporaryDirectory() as td:
+ tmp = Path(td)
+ if "COLI_USAGE" not in extra:
+ eng.extra["COLI_USAGE"] = str(tmp / "usage")
+ oracle, base_out, base_logits = build_reference(eng, ref, a.n_new, tmp)
+ print(f"reference: {a.n_new} tokens decoded plainly{(' with ' + ' '.join(a.env)) if a.env else ''}")
+ if a.device_loss:
+ check_device_loss(eng, oracle, base_out, tmp)
+ else:
+ if a.mtp:
+ check_mtp(eng, oracle, base_out, base_logits, a.n_new, tmp, a.quick)
+ check_lookup(eng, oracle, base_out, base_logits, a.n_new, tmp, a.mtp, a.quick)
+ if a.serve:
+ check_serve(eng, a.mtp, a.quick)
+ if FAILS:
+ print(f"spec drafts: {len(FAILS)} check(s) failed")
+ return 1
+ print("spec drafts: device-loss replay preserves plain decoding's tokens" if a.device_loss else
+ "spec drafts: every configuration gives plain decoding's tokens and last logits")
+ return 0
+
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/c/tests/test_jev_sdk.py b/c/tests/test_jev_sdk.py
index e44d7986..5239040d 100644
--- a/c/tests/test_jev_sdk.py
+++ b/c/tests/test_jev_sdk.py
@@ -118,6 +118,11 @@ class ColiServe:
except subprocess.TimeoutExpired:
self.process.kill()
self.process.wait(timeout=10)
+ keep = os.environ.get("JEV_SDK_SERVE_LOG") # tests/vulkan_engines.sh decide reads the engines' lines
+ if keep:
+ self.log.seek(0)
+ with open(keep, "ab") as f:
+ f.write(self.log.read())
self.log.close()
diff --git a/c/tests/test_qwen36_weight_alloc.c b/c/tests/test_qwen36_weight_alloc.c
new file mode 100644
index 00000000..cf7983bb
--- /dev/null
+++ b/c/tests/test_qwen36_weight_alloc.c
@@ -0,0 +1,108 @@
+/* Vulkan's page-aligned q/h rows must use the matching release on Windows.
+ * Track the allocation family even on Linux, where free would hide a mismatch. */
+#define _GNU_SOURCE
+#include <stdio.h>
+#include <stdlib.h>
+#include "../compat.h"
+
+static void *aligned_rows[32];
+static int nrows, aligned_calls, wrong_free;
+static int weight_memalign(void **p, size_t al, size_t n) {
+ int rc = posix_memalign(p, al, n);
+ if (!rc) { if (nrows == 32) abort(); aligned_rows[nrows++] = *p; aligned_calls++; }
+ return rc;
+}
+static int remove_row(void *p) {
+ for (int i = 0; i < nrows; i++) if (aligned_rows[i] == p) {
+ aligned_rows[i] = aligned_rows[--nrows]; return 1;
+ }
+ return 0;
+}
+static void weight_aligned_free(void *p) {
+ if (p && !remove_row(p)) abort();
+ compat_aligned_free(p);
+}
+static void weight_free(void *p) {
+ if (p && remove_row(p)) { wrong_free++; compat_aligned_free(p); }
+ else free(p);
+}
+#undef posix_memalign
+#undef compat_aligned_free
+#define posix_memalign weight_memalign
+#define compat_aligned_free weight_aligned_free
+#define free weight_free
+#define QWEN36_NO_MAIN
+#include "../qwen36.c"
+#undef free
+
+#define CHECK(x) do { if (!(x)) { fprintf(stderr, "%s:%d: %s\n", __FILE__, __LINE__, #x); return 1; } } while (0)
+#ifdef COLI_VULKAN
+/* Combine explicit import with host removal. The freed pages must never remain a
+ * Vulkan tensor's backing storage, and aligned int8/f16 rows use the right release. */
+static int check_device_only(void) {
+ setenv("COLI_VK_DENSE_HOST", "0", 1);
+ g_vk_ready = coli_vk_init_env_tier("qwen36", 0);
+ CHECK(g_vk_ready);
+ CHECK(coli_vk_dense_host_decide("qwen36", 1, 4096));
+ g_vk_import = 1;
+ for (int half = 0; half < 2; half++) {
+ QW w = {0}; w.I = 64; w.O = 16;
+ w.vk_name = strdup("test.dense.weight");
+ float x[64], y[16], want[16];
+ for (int i = 0; i < 64; i++) x[i] = 1.f;
+ if (half) {
+ w.h = q36_walloc(64 * 16 * sizeof(uint16_t));
+ CHECK(w.h);
+ for (int i = 0; i < 64 * 16; i++) w.h[i] = 0x3c00; /* 1.0 */
+ for (int o = 0; o < 16; o++) want[o] = 64.f;
+ } else {
+ w.q = q36_walloc(64 * 16); w.sc = malloc(16 * sizeof(float));
+ CHECK(w.q && w.sc);
+ for (int o = 0; o < 16; o++) {
+ w.sc[o] = 0.125f; want[o] = 0.f;
+ for (int i = 0; i < 64; i++) {
+ w.q[o * 64 + i] = (i + o) % 7 - 3;
+ want[o] += w.q[o * 64 + i] * w.sc[o];
+ }
+ }
+ }
+ size_t bytes = 0; int n = 0;
+ q36_dho_drop(&w, &bytes, &n);
+ CHECK(n == 1 && bytes > 0 && w.vk_gone && !w.vk_imported);
+ CHECK(!w.q && !w.h && !wrong_free && !nrows);
+ CHECK(vk_dense_matmul(y, x, &w, 1, 64, 16));
+ for (int o = 0; o < 16; o++) CHECK(fabsf(y[o] - want[o]) < 1e-4f);
+ qw_free(&w);
+ }
+ coli_vk_shutdown();
+ return 0;
+}
+#endif
+int main(void) {
+ QW w = {0};
+ w.q = q36_walloc(64);
+ w.h = q36_walloc(128);
+ CHECK(w.q && w.h);
+ qw_free(&w);
+ CHECK(!wrong_free && !nrows && !w.q && !w.h);
+ /* The int8 intermediate is released as soon as int4 packing succeeds. */
+ setenv("COLI_DENSE_BITS", "4", 1);
+ unsetenv("COLI_DENSE_INT4");
+ unsetenv("COLI_DENSE_KEEP_I8");
+ unsetenv("COLI_CUDA");
+ float values[64 * 16];
+ for (int i = 0; i < 64 * 16; i++) values[i] = (float)(i % 19 - 9) / 9.f;
+ qw_quantize(values, 64, 16, "model.layers.0.self_attn.q_proj.weight", &w);
+ CHECK(w.q4 && !w.q);
+ qw_free(&w);
+ CHECK(!wrong_free && !nrows);
+#ifdef COLI_VULKAN
+ const char *e = getenv("COLI_VULKAN");
+ CHECK(aligned_calls == ((e && atoi(e)) ? 3 : 0));
+ if (e && atoi(e)) CHECK(!check_device_only());
+#else
+ CHECK(aligned_calls == 0);
+#endif
+ puts("qwen36 weight allocation: matching release and independent device storage after host removal");
+ return 0;
+}
diff --git a/c/tests/test_qwen38_spec_alloc.c b/c/tests/test_qwen38_spec_alloc.c
new file mode 100644
index 00000000..7d750ff1
--- /dev/null
+++ b/c/tests/test_qwen38_spec_alloc.c
@@ -0,0 +1,68 @@
+/* A failed deeper verify must release its unfinished rollback slot, preserve
+ * complete slots, and permit a later retry. Inject every allocation failure
+ * while growing from one slot to three, including the PLE copy. */
+#define _GNU_SOURCE
+#include <stdio.h>
+#include <stdlib.h>
+
+static void *live[128];
+static int nlive, track, fail_at = -1, attempts;
+
+static int refuse(void) { return track && attempts++ == fail_at; }
+static void *remember(void *p) {
+ if (track && p) {
+ if (nlive == 128) abort();
+ live[nlive++] = p;
+ }
+ return p;
+}
+static void *spec_malloc(size_t n) { return refuse() ? NULL : remember(malloc(n)); }
+static void *spec_calloc(size_t n, size_t s) { return refuse() ? NULL : remember(calloc(n, s)); }
+static void spec_free(void *p) {
+ for (int i = 0; i < nlive; i++) if (live[i] == p) { live[i] = live[--nlive]; break; }
+ free(p);
+}
+
+#define malloc spec_malloc
+#define calloc spec_calloc
+#define free spec_free
+#define QWEN38_NO_MAIN
+#include "../qwen38.c"
+#undef malloc
+#undef calloc
+#undef free
+
+#define CHECK(x) do { if (!(x)) { fprintf(stderr, "%s:%d: %s\n", __FILE__, __LINE__, #x); return 1; } } while (0)
+
+int main(void) {
+ const int per_slot = 7; /* two pointer tables, two recurrent layers, PLE */
+ for (int failure = 0; failure < 2 * per_slot; failure++) {
+ Model m = {0};
+ m.c.layers = 3;
+ m.c.is_attn = calloc(3, sizeof(*m.c.is_attn));
+ m.c.is_attn[1] = 1;
+ m.c.dn_vheads = m.c.dn_kdim = m.c.dn_vdim = m.c.dn_conv_dim = 2;
+ m.c.dn_convk = m.c.ple_convk = m.c.hc_width = m.c.ngram_size = 2;
+ m.PLE_conv_state = calloc(4, sizeof(float));
+ track = 1; fail_at = -1; attempts = 0;
+ CHECK(q38_spec_alloc(&m, 1));
+ CHECK(nlive == per_slot);
+ float *first = m.snap_rec[0][0];
+ first[0] = 19.0f;
+ fail_at = failure; attempts = 0;
+ CHECK(!q38_spec_alloc(&m, 3));
+ CHECK(m.snap_slots == 1 + failure / per_slot);
+ CHECK(nlive == m.snap_slots * per_slot);
+ CHECK(!m.snap_rec[m.snap_slots] && !m.snap_conv[m.snap_slots] && !m.snap_ple[m.snap_slots]);
+ CHECK(m.snap_rec[0][0] == first && first[0] == 19.0f);
+ fail_at = -1; attempts = 0;
+ CHECK(q38_spec_alloc(&m, 3));
+ CHECK(m.snap_slots == 3 && nlive == 3 * per_slot);
+ CHECK(m.snap_rec[0][0] == first && first[0] == 19.0f);
+ track = 0;
+ q38_model_free(&m);
+ CHECK(nlive == 0);
+ }
+ puts("qwen38 speculative allocation: 14 failures release unfinished slots, preserve state and retry cleanly");
+ return 0;
+}
diff --git a/c/tests/test_resource_plan.py b/c/tests/test_resource_plan.py
index fc6d9052..a8930ee5 100644
--- a/c/tests/test_resource_plan.py
+++ b/c/tests/test_resource_plan.py
@@ -33,7 +33,8 @@ def write_shard(path, tensors):
for tensor in tensors:
name, size, *metadata = tensor
dtype = metadata[0] if metadata else "U8"
- header[name] = {"dtype": dtype, "shape": [size],
+ shape = metadata[1] if len(metadata) > 1 else [size]
+ header[name] = {"dtype": dtype, "shape": shape,
"data_offsets": [offset, offset + size]}
offset += size
raw = json.dumps(header).encode()
@@ -344,6 +345,51 @@ class ResourcePlanTest(unittest.TestCase):
kept = environment_for_plan(plan, {"K3_EXPERT_GB": "3.5"})
self.assertEqual(kept["K3_EXPERT_GB"], "3.5")
+ def test_glm53_dense_ram_uses_loaded_components_and_each_plans_environment(self):
+ model = self._glm53_model()
+ write_shard(model / "model.safetensors", [
+ ("model.language_model.embed_tokens.weight", 2048, "BF16", [16, 64]),
+ ("model.language_model.layers.0.self_attn.q_proj.weight", 2048, "BF16", [16, 64]),
+ # The eight-column projection falls back to int8 even at GLM53_BITS=4.
+ ("model.language_model.layers.0.self_attn.g_b_proj.weight", 512, "F32", [16, 8]),
+ ("lm_head.weight", 2048, "F16", [16, 64]),
+ ("model.language_model.layers.0.mlp.gate.weight", 256, "BF16", [2, 64]),
+ ("model.language_model.layers.0.input_layernorm.weight", 128, "BF16", [64]),
+ ("model.visual.blocks.0.attn.qkv.weight", 2048, "BF16", [16, 64]),
+ # Absorbed kv_b, unknown components and packed source weights never
+ # receive a guessed checkpoint-wide shrink factor.
+ ("model.language_model.layers.1.self_attn.kv_b_proj.weight", 1024, "BF16", [64, 8]),
+ ("model.language_model.layers.0.unknown_projection.weight", 128, "BF16", [8, 8]),
+ ("model.language_model.layers.0.mlp.up_proj.weight", 512, "U8", [16, 32]),
+ ("model.language_model.layers.0.mlp.up_proj.weight.qs", 64, "F32", [16, 1]),
+ ])
+ kwargs = dict(context=32, available_memory=16 * GB, available_disk=16 * GB, gpus=[])
+ retained = 4096 + 512 + 256 + 4096 + 2048 + 256 + 512 + 64
+ expected = {4: retained + 2 * (512 + 64) + (128 + 64),
+ 8: retained + 2 * (1024 + 64) + (128 + 64),
+ 32: retained + 2 * 4096 + 512}
+ with mock.patch.dict(os.environ, {"GLM53_BITS": "32"}):
+ scan = analyze_model(model)
+ self.assertEqual(scan["dense_bytes"], expected[4])
+ self.assertEqual(scan["embed_bytes"], 4096)
+ # The cached scan is reused for all plans. An explicit env wins over
+ # os.environ, and none of these calls bakes its format into the cache.
+ with mock.patch("resource_plan._tensor_sizes", side_effect=AssertionError("cache miss")):
+ for bits in (4, 32, 8, 4):
+ plan = build_plan(model, env={"GLM53_BITS": str(bits)}, **kwargs)
+ self.assertEqual(plan["tiers"]["ram"]["dense_bytes"], expected[bits])
+ inherited = build_plan(model, **kwargs)
+ self.assertEqual(inherited["tiers"]["ram"]["dense_bytes"], expected[32])
+ with mock.patch.dict(os.environ, {"GLM53_BITS": "8"}):
+ self.assertEqual(analyze_model(model)["dense_bytes"], scan["dense_bytes"])
+
+ def test_glm53_rejects_a_dense_precision_the_engine_cannot_load(self):
+ model = self._glm53_model()
+ for bits in ("0", "16", "", "invalid"):
+ with self.subTest(bits=bits), self.assertRaisesRegex(ValueError, "GLM53_BITS"):
+ build_plan(model, context=32, env={"GLM53_BITS": bits},
+ available_memory=16 * GB, available_disk=16 * GB, gpus=[])
+
def test_glm53_plan_exports_the_expert_cache_knob_the_engine_reads(self):
"""glm53.c reads GLM53_EXPERT_GB, not RAM_GB. --auto-tier exported
RAM_GB (inert) and relied on COLI_PLAN_CAP as argv, which a direct
@@ -1005,14 +1051,14 @@ memInfo.free: 23.50 GB (97%)
}
(self.model / "config.json").write_text(json.dumps(config))
MiB = 1 << 20
- tensors = [("model.embed_tokens.weight", 256, "BF16"),
+ tensors = [("model.embed_tokens.weight", 256, "BF16", [16, 8]),
# dense matmul matrices the engine offers to the tier: one
# big enough to go (4 MiB BF16 -> 2 MiB int8), one under the
# 1 MiB line that stays on the CPU, one PLE projection that
# is never offered
- ("model.layers.0.linear_attn.in_proj_qkv.weight", 4 * MiB, "BF16"),
- ("model.layers.0.mlp.gate.weight", 1024, "BF16"),
- ("model.layers.1.ple.key_proj.weight", 4 * MiB, "BF16")]
+ ("model.layers.0.linear_attn.in_proj_qkv.weight", 4 * MiB, "BF16", [1024, 2048]),
+ ("model.layers.0.mlp.gate.weight", 1024, "BF16", [2, 256]),
+ ("model.layers.1.ple.key_proj.weight", 4 * MiB, "BF16", [1024, 2048])]
for projection in ("gate_proj", "up_proj", "down_proj"):
prefix = f"model.layers.0.mlp.experts.0.{projection}"
tensors.append((prefix + ".weight", 32, "F8_E4M3"))
@@ -1078,6 +1124,203 @@ memInfo.free: 23.50 GB (97%)
available_disk=16 * GB, gpus=[gpu])
self.assertEqual(tiny["tiers"]["vram"]["trunk_bytes"], 0)
+ def test_qwen38_plan_holds_the_trunk_as_int8_rows(self):
+ # q38_trunk_cpu_int8 (on by default) keeps the trunk's large matrices as
+ # int8 rows: the plan's RAM is the checkpoint's BF16 less those bytes.
+ self.write_qwen38()
+ MiB = 1 << 20
+ analysis = analyze_model(self.model)
+ kwargs = dict(context=64, available_memory=16 * GB, available_disk=16 * GB, gpus=[])
+ plan = build_plan(self.model, env={}, **kwargs)
+ self.assertEqual(plan["tiers"]["ram"]["dense_bytes"], analysis["dense_bytes"] - 2 * MiB + 4 * 1024)
+ bf16 = build_plan(self.model, env={"Q38_TRUNK_CPU_INT8": "0"}, **kwargs)
+ self.assertEqual(bf16["tiers"]["ram"]["dense_bytes"], analysis["dense_bytes"])
+
+ def test_qwen38_cpu_int8_selection_is_recomputed_for_each_plan(self):
+ self.write_qwen38()
+ kwargs = dict(context=64, available_memory=16 * GB, available_disk=16 * GB, gpus=[])
+ baseline = analyze_model(self.model)["dense_bytes"]
+ # Every call after the scan hits the same cache. These knobs must still
+ # affect RAM: neither the GPU inventory nor a cached default is the CPU table.
+ for env in ({"Q38_TRUNK_SKIP": "dnqkv"},
+ {"Q38_TRUNK_SKIP": "router,dnqkv,attnq"},
+ {"Q38_TRUNK_MIN_KB": "999999999"},
+ {"Q38_TRUNK_MIN_KB": "-1"}):
+ with self.subTest(env=env):
+ plan = build_plan(self.model, env=env, **kwargs)
+ self.assertEqual(plan["tiers"]["ram"]["dense_bytes"], baseline)
+ regular = build_plan(self.model, env={}, **kwargs)
+ no_gpu = build_plan(self.model, env={"Q38_TRUNK_GPU": "0"}, **kwargs)
+ self.assertEqual(regular["tiers"]["ram"]["dense_bytes"],
+ no_gpu["tiers"]["ram"]["dense_bytes"])
+ self.assertLess(regular["tiers"]["ram"]["dense_bytes"], baseline)
+
+ def test_qwen38_cpu_int8_threshold_includes_scales_and_loaded_dtype(self):
+ self.write_qwen38()
+ kwargs = dict(context=64, available_memory=16 * GB, available_disk=16 * GB, gpus=[])
+ rows, cols = 256, 4092
+ # At exactly 1 MiB including row scales, but below 1 MiB of values.
+ # F16 is expanded to F32 by q38_load_weight, BF16 remains native.
+ for dtype, width, loaded_width in (("BF16", 2, 2), ("F16", 2, 4), ("F32", 4, 4)):
+ with self.subTest(dtype=dtype):
+ write_shard(self.model / "model.safetensors", [
+ ("model.layers.0.linear_attn.in_proj_qkv.weight", rows * cols * width,
+ dtype, [rows, cols]),
+ # Embedding and PLE are never CPU-int8 trunk candidates,
+ # even when they meet its size threshold.
+ ("model.embed_tokens.weight", rows * cols * 2, "BF16", [rows, cols]),
+ ("model.layers.1.ple.key_proj.weight", rows * cols * 2, "BF16", [rows, cols]),
+ ])
+ before = build_plan(self.model, env={"Q38_TRUNK_CPU_INT8": "0"}, **kwargs)
+ after = build_plan(self.model, env={}, **kwargs)
+ self.assertEqual(before["tiers"]["ram"]["dense_bytes"] - after["tiers"]["ram"]["dense_bytes"],
+ rows * cols * loaded_width - (rows * cols + 4 * rows))
+ above = build_plan(self.model, env={"Q38_TRUNK_MIN_KB": "1025"}, **kwargs)
+ self.assertEqual(above["tiers"]["ram"]["dense_bytes"], before["tiers"]["ram"]["dense_bytes"])
+
+ def test_vulkan_device_only_reserves_cpu_components_and_unknown_matrices(self):
+ self.write_qwen38()
+ write_shard(self.model / "retained.safetensors", [
+ ("model.layers.0.input_layernorm.weight", 32, "BF16", [16]),
+ ("model.layers.0.linear_attn.conv1d.weight", 256, "BF16", [4, 4, 8]),
+ ("model.layers.0.unknown_projection.weight", 2048, "F32", [16, 32]),
+ ])
+ kwargs = dict(context=64, ram_gb=4, available_memory=16 * GB,
+ available_disk=16 * GB, gpus=[],
+ vulkan={"type": "discrete", "budget_bytes": 16 * GB})
+ on = {"COLI_VULKAN": "1", "COLI_VK_CHAIN": "1"}
+ kept = build_plan(self.model, env=dict(on, COLI_VK_DENSE_HOST="1"), **kwargs)["tiers"]["ram"]
+ dropped = build_plan(self.model, env=dict(on, COLI_VK_DENSE_HOST="0"), **kwargs)["tiers"]["ram"]
+ # Small BF16 vectors/conv weights expand to F32 in this loader.
+ retained = 256 + 32 * 2 + 256 * 2 + 2048
+ self.assertEqual(dropped["dense_bytes"], retained)
+ self.assertEqual(dropped["dense_on_device_bytes"], kept["dense_bytes"] - retained)
+ self.assertEqual(dropped["expert_cache_bytes"] - kept["expert_cache_bytes"],
+ dropped["dense_on_device_bytes"])
+
+ def test_qwen36_vulkan_credit_keeps_vision_vectors_and_unsupported_storage(self):
+ model = self._qwen_dense_model()
+ write_shard(model / "model.safetensors", [
+ ("model.language_model.embed_tokens.weight", 8192, "BF16", [64, 64]),
+ ("model.language_model.layers.0.self_attn.q_proj.weight", 8192, "BF16", [64, 64]),
+ ("model.language_model.layers.0.linear_attn.in_proj_a.weight", 8192, "BF16", [64, 64]),
+ ("model.language_model.layers.0.mlp.shared_expert_gate.weight", 128, "BF16", [64]),
+ ("model.visual.blocks.0.attn.qkv.weight", 8192, "BF16", [64, 64]),
+ ("model.language_model.layers.0.self_attn.k_proj.weight", 2048, "U8", [64, 32]),
+ ])
+ kwargs = dict(context=32, available_memory=16 * GB, available_disk=16 * GB, gpus=[],
+ vulkan={"type": "discrete", "budget_bytes": 16 * GB})
+ on = {"COLI_VULKAN": "1", "COLI_VK_DENSE_HOST": "0"}
+ host = build_plan(model, env=dict(on, COLI_VK_DENSE_HOST="1"), **kwargs)["tiers"]["ram"]
+ for bits, credit in (("8", 4096), ("4", 2048), ("16", 0)):
+ with self.subTest(bits=bits):
+ ram = build_plan(model, env=dict(on, COLI_DENSE_BITS=bits), **kwargs)["tiers"]["ram"]
+ self.assertEqual(ram["dense_on_device_bytes"], credit)
+ self.assertEqual(ram["dense_bytes"], host["dense_bytes"] - credit)
+
+ def test_vulkan_device_only_dense_leaves_the_embedding_in_ram(self):
+ # COLI_VULKAN=1 with the dense weights on the device only: the RAM budget
+ # holds the embedding table alone, and the experts' cache takes the rest.
+ self.write_qwen38()
+ MiB = 1 << 20
+ kwargs = dict(context=64, ram_gb=4, available_memory=16 * GB,
+ available_disk=16 * GB, gpus=[])
+ host = build_plan(self.model, env={"COLI_VULKAN": "1", "COLI_VK_CHAIN": "1",
+ "COLI_VK_DENSE_HOST": "1"}, **kwargs)
+ only = build_plan(self.model, env={"COLI_VULKAN": "1", "COLI_VK_CHAIN": "1",
+ "COLI_VK_DENSE_HOST": "0"}, **kwargs)
+ self.assertEqual(only["tiers"]["ram"]["dense_bytes"], 256)
+ self.assertEqual(only["tiers"]["ram"]["dense_on_device_bytes"],
+ host["tiers"]["ram"]["dense_bytes"] - 256)
+ self.assertEqual(host["tiers"]["ram"]["dense_on_device_bytes"], 0)
+ self.assertGreater(only["tiers"]["ram"]["expert_cache_bytes"],
+ host["tiers"]["ram"]["expert_cache_bytes"])
+ self.assertIn("on the Vulkan device only", format_plan(only))
+ self.assertNotIn("on the Vulkan device only", format_plan(host))
+ # unset: an integrated GPU holds them when the chain is on (no host copy:
+ # the same RAM twice); a discrete one when they fit its free memory
+ igpu = build_plan(self.model, env={"COLI_VULKAN": "1", "COLI_VK_CHAIN": "1"},
+ vulkan={"type": "integrated"}, **kwargs)
+ self.assertEqual(igpu["tiers"]["ram"]["dense_bytes"], 256)
+ small = build_plan(self.model, env={"COLI_VULKAN": "1"},
+ vulkan={"type": "discrete", "budget_bytes": GB + MiB}, **kwargs)
+ self.assertEqual(small["tiers"]["ram"]["dense_bytes"], host["tiers"]["ram"]["dense_bytes"])
+ # without COLI_VULKAN nothing changes
+ cpu = build_plan(self.model, env={"COLI_VK_DENSE_HOST": "0"}, **kwargs)
+ self.assertEqual(cpu["tiers"]["ram"]["dense_bytes"], host["tiers"]["ram"]["dense_bytes"])
+
+ def test_vulkan_device_only_decision_follows_the_engine_rule(self):
+ from resource_plan import vk_dense_device_only
+ on = {"COLI_VULKAN": "1"}
+ igpu, dgpu = {"type": "integrated"}, {"type": "discrete", "budget_bytes": 16 * GB}
+ cases = [
+ ({}, dgpu, 1, False), # no Vulkan
+ (dict(on), dgpu, 5 * GB, True), # discrete, fits
+ (dict(on), dgpu, 15.5 * GB, False), # discrete, 1 GiB short
+ (dict(on, COLI_VK_DENSE_HOST="1"), dgpu, 1, False), # copies kept
+ (dict(on, COLI_VK_CHAIN="0", COLI_VK_DENSE="0"), dgpu, 1, False), # dense on the CPU
+ (dict(on), igpu, 1, False), # qwen38's chain is off there
+ (dict(on, COLI_VK_CHAIN="1"), igpu, 1, True),
+ (dict(on, COLI_VK_DENSE="1"), igpu, 1, True),
+ (dict(on, COLI_VK_DENSE_HOST="0", COLI_VK_CHAIN="1"), {"type": "cpu"}, 1, True),
+ (dict(on, COLI_VK_CHAIN="1"), {"type": "cpu"}, 1, False), # Lavapipe keeps them
+ ]
+ for env, device, dense, want in cases:
+ with self.subTest(env=env, device=device["type"], dense=dense):
+ self.assertEqual(vk_dense_device_only(dense, "qwen38", env, device)[0], want)
+ # qwen36 and olmoe run the chain by default on an integrated GPU
+ self.assertTrue(vk_dense_device_only(1, "qwen36", dict(on), igpu)[0])
+
+ def test_vulkan_host_copies_use_free_budget_and_supported_engine(self):
+ from resource_plan import vk_dense_device_only
+ GiB = 1 << 30
+ on = {"COLI_VULKAN": "1"}
+ device = {"type": "discrete", "budget_bytes": 8 * GiB,
+ "heaps": [{"device_local": True, "usage": 4 * GiB}]}
+ self.assertFalse(vk_dense_device_only(4 * GiB, "qwen38", on, device)[0])
+ self.assertTrue(vk_dense_device_only(3 * GiB, "qwen38", on, device)[0])
+ self.assertFalse(vk_dense_device_only(3 * GiB + 1, "qwen38", on, device)[0])
+ forced = dict(on, COLI_VK_DENSE_HOST="0")
+ for family in ("laya", "gliner_decide", "qwen_image", None):
+ with self.subTest(family=family):
+ self.assertFalse(vk_dense_device_only(1, family, forced, device)[0])
+ self.assertFalse(vk_dense_device_only(
+ 1, "qwen38", dict(forced, COLI_CUDA="1"), device)[0])
+ igpu = {"type": "integrated"}
+ # Dense and chain are independent decisions in the runtime. Disabling
+ # the per-matrix path alone still leaves qwen36's default chain on.
+ self.assertTrue(vk_dense_device_only(
+ 1, "qwen36", dict(on, COLI_VK_DENSE="0"), igpu)[0])
+ self.assertTrue(vk_dense_device_only(
+ 1, "qwen38", dict(on, COLI_VK_CHAIN="0", COLI_VK_TIER="0"), igpu)[0])
+ for gpu in (device, igpu):
+ with self.subTest(glm_device=gpu["type"]):
+ off = dict(on, COLI_VK_CHAIN="0", COLI_VK_TIER="0")
+ self.assertFalse(vk_dense_device_only(1, "glm", off, gpu)[0])
+ self.assertTrue(vk_dense_device_only(1, "glm", dict(off, COLI_VK_DENSE="1"), gpu)[0])
+
+ def test_vulkan_integrated_dense_copy_is_still_physical_ram(self):
+ self.write_qwen38()
+ kwargs = dict(context=64, ram_gb=4, available_memory=16 * GB,
+ available_disk=16 * GB, gpus=[], vulkan={"type": "integrated"})
+ cpu = build_plan(self.model, env={}, **kwargs)
+ on = {"COLI_VULKAN": "1", "COLI_VK_CHAIN": "1"}
+ kept = build_plan(self.model, env=dict(on, COLI_VK_DENSE_HOST="1"), **kwargs)
+ dropped = build_plan(self.model, env=dict(on, COLI_VK_DENSE_HOST="0"), **kwargs)
+ cpu_ram, kept_ram, dropped_ram = (p["tiers"]["ram"] for p in (cpu, kept, dropped))
+ device_copy = dropped_ram["dense_on_device_bytes"]
+ self.assertGreater(device_copy, 0)
+ self.assertEqual(dropped_ram["shared_device_dense_bytes"], device_copy)
+ self.assertEqual(kept_ram["shared_device_dense_bytes"], device_copy)
+ self.assertEqual(dropped_ram["runtime_bytes"], cpu_ram["runtime_bytes"] + device_copy)
+ self.assertEqual(kept_ram["runtime_bytes"], dropped_ram["runtime_bytes"])
+ # One GPU copy replaces one CPU copy, so dropping the duplicate recovers
+ # exactly its bytes without pretending that the weights vanished from RAM.
+ self.assertEqual(dropped_ram["expert_cache_bytes"], cpu_ram["expert_cache_bytes"])
+ self.assertEqual(dropped_ram["expert_cache_bytes"] - kept_ram["expert_cache_bytes"],
+ device_copy)
+ self.assertTrue(dropped["memory"]["unified"])
+
def test_qwen38_int4_sidecar_prices_the_cache_with_its_records(self):
# tools/convert_qwen38_experts_int4.py's index for this geometry
# (2 layers, 2 experts, hidden 8, moe_intermediate 4): codes 16+16+16
diff --git a/c/tests/test_spec_draft.c b/c/tests/test_spec_draft.c
new file mode 100644
index 00000000..12954860
--- /dev/null
+++ b/c/tests/test_spec_draft.c
@@ -0,0 +1,137 @@
+/* spec_draft.h on its own: what prompt lookup proposes, and what the gate decides.
+ *
+ * lookup the longest suffix wins over a shorter one, the most recent occurrence
+ * over an older one, the proposal stops at the history's end (a
+ * continuation that runs into the suffix itself is fine), k caps it,
+ * min_n and max_n bound the n-gram, and nothing is proposed without an
+ * earlier occurrence or a token after it;
+ * gate the expected tokens E(k) and the value of a depth follow the formula
+ * in the header; acceptance moves toward what was judged (and only the
+ * judged positions move); a measured forward cost replaces the fitted
+ * line; drafting stops when the measured cost outweighs the acceptance,
+ * comes back through the probe (sooner while the deeper verify's cost is
+ * unmeasured), and a plain step is forced after SPEC_REFRESH forwards
+ * without one; off drafts every proposal in full.
+ *
+ * cc -O2 tests/test_spec_draft.c -o tests/test_spec_draft -lm && ./tests/test_spec_draft */
+#include <math.h>
+#include <stdio.h>
+#include "../spec_draft.h"
+
+static int fails;
+#define CHECK(c, ...) do { if (!(c)) { fails++; printf("FAIL %s:%d: ", __FILE__, __LINE__); printf(__VA_ARGS__); printf("\n"); } } while (0)
+
+static int same(const int *a, const int *b, int n) { for (int i = 0; i < n; i++) if (a[i] != b[i]) return 0; return 1; }
+
+static void test_lookup(void) {
+ int out[8] = {0};
+ /* the suffix "3 4" occurred at 2..3 and is followed by 5 6 7 */
+ int h1[] = {1, 2, 3, 4, 5, 6, 7, 9, 3, 4};
+ int n = spec_lookup(h1, 10, 2, 4, 5, out);
+ int w1[] = {5, 6, 7, 9, 3};
+ CHECK(n == 5 && same(out, w1, 5), "bigram continuation: n=%d", n);
+ n = spec_lookup(h1, 10, 2, 4, 2, out);
+ CHECK(n == 2 && out[0] == 5 && out[1] == 6, "k caps the proposal: n=%d", n);
+ /* the most recent earlier occurrence wins: "8 9" at 0 (then 1) and at 3 (then 2) */
+ int h2[] = {8, 9, 1, 8, 9, 2, 0, 8, 9};
+ n = spec_lookup(h2, 9, 2, 4, 3, out);
+ CHECK(n == 3 && out[0] == 2 && out[1] == 0 && out[2] == 8, "most recent occurrence: n=%d first %d", n, out[0]);
+ /* the longest suffix wins: "7 8 9" occurs once (then 5), "8 9" more recently (then 6) */
+ int h3[] = {7, 8, 9, 5, 1, 8, 9, 6, 2, 7, 8, 9};
+ n = spec_lookup(h3, 12, 2, 4, 1, out);
+ CHECK(n == 1 && out[0] == 5, "longest suffix: n=%d got %d", n, out[0]);
+ n = spec_lookup(h3, 12, 2, 2, 1, out);
+ CHECK(n == 1 && out[0] == 6, "max_n 2 takes the bigram: n=%d got %d", n, out[0]);
+ /* a periodic history: the continuation runs up to the end of the history */
+ int h4[] = {1, 2, 1, 2, 1, 2};
+ n = spec_lookup(h4, 6, 2, 4, 5, out);
+ CHECK(n >= 1 && out[0] == 1, "periodic: n=%d first %d", n, out[0]);
+ for (int i = 0; i < n; i++) CHECK(out[i] == (i % 2 ? 2 : 1), "periodic token %d = %d", i, out[i]);
+ /* nothing to find: no earlier occurrence, a history too short, min_n above what matches */
+ int h5[] = {1, 2, 3, 4, 5};
+ CHECK(spec_lookup(h5, 5, 2, 4, 5, out) == 0, "no repeat, no proposal");
+ CHECK(spec_lookup(h5, 1, 1, 4, 5, out) == 0, "one token, no proposal");
+ int h6[] = {4, 1, 2, 4}; /* unigram 4 repeats, no bigram does */
+ CHECK(spec_lookup(h6, 4, 2, 4, 5, out) == 0, "min_n 2: a unigram match is not enough");
+ n = spec_lookup(h6, 4, 1, 4, 5, out);
+ CHECK(n == 3 && out[0] == 1 && out[1] == 2 && out[2] == 4, "min_n 1: the unigram's continuation, n=%d", n);
+ CHECK(spec_lookup(h1, 10, 2, 4, 0, out) == 0, "k 0");
+}
+
+static void test_gate(void) {
+ SpecGate g;
+ spec_gate_init(&g, 0);
+ double e;
+ /* the priors: MTP 0.8, 0.6, 0.5; no forward measured, rho = SPEC_RHO0 */
+ double v1 = spec_gate_value(&g, SPEC_SRC_MTP, 1, &e);
+ CHECK(fabs(e - 1.8) < 1e-12 && fabs(v1 - 1.8 / (1.0 + SPEC_RHO0)) < 1e-12, "E(1) %g value %g", e, v1);
+ double v3 = spec_gate_value(&g, SPEC_SRC_MTP, 3, &e);
+ CHECK(fabs(e - (1 + 0.8 + 0.48 + 0.24)) < 1e-12 && fabs(v3 - e / (1.0 + 3 * SPEC_RHO0)) < 1e-12, "E(3) %g value %g", e, v3);
+ CHECK(spec_gate_value(&g, SPEC_SRC_MTP, 0, NULL) == 1.0, "a plain step is worth 1");
+ /* acceptance moves toward what was judged, and only the judged positions move */
+ double a0 = g.acc[SPEC_SRC_LOOKUP][0], a2 = g.acc[SPEC_SRC_LOOKUP][2];
+ spec_gate_result(&g, SPEC_SRC_LOOKUP, 2, 1); /* draft 1 stood, draft 2 did not, 3+ unseen */
+ CHECK(g.acc[SPEC_SRC_LOOKUP][0] > a0 && g.acc[SPEC_SRC_LOOKUP][1] < 0.5 && g.acc[SPEC_SRC_LOOKUP][2] == a2,
+ "judged positions: %g %g %g", g.acc[SPEC_SRC_LOOKUP][0], g.acc[SPEC_SRC_LOOKUP][1], g.acc[SPEC_SRC_LOOKUP][2]);
+ CHECK(g.prop[SPEC_SRC_LOOKUP][0] == 1 && g.hit[SPEC_SRC_LOOKUP][0] == 1 && g.prop[SPEC_SRC_LOOKUP][1] == 1 &&
+ g.hit[SPEC_SRC_LOOKUP][1] == 0 && g.prop[SPEC_SRC_LOOKUP][2] == 0, "counts");
+ /* measured costs: a row that costs nothing extra makes every depth pay */
+ spec_gate_init(&g, 0);
+ for (int i = 0; i < 4; i++) { spec_gate_forward(&g, 1, 0.100); spec_gate_forward(&g, 4, 0.100); }
+ CHECK(fabs(spec_gate_rows_cost(&g, 4) - 1.0) < 1e-9 && g.rho < 0.1 + 1e-9, "free rows: cost %g rho %g",
+ spec_gate_rows_cost(&g, 4), g.rho);
+ double v;
+ CHECK(spec_gate_pick(&g, SPEC_SRC_MTP, 3, &v) == 3 && v > 2.0, "free rows: the deepest verify (value %g)", v);
+ /* rows that cost a full forward each, with 50% acceptance, never pay */
+ spec_gate_init(&g, 0);
+ for (int i = 0; i < 8; i++) {
+ spec_gate_forward(&g, 1, 0.1);
+ for (int r = 2; r <= 6; r++) spec_gate_forward(&g, r, 0.1 * r);
+ spec_gate_result(&g, SPEC_SRC_LOOKUP, 1, 0);
+ spec_gate_result(&g, SPEC_SRC_LOOKUP, 2, 1);
+ }
+ int picked = 0, probes = 0;
+ for (int i = 0; i < SPEC_PROBE * 3; i++) {
+ int k = spec_gate_pick(&g, SPEC_SRC_LOOKUP, 5, &v);
+ if (k) { picked++; CHECK(k == 1, "a probe drafts one deeper than the best (0), got %d", k); }
+ spec_gate_forward(&g, 1, 0.1); /* the plain steps keep T(1) fresh */
+ }
+ probes = (int)g.probes[SPEC_SRC_LOOKUP];
+ CHECK(picked == 3 && probes == 3, "costly rows: only the probes draft (%d picked, %d probes)", picked, probes);
+ /* a deeper verify whose cost is barely measured is probed every SPEC_PROBE_NEW picks */
+ spec_gate_init(&g, 0);
+ for (int i = 0; i < 8; i++) { spec_gate_forward(&g, 1, 0.1); spec_gate_forward(&g, 2, 0.13); spec_gate_result(&g, SPEC_SRC_MTP, 2, 1); }
+ int deeper = 0;
+ for (int i = 0; i < 16; i++) {
+ int k = spec_gate_pick(&g, SPEC_SRC_MTP, 3, NULL);
+ if (k == 2) deeper++;
+ CHECK(k == 1 || k == 2, "best 1 or its probe, got %d", k);
+ }
+ CHECK(deeper == 16 / SPEC_PROBE_NEW, "an unmeasured depth probed every %d picks: %d probes in 16", SPEC_PROBE_NEW, deeper);
+ /* T(1) is kept measured: SPEC_REFRESH forwards without a plain step force one */
+ spec_gate_init(&g, 0);
+ spec_gate_forward(&g, 1, 0.1);
+ for (int i = 0; i < SPEC_REFRESH; i++) spec_gate_forward(&g, 3, 0.11);
+ CHECK(spec_gate_pick(&g, SPEC_SRC_MTP, 3, NULL) == 0, "a plain step after %d verifies", SPEC_REFRESH);
+ spec_gate_forward(&g, 1, 0.1);
+ CHECK(spec_gate_pick(&g, SPEC_SRC_MTP, 3, NULL) > 0, "drafting again once T(1) is fresh");
+ /* the extra cost of a source's deeper drafts counts */
+ spec_gate_init(&g, 0);
+ spec_gate_forward(&g, 1, 0.1); spec_gate_forward(&g, 3, 0.1);
+ double before = spec_gate_value(&g, SPEC_SRC_MTP, 2, NULL);
+ spec_gate_draft_cost(&g, SPEC_SRC_MTP, 0.05);
+ double after = spec_gate_value(&g, SPEC_SRC_MTP, 2, NULL);
+ CHECK(after < before && fabs(before / after - 1.5) < 1e-9, "draft cost: %g -> %g", before, after);
+ /* off: every proposal in full */
+ spec_gate_init(&g, 1);
+ for (int i = 0; i < 4; i++) spec_gate_forward(&g, 6, 10.0);
+ CHECK(spec_gate_pick(&g, SPEC_SRC_LOOKUP, 5, NULL) == 5, "gate off drafts the whole proposal");
+}
+
+int main(void) {
+ test_lookup();
+ test_gate();
+ if (fails) { printf("test_spec_draft: %d failure(s)\n", fails); return 1; }
+ printf("test_spec_draft: prompt lookup and the gate behave as spec_draft.h says\n");
+ return 0;
+}
diff --git a/c/tests/test_vk_chain.c b/c/tests/test_vk_chain.c
index 53142cea..6b6e4f0b 100644
--- a/c/tests/test_vk_chain.c
+++ b/c/tests/test_vk_chain.c
@@ -48,6 +48,11 @@
* window and compressed rows (V4's bf16 roundings too), interleaved RoPE
* both ways, the compressor's ring (V4's overlapping form), the indexer's
* scores, candidate blocks and top-k slot for slot, the engram gate
+ * kvs the KV cache split between the device and the host (vk_kvsplit.h): decode
+ * and prefill steps over a device holding a few blocks, the device's part, the
+ * host's and their merge against the full attention, GQA (a gate, a sink,
+ * position-major rows, a window, lists with pinned blocks) and MLA (NoPE, a
+ * nonzero start, lists, latents up to 1024)
* dsv4 rounding DeepSeek V4's roundings (deepseek_v4.c): bf16, E4M3 and E2M1 per block,
* the Hadamard transform, bit for bit; its SwiGLU within one bf16 step
*
@@ -59,6 +64,7 @@
#include <float.h>
#include "../backend_vulkan.h"
#include "../vk_chain.h"
+#include "../vk_kvsplit.h" /* the KV cache split between the device and the host */
#include "../delta_attention.h" /* the KDA step the KDA ops follow */
#include "../hyper_connections.h" /* the mHC arithmetic */
#include "../sparse_index.h" /* GLM-5.3's k-pooled indexer */
@@ -192,9 +198,23 @@ static void test_rope(void) {
vkc_free(xb); vkc_free(cb); free(x); free(ref); free(cs); free(y);
}
+/* An automatic chunk must not force 64 rows when even one row exhausts the
+ * measured budget. Explicit overrides and small caps remain exact. */
+static void test_chunk_rows(void) {
+ setenv("COLI_VK_CHAIN_ROWS", "auto", 1);
+ setenv("COLI_VK_CHAIN_ROWS_MAX", "8192", 1);
+ CHECK(vkc_chunk_rows("test-tight-memory", SIZE_MAX / 4) == 1,
+ "automatic chunk exceeded a one-row memory budget");
+ setenv("COLI_VK_CHAIN_ROWS_MAX", "7", 1);
+ CHECK(vkc_chunk_rows("test-small-cap", 1) == 7, "automatic chunk exceeded its cap");
+ setenv("COLI_VK_CHAIN_ROWS", "3", 1);
+ CHECK(vkc_chunk_rows("test-explicit", SIZE_MAX / 4) == 3, "explicit chunk ignored");
+ unsetenv("COLI_VK_CHAIN_ROWS"); unsetenv("COLI_VK_CHAIN_ROWS_MAX");
+}
+
/* ---- attention ---------------------------------------------------------------------- */
-static void test_attn(int S, int pos_base, int hd, int use_list) {
- int H = 4, KVH = 2, cap = 300, gsz = hd, qseg = hd + gsz, koff = 64;
+static void test_attn_hk(int S, int pos_base, int H, int KVH, int hd, int use_list) {
+ int cap = 300, gsz = hd, qseg = hd + gsz, koff = 64;
int T = pos_base + S;
size_t kn = (size_t)koff + (size_t)KVH * cap * hd;
float *q = fvec((size_t)S * H * qseg, 1.f), *kc = fvec(kn, 1.f), *vc = fvec(kn, 1.f);
@@ -234,10 +254,12 @@ static void test_attn(int S, int pos_base, int hd, int use_list) {
vkc_begin(); int ok = vkc_attn(qb, kb, vb, ob, qb, lb, &p); vkc_submit(1);
float *o = vkc_ptr(ob);
double e = relerr(o, ref, (size_t)S * H * hd, 1e-3);
- CHECK(ok && e < 2e-5, "attn S %d pos %d hd %d list %d: err %.2e", S, pos_base, hd, use_list, e);
+ CHECK(ok && e < 2e-5, "attn S %d pos %d H %d/%d hd %d list %d (block from %d): err %.2e", S, pos_base, H, KVH, hd,
+ use_list, vkc_attn_block_rows(), e);
vkc_free(qb); vkc_free(kb); vkc_free(vb); vkc_free(ob); vkc_free(lb);
free(q); free(kc); free(vc); free(sel); free(ref);
}
+static void test_attn(int S, int pos_base, int hd, int use_list) { test_attn_hk(S, pos_base, 4, 2, hd, use_list); }
/* MiMo's attention (vkc_attn_w): row s at pos = pos_base + s sees the positions
* max(0, pos - win + 1)..pos (win 0: from 0); position t sits in row t % ring of the
@@ -280,6 +302,16 @@ static void test_attn_win(int S, int pos_base, int H, int KVH, int hd, int vd, i
double e = relerr(o, ref, (size_t)S * H * vd, 1e-3);
CHECK(ok && e < 2e-5, "attn window S %d pos %d hd %d vd %d win %d ring %d sink %d pm %d: err %.2e",
S, pos_base, hd, vd, win, ring, sink, kv_pm, e);
+ if (S >= 16 && !getenv("COLI_VK_ATTN_SLICE")) {
+ size_t bytes = (size_t)S * H * vd * sizeof(float);
+ float *whole = malloc(bytes); memcpy(whole, o, bytes);
+ setenv("COLI_VK_ATTN_SLICE", "1", 1);
+ vkc_begin(); ok = vkc_attn_w(qb, kb, vb, ob, NULL, NULL, sink ? sb : NULL, &p);
+ ok = vkc_submit(1) && ok;
+ CHECK(ok && memcmp(whole, vkc_ptr(ob), bytes) == 0,
+ "attention slices changed bytes S %d H %d/%d win %d sink %d", S, H, KVH, win, sink);
+ unsetenv("COLI_VK_ATTN_SLICE"); free(whole);
+ }
vkc_free(qb); vkc_free(kb); vkc_free(vb); vkc_free(sb); vkc_free(ob);
free(q); free(kc); free(vc); free(snk); free(ref);
}
@@ -942,6 +974,37 @@ static void test_mla(MlaCase c) {
"ok %d err %.2e latent %.2e rope %.2e", H, Q, R, V, K, c.q_lora, S, pb, c.kv_start, c.style, c.split, c.list, c.gate,
c.fq, c.fkv, c.fo, ok, e, el, er);
if (getenv("VKC_TEST_VERBOSE")) printf("mla case H %d K %d S %d: err %.2e latent %.2e rope %.2e\n", H, K, S, e, el, er);
+ /* the same layer over the split cache (vk_kvsplit.h): a device of a few blocks of 4,
+ * the positions before them in the host's rows (lat, rope as given) */
+ if (vkc_kvs_ready()) {
+ int rows = (c.list ? 16 : 8) * S + 2; /* blocks of 1, a step of S rows an eighth of the window */
+ char rv[32]; snprintf(rv, sizeof rv, "%d", rows); setenv("COLI_VK_KV_DEVICE_ROWS", rv, 1); setenv("COLI_VK_KV_BLOCK", "1", 1);
+ VkcKvSplit ks; memset(&ks, 0, sizeof ks);
+ int plan = vkc_kv_plan(&ks, "test", 1, (size_t)KR * 4, cap, S, c.list, 0);
+ unsetenv("COLI_VK_KV_DEVICE_ROWS"); unsetenv("COLI_VK_KV_BLOCK");
+ if (plan == 2) {
+ VkcMlaCache dc = {vkc_buf((size_t)ks.rows * K * 4, VKC_DEV), R ? vkc_buf((size_t)ks.rows * R * 4, VKC_DEV) : NULL, ks.rows};
+ VkcMlaCache tc = {vkc_buf((size_t)S * K * 4, VKC_DEV), R ? vkc_buf((size_t)S * R * 4, VKC_DEV) : NULL, S};
+ VkcKvPart pt[2] = {{1, K, lat, 0, dc.lat, 0}, {1, R ? R : 1, rope, 0, dc.rope, 0}};
+ VkcBuf *ob2 = vkc_buf((size_t)S * D * 4, VKC_DEV), *dn2 = vkc_buf((size_t)S * KR * 4, VKC_DOWN);
+ vkc_kv_place(&ks, 0, pb, S);
+ unsigned long long sp = ks.splits;
+ int ok2 = vkc_kv_parts(&ks, (size_t)S * H * (K + 2)) && vkc_begin() && vkc_kv_push(&ks, 0, pt, R ? 2 : 1, pb) &&
+ vkc_write(sb, 0, sel, (size_t)S * selrow * 4) &&
+ vkc_kv_mla_qkv(&ks, 0, &m, &scr, xb, 16, S, pb, csb, &dc, &tc, dn2, 0) &&
+ vkc_kv_mla_attn(&ks, 0, &m, &scr, S, pb, c.kv_start, &dc, c.list ? sb : NULL, 0, selrow, c.gate ? gb : NULL, 0,
+ ob2, 0, lat, rope) && vkc_submit(1);
+ float *y2 = down(ob2, 0, (size_t)S * D);
+ double e2 = relerr(y2, ref, (size_t)S * D, 1e-3), el2 = 0;
+ float *nl2 = (float *)vkc_ptr(dn2);
+ for (size_t i = 0; i < (size_t)S * KR; i++) { double d = fabs(nl2[i] - nl[i]); if (d > el2) el2 = d; }
+ CHECK(ok2 && e2 < 2e-5 && el2 == 0 && (pb <= ks.rows || ks.splits > sp),
+ "mla over the split cache H %d K %d S %d pos %d list %d: ok %d err %.2e, new rows %.2e, %llu host parts",
+ H, K, S, pb, c.list, ok2, e2, el2, ks.splits - sp);
+ vkc_free(dc.lat); vkc_free(dc.rope); vkc_free(tc.lat); vkc_free(tc.rope); vkc_free(ob2); vkc_free(dn2); free(y2);
+ } else CHECK(cap <= rows, "mla over the split cache: no split for cap %d rows %d", cap, rows);
+ vkc_kv_free(&ks);
+ }
vkc_mla_scratch_free(&scr);
vkc_free(cc.lat); vkc_free(cc.rope); vkc_free(xb); vkc_free(csb); vkc_free(gb); vkc_free(ob); vkc_free(dn); vkc_free(sb);
vkc_free(m.prm);
@@ -1093,14 +1156,18 @@ static void test_mhc(int H, int D, int iters) {
vkc_mhc(VKC_MHC_COLLAPSE, xb, NULL, hb, NULL, cb, &sp) && vkc_mhc(VKC_MHC_POST, xb, bb, hb, NULL, ob, &po) &&
vkc_mhc(VKC_MHC_MEAN, xb, NULL, NULL, NULL, eb, &sp) && vkc_submit(1);
float *hp = down(hb, 0, (size_t)S * hrow), *col = down(cb, 0, (size_t)S * D), *out = down(ob, 0, (size_t)S * HD), *mean = down(eb, 0, (size_t)S * D);
- /* pre, post and comb from coli_hc_split_sinkhorn on the CPU's own mixes */
+ /* Isolate the nonlinear split from the preceding matrix product: use the
+ * device's raw mixes here. End-to-end pre/post/collapse/write-back references
+ * above still include the CPU matrix product and retain their own bounds. */
float *rp = malloc((size_t)S * hrow * 4), *mixs = malloc((size_t)nm * 4);
+ float *device_mix = down(mb, 0, (size_t)S * nm);
for (int s = 0; s < S; s++) {
float ms = 0; for (int i = 0; i < HD; i++) ms += x[s * HD + i] * x[s * HD + i];
float ir = 1.f / sqrtf(ms / HD + eps);
- for (int r = 0; r < nm; r++) { float a = 0; for (int i = 0; i < HD; i++) a += fn[(size_t)r * HD + i] * x[s * HD + i]; mixs[r] = a * ir; }
+ for (int r = 0; r < nm; r++) mixs[r] = device_mix[s * nm + r] * ir;
coli_hc_split_sinkhorn(rp + s * hrow, rp + s * hrow + H, rp + s * hrow + 2 * H, mixs, prm, prm + 3, H, iters, hce);
}
+ free(device_mix);
double epre = 0;
for (int s = 0; s < S; s++) for (int i = 0; i < hrow; i++) { double d = fabs(hp[s * hrow + i] - rp[s * hrow + i]); if (d > epre) epre = d; }
free(rp); free(mixs);
@@ -1287,6 +1354,8 @@ static void test_ares(int S, int D, int nb) {
static void test_situ(float b1, float b2) {
int n = 777;
float *ga = fvec(n, 3.f * b1), *ua = fvec(n, 3.f * b2), *rr = malloc((size_t)n * 4);
+ const float tiny[] = {0.f, 0x1p-40f, -0x1p-40f, 0x1p-20f, -0x1p-20f, 0x1p-13f, -0x1p-13f};
+ for (int i = 0; i < 7; i++) { ga[i] = b1 * tiny[i]; ua[i] = b2 * tiny[i]; }
for (int i = 0; i < n; i++) rr[i] = b1 * tanhf(ga[i] / b1) * sigm(ga[i]) * b2 * tanhf(ua[i] / b2);
VkcBuf *gb = up(ga, n), *ub = up(ua, n), *yb = vkc_buf((size_t)n * 4 + 64, VKC_DEV);
VkcSitu sp = {n, 0, 0, 16, b1, b2};
@@ -1294,6 +1363,9 @@ static void test_situ(float b1, float b2) {
float *y = down(yb, 16, n);
double e = relerr(y, rr, n, 1e-3);
CHECK(ok && e < 1e-6 && !bad(y, n), "situ b1 %g b2 %g: ok %d err %.2e", b1, b2, ok, e);
+ for (int i = 0; i < 7; i++)
+ CHECK(fabs((double)y[i] - rr[i]) <= 2e-6 * fabs(rr[i]),
+ "situ tiny b1 %g b2 %g at %d: got %.9g ref %.9g", b1, b2, i, y[i], rr[i]);
vkc_free(gb); vkc_free(ub); vkc_free(yb); free(ga); free(ua); free(rr); free(y);
}
@@ -1758,6 +1830,481 @@ static void test_frames(void) {
vkc_free(x); vkc_free(y); free(a); free(acc); free(got);
}
+/* ---- the KV cache split between the device and the host (chain_kvs, vk_kvsplit.h) --
+ * A host cache of every position, a device holding `rows` of them (vkc_kv_plan under
+ * COLI_VK_KV_DEVICE_ROWS), decode or prefill steps one after another as an engine runs
+ * them: the window placed, the rows below the step uploaded, the step's own rows stored
+ * from a scratch, then the device's part (F2) and the host's part (from the queries F1
+ * brought down) merged in F3, or one frame when the device holds every visible position.
+ * Each step's output against a double-precision reference of the full attention. */
+/* row s's positions: its list (count >= 0) or the causal range */
+static int kvs_positions(const int *sel, int sel_row, int s, int lo, int pos, int *out) {
+ if (sel && sel[(size_t)s * sel_row] >= 0) {
+ int n = 0;
+ for (int j = 0; j < sel[(size_t)s * sel_row]; j++) { int t = sel[(size_t)s * sel_row + 1 + j]; if (t >= 0) out[n++] = t; }
+ return n;
+ }
+ int n = 0;
+ for (int t = lo; t <= pos; t++) out[n++] = t;
+ return n;
+}
+
+/* A query value fixed by its position and index (not by the order the steps draw them):
+ * the same row gets the same query however the positions are cut into steps. */
+static float hval(unsigned a, unsigned b) {
+ unsigned x = a * 2654435761u ^ (b + 0x9e3779b9u) * 2246822519u;
+ x ^= x >> 15; x *= 2654435761u; x ^= x >> 13;
+ return (float)((int)(x % 2001u) - 1000) / 1000.0f;
+}
+/* each step's output rows, kept by position (kvs_keep_row floats a position) */
+static float *kvs_keep; static size_t kvs_keep_row;
+static int kvs_start; /* seeded canonical prefix; zero for the sequential tests */
+static float *hvec(size_t n, unsigned seed, float scale) { float *v = malloc(n * sizeof *v); for (size_t i = 0; i < n; i++) v[i] = hval(seed, (unsigned)i) * scale; return v; }
+/* The host's cache as an engine's holds it: a step's rows arrive after the step (the
+ * rows not there yet are NaN, which a host part reading them would carry into the
+ * result). copy rows [t0, t1) of a [seg][T][len] or [T][len] array. */
+static float *kvs_host(size_t n) { float *v = malloc(n * sizeof *v); for (size_t i = 0; i < n; i++) v[i] = NAN; return v; }
+static void kvs_arrive(float *dst, const float *src, int nseg, size_t seg, int len, int t0, int t1) {
+ for (int h = 0; h < nseg; h++) memcpy(dst + h * seg + (size_t)t0 * len, src + h * seg + (size_t)t0 * len, (size_t)(t1 - t0) * len * 4);
+}
+static void kvs_keep_rows(const float *o, int pb, int S) { if (kvs_keep) memcpy(kvs_keep + (size_t)pb * kvs_keep_row, o, (size_t)S * kvs_keep_row * 4); }
+
+/* GQA: steps of `step` rows from position 0 to T, the device holding `rows` positions. */
+static void test_kvs_gqa(int T, int step, int H, int KVH, int hd, int vd, int B, int rows, int win, int sink, int kv_pm,
+ int gated, int lists, int selects) {
+ char rv[32]; snprintf(rv, sizeof rv, "%d", rows); setenv("COLI_VK_KV_DEVICE_ROWS", rv, 1);
+ char bv[32]; snprintf(bv, sizeof bv, "%d", B); setenv("COLI_VK_KV_BLOCK", bv, 1);
+ VkcKvSplit ks; memset(&ks, 0, sizeof ks);
+ int r = vkc_kv_plan(&ks, "test", 1, (size_t)KVH * (hd + vd) * 4, T, step, selects, 0);
+ unsetenv("COLI_VK_KV_DEVICE_ROWS"); unsetenv("COLI_VK_KV_BLOCK");
+ CHECK(r == 2 && ks.on, "kvs gqa: the plan did not split (%d)", r);
+ if (r != 2) { vkc_kv_free(&ks); return; }
+ if (step > ks.chunk) step = ks.chunk; /* as the engines clamp a step to the split's */
+ float *Kh = hvec((size_t)T * KVH * hd, 1u << 30, 1.f), *Vh = hvec((size_t)T * KVH * vd, 1u << 29, 1.f), *snk = hvec(H + 3, 1u << 28, 2.f);
+ float *Kx = kvs_host((size_t)T * KVH * hd), *Vx = kvs_host((size_t)T * KVH * vd);
+ int qrow = H * 2 * hd, sel_row = lists ? 1 + T : 0;
+ float scale = 1.f / sqrtf((float)hd);
+ VkcBuf *kc = vkc_buf((size_t)KVH * ks.rows * hd * 4, VKC_DEV), *vc = vkc_buf((size_t)KVH * ks.rows * vd * 4, VKC_DEV);
+ VkcBuf *qb = vkc_buf((size_t)step * qrow * 4, VKC_DEV), *nk = vkc_buf((size_t)step * KVH * hd * 4, VKC_DEV);
+ VkcBuf *nv = vkc_buf((size_t)step * KVH * vd * 4, VKC_DEV), *ob = vkc_buf((size_t)step * H * vd * 4, VKC_DEV);
+ VkcBuf *sb = up(snk, H + 3), *lb = vkc_buf((size_t)step * (sel_row ? sel_row : 1) * 4, VKC_DEV);
+ int ok = vkc_kv_parts(&ks, (size_t)step * H * (vd + 2));
+ double worst = 0; int splits = 0, fins = 0;
+ int *posl = malloc((size_t)(T + 1) * sizeof *posl), *sel = calloc((size_t)step * (sel_row ? sel_row : 1), sizeof *sel);
+ float *ref = malloc((size_t)step * H * vd * sizeof *ref), *nkh = malloc((size_t)step * KVH * hd * 4), *nvh = malloc((size_t)step * KVH * vd * 4);
+ if (kvs_start) {
+ kvs_arrive(Kx, Kh, kv_pm ? 1 : KVH, (size_t)T * hd, kv_pm ? KVH * hd : hd, 0, kvs_start);
+ kvs_arrive(Vx, Vh, kv_pm ? 1 : KVH, (size_t)T * vd, kv_pm ? KVH * vd : vd, 0, kvs_start);
+ }
+ for (int pb = kvs_start; pb < T && ok; pb += step) {
+ int S = T - pb < step ? T - pb : step;
+ float *q = malloc((size_t)S * qrow * 4);
+ for (int s = 0; s < S; s++) for (int i = 0; i < qrow; i++) q[(size_t)s * qrow + i] = hval(pb + s, i);
+ for (int s = 0; s < S; s++) for (int h = 0; h < KVH; h++) { /* the step's rows, as its projections leave them */
+ const float *ks_ = kv_pm ? Kh + ((size_t)(pb + s) * KVH + h) * hd : Kh + ((size_t)h * T + pb + s) * hd;
+ const float *vs_ = kv_pm ? Vh + ((size_t)(pb + s) * KVH + h) * vd : Vh + ((size_t)h * T + pb + s) * vd;
+ memcpy(nkh + ((size_t)s * KVH + h) * hd, ks_, hd * 4); memcpy(nvh + ((size_t)s * KVH + h) * vd, vs_, vd * 4);
+ }
+ if (lists) for (int s = 0; s < S; s++) { /* odd rows causal; even rows every third position and a few far ones */
+ int pos = pb + s, n = 0, *rw = sel + (size_t)s * sel_row;
+ if (pos % 2) { rw[0] = -1; continue; }
+ for (int t = 0; t <= pos; t++) if (t % 3 == 0 || t == pos || t < 2 * B) rw[1 + n++] = t;
+ rw[1 + n++] = -1; /* a skipped entry */
+ rw[0] = n;
+ }
+ for (int s = 0; s < S; s++) for (int h = 0; h < H; h++) {
+ int kvh = h / (H / KVH), pos = pb + s, lo = win ? (pos - win + 1 > 0 ? pos - win + 1 : 0) : 0;
+ int n = kvs_positions(lists ? sel : NULL, sel_row, s, lo, pos, posl);
+ double mx = sink ? snk[3 + h] : -1e300, sum = 0, *sc = malloc((n + 1) * sizeof *sc);
+ for (int j = 0; j < n; j++) {
+ const float *kr = kv_pm ? Kh + ((size_t)posl[j] * KVH + kvh) * hd : Kh + ((size_t)kvh * T + posl[j]) * hd;
+ double a = 0; for (int d = 0; d < hd; d++) a += (double)q[(size_t)s * qrow + h * 2 * hd + d] * kr[d];
+ sc[j] = a * scale; if (sc[j] > mx) mx = sc[j];
+ }
+ if (sink) sum = exp(snk[3 + h] - mx);
+ for (int j = 0; j < n; j++) { sc[j] = exp(sc[j] - mx); sum += sc[j]; }
+ for (int d = 0; d < vd; d++) {
+ double a = 0;
+ for (int j = 0; j < n; j++) {
+ const float *vr = kv_pm ? Vh + ((size_t)posl[j] * KVH + kvh) * vd : Vh + ((size_t)kvh * T + posl[j]) * vd;
+ a += sc[j] * vr[d];
+ }
+ double v = sum > 0 ? a / sum : 0;
+ if (gated) v *= sigm(q[(size_t)s * qrow + h * 2 * hd + hd + d]);
+ ref[((size_t)s * H + h) * vd + d] = (float)v;
+ }
+ free(sc);
+ }
+ /* the engine's order: the window placed, the rows below uploaded, the new rows stored */
+ VkcKvPart pk = {kv_pm ? 1 : KVH, kv_pm ? KVH * hd : hd, Kx, (size_t)T * hd, kc, 0};
+ VkcKvPart pv = {kv_pm ? 1 : KVH, kv_pm ? KVH * vd : vd, Vx, (size_t)T * vd, vc, 0};
+ VkcKvPart parts[2] = {pk, pv};
+ vkc_kv_place(&ks, 0, pb, S);
+ ok = vkc_begin() && vkc_kv_push(&ks, 0, parts, 2, pb) &&
+ vkc_write(qb, 0, q, (size_t)S * qrow * 4) && vkc_write(nk, 0, nkh, (size_t)S * KVH * hd * 4) &&
+ vkc_write(nv, 0, nvh, (size_t)S * KVH * vd * 4) && (!lists || vkc_write(lb, 0, sel, (size_t)S * sel_row * 4)) &&
+ vkc_kv_store(&ks, 0, &pk, nk, 0, (size_t)KVH * hd, kv_pm ? 0 : (size_t)hd, pb, S) &&
+ vkc_kv_store(&ks, 0, &pv, nv, 0, (size_t)KVH * vd, kv_pm ? 0 : (size_t)vd, pb, S);
+ unsigned long long sp = ks.splits;
+ VkcKvGqa ga = {&ks, 0, qb, kc, vc, qb, lists ? lb : NULL, sb, ob, S, H, KVH, hd, vd, pb, win, kv_pm, sink, 3,
+ qrow, 2 * hd, gated, hd, qrow, 2 * hd, H * vd, 0, sel_row, scale,
+ Kx, Vx, kv_pm ? (size_t)hd : (size_t)T * hd, kv_pm ? (size_t)KVH * hd : (size_t)hd,
+ kv_pm ? (size_t)vd : (size_t)T * vd, kv_pm ? (size_t)KVH * vd : (size_t)vd};
+ ok = ok && vkc_kv_gqa(&ga);
+ if (ks.splits > sp) splits++; else fins++;
+ ok = ok && vkc_submit(1);
+ vkc_kv_done(&ks, 0, pb + S);
+ if (kv_pm) { kvs_arrive(Kx, Kh, 1, 0, KVH * hd, pb, pb + S); kvs_arrive(Vx, Vh, 1, 0, KVH * vd, pb, pb + S); }
+ else { kvs_arrive(Kx, Kh, KVH, (size_t)T * hd, hd, pb, pb + S); kvs_arrive(Vx, Vh, KVH, (size_t)T * vd, vd, pb, pb + S); }
+ float *o = down(ob, 0, (size_t)S * H * vd);
+ kvs_keep_rows(o, pb, S);
+ double e = relerr(o, ref, (size_t)S * H * vd, 1e-3);
+ if (bad(o, (size_t)S * H * vd)) e = 1e9; /* a NaN: the host part read a row not there yet */
+ if (e > worst) worst = e;
+ free(o); free(q);
+ }
+ CHECK(ok && worst < 2e-5 && splits > 0, "kvs gqa T %d step %d H %d/%d hd %d vd %d B %d rows %d win %d sink %d pm %d gate %d lists %d: "
+ "err %.2e (%d split steps, %d whole)", T, step, H, KVH, hd, vd, B, rows, win, sink, kv_pm, gated, lists, worst, splits, fins);
+ printf(" kvs gqa T %d step %d rows %d lists %d: rel err %.2e, %d split steps, %d whole, %llu host positions, %llu pinned\n",
+ T, step, rows, lists, worst, splits, fins, ks.host_pos, ks.pins);
+ CHECK(!ks.np || !selects || ks.pins > 0, "kvs gqa lists: no block was pinned");
+ vkc_free(kc); vkc_free(vc); vkc_free(qb); vkc_free(nk); vkc_free(nv); vkc_free(ob); vkc_free(sb); vkc_free(lb);
+ vkc_kv_free(&ks);
+ free(Kh); free(Vh); free(Kx); free(Vx); free(snk); free(posl); free(sel); free(ref); free(nkh); free(nvh);
+}
+
+/* MLA: the latent and the rope key per position; lists with skipped entries; kv_start. */
+static void test_kvs_mla(int T, int step, int H, int K, int R, int B, int rows, int kv_start, int lists) {
+ char rv[32]; snprintf(rv, sizeof rv, "%d", rows); setenv("COLI_VK_KV_DEVICE_ROWS", rv, 1);
+ char bv[32]; snprintf(bv, sizeof bv, "%d", B); setenv("COLI_VK_KV_BLOCK", bv, 1);
+ VkcKvSplit ks; memset(&ks, 0, sizeof ks);
+ int r = vkc_kv_plan(&ks, "test", 2, (size_t)(K + R) * 4, T, step, lists, 0);
+ unsetenv("COLI_VK_KV_DEVICE_ROWS"); unsetenv("COLI_VK_KV_BLOCK");
+ CHECK(r == 2, "kvs mla: the plan did not split (%d)", r);
+ if (r != 2) { vkc_kv_free(&ks); return; }
+ if (step > ks.chunk) step = ks.chunk;
+ float *Lh = hvec((size_t)T * K, 1u << 30, 1.f), *Rh = hvec((size_t)T * (R ? R : 1), 1u << 29, 1.f);
+ float *Lx = kvs_host((size_t)T * K), *Rx = kvs_host((size_t)T * (R ? R : 1));
+ int sel_row = lists ? 1 + T : 0, li = 1;
+ float scale = 0.7f / sqrtf((float)(K + R));
+ VkcBuf *lat = vkc_buf((size_t)ks.rows * K * 4, VKC_DEV), *rope = vkc_buf((size_t)ks.rows * (R ? R : 1) * 4, VKC_DEV);
+ VkcBuf *qab = vkc_buf((size_t)step * H * (K + R) * 4, VKC_DEV), *nl = vkc_buf((size_t)step * (K + R) * 4, VKC_DEV);
+ VkcBuf *ob = vkc_buf((size_t)step * H * K * 4, VKC_DEV), *lb = vkc_buf((size_t)step * (sel_row ? sel_row : 1) * 4, VKC_DEV);
+ int ok = vkc_kv_parts(&ks, (size_t)step * H * (K + 2));
+ double worst = 0; int splits = 0;
+ int *posl = malloc((size_t)(T + 1) * sizeof *posl), *sel = calloc((size_t)step * (sel_row ? sel_row : 1), sizeof *sel);
+ float *ref = malloc((size_t)step * H * K * sizeof *ref), *nh = malloc((size_t)step * (K + R) * 4);
+ for (int pb = 0; pb < T && ok; pb += step) {
+ int S = T - pb < step ? T - pb : step;
+ float *q = malloc((size_t)S * H * (K + R) * 4); /* qa [S][H][K], then qr [S][H][R] */
+ for (int s = 0; s < S; s++) {
+ for (int i = 0; i < H * K; i++) q[(size_t)s * H * K + i] = hval(pb + s, i);
+ for (int i = 0; i < H * R; i++) q[(size_t)S * H * K + (size_t)s * H * R + i] = hval(pb + s, 100000 + i);
+ }
+ for (int s = 0; s < S; s++) {
+ memcpy(nh + (size_t)s * K, Lh + (size_t)(pb + s) * K, K * 4);
+ if (R) memcpy(nh + (size_t)S * K + (size_t)s * R, Rh + (size_t)(pb + s) * R, R * 4);
+ }
+ if (lists) for (int s = 0; s < S; s++) {
+ int pos = pb + s, n = 0, *rw = sel + (size_t)s * sel_row;
+ if (pos % 3 == 2) { rw[0] = -1; continue; }
+ for (int t = 0; t <= pos; t++) if (t % 4 == 1 || t == pos || (t >= B && t < 3 * B)) rw[1 + n++] = t;
+ rw[1 + n++] = -1;
+ rw[0] = n;
+ }
+ for (int s = 0; s < S; s++) for (int h = 0; h < H; h++) {
+ int n = kvs_positions(lists ? sel : NULL, sel_row, s, kv_start, pb + s, posl);
+ double mx = -1e300, sum = 0, *sc = malloc((n + 1) * sizeof *sc);
+ for (int j = 0; j < n; j++) {
+ double a = 0;
+ for (int d = 0; d < K; d++) a += (double)q[((size_t)s * H + h) * K + d] * Lh[(size_t)posl[j] * K + d];
+ for (int d = 0; d < R; d++) a += (double)q[(size_t)S * H * K + ((size_t)s * H + h) * R + d] * Rh[(size_t)posl[j] * R + d];
+ sc[j] = a * scale; if (sc[j] > mx) mx = sc[j];
+ }
+ for (int j = 0; j < n; j++) { sc[j] = exp(sc[j] - mx); sum += sc[j]; }
+ for (int d = 0; d < K; d++) {
+ double a = 0;
+ for (int j = 0; j < n; j++) a += sc[j] * Lh[(size_t)posl[j] * K + d];
+ ref[((size_t)s * H + h) * K + d] = (float)(sum > 0 ? a / sum : 0);
+ }
+ free(sc);
+ }
+ VkcKvPart pl = {1, K, Lx, 0, lat, 0}, pr = {1, R ? R : 1, Rx, 0, rope, 0};
+ VkcKvPart parts[2] = {pl, pr};
+ vkc_kv_place(&ks, li, pb, S);
+ ok = vkc_begin() && vkc_kv_push(&ks, li, parts, R ? 2 : 1, pb) && vkc_write(qab, 0, q, (size_t)S * H * (K + R) * 4) &&
+ vkc_write(nl, 0, nh, (size_t)S * (K + R) * 4) && (!lists || vkc_write(lb, 0, sel, (size_t)S * sel_row * 4)) &&
+ vkc_kv_store(&ks, li, &pl, nl, 0, (size_t)K, 0, pb, S) &&
+ (!R || vkc_kv_store(&ks, li, &pr, nl, (size_t)S * K, (size_t)R, 0, pb, S));
+ unsigned long long sp = ks.splits;
+ VkcKvMla ma = {&ks, li, qab, qab, lat, rope, lists ? lb : NULL, ob, S, H, K, R, pb, kv_start,
+ 0, H * K, K, S * H * K, H * R, R, 0, H * K, K, 0, sel_row, scale, Lx, Rx, 0};
+ ok = ok && vkc_kv_mla(&ma);
+ if (ks.splits > sp) splits++;
+ ok = ok && vkc_submit(1);
+ vkc_kv_done(&ks, li, pb + S);
+ kvs_arrive(Lx, Lh, 1, 0, K, pb, pb + S);
+ if (R) kvs_arrive(Rx, Rh, 1, 0, R, pb, pb + S);
+ float *o = down(ob, 0, (size_t)S * H * K);
+ kvs_keep_rows(o, pb, S);
+ double e = relerr(o, ref, (size_t)S * H * K, 1e-3);
+ if (bad(o, (size_t)S * H * K)) e = 1e9; /* a NaN: the host part read a row not there yet */
+ if (e > worst) worst = e;
+ free(o); free(q);
+ }
+ CHECK(ok && worst < 2e-5 && splits > 0, "kvs mla T %d step %d H %d K %d R %d B %d rows %d start %d lists %d: err %.2e (%d split steps)",
+ T, step, H, K, R, B, rows, kv_start, lists, worst, splits);
+ printf(" kvs mla T %d step %d K %d rows %d lists %d: rel err %.2e, %d split steps, %llu host positions, %llu pinned\n",
+ T, step, K, rows, lists, worst, splits, ks.host_pos, ks.pins);
+ CHECK(!lists || ks.pins > 0, "kvs mla lists: no block was pinned");
+ vkc_free(lat); vkc_free(rope); vkc_free(qab); vkc_free(nl); vkc_free(ob); vkc_free(lb);
+ vkc_kv_free(&ks);
+ free(Lh); free(Rh); free(Lx); free(Rx); free(posl); free(sel); free(ref); free(nh);
+}
+/* Inkling's global layer over the split cache: the relative-position bias for distances
+ * below ext, tau per row; steps from position 0 to T. */
+static void test_kvs_rel(int T, int step, int H, int KVH, int hd, int ext, int d_rel, int B, int rows) {
+ char rv[32]; snprintf(rv, sizeof rv, "%d", rows); setenv("COLI_VK_KV_DEVICE_ROWS", rv, 1);
+ char bv[32]; snprintf(bv, sizeof bv, "%d", B); setenv("COLI_VK_KV_BLOCK", bv, 1);
+ VkcKvSplit ks; memset(&ks, 0, sizeof ks);
+ int r = vkc_kv_plan(&ks, "test", 1, (size_t)KVH * hd * 2 * 4, T, step, 0, 0);
+ unsetenv("COLI_VK_KV_DEVICE_ROWS"); unsetenv("COLI_VK_KV_BLOCK");
+ CHECK(r == 2, "kvs rel: the plan did not split (%d)", r);
+ if (r != 2) { vkc_kv_free(&ks); return; }
+ if (step > ks.chunk) step = ks.chunk;
+ float *Kh = hvec((size_t)T * KVH * hd, 1u << 30, 1.f), *Vh = hvec((size_t)T * KVH * hd, 1u << 29, 1.f);
+ float *relp = hvec((size_t)d_rel * ext + 3, 1u << 28, 0.5f);
+ float *Kx = kvs_host((size_t)T * KVH * hd), *Vx = kvs_host((size_t)T * KVH * hd);
+ float scale = 1.f / (float)hd;
+ VkcBuf *kc = vkc_buf((size_t)KVH * ks.rows * hd * 4, VKC_DEV), *vc = vkc_buf((size_t)KVH * ks.rows * hd * 4, VKC_DEV);
+ VkcBuf *qb = vkc_buf((size_t)step * H * hd * 4, VKC_DEV), *rb = vkc_buf((size_t)step * H * (d_rel ? d_rel : 1) * 4, VKC_DEV);
+ VkcBuf *nk = vkc_buf((size_t)step * KVH * hd * 4, VKC_DEV), *nv = vkc_buf((size_t)step * KVH * hd * 4, VKC_DEV);
+ VkcBuf *ob = vkc_buf((size_t)step * H * hd * 4, VKC_DEV), *tb = vkc_buf((size_t)(step + 2) * 4, VKC_UP), *pb_ = up(relp, (size_t)d_rel * ext + 3);
+ int ok = vkc_kv_parts(&ks, (size_t)step * H * (hd + 2));
+ double worst = 0; int splits = 0;
+ float *ref = malloc((size_t)step * H * hd * 4), *nkh = malloc((size_t)step * KVH * hd * 4), *nvh = malloc((size_t)step * KVH * hd * 4);
+ float *tau = malloc((size_t)(step + 2) * 4);
+ for (int pb = 0; pb < T && ok; pb += step) {
+ int S = T - pb < step ? T - pb : step;
+ float *q = malloc((size_t)S * H * hd * 4), *rr = malloc((size_t)S * H * (d_rel ? d_rel : 1) * 4);
+ for (int s = 0; s < S; s++) {
+ for (int i = 0; i < H * hd; i++) q[(size_t)s * H * hd + i] = hval(pb + s, i);
+ for (int i = 0; i < H * (d_rel ? d_rel : 1); i++) rr[(size_t)s * H * (d_rel ? d_rel : 1) + i] = hval(pb + s, 50000 + i);
+ }
+ for (int s = 0; s < S; s++) { tau[2 + s] = 1.f + 0.3f * logf(1.f + (float)(pb + s) / 7.f); }
+ for (int s = 0; s < S; s++) for (int h = 0; h < KVH; h++) {
+ memcpy(nkh + ((size_t)s * KVH + h) * hd, Kh + ((size_t)h * T + pb + s) * hd, hd * 4);
+ memcpy(nvh + ((size_t)s * KVH + h) * hd, Vh + ((size_t)h * T + pb + s) * hd, hd * 4);
+ }
+ for (int s = 0; s < S; s++) for (int h = 0; h < H; h++) {
+ int kvh = h / (H / KVH), pos = pb + s;
+ double mx = -1e300, sum = 0, *sc = malloc((pos + 1) * sizeof *sc);
+ for (int t = 0; t <= pos; t++) {
+ double a = 0; for (int d = 0; d < hd; d++) a += (double)q[((size_t)s * H + h) * hd + d] * Kh[((size_t)kvh * T + t) * hd + d];
+ double bias = 0; int dist = pos - t;
+ if (dist < ext) for (int j = 0; j < d_rel; j++) bias += (double)rr[((size_t)s * H + h) * d_rel + j] * relp[3 + (size_t)j * ext + dist];
+ sc[t] = tau[2 + s] * (a * scale + bias); if (sc[t] > mx) mx = sc[t];
+ }
+ for (int t = 0; t <= pos; t++) { sc[t] = exp(sc[t] - mx); sum += sc[t]; }
+ for (int d = 0; d < hd; d++) {
+ double a = 0; for (int t = 0; t <= pos; t++) a += sc[t] * Vh[((size_t)kvh * T + t) * hd + d];
+ ref[((size_t)s * H + h) * hd + d] = (float)(a / sum);
+ }
+ free(sc);
+ }
+ VkcKvPart pk = {KVH, hd, Kx, (size_t)T * hd, kc, 0}, pv = {KVH, hd, Vx, (size_t)T * hd, vc, 0};
+ VkcKvPart parts[2] = {pk, pv};
+ vkc_kv_place(&ks, 0, pb, S);
+ memcpy(vkc_ptr(tb), tau, (size_t)(S + 2) * 4);
+ ok = vkc_begin() && vkc_kv_push(&ks, 0, parts, 2, pb) && vkc_write(qb, 0, q, (size_t)S * H * hd * 4) &&
+ (!d_rel || vkc_write(rb, 0, rr, (size_t)S * H * d_rel * 4)) &&
+ vkc_write(nk, 0, nkh, (size_t)S * KVH * hd * 4) && vkc_write(nv, 0, nvh, (size_t)S * KVH * hd * 4) &&
+ vkc_kv_store(&ks, 0, &pk, nk, 0, (size_t)KVH * hd, (size_t)hd, pb, S) &&
+ vkc_kv_store(&ks, 0, &pv, nv, 0, (size_t)KVH * hd, (size_t)hd, pb, S);
+ unsigned long long sp = ks.splits;
+ VkcKvRel a = {&ks, 0, qb, kc, vc, rb, tb, pb_, ob, S, H, KVH, hd, pb, ext, d_rel, H * hd, H * d_rel, 3, 2, H * hd, scale,
+ Kx, Vx, (size_t)T * hd, (size_t)hd, (size_t)T * hd, (size_t)hd, tau + 2, relp + 3, 0};
+ ok = ok && vkc_kv_rel(&a) && vkc_submit(1);
+ if (ks.splits > sp) splits++;
+ vkc_kv_done(&ks, 0, pb + S);
+ kvs_arrive(Kx, Kh, KVH, (size_t)T * hd, hd, pb, pb + S); kvs_arrive(Vx, Vh, KVH, (size_t)T * hd, hd, pb, pb + S);
+ float *o = down(ob, 0, (size_t)S * H * hd);
+ kvs_keep_rows(o, pb, S);
+ double e = relerr(o, ref, (size_t)S * H * hd, 1e-3);
+ if (bad(o, (size_t)S * H * hd)) e = 1e9; /* a NaN: the host part read a row not there yet */
+ if (e > worst) worst = e;
+ free(o); free(q); free(rr);
+ }
+ CHECK(ok && worst < 2e-5 && splits > 0, "kvs rel T %d step %d H %d/%d hd %d ext %d d_rel %d B %d rows %d: err %.2e (%d split steps)",
+ T, step, H, KVH, hd, ext, d_rel, B, rows, worst, splits);
+ printf(" kvs rel T %d step %d rows %d ext %d: rel err %.2e, %d split steps, %llu host positions\n", T, step, rows, ext, worst, splits, ks.host_pos);
+ vkc_free(kc); vkc_free(vc); vkc_free(qb); vkc_free(rb); vkc_free(nk); vkc_free(nv); vkc_free(ob); vkc_free(tb); vkc_free(pb_);
+ vkc_kv_free(&ks);
+ free(Kh); free(Vh); free(Kx); free(Vx); free(relp); free(ref); free(nkh); free(nvh); free(tau);
+}
+
+/* DeepSeek's sparse attention over a window ring and compressed rows split between the
+ * device and the host: against the scalar kernel (V4's rounding: its reference), and with
+ * every compressed row on the device, bit for bit against vkc_dsv4_attn. */
+static void test_kvs_ds(int S, int H, int hd, int nwin, int ncmp, int cnt, int v4, int B, int rows) {
+ int qrow = H * hd + 3, lrow = cnt + 2, orow = H * hd + 3;
+ float *q = fvec((size_t)S * qrow, 1.f), *win = fvec((size_t)nwin * hd, 1.f), *cmp = fvec((size_t)ncmp * hd, 1.f);
+ float *sink = fvec((size_t)H + 2, 2.f), *ref = calloc((size_t)S * orow, 4), *sc = malloc((size_t)cnt * 4);
+ int *list = malloc((size_t)S * lrow * 4);
+ float *kvall = malloc((size_t)(nwin + ncmp) * hd * 4);
+ memcpy(kvall, win, (size_t)nwin * hd * 4); memcpy(kvall + (size_t)nwin * hd, cmp, (size_t)ncmp * hd * 4);
+ for (int s = 0; s < S; s++)
+ for (int j = 0; j < lrow; j++) {
+ int r = (int)(rnd() % 6);
+ list[s * lrow + j] = r == 0 ? -1 - (int)(rnd() % 7) : r < 2 ? (int)(rnd() % (unsigned)nwin) : nwin + (int)(rnd() % (unsigned)ncmp);
+ }
+ float scale = 1.f / sqrtf((float)hd);
+ for (int s = 0; s < S; s++) for (int h = 0; h < H; h++) {
+ const float *qq = q + (size_t)s * qrow + 1 + h * hd;
+ float *o = ref + (size_t)s * orow + 2 + h * hd;
+ const int *idx = list + s * lrow + 1;
+ if (!v4) { coli_sparse_attend_scalar(o, qq, kvall, idx, cnt, hd, sink[1 + h], scale, sc); continue; }
+ float best = -1e30f;
+ for (int j = 0; j < cnt; j++) {
+ if (idx[j] < 0) { sc[j] = -INFINITY; continue; }
+ float d = 0; for (int i = 0; i < hd; i++) d += qq[i] * kvall[(size_t)idx[j] * hd + i];
+ sc[j] = d * scale; if (sc[j] > best) best = sc[j];
+ }
+ float den = expf(sink[1 + h] - best);
+ for (int i = 0; i < hd; i++) o[i] = 0;
+ for (int j = 0; j < cnt; j++) {
+ if (idx[j] < 0) continue;
+ float w = expf(sc[j] - best); den += w; w = bf16r(w);
+ for (int i = 0; i < hd; i++) o[i] += w * kvall[(size_t)idx[j] * hd + i];
+ }
+ for (int i = 0; i < hd; i++) o[i] = bf16r(o[i] / den);
+ }
+ VkcBuf *qb = up(q, (size_t)S * qrow), *wb = up(win, (size_t)nwin * hd), *cb = up(cmp, (size_t)ncmp * hd);
+ VkcBuf *sb = up(sink, (size_t)H + 2), *lb = vkc_buf((size_t)S * lrow * 4, VKC_DEV);
+ VkcBuf *ow = vkc_buf((size_t)S * orow * 4, VKC_DEV), *os = vkc_buf((size_t)S * orow * 4, VKC_DEV);
+ vkc_begin(); vkc_write(lb, 0, list, (size_t)S * lrow * 4); vkc_submit(1);
+ VkcDsAttn p = {S, H, hd, cnt, 1, lrow, nwin, 0, 0, 1, qrow, 2, orow, 1, v4, scale};
+ int ok = vkc_begin() && vkc_dsv4_attn(qb, wb, cb, lb, sb, ow, &p) && vkc_submit(1);
+ float *whole = down(ow, 0, (size_t)S * orow);
+ /* twice: a device holding `rows` compressed rows (a host part), then all of them (fin) */
+ for (int pass = 0; pass < 2; pass++) {
+ int rws = pass ? B * ((ncmp + B - 1) / B) : rows;
+ char rv[32]; snprintf(rv, sizeof rv, "%d", rws); setenv("COLI_VK_KV_DEVICE_ROWS", rv, 1);
+ char bv[32]; snprintf(bv, sizeof bv, "%d", B); setenv("COLI_VK_KV_BLOCK", bv, 1);
+ VkcKvSplit ks; memset(&ks, 0, sizeof ks);
+ int r = vkc_kv_plan(&ks, "test", 1, (size_t)hd * 4, ncmp + 2 * B, 1, 1, 0);
+ unsetenv("COLI_VK_KV_DEVICE_ROWS"); unsetenv("COLI_VK_KV_BLOCK");
+ CHECK(r == 2, "kvs ds: the plan did not split (%d)", r);
+ if (r != 2) { vkc_kv_free(&ks); continue; }
+ if (pass) { ks.nw = ks.ns; ks.np = 0; } /* every block in the window */
+ VkcBuf *dc = vkc_buf((size_t)ks.rows * hd * 4, VKC_DEV);
+ VkcKvPart pc = {1, hd, cmp, 0, dc, 0};
+ vkc_kv_place(&ks, 0, ncmp - 1, 1);
+ unsigned long long sp = ks.splits;
+ ok = vkc_kv_parts(&ks, (size_t)S * H * (hd + 2)) && vkc_begin() && vkc_kv_push(&ks, 0, &pc, 1, ncmp);
+ /* The current chunk's resident rows may not have reached canonical RAM
+ * yet. Poison them there: staging must read only nonresident rows. */
+ float *hostcmp = malloc((size_t)ncmp * hd * sizeof(float));
+ memcpy(hostcmp, cmp, (size_t)ncmp * hd * sizeof(float));
+ for (int j = 0; j < ncmp; j++) if (ks.t[0].bt[j / B] >= 0)
+ for (int d = 0; d < hd; d++) hostcmp[(size_t)j * hd + d] = NAN;
+ VkcKvDs a = {&ks, 0, qb, wb, dc, sb, lb, os, S, H, hd, cnt, 1, lrow, nwin, 0, 1, qrow, 1, v4, 2, orow, ncmp, scale, hostcmp};
+ ok = ok && vkc_kv_ds(&a) && vkc_submit(1);
+ float *got = down(os, 0, (size_t)S * orow);
+ double e = 0, ew = 0;
+ for (int s = 0; s < S; s++) {
+ double r1 = relerr(got + (size_t)s * orow + 2, ref + (size_t)s * orow + 2, (size_t)H * hd, 1e-3); if (r1 > e) e = r1;
+ double r2 = relerr(got + (size_t)s * orow + 2, whole + (size_t)s * orow + 2, (size_t)H * hd, 1e-3); if (r2 > ew) ew = r2;
+ }
+ int same = 1;
+ for (int s = 0; s < S; s++) same &= !memcmp(got + (size_t)s * orow + 2, whole + (size_t)s * orow + 2, (size_t)H * hd * 4);
+ double tol = 2e-5;
+ if (pass) CHECK(ok && same && ks.splits == sp, "kvs ds fin S %d H %d hd %d cnt %d v4 %d: not vkc_dsv4_attn's bits (err %.2e)", S, H, hd, cnt, v4, ew);
+ else CHECK(ok && same && e < tol && ks.splits > sp && ks.staged_rows > 0, "kvs ds S %d H %d hd %d cnt %d v4 %d rows %d: err %.2e (%llu host parts)", S, H, hd, cnt, v4, rows, e, ks.splits - sp);
+ printf(" kvs ds S %d hd %d cnt %d v4 %d rows %d: rel err %.2e against the reference, %.2e against the whole op%s\n",
+ S, hd, cnt, v4, rws, e, ew, pass ? (same ? " (its bits)" : " (NOT its bits)") : "");
+ vkc_free(dc); free(got); free(hostcmp);
+ vkc_kv_free(&ks);
+ }
+ vkc_free(qb); vkc_free(wb); vkc_free(cb); vkc_free(sb); vkc_free(lb); vkc_free(ow); vkc_free(os);
+ free(q); free(win); free(cmp); free(sink); free(ref); free(sc); free(list); free(kvall); free(whole);
+}
+/* The same positions in steps of different sizes give the same bits (the split's
+ * partition follows each row's own position): GQA, MLA and Inkling's, causal, and a
+ * list engine's with no pins. */
+static void test_kvs_steps(void) {
+ int T = 70, H = 4, hd = 32;
+ float *a = malloc((size_t)T * H * 64 * 4), *b = malloc((size_t)T * H * 64 * 4);
+ kvs_keep_row = (size_t)H * hd;
+ kvs_keep = a; test_kvs_gqa(T, 1, H, 2, hd, hd, 4, 32, 0, 1, 0, 1, 0, 0);
+ kvs_keep = b; test_kvs_gqa(T, 3, H, 2, hd, hd, 4, 32, 0, 1, 0, 1, 0, 0);
+ CHECK(!memcmp(a, b, (size_t)T * kvs_keep_row * 4), "kvs: GQA rows in steps of 1 and 3 differ");
+ setenv("COLI_VK_KV_SLICE", "20000", 1); /* the device's rows in slices of a row or two, a submission each */
+ kvs_keep = b; test_kvs_gqa(T, 3, H, 2, hd, hd, 4, 32, 0, 1, 0, 1, 0, 0);
+ unsetenv("COLI_VK_KV_SLICE");
+ CHECK(!memcmp(a, b, (size_t)T * kvs_keep_row * 4), "kvs: GQA rows in slices differ");
+ unsetenv("COLI_VK_KV_PIN"); /* default: a deterministic partition even for lists */
+ kvs_keep = a; test_kvs_gqa(T, 1, H, 2, hd, hd, 4, 32, 0, 0, 0, 1, 1, 1);
+ kvs_keep = b; test_kvs_gqa(T, 4, H, 2, hd, hd, 4, 32, 0, 0, 0, 1, 1, 1);
+ unsetenv("COLI_VK_KV_PIN");
+ CHECK(!memcmp(a, b, (size_t)T * kvs_keep_row * 4), "kvs: listed GQA rows in steps of 1 and 4 (no pins) differ");
+ kvs_keep_row = (size_t)2 * 64;
+ kvs_keep = a; test_kvs_mla(T, 1, 2, 64, 8, 4, 32, 0, 0);
+ kvs_keep = b; test_kvs_mla(T, 2, 2, 64, 8, 4, 32, 0, 0);
+ CHECK(!memcmp(a, b, (size_t)T * kvs_keep_row * 4), "kvs: MLA rows in steps of 1 and 2 differ");
+ kvs_keep_row = (size_t)H * hd;
+ kvs_keep = a; test_kvs_rel(T, 1, H, 2, hd, 16, 4, 4, 32);
+ kvs_keep = b; test_kvs_rel(T, 4, H, 2, hd, 16, 4, 4, 32);
+ CHECK(!memcmp(a, b, (size_t)T * kvs_keep_row * 4), "kvs: Inkling's rows in steps of 1 and 4 differ");
+ kvs_keep = NULL;
+ free(a); free(b);
+ printf(" kvs: the same bits for every position in steps of 1, 2, 3 and 4, and in slices of rows\n");
+}
+static void test_kvs(void) {
+ setenv("COLI_VK_KV_PIN", "1", 1); /* explicitly exercise read-based pins */
+ /* T step H KVH hd vd B rows win sink pm gate lists selects */
+ test_kvs_gqa(40, 1, 4, 2, 32, 32, 4, 12, 0, 0, 0, 1, 0, 0); /* decode over a window of three blocks, a gate */
+ test_kvs_gqa(41, 3, 4, 2, 32, 32, 4, 8, 0, 0, 0, 1, 0, 0); /* the smallest device: steps across a block */
+ test_kvs_gqa(70, 7, 4, 2, 48, 32, 8, 32, 0, 1, 1, 0, 0, 0); /* prefill chunks, MiMo's position-major rows, a sink */
+ test_kvs_gqa(300, 1, 8, 2, 64, 64, 16, 64, 0, 0, 0, 1, 0, 0); /* past a tile of the device's rows */
+ test_kvs_gqa(600, 50, 4, 4, 16, 16, 16, 128, 0, 0, 0, 0, 0, 0); /* host tiles of many positions, prefill rows */
+ test_kvs_gqa(60, 3, 4, 2, 32, 32, 4, 24, 0, 0, 0, 1, 1, 1); /* selection lists, pinned blocks */
+ test_kvs_gqa(50, 2, 4, 2, 256, 256, 4, 16, 0, 0, 0, 1, 1, 1); /* the largest head */
+ test_kvs_gqa(60, 2, 4, 2, 32, 24, 4, 16, 20, 1, 1, 0, 0, 0); /* a window wider than the device's rows */
+ test_kvs_gqa(48, 12, 4, 2, 32, 32, 4, 40, 0, 0, 0, 1, 0, 0); /* prompt chunks on the device whole, then split */
+ /* T step H K R B rows kv_start lists */
+ test_kvs_mla(40, 1, 4, 32, 8, 4, 12, 0, 0);
+ test_kvs_mla(90, 5, 2, 64, 16, 8, 40, 0, 1);
+ test_kvs_mla(60, 4, 3, 48, 0, 4, 16, 0, 0); /* NoPE */
+ test_kvs_mla(70, 2, 2, 512, 64, 8, 32, 5, 0); /* GLM-5.2's latent, a nonzero start */
+ test_kvs_mla(40, 3, 1, 1024, 8, 4, 16, 0, 1); /* the largest latent */
+ /* T step H KVH hd ext d_rel B rows */
+ test_kvs_rel(60, 1, 4, 2, 32, 8, 4, 4, 16); /* decode: the bias reaches back 8, the device holds 16 */
+ test_kvs_rel(80, 5, 4, 2, 16, 40, 16, 8, 32); /* the bias reaching into the host's rows */
+ test_kvs_rel(50, 3, 2, 1, 128, 16, 64, 4, 24); /* the largest bank, Inkling's head */
+ /* S H hd nwin ncmp cnt v4 B rows */
+ test_kvs_ds(1, 4, 64, 8, 40, 30, 0, 4, 12);
+ test_kvs_ds(3, 2, 512, 16, 200, 300, 0, 16, 64);
+ test_kvs_ds(2, 3, 128, 8, 90, 2900, 0, 8, 32);
+ test_kvs_ds(3, 4, 64, 8, 60, 40, 1, 4, 16); /* DeepSeek V4's roundings */
+ test_kvs_ds(2, 2, 1024, 4, 30, 20, 1, 4, 12); /* the largest head */
+ unsetenv("COLI_VK_KV_PIN");
+ test_kvs_steps();
+ /* Cross 64K with only 512 device positions, starting from a canonical RAM
+ * prefix. This also crosses host tiles and changes OpenMP task grouping.
+ * Rows not yet produced are NaN, as in the shorter sequential tests. */
+ int T = 65543, first = 65531, H = 2, hd = 16;
+ float *a = calloc((size_t)T * H * hd, sizeof(float));
+ float *b = calloc((size_t)T * H * hd, sizeof(float));
+ kvs_start = first; kvs_keep_row = (size_t)H * hd;
+ kvs_keep = a; test_kvs_gqa(T, 1, H, 1, hd, hd, 64, 512, 0, 0, 0, 0, 0, 0);
+ kvs_keep = b; test_kvs_gqa(T, 7, H, 1, hd, hd, 64, 512, 0, 0, 0, 0, 0, 0);
+ CHECK(!memcmp(a + (size_t)first * H * hd, b + (size_t)first * H * hd,
+ (size_t)(T - first) * H * hd * sizeof(float)), "kvs: rows crossing 64K differ with chunk size");
+ printf(" kvs: compared all %d rows crossing 64K bit for bit across chunk sizes\n", T - first);
+ kvs_start = 0; kvs_keep = NULL; free(a); free(b);
+}
+
/* COLI_VK_CHAIN_BENCH=1: decode-shaped GEMVs back to back in one frame (the chain's
* situation: no host gap between matrices), per weight format, in GB/s of weights. */
#include <time.h>
@@ -1807,6 +2354,7 @@ int main(int argc, char **argv) {
test_norm(VKC_NORM_ADD1, 0); test_norm(VKC_NORM_ADD1, 1); test_norm(0, 0); test_norm(VKC_NORM_NOW, 1); test_norm(VKC_NORM_L2 | VKC_NORM_NOW, 0);
printf("norm done\n");
test_rope();
+ test_chunk_rows();
test_attn(1, 0, 16, 0); test_attn(1, 140, 32, 0); test_attn(5, 200, 64, 0); test_attn(6, 9, 256, 1); test_attn(130, 3, 24, 0);
/* MiMo: a prompt's rows through a window (from position 0, and past it), a decode row
* over a ring as large as the window, prefill rows over a larger ring, full attention
@@ -1817,6 +2365,35 @@ int main(int argc, char **argv) {
test_attn_win(3, 40, 4, 2, 48, 32, 8, 10, 0, 1); test_attn_win(4, 6, 4, 2, 64, 48, 0, 0, 1, 1);
test_attn_win(2, 300, 4, 2, 16, 16, 150, 0, 1, 1); test_attn_win(130, 3, 4, 2, 24, 40, 16, 0, 0, 0);
test_attn_win(1, 500, 8, 4, 192, 128, 128, 128, 1, 1);
+ /* prompt chunks through the blocked attention (chain_attnb.comp, from 16 rows): Qwen3.6's
+ * and Qwen3.8's head groups (8 and 12 a KV head), one head a KV head, lists, the window,
+ * the ring, position-major rows and the sink; then the same cases on chain_attn */
+ for (int pass = 0; pass < 2; pass++) {
+ setenv("COLI_VK_ATTN_BLOCK", pass ? "0" : "16", 1);
+ test_attn_hk(40, 7, 16, 2, 256, 0); test_attn_hk(33, 0, 24, 2, 64, 0); test_attn_hk(64, 100, 4, 4, 32, 0);
+ test_attn_hk(37, 5, 16, 2, 128, 1); test_attn_hk(16, 200, 24, 2, 24, 1);
+ test_attn_win(40, 0, 4, 2, 48, 32, 8, 0, 1, 1); test_attn_win(33, 20, 8, 2, 64, 48, 0, 0, 1, 0);
+ test_attn_win(50, 100, 4, 4, 48, 32, 16, 40, 0, 1); test_attn_win(20, 3, 8, 4, 192, 128, 128, 0, 1, 1);
+ }
+ unsetenv("COLI_VK_ATTN_BLOCK");
+ { VkcStats st; vkc_stats(&st); CHECK(st.attn_blocked >= 9, "attn: %llu calls took the blocked attention", st.attn_blocked);
+ printf(" blocked attention: %llu calls\n", st.attn_blocked); }
+ /* a long attention cut over several submissions (COLI_VK_ATTN_SLICE): the same cases,
+ * the blocked attention and chain_attn, slices of a few rows each */
+ {
+ VkcStats s0; vkc_stats(&s0);
+ setenv("COLI_VK_ATTN_SLICE", "60000", 1);
+ for (int pass = 0; pass < 2; pass++) {
+ setenv("COLI_VK_ATTN_BLOCK", pass ? "0" : "16", 1);
+ test_attn_hk(40, 7, 16, 2, 256, 0); test_attn_hk(37, 5, 16, 2, 128, 1); test_attn_hk(64, 100, 4, 4, 32, 0);
+ test_attn_win(50, 100, 4, 4, 48, 32, 16, 40, 0, 1); test_attn_win(33, 20, 8, 2, 64, 48, 0, 0, 1, 0);
+ }
+ unsetenv("COLI_VK_ATTN_BLOCK");
+ VkcStats s1; vkc_stats(&s1);
+ CHECK(s1.attn_slices > s0.attn_slices + 10, "attn: %llu slices", s1.attn_slices - s0.attn_slices);
+ printf(" attention in slices: %llu extra submissions\n", s1.attn_slices - s0.attn_slices);
+ unsetenv("COLI_VK_ATTN_SLICE");
+ }
printf("attn done\n");
test_dnconv(0); test_dnconv(1);
test_dnrec(8, 8, 8, 4, 0); test_dnrec(4, 4, 4, 2, 1); test_dnrec(128, 128, 4, 2, 0); test_dnrec(32, 100, 6, 3, 1);
@@ -1843,6 +2420,17 @@ int main(int argc, char **argv) {
{1, 16, 8, 16, 1024, 32, 64, 1, 20, 0, 1, 0, 0, 1, 10, 5, 10}, /* the largest latent, int3-g64 */
};
for (size_t k = 0; k < sizeof cases / sizeof *cases; k++) test_mla(cases[k]);
+ { /* the core over a long chunk in slices of rows (COLI_VK_ATTN_SLICE) */
+ VkcStats s0; vkc_stats(&s0);
+ setenv("COLI_VK_ATTN_SLICE", "200000", 1);
+ test_mla(cases[1]); test_mla(cases[2]); test_mla(cases[4]); test_mla(cases[6]);
+ MlaCase big = {4, 24, 8, 32, 32, 64, 128, 40, 30, 0, 1, 0, 0, 0, 10, 10, 10}; /* 40 rows after 30 */
+ test_mla(big);
+ unsetenv("COLI_VK_ATTN_SLICE");
+ VkcStats s1; vkc_stats(&s1);
+ CHECK(s1.attn_slices > s0.attn_slices, "mla: no slice");
+ printf(" mla core in slices: %llu extra submissions\n", s1.attn_slices - s0.attn_slices);
+ }
printf("mla done\n");
test_dsa(1, 30, 2, 16, 8, 0, 0); test_dsa(4, 40, 4, 32, 16, 0, 0); test_dsa(3, 5, 2, 16, 64, 0, 0);
test_dsa(3, 5, 2, 16, 64, 1, 0); test_dsa(2, 700, 3, 64, 100, 0, 0); test_dsa(1, 9, 64, 64, 4, 0, 0);
@@ -1869,7 +2457,17 @@ int main(int argc, char **argv) {
test_situ(1.f, 1.f); test_situ(4.f, 25.f);
printf("ares done\n");
}
- if (vkc_sconv_ready() && vkc_relattn_ready()) { test_inkling(); printf("inkling done\n"); }
+ if (vkc_sconv_ready() && vkc_relattn_ready()) {
+ test_inkling();
+ VkcStats s0; vkc_stats(&s0);
+ setenv("COLI_VK_ATTN_SLICE", "30000", 1); /* the relative attention over slices of rows */
+ test_relattn(20, 30, 16, 16, 16, 4, 16); test_relattn(130, 3, 0, 200, 24, 4, 200); test_relattn(40, 10, 0, 64, 32, 16, 64);
+ unsetenv("COLI_VK_ATTN_SLICE");
+ VkcStats s1; vkc_stats(&s1);
+ CHECK(s1.attn_slices > s0.attn_slices, "relattn: no slice");
+ printf(" relative attention in slices: %llu extra submissions\n", s1.attn_slices - s0.attn_slices);
+ printf("inkling done\n");
+ }
else CHECK(0, "inkling's ops: chain_sconv.spv or chain_relattn.spv did not load");
if (!vkc_dsv4_ready()) { fails++; printf("FAIL: the DeepSeek V4 shader did not load\n"); }
else {
@@ -1895,6 +2493,8 @@ int main(int argc, char **argv) {
test_ds_swiglu(5000, 10.0f); test_ds_swiglu(3000, 0.0f);
printf("dsv4 rounding done\n");
}
+ if (!vkc_kvs_ready()) { fails++; printf("FAIL: the split KV shader did not load\n"); }
+ else { test_kvs(); printf("kvs done\n"); }
VkcStats st; vkc_stats(&st);
printf("chain: %llu frames, %llu ops, %llu matmuls (%llu GEMM), %llu barriers\n", st.frames, st.ops, st.matmuls, st.gemms, st.barriers);
vkc_shutdown();
diff --git a/c/tests/test_vk_tier.c b/c/tests/test_vk_tier.c
index c211e302..ee59dbc9 100644
--- a/c/tests/test_vk_tier.c
+++ b/c/tests/test_vk_tier.c
@@ -17,7 +17,12 @@
* the join (no failed upload, the budget holds);
* books device + CPU = routed, resident <= budget, no failed upload;
* v4 DeepSeek V4's activation (route weight on the device, its bf16 and E4M3
- * roundings): device rows against the engine's CPU arithmetic.
+ * roundings): device rows against the engine's CPU arithmetic;
+ * stream big prefill steps with streaming (vkt_issue's sub-batches, the staging
+ * slots, prefetch, experts cut into parts): every device row, resident or
+ * streamed, equals the reference, the sum the all-CPU sum, the cold experts
+ * below the rule's rows stay on the CPU, and the resident set is the warm
+ * start's before and after (streaming promotes nothing).
*
* make tests/test_vk_tier VK=1 && VK_ICD_FILENAMES=.../lvp_icd.json ./tests/test_vk_tier */
#include <stdio.h>
@@ -509,20 +514,206 @@ static void v4_act(void) {
vkt_shutdown(); model_free();
}
+/* ---- streaming ----------------------------------------------------------------------
+ * The engine's load hook: the expert's bytes as its RAM holds them. */
+static int g_loads, g_batch_left, g_partial_serial;
+static int load_cb(void *ctx, int layer, int eid, VktExpertSrc *src, void **h) {
+ if (g_batch_left >= 2) g_partial_serial++;
+ if (g_batch_left) g_batch_left--;
+ (void)ctx; *src = src_of(&ex[layer][eid]); *h = &ex[layer][eid]; g_loads++; return 1;
+}
+static void release_cb(void *ctx, void *h) { (void)ctx; (void)h; }
+/* An engine with two RAM slots: a device upload group needs several batches. */
+static int g_batches;
+static int load_batch_cb(void *ctx, int layer, const int *eids, int n, VktExpertSrc *srcs, void **h) {
+ (void)ctx; g_batches++;
+ int got = n < 2 ? n : 2;
+ g_batch_left = n - got;
+ for (int i = 0; i < got; i++) { srcs[i] = src_of(&ex[layer][eids[i]]); h[i] = &ex[layer][eids[i]]; g_loads++; }
+ return got;
+}
+/* One big step (vk_tier.h step 3, with the prefetch while "attention" runs); the device
+ * rows of cold experts counted apart. */
+static double stream_step(int layer, int S, const int *idx, const float *w, int v4, Books *bk,
+ unsigned long long *streamed, unsigned long long *kept, int prefetch) {
+ float *x = calloc((size_t)S * H, sizeof(float)), *xq = calloc((size_t)S * H, sizeof(float));
+ float *cpu = calloc((size_t)S * K * H, sizeof(float)), *out = calloc((size_t)S * H, sizeof(float));
+ float *ref = calloc((size_t)S * H, sizeof(float)), *y = malloc(sizeof(float) * H);
+ uint8_t *taken = malloc((size_t)S * K); const float **dev = malloc(sizeof(*dev) * S * K);
+ uint8_t *res = malloc((size_t)S * K);
+ for (int i = 0; i < S * H; i++) x[i] = frnd() * (v4 ? 4.0f : 1.0f);
+ memcpy(xq, x, sizeof(float) * S * H);
+ if (v4) for (int s = 0; s < S; s++) qdq128(xq + (size_t)s * H, H);
+ for (int i = 0; i < S * K; i++) res[i] = (uint8_t)vkt_resident(layer, idx[i]);
+ if (prefetch) vkt_stream_prefetch(layer, S);
+ int n = v4 ? vkt_issue_w(layer, xq, S, K, idx, w, taken) : vkt_issue(layer, xq, S, K, idx, taken);
+ for (int i = 0; i < S * K; i++) {
+ if (taken[i]) continue;
+ if (v4) expert_v4(&ex[layer][idx[i]], xq + (size_t)(i / K) * H, w[i], 10.0f, cpu + (size_t)i * H);
+ else expert_ref(&ex[layer][idx[i]], xq + (size_t)(i / K) * H, cpu + (size_t)i * H);
+ VktExpertSrc s = src_of(&ex[layer][idx[i]]); vkt_note(layer, idx[i], &s);
+ (*kept)++;
+ }
+ int joined = n ? vkt_join(dev) : 1;
+ CHECK(joined, "stream: join failed");
+ double worst = 0;
+ for (int s = 0; s < S; s++)
+ for (int k = 0; k < K; k++) {
+ int i = s * K + k;
+ if (v4) expert_v4(&ex[layer][idx[i]], xq + (size_t)s * H, w[i], 10.0f, y);
+ else expert_ref(&ex[layer][idx[i]], xq + (size_t)s * H, y);
+ float dr[H];
+ const float *c = cpu + (size_t)i * H;
+ if (taken[i] && joined) {
+ for (int d = 0; d < H; d++) dr[d] = v4 ? bf16r(dev[i][d]) : dev[i][d];
+ c = dr;
+ double r = rel(c, y, H); if (r > worst) worst = r;
+ if (!res[i]) (*streamed)++;
+ }
+ for (int d = 0; d < H; d++) { ref[(size_t)s * H + d] += (v4 ? 1.0f : w[i]) * y[d]; out[(size_t)s * H + d] += (v4 ? 1.0f : w[i]) * c[d]; }
+ bk->routed++; bk->dev += taken[i] != 0;
+ }
+ double r = rel(out, ref, S * H);
+ if (r > worst) worst = r;
+ free(x); free(xq); free(cpu); free(out); free(ref); free(y); free(taken); free(dev); free(res);
+ return worst;
+}
+static void stream(void) {
+ struct { VktFmt gu, dn; int act; const char *name; } cs[] = {
+ {{VKT_SRC_I4U_PLANAR64, 64}, {VKT_SRC_I4U_PLANAR64, 64}, VKT_ACT_SWIGLU, "planar int4-g64"},
+ {{VKT_SRC_I8_AS_I4_GS, 64}, {VKT_SRC_I8_GS, 32}, VKT_ACT_SWIGLU, "unpacked int4 gs64, int8 down"},
+ {{VKT_SRC_FP8_BLOCK, 128}, {VKT_SRC_FP8_BLOCK, 128}, VKT_ACT_SWIGLU, "fp8 128x128 blocks"},
+ {{VKT_SRC_MXFP4_E8M0, 32}, {VKT_SRC_MXFP4_E8M0, 32}, VKT_ACT_SITU, "MXFP4 ue8m0, SiTU-GLU"},
+ {{VKT_SRC_BF16, 0}, {VKT_SRC_BF16, 0}, VKT_ACT_SWIGLU, "bf16"},
+ {{VKT_SRC_MXFP4_E8M0, 32}, {VKT_SRC_MXFP4_E8M0, 32}, VKT_ACT_SWIGLU_V4, "MXFP4, DeepSeek V4 roundings"},
+ };
+ for (size_t c = 0; c < sizeof cs / sizeof *cs; c++) {
+ int v4 = cs[c].act == VKT_ACT_SWIGLU_V4;
+ model_make(cs[c].gu, cs[c].dn); g_act = cs[c].act; g_limit = 0;
+ /* Eight residents at most, four warm; batch cases have eight upload slots
+ * but only two RAM slots, so a group must continue after a partial load. */
+ set_budget(c % 2 ? 16 : 12, cs[c].gu, cs[c].dn);
+ setenv("COLI_VK_TIER_RATE", "0", 1);
+ setenv("COLI_VK_TIER_STREAM_SLOTS", c % 2 ? "8" : "4", 1);
+ setenv("COLI_VK_TIER_STREAM_ROWS", "3", 1); /* a cold expert with 3 rows or more streams */
+ setenv("COLI_VK_TIER_STREAM_HALF", "40", 1); /* sub-batches of 40 rows: several a step, experts in parts */
+ VktConfig vc = cfg_of(cs[c].gu, cs[c].dn, cs[c].act, v4 ? 10.0f : 0.f);
+ vc.load = load_cb; vc.release = release_cb;
+ if (c % 2) vc.load_batch = load_batch_cb; /* half the cases: the batch hook too */
+ g_batches = g_batch_left = g_partial_serial = 0;
+ uint32_t hist[L][E], *rows[L];
+ for (int l = 0; l < L; l++) { rows[l] = hist[l]; for (int e = 0; e < E; e++) hist[l][e] = e < 2 ? 100 - e : 0; }
+ int on = vkt_init(&vc, rows);
+ unsetenv("COLI_VK_TIER_STREAM_HALF");
+ CHECK(on, "stream %s: the tier did not start", cs[c].name);
+ if (!on) { model_free(); continue; }
+ int pl[L * E], pe[L * E], np = vkt_plan(pl, pe, L * E);
+ for (int i = 0; i < np; i++) { VktExpertSrc s = src_of(&ex[pl[i]][pe[i]]); vkt_put(pl[i], pe[i], &s); }
+ vkt_put_done();
+ uint8_t before[L][E];
+ for (int l = 0; l < L; l++) for (int e = 0; e < E; e++) before[l][e] = (uint8_t)vkt_resident(l, e);
+ Books bk = {0, 0}; unsigned long long streamed = 0, kept = 0; double worst = 0;
+ g_loads = 0;
+ static const int Ss[] = {48, 33, 1, 64, 2, 40};
+ for (int t = 0; t < 6; t++)
+ for (int l = 0; l < L; l++) {
+ int S = Ss[t], idx[64 * K]; float w[64 * K];
+ route(S, 0, E, idx, w);
+ vkt_begin_forward();
+ double r = stream_step(l, S, idx, w, v4, &bk, &streamed, &kept, t >= 3);
+ if (r > worst) worst = r;
+ }
+ int same = 1;
+ for (int l = 0; l < L; l++) for (int e = 0; e < E; e++) same &= before[l][e] == (uint8_t)vkt_resident(l, e);
+ printf(" %-34s device %3llu of %3llu assignments (%llu streamed), %llu on the CPU, %d loads (%d batches), %d warm, worst %.2e\n",
+ cs[c].name, bk.dev, bk.routed, streamed, kept, g_loads, g_batches, np, worst);
+ vkt_report("test", 0, 0);
+ CHECK(np == 4, "stream %s: %d warm experts", cs[c].name, np);
+ CHECK(!(c % 2) || g_batches > 0, "stream %s: the batch hook was never called", cs[c].name);
+ CHECK(g_partial_serial == 0, "stream %s: %d reads serialized after a partial batch", cs[c].name, g_partial_serial);
+ CHECK(streamed > 0, "stream %s: nothing streamed", cs[c].name);
+ CHECK(kept > 0, "stream %s: no cold expert stayed on the CPU", cs[c].name);
+ CHECK(same, "stream %s: the resident set moved", cs[c].name);
+ CHECK(worst < (v4 ? 2e-2 : 2e-3), "stream %s: device rows off the reference (%.3g)", cs[c].name, worst);
+ vkt_shutdown(); model_free();
+ unsetenv("COLI_VK_TIER_STREAM_SLOTS"); unsetenv("COLI_VK_TIER_STREAM_ROWS");
+ }
+}
+
+/* A large requested step under a deliberately tiny scratch budget: reserve a
+ * smaller batch before allocating, then keep the backend usable for the format
+ * and streaming reference checks below. Reusing those buffers needs no growth. */
+static void scratch_budget(void) {
+ CHECK(coli_vk_xb_init(H, F, COLI_VK_ACT_SWIGLU, 0, 0, 0), "scratch budget: init failed");
+ const size_t budget = 512 * 1024;
+ ColiVkXbStats before, after; coli_vk_xb_stats(&before);
+ int rows = coli_vk_xb_sub_fit(4096, 2, budget);
+ CHECK(rows > 0 && rows < 4096, "scratch budget: got %d rows", rows);
+ CHECK(rows > 0 && coli_vk_xb_sub_reserve(rows, 2), "scratch budget: reserve failed");
+ coli_vk_xb_stats(&after);
+ CHECK(after.scratch_bytes > before.scratch_bytes && after.scratch_bytes - before.scratch_bytes <= budget,
+ "scratch budget: grew from %zu to %zu beyond %zu bytes", before.scratch_bytes, after.scratch_bytes, budget);
+ CHECK(coli_vk_xb_sub_fit(rows, 2, 0) == rows, "scratch budget: existing buffers should need no extra bytes");
+ printf("scratch budget: %d of 4096 rows, %zu additional bytes (limit %zu)\n", rows,
+ after.scratch_bytes - before.scratch_bytes, budget);
+}
+
+/* A failed multi-expert upload may have copied only one projection. None of
+ * that group may run; its complete CPU replay must be exact. Reusing the same
+ * staging slots on the next step must replace every matrix before publishing. */
+static void stream_commit_failure(const char *spv) {
+ setenv("COLI_VK_STAGED", "1", 1);
+ setenv("COLI_VK_STAGED_FAULT", "commit:1", 1);
+ int ready = coli_vk_init(spv);
+ CHECK(ready && coli_vk_staged(), "stream fault: staged device did not start");
+ if (!ready) return;
+ VktFmt f = {VKT_SRC_FP8_BLOCK, 128};
+ model_make(f, f); g_act = VKT_ACT_SWIGLU; g_limit = 0;
+ set_budget(12, f, f);
+ setenv("COLI_VK_TIER_RATE", "0", 1);
+ setenv("COLI_VK_TIER_STREAM_SLOTS", "4", 1);
+ setenv("COLI_VK_TIER_STREAM_ROWS", "1", 1);
+ VktConfig vc = cfg_of(f, f, VKT_ACT_SWIGLU, 0);
+ vc.load = load_cb; vc.release = release_cb; vc.load_batch = load_batch_cb;
+ int on = vkt_init(&vc, NULL);
+ CHECK(on, "stream fault: tier did not start");
+ if (on) {
+ enum { S = 32 };
+ int idx[S * K]; float w[S * K];
+ for (int i = 0; i < S * K; i++) { idx[i] = i % 2; w[i] = 1.0f / K; }
+ Books books = {0, 0}; unsigned long long streamed = 0, kept = 0;
+ double error = stream_step(0, S, idx, w, 0, &books, &streamed, &kept, 0);
+ CHECK(books.dev == 0 && kept == S * K && error == 0,
+ "stream fault: partial upload used (GPU %llu, CPU %llu, error %.3g)", books.dev, kept, error);
+ books = (Books){0, 0}; streamed = kept = 0;
+ vkt_begin_forward();
+ error = stream_step(0, S, idx, w, 0, &books, &streamed, &kept, 0);
+ CHECK(books.dev == S * K && !kept && error < 2e-3,
+ "stream fault: refill did not recover (GPU %llu, CPU %llu, error %.3g)", books.dev, kept, error);
+ puts("stream fault: whole group replayed on CPU, staging slots refilled correctly");
+ }
+ vkt_shutdown(); model_free(); coli_vk_shutdown();
+ unsetenv("COLI_VK_STAGED_FAULT");
+ unsetenv("COLI_VK_TIER_STREAM_SLOTS"); unsetenv("COLI_VK_TIER_STREAM_ROWS");
+}
+
int main(int argc, char **argv) {
char buf[1024];
const char *spv = argc > 1 ? argv[1] : coli_vk_shader_path(buf, sizeof buf);
if (!coli_vk_init(spv)) { printf("FAIL: no Vulkan device (shaders %s)\n", spv); return 1; }
setenv("COLI_VK_TIER_RESERVE_GB", "0", 1);
+ scratch_budget();
printf("formats:\n"); formats();
printf("warm:\n"); warm();
printf("adapt:\n"); adapt();
printf("partial:\n"); partial();
printf("sync:\n"); sync_evict();
printf("DeepSeek V4:\n"); v4_act();
+ printf("stream:\n"); stream();
ColiVkPoolStats ps; coli_vk_pool_stats(1, &ps);
CHECK(ps.live == 0, "%d tier ranges still live after every shutdown", ps.live);
coli_vk_shutdown(); /* with staged uploads: where the experts were ("[VK] memory at exit") */
+ stream_commit_failure(spv);
printf(fails ? "FAIL (%d)\n" : "PASS\n", fails);
return fails != 0;
}
diff --git a/c/tests/vulkan_decide_compare.py b/c/tests/vulkan_decide_compare.py
new file mode 100644
index 00000000..3132028b
--- /dev/null
+++ b/c/tests/vulkan_decide_compare.py
@@ -0,0 +1,201 @@
+"""A decision engine on Vulkan against the same engine on the CPU, on its tiny fixture.
+
+ python3 tests/vulkan_decide_compare.py laya|gliner|clef [ENV=VALUE ...]
+
+Runs every case of the fixture's ref.json (the cases the tiny oracle tests read) through
+the engine's --records mode twice: once with the environment as it is minus COLI_VULKAN
+plus the pairs given that are not the device's own (COLI_VK_*), the CPU; once with
+COLI_VULKAN=1 and every pair given. Gates:
+ - the same answer to every question: the same decision (argmax), the same refusals;
+ - every probability within PROB_TOL and every logit within LOGIT_TOL of the CPU's
+ (relative to the largest |logit| of the question, at least 1): f32 activations on
+ both, so the two differ only in the order of the sums (tests/vulkan_engines.sh decide
+ records the measured worst case);
+ - the Vulkan run's "[VK] <engine>: N matmuls on the GPU" line has N > 0, and with
+ COLI_VK_CHAIN=1 its "chain: K forwards on the device" has K > 0 and none on the CPU
+ (laya and gliner; Clef's chain is qwen36's, whose own line counts its forwards).
+Prints one summary line with the worst differences.
+
+ python3 tests/vulkan_decide_compare.py --off laya|gliner|clef [ENV=VALUE ...]
+
+The device opened and every Vulkan path of the engine turned off by the pairs given
+(COLI_VK_CHAIN=0 COLI_VK_DENSE=0): the answers must be the CPU run's byte for byte (the
+elapsed time aside) and no matmul may run on the device.
+"""
+import json
+import os
+import re
+import subprocess
+import sys
+import tempfile
+from pathlib import Path
+
+HERE = Path(__file__).resolve().parent.parent
+sys.path.insert(0, str(HERE))
+from openai_server import systemone_decision_record # noqa: E402
+
+PROB_TOL = 1e-5
+LOGIT_TOL = 1e-4
+EXE = ".exe" if os.name == "nt" else ""
+
+
+def records_of(cases, form=None):
+ out = []
+ for case in cases:
+ rec = systemone_decision_record({"state": case["state"], "questions": case["questions"]},
+ *(() if form is None else (form,)))
+ out.append({"payload": json.dumps(rec, ensure_ascii=False)})
+ return out
+
+
+def run(engine, records, env, max_length=None):
+ with tempfile.TemporaryDirectory() as tmp:
+ path = Path(tmp) / "records.json"
+ path.write_text(json.dumps(records, ensure_ascii=False), encoding="utf-8")
+ e = dict(env, OMP_NUM_THREADS=env.get("OMP_NUM_THREADS", "2"), COLI_NO_OMP_TUNE="1")
+ if engine == "laya":
+ argv = [str(HERE / ("laya" + EXE)), "--model", str(HERE / "laya_tiny"), "--records", str(path)]
+ elif engine == "gliner":
+ ref = json.loads((HERE / "gliner_decide_tiny" / "ref.json").read_text(encoding="utf-8"))
+ e["COLI_GLINER_MAX_LEN"] = str(ref["max_len"])
+ argv = [str(HERE / ("gliner_decide" + EXE)), "--model", str(HERE / "gliner_decide_tiny"),
+ "--records", str(path)]
+ else:
+ e.update(SNAP=str(HERE / "clef_tiny_c"), CLEF_RECORDS=str(path),
+ COLI_DENSE_I8=env.get("COLI_DENSE_I8", "0"))
+ if max_length:
+ e["COLI_CLEF_MAX_LEN"] = str(max_length)
+ argv = [str(HERE / ("qwen36" + EXE)), "8", "8"]
+ r = subprocess.run(argv, capture_output=True, text=True, encoding="utf-8", env=e, timeout=1800)
+ if r.returncode:
+ raise SystemExit(f"FAIL: {engine} exited {r.returncode}\n{r.stderr[-3000:]}")
+ return [json.loads(line) for line in r.stdout.splitlines()], r.stderr
+
+
+def strip_ms(out):
+ return [re.sub(r'"[a-z_]*ms": ?[0-9.eE+-]+', '"ms":0', json.dumps(o, sort_keys=True)) for o in out]
+
+
+def main():
+ off = sys.argv[1] == "--off"
+ args = sys.argv[2:] if off else sys.argv[1:]
+ engine = args[0]
+ extra = dict(a.split("=", 1) for a in args[1:])
+ fixture = {"laya": "laya_tiny", "gliner": "gliner_decide_tiny", "clef": "clef_tiny"}[engine]
+ ref = json.loads((HERE / fixture / "ref.json").read_text(encoding="utf-8"))
+ groups = {}
+ for case in ref["cases"]:
+ groups.setdefault(case.get("max_length") if engine == "clef" else None, []).append(case)
+ base = {k: v for k, v in os.environ.items() if k != "COLI_VULKAN"}
+ vk_env = dict(base, COLI_VULKAN="1", **extra)
+ # the CPU arm takes every pair but the device's own switches: the same weights, and
+ # the same activation arithmetic where the engine has a choice (COLI_DENSE_IDOT=0)
+ cpu_env = dict(base, **{k: v for k, v in extra.items() if not k.startswith("COLI_VK")})
+ name = {"laya": "laya", "gliner": "gliner_decide", "clef": "qwen36"}[engine]
+ worst_p = worst_l = 0.0
+ decisions = refusals = 0
+ logs = []
+ for max_length, cases in groups.items():
+ recs = records_of(cases, "raw" if engine == "clef" else None)
+ cpu, _ = run(engine, recs, cpu_env, max_length)
+ vk, err = run(engine, recs, vk_env, max_length)
+ logs.append(err)
+ if off:
+ if strip_ms(cpu) != strip_ms(vk):
+ raise SystemExit(f"FAIL: with every Vulkan path off the answers differ from the CPU's\n{err[-3000:]}")
+ decisions += sum(len(o.get("answers", [])) for o in vk)
+ continue
+ if len(cpu) != len(cases) or len(vk) != len(cases):
+ raise SystemExit(f"FAIL: {len(cpu)} CPU and {len(vk)} Vulkan answers for {len(cases)} cases\n{err[-3000:]}")
+ for case, a, b in zip(cases, cpu, vk):
+ if "error" in a or "error" in b:
+ if a.get("error") != b.get("error"):
+ raise SystemExit(f"FAIL: {case['name']}: CPU {a.get('error')!r}, Vulkan {b.get('error')!r}")
+ refusals += 1
+ continue
+ for qa, qb in zip(a["answers"], b["answers"]):
+ where = f"{case['name']}.{qa['id']}"
+ pa, pb = qa["probs"], qb["probs"]
+ if max(range(len(pa)), key=lambda i: (pa[i], -i)) != max(range(len(pb)), key=lambda i: (pb[i], -i)):
+ raise SystemExit(f"FAIL: {where}: a different decision, CPU {pa} Vulkan {pb}")
+ dp = max(abs(x - y) for x, y in zip(pa, pb))
+ scale = max(1.0, max(abs(x) for x in qa["logits"]))
+ dl = max(abs(x - y) for x, y in zip(qa["logits"], qb["logits"])) / scale
+ worst_p, worst_l = max(worst_p, dp), max(worst_l, dl)
+ if dp > PROB_TOL or dl > LOGIT_TOL:
+ raise SystemExit(f"FAIL: {where}: |dp| {dp:.2e}, |dlogit|/scale {dl:.2e}\n"
+ f" CPU {qa['logits']}\n Vulkan {qb['logits']}")
+ decisions += 1
+ err = "\n".join(logs)
+ counts = [int(n) for n in re.findall(rf"^\[VK\] {name}: (\d+) matmuls on the GPU", err, re.M)]
+ if off:
+ if not re.search(rf"^\[VK\] {name}: device ready", err, re.M) or any(counts):
+ raise SystemExit(f"FAIL: the device did not open, or a matmul ran on it\n{err[-3000:]}")
+ print(f"OK {engine} {' '.join(args[1:])}: the device open and every path off, {decisions} answers "
+ f"byte for byte the CPU's, no matmul on the device")
+ return
+ chain = ""
+ if engine == "clef" and extra.get("COLI_VK_CHAIN", "") not in ("", "0"):
+ # qwen36's chain records its matrices in its frames: its line counts the forwards
+ fw = [int(n) for n in re.findall(r"^\[VK\] qwen36 chain: (\d+) forwards", err, re.M)]
+ if "COLI_VK_CHAIN_FAULT" in extra:
+ # A failed prompt can have completed rows without a completed forward.
+ # Require the injected submit failure in every process, and prove that
+ # the discarded host weights were reloaded before checking its answers.
+ for log in logs:
+ if "queue submit (COLI_VK_CHAIN_FAULT) failed" not in log:
+ raise SystemExit("FAIL: Clef did not reach the injected device loss")
+ if extra.get("COLI_VK_DENSE_HOST") == "0":
+ restored = re.search(r"dense weights at exit: (\d+) matrices.*?, (\d+) read back", log)
+ if not restored or min(map(int, restored.groups())) <= 0:
+ raise SystemExit("FAIL: Clef did not drop and restore its host weights")
+ chain = ", failed prompt replayed on the CPU after the injected device loss"
+ elif not fw or fw[-1] <= 0:
+ raise SystemExit(f"FAIL: qwen36's chain did not run\n{err[-3000:]}")
+ else:
+ chain = f", {fw[-1]} chain forwards"
+ layers = json.loads((HERE / "clef_tiny_c" / "qwen36_meta.json").read_text())["n_layers"]
+ # A dense backbone must not put a whole model in one submission:
+ # on the real 27B checkpoint that exceeded the driver watchdog.
+ # Compare successive reports to exclude one-time allocation frames.
+ for log in logs:
+ snapshots = [tuple(map(int, x)) for x in re.findall(
+ r"^\[VK\] qwen36 chain: (\d+) forwards, (\d+) frames", log, re.M)]
+ for (f0, s0), (f1, s1) in zip(snapshots, snapshots[1:]):
+ if s1 - s0 < (f1 - f0) * layers:
+ raise SystemExit("FAIL: Clef accumulated multiple dense layers in one submission")
+ counts = counts or [0]
+ elif not counts or min(counts) <= 0:
+ raise SystemExit(f"FAIL: no matmul ran on the device\n{err[-3000:]}")
+ if engine == "clef":
+ # Import follows the actual device policy: default on shared-memory devices,
+ # copies on discrete devices. Device-only mode keeps imported pages only on
+ # an integrated GPU. An unsupported import (e.g. Dozen) falls back to copies.
+ inplace = [float(x) for x in re.findall(r"^\[VK\] qwen36: \d+ matmuls.*; ([0-9.]+) MiB read in place", err, re.M)]
+ rows = extra.get("COLI_DENSE_I8", "0") != "0" and extra.get("COLI_DENSE_BITS", "8") in ("8", "16")
+ shared = "dense rows shared with the integrated GPU" in err
+ requested = "dense rows read in place, not copied" in err
+ copy_required = "qwen36: dense weights on the device only" in err and not shared
+ want = (rows and requested and not copy_required
+ and "cannot import host memory" not in err)
+ if want != bool(inplace and inplace[-1] > 0):
+ raise SystemExit(f"FAIL: rows read in place: expected {want}, got {inplace[-1:] or 'none'}\n{err[-3000:]}")
+ chain += f", {inplace[-1]:.1f} MiB read in place" if want else ", rows copied"
+ if engine != "clef" and extra.get("COLI_VK_CHAIN", "") not in ("", "0"):
+ fw = re.findall(rf"^\[VK\] {name} chain: (\d+) forwards on the device .*?, (\d+) on the CPU", err, re.M)
+ if "COLI_VK_CHAIN_FAULT" in extra:
+ # a device lost mid-run: forwards before it on the device, the rest on the CPU
+ if not re.search(rf"^\[VK\] {name}: the device was lost", err, re.M) or not fw or \
+ int(fw[-1][0]) <= 0 or int(fw[-1][1]) <= 0:
+ raise SystemExit(f"FAIL: the device was not lost in the middle of the run\n{err[-3000:]}")
+ chain = f", {fw[-1][0]} forwards on the device and {fw[-1][1]} on the CPU after the loss"
+ elif not fw or int(fw[-1][0]) <= 0 or int(fw[-1][1]) != 0:
+ raise SystemExit(f"FAIL: the forward did not run on the device\n{err[-3000:]}")
+ else:
+ chain = f", {fw[-1][0]} forwards on the device"
+ print(f"OK {engine} {' '.join(args[1:])}: {decisions} decisions and {refusals} refusals as on the CPU; "
+ f"max |dp| {worst_p:.2e}, max |dlogit| {worst_l:.2e} of the largest; {counts[-1]} matmuls on the GPU{chain}")
+
+
+if __name__ == "__main__":
+ main()
diff --git a/c/tests/vulkan_dense_only_deepseek.sh b/c/tests/vulkan_dense_only_deepseek.sh
new file mode 100644
index 00000000..c7b28f6a
--- /dev/null
+++ b/c/tests/vulkan_dense_only_deepseek.sh
@@ -0,0 +1,261 @@
+# Sourced by tests/vulkan_engines.sh: DeepSeek V4 (deepseek_v4) and DeepSeek V4.1 Flash
+# (deepseek_v41) with their dense weights on the device only (COLI_VK_DENSE_HOST=0,
+# docs/vulkan.md "Dense weights on the device only"), on the deepseek-chain family's
+# fixtures and gates (v41_gate, v4_chain_gate, v4_chain_same: the CPU's tokens, the
+# reference's, the logits within each engine's bound, the chain against itself), plus:
+# - the matrices went to the device only and none came back from disk on a healthy run;
+# - the same bits as the run that keeps its host copies (the device holds the same
+# tensors either way);
+# - a lost device: the CPU's tokens, the matrices read back from disk (deepseek_v4: its
+# dense layers read per forward from there on, as a low-memory plan does);
+# - deepseek_v4 under a RAM limit (--memory-gb) where the plan streams the dense layers
+# with the host copies and the chain stays off; on the device only the plan keeps them
+# there and the chain runs; and one where the experts' cache gets the dense layers' RAM;
+# - staged uploads, serve sessions, prompts only, the per-matrix path alone.
+
+dho_v41_check() { # <tag> <log> <lost 0|1>
+ local n r; n=$(dho_dropped "$2"); r=$(dho_reloaded "$2")
+ [ "$n" -gt 0 ] || { cat "$2"; fail "$1: no matrix went to the device only"; }
+ if [ "$3" = 1 ]; then [ "$r" -gt 0 ] || { grep '^\[VK\]' "$2"; fail "$1: nothing was read back after the loss"; }
+ else [ "$r" = 0 ] || { grep '^\[VK\]' "$2"; fail "$1: $r matrices read back from disk on a healthy run"; }; fi
+ echo " $n matrices on the device only, $r read back"
+}
+# dho_v41 <tag> <env...> -- <argv...>: v41_gate with the host copies dropped, then the
+# same run keeping them (COLI_VK_DENSE_HOST=1): the same logits bit for bit (the tier's
+# balancer off in both: it moves experts between the device and the CPU by measured
+# times, and V4.1's two sides sum an expert in different orders). Not with prompts only
+# (CHAINMODE=2): there the run with host copies decodes its trunk on the CPU (this
+# device's default with the tier on), the run without them on the device.
+dho_v41() {
+ local tag=$1; shift
+ # The production streaming threshold is timed. Fix it when comparing bits so
+ # host-copy allocation timing cannot move experts between CPU and GPU arithmetic.
+ local COLI_VK_TIER_STREAM_ROWS=16
+ export COLI_VK_TIER_STREAM_ROWS
+ v41_gate "$tag" COLI_VK_DENSE_HOST=0 COLI_VK_TIER_BALANCE=0 "$@"
+ dho_v41_check "$tag" vk.log "$([ -n "${FAULT_BACK:-}" ] && echo 1 || echo 0)"
+ if [ -z "${FAULT_BACK:-}" ] && [ "${CHAINMODE:-1}" = 1 ]; then
+ local envs=(); while [ "$1" != "--" ]; do envs+=("$1"); shift; done; shift
+ cp vk.f32 dho.f32
+ rm -f chain.usage host.f32
+ env "${envs[@]}" COLI_VK_DENSE_HOST=1 COLI_VK_TIER_BALANCE=0 DUMP=host.f32 COLI_USAGE=chain.usage COLI_VK_TIER_SYNC=1 COLI_VULKAN=1 \
+ COLI_VK_CHAIN=${CHAINMODE:-1} ./deepseek_v41 "$@" > /dev/null 2> host.log || true
+ cmp -s dho.f32 host.f32 || fail "$tag: the logits differ from the run that keeps its host copies"
+ echo " logits = the host-copy run's, bit for bit"
+ fi
+}
+# the per-matrix path alone (no chain), the device's matrices still its own
+dho_v41_matrix() { # <tag> <env...> -- <argv...>
+ local tag=$1; shift
+ local envs=(); while [ "$1" != "--" ]; do envs+=("$1"); shift; done; shift
+ rm -f chain.usage cpu.f32 vk.f32
+ local rc_cpu=0 rc_vk=0
+ env "${envs[@]}" DUMP=cpu.f32 ./deepseek_v41 "$@" > cpu.txt 2> cpu.log || rc_cpu=$?
+ env "${envs[@]}" COLI_VK_DENSE_HOST=0 COLI_VK_CHAIN=0 COLI_VK_DENSE=1 DUMP=vk.f32 COLI_USAGE=chain.usage \
+ COLI_VK_TIER_SYNC=1 COLI_VULKAN=1 ./deepseek_v41 "$@" > vk.txt 2> vk.log || rc_vk=$?
+ [ "$rc_cpu" = "$rc_vk" ] && cmp -s cpu.txt vk.txt || { cat cpu.txt vk.txt; tail -20 vk.log; fail "$tag: tokens"; }
+ local lg; lg=$(logits_close cpu.f32 vk.f32) || { echo "$lg"; fail "$tag: logits"; }
+ dho_v41_check "$tag" vk.log 0 > /dev/null
+ echo "OK $tag: tokens = CPU, $lg, $(vk_count deepseek_v41 vk.log) matmuls on the device, $(dho_dropped vk.log) matrices on it only"
+}
+
+dho_v4_check() { # <tag> <log> <lost 0|1>
+ local n r; n=$(dho_dropped "$2"); r=$(dho_reloaded "$2")
+ [ "$n" -gt 0 ] || { cat "$2"; fail "$1: no matrix went to the device only"; }
+ grep -q '^ram_tiers .*dense=device(' "$2" || { grep ram_tiers "$2"; fail "$1: the plan did not count the dense layers on the device"; }
+ if [ "$3" = 1 ]; then
+ [ "$r" -gt 0 ] || { grep '^\[VK\]' "$2"; fail "$1: nothing was read back after the loss"; }
+ grep -q 'the device that held the dense layers is gone' "$2" || { grep '^\[VK\]' "$2"; fail "$1: the per-forward reads did not take over"; }
+ else [ "$r" = 0 ] || { grep '^\[VK\]' "$2"; fail "$1: $r matrices read back from disk on a healthy run"; }; fi
+ echo " $n matrices on the device only, $r read back, $(grep -o 'dense=device([^)]*)' "$2" | head -1)"
+}
+dho_v4() { # <tag> <fixture> <case> <env...>: v4_chain_gate on the device only
+ local tag=$1 fx=$2 c=$3; shift 3
+ v4_chain_gate "$tag" "$fx" "$c" COLI_VK_DENSE_HOST=0 "$@"
+ dho_v4_check "$tag" v4-vk.err "$([ -n "${FAULT_BACK:-}" ] && echo 1 || echo 0)"
+}
+# the per-matrix path alone on the device only: ids = CPU = reference, logits within V4's bound
+dho_v4_matrix() { # <tag> <fixture> <case> <env...>
+ local tag=$1 fx=$2 c=$3; shift 3
+ local p mt
+ p=$(v4_chain_prompt "$fx" "$c")
+ mt=$($PY -c 'import json,sys; print(json.load(open(sys.argv[1]+"/ref.json"))["cases"][sys.argv[2]]["max_new_tokens"])' "$fx" "$c")
+ rm -f "$fx/.coli_usage" cpu.f32 vk.f32
+ env "$@" DUMP=cpu.f32 ./deepseek_v4 "./$fx" "$p" --raw-prompt --max-tokens "$mt" --record-oracle v4-cpu.json > /dev/null 2> v4-cpu.err || { cat v4-cpu.err; fail "$tag: CPU"; }
+ rm -f "$fx/.coli_usage"
+ env "$@" COLI_VK_DENSE_HOST=0 COLI_VK_CHAIN=0 COLI_VK_DENSE=1 DUMP=vk.f32 COLI_VK_TIER_SYNC=1 COLI_VULKAN=1 \
+ ./deepseek_v4 "./$fx" "$p" --raw-prompt --max-tokens "$mt" --record-oracle v4-vk.json > /dev/null 2> v4-vk.err || { tail -20 v4-vk.err; fail "$tag: device"; }
+ rm -f "$fx/.coli_usage"
+ $PY -c 'import json,sys; a=json.load(open("v4-cpu.json")); b=json.load(open("v4-vk.json")); r=json.load(open(sys.argv[1]+"/ref.json"))["cases"][sys.argv[2]]["greedy_full_ids"]; sys.exit(not (a["full_ids"]==b["full_ids"]==r))' "$fx" "$c" ||
+ fail "$tag: the ids differ from the CPU's"
+ local lg; lg=$(v4_chain_logits cpu.f32 vk.f32 "$($PY -c 'import json,sys; print(json.load(open(sys.argv[1]+"/config.json"))["vocab_size"])' "$fx")") || { echo "$lg"; fail "$tag: logits"; }
+ [ "$(vk_count deepseek_v4 v4-vk.err)" -gt 0 ] || { cat v4-vk.err; fail "$tag: no matmul on the device"; }
+ dho_v4_check "$tag" v4-vk.err 0 > /dev/null
+ echo "OK $tag (${c}): ids = CPU = reference, $lg, $(vk_count deepseek_v4 v4-vk.err) matmuls on the device, $(dho_dropped v4-vk.err) matrices on it only"
+}
+# The RAM limit of the user's case (#1852), on the 8-expert fixture: at LOW GiB the plan
+# with the host copies streams the dense layers (read per forward, the chain declines),
+# on the device only it keeps them there and the chain runs; at MID GiB the host copies
+# fit and take RAM the experts' cache gets on the device only (one slot more a layer).
+# The two limits are this fixture's plan arithmetic (found by bisection on --memory-gb;
+# the plan is exact integer arithmetic on the configured limit, no machine reading).
+DHO_V4_LOW=0.500917917
+DHO_V4_MID=0.501555663
+dho_v4_lowram() { # <tag> <GiB> <expect: stream|slots>
+ local tag=$1 gib=$2 expect=$3 fx=deepseek_v4_tiny_e8 c=long
+ local p mt side
+ p=$(v4_chain_prompt "$fx" "$c")
+ mt=$($PY -c 'import json,sys; print(json.load(open(sys.argv[1]+"/ref.json"))["cases"][sys.argv[2]]["max_new_tokens"])' "$fx" "$c")
+ for side in cpu host dev; do
+ rm -f "$fx/.coli_usage" "lr-$side.f32" "lr-$side.json"
+ local e=()
+ case $side in
+ host) e=(COLI_VULKAN=1 COLI_VK_CHAIN=1 COLI_VK_TIER_SYNC=1 COLI_VK_DENSE_HOST=1) ;;
+ dev) e=(COLI_VULKAN=1 COLI_VK_CHAIN=1 COLI_VK_TIER_SYNC=1 COLI_VK_DENSE_HOST=0) ;;
+ esac
+ env "${e[@]}" DUMP=lr-$side.f32 ./deepseek_v4 "./$fx" "$p" --raw-prompt --max-tokens "$mt" --memory-gb "$gib" \
+ --record-oracle lr-$side.json > /dev/null 2> lr-$side.err || { tail -20 lr-$side.err; fail "$tag: $side run"; }
+ done
+ rm -f "$fx/.coli_usage"
+ $PY -c 'import json,sys; r=json.load(open(sys.argv[1]+"/ref.json"))["cases"][sys.argv[2]]["greedy_full_ids"]; ids=[json.load(open("lr-%s.json"%s))["full_ids"] for s in ("cpu","host","dev")]; sys.exit(not all(i==r for i in ids))' "$fx" "$c" ||
+ fail "$tag: the ids differ from the CPU's"
+ local V; V=$($PY -c 'import json,sys; print(json.load(open(sys.argv[1]+"/config.json"))["vocab_size"])' "$fx")
+ local lg; lg=$(v4_chain_logits lr-cpu.f32 lr-dev.f32 "$V") || { echo "$lg"; fail "$tag: logits"; }
+ local sh sd; sh=$(sed -n 's/^ram_tiers .*target_slots=\([0-9]*\).*/\1/p' lr-host.err); sd=$(sed -n 's/^ram_tiers .*target_slots=\([0-9]*\).*/\1/p' lr-dev.err)
+ grep -q '^ram_tiers .*dense=device(' lr-dev.err || { grep ram_tiers lr-dev.err; fail "$tag: the plan did not keep the dense layers on the device"; }
+ [ "$(chain_count deepseek_v4 lr-dev.err)" -gt 0 ] || { grep '\[VK\]' lr-dev.err; fail "$tag: the chain did not run on the device only"; }
+ if [ "$expect" = stream ]; then
+ grep -q '^ram_tiers .*dense=streamed' lr-host.err || { grep ram_tiers lr-host.err; fail "$tag: the host-copy plan did not stream at this limit"; }
+ grep -q 'deepseek_v4 chain: the dense layers are not resident' lr-host.err || { grep '\[VK\]' lr-host.err; fail "$tag: the host-copy chain did not decline"; }
+ echo "OK $tag: --memory-gb $gib: host copies: dense=streamed, the chain declined, $sh slots; device only: $(grep -o 'dense=device([^)]*)' lr-dev.err), $(chain_count deepseek_v4 lr-dev.err) chain forwards, $sd slots; ids = CPU = reference, $lg"
+ else
+ grep -q '^ram_tiers .*dense=resident' lr-host.err || { grep ram_tiers lr-host.err; fail "$tag: the host copies did not fit at this limit"; }
+ [ "$sd" -gt "$sh" ] || fail "$tag: the experts' cache got no more slots on the device only ($sh, $sd)"
+ echo "OK $tag: --memory-gb $gib: host copies resident, $sh slots a layer; device only $sd slots a layer; ids = CPU = reference, $lg"
+ fi
+}
+# A serve session's engine on the device only: the mode is on at READY (the frames
+# themselves are v4_chain_serve's, below, which runs in the same environment).
+dho_v4_serve_mode() { # <fixture>
+ rm -f "$1/.coli_usage"
+ SERVE=1 SNAP=$1 CTX=128 COLI_VULKAN=1 COLI_VK_CHAIN=1 COLI_VK_DENSE_HOST=0 ./deepseek_v4 < /dev/null > /dev/null 2> sv.err || true
+ rm -f "$1/.coli_usage"
+ [ "$(dho_dropped sv.err)" -gt 0 ] && grep -q '^ram_tiers .*dense=device(' sv.err || { cat sv.err; fail "deepseek_v4 serve: not on the device only"; }
+ echo "OK deepseek_v4 serve on the device only: $(dho_dropped sv.err) matrices, $(grep -o 'dense=device([^)]*)' sv.err)"
+}
+
+dho_family_deepseek() {
+ export OMP_NUM_THREADS=2
+ make deepseek_v41 VK=1
+ make deepseek-v4 VK=1
+ # ---- deepseek_v41
+ v41_chain_fixtures
+ local cap f
+ for cap in 1 8; do dho_v41 "dense-only deepseek_v41 cap=$cap" SNAP=dsv41_tiny -- $cap dsv41_tiny/ref.json; done
+ dho_v41 "dense-only deepseek_v41 40-token prompt" SNAP=dsv41_long -- 8 dsv41_long/ref.json
+ for f in 2 5; do dho_v41 "dense-only deepseek_v41 DSpark spec=$f" SNAP=dsv41_tiny V41_DSPARK=1 V41_SPEC_FORCE=$f -- 8 dsv41_tiny/ref.json; done
+ dho_v41 "dense-only deepseek_v41 prompt in chunks of 3" SNAP=dsv41_long COLI_VK_CHAIN_ROWS=3 -- 8 dsv41_long/ref.json
+ dho_v41 "dense-only deepseek_v41 tier off" SNAP=dsv41_long COLI_VK_TIER=0 -- 8 dsv41_long/ref.json
+ CHAINMODE=2 dho_v41 "dense-only deepseek_v41 prompts only" SNAP=dsv41_long -- 8 dsv41_long/ref.json
+ COLI_VK_STAGED=1 dho_v41 "dense-only deepseek_v41 staged" SNAP=dsv41_long -- 8 dsv41_long/ref.json
+ dho_v41_matrix "dense-only deepseek_v41 the per-matrix path alone" SNAP=dsv41_long -- 8 dsv41_long/ref.json
+ FAULT_BACK=5 dho_v41 "dense-only deepseek_v41 device lost mid-decode" SNAP=dsv41_long -- 8 dsv41_long/ref.json
+ FAULT_BACK=100 dho_v41 "dense-only deepseek_v41 device lost in the prompt" SNAP=dsv41_long COLI_VK_CHAIN_ROWS=7 -- 8 dsv41_long/ref.json
+ CHAIN_SERVE_EXPECT='dense matrices on the device only' COLI_VK_DENSE_HOST=0 \
+ $PY tests/vulkan_chain_serve.py ./deepseek_v41 dsv41_tiny V41_DSPARK=1
+ # unset on a CPU device the host copies stay (the auto rule)
+ rm -f chain.usage
+ env COLI_USAGE=chain.usage COLI_VULKAN=1 COLI_VK_CHAIN=1 SNAP=dsv41_tiny ./deepseek_v41 8 dsv41_tiny/ref.json > /dev/null 2> vk.log || true
+ grep -q 'deepseek_v41: dense weights on the device and in host RAM (a CPU device' vk.log && [ "$(dho_dropped vk.log)" = 0 ] ||
+ { grep '^\[VK\]' vk.log; fail "deepseek_v41: the auto rule dropped the host copies on a CPU device"; }
+ echo "OK dense-only deepseek_v41 auto on a CPU device: host copies kept"
+ rm -f chain.usage chain-serve.usage dho.f32 host.f32 host.log
+ # ---- deepseek_v4
+ v4_chain_fixtures
+ local c fx
+ for c in short compressed long; do
+ dho_v4 "dense-only deepseek_v4 4 experts" deepseek_v4_tiny_t $c
+ dho_v4 "dense-only deepseek_v4 8 experts, pinned rows16" deepseek_v4_tiny_e8 $c
+ done
+ dho_v4 "dense-only deepseek_v4 2 output groups (wo_a per group)" deepseek_v4_tiny_g2 long
+ dho_v4 "dense-only deepseek_v4 a window of 4, ratios 4 and 2" deepseek_v4_tiny_w4 long
+ dho_v4 "dense-only deepseek_v4 an indexer of 64 heads of 128" deepseek_v4_tiny_ix long
+ dho_v4 "dense-only deepseek_v4 tier off" deepseek_v4_tiny_t long COLI_VK_TIER=0
+ dho_v4 "dense-only deepseek_v4 chain chunks of 3" deepseek_v4_tiny_e8 long COLI_VK_CHAIN_ROWS=3
+ CHAINMODE=2 dho_v4 "dense-only deepseek_v4 prompts only" deepseek_v4_tiny_t long
+ COLI_VK_STAGED=1 dho_v4 "dense-only deepseek_v4 staged" deepseek_v4_tiny_t long
+ COLI_VK_STAGED=1 dho_v4 "dense-only deepseek_v4 staged, 2 output groups" deepseek_v4_tiny_g2 long
+ local M=ids:30,31,32,33,34,35,60,61,30,31,32,33,34,35,36,37,38,39,40,41,30,31,32:12
+ dho_v4 "dense-only deepseek_v4 n-gram drafts, a window of 4" deepseek_v4_tiny_w4 $M V4_DRAFT=4 V4_NGRAM_PARTIAL_KEEP=1
+ dho_v4_matrix "dense-only deepseek_v4 the per-matrix path alone" deepseek_v4_tiny_t long
+ dho_v4_matrix "dense-only deepseek_v4 the per-matrix path alone, 2 output groups" deepseek_v4_tiny_g2 long
+ # the same bits as the run that keeps its host copies
+ for fx in deepseek_v4_tiny_t deepseek_v4_tiny_g2; do
+ v4_chain_same "dense-only deepseek_v4 host copies against device only, $fx" $fx long "COLI_VK_DENSE_HOST=1" "COLI_VK_DENSE_HOST=0"
+ done
+ FAULT_BACK=3 dho_v4 "dense-only deepseek_v4 device lost mid-decode" deepseek_v4_tiny_t long
+ FAULT_BACK=40 dho_v4 "dense-only deepseek_v4 device lost in the prompt" deepseek_v4_tiny_t long COLI_VK_CHAIN_ROWS=7
+ dho_v4_lowram "dense-only deepseek_v4 a low-RAM plan (#1852)" $DHO_V4_LOW stream
+ dho_v4_lowram "dense-only deepseek_v4 the dense layers' RAM to the experts" $DHO_V4_MID slots
+ dho_v4_serve_mode deepseek_v4_tiny_t
+ COLI_VK_DENSE_HOST=0 v4_chain_serve deepseek_v4_tiny_t
+ local t
+ for t in test_deepseek_v4_prefix test_deepseek_v4_brio; do
+ COLI_VULKAN=1 COLI_VK_CHAIN=1 COLI_VK_DENSE_HOST=0 COLI_VK_TIER_SYNC=1 COLI_USAGE=$PWD/chain.usage \
+ $PY tests/$t.py --binary "$PWD/deepseek_v4" --fixture "$PWD/deepseek_v4_tiny_t" > v4-test.log 2>&1 ||
+ { cat v4-test.log; fail "deepseek_v4 $t on the device only"; }
+ echo "OK deepseek_v4 $t on the device only: $(tail -1 v4-test.log)"
+ done
+ # unset on a CPU device the host copies stay
+ rm -f deepseek_v4_tiny_t/.coli_usage
+ COLI_VULKAN=1 COLI_VK_CHAIN=1 ./deepseek_v4 ./deepseek_v4_tiny_t "$(v4_chain_prompt deepseek_v4_tiny_t short)" --raw-prompt --max-tokens 2 > /dev/null 2> vk.log || true
+ grep -q 'deepseek_v4: dense weights on the device and in host RAM (a CPU device' vk.log && [ "$(dho_dropped vk.log)" = 0 ] &&
+ grep -q '^ram_tiers .*dense=resident' vk.log || { grep -E '^\[VK\]|ram_tiers' vk.log; fail "deepseek_v4: the auto rule dropped the host copies on a CPU device"; }
+ echo "OK dense-only deepseek_v4 auto on a CPU device: host copies kept"
+ rm -rf deepseek_v4_tiny_t deepseek_v4_tiny_e8 deepseek_v4_tiny_g2 deepseek_v4_tiny_w4 deepseek_v4_tiny_ix chain.usage
+ rm -f cpu.f32 vk.f32 v4-*.f32 v4-*.json v4-*.err v4-tf.txt v4-test.log lr-*.f32 lr-*.json lr-*.err sv.err
+ unset OMP_NUM_THREADS
+}
+
+# The same under ASan and UBSan (LTO off for deepseek_v4, as its chain family builds it):
+# no diagnostic, the matrices on the device only, and after a loss the read-back.
+dho_family_deepseek_sanitize() {
+ local SAN="-fsanitize=address,undefined -fno-omit-frame-pointer -g"
+ make clean >/dev/null 2>&1 || true
+ make deepseek-v4-clean >/dev/null 2>&1 || true
+ make deepseek_v41 VK=1 EXTRA_CFLAGS="$SAN"
+ make deepseek-v4 VK=1 LTO=0 EXTRA_CFLAGS="$SAN" EXTRA_LDFLAGS="$SAN"
+ export ASAN_OPTIONS=detect_leaks=0:detect_stack_use_after_return=0 UBSAN_OPTIONS=print_stacktrace=1 OMP_NUM_THREADS=2
+ v41_chain_fixtures
+ dsan41() { # <tag> <lost 0|1> <env and argv...>
+ local tag=$1 lost=$2; shift 2
+ rm -f chain.usage
+ env COLI_USAGE=chain.usage COLI_VK_TIER_SYNC=1 COLI_VULKAN=1 COLI_VK_CHAIN=1 COLI_VK_DENSE_HOST=0 "$@" > san.log 2>&1 || true
+ if grep -qE "ERROR: AddressSanitizer|runtime error:" san.log; then cat san.log; fail "$tag: sanitizer diagnostic"; fi
+ dho_v41_check "$tag" san.log "$lost" > /dev/null
+ echo "OK $tag: sanitizers clean, $(dho_dropped san.log) matrices on the device only, $(dho_reloaded san.log) read back"
+ }
+ dsan41 "asan dense-only deepseek_v41 40-token prompt" 0 SNAP=dsv41_long ./deepseek_v41 8 dsv41_long/ref.json
+ dsan41 "asan dense-only deepseek_v41 DSpark spec=5" 0 SNAP=dsv41_tiny V41_DSPARK=1 V41_SPEC_FORCE=5 ./deepseek_v41 1 dsv41_tiny/ref.json
+ dsan41 "asan dense-only deepseek_v41 prompts only" 0 COLI_VK_CHAIN=2 SNAP=dsv41_long ./deepseek_v41 8 dsv41_long/ref.json
+ dsan41 "asan dense-only deepseek_v41 device lost" 1 COLI_VK_CHAIN_FAULT=40 SNAP=dsv41_long ./deepseek_v41 8 dsv41_long/ref.json
+ CHAIN_SERVE_EXPECT='dense matrices on the device only' COLI_VK_DENSE_HOST=0 \
+ $PY tests/vulkan_chain_serve.py ./deepseek_v41 dsv41_tiny V41_DSPARK=1 > san.log 2>&1 || { cat san.log; fail "asan dense-only deepseek_v41 serve"; }
+ echo "OK asan dense-only deepseek_v41 serve: $(tail -1 san.log | cut -c1-120)"
+ v4_chain_fixtures
+ local p; p=$(v4_chain_prompt deepseek_v4_tiny_t long)
+ v4_chain_san "asan dense-only deepseek_v4 long" COLI_VK_DENSE_HOST=0 ./deepseek_v4 ./deepseek_v4_tiny_t "$p" --raw-prompt --max-tokens 4 --record-oracle san.json
+ dho_v4_check "asan dense-only deepseek_v4 long" san.log 0
+ v4_chain_san "asan dense-only deepseek_v4 2 output groups, prompts only" COLI_VK_DENSE_HOST=0 COLI_VK_CHAIN=2 ./deepseek_v4 ./deepseek_v4_tiny_g2 "$(v4_chain_prompt deepseek_v4_tiny_g2 long)" --raw-prompt --max-tokens 4
+ dho_v4_check "asan dense-only deepseek_v4 2 output groups, prompts only" san.log 0
+ FAULT_BACK=10 v4_chain_san "asan dense-only deepseek_v4 device lost" COLI_VK_DENSE_HOST=0 ./deepseek_v4 ./deepseek_v4_tiny_t "$p" --raw-prompt --max-tokens 4
+ dho_v4_check "asan dense-only deepseek_v4 device lost" san.log 1
+ v4_chain_san "asan dense-only deepseek_v4 a low-RAM plan" COLI_VK_DENSE_HOST=0 ./deepseek_v4 ./deepseek_v4_tiny_e8 "$(v4_chain_prompt deepseek_v4_tiny_e8 long)" --raw-prompt --max-tokens 4 --memory-gb $DHO_V4_LOW
+ dho_v4_check "asan dense-only deepseek_v4 a low-RAM plan" san.log 0
+ COLI_VK_DENSE_HOST=0 v4_chain_serve deepseek_v4_tiny_t > san.log 2>&1 || { cat san.log; fail "asan dense-only deepseek_v4 served"; }
+ echo "OK asan dense-only deepseek_v4 served: $(tail -1 san.log | cut -c1-120)"
+ rm -rf deepseek_v4_tiny_t deepseek_v4_tiny_e8 deepseek_v4_tiny_g2 deepseek_v4_tiny_w4 deepseek_v4_tiny_ix san.json san.log chain.usage chain-serve.usage
+ unset OMP_NUM_THREADS
+ make deepseek-v4-clean >/dev/null 2>&1 || true
+ make clean >/dev/null 2>&1 || true
+}
diff --git a/c/tests/vulkan_dense_only_glm.sh b/c/tests/vulkan_dense_only_glm.sh
new file mode 100644
index 00000000..4b8cd758
--- /dev/null
+++ b/c/tests/vulkan_dense_only_glm.sh
@@ -0,0 +1,258 @@
+# Sourced by tests/vulkan_engines.sh: colibri (GLM-5.2) and glm53 (GLM-5.3 Flash) with
+# their dense weights on the device only (COLI_VK_DENSE_HOST=0, docs/vulkan.md "Dense
+# weights on the device only"). The fixtures and the token lines are the glm-chain
+# family's (glm_chain_fixtures, mla_toks): every expert and trunk format, prefill in
+# chunks, the DSA selection, n-gram and MTP drafts, the tier off, prompts only and the
+# per-matrix path alone (the steps the chain does not run take the device: the CPU has no
+# copy), the device lost (the matrices read back from the checkpoint, glm53's KDA state
+# rebuilt), serve sessions frame for frame (two KV slots, prompts only), glm53's image and
+# pin-branch harness, staged uploads, the RAM the experts' cache gets back, and the
+# resident set at exit on a fixture whose head is large enough to see.
+
+# dho_mla_gate <engine> <tag> <tol 0|1> <env...> -- <argv...>: the CPU, the device keeping
+# its host copies (COLI_VK_DENSE_HOST=1) and the device only (=0), the tier deterministic
+# and each run from a fresh history. Gates: the CPU's tokens in both device runs; matrices
+# on the device only and none read back; chain forwards (CHAINMODE, default 1; 0 = the
+# per-matrix path alone with COLI_VK_DENSE=1); tol 1: the device-only logits within 1e-4
+# of the CPU's largest (logits_close).
+dho_mla_gate() {
+ local eng=$1 tag=$2 tol=$3; shift 3
+ local envs=(); while [ "$1" != "--" ]; do envs+=("$1"); shift; done; shift
+ local mode=${CHAINMODE:-1}
+ local vk=(USAGE_SAVE=0 COLI_USAGE=chain.usage COLI_VK_TIER_SYNC=1 COLI_VK_TIER_BALANCE=0 COLI_VULKAN=1 COLI_VK_CHAIN=$mode)
+ [ "$mode" = 0 ] && vk+=(COLI_VK_DENSE=1)
+ rm -f chain.usage cpu.f32 vk.f32
+ env "${envs[@]}" USAGE_SAVE=0 COLI_USAGE=chain.usage DUMP=cpu.f32 ./"$eng" "$@" > cpu.log 2>&1 || true
+ rm -f chain.usage
+ env "${envs[@]}" "${vk[@]}" COLI_VK_DENSE_HOST=1 ./"$eng" "$@" > host.log 2>&1 || true
+ rm -f chain.usage
+ env "${envs[@]}" "${vk[@]}" COLI_VK_DENSE_HOST=0 DUMP=vk.f32 ./"$eng" "$@" > vk.log 2>&1 || true
+ rm -f chain.usage
+ mla_toks "$eng" cpu.log > cpu.tok; mla_toks "$eng" host.log > host.tok; mla_toks "$eng" vk.log > vk.tok
+ { [ -s cpu.tok ] && cmp -s cpu.tok vk.tok; } || { cat cpu.tok vk.tok; tail -20 vk.log; fail "$tag: the device-only run's tokens differ from the CPU"; }
+ cmp -s cpu.tok host.tok || { cat cpu.tok host.tok; tail -20 host.log; fail "$tag: the host-copy run's tokens differ from the CPU"; }
+ local n; n=$(dho_dropped vk.log)
+ [ "$n" -gt 0 ] || { cat vk.log; fail "$tag: no dense matrix went to the device only"; }
+ [ "$(dho_reloaded vk.log)" = 0 ] || { grep -a '^\[VK\]' vk.log; fail "$tag: a matrix was read back from disk without a lost device"; }
+ if [ "$mode" != 0 ]; then [ "$(chain_count "$eng" vk.log)" -gt 0 ] || { cat vk.log; fail "$tag: the chain did not stay on"; }; fi
+ local lg=""
+ if [ "$tol" = 1 ]; then lg=$(logits_close cpu.f32 vk.f32) || { echo "$lg"; fail "$tag: logits"; }; lg=", $lg"; fi
+ echo "OK $tag: tokens = CPU$lg, $n matrices on the device only, $(chain_count "$eng" vk.log) chain forwards"
+}
+# dho_mla_lost <engine> <tag> <frames back> <env...> -- <argv...>: device only, the device
+# lost `back` frames before the end of a run (COLI_VK_CHAIN_FAULT, counted on a fault-free
+# run of the same configuration): the CPU's tokens, the loss handled, matrices read back
+# from disk for the CPU; REBUILD=1: glm53's KDA state rebuilt on the CPU.
+dho_mla_lost() {
+ local eng=$1 tag=$2 back=$3; shift 3
+ local envs=(); while [ "$1" != "--" ]; do envs+=("$1"); shift; done; shift
+ local vk=(USAGE_SAVE=0 COLI_USAGE=chain.usage COLI_VK_TIER_SYNC=1 COLI_VULKAN=1 COLI_VK_CHAIN=1 COLI_VK_DENSE_HOST=0)
+ rm -f chain.usage
+ env "${envs[@]}" USAGE_SAVE=0 COLI_USAGE=chain.usage ./"$eng" "$@" > cpu.log 2>&1 || true
+ rm -f chain.usage
+ env "${envs[@]}" "${vk[@]}" ./"$eng" "$@" > vk.log 2>&1 || true
+ local f; f=$(sed -n "s/^\[VK\] $eng chain: [0-9]* forwards, \([0-9]*\) frames.*/\1/p" vk.log | tail -1)
+ { [ -n "$f" ] && [ "$f" -gt "$back" ]; } || { cat vk.log; fail "$tag: no fault-free run to count frames from"; }
+ rm -f chain.usage
+ env "${envs[@]}" "${vk[@]}" COLI_VK_CHAIN_FAULT=$((f - back + 1)) ./"$eng" "$@" > vk.log 2>&1 || true
+ rm -f chain.usage
+ mla_toks "$eng" cpu.log > cpu.tok; mla_toks "$eng" vk.log > vk.tok
+ { [ -s cpu.tok ] && cmp -s cpu.tok vk.tok; } || { cat cpu.tok vk.tok; tail -20 vk.log; fail "$tag: the tokens after the loss differ from the CPU"; }
+ grep -q "$eng chain: the device was lost" vk.log || { cat vk.log; fail "$tag: no loss was handled"; }
+ if [ "${REBUILD:-0}" = 1 ]; then grep -q "rebuilding the state of [1-9]" vk.log || { cat vk.log; fail "$tag: no state was rebuilt"; }; fi
+ local r; r=$(dho_reloaded vk.log)
+ [ "$r" -gt 0 ] || { grep -a '^\[VK\]' vk.log; fail "$tag: nothing was read back for the CPU"; }
+ echo "OK $tag: tokens = CPU after the loss, $r matrices read back from disk$(grep -o 'rebuilding the state of [0-9]* positions' vk.log | sed 's/^/, /')"
+}
+# dho_widen <src> <dst> <vocab>: a copy of a fixture whose embedding and head have `vocab`
+# rows (the rows past the original vocabulary repeat the first ones): a head of tens of MiB
+# whose host copy shows in the resident set; the prompts keep their ids, so the runs are
+# the original's model with a wider (never chosen) vocabulary tail.
+dho_widen() {
+ $PY - "$1" "$2" "$3" <<'PY'
+import json, shutil, struct, sys
+from pathlib import Path
+src, dst, vocab = Path(sys.argv[1]), Path(sys.argv[2]), int(sys.argv[3])
+shutil.rmtree(dst, ignore_errors=True); shutil.copytree(src, dst)
+cfg = json.loads((dst / "config.json").read_text())
+(cfg["text_config"] if "text_config" in cfg else cfg)["vocab_size"] = vocab
+(dst / "config.json").write_text(json.dumps(cfg))
+for shard in sorted(dst.glob("*.safetensors")):
+ raw = shard.read_bytes(); n = struct.unpack("<Q", raw[:8])[0]
+ head = json.loads(raw[8:8 + n]); body = raw[8 + n:]
+ meta = head.pop("__metadata__", None)
+ out, off, blobs = {}, 0, []
+ for name, t in sorted(head.items(), key=lambda kv: kv[1]["data_offsets"][0]):
+ a, b = t["data_offsets"]; data = body[a:b]; shape = list(t["shape"])
+ if (name.endswith("embed_tokens.weight") or name == "lm_head.weight") and t["dtype"] == "F32":
+ row = len(data) // shape[0]
+ data = b"".join(data[(r % shape[0]) * row:(r % shape[0] + 1) * row] for r in range(vocab))
+ shape[0] = vocab
+ out[name] = {"dtype": t["dtype"], "shape": shape, "data_offsets": [off, off + len(data)]}
+ blobs.append(data); off += len(data)
+ if meta is not None: out["__metadata__"] = meta
+ h = json.dumps(out).encode(); h += b" " * (-len(h) % 8)
+ shard.write_bytes(struct.pack("<Q", len(h)) + h + b"".join(blobs))
+PY
+}
+
+dho_family_glm() {
+ make colibri glm53 VK=1
+ glm_chain_fixtures
+ export OMP_NUM_THREADS=2 CAP_RAISE=0
+ local cap b fx d
+ # colibri: the f32 trunk and experts, 8/4/3-bit trunks (IDOT=0: the device multiplies f32
+ # activations), the expert containers, prefill, the DSA selection, drafts
+ for cap in 1 64; do dho_mla_gate colibri "dense-only colibri f32 cap=$cap" 1 SNAP=glm_tiny REF=ref_glm.json -- $cap 16 16; done
+ dho_mla_gate colibri "dense-only colibri f32 prefill, chunks of 3" 1 SNAP=glm_tiny REF=ref_glm.json TF=1 COLI_VK_CHAIN_ROWS=3 -- 64 16 16
+ for b in 8 4 3; do
+ dho_mla_gate colibri "dense-only colibri ${b}-bit trunk and experts" 1 SNAP=glm_tiny REF=ref_glm.json IDOT=0 -- 2 $b $b
+ done
+ dho_mla_gate colibri "dense-only colibri int4-g64 experts" 1 SNAP=glm_tiny_fmt4 REF=glm_tiny_fmt4/ref_glm.json -- 2 16 16
+ dho_mla_gate colibri "dense-only colibri E8/IQ3 experts on the CPU" 1 SNAP=glm_tiny_fmt6 REF=glm_tiny_fmt6/ref_glm.json -- 2 16 16
+ for fx in i4 i4r i3 d3; do
+ dho_mla_gate colibri "dense-only colibri $fx container" 1 SNAP=glm_tiny_$fx REF=glm_tiny_$fx/ref_glm.json IDOT=0 -- 1 4 4
+ done
+ dho_mla_gate colibri "dense-only colibri DSA top-4" 1 SNAP=glm_tiny REF=ref_glm.json DSA_TOPK=4 -- 64 16 16
+ dho_mla_gate colibri "dense-only colibri DSA top-4, prefill in chunks of 5" 1 SNAP=glm_tiny REF=ref_glm.json DSA_TOPK=4 TF=1 COLI_VK_CHAIN_ROWS=5 -- 64 16 16
+ dho_mla_gate colibri "dense-only colibri n-gram drafts" 1 SNAP=glm_tiny REF=ref_glm.json DRAFT=3 -- 64 16 16
+ # the MTP head runs per matrix: its layer, eh_proj and lm_head take the device here
+ for d in 1 3; do dho_mla_gate colibri "dense-only colibri MTP depth $d" 1 SNAP=glm_tiny_mtp REF=glm_tiny_mtp/ref_glm.json DRAFT=$d -- 64 16 16; done
+ dho_mla_gate colibri "dense-only colibri MTP, 4-bit" 1 SNAP=glm_tiny_mtp REF=glm_tiny_mtp/ref_glm.json IDOT=0 DRAFT=2 -- 2 4 4
+ dho_mla_gate colibri "dense-only colibri tier off" 1 SNAP=glm_tiny REF=ref_glm.json COLI_VK_TIER=0 -- 64 16 16
+ dho_mla_gate colibri "dense-only colibri beside COLI_VK_ATTN=1" 1 SNAP=glm_tiny_i4 REF=glm_tiny_i4/ref_glm.json IDOT=0 COLI_VK_ATTN=1 -- 2 4 4
+ # prompts only: decode runs per matrix on the device; kv_b and the indexer stay on the
+ # host (the CPU's attention reads them by row)
+ CHAINMODE=2 dho_mla_gate colibri "dense-only colibri prompts only, drafts" 1 SNAP=glm_tiny REF=ref_glm.json DRAFT=3 -- 64 16 16
+ CHAINMODE=0 dho_mla_gate colibri "dense-only colibri per-matrix path, absorb core" 1 SNAP=glm_tiny_i4 REF=glm_tiny_i4/ref_glm.json IDOT=0 COLI_VK_ATTN=1 -- 2 4 4
+ CHAINMODE=0 dho_mla_gate colibri "dense-only colibri per-matrix path, MTP" 1 SNAP=glm_tiny_mtp REF=glm_tiny_mtp/ref_glm.json DRAFT=2 -- 64 16 16
+ # staged uploads (a card without Resizable BAR): the drop happens after the copy returned
+ COLI_VK_STAGED=1 dho_mla_gate colibri "dense-only colibri staged" 1 SNAP=glm_tiny REF=ref_glm.json DSA_TOPK=4 -- 64 16 16
+ grep -q '^\[VK\] memory: staged uploads' vk.log || { grep '^\[VK\]' vk.log; fail "dense-only colibri staged: not staged"; }
+ # the device lost: mid-decode, in the prompt (one teacher-forced forward), with MTP drafts
+ dho_mla_lost colibri "dense-only colibri device lost mid-decode" 12 SNAP=glm_tiny REF=ref_glm.json -- 64 16 16
+ dho_mla_lost colibri "dense-only colibri device lost in the prompt" 3 SNAP=glm_tiny REF=ref_glm.json TF=1 -- 64 16 16
+ dho_mla_lost colibri "dense-only colibri device lost with MTP" 12 SNAP=glm_tiny_mtp REF=glm_tiny_mtp/ref_glm.json -- 64 16 16
+ COLI_VK_STAGED=1 dho_mla_lost colibri "dense-only colibri device lost, staged" 12 SNAP=glm_tiny REF=ref_glm.json -- 64 16 16
+ # serve sessions: pins, the prompt cache, the prefill read-out, drafts, two KV slots (the
+ # ragged batch declines the chain: per matrix on the device, kv_b on the host)
+ export COLI_VK_DENSE_HOST=0 CHAIN_SERVE_EXPECT='dense matrices on the device only'
+ CHAIN_SERVE_DIALECT=colibri $PY tests/vulkan_chain_serve.py ./colibri glm_tiny_serve SERVE_BATCH=1 IDOT=0
+ CHAIN_SERVE_DIALECT=colibri $PY tests/vulkan_chain_serve.py ./colibri glm_tiny_serve SERVE_BATCH=1 IDOT=0 DSA_TOPK=4 DRAFT=3
+ CHAIN_SERVE_SLOTS=2 CHAIN_SERVE_DIALECT=colibri $PY tests/vulkan_chain_serve.py ./colibri glm_tiny_serve SERVE_BATCH=1 IDOT=0 KV_SLOTS=2
+ COLI_VK_CHAIN=2 CHAIN_SERVE_DIALECT=colibri $PY tests/vulkan_chain_serve.py ./colibri glm_tiny_serve SERVE_BATCH=1 IDOT=0 DSA_TOPK=4
+ unset COLI_VK_DENSE_HOST CHAIN_SERVE_EXPECT
+ # the RAM the experts' cache gets back: RAM_GB just above the fixture's fixed reserve, the
+ # cap cap_for_ram leaves with the host copies against the one it leaves without them
+ local c1 c0 h
+ for h in 1 0; do
+ rm -f chain.usage
+ env RAM_GB=3.7145 SNAP=glm_tiny REF=ref_glm.json USAGE_SAVE=0 COLI_USAGE=chain.usage COLI_VK_TIER_SYNC=1 COLI_VULKAN=1 \
+ COLI_VK_CHAIN=1 COLI_VK_DENSE_HOST=$h ./colibri 64 16 16 > vk$h.log 2>&1 || true
+ done
+ rm -f chain.usage
+ c1=$(sed -n 's/.*cap lowered 64->\([0-9]*\).*/\1/p' vk1.log | tail -1); c0=$(sed -n 's/.*cap lowered 64->\([0-9]*\).*/\1/p' vk0.log | tail -1)
+ { [ -n "$c1" ] && [ -n "$c0" ] && [ "$c0" -gt "$c1" ]; } || { grep -a 'RAM_GB\|resident dense' vk1.log vk0.log; fail "dense-only colibri RAM_GB: the cap did not grow"; }
+ mla_toks colibri vk1.log > host.tok; mla_toks colibri vk0.log > vk.tok
+ cmp -s host.tok vk.tok || fail "dense-only colibri RAM_GB: the tokens differ between the two caps"
+ echo "OK dense-only colibri RAM_GB=3.7145: cap $c1 with the host copies, $c0 on the device only ($(grep -ao 'resident dense: [0-9.]* MB' vk1.log) -> $(grep -ao '[0-9.]* MB' <<< "$(grep -ao 'resident dense: [0-9.]* MB' vk0.log)"))"
+ # the resident set at exit: a 65536-row f32 head (32 MiB) on the device only
+ dho_widen glm_tiny glm_tiny_wide 65536
+ dho_rss_gate colibri "dense-only colibri resident set" 24 SNAP=glm_tiny_wide REF=ref_glm.json USAGE_SAVE=0 -- 64 16 16
+ # unset, on a CPU device the host copies stay (the auto rule), and the line says why
+ rm -f chain.usage
+ env SNAP=glm_tiny REF=ref_glm.json USAGE_SAVE=0 COLI_USAGE=chain.usage COLI_VULKAN=1 COLI_VK_CHAIN=1 ./colibri 64 16 16 > vk.log 2>&1 || true
+ rm -f chain.usage
+ grep -q '^\[VK\] colibri: dense weights on the device and in host RAM (a CPU device' vk.log || { grep '^\[VK\]' vk.log; fail "colibri: the auto rule dropped the host copies on a CPU device"; }
+ [ "$(dho_dropped vk.log)" = 0 ] || fail "colibri: matrices dropped without COLI_VK_DENSE_HOST=0 on a CPU device"
+ echo "OK dense-only colibri auto on a CPU device: host copies kept"
+
+ # glm53: f32, int8 and int4 trunks, streamed and resident experts, prefill and chain
+ # chunks, the tier off, an image, swiglu_limit 0, prompts only, the per-matrix path alone
+ local ids bits
+ ids=$($PY -c "print(','.join(str((i*37+5)%120+2) for i in range(100)))")
+ dho_mla_gate glm53 "dense-only glm53 decode" 1 GLM53_BITS=32 -- --model glm53_stream-i4 --ids 5,7,9,11,13,17,19,23 --greedy 8
+ for bits in 32 8 4; do
+ dho_mla_gate glm53 "dense-only glm53 100-token prompt, ${bits}-bit trunk" 1 GLM53_BITS=$bits -- --model glm53_stream-i4 --ids $ids --greedy 4
+ done
+ dho_mla_gate glm53 "dense-only glm53 prefill chunks of 7, one cache slot" 1 GLM53_BITS=4 GLM53_PREFILL_CHUNK=7 GLM53_EXPERT_GB=0.000001 -- --model glm53_stream-i4 --ids $ids --greedy 4
+ dho_mla_gate glm53 "dense-only glm53 chain chunks of 3" 1 GLM53_BITS=32 COLI_VK_CHAIN_ROWS=3 -- --model glm53_stream-i4 --ids $ids --greedy 4
+ dho_mla_gate glm53 "dense-only glm53 resident experts" 1 GLM53_BITS=32 -- --model glm53_tiny --ids $ids --greedy 6
+ dho_mla_gate glm53 "dense-only glm53 tier off" 1 GLM53_BITS=32 COLI_VK_TIER=0 -- --model glm53_stream-i4 --ids $ids --greedy 6
+ dho_mla_gate glm53 "dense-only glm53 an image" 1 GLM53_BITS=32 -- --model glm53_mm_tiny --ids 103,117,268,268,268,268,120,121 --patches glm53_mm_tiny/patches.f32 --grid 4x4 --greedy 4
+ dho_mla_gate glm53 "dense-only glm53 swiglu_limit 0" 1 GLM53_BITS=32 -- --model glm53_lim0 --ids $ids --greedy 4
+ CHAINMODE=2 dho_mla_gate glm53 "dense-only glm53 prompts only" 1 GLM53_BITS=32 GLM53_PREFILL_CHUNK=16 -- --model glm53_stream-i4 --ids $ids --greedy 6
+ CHAINMODE=0 dho_mla_gate glm53 "dense-only glm53 per-matrix path, 4-bit trunk" 1 GLM53_BITS=4 -- --model glm53_stream-i4 --ids $ids --greedy 6
+ COLI_VK_STAGED=1 dho_mla_gate glm53 "dense-only glm53 staged" 1 GLM53_BITS=4 -- --model glm53_stream-i4 --ids $ids --greedy 4
+ grep -q '^\[VK\] memory: staged uploads' vk.log || { grep '^\[VK\]' vk.log; fail "dense-only glm53 staged: not staged"; }
+ REBUILD=1 dho_mla_lost glm53 "dense-only glm53 device lost, the KDA state rebuilt" 7 GLM53_BITS=32 GLM53_PREFILL_CHUNK=16 -- --model glm53_stream-i4 --ids $ids --greedy 6
+ REBUILD=1 dho_mla_lost glm53 "dense-only glm53 device lost, 4-bit trunk" 7 GLM53_BITS=4 GLM53_PREFILL_CHUNK=16 -- --model glm53_stream-i4 --ids $ids --greedy 6
+ REBUILD=1 dho_mla_lost glm53 "dense-only glm53 device lost after an image" 5 GLM53_BITS=32 -- --model glm53_mm_tiny --ids 103,117,268,268,268,268,120,121 --patches glm53_mm_tiny/patches.f32 --grid 4x4 --greedy 4
+ export COLI_VK_DENSE_HOST=0 CHAIN_SERVE_EXPECT='dense matrices on the device only'
+ CHAIN_SERVE_DIALECT=numeric $PY tests/vulkan_chain_serve.py ./glm53 glm53_serve GLM53_BITS=32
+ CHAIN_SERVE_DIALECT=numeric $PY tests/vulkan_chain_serve.py ./glm53 glm53_serve GLM53_BITS=4 COLI_VK_CHAIN_ROWS=3
+ CHAIN_SERVE_SLOTS=2 CHAIN_SERVE_DIALECT=numeric $PY tests/vulkan_chain_serve.py ./glm53 glm53_serve GLM53_BITS=32 KV_SLOTS=2
+ COLI_VK_CHAIN=2 CHAIN_SERVE_DIALECT=numeric $PY tests/vulkan_chain_serve.py ./glm53 glm53_serve GLM53_BITS=32
+ unset CHAIN_SERVE_EXPECT
+ COLI_VULKAN=1 COLI_VK_CHAIN=1 COLI_USAGE=$PWD/chain.usage $PY tests/glm53_pin_branch_harness.py --binary ./glm53 --fixture glm53_mm_tiny
+ unset COLI_VK_DENSE_HOST
+ # the expert cache sizes itself from the free memory after the host copies went (on a
+ # device that shares the RAM the device's copy takes the same RAM: no gain to show there;
+ # on a discrete GPU the RAM given back is free memory the cache measures)
+ rm -f chain.usage
+ env GLM53_VERBOSE=1 GLM53_BITS=32 USAGE_SAVE=0 COLI_USAGE=chain.usage COLI_VK_TIER_SYNC=1 COLI_VULKAN=1 COLI_VK_CHAIN=1 \
+ COLI_VK_DENSE_HOST=0 ./glm53 --model glm53_stream-i4 --ids 5,7,9 --greedy 2 > vk.log 2>&1 || true
+ rm -f chain.usage
+ local given budget
+ given=$(grep -n 'dense matrices on the device only' vk.log | head -1 | cut -d: -f1); budget=$(grep -n '^expert budget:' vk.log | head -1 | cut -d: -f1)
+ { [ -n "$given" ] && [ -n "$budget" ] && [ "$given" -lt "$budget" ]; } || { cat vk.log; fail "dense-only glm53: the expert cache sized itself before the host copies went"; }
+ echo "OK dense-only glm53: the expert cache sized after the host copies went ($(grep -a '^expert budget:' vk.log | head -1))"
+ # The log rounds RSS to 0.01 GiB. A 64 MiB head leaves enough separation
+ # for that rounding and allocator bookkeeping without weakening the gate.
+ dho_widen glm53_stream-i4 glm53_wide 131072
+ dho_rss_gate glm53 "dense-only glm53 resident set" 48 GLM53_BITS=32 USAGE_SAVE=0 -- --model glm53_wide --ids 5,7,9,11,13 --greedy 4
+ unset OMP_NUM_THREADS CAP_RAISE
+}
+
+# The same under ASan and UBSan: memory safety is the gate, with the host copies dropped,
+# the read-back after a lost device, the CPU's attention reading kv_b back (prompts only),
+# drafts, an image and serve sessions.
+dho_family_glm_sanitize() {
+ make clean >/dev/null 2>&1 || true
+ make colibri glm53 VK=1 EXTRA_CFLAGS="-fsanitize=address,undefined -fno-omit-frame-pointer -g"
+ export ASAN_OPTIONS=detect_leaks=0:detect_stack_use_after_return=0 UBSAN_OPTIONS=print_stacktrace=1
+ glm_chain_fixtures
+ export OMP_NUM_THREADS=2 CAP_RAISE=0
+ dsan() { # <engine> <tag> <lost 0|1> <env and argv...>
+ local eng=$1 tag=$2 lost=$3; shift 3
+ rm -f chain.usage
+ env COLI_USAGE=chain.usage USAGE_SAVE=0 COLI_VK_TIER_SYNC=1 COLI_VULKAN=1 COLI_VK_CHAIN=${CHAINMODE:-1} COLI_VK_DENSE_HOST=0 "$@" > san.log 2>&1 || true
+ rm -f chain.usage
+ if grep -qE "ERROR: AddressSanitizer|runtime error:" san.log; then cat san.log; fail "$tag: sanitizer diagnostic"; fi
+ [ "$(dho_dropped san.log)" -gt 0 ] || { cat san.log; fail "$tag: nothing on the device only"; }
+ if [ "$lost" = 1 ]; then [ "$(dho_reloaded san.log)" -gt 0 ] || { cat san.log; fail "$tag: nothing read back"; }; fi
+ echo "OK $tag: sanitizers clean, $(dho_dropped san.log) matrices on the device only, $(dho_reloaded san.log) read back"
+ }
+ local ids
+ ids=$($PY -c "print(','.join(str((i*37+5)%120+2) for i in range(100)))")
+ dsan colibri "asan dense-only colibri f32, DSA top-4 prefill" 0 SNAP=glm_tiny REF=ref_glm.json TF=1 DSA_TOPK=4 COLI_VK_CHAIN_ROWS=3 ./colibri 64 16 16
+ dsan colibri "asan dense-only colibri i4 container" 0 SNAP=glm_tiny_i4 REF=glm_tiny_i4/ref_glm.json IDOT=0 ./colibri 2 4 4
+ dsan colibri "asan dense-only colibri MTP depth 2" 0 SNAP=glm_tiny_mtp REF=glm_tiny_mtp/ref_glm.json DRAFT=2 ./colibri 64 16 16
+ CHAINMODE=2 dsan colibri "asan dense-only colibri prompts only" 0 SNAP=glm_tiny REF=ref_glm.json DRAFT=3 ./colibri 64 16 16
+ dsan colibri "asan dense-only colibri device lost" 1 SNAP=glm_tiny REF=ref_glm.json COLI_VK_CHAIN_FAULT=30 ./colibri 64 16 16
+ dsan glm53 "asan dense-only glm53 int4 trunk, chain chunks of 3" 0 GLM53_BITS=4 COLI_VK_CHAIN_ROWS=3 ./glm53 --model glm53_stream-i4 --ids $ids --greedy 4
+ dsan glm53 "asan dense-only glm53 an image" 0 GLM53_BITS=32 ./glm53 --model glm53_mm_tiny --ids 103,117,268,268,268,268,120,121 --patches glm53_mm_tiny/patches.f32 --grid 4x4 --greedy 4
+ CHAINMODE=2 dsan glm53 "asan dense-only glm53 prompts only" 0 GLM53_BITS=32 GLM53_PREFILL_CHUNK=16 ./glm53 --model glm53_stream-i4 --ids $ids --greedy 6
+ dsan glm53 "asan dense-only glm53 device lost, rebuilt" 1 GLM53_BITS=32 GLM53_PREFILL_CHUNK=16 COLI_VK_CHAIN_FAULT=20 ./glm53 --model glm53_stream-i4 --ids $ids --greedy 6
+ local args
+ for args in "./colibri glm_tiny_serve SERVE_BATCH=1 IDOT=0 KV_SLOTS=2" "./glm53 glm53_serve GLM53_BITS=32 KV_SLOTS=2"; do
+ local dialect=colibri; [ "${args%% *}" = ./glm53 ] && dialect=numeric
+ # shellcheck disable=SC2086
+ COLI_VK_DENSE_HOST=0 CHAIN_SERVE_EXPECT='dense matrices on the device only' CHAIN_SERVE_SLOTS=2 CHAIN_SERVE_DIALECT=$dialect \
+ $PY tests/vulkan_chain_serve.py $args > san.log 2>&1 || { cat san.log; fail "asan dense-only serve ${args%% *}"; }
+ if grep -qE "ERROR: AddressSanitizer|runtime error:" san.log; then cat san.log; fail "asan dense-only serve ${args%% *}: sanitizer diagnostic"; fi
+ echo "OK asan dense-only serve ${args%% *}: $(tail -1 san.log)"
+ done
+ unset OMP_NUM_THREADS CAP_RAISE
+ make clean >/dev/null 2>&1 || true
+}
diff --git a/c/tests/vulkan_dense_only_inkling-mimo-kimi.sh b/c/tests/vulkan_dense_only_inkling-mimo-kimi.sh
new file mode 100644
index 00000000..449d559d
--- /dev/null
+++ b/c/tests/vulkan_dense_only_inkling-mimo-kimi.sh
@@ -0,0 +1,292 @@
+# Sourced by tests/vulkan_engines.sh: inkling, MiMo and Kimi K3 with their dense matrices
+# on the device only (COLI_VK_DENSE_HOST=0, docs/vulkan.md "Dense weights on the device
+# only"), run as `bash tests/vulkan_engines.sh dense-only-inkling-mimo-kimi` (and
+# `...-sanitize`). Each configuration against the CPU's tokens and against the run that
+# keeps its host copies (the same device copies: the logits are its bytes); a lost device
+# reads the matrices back from disk and the CPU's tokens follow; serve sessions frame for
+# frame; staged uploads; the resident set and the expert cache where the engine sizes it.
+
+# mimo_dho_gate <tag> <case> <env...>: MiMo's CLI (mimo_tiny, --ids, its text on stdout),
+# on the CPU, keeping the host copies and on the device only; the text the CPU's, the
+# logits of the greedy stream (MIMO_LOGITS) the host-copy run's bytes, or with prompts
+# only (CHAINMODE=2: decode on the per-matrix path, the device's on the device only and
+# the CPU's with the host copies on this device) within 1e-4 of the CPU's.
+mimo_dho_gate() {
+ local tag=$1 c=$2 extra=() h n; shift 2
+ [ "$c" = image ] && extra=(--image mimo_tiny/patches.f32 --grid $MGRID)
+ local vk=(COLI_TEMP=0 COLI_VULKAN=1 COLI_VK_CHAIN=${CHAINMODE:-1} COLI_VK_TIER_SYNC=1 COLI_VK_TIER_BALANCE=0 COLI_VK_TIER_STREAM_ROWS=16)
+ # A dropped trunk also enables per-matrix GPU work, including the vision tower.
+ # Keep that arithmetic identical in both arms of this byte-exact comparison.
+ if [ "${CHAINMODE:-1}" = 0 ] || [ "$c" = image ]; then vk+=(COLI_VK_DENSE=1); fi
+ env "$@" COLI_TEMP=0 ./mimo mimo_tiny --ids "$(mimo_ids $c prompt_ids)" --ngen 6 "${extra[@]}" > mimo-cpu.txt 2>/dev/null
+ for h in 1 0; do
+ env "$@" "${vk[@]}" COLI_VK_DENSE_HOST=$h ./mimo mimo_tiny --ids "$(mimo_ids $c prompt_ids)" --ngen 6 "${extra[@]}" \
+ > mimo-vk$h.txt 2> mimo-vk$h.err
+ { [ -s mimo-cpu.txt ] && cmp -s mimo-cpu.txt mimo-vk$h.txt; } || { cat mimo-cpu.txt mimo-vk$h.txt mimo-vk$h.err; fail "$tag $c: tokens differ from the CPU (COLI_VK_DENSE_HOST=$h)"; }
+ rm -f host$h.f32
+ env "$@" "${vk[@]}" COLI_VK_DENSE_HOST=$h MIMO_LOGITS=host$h.f32 \
+ ./mimo mimo_tiny --ids "$(mimo_ids $c greedy_full_ids)" --ngen 0 "${extra[@]}" > /dev/null 2>> mimo-vk$h.err
+ done
+ local lg="logits = the host-copy run's"
+ if [ "${CHAINMODE:-1}" = 2 ]; then
+ rm -f cpu.f32
+ env "$@" MIMO_LOGITS=cpu.f32 ./mimo mimo_tiny --ids "$(mimo_ids $c greedy_full_ids)" --ngen 0 "${extra[@]}" > /dev/null 2>&1
+ lg=$(logits_close cpu.f32 host0.f32) || { echo "$lg"; fail "$tag $c: logits"; }
+ else cmp -s host1.f32 host0.f32 || fail "$tag $c: the logits differ from the run that keeps its host copies"; fi
+ n=$(dho_dropped mimo-vk0.err)
+ [ "$n" -gt 0 ] || { cat mimo-vk0.err; fail "$tag $c: no dense matrix went to the device only"; }
+ [ "$(dho_reloaded mimo-vk0.err)" = 0 ] || { grep '^\[VK\]' mimo-vk0.err; fail "$tag $c: a matrix was read back without a lost device"; }
+ if [ "${CHAINMODE:-1}" != 0 ]; then [ "$(chain_count mimo mimo-vk0.err)" -gt 0 ] || { cat mimo-vk0.err; fail "$tag $c: the chain did not stay on"; }; fi
+ echo "OK $tag $c: tokens = CPU, $lg, $n matrices on the device only"
+}
+# mimo_dho_lost <tag> <case> <back> <env...>: the device lost <back> frames before the end
+mimo_dho_lost() {
+ local tag=$1 c=$2 back=$3 extra=() frames r; shift 3
+ [ "$c" = image ] && extra=(--image mimo_tiny/patches.f32 --grid $MGRID)
+ local vk=(COLI_TEMP=0 COLI_VULKAN=1 COLI_VK_CHAIN=1 COLI_VK_TIER_SYNC=1 COLI_VK_DENSE_HOST=0)
+ env "$@" COLI_TEMP=0 ./mimo mimo_tiny --ids "$(mimo_ids $c prompt_ids)" --ngen 6 "${extra[@]}" > mimo-cpu.txt 2>/dev/null
+ env "$@" "${vk[@]}" ./mimo mimo_tiny --ids "$(mimo_ids $c prompt_ids)" --ngen 6 "${extra[@]}" > /dev/null 2> mimo-vk.err
+ frames=$(sed -n 's/^\[VK\] mimo chain: [0-9]* forwards, \([0-9]*\) frames.*/\1/p' mimo-vk.err | tail -1)
+ [ "${frames:-0}" -gt "$back" ] || { cat mimo-vk.err; fail "$tag: the run had no frame $back before its end"; }
+ env "$@" "${vk[@]}" COLI_VK_CHAIN_FAULT=$((frames - back)) ./mimo mimo_tiny --ids "$(mimo_ids $c prompt_ids)" --ngen 6 "${extra[@]}" \
+ > mimo-vk.txt 2> mimo-vk.err
+ { [ -s mimo-cpu.txt ] && cmp -s mimo-cpu.txt mimo-vk.txt; } || { cat mimo-cpu.txt mimo-vk.txt mimo-vk.err; fail "$tag: tokens differ from the CPU"; }
+ grep -q "the device was lost at position [0-9]" mimo-vk.err || { cat mimo-vk.err; fail "$tag: the device was never lost"; }
+ r=$(dho_reloaded mimo-vk.err)
+ [ "$r" -gt 0 ] || { grep '^\[VK\]' mimo-vk.err; fail "$tag: nothing was read back for the CPU"; }
+ echo "OK $tag: tokens = CPU after the loss, $r matrices read back from disk"
+}
+
+# k3_dho_gate <tag> <env...>: Kimi K3's CLI on the fixture's cases (K3C_CASES), on the CPU,
+# keeping the host copies and on the device only: the CPU's tokens, the logits within the
+# chain family's bound of the CPU's (k3c_close); the decode logits are the host-copy
+# run's bytes. The earlier prompt rows are echoed by the CPU head with host copies,
+# and by the device head without them: those retain the CPU-reference numeric gate.
+# The last prompt row and every generated step use the chain head in both runs.
+# RELOADS=n: the
+# matrices a run without a lost device must read back (the CPU's MLA path reads kv_b's rows
+# where the chain does not run: prompts only, the per-matrix path); BYTES=0: the logits held
+# to the CPU's bound only (a forward the chain does not run multiplies on the device here
+# and on the CPU with the host copies kept); FAULT_BACK=k: the device lost k frames before
+# the end, the matrices read back and the state rebuilt.
+k3_dho_gate() {
+ local tag=$1 c frames fault h n r; shift
+ local vk=(COLI_VK_TIER_SYNC=1 COLI_VK_TIER_BALANCE=0 COLI_VK_TIER_STREAM_ROWS=16 COLI_VULKAN=1 COLI_VK_CHAIN=${CHAINMODE:-1})
+ [ "${CHAINMODE:-1}" = 0 ] && vk+=(COLI_VK_DENSE=1)
+ for c in ${K3C_CASES:-short long}; do
+ rm -f k3c.usage cpu.f32 host1.f32 host0.f32
+ env "$@" COLI_USAGE=$PWD/k3c.usage USAGE_SAVE=0 K3_VAL_LOGITS=cpu.f32 ./kimi_k3 kimi_k3_tiny --ids "$(k3c_ids $c)" --ngen 8 \
+ 2> cpu.log | sed 's/ *TUNE.*//' > cpu.tok
+ fault=""
+ if [ -n "${FAULT_BACK:-}" ]; then
+ env "$@" "${vk[@]}" COLI_VK_DENSE_HOST=0 COLI_USAGE=$PWD/k3c.usage USAGE_SAVE=0 \
+ ./kimi_k3 kimi_k3_tiny --ids "$(k3c_ids $c)" --ngen 8 2> vk.log > /dev/null
+ frames=$(sed -n 's/^\[VK\] kimi_k3 chain: [0-9]* forwards, \([0-9]*\) frames.*/\1/p' vk.log | tail -1)
+ [ -n "$frames" ] && [ "$frames" -gt "$FAULT_BACK" ] || { cat vk.log; fail "$tag $c: no fault-free run to count frames from"; }
+ fault="COLI_VK_CHAIN_FAULT=$((frames - FAULT_BACK + 1))"
+ fi
+ for h in 1 0; do
+ [ -n "$fault" ] && [ $h = 1 ] && continue
+ rm -f k3c.usage
+ env "$@" "${vk[@]}" $fault COLI_VK_DENSE_HOST=$h COLI_USAGE=$PWD/k3c.usage USAGE_SAVE=0 K3_VAL_LOGITS=host$h.f32 \
+ ./kimi_k3 kimi_k3_tiny --ids "$(k3c_ids $c)" --ngen 8 2> vk$h.log | sed 's/ *TUNE.*//' > vk$h.tok
+ { [ -s cpu.tok ] && cmp -s cpu.tok vk$h.tok; } || { cat cpu.tok vk$h.tok; tail -20 vk$h.log; fail "$tag $c: tokens differ from the CPU (COLI_VK_DENSE_HOST=$h)"; }
+ done
+ n=$(dho_dropped vk0.log); r=$(dho_reloaded vk0.log)
+ [ "$n" -gt 0 ] || { cat vk0.log; fail "$tag $c: no dense matrix went to the device only"; }
+ local lg; lg=$(k3c_close cpu.f32 host0.f32) || { echo "$lg"; fail "$tag $c: logits"; }
+ if [ -n "$fault" ]; then
+ grep -q "kimi_k3 chain: the device was lost" vk0.log || { cat vk0.log; fail "$tag $c: no loss was handled"; }
+ [ "$r" -gt 0 ] || { grep '^\[VK\]' vk0.log; fail "$tag $c: nothing was read back for the CPU"; }
+ echo "OK $tag $c: tokens = CPU after the loss, $lg, $r matrices read back from disk"
+ continue
+ fi
+ [ "$r" = "${RELOADS:-0}" ] || { grep '^\[VK\]' vk0.log; fail "$tag $c: $r matrices read back from disk, expected ${RELOADS:-0}"; }
+ if [ "${BYTES:-1}" = 1 ]; then
+ $PY - <<'PY' || fail "$tag $c: decode logits differ from the run that keeps its host copies"
+import json
+from pathlib import Path
+vocab = json.loads(Path("kimi_k3_tiny/config.json").read_text())["vocab_size"]
+steps = len(Path("vk0.tok").read_text().split())
+kept, dropped = (Path(f"host{h}.f32").read_bytes() for h in (1, 0))
+assert steps > 0 and len(kept) == len(dropped) and len(kept) % (4 * vocab) == 0
+size = steps * vocab * 4
+assert len(kept) >= size and kept[-size:] == dropped[-size:]
+PY
+ lg="$lg, decode logits = the host-copy run's"
+ fi
+ if [ "${CHAIN_EXPECT:-${CHAINMODE:-1}}" != 0 ]; then
+ [ "$(chain_count kimi_k3 vk0.log)" -gt 0 ] || { cat vk0.log; fail "$tag $c: the chain did not stay on"; }
+ else
+ [ "$(chain_count kimi_k3 vk0.log)" = 0 ] || fail "$tag $c: the chain ran despite the CPU fallback"
+ need_gpu kimi_k3 vk0.log "$tag $c per-matrix fallback"
+ fi
+ echo "OK $tag $c: tokens = CPU, $lg, $n matrices on the device only, $r read back"
+ done
+}
+# The expert cache Kimi K3 sizes from RAM_GB counts the dropped host copies out: a budget
+# set (from a probe run's own figures) between the plan with the host copies (cap 1) and
+# the plan without them, and the cap the run takes is the larger.
+k3_dho_cap() {
+ local O="K3_BITS=32 K3_MLA_BITS=32 K3_HEAD_BITS=32 K3_IDOT=0 COLI_TEMP=0" line ram a b off
+ rm -f k3c.usage
+ # shellcheck disable=SC2086
+ env $O COLI_USAGE=$PWD/k3c.usage USAGE_SAVE=0 COLI_VULKAN=1 COLI_VK_CHAIN=1 COLI_VK_DENSE_HOST=0 \
+ ./kimi_k3 kimi_k3_tiny --ids "$(k3c_ids short)" --ngen 2 > /dev/null 2> vk.log
+ line=$(grep '^\[K3\] dense weights on the device only:' vk.log) || { cat vk.log; fail "kimi_k3 cap: no plan line"; }
+ ram=$($PY - "$line" <<'PY'
+import re, sys
+d, r, s = (float(x) for x in re.search(r"only: ([0-9.]+) GB of host copies out of the resident ([0-9.]+) GB \(reserve ([0-9.]+) GB\)", sys.argv[1]).groups())
+print(f"{r + s - d / 2:.6f}")
+PY
+)
+ rm -f k3c.usage
+ # shellcheck disable=SC2086
+ env $O RAM_GB=$ram COLI_USAGE=$PWD/k3c.usage USAGE_SAVE=0 COLI_VULKAN=1 COLI_VK_CHAIN=1 COLI_VK_DENSE_HOST=0 \
+ ./kimi_k3 kimi_k3_tiny --ids "$(k3c_ids short)" --ngen 2 > /dev/null 2> vk.log
+ line=$(grep '^\[K3\] dense weights on the device only:' vk.log) || { cat vk.log; fail "kimi_k3 cap: no plan line under RAM_GB=$ram"; }
+ a=$(echo "$line" | sed -n 's/.*expert cache \([0-9]*\)\/layer instead of \([0-9]*\)$/\1/p')
+ b=$(echo "$line" | sed -n 's/.*expert cache \([0-9]*\)\/layer instead of \([0-9]*\)$/\2/p')
+ { [ -n "$a" ] && [ -n "$b" ] && [ "$a" -gt "$b" ]; } || { echo "$line"; fail "kimi_k3 cap: the cache did not grow with the host copies out"; }
+ grep -q "expert cache $a/layer" vk.log || { grep '\[K3\]' vk.log; fail "kimi_k3 cap: the run did not take the larger cache"; }
+ rm -f k3c.usage
+ # shellcheck disable=SC2086
+ env $O RAM_GB=$ram COLI_USAGE=$PWD/k3c.usage USAGE_SAVE=0 COLI_VULKAN=1 COLI_VK_CHAIN=1 COLI_VK_DENSE_HOST=1 \
+ ./kimi_k3 kimi_k3_tiny --ids "$(k3c_ids short)" --ngen 2 > /dev/null 2> vk.log
+ off=$(sed -n 's/.*expert cache \([0-9]*\)\/layer (.*/\1/p' vk.log | tail -1)
+ [ "$off" = "$b" ] || { grep '\[K3\]' vk.log; fail "kimi_k3 cap: with the host copies kept the cache is $off, the plan said $b"; }
+ rm -f k3c.usage
+ echo "OK dense-only kimi_k3 expert cache under RAM_GB=$ram: $a/layer on the device only, $b/layer with the host copies"
+}
+
+dho_family_inkling_mimo_kimi() {
+ export OMP_NUM_THREADS=2
+ make inkling mimo kimi_k3 VK=1
+ # ---- inkling: f32, the dense-int4g64 container (int8 and int4-g64), bf16, the expert
+ # containers and runtime quantization, the tier off, chunks, prompts only, the
+ # per-matrix path alone, D = 6144, staged, a lost device, a serve session
+ inkling_olmoe_chain_fixtures
+ local fx bits c R=tiny_inkling/ref_inkling.json W=tiny_inkling_wide/ref_inkling.json
+ for fx in tiny_inkling tiny_inkling_q tiny_inkling_bf16 tiny_inkling_x-i4 tiny_inkling_x-i8; do
+ dho_gate inkling "dense-only inkling $fx" 2 SNAP=$fx -- 8 0 $R
+ done
+ dho_gate inkling "dense-only inkling runtime int8 experts" 2 SNAP=tiny_inkling -- 2 8 $R
+ dho_gate inkling "dense-only inkling tier off" 2 COLI_VK_TIER=0 SNAP=tiny_inkling_q -- 8 0 $R
+ dho_gate inkling "dense-only inkling prefill in chunks of 3" 2 COLI_VK_CHAIN_ROWS=3 SNAP=tiny_inkling -- 8 0 $R
+ # prompts only: decode on the per-matrix path, the device's here and the CPU's with the
+ # host copies kept on this device, so its logits are held to the CPU's within 1e-4
+ CHAINMODE=2 dho_gate inkling "dense-only inkling prompts only" 1 SNAP=tiny_inkling -- 8 0 $R
+ CHAINMODE=0 dho_gate inkling "dense-only inkling per-matrix path" 2 SNAP=tiny_inkling_q -- 8 0 $R
+ dho_gate inkling "dense-only inkling D=6144" 2 SNAP=tiny_inkling_wide -- 8 0 $W
+ dho_gate inkling "dense-only inkling staged" 2 COLI_VK_STAGED=1 SNAP=tiny_inkling_q -- 8 0 $R
+ dho_lost inkling "dense-only inkling device lost" 20 SNAP=tiny_inkling -- 8 0 $R
+ dho_lost inkling "dense-only inkling device lost, int4-g64 trunk" 19 SNAP=tiny_inkling_q -- 8 0 $R
+ CHAIN_SERVE_EXPECT='dense matrices on the device only' COLI_VK_DENSE_HOST=0 \
+ $PY tests/vulkan_chain_serve.py ./inkling tiny_inkling INK_PREFIX_LOG=1
+ # the resident set at exit: the host copies of D = 6144's trunk are gone
+ dho_rss_gate inkling "dense-only inkling D=6144 resident set" 80 SNAP=tiny_inkling_wide -- 8 0 $W
+ # the auto cap measures RAM after the drop (here a CPU device: the device copy holds it)
+ rm -f chain.usage
+ env COLI_USAGE=chain.usage COLI_VULKAN=1 COLI_VK_CHAIN=1 COLI_VK_DENSE_HOST=0 SNAP=tiny_inkling_wide ./inkling 0 0 $W > vk.log 2>&1 || true
+ grep -q '^\[cap auto\] dense weights on the device only: RAM measured after 0\.[0-9]*[1-9][0-9]* GB of host copies went' vk.log ||
+ { grep 'cap auto\|^\[VK\]' vk.log; fail "dense-only inkling auto cap: not measured after the drop"; }
+ echo "OK dense-only inkling auto cap: $(grep -o 'RAM measured after.*' vk.log)"
+ rm -f chain.usage
+
+ # ---- mimo: every dense form, text and picture, the tower on the device, the tier off,
+ # chunks, prompts only, the per-matrix path alone, staged, the vendor oracle, lost
+ # devices, a serve session
+ $PY tools/make_mimo_tiny.py --output ./mimo_tiny --force --vision
+ MGRID=$($PY -c "import json;i=json.load(open('mimo_tiny/ref.json'))['image'];print(i['grid_h'],i['grid_w'])")
+ COLI_VULKAN=1 COLI_VK_CHAIN=1 COLI_VK_TIER_SYNC=1 COLI_VK_DENSE_HOST=0 $PY tests/mimo_tiny_harness.py --binary ./mimo --fixture ./mimo_tiny
+ echo "OK dense-only mimo vendor oracle"
+ for bits in 32 0 8; do for c in short long image; do
+ mimo_dho_gate "dense-only mimo bits=$bits" $c MIMO_DENSE_BITS=$bits
+ done; done
+ mimo_dho_gate "dense-only mimo tower on the device" image MIMO_DENSE_BITS=0 COLI_VK_DENSE=1
+ mimo_dho_gate "dense-only mimo tier off" long MIMO_DENSE_BITS=0 COLI_VK_TIER=0
+ mimo_dho_gate "dense-only mimo chunks of 3 rows" long MIMO_DENSE_BITS=32 COLI_VK_CHAIN_ROWS=3
+ mimo_dho_gate "dense-only mimo staged" image MIMO_DENSE_BITS=0 COLI_VK_STAGED=1
+ CHAINMODE=2 mimo_dho_gate "dense-only mimo prompts only" long MIMO_DENSE_BITS=0
+ CHAINMODE=0 mimo_dho_gate "dense-only mimo per-matrix path" image MIMO_DENSE_BITS=0
+ mimo_dho_lost "dense-only mimo device lost mid-decode" long 3 MIMO_DENSE_BITS=0
+ mimo_dho_lost "dense-only mimo device lost in a picture's prompt" image 33 MIMO_DENSE_BITS=8
+ mimo_served_fixture
+ CHAIN_SERVE_TOL=1e-3 CHAIN_SERVE_EXPECT='dense matrices on the device only' COLI_VK_DENSE_HOST=0 \
+ $PY tests/vulkan_chain_serve.py ./mimo mimo_tiny_served OMP_NUM_THREADS=2
+
+ # ---- kimi_k3: Moonshot's oracle, every dense format, the tier off, chunks, prompts
+ # only and the per-matrix path alone (kv_b read back for the CPU's MLA path), a trace
+ # (the chain declines), staged, lost devices, serve sessions, the expert cache
+ $PY tools/make_kimi_k3_tiny.py --output ./kimi_k3_tiny --force
+ k3c_serve_fixture
+ COLI_USAGE=$PWD/k3c.usage USAGE_SAVE=0 COLI_VULKAN=1 COLI_VK_CHAIN=1 COLI_VK_TIER_SYNC=1 COLI_VK_DENSE_HOST=0 \
+ $PY tests/test_kimi_k3_tiny.py --binary ./kimi_k3 --fixture ./kimi_k3_tiny
+ echo "OK dense-only kimi_k3 vendor oracle"
+ local O="K3_BITS=32 K3_MLA_BITS=32 K3_HEAD_BITS=32 K3_IDOT=0 COLI_TEMP=0" b
+ # shellcheck disable=SC2086
+ {
+ k3_dho_gate "dense-only kimi_k3 f32" $O
+ for b in 8 4; do k3_dho_gate "dense-only kimi_k3 ${b}-bit trunk" K3_BITS=$b K3_MLA_BITS=$b K3_HEAD_BITS=$b K3_IDOT=0 COLI_TEMP=0; done
+ k3_dho_gate "dense-only kimi_k3 int4 KDA and MoE, int8 MLA and head" K3_BITS=4 K3_IDOT=0 COLI_TEMP=0
+ k3_dho_gate "dense-only kimi_k3 tier off" $O COLI_VK_TIER=0
+ k3_dho_gate "dense-only kimi_k3 chain chunks of 3" $O COLI_VK_CHAIN_ROWS=3
+ k3_dho_gate "dense-only kimi_k3 staged" K3_BITS=4 K3_IDOT=0 COLI_TEMP=0 COLI_VK_STAGED=1
+ CHAINMODE=2 RELOADS=4 BYTES=0 k3_dho_gate "dense-only kimi_k3 prompts only" $O
+ CHAINMODE=0 RELOADS=4 BYTES=0 k3_dho_gate "dense-only kimi_k3 per-matrix path" $O
+ CHAIN_EXPECT=0 RELOADS=4 BYTES=0 k3_dho_gate "dense-only kimi_k3 K3_TRACE (the chain declines)" $O K3_TRACE=/dev/null
+ FAULT_BACK=3 k3_dho_gate "dense-only kimi_k3 device lost mid-decode" $O
+ K3C_CASES=long FAULT_BACK=77 k3_dho_gate "dense-only kimi_k3 device lost in a later prompt chunk" K3_BITS=4 K3_IDOT=0 COLI_TEMP=0 COLI_VK_CHAIN_ROWS=7
+ }
+ local S="K3_BITS=32 K3_MLA_BITS=32 K3_HEAD_BITS=32 K3_IDOT=0 K3_PREFIX_LOG=1 USAGE_SAVE=0"
+ # shellcheck disable=SC2086
+ {
+ CHAIN_SERVE_TOL=2e-2 CHAIN_SERVE_EXPECT='dense matrices on the device only' COLI_VK_DENSE_HOST=0 \
+ $PY tests/vulkan_chain_serve.py ./kimi_k3 kimi_k3_serve $S COLI_K3_CKPT=4
+ CHAIN_SERVE_TOL=2e-2 CHAIN_SERVE_EXPECT='dense matrices on the device only' COLI_VK_DENSE_HOST=0 COLI_VK_CHAIN=2 \
+ $PY tests/vulkan_chain_serve.py ./kimi_k3 kimi_k3_serve $S
+ }
+ k3_dho_cap
+ rm -f k3c.usage chain.usage chain-serve.usage
+ unset OMP_NUM_THREADS
+}
+
+# The same under ASan and UBSan: memory safety is the gate, each run with its host copies
+# dropped (and, where the device is lost, read back).
+dho_family_inkling_mimo_kimi_sanitize() {
+ make clean >/dev/null 2>&1 || true
+ make inkling mimo kimi_k3 VK=1 EXTRA_CFLAGS="-fsanitize=address,undefined -fno-omit-frame-pointer -g"
+ export ASAN_OPTIONS=detect_leaks=0:detect_stack_use_after_return=0 UBSAN_OPTIONS=print_stacktrace=1 OMP_NUM_THREADS=2
+ $PY tools/make_tiny_inkling.py tiny_inkling
+ $PY tools/make_mimo_tiny.py --output ./mimo_tiny --force --vision
+ MGRID=$($PY -c "import json;i=json.load(open('mimo_tiny/ref.json'))['image'];print(i['grid_h'],i['grid_w'])")
+ $PY tools/make_kimi_k3_tiny.py --output ./kimi_k3_tiny --force
+ k3c_serve_fixture
+ dsan_run() { # <tag> <lost 0|1> <env and argv...>
+ local tag=$1 lost=$2; shift 2
+ rm -f chain.usage
+ env COLI_USAGE=chain.usage USAGE_SAVE=0 COLI_VK_TIER_SYNC=1 COLI_VULKAN=1 COLI_VK_CHAIN=1 COLI_VK_DENSE_HOST=0 "$@" > san.log 2>&1 || true
+ if grep -qE "ERROR: AddressSanitizer|runtime error:" san.log; then cat san.log; fail "asan $tag: sanitizer diagnostic"; fi
+ [ "$(dho_dropped san.log)" -gt 0 ] || { cat san.log; fail "asan $tag: nothing on the device only"; }
+ if [ "$lost" = 1 ]; then [ "$(dho_reloaded san.log)" -gt 0 ] || { cat san.log; fail "asan $tag: nothing read back"; }; fi
+ echo "OK asan $tag: sanitizers clean, $(dho_dropped san.log) matrices on the device only, $(dho_reloaded san.log) read back"
+ }
+ local R=tiny_inkling/ref_inkling.json
+ dsan_run "dense-only inkling f32" 0 SNAP=tiny_inkling ./inkling 8 0 $R
+ dsan_run "dense-only inkling per-matrix path" 0 COLI_VK_CHAIN=0 COLI_VK_DENSE=1 SNAP=tiny_inkling ./inkling 8 0 $R
+ dsan_run "dense-only inkling device lost" 1 COLI_VK_CHAIN_FAULT=150 SNAP=tiny_inkling ./inkling 8 0 $R
+ dsan_run "dense-only mimo picture" 0 MIMO_DENSE_BITS=0 COLI_VK_DENSE=1 ./mimo mimo_tiny --ids "$(mimo_ids image prompt_ids)" --ngen 4 --image mimo_tiny/patches.f32 --grid $MGRID
+ dsan_run "dense-only mimo device lost" 1 MIMO_DENSE_BITS=0 COLI_VK_CHAIN_FAULT=20 ./mimo mimo_tiny --ids "$(mimo_ids long prompt_ids)" --ngen 4
+ local O=(K3_BITS=4 K3_IDOT=0 COLI_TEMP=0)
+ dsan_run "dense-only kimi_k3 int4" 0 "${O[@]}" ./kimi_k3 kimi_k3_tiny --ids "$(k3c_ids long)" --ngen 4
+ dsan_run "dense-only kimi_k3 prompts only" 0 "${O[@]}" COLI_VK_CHAIN=2 ./kimi_k3 kimi_k3_tiny --ids "$(k3c_ids long)" --ngen 4
+ dsan_run "dense-only kimi_k3 device lost" 1 "${O[@]}" COLI_VK_CHAIN_FAULT=40 ./kimi_k3 kimi_k3_tiny --ids "$(k3c_ids long)" --ngen 4
+ CHAIN_SERVE_TOL=2e-2 CHAIN_SERVE_EXPECT='dense matrices on the device only' COLI_VK_DENSE_HOST=0 \
+ $PY tests/vulkan_chain_serve.py ./kimi_k3 kimi_k3_serve K3_BITS=32 K3_MLA_BITS=32 K3_HEAD_BITS=32 K3_IDOT=0 USAGE_SAVE=0 > san.log 2>&1 ||
+ { cat san.log; fail "asan dense-only kimi_k3 serve"; }
+ if grep -qE "ERROR: AddressSanitizer|runtime error:" san.log; then cat san.log; fail "asan dense-only kimi_k3 serve: sanitizer diagnostic"; fi
+ echo "OK asan dense-only kimi_k3 serve: $(tail -1 san.log)"
+ rm -f chain.usage chain-serve.usage
+ unset OMP_NUM_THREADS
+ make clean >/dev/null 2>&1 || true
+}
diff --git a/c/tests/vulkan_dense_only_qwen36-olmoe.sh b/c/tests/vulkan_dense_only_qwen36-olmoe.sh
new file mode 100644
index 00000000..2c07f969
--- /dev/null
+++ b/c/tests/vulkan_dense_only_qwen36-olmoe.sh
@@ -0,0 +1,184 @@
+# Sourced by tests/vulkan_engines.sh: qwen36 (Qwen3.6, Qwen3-Coder, Qwen3.8-27B dense,
+# Clef) and olmoe with their dense matrices on the device only (COLI_VK_DENSE_HOST=0,
+# docs/vulkan.md "Dense weights on the device only").
+#
+# qwen36: f32, int8 and int4-g64 dense rows, int4-g64 experts, the Qwen3-Coder and the
+# dense 27B geometries, prefill in chunks, the tier off, prompts only and the per-matrix
+# path alone (the steps the chain does not run take the device too: the CPU has no copy),
+# an image, the device lost (the DeltaNet state rebuilt on the CPU with every matrix read
+# back from disk), a serve session frame for frame, and Clef's oracle.
+# olmoe: caps 1 and 8, PILOT, the device lost (the step redone on the CPU from matrices
+# read back), a serve session, an automatic cache that takes the RAM the dense weights
+# held, and the resident set falling by what was dropped (a wide fixture).
+
+# A wide OLMoE: a 32768-row lm_head of 512 columns (64 MiB of f32) and small experts, so
+# the dense weights are most of the process and an expert slot is small beside them.
+dho_olmoe_wide_fixture() {
+ [ -f olmoe_wide_c/config.json ] && return 0
+ rm -rf olmoe_wide olmoe_wide_c
+ $PY - <<'PY'
+import json, torch
+from pathlib import Path
+from transformers import OlmoeConfig, OlmoeForCausalLM
+torch.manual_seed(20261004); torch.set_num_threads(2)
+cfg = OlmoeConfig(vocab_size=32768, hidden_size=512, intermediate_size=64, num_hidden_layers=2,
+ num_attention_heads=8, num_key_value_heads=8, max_position_embeddings=128,
+ num_experts=64, num_experts_per_tok=2, norm_topk_prob=True,
+ eos_token_id=None, pad_token_id=None, tie_word_embeddings=False)
+model = OlmoeForCausalLM(cfg).eval().float()
+prompt = [3, 11, 29, 7, 41, 19]
+with torch.no_grad():
+ full = model.generate(torch.tensor([prompt]), max_new_tokens=6, do_sample=False, use_cache=True)[0].tolist()
+out = Path("olmoe_wide"); out.mkdir()
+model.save_pretrained(out, safe_serialization=True)
+(out / "ref_olmoe.json").write_text(json.dumps({"prompt_ids": prompt, "full_ids": full}) + "\n")
+PY
+ $PY tools/convert_olmoe_merged.py --model olmoe_wide --out olmoe_wide_c > /dev/null
+}
+
+dho_qwen36_olmoe_fixtures() {
+ $PY tools/make_qwen36_tiny.py --out qwen36_tiny --ref-mode full --emit-ref qwen36_tiny/ref_full.json
+ $PY tools/convert_qwen36.py --model qwen36_tiny --out qwen36_tiny_c --ebits 8
+ $PY -c 'import sys; from pathlib import Path; sys.path.insert(0, "tests"); from prefix_serve_harness import ensure_byte_tokenizer; ensure_byte_tokenizer(Path("qwen36_tiny_c"))' # serve speaks text
+ $PY tools/make_qwen36_tiny.py --geometry qwen3-coder-30b --out qwen3_coder_tiny --ref-mode full --emit-ref qwen3_coder_tiny/ref_full.json
+ $PY tools/convert_qwen36.py --model qwen3_coder_tiny --out qwen3_coder_tiny_c --ebits 8
+ $PY tools/make_qwen36_tiny.py --geometry qwen38-27b-dense --out qwen38_27b_tiny --ref-mode full --emit-ref qwen38_27b_tiny/ref_full.json
+ $PY tools/convert_qwen36.py --model qwen38_27b_tiny --out qwen38_27b_tiny_c --ebits 8
+ $PY tools/make_qwen36_tiny.py --out qwen36_tiny64 --ref-mode full --inter 64 --emit-ref qwen36_tiny64/ref_full.json
+ $PY tools/convert_qwen36.py --model qwen36_tiny64 --out qwen36_tiny64_c --ebits 4 --gs 64
+ $PY tools/make_qwen36_vl_tiny.py --out qwen38_27b_vl_tiny
+ $PY tools/convert_qwen36.py --model qwen38_27b_vl_tiny --out qwen38_27b_vl_tiny_c --ebits 8
+ $PY tools/make_olmoe_tiny.py --output olmoe_tiny --force
+ $PY tools/convert_olmoe_merged.py --model olmoe_tiny --out olmoe_tiny_c
+ $PY tools/make_edge_tiny_tokenizer.py --vocab-size 128 olmoe_tiny_c
+ dho_olmoe_wide_fixture
+}
+
+# dho_olmoe_cap_gate: olmoe's automatic cache (cap 0) under a RAM_GB that leaves the
+# experts almost nothing beside the dense weights: with the host copies kept it stays
+# below every expert of a layer, on the device only it takes them all (the 64 MiB the
+# dense weights held hold them many times over).
+dho_olmoe_cap_gate() {
+ local R=olmoe_wide/ref_olmoe.json
+ SNAP=olmoe_wide_c ./olmoe 8 8 $R > cpu.log 2>&1 || true
+ local rss kv ram
+ rss=$(sed -n 's/.*RSS after load: \([0-9.]*\) GB.*/\1/p' cpu.log | tail -1)
+ [ -n "$rss" ] || { cat cpu.log; fail "olmoe automatic cache: no RSS after load"; }
+ kv=$($PY -c 'import json; c=json.load(open("olmoe_wide_c/config.json")); print(2*c["num_hidden_layers"]*c["num_attention_heads"]*4096*(c["hidden_size"]//c["num_attention_heads"])*4/1e9)')
+ ram=$($PY -c "print(f'{$rss + $kv + 0.5:.4f}')")
+ local vk=(COLI_VK_TIER_SYNC=1 COLI_VULKAN=1 COLI_VK_CHAIN=1 RAM_GB=$ram SNAP=olmoe_wide_c)
+ rm -f chain.usage
+ env "${vk[@]}" COLI_USAGE=chain.usage COLI_VK_DENSE_HOST=1 ./olmoe 0 8 $R > host.log 2>&1 || true
+ rm -f chain.usage
+ env "${vk[@]}" COLI_USAGE=chain.usage COLI_VK_DENSE_HOST=0 ./olmoe 0 8 $R > vk.log 2>&1 || true
+ rm -f chain.usage
+ same_tokens cpu.log vk.log "olmoe automatic cache"
+ local a b
+ a=$(sed -n 's/^\[cache\] \([0-9]*\) slots\/layer of .*/\1/p' host.log | tail -1)
+ b=$(sed -n 's/^\[cache\] \([0-9]*\) slots\/layer of .*/\1/p' vk.log | tail -1)
+ { [ -n "$a" ] && [ -n "$b" ] && [ "$b" -gt "$a" ]; } || { grep '\[cache\]' host.log vk.log; fail "olmoe automatic cache: the cache did not grow on the device only"; }
+ grep -q '^\[cache\] .* (was [0-9]*): the dense weights on the device only gave' vk.log || { grep '\[cache\]' vk.log; fail "olmoe automatic cache: no line for the growth"; }
+ echo "OK dense-only olmoe automatic cache, RAM_GB=$ram: $a slots/layer with the host copies, $b on the device only"
+}
+
+dho_family_qwen36_olmoe() {
+ export OMP_NUM_THREADS=2
+ make qwen36 olmoe VK=1
+ dho_qwen36_olmoe_fixtures
+ local fx R=qwen36_tiny/ref_full.json
+ # qwen36: every geometry with f32 dense rows, then int8 rows (fmt 1; the CPU arm with
+ # f32 activations, as the device) and the int4-g64 copy (COLI_DENSE_BITS=4, fmt 4)
+ for fx in qwen36_tiny qwen3_coder_tiny qwen38_27b_tiny; do
+ dho_gate qwen36 "dense-only qwen36 $fx f32" 2 COLI_DENSE_I8=0 SNAP=${fx}_c -- 8 8 $fx/ref_full.json
+ CPUENV=COLI_DENSE_IDOT=0 dho_gate qwen36 "dense-only qwen36 $fx int8" 2 SNAP=${fx}_c -- 8 8 $fx/ref_full.json
+ done
+ dho_gate qwen36 "dense-only qwen36 int4-g64 dense" 2 COLI_DENSE_BITS=4 SNAP=qwen36_tiny64_c -- 8 4 qwen36_tiny64/ref_full.json
+ dho_gate qwen36 "dense-only qwen36 int4-g64 experts" 2 QWEN_EXPERT_ACT=f32 COLI_DENSE_I8=0 SNAP=qwen36_tiny64_c -- 8 4 qwen36_tiny64/ref_full.json
+ dho_gate qwen36 "dense-only qwen36 cap 1" 2 COLI_DENSE_I8=0 SNAP=qwen36_tiny_c -- 1 8 $R
+ dho_gate qwen36 "dense-only qwen36 prefill in chunks of 3" 2 COLI_VK_CHAIN_ROWS=3 COLI_DENSE_I8=0 SNAP=qwen36_tiny_c -- 8 8 $R
+ dho_gate qwen36 "dense-only qwen36 tier off" 2 COLI_VK_TIER=0 COLI_DENSE_I8=0 SNAP=qwen36_tiny_c -- 8 8 $R
+ # prompts on the chain, decode on the per-matrix path (the device's in this mode, the
+ # CPU's with the host copies kept on this device): the CPU's logits
+ CHAINMODE=2 dho_gate qwen36 "dense-only qwen36 prompts only" 1 COLI_DENSE_I8=0 SNAP=qwen36_tiny_c -- 8 8 $R
+ CHAINMODE=0 dho_gate qwen36 "dense-only qwen36 per-matrix path" 2 COLI_DENSE_I8=0 SNAP=qwen36_tiny_c -- 8 8 $R
+ # PILOT keeps the chain off: every matrix per matrix on the device
+ CHAINMODE=0 dho_gate qwen36 "dense-only qwen36 PILOT" 2 PILOT=1 COLI_DENSE_I8=0 SNAP=qwen36_tiny_c -- 8 8 $R
+ dho_gate qwen36 "dense-only qwen36 vision" 1 COLI_DENSE_I8=0 SNAP=qwen38_27b_vl_tiny_c -- 8 8 qwen38_27b_vl_tiny/ref.json
+ # f16 acquired a Vulkan form with the decision models. Import requested alongside
+ # host removal must copy first: the device cannot retain pages that the engine frees.
+ dho_gate qwen36 "dense-only qwen36 f16 dense" 2 COLI_DENSE_BITS=16 COLI_VK_IMPORT=1 SNAP=qwen36_tiny_c -- 8 8 $R
+ dho_gate qwen36 "dense-only qwen36 int8 import requested" 2 COLI_DENSE_IDOT=0 COLI_VK_IMPORT=1 SNAP=qwen36_tiny_c -- 8 8 $R
+ dho_lost qwen36 "dense-only qwen36 f16 device lost" 20 COLI_DENSE_BITS=16 COLI_VK_IMPORT=1 SNAP=qwen36_tiny_c -- 8 8 $R
+ dho_lost qwen36 "dense-only qwen36 device lost" 20 COLI_DENSE_I8=0 SNAP=qwen36_tiny_c -- 8 8 $R
+ dho_lost qwen36 "dense-only qwen36 int8 device lost" 20 COLI_DENSE_IDOT=0 SNAP=qwen36_tiny_c -- 8 8 $R
+ CHAIN_SERVE_EXPECT='dense matrices on the device only' COLI_VK_DENSE_HOST=0 \
+ $PY tests/vulkan_chain_serve.py ./qwen36 qwen36_tiny_c COLI_DENSE_I8=0
+ # Clef's oracle (the decision head on the CPU, the backbone on the device only)
+ COLI_VULKAN=1 COLI_VK_CHAIN=1 COLI_VK_DENSE_HOST=0 COLI_VK_TIER_SYNC=1 COLI_USAGE=$PWD/chain.usage \
+ make clef-tiny-check VK=1
+ rm -f chain.usage chain-serve.usage
+
+ # olmoe
+ local OR=olmoe_tiny/ref_olmoe.json cap
+ for cap in 1 8; do dho_gate olmoe "dense-only olmoe cap=$cap" 2 SNAP=olmoe_tiny_c -- $cap 8 $OR; done
+ dho_gate olmoe "dense-only olmoe PILOT" 2 PILOT=1 WIDE=2 SNAP=olmoe_tiny_c -- 1 8 $OR
+ dho_gate olmoe "dense-only olmoe prefill in chunks of 3" 2 COLI_VK_CHAIN_ROWS=3 SNAP=olmoe_tiny_c -- 8 8 $OR
+ CHAINMODE=2 dho_gate olmoe "dense-only olmoe prompts only" 1 SNAP=olmoe_tiny_c -- 8 8 $OR
+ CHAINMODE=0 dho_gate olmoe "dense-only olmoe per-matrix path" 2 SNAP=olmoe_tiny_c -- 8 8 $OR
+ dho_lost olmoe "dense-only olmoe device lost" 10 SNAP=olmoe_tiny_c -- 8 8 $OR
+ CHAIN_SERVE_EXPECT='dense matrices on the device only' COLI_VK_DENSE_HOST=0 \
+ $PY tests/vulkan_chain_serve.py ./olmoe olmoe_tiny_c
+ dho_gate olmoe "dense-only olmoe wide" 2 SNAP=olmoe_wide_c -- 8 8 olmoe_wide/ref_olmoe.json
+ dho_olmoe_cap_gate
+ # the RSS: on Lavapipe the device copies are this process's memory too, so the
+ # resident set falls by the host copies alone (64 MiB of lm_head, 4 MiB of layers)
+ dho_rss_gate olmoe "dense-only olmoe RSS" 48 SNAP=olmoe_wide_c -- 8 8 olmoe_wide/ref_olmoe.json
+ rm -f chain.usage chain-serve.usage
+ unset OMP_NUM_THREADS
+}
+
+# The same under ASan and UBSan: memory safety is the gate, with the host copies dropped,
+# a lost device's read-back, PILOT, the grown automatic cache and a serve session each.
+dho_family_qwen36_olmoe_sanitize() {
+ make clean >/dev/null 2>&1 || true
+ make qwen36 olmoe VK=1 EXTRA_CFLAGS="-fsanitize=address,undefined -fno-omit-frame-pointer -g"
+ export ASAN_OPTIONS=detect_leaks=0:detect_stack_use_after_return=0 UBSAN_OPTIONS=print_stacktrace=1 OMP_NUM_THREADS=2
+ $PY tools/make_qwen36_tiny.py --out qwen36_tiny --ref-mode full --emit-ref qwen36_tiny/ref_full.json
+ $PY tools/convert_qwen36.py --model qwen36_tiny --out qwen36_tiny_c --ebits 8
+ $PY -c 'import sys; from pathlib import Path; sys.path.insert(0, "tests"); from prefix_serve_harness import ensure_byte_tokenizer; ensure_byte_tokenizer(Path("qwen36_tiny_c"))'
+ $PY tools/make_qwen36_tiny.py --out qwen36_tiny64 --ref-mode full --inter 64 --emit-ref qwen36_tiny64/ref_full.json
+ $PY tools/convert_qwen36.py --model qwen36_tiny64 --out qwen36_tiny64_c --ebits 4 --gs 64
+ $PY tools/make_olmoe_tiny.py --output olmoe_tiny --force
+ $PY tools/convert_olmoe_merged.py --model olmoe_tiny --out olmoe_tiny_c
+ $PY tools/make_edge_tiny_tokenizer.py --vocab-size 128 olmoe_tiny_c
+ local tag args eng R=qwen36_tiny/ref_full.json OR=olmoe_tiny/ref_olmoe.json
+ for tag in "qwen36 int8" "qwen36 int4-g64 dense" "qwen36 PILOT" "qwen36 device lost" \
+ "olmoe cap 1" "olmoe PILOT" "olmoe device lost" "olmoe automatic cache"; do
+ case $tag in
+ "qwen36 int8") args=(SNAP=qwen36_tiny_c ./qwen36 8 8 $R) ;;
+ "qwen36 int4-g64 dense") args=(COLI_DENSE_BITS=4 SNAP=qwen36_tiny64_c ./qwen36 8 4 qwen36_tiny64/ref_full.json) ;;
+ "qwen36 PILOT") args=(PILOT=1 COLI_DENSE_I8=0 SNAP=qwen36_tiny_c ./qwen36 8 8 $R) ;;
+ "qwen36 device lost") args=(COLI_VK_CHAIN_FAULT=60 COLI_DENSE_I8=0 SNAP=qwen36_tiny_c ./qwen36 8 8 $R) ;;
+ "olmoe cap 1") args=(SNAP=olmoe_tiny_c ./olmoe 1 8 $OR) ;;
+ "olmoe PILOT") args=(PILOT=1 WIDE=2 SNAP=olmoe_tiny_c ./olmoe 1 8 $OR) ;;
+ "olmoe device lost") args=(COLI_VK_CHAIN_FAULT=30 SNAP=olmoe_tiny_c ./olmoe 8 8 $OR) ;;
+ "olmoe automatic cache") args=(RAM_GB=0.7 SNAP=olmoe_tiny_c ./olmoe 0 8 $OR) ;;
+ esac
+ rm -f chain.usage
+ env COLI_USAGE=chain.usage COLI_VK_TIER_SYNC=1 COLI_VULKAN=1 COLI_VK_CHAIN=1 COLI_VK_DENSE_HOST=0 "${args[@]}" > san.log 2>&1 || true
+ if grep -qE "ERROR: AddressSanitizer|runtime error:" san.log; then cat san.log; fail "asan dense-only $tag: sanitizer diagnostic"; fi
+ [ "$(dho_dropped san.log)" -gt 0 ] || { cat san.log; fail "asan dense-only $tag: nothing on the device only"; }
+ case $tag in *lost) [ "$(dho_reloaded san.log)" -gt 0 ] || { cat san.log; fail "asan dense-only $tag: nothing read back"; } ;; esac
+ echo "OK asan dense-only $tag: sanitizers clean, $(dho_dropped san.log) matrices on the device only, $(dho_reloaded san.log) read back"
+ done
+ for eng in qwen36 olmoe; do
+ local snap=qwen36_tiny_c; [ $eng = olmoe ] && snap=olmoe_tiny_c
+ CHAIN_SERVE_EXPECT='dense matrices on the device only' COLI_VK_DENSE_HOST=0 \
+ $PY tests/vulkan_chain_serve.py ./$eng $snap COLI_DENSE_I8=0 > san.log 2>&1 || { cat san.log; fail "asan dense-only $eng serve"; }
+ if grep -qE "ERROR: AddressSanitizer|runtime error:" san.log; then cat san.log; fail "asan dense-only $eng serve: sanitizer diagnostic"; fi
+ echo "OK asan dense-only $eng serve: $(tail -1 san.log)"
+ done
+ rm -f chain.usage chain-serve.usage
+ unset OMP_NUM_THREADS
+ make clean >/dev/null 2>&1 || true
+}
diff --git a/c/tests/vulkan_dense_only_qwen38.sh b/c/tests/vulkan_dense_only_qwen38.sh
new file mode 100644
index 00000000..b26de0b2
--- /dev/null
+++ b/c/tests/vulkan_dense_only_qwen38.sh
@@ -0,0 +1,74 @@
+# Sourced by tests/vulkan_engines.sh: qwen38 (Qwen3.8 Flash Next) with its trunk on the
+# device only (COLI_VK_DENSE_HOST=0, docs/vulkan.md "Dense weights on the device only").
+# Every resident format, the int4-g64 experts, prefill in chunks, the tier off, an MTP
+# verify, prompts only and the per-matrix path alone (the steps the chain does not run
+# take the device too: the CPU has no copy), the device lost (the trunk read back from
+# the checkpoint, the CPU's tokens from there), and a serve session frame for frame.
+dho_family_qwen38() {
+ export OMP_NUM_THREADS=2
+ make qwen38 VK=1
+ $PY tools/make_qwen38_tiny.py --out qwen38_tiny
+ $PY tools/make_qwen38_tiny.py --out qwen38_tiny_int4 --fp8-experts --int4-experts --expert-gain 3
+ $PY tools/make_qwen38_tiny.py --out qwen38_tiny_mtp --mtp
+ $PY tools/make_edge_tiny_tokenizer.py --vocab-size 64 ./qwen38_tiny_mtp # serve speaks text
+ local bf16
+ for bf16 in 0 1; do
+ dho_gate qwen38 "dense-only qwen38 bf16=$bf16" 2 Q38_NATIVE_BF16=$bf16 SNAP=qwen38_tiny -- 4 8 qwen38_tiny/ref.json
+ done
+ # the int8 trunk (fmt 1): every matrix of the tiny model through the int8 conversion
+ dho_gate qwen38 "dense-only qwen38 int8 trunk" 2 Q38_TRUNK_MIN_KB=0 SNAP=qwen38_tiny -- 4 8 qwen38_tiny/ref.json
+ dho_gate qwen38 "dense-only qwen38 int4-g64 experts" 2 SNAP=qwen38_tiny_int4 -- 2 8 qwen38_tiny_int4/ref_int4.json
+ dho_gate qwen38 "dense-only qwen38 prefill in chunks of 3" 2 COLI_VK_CHAIN_ROWS=3 SNAP=qwen38_tiny -- 4 8 qwen38_tiny/ref.json
+ dho_gate qwen38 "dense-only qwen38 tier off" 2 COLI_VK_TIER=0 SNAP=qwen38_tiny -- 4 8 qwen38_tiny/ref.json
+ # the MTP head's matrices run per matrix on the device here (on the CPU with the host
+ # copies kept on this device): tokens and the CPU's logits
+ dho_gate qwen38 "dense-only qwen38 MTP mixed" 1 Q38_MTP=1 Q38_MTP_FORCE=mixed SNAP=qwen38_tiny_mtp -- 2 8 qwen38_tiny_mtp/ref.json
+ # prompts on the chain, decode on the per-matrix path: the device's in this mode
+ CHAINMODE=2 dho_gate qwen38 "dense-only qwen38 prompts only" 1 SNAP=qwen38_tiny -- 4 8 qwen38_tiny/ref.json
+ CHAINMODE=0 dho_gate qwen38 "dense-only qwen38 per-matrix path" 2 Q38_TRUNK_MIN_KB=0 SNAP=qwen38_tiny -- 4 8 qwen38_tiny/ref.json
+ # the device lost mid-decode and inside an MTP verify: the trunk comes back from disk
+ dho_lost qwen38 "dense-only qwen38 device lost" 10 Q38_TRUNK_MIN_KB=0 SNAP=qwen38_tiny -- 4 8 qwen38_tiny/ref.json
+ dho_lost qwen38 "dense-only qwen38 device lost in a verify" 15 Q38_MTP=1 Q38_MTP_FORCE=mixed SNAP=qwen38_tiny_mtp -- 2 8 qwen38_tiny_mtp/ref.json
+ CHAIN_SERVE_EXPECT='dense matrices on the device only' COLI_VK_DENSE_HOST=0 \
+ $PY tests/vulkan_chain_serve.py ./qwen38 qwen38_tiny_mtp Q38_MTP=1
+ # unset on a CPU device the host copies stay (the auto rule), and the line says why
+ rm -f chain.usage
+ env COLI_USAGE=chain.usage COLI_VULKAN=1 COLI_VK_CHAIN=1 SNAP=qwen38_tiny ./qwen38 4 8 qwen38_tiny/ref.json > vk.log 2>&1
+ grep -q '^\[VK\] qwen38: dense weights on the device and in host RAM (a CPU device' vk.log || { grep '^\[VK\]' vk.log; fail "qwen38: the auto rule dropped the host copies on a CPU device"; }
+ [ "$(dho_dropped vk.log)" = 0 ] || fail "qwen38: matrices dropped without COLI_VK_DENSE_HOST=0 on a CPU device"
+ echo "OK dense-only qwen38 auto on a CPU device: host copies kept"
+ rm -f chain.usage
+ unset OMP_NUM_THREADS
+}
+
+# The same under ASan and UBSan: memory safety is the gate, with the host copies dropped,
+# a lost device's read-back and a serve session.
+dho_family_qwen38_sanitize() {
+ make clean >/dev/null 2>&1 || true
+ make qwen38 VK=1 EXTRA_CFLAGS="-fsanitize=address,undefined -fno-omit-frame-pointer -g"
+ export ASAN_OPTIONS=detect_leaks=0:detect_stack_use_after_return=0 UBSAN_OPTIONS=print_stacktrace=1 OMP_NUM_THREADS=2
+ $PY tools/make_qwen38_tiny.py --out qwen38_tiny
+ $PY tools/make_qwen38_tiny.py --out qwen38_tiny_mtp --mtp
+ $PY tools/make_edge_tiny_tokenizer.py --vocab-size 64 ./qwen38_tiny_mtp
+ local tag args f
+ for tag in "int8 trunk" "MTP mixed" "device lost"; do
+ case $tag in
+ "int8 trunk") args=(Q38_TRUNK_MIN_KB=0 SNAP=qwen38_tiny ./qwen38 4 8 qwen38_tiny/ref.json) ;;
+ "MTP mixed") args=(Q38_MTP=1 Q38_MTP_FORCE=mixed SNAP=qwen38_tiny_mtp ./qwen38 2 8 qwen38_tiny_mtp/ref.json) ;;
+ "device lost") args=(COLI_VK_CHAIN_FAULT=40 Q38_TRUNK_MIN_KB=0 SNAP=qwen38_tiny ./qwen38 4 8 qwen38_tiny/ref.json) ;;
+ esac
+ rm -f chain.usage
+ env COLI_USAGE=chain.usage COLI_VK_TIER_SYNC=1 COLI_VULKAN=1 COLI_VK_CHAIN=1 COLI_VK_DENSE_HOST=0 "${args[@]}" > san.log 2>&1 || true
+ if grep -qE "ERROR: AddressSanitizer|runtime error:" san.log; then cat san.log; fail "asan dense-only qwen38 $tag: sanitizer diagnostic"; fi
+ [ "$(dho_dropped san.log)" -gt 0 ] || { cat san.log; fail "asan dense-only qwen38 $tag: nothing on the device only"; }
+ if [ "$tag" = "device lost" ]; then [ "$(dho_reloaded san.log)" -gt 0 ] || { cat san.log; fail "asan dense-only qwen38 $tag: nothing read back"; }; fi
+ echo "OK asan dense-only qwen38 $tag: sanitizers clean, $(dho_dropped san.log) matrices on the device only, $(dho_reloaded san.log) read back"
+ done
+ CHAIN_SERVE_EXPECT='dense matrices on the device only' COLI_VK_DENSE_HOST=0 \
+ $PY tests/vulkan_chain_serve.py ./qwen38 qwen38_tiny_mtp Q38_MTP=1 > san.log 2>&1 || { cat san.log; fail "asan dense-only qwen38 serve"; }
+ if grep -qE "ERROR: AddressSanitizer|runtime error:" san.log; then cat san.log; fail "asan dense-only qwen38 serve: sanitizer diagnostic"; fi
+ echo "OK asan dense-only qwen38 serve: $(tail -1 san.log)"
+ rm -f chain.usage
+ unset OMP_NUM_THREADS
+ make clean >/dev/null 2>&1 || true
+}
diff --git a/c/tests/vulkan_engines.sh b/c/tests/vulkan_engines.sh
index 26d42aab..b3aa5490 100755
--- a/c/tests/vulkan_engines.sh
+++ b/c/tests/vulkan_engines.sh
@@ -7,15 +7,24 @@
# bash tests/vulkan_engines.sh glm | glm-sanitize # GLM-5.2 (colibri) and GLM-5.3 Flash (glm53)
# bash tests/vulkan_engines.sh shader # the qmatmul formats, the expert batch and the tier, no engine
# bash tests/vulkan_engines.sh qwen-chain | qwen-chain-sanitize # the dense chain (COLI_VK_CHAIN=1)
+# bash tests/vulkan_engines.sh qwen-spec | qwen-spec-sanitize # MTP and prompt-lookup verifies (docs/speculative.md)
# bash tests/vulkan_engines.sh mimo-chain | mimo-chain-sanitize # MiMo-V2.6's dense chain
# bash tests/vulkan_engines.sh inkling-olmoe-chain | inkling-olmoe-chain-sanitize
# bash tests/vulkan_engines.sh glm-chain | glm-chain-sanitize # the same for colibri and glm53
# bash tests/vulkan_engines.sh kimi-chain | kimi-chain-sanitize # the same for Kimi K3
# bash tests/vulkan_engines.sh deepseek-chain | deepseek-chain-sanitize # deepseek_v41 and deepseek_v4
+# bash tests/vulkan_engines.sh prefill-qwen | prefill-qwen-sanitize | prefill-inkling-olmoe | prefill-mimo-kimi
+# bash tests/vulkan_engines.sh prefill-glm | prefill-deepseek # big prompt chunks and expert streaming
+# bash tests/vulkan_engines.sh kv-split | kv-split-sanitize # the KV cache split between the device and
+# # the host past the device's budget, every chain engine
+# bash tests/vulkan_engines.sh kv-split-deepseek | kv-split-deepseek-sanitize # the same for DeepSeek's
# bash tests/vulkan_engines.sh staged # staged uploads: the same bits as mapped memory, the decision
# bash tests/vulkan_engines.sh <family>-staged # a family with COLI_VK_STAGED=1 (qwen-staged: and
# # an engine under an emulated small window)
# bash tests/vulkan_engines.sh staged-faults | staged-faults-sanitize # staged uploads failing
+# bash tests/vulkan_engines.sh dense-only-<group> | dense-only-<group>-sanitize # the dense weights
+# on the device only (COLI_VK_DENSE_HOST=0), tests/vulkan_dense_only_<group>.sh
+# bash tests/vulkan_engines.sh decide | decide-sanitize # Laya, GLiNER2.5-Decide and Clef's DECIDE
#
# Needs libvulkan-dev, glslc and mesa-vulkan-drivers, plus the Python packages of
# the family's tiny fixtures (see the vulkan-engines job in .github/workflows/ci.yml).
@@ -1209,10 +1218,12 @@ family_mimo_chain() {
# off the chain's logits are the same bytes whatever the blocks -- one row reading the
# ring in place, blocks of 3, of the window's 8 and of 9 rows gathering it, the whole
# prompt -- so a row's window is the same rows in the same order on every path.
+ # Keep the per-row attention too: blocked attention has its own reduction order;
+ # the default blocked path's numerical gates above remain enabled.
for c in window long image; do
local x=(); [ $c = image ] && x=(--image mimo_tiny/patches.f32 --grid $MGRID)
for ch in 64 1 3 8 9; do
- COLI_VULKAN=1 COLI_VK_CHAIN=1 COLI_VK_TIER=0 COLI_VK_GEMM_MIN_S=0 MIMO_DENSE_BITS=32 MIMO_CHUNK=$ch MIMO_LOGITS=blk$ch.f32 \
+ COLI_VULKAN=1 COLI_VK_CHAIN=1 COLI_VK_TIER=0 COLI_VK_GEMM_MIN_S=0 COLI_VK_ATTN_BLOCK=0 MIMO_DENSE_BITS=32 MIMO_CHUNK=$ch MIMO_LOGITS=blk$ch.f32 \
./mimo mimo_tiny --ids "$(mimo_ids $c greedy_full_ids)" --ngen 0 "${x[@]}" > /dev/null 2> mimo-vk.err
[ "$(chain_count mimo mimo-vk.err)" -gt 0 ] || { cat mimo-vk.err; fail "chain mimo blocks $ch $c: the chain never ran"; }
[ $ch = 64 ] || cmp -s blk64.f32 blk$ch.f32 || fail "chain mimo $c: blocks of $ch are not the bytes of one block"
@@ -1232,7 +1243,7 @@ family_mimo_chain() {
# the prefix-reuse contract and Brio photos (the rings restored into the device's
# mirrors), compared bit for bit with a cold engine: the per-row GEMV and no tier, so
# a resumed prompt's rows get the bits of the same rows computed cold
- COLI_VULKAN=1 COLI_VK_CHAIN=1 COLI_VK_TIER=0 COLI_VK_GEMM_MIN_S=0 $PY -m unittest tests.test_mimo_prefix_serve
+ COLI_VULKAN=1 COLI_VK_CHAIN=1 COLI_VK_TIER=0 COLI_VK_GEMM_MIN_S=0 COLI_VK_ATTN_BLOCK=0 $PY -m unittest tests.test_mimo_prefix_serve
# a serve session against the CPU's, frame for frame, in both modes and with the
# device lost mid-session. The fixture's logits run to 300 (its head is scaled for wide
# greedy margins), so its logprobs compare within 1e-3: about 3e-6 of the largest logit
@@ -1659,6 +1670,111 @@ family_qwen_chain_sanitize() {
make clean >/dev/null 2>&1 || true
}
+# Speculative decoding with deeper verifies (docs/speculative.md): the MTP head's drafts
+# up to 3 per verify (qwen38, Q38_MTP_DRAFTS) and prompt-lookup drafts up to 5 (qwen38
+# and qwen36, COLI_LOOKUP=1), a verify of up to 6 rows with the recurrent state copied
+# after each row; and colibri's n-gram drafts from the same lookup and gate. tests/spec_drafts_harness.py (make spec-drafts-check) compares every
+# run with drafts against the same run without, tokens and last logits byte for byte:
+# - on the CPU (this VK=1 build with COLI_VULKAN unset);
+# - in the dense chain, the tier off (with it on, the tier's residency follows the
+# routed rows, the rejected drafts' included, and an expert on the device sums in
+# another order than on the CPU, so the last bits depend on the history); and the
+# chain in chunks of 3 rows (a verify of 6 rows split over two chunks), the
+# per-matrix path (a verify's rows one GEMV at a time) and prompts only
+# (COLI_VK_CHAIN=2: verifies on the per-matrix path);
+# - against the CPU with the tier on (chain_gate: the CPU's tokens, logits within
+# tolerance), and the device lost inside a deep verify (rebuilt on the CPU).
+spec_fixtures() {
+ $PY tools/make_qwen38_tiny.py --out qwen38_tiny_mtp --mtp
+ $PY tools/make_edge_tiny_tokenizer.py --vocab-size 64 ./qwen38_tiny_mtp # serve speaks text
+ $PY tools/make_qwen38_tiny.py --out qwen38_tiny_int4_mtp --fp8-experts --int4-experts --expert-gain 3 --mtp
+ make qwen36-tiny-v128-generate
+}
+family_qwen_spec() {
+ make qwen36 qwen38 tests/test_spec_draft VK=1
+ make spec-drafts-check VK=1
+ local chain="--env COLI_VULKAN=1 --env COLI_VK_CHAIN=1 --env COLI_VK_TIER=0 --env COLI_VK_TIER_SYNC=1"
+ make spec-drafts-run VK=1 SPEC_ENV="$chain"
+ local h="$PY tests/spec_drafts_harness.py" e
+ for e in "--env COLI_VULKAN=1 --env COLI_VK_CHAIN=1 --env COLI_VK_TIER=0 --env COLI_VK_CHAIN_ROWS=3" \
+ "--env COLI_VULKAN=1 --env COLI_VK_CHAIN=0 --env COLI_VK_DENSE=1 --env COLI_VK_TIER=0" \
+ "--env COLI_VULKAN=1 --env COLI_VK_CHAIN=2 --env COLI_VK_TIER=0"; do
+ $h --engine ./qwen38 --kind qwen38 --fixture ./qwen38_tiny_mtp --mtp --quick $e
+ $h --engine ./qwen36 --kind qwen36 --fixture ./qwen36_tiny_v128_c --ref ./qwen36_tiny_v128/ref_full.json \
+ --cap 8 --n-new 64 --quick --env COLI_DENSE_I8=0 $e
+ done
+ # A six-row verify crosses several one-row KV chunks. Rollback must lower both
+ # the mirror and split-cache validity, including rejected rows already uploaded.
+ # Keep the same attention kernel in plain/speculative arms for byte comparisons.
+ e="$chain --env COLI_VK_KV_DEVICE_ROWS=8 --env COLI_VK_KV_BLOCK=2 --env COLI_VK_CHAIN_ROWS=1 --env COLI_VK_ATTN_BLOCK=0"
+ $h --engine ./qwen38 --kind qwen38 --fixture ./qwen38_tiny_mtp --mtp --serve --quick --require-kv-split $e
+ $h --engine ./qwen36 --kind qwen36 --fixture ./qwen36_tiny_v128_c --ref ./qwen36_tiny_v128/ref_full.json \
+ --cap 8 --n-new 64 --serve --quick --require-kv-split --env COLI_DENSE_I8=0 $e
+ # A completed MTP chunk has already returned its hidden streams. A loss in
+ # the next chunk must restart the CPU replay from the original embeddings.
+ $h --engine ./qwen38 --kind qwen38 --fixture ./qwen38_tiny_mtp --mtp --n-new 32 \
+ --device-loss --require-kv-split --env COLI_VK_DENSE_HOST=0 $e
+ local f
+ for f in cycle row2 row3; do
+ chain_gate qwen38 "spec chain qwen38 MTP depth 3 $f, tier on" 1 OMP_NUM_THREADS=2 Q38_MTP=1 Q38_MTP_DRAFTS=3 \
+ Q38_MTP_FORCE=$f SNAP=qwen38_tiny_mtp -- 2 8 qwen38_tiny_mtp/ref.json
+ chain_gate qwen36 "spec chain qwen36 lookup $f, tier on" 1 COLI_DENSE_I8=0 COLI_LOOKUP=1 COLI_SPEC_GATE=0 \
+ COLI_LOOKUP_FORCE=$f SNAP=qwen36_tiny_v128_c -- 8 8 qwen36_tiny_v128/ref_full.json
+ done
+ chain_gate qwen38 "spec chain qwen38 lookup row4, int4 experts, tier on" 1 OMP_NUM_THREADS=2 Q38_EXPERT_INT4=1 \
+ COLI_VK_TIER_BALANCE=0 COLI_LOOKUP=1 COLI_SPEC_GATE=0 COLI_LOOKUP_FORCE=row4 SNAP=qwen38_tiny_int4_mtp -- 2 8 \
+ qwen38_tiny_int4_mtp/ref_int4.json
+ # colibri (GLM-5.2): its n-gram drafts from spec_draft.h under COLI_LOOKUP=1, on the
+ # CPU and in the chain, against the oracle (every token) and against DRAFT=0
+ make colibri VK=1
+ $PY tools/make_glm_oracle.py > /dev/null
+ local v base
+ base=$(env OMP_NUM_THREADS=2 SNAP=glm_tiny REF=ref_glm.json DRAFT=0 ./colibri 64 16 16 2>/dev/null | grep -a '^GLM C engine')
+ for v in "" "COLI_VULKAN=1 COLI_VK_CHAIN=1 COLI_VK_TIER=0" "COLI_VULKAN=1 COLI_VK_CHAIN=1"; do
+ for e in "COLI_SPEC_GATE=0" "COLI_SPEC_GATE=0 DRAFT=2" ""; do
+ # shellcheck disable=SC2086
+ env OMP_NUM_THREADS=2 SNAP=glm_tiny REF=ref_glm.json COLI_USAGE=chain.usage COLI_LOOKUP=1 $e $v ./colibri 64 16 16 > vk.log 2>&1 \
+ || { cat vk.log; fail "spec colibri lookup $e $v: misses the oracle"; }
+ [ "$(grep -a '^GLM C engine' vk.log)" = "$base" ] || { cat vk.log; fail "spec colibri lookup $e $v: tokens differ from DRAFT=0"; }
+ grep -q "colibri lookup\] acceptance" vk.log || { cat vk.log; fail "spec colibri lookup $e $v: no lookup line"; }
+ echo "OK spec colibri lookup ${e:-gated} ${v:-CPU}: oracle and DRAFT=0's tokens, $(grep -o 'acceptance [^|]*' vk.log | head -1)"
+ done
+ done
+ lost_late qwen38 "spec chain qwen38 device lost mid-decode, verifies of 4 rows" 15 rebuild OMP_NUM_THREADS=2 Q38_MTP=1 \
+ Q38_MTP_DRAFTS=3 Q38_MTP_FORCE=cycle SNAP=qwen38_tiny_mtp -- 2 8 qwen38_tiny_mtp/ref.json
+ lost_late qwen36 "spec chain qwen36 device lost mid-decode, lookup verifies" 20 rebuild COLI_DENSE_I8=0 COLI_LOOKUP=1 \
+ COLI_SPEC_GATE=0 COLI_LOOKUP_FORCE=cycle SNAP=qwen36_tiny_v128_c -- 8 8 qwen36_tiny_v128/ref_full.json
+}
+# The same under ASan and UBSan: every run of the harness's quick subset (depth 3
+# rejected at rows 1 and 2 and cycling, lookup cycling and at row 3, a serve session
+# per source) on the CPU and in the chain, sanitizer-clean and still equal to its own
+# plain decoding.
+family_qwen_spec_sanitize() {
+ make clean >/dev/null 2>&1 || true
+ make qwen36 qwen38 tests/test_spec_draft tests/test_qwen38_spec_alloc VK=1 EXTRA_CFLAGS="-fsanitize=address,undefined -fno-omit-frame-pointer -g"
+ export ASAN_OPTIONS=detect_leaks=0:detect_stack_use_after_return=0 UBSAN_OPTIONS=print_stacktrace=1
+ ./tests/test_spec_draft > san.log 2>&1 || true
+ if grep -qE "ERROR: AddressSanitizer|runtime error:" san.log || ! tail -1 san.log | grep -q "behave as"; then cat san.log; fail "asan: spec_draft.h"; fi
+ echo "OK asan: spec_draft.h"
+ ./tests/test_qwen38_spec_alloc > san.log 2>&1 || { cat san.log; fail "asan: speculative allocation"; }
+ if grep -qE "ERROR: AddressSanitizer|runtime error:" san.log; then cat san.log; fail "asan: speculative allocation"; fi
+ echo "OK asan: speculative allocation"
+ spec_fixtures
+ local h="$PY tests/spec_drafts_harness.py --sanitize --quick" e
+ for e in "" "--env COLI_VULKAN=1 --env COLI_VK_CHAIN=1 --env COLI_VK_TIER=0" \
+ "--require-kv-split --env COLI_VULKAN=1 --env COLI_VK_CHAIN=1 --env COLI_VK_TIER=0 --env COLI_VK_KV_DEVICE_ROWS=8 --env COLI_VK_KV_BLOCK=2 --env COLI_VK_CHAIN_ROWS=1 --env COLI_VK_ATTN_BLOCK=0"; do
+ $h --engine ./qwen38 --kind qwen38 --fixture ./qwen38_tiny_mtp --mtp --serve $e
+ $h --engine ./qwen38 --kind qwen38 --fixture ./qwen38_tiny_int4_mtp --ref ./qwen38_tiny_int4_mtp/ref_int4.json \
+ --mtp --env Q38_EXPERT_INT4=1 --env Q38_TRUNK_MIN_KB=0 $e
+ $h --engine ./qwen36 --kind qwen36 --fixture ./qwen36_tiny_v128_c --ref ./qwen36_tiny_v128/ref_full.json \
+ --cap 8 --n-new 64 --serve --env COLI_DENSE_I8=0 $e
+ done
+ $h --engine ./qwen38 --kind qwen38 --fixture ./qwen38_tiny_mtp --mtp --n-new 32 --device-loss \
+ --require-kv-split --env COLI_VULKAN=1 --env COLI_VK_CHAIN=1 --env COLI_VK_TIER=0 --env COLI_VK_DENSE_HOST=0 \
+ --env COLI_VK_KV_DEVICE_ROWS=8 --env COLI_VK_KV_BLOCK=2 --env COLI_VK_CHAIN_ROWS=1 --env COLI_VK_ATTN_BLOCK=0
+ make clean >/dev/null 2>&1 || true
+}
+
# inkling's and olmoe's fixtures for their dense chain: the family inkling-olmoe's (the
# bf16 copy of the f32 snapshot, the dense-int4g64 container, the expert containers and
# the tokenizers), and Inkling at its real width (--wide: D = 6144, head dim 128).
@@ -1808,10 +1924,11 @@ family_inkling_olmoe_chain() {
COLI_VK_CHAIN=2 $PY tests/vulkan_chain_serve.py ./olmoe olmoe_tiny_c
# (these compare a warm session's logprobs with a cold one's to the last printed
# digit: no history file and COLI_VK_TIER_BALANCE=0, so both sessions split their
- # experts between the device and the CPU alike)
- COLI_VULKAN=1 COLI_VK_CHAIN=1 COLI_VK_TIER_SYNC=1 COLI_VK_TIER_BALANCE=0 INKLING_TINY=tiny_inkling \
+ # experts between the device and the CPU alike. Use per-row attention on both:
+ # a cold prefill and a short suffix must share a reduction order for exact bits.)
+ COLI_VULKAN=1 COLI_VK_CHAIN=1 COLI_VK_TIER_SYNC=1 COLI_VK_TIER_BALANCE=0 COLI_VK_ATTN_BLOCK=0 INKLING_TINY=tiny_inkling \
$PY -m unittest tests.test_inkling_prefix_serve tests.test_inkling_dashboard_hits
- COLI_VULKAN=1 COLI_VK_CHAIN=1 COLI_VK_TIER_SYNC=1 COLI_VK_TIER_BALANCE=0 OLMOE_TINY=olmoe_tiny_c \
+ COLI_VULKAN=1 COLI_VK_CHAIN=1 COLI_VK_TIER_SYNC=1 COLI_VK_TIER_BALANCE=0 COLI_VK_ATTN_BLOCK=0 OLMOE_TINY=olmoe_tiny_c \
$PY -m unittest tests.test_olmoe_prefix_serve tests.test_olmoe_dashboard_hits tests.test_brio_serve
}
@@ -2052,7 +2169,10 @@ family_glm_chain_sanitize() {
# |logit|, the logprobs of the serve sessions within 2e-2, the tokens exactly. With the CPU's int8 expert activations
# (K3_IDOT=1) a rounding that flips an int8 step moves the logits further (8e-3, the
# tokens unchanged): that configuration gates on its tokens (TOKENS=1).
-k3c_ids() { $PY -c "import json,sys;print(' '.join(map(str,json.load(open('kimi_k3_tiny/ref.json'))['cases'][sys.argv[1]]['prompt_ids'])))" "$1"; }
+k3c_ids() { # case "ids": the prompt K3C_IDS holds (a prompt past the oracle's)
+ if [ "$1" = ids ] && [ -n "${K3C_IDS:-}" ]; then echo "$K3C_IDS"; return; fi
+ $PY -c "import json,sys;print(' '.join(map(str,json.load(open('kimi_k3_tiny/ref.json'))['cases'][sys.argv[1]]['prompt_ids'])))" "$1"
+}
k3c_close() { # <cpu.f32> <vk.f32>: max |diff| within 2e-3 of the largest |logit|
$PY - "$1" "$2" <<'PY'
import array, sys
@@ -2152,8 +2272,8 @@ family_kimi_chain() {
# the device lost: 9 frames a forward on this fixture (two per sparse layer and the
# head); the long prompt is three forwards (32, 32 and 8 rows), then 7 decode steps
FAULT_BACK=3 REBUILD=1 k3c_gate "chain kimi_k3 device lost mid-decode" $O
- K3C_CASES=long FAULT_BACK=86 k3c_gate "chain kimi_k3 device lost in the first prompt chunk" $O
- K3C_CASES=long FAULT_BACK=77 REBUILD=1 k3c_gate "chain kimi_k3 device lost in a later prompt chunk" $O
+ K3C_CASES=long FAULT_BACK=86 k3c_gate "chain kimi_k3 device lost in the first prompt chunk" $O K3_CHUNK=32
+ K3C_CASES=long FAULT_BACK=77 REBUILD=1 k3c_gate "chain kimi_k3 device lost in a later prompt chunk" $O K3_CHUNK=32
# in chain chunks of 5 rows (seven chunks a 32-row forward), the loss in the third
# chunk of the second forward: the state its first chunks advanced is not used
K3C_CASES=long FAULT_BACK=124 REBUILD=1 k3c_gate "chain kimi_k3 device lost inside a chunked forward" $O COLI_VK_CHAIN_ROWS=5
@@ -2668,6 +2788,1162 @@ family_deepseek_chain_sanitize() {
make clean >/dev/null 2>&1 || true
}
+# ---- the dense weights on the device only (COLI_VK_DENSE_HOST=0; docs/vulkan.md,
+# "Dense weights on the device only"): one file per engine group,
+# tests/vulkan_dense_only_<group>.sh, each defining dho_family_<group> and
+# dho_family_<group>_sanitize (run as dense-only-<group> and dense-only-<group>-sanitize).
+#
+# dho_dropped <log>: matrices the engine placed on the device only (its startup line)
+dho_dropped() {
+ local n; n=$(sed -n 's/^\[VK\] [a-z0-9_]*: \([0-9][0-9]*\) dense matrices on the device only.*/\1/p' "$1" | tail -1)
+ echo "${n:-0}"
+}
+# dho_reloaded <log>: host copies read back from disk for the CPU (the exit line)
+dho_reloaded() {
+ local n; n=$(sed -n 's/^\[VK\] [a-z0-9_]*: dense weights at exit: .*), \([0-9][0-9]*\) read back from disk.*/\1/p' "$1" | tail -1)
+ echo "${n:--1}"
+}
+# dho_rss <log>: the resident set the exit line reports, in GiB
+dho_rss() { sed -n 's/^\[VK\] [a-z0-9_]*: dense weights at exit: .*; RSS \([0-9.]*\) GiB$/\1/p' "$1" | tail -1; }
+# dho_gate <engine> <tag> <tol 0|1|2> <env...> -- <argv...>: one configuration three
+# times, on the CPU, on the device keeping its host copies (COLI_VK_DENSE_HOST=1) and on
+# the device only (=0), the tier deterministic (COLI_VK_TIER_SYNC=1 COLI_VK_TIER_BALANCE=0)
+# and each run from a fresh history. Gates: the CPU's tokens; matrices dropped and none read
+# back; with the chain asked for (CHAINMODE, default 1; 0 = the per-matrix path alone with
+# COLI_VK_DENSE=1), chain forwards; tol 2: the logits the bytes of the host-copy run (what
+# runs on the device is the same), 1: within 1e-4 of the CPU's largest logit (logits_close),
+# 0: tokens only. CPUENV: the CPU arm's own settings; DUMPENV: the variable the engine
+# writes its logits to (default DUMP).
+dho_gate() {
+ local eng=$1 tag=$2 tol=$3; shift 3
+ local envs=(); while [ "$1" != "--" ]; do envs+=("$1"); shift; done; shift
+ local dv=${DUMPENV:-DUMP} mode=${CHAINMODE:-1}
+ local vk=(COLI_VK_TIER_SYNC=1 COLI_VK_TIER_BALANCE=0 COLI_VULKAN=1 COLI_VK_CHAIN=$mode)
+ [ "$mode" = 0 ] && vk+=(COLI_VK_DENSE=1)
+ rm -f chain.usage cpu.f32 vk.f32 host.f32
+ env "${envs[@]}" ${CPUENV:-} $dv=cpu.f32 ./"$eng" "$@" > cpu.log 2>&1 || true
+ env "${envs[@]}" "${vk[@]}" COLI_USAGE=chain.usage COLI_VK_DENSE_HOST=1 $dv=host.f32 ./"$eng" "$@" > host.log 2>&1 || true
+ rm -f chain.usage
+ env "${envs[@]}" "${vk[@]}" COLI_USAGE=chain.usage COLI_VK_DENSE_HOST=0 $dv=vk.f32 ./"$eng" "$@" > vk.log 2>&1 || true
+ rm -f chain.usage
+ same_tokens cpu.log vk.log "$tag"
+ same_tokens cpu.log host.log "$tag (host copies)"
+ local n; n=$(dho_dropped vk.log)
+ [ "$n" -gt 0 ] || { cat vk.log; fail "$tag: no dense matrix went to the device only"; }
+ [ "$(dho_reloaded vk.log)" = 0 ] || { grep '^\[VK\]' vk.log; fail "$tag: a matrix was read back from disk without a lost device"; }
+ if [ "$mode" != 0 ]; then [ "$(chain_count "$eng" vk.log)" -gt 0 ] || { cat vk.log; fail "$tag: the chain did not stay on"; }; fi
+ local lg=""
+ case $tol in
+ 2) cmp -s host.f32 vk.f32 || fail "$tag: the logits differ from the run that keeps its host copies"; lg=", logits = the host-copy run's" ;;
+ 1) lg=$(logits_close cpu.f32 vk.f32) || { echo "$lg"; fail "$tag: logits"; }; lg=", $lg" ;;
+ esac
+ echo "OK $tag: tokens = CPU$lg, $n matrices on the device only, $(chain_count "$eng" vk.log) chain forwards"
+}
+# dho_lost <engine> <tag> <frames back> <env...> -- <argv...>: device only, the device lost
+# `back` frames before the end of a run (COLI_VK_CHAIN_FAULT, counted on a probe run): the
+# CPU's tokens, and the matrices the CPU then needed read back from disk.
+dho_lost() {
+ local eng=$1 tag=$2 back=$3; shift 3
+ local envs=(); while [ "$1" != "--" ]; do envs+=("$1"); shift; done; shift
+ local vk=(COLI_VK_TIER_SYNC=1 COLI_VULKAN=1 COLI_VK_CHAIN=1 COLI_VK_DENSE_HOST=0)
+ rm -f chain.usage
+ env "${envs[@]}" ./"$eng" "$@" > cpu.log 2>&1 || true
+ env "${envs[@]}" "${vk[@]}" COLI_USAGE=chain.usage ./"$eng" "$@" > vk.log 2>&1 || true
+ local f; f=$(sed -n "s/^\[VK\] $eng chain: [0-9]* forwards, \([0-9]*\) frames.*/\1/p" vk.log | tail -1)
+ { [ -n "$f" ] && [ "$f" -gt "$back" ]; } || { cat vk.log; fail "$tag: the probe run reports no frames"; }
+ rm -f chain.usage
+ env "${envs[@]}" "${vk[@]}" COLI_VK_CHAIN_FAULT=$((f - back)) COLI_USAGE=chain.usage ./"$eng" "$@" > vk.log 2>&1 || true
+ rm -f chain.usage
+ same_tokens cpu.log vk.log "$tag"
+ local r; r=$(dho_reloaded vk.log)
+ [ "$r" -gt 0 ] || { grep '^\[VK\]' vk.log; fail "$tag: nothing was read back for the CPU"; }
+ echo "OK $tag: tokens = CPU after the loss, $r matrices read back from disk"
+}
+# dho_rss_gate <engine> <tag> <MiB at least> <env...> -- <argv...>: the resident set at
+# exit falls by most of what was dropped (the device copies are host memory on a CPU
+# device, the host copies are not there twice): RSS(host copies) - RSS(device only) is at
+# least the given MiB.
+dho_rss_gate() {
+ local eng=$1 tag=$2 mib=$3; shift 3
+ local envs=(); while [ "$1" != "--" ]; do envs+=("$1"); shift; done; shift
+ local vk=(COLI_VK_TIER_SYNC=1 COLI_VULKAN=1 COLI_VK_CHAIN=${CHAINMODE:-1})
+ rm -f chain.usage
+ env "${envs[@]}" "${vk[@]}" COLI_USAGE=chain.usage COLI_VK_DENSE_HOST=1 ./"$eng" "$@" > host.log 2>&1 || true
+ rm -f chain.usage
+ env "${envs[@]}" "${vk[@]}" COLI_USAGE=chain.usage COLI_VK_DENSE_HOST=0 ./"$eng" "$@" > vk.log 2>&1 || true
+ rm -f chain.usage
+ local a b; a=$(dho_rss host.log); b=$(dho_rss vk.log)
+ { [ -n "$a" ] && [ -n "$b" ]; } || { grep '^\[VK\]' host.log vk.log; fail "$tag: no exit line with the resident set"; }
+ $PY - "$a" "$b" "$mib" <<'PY' || { grep '^\[VK\].*dense' host.log vk.log; fail "$tag: the resident set did not fall"; }
+import sys
+a, b, mib = float(sys.argv[1]), float(sys.argv[2]), float(sys.argv[3])
+print(f"RSS at exit {a:.2f} GiB with the host copies, {b:.2f} GiB on the device only")
+sys.exit(0 if (a - b) * 1024 >= mib else 1)
+PY
+ echo "OK $tag: $(grep -o 'given back.*' vk.log | head -1)"
+}
+for f in tests/vulkan_dense_only_*.sh; do [ -e "$f" ] && . "$f"; done
+# ---- big prompt chunks and expert streaming (docs/vulkan.md, "Big prompt chunks and
+# expert streaming") ----
+# Every engine with the chain, on a prompt longer than its usual block, against its CPU
+# run with the same snapshot and settings:
+# default the chunk from the budget (the whole prompt in one chunk here) and the cold
+# experts the rule streams: the CPU's tokens and logits, the chunk's line,
+# and one tier step of the whole prompt;
+# forced a small chunk cap and a forced streaming threshold (PF_FORCE: chunks of
+# PF_ROWS rows, a tier of a few experts, cold experts with 3 rows or more
+# streamed through four slots in sub-batches of 24 rows, so experts are cut
+# into parts): the CPU's tokens and logits, streamed experts counted, and the
+# later chunks' prefetch from the earlier ones' routing (PF_PREFETCH=1);
+# staged the forced configuration with COLI_VK_STAGED=1 (the staging slots filled
+# through host images): the same, nothing resident left in host memory;
+# off COLI_VK_CHAIN_ROWS=512 COLI_VK_TIER_STREAM=0 COLI_VK_ATTN_BLOCK=0, the
+# build before's behaviour: the CPU's tokens, nothing streamed.
+# <family>-sanitize runs the forced configurations under ASan and UBSan.
+PF_FORCE="COLI_VK_TIER_STREAM_ROWS=3 COLI_VK_TIER_STREAM_SLOTS=4 COLI_VK_TIER_STREAM_HALF=24"
+PF_OFF="COLI_VK_CHAIN_ROWS=512 COLI_VK_TIER_STREAM=0 COLI_VK_ATTN_BLOCK=0"
+stream_count() { # <engine> <log>: cold experts streamed in the run (the last report)
+ local n; n=$(sed -n "s/^\[VK\] tier $1 run: .* stream: [0-9]* steps, \([0-9]*\) cold experts.*/\1/p" "$2" | tail -1)
+ echo "${n:-0}"
+}
+prefetch_count() {
+ local n; n=$(sed -n "s/^\[VK\] tier $1 run: .* prefetched \([0-9]*\) (\([0-9]*\) used).*/\2/p" "$2" | tail -1)
+ echo "${n:-0}"
+}
+# pf_check <engine> <log> <tag> <kind> [rows]: what the run must show besides the CPU's tokens
+pf_check() {
+ local eng=$1 log=$2 tag=$3 kind=$4 rows=${5:-}
+ case $kind in
+ default)
+ grep -q "^\[VK\] $eng chain: prompt chunks of up to 8192 rows" "$log" || { grep '\[VK\]' "$log"; fail "$tag: the chunk is not the budget's"; }
+ if [ -n "$rows" ]; then grep -q "^\[VK\] tier $eng stream: a step of $rows rows" "$log" ||
+ { grep '\[VK\] tier' "$log"; fail "$tag: no tier step of the whole prompt ($rows rows)"; }; fi ;;
+ forced|staged)
+ [ "$(stream_count "$eng" "$log")" -gt 0 ] || { grep '\[VK\] tier' "$log"; fail "$tag: nothing streamed"; }
+ if [ "${PF_PREFETCH:-0}" = 1 ]; then [ "$(prefetch_count "$eng" "$log")" -gt 0 ] ||
+ { grep '\[VK\] tier' "$log"; fail "$tag: no prefetched expert was used"; }; fi
+ if [ $kind = staged ]; then staged_check "$log" "$tag"; fi ;;
+ off)
+ ! grep -q "^\[VK\] tier $eng stream:" "$log" || { grep '\[VK\] tier' "$log"; fail "$tag: streamed with the feature off"; } ;;
+ esac
+ echo " $tag: $(grep -o 'stream: [0-9]* steps, [0-9]* cold experts[^|]*' "$log" | tail -1 | cut -c1-160)"
+}
+# pf_runs <gate> <engine> <tag> <log> <rows> <budget env> <env...>: the four runs above.
+# <gate> is a command line (eval'd: its quotes hold) taking <tag> <tol> <env...> after it.
+pf_runs() {
+ local g=$1 eng=$2 tag=$3 log=$4 rows=$5 bud=$6; shift 6
+ local rw="COLI_VK_CHAIN_ROWS=${PF_ROWS:-64}"
+ eval "$g"' "pf $tag default" 1 "$@"'; pf_check "$eng" "$log" "pf $tag default" default "$rows"
+ eval "$g"' "pf $tag forced" 1 $PF_FORCE $bud $rw "$@"'; pf_check "$eng" "$log" "pf $tag forced" forced
+ eval "$g"' "pf $tag staged" 1 $PF_FORCE $bud $rw COLI_VK_STAGED=1 "$@"'; pf_check "$eng" "$log" "pf $tag staged" staged
+ eval "$g"' "pf $tag off" 1 $PF_OFF "$@"'; pf_check "$eng" "$log" "pf $tag off" off
+}
+# the engines' gates in pf_runs' calling convention: <fixed args...> <tag> <tol> <env...>
+pf_chain() { # <engine> <argv as one string> <tag> <tol> <env...>
+ local eng=$1 argv=$2 tag=$3 tol=$4; shift 4
+ # shellcheck disable=SC2086
+ chain_gate "$eng" "$tag" "$tol" "$@" -- $argv
+}
+pf_mla() { # <engine> <argv as one string> <tag> <tol> <env...>
+ local eng=$1 argv=$2 tag=$3 tol=$4; shift 4
+ # shellcheck disable=SC2086
+ mla_gate "$eng" "$tag" "$tol" "$@" -- $argv
+}
+pf_ids() { $PY -c "print(','.join(str($2 + (i * 37) % $3) for i in range($1)))"; }
+# a long prompt's ref.json for the engines whose oracle reads one: the ids and a few more
+pf_ref() { # <out> <n> <first id> <span>
+ $PY - "$1" "$2" "$3" "$4" <<'PY'
+import json, sys
+out, n, lo, span = sys.argv[1], int(sys.argv[2]), int(sys.argv[3]), int(sys.argv[4])
+p = [lo + (i * 37) % span for i in range(n)]
+full = p + [lo] * 8
+json.dump({"prompt_ids": p, "full_ids": full, "tf_pred": full[1:] + [lo]}, open(out, "w"))
+PY
+}
+# MiMo on a prompt of ids (its cases are at most 72 tokens): tokens with --ngen 6, every
+# position's logits with --ngen 0, against the CPU
+pf_mimo() { # <ids> <tag> <tol> <env...>
+ local ids=$1 tag=$2; shift 3
+ env "$@" COLI_TEMP=0 ./mimo mimo_tiny --ids "$ids" --ngen 6 > mimo-cpu.txt 2>/dev/null
+ env "$@" COLI_TEMP=0 COLI_VULKAN=1 COLI_VK_CHAIN=1 COLI_VK_TIER_SYNC=1 ./mimo mimo_tiny --ids "$ids" --ngen 6 > mimo-vk.txt 2> mimo-vk.err
+ [ -s mimo-cpu.txt ] && cmp -s mimo-cpu.txt mimo-vk.txt || { cat mimo-cpu.txt mimo-vk.txt mimo-vk.err; fail "$tag: tokens differ from the CPU"; }
+ rm -f cpu.f32 vk.f32
+ env "$@" MIMO_LOGITS=cpu.f32 ./mimo mimo_tiny --ids "$ids" --ngen 0 > /dev/null 2>&1
+ env "$@" MIMO_LOGITS=vk.f32 COLI_VULKAN=1 COLI_VK_CHAIN=1 COLI_VK_TIER_SYNC=1 ./mimo mimo_tiny --ids "$ids" --ngen 0 > /dev/null 2> mimo-vk2.err
+ local lg; lg=$(logits_close cpu.f32 vk.f32) || { echo "$lg"; fail "$tag: logits"; }
+ echo "OK $tag: tokens = CPU, $lg, $(chain_count mimo mimo-vk.err) chain forwards"
+}
+pf_k3() { # <ids> <tag> <tol> <env...>: k3c_gate on one prompt of ids
+ local ids=$1 tag=$2; shift 3
+ K3C_IDS="$ids" K3C_CASES=ids k3c_gate "$tag" "$@"
+}
+pf_v41() { # <snap> <tag> <tol> <env...>
+ local snap=$1 tag=$2; shift 3
+ v41_gate "$tag" SNAP="$snap" "$@" -- 2 "$snap"/ref.json
+}
+pf_v4() { # <fixture> <case> <tag> <tol> <env...>
+ local fx=$1 c=$2 tag=$3; shift 4
+ v4_chain_gate "$tag" "$fx" "$c" "$@"
+}
+
+family_prefill_qwen() {
+ make qwen36 qwen38 tests/test_vk_chain tests/test_vk_tier VK=1
+ ./tests/test_vk_tier shaders/qmatmul.spv | tee vk_tier.log
+ tail -1 vk_tier.log | grep -qx PASS || fail "the tier (its streaming section)"
+ ./tests/test_vk_chain shaders/qmatmul.spv | tee vk_chain.log
+ tail -1 vk_chain.log | grep -qx PASS || fail "the chain's ops (the blocked attention)"
+ $PY tools/make_qwen36_tiny.py --out qwen36_tiny --ref-mode full --emit-ref qwen36_tiny/ref_full.json
+ $PY tools/convert_qwen36.py --model qwen36_tiny --out qwen36_tiny_c --ebits 8
+ $PY tools/make_qwen36_tiny.py --out qwen36_tiny64 --ref-mode full --inter 64 --emit-ref qwen36_tiny64/ref_full.json
+ $PY tools/convert_qwen36.py --model qwen36_tiny64 --out qwen36_tiny64_c --ebits 4 --gs 64
+ pf_ref qwen36_tiny/ref_long.json 300 1 250
+ local ids; ids=$(pf_ids 110 1 60)
+ $PY tools/make_qwen38_tiny.py --out qwen38_long_int4 --prompt-ids "$ids" --fp8-experts --int4-experts --expert-gain 3
+ $PY tools/make_qwen38_tiny.py --out qwen38_long --prompt-ids "$ids"
+ export OMP_NUM_THREADS=2
+ pf_runs "pf_chain qwen36 '8 8 qwen36_tiny/ref_long.json'" qwen36 "qwen36 int8 experts" vk.log 300 COLI_VK_TIER_GB=0.00015 COLI_DENSE_I8=0 SNAP=qwen36_tiny_c
+ pf_runs "pf_chain qwen36 '2 4 qwen36_tiny/ref_long.json'" qwen36 "qwen36 planar int4-g64, cap 2" vk.log 300 COLI_VK_TIER_GB=0.0003 QWEN_EXPERT_ACT=f32 COLI_DENSE_I8=0 SNAP=qwen36_tiny64_c
+ pf_runs "pf_chain qwen38 '4 8 qwen38_long/ref.json'" qwen38 "qwen38 bf16" vk.log 110 COLI_VK_TIER_GB=0.00005 SNAP=qwen38_long
+ pf_runs "pf_chain qwen38 '2 8 qwen38_long_int4/ref_int4.json'" qwen38 "qwen38 int4 sidecar" vk.log 110 COLI_VK_TIER_GB=0.00005 SNAP=qwen38_long_int4
+ PF_PREFETCH=1 chain_gate qwen36 "pf qwen36 prefetch across chunks" 1 $PF_FORCE COLI_VK_TIER_GB=0.00015 COLI_VK_CHAIN_ROWS=100 COLI_DENSE_I8=0 SNAP=qwen36_tiny_c -- 8 8 qwen36_tiny/ref_long.json
+ PF_PREFETCH=1 pf_check qwen36 vk.log "pf qwen36 prefetch across chunks" forced
+ PF_PREFETCH=1 chain_gate qwen38 "pf qwen38 prefetch across chunks" 1 $PF_FORCE COLI_VK_TIER_GB=0.00005 COLI_VK_CHAIN_ROWS=40 SNAP=qwen38_long_int4 -- 2 8 qwen38_long_int4/ref_int4.json
+ PF_PREFETCH=1 pf_check qwen38 vk.log "pf qwen38 prefetch across chunks" forced
+ # the CPU's prefill batching off (Q38_PREFILL_BATCH=0): the chain's MoE step still whole
+ chain_gate qwen38 "pf qwen38 batch=0" 1 Q38_PREFILL_BATCH=0 COLI_VK_TIER_GB=0.00005 COLI_VK_TIER_STREAM_ROWS=3 SNAP=qwen38_long -- 4 8 qwen38_long/ref.json
+ # the device lost while a chunk's streamed sub-batches are in flight: the state rebuilt
+ # on the CPU, the run finishes there
+ lost_late qwen36 "pf qwen36 device lost in a streamed prompt" 60 rebuild $PF_FORCE COLI_VK_TIER_GB=0.00015 COLI_VK_CHAIN_ROWS=64 COLI_DENSE_I8=0 SNAP=qwen36_tiny_c -- 8 8 qwen36_tiny/ref_long.json
+ unset OMP_NUM_THREADS
+}
+family_prefill_qwen_sanitize() {
+ make clean >/dev/null 2>&1 || true
+ make qwen36 qwen38 tests/test_vk_tier VK=1 EXTRA_CFLAGS="-fsanitize=address,undefined -fno-omit-frame-pointer -g"
+ export ASAN_OPTIONS=detect_leaks=0:detect_stack_use_after_return=0 UBSAN_OPTIONS=print_stacktrace=1
+ ./tests/test_vk_tier shaders/qmatmul.spv > san.log 2>&1 || true
+ if grep -qE "ERROR: AddressSanitizer|runtime error:" san.log || ! tail -1 san.log | grep -qx PASS; then cat san.log; fail "asan: the tier"; fi
+ echo "OK asan: the tier, its streaming section included"
+ COLI_VK_STAGED=1 ./tests/test_vk_tier shaders/qmatmul.spv > san.log 2>&1 || true
+ if grep -qE "ERROR: AddressSanitizer|runtime error:" san.log || ! tail -1 san.log | grep -qx PASS; then cat san.log; fail "asan: the tier staged"; fi
+ echo "OK asan: the tier staged"
+ $PY tools/make_qwen36_tiny.py --out qwen36_tiny --ref-mode full --emit-ref qwen36_tiny/ref_full.json
+ $PY tools/convert_qwen36.py --model qwen36_tiny --out qwen36_tiny_c --ebits 8
+ pf_ref qwen36_tiny/ref_long.json 300 1 250
+ local ids; ids=$(pf_ids 110 1 60)
+ $PY tools/make_qwen38_tiny.py --out qwen38_long_int4 --prompt-ids "$ids" --fp8-experts --int4-experts --expert-gain 3
+ export OMP_NUM_THREADS=2
+ pfsan qwen36 "asan pf qwen36 forced" COLI_VK_TIER_GB=0.00015 COLI_VK_CHAIN_ROWS=64 COLI_DENSE_I8=0 SNAP=qwen36_tiny_c ./qwen36 8 8 qwen36_tiny/ref_long.json
+ pfsan qwen36 "asan pf qwen36 forced, staged" COLI_VK_STAGED=1 COLI_VK_TIER_GB=0.00015 COLI_VK_CHAIN_ROWS=64 COLI_DENSE_I8=0 SNAP=qwen36_tiny_c ./qwen36 8 8 qwen36_tiny/ref_long.json
+ pfsan qwen38 "asan pf qwen38 forced" COLI_VK_TIER_GB=0.00005 COLI_VK_CHAIN_ROWS=40 SNAP=qwen38_long_int4 ./qwen38 2 8 qwen38_long_int4/ref_int4.json
+ unset OMP_NUM_THREADS
+ make clean >/dev/null 2>&1 || true
+}
+# pfsan <engine> <tag> <env and argv...>: the forced configuration under the sanitizers
+pfsan() {
+ local eng=$1 tag=$2; shift 2
+ rm -f chain.usage
+ # shellcheck disable=SC2086
+ env $PF_FORCE COLI_USAGE=chain.usage COLI_VK_TIER_SYNC=1 COLI_VULKAN=1 COLI_VK_CHAIN=1 "$@" > san.log 2>&1 || true
+ if grep -qE "ERROR: AddressSanitizer|runtime error:" san.log; then cat san.log; fail "$tag: sanitizer diagnostic"; fi
+ [ "$(chain_count "$eng" san.log)" -gt 0 ] || { cat san.log; fail "$tag: the chain never ran"; }
+ [ "$(stream_count "$eng" san.log)" -gt 0 ] || { grep '\[VK\]' san.log; fail "$tag: nothing streamed"; }
+ echo "OK $tag: sanitizers clean, $(chain_count "$eng" san.log) chain forwards, $(stream_count "$eng" san.log) experts streamed"
+}
+
+family_prefill_inkling_olmoe() {
+ export OMP_NUM_THREADS=2
+ make inkling olmoe VK=1
+ $PY tools/make_tiny_inkling.py tiny_inkling
+ $PY tools/make_olmoe_tiny.py --output olmoe_tiny --force
+ $PY tools/convert_olmoe_merged.py --model olmoe_tiny --out olmoe_tiny_c
+ pf_ref tiny_inkling/ref_long.json 200 1 250
+ pf_ref olmoe_tiny/ref_long.json 200 1 120
+ pf_runs "pf_chain inkling '8 0 tiny_inkling/ref_long.json'" inkling "inkling f32" vk.log 200 COLI_VK_TIER_GB=0.0001 SNAP=tiny_inkling
+ pf_runs "pf_chain inkling '2 8 tiny_inkling/ref_long.json'" inkling "inkling runtime int8, cap 2" vk.log 200 COLI_VK_TIER_GB=0.0001 SNAP=tiny_inkling
+ pf_runs "pf_chain olmoe '8 8 olmoe_tiny/ref_long.json'" olmoe "olmoe" vk.log 200 COLI_VK_TIER_GB=0.0001 SNAP=olmoe_tiny_c
+ pf_runs "pf_chain olmoe '1 4 olmoe_tiny/ref_long.json'" olmoe "olmoe 4-bit experts, cap 1" vk.log 200 COLI_VK_TIER_GB=0.0001 SNAP=olmoe_tiny_c
+ PF_PREFETCH=1 chain_gate olmoe "pf olmoe prefetch across chunks" 1 $PF_FORCE COLI_VK_TIER_GB=0.0001 COLI_VK_CHAIN_ROWS=64 SNAP=olmoe_tiny_c -- 8 8 olmoe_tiny/ref_long.json
+ PF_PREFETCH=1 pf_check olmoe vk.log "pf olmoe prefetch across chunks" forced
+ chain_gate olmoe "pf olmoe PILOT" 1 PILOT=1 WIDE=2 $PF_FORCE COLI_VK_TIER_GB=0.0001 SNAP=olmoe_tiny_c -- 2 8 olmoe_tiny/ref_long.json
+ pf_check olmoe vk.log "pf olmoe PILOT" forced
+ # sanitizers
+ make clean >/dev/null 2>&1 || true
+ make inkling olmoe VK=1 EXTRA_CFLAGS="-fsanitize=address,undefined -fno-omit-frame-pointer -g"
+ export ASAN_OPTIONS=detect_leaks=0:detect_stack_use_after_return=0 UBSAN_OPTIONS=print_stacktrace=1
+ pfsan inkling "asan pf inkling forced" COLI_VK_TIER_GB=0.0001 COLI_VK_CHAIN_ROWS=64 SNAP=tiny_inkling ./inkling 2 8 tiny_inkling/ref_long.json
+ pfsan olmoe "asan pf olmoe forced, staged" COLI_VK_STAGED=1 COLI_VK_TIER_GB=0.0001 COLI_VK_CHAIN_ROWS=64 SNAP=olmoe_tiny_c ./olmoe 1 8 olmoe_tiny/ref_long.json
+ unset OMP_NUM_THREADS
+ make clean >/dev/null 2>&1 || true
+}
+
+family_prefill_mimo_kimi() {
+ export OMP_NUM_THREADS=2
+ make mimo kimi_k3 VK=1
+ $PY tools/make_mimo_tiny.py --output ./mimo_tiny --force --vision
+ $PY tools/make_kimi_k3_tiny.py --output ./kimi_k3_tiny --force
+ local mids kids
+ mids=$(pf_ids 200 1 300 | tr ',' ' '); kids=$(pf_ids 200 1 300 | tr ',' ' ')
+ pf_runs "pf_mimo '$mids'" mimo "mimo f32" mimo-vk.err 200 MIMO_VK_EXPERTS=6 MIMO_DENSE_BITS=32
+ pf_runs "pf_mimo '$mids'" mimo "mimo fp8/bf16" mimo-vk.err 200 MIMO_VK_EXPERTS=6 MIMO_DENSE_BITS=0
+ local O="K3_BITS=32 K3_MLA_BITS=32 K3_HEAD_BITS=32 K3_IDOT=0 COLI_TEMP=0"
+ # shellcheck disable=SC2086
+ pf_runs "pf_k3 '$kids'" kimi_k3 "kimi_k3 f32" vk.log 200 COLI_VK_TIER_GB=0.0005 $O
+ PF_PREFETCH=1 pf_mimo "$mids" "pf mimo prefetch across chunks" 1 $PF_FORCE MIMO_VK_EXPERTS=6 MIMO_DENSE_BITS=32 COLI_VK_CHAIN_ROWS=50
+ PF_PREFETCH=1 pf_check mimo mimo-vk.err "pf mimo prefetch across chunks" forced
+ # shellcheck disable=SC2086
+ PF_PREFETCH=1 pf_k3 "$kids" "pf kimi_k3 prefetch across chunks" 1 $PF_FORCE COLI_VK_TIER_GB=0.0005 COLI_VK_CHAIN_ROWS=50 $O
+ PF_PREFETCH=1 pf_check kimi_k3 vk.log "pf kimi_k3 prefetch across chunks" forced
+ make clean >/dev/null 2>&1 || true
+ make mimo kimi_k3 VK=1 EXTRA_CFLAGS="-fsanitize=address,undefined -fno-omit-frame-pointer -g"
+ export ASAN_OPTIONS=detect_leaks=0:detect_stack_use_after_return=0 UBSAN_OPTIONS=print_stacktrace=1
+ pfsan mimo "asan pf mimo forced" MIMO_VK_EXPERTS=6 MIMO_DENSE_BITS=32 COLI_VK_CHAIN_ROWS=50 COLI_TEMP=0 ./mimo mimo_tiny --ids "$mids" --ngen 4
+ # shellcheck disable=SC2086
+ pfsan kimi_k3 "asan pf kimi_k3 forced, staged" COLI_VK_STAGED=1 COLI_VK_TIER_GB=0.0005 COLI_VK_CHAIN_ROWS=50 $O ./kimi_k3 kimi_k3_tiny --ids "$kids" --ngen 4
+ unset OMP_NUM_THREADS
+ make clean >/dev/null 2>&1 || true
+}
+
+family_prefill_glm() {
+ export OMP_NUM_THREADS=2 CAP_RAISE=0
+ make colibri glm53 VK=1
+ glm_fixtures
+ pf_ref glm_tiny/ref_long.json 200 1 250
+ local ids; ids=$(pf_ids 200 2 120)
+ pf_runs "pf_mla colibri '64 16 16'" colibri "colibri f32, teacher-forced" vk.log 208 COLI_VK_TIER_GB=0.0003 SNAP=glm_tiny REF=glm_tiny/ref_long.json TF=1
+ pf_runs "pf_mla colibri '2 16 16'" colibri "colibri int4-g64 experts" vk.log 200 COLI_VK_TIER_GB=0.0012 SNAP=glm_tiny_fmt4 REF=glm_tiny/ref_long.json
+ pf_runs "pf_mla glm53 '--model glm53_stream-i4 --ids $ids --greedy 4'" glm53 "glm53 f32 trunk" vk.log 200 COLI_VK_TIER_GB=0.0002 GLM53_BITS=32
+ pf_runs "pf_mla glm53 '--model glm53_stream-i4 --ids $ids --greedy 4'" glm53 "glm53 int4 trunk" vk.log 200 COLI_VK_TIER_GB=0.0002 GLM53_BITS=4
+ PF_PREFETCH=1 mla_gate colibri "pf colibri prefetch across chunks" 1 $PF_FORCE COLI_VK_TIER_GB=0.0003 COLI_VK_CHAIN_ROWS=64 SNAP=glm_tiny REF=glm_tiny/ref_long.json TF=1 -- 2 16 16
+ PF_PREFETCH=1 pf_check colibri vk.log "pf colibri prefetch across chunks" forced
+ PF_PREFETCH=1 mla_gate glm53 "pf glm53 prefetch across chunks" 1 $PF_FORCE COLI_VK_TIER_GB=0.0002 COLI_VK_CHAIN_ROWS=64 GLM53_BITS=4 GLM53_PREFILL_CHUNK=200 -- --model glm53_stream-i4 --ids "$ids" --greedy 4
+ PF_PREFETCH=1 pf_check glm53 vk.log "pf glm53 prefetch across chunks" forced
+ make clean >/dev/null 2>&1 || true
+ make colibri glm53 VK=1 EXTRA_CFLAGS="-fsanitize=address,undefined -fno-omit-frame-pointer -g"
+ export ASAN_OPTIONS=detect_leaks=0:detect_stack_use_after_return=0 UBSAN_OPTIONS=print_stacktrace=1
+ pfsan colibri "asan pf colibri forced" COLI_VK_TIER_GB=0.0003 COLI_VK_CHAIN_ROWS=64 SNAP=glm_tiny REF=glm_tiny/ref_long.json TF=1 ./colibri 2 16 16
+ pfsan glm53 "asan pf glm53 forced, staged" COLI_VK_STAGED=1 COLI_VK_TIER_GB=0.0002 COLI_VK_CHAIN_ROWS=64 GLM53_BITS=4 ./glm53 --model glm53_stream-i4 --ids "$ids" --greedy 4
+ unset OMP_NUM_THREADS CAP_RAISE
+ make clean >/dev/null 2>&1 || true
+}
+
+family_prefill_deepseek() {
+ export OMP_NUM_THREADS=2
+ make deepseek_v41 VK=1
+ make deepseek-v4 VK=1
+ $PY tools/make_dsv41_tiny.py --out dsv41_vlong --emit-ref dsv41_vlong/ref.json --prompt-len 160 --max-new 6 > /dev/null
+ pf_v4_fixture
+ local ids; ids=$(pf_ids 300 3 120)
+ pf_runs "pf_v41 dsv41_vlong" deepseek_v41 "deepseek_v41" vk.log 160 COLI_VK_TIER_GB=0.0005
+ pf_runs "pf_v4 deepseek_v4_tiny_p ids:$ids:4" deepseek_v4 "deepseek_v4" v4-vk.err 300 COLI_VK_TIER_GB=0.0005
+ PF_PREFETCH=1 v41_gate "pf deepseek_v41 prefetch across chunks" SNAP=dsv41_vlong $PF_FORCE COLI_VK_TIER_GB=0.0005 COLI_VK_CHAIN_ROWS=64 -- 2 dsv41_vlong/ref.json
+ PF_PREFETCH=1 pf_check deepseek_v41 vk.log "pf deepseek_v41 prefetch across chunks" forced
+ PF_PREFETCH=1 v4_chain_gate "pf deepseek_v4 prefetch across chunks" deepseek_v4_tiny_p "ids:$ids:4" $PF_FORCE COLI_VK_TIER_GB=0.0005 COLI_VK_CHAIN_ROWS=100
+ PF_PREFETCH=1 pf_check deepseek_v4 v4-vk.err "pf deepseek_v4 prefetch across chunks" forced
+ make clean >/dev/null 2>&1 || true
+ make deepseek-v4-clean >/dev/null 2>&1 || true
+ local SAN="-fsanitize=address,undefined -fno-omit-frame-pointer -g"
+ make deepseek_v41 VK=1 EXTRA_CFLAGS="$SAN"
+ make deepseek-v4 VK=1 LTO=0 EXTRA_CFLAGS="$SAN" EXTRA_LDFLAGS="$SAN"
+ export ASAN_OPTIONS=detect_leaks=0:detect_stack_use_after_return=0 UBSAN_OPTIONS=print_stacktrace=1
+ pfsan deepseek_v41 "asan pf deepseek_v41 forced" COLI_VK_TIER_GB=0.0005 COLI_VK_CHAIN_ROWS=64 SNAP=dsv41_vlong ./deepseek_v41 2 dsv41_vlong/ref.json
+ pfsan deepseek_v4 "asan pf deepseek_v4 forced, staged" COLI_VK_STAGED=1 COLI_VK_TIER_GB=0.0005 COLI_VK_CHAIN_ROWS=100 ./deepseek_v4 ./deepseek_v4_tiny_p \
+ "$(v4_chain_prompt deepseek_v4_tiny_p "ids:$ids:4")" --raw-prompt --max-tokens 4
+ unset OMP_NUM_THREADS
+ make clean >/dev/null 2>&1 || true
+ make deepseek-v4-clean >/dev/null 2>&1 || true
+}
+# DeepSeek V4's tiny model at 512 positions, 8 experts: room for a prompt past its 128-row block
+pf_v4_fixture() {
+ $PY - tools/make_deepseek_v4_tiny.py <<'PY' > /dev/null
+import importlib.util, sys
+spec = importlib.util.spec_from_file_location("gen", sys.argv[1])
+gen = importlib.util.module_from_spec(spec); spec.loader.exec_module(gen)
+gen.EXPERTS = 8; gen.MAX_POSITIONS = 512
+sys.argv = ["make_deepseek_v4_tiny.py", "--output", "deepseek_v4_tiny_p", "--force"]
+gen.main()
+PY
+}
+# ---- the KV cache split between the device and the host (vk_kvsplit.h) ----------------
+# Past the device's budget each chain engine keeps a few blocks of every attention
+# layer's cache on the device and attends over the rest on the CPU, the two parts merged
+# through their softmax statistics. The tiny fixtures emulate a small device with
+# COLI_VK_KV_DEVICE_ROWS (rows a layer keeps) and COLI_VK_KV_BLOCK (positions a block),
+# over prompts and generations long enough that most steps have a host part.
+# kv_hostparts <engine> <log>: M from "[VK] <engine> chain: KV split: N layer steps, M with a host part"
+kv_hostparts() {
+ local n
+ n=$(sed -n "s/^\[VK\] $1 chain: KV split: [0-9]* layer steps, \([0-9]*\) with a host part.*/\1/p" "$2" | tail -1)
+ echo "${n:-0}"
+}
+# kv_gate <engine> <tag> <tol 0|1> <rows> <block> <env...> -- <argv...>: chain_gate (the
+# CPU's tokens, logits within 1e-4 of the largest with tol 1) with the split on, which
+# must have run steps with a host part.
+kv_gate() {
+ local eng=$1 tag=$2 tol=$3 rows=$4 blk=$5; shift 5
+ chain_gate "$eng" "$tag" "$tol" COLI_VK_KV_DEVICE_ROWS=$rows COLI_VK_KV_BLOCK=$blk "$@"
+ [ "$(kv_hostparts "$eng" vk.log)" -gt 0 ] || { cat vk.log; fail "$tag: the split never ran a host part"; }
+ echo " $tag: $(grep -a "KV split:" vk.log | tail -1 | sed 's/.*KV split: //')"
+}
+# kv_san <engine> <tag> <env and argv...>: a sanitized run under the split (the build is
+# the caller's): no diagnostic, and steps with a host part
+kv_san() {
+ local eng=$1 tag=$2; shift 2
+ rm -f chain.usage
+ env OMP_NUM_THREADS=2 COLI_USAGE=chain.usage COLI_VK_TIER_SYNC=1 COLI_VULKAN=1 COLI_VK_CHAIN=1 "$@" > san.log 2>&1 || true
+ if grep -qE "ERROR: AddressSanitizer|runtime error:" san.log; then cat san.log; fail "$tag: sanitizer diagnostic"; fi
+ [ "$(kv_hostparts "$eng" san.log)" -gt 0 ] || { cat san.log; fail "$tag: the split never ran a host part"; }
+ echo "OK $tag: sanitizers clean, $(kv_hostparts "$eng" san.log) layer steps with a host part"
+}
+
+# qwen36: the hybrid and the dense geometry, a 40-token prompt and 40 new tokens
+kv_qwen36_fixtures() {
+ local P; P=$($PY -c "print(','.join(str(3 + i * 7 % 60) for i in range(40)))")
+ $PY tools/make_qwen36_tiny.py --out qwen36_kv --ref-mode full --prompt-ids "$P" --max-new 40 --emit-ref qwen36_kv/ref_full.json
+ $PY tools/convert_qwen36.py --model qwen36_kv --out qwen36_kv_c --ebits 8
+ $PY tools/make_qwen36_tiny.py --geometry qwen38-27b-dense --out qwen38_27b_kv --ref-mode full --prompt-ids "$P" --max-new 40 \
+ --emit-ref qwen38_27b_kv/ref_full.json
+ $PY tools/convert_qwen36.py --model qwen38_27b_kv --out qwen38_27b_kv_c --ebits 8
+ $PY tools/make_qwen36_tiny.py --out qwen36_tiny --ref-mode full --emit-ref qwen36_tiny/ref_full.json
+ $PY tools/convert_qwen36.py --model qwen36_tiny --out qwen36_tiny_c --ebits 8
+}
+kv_split_qwen36() {
+ kv_qwen36_fixtures
+ local R=qwen36_kv/ref_full.json
+ kv_gate qwen36 "kv qwen36 16 rows on the device" 1 16 4 COLI_DENSE_I8=0 SNAP=qwen36_kv_c -- 8 8 $R
+ kv_gate qwen36 "kv qwen36 two blocks on the device" 1 8 4 COLI_DENSE_I8=0 SNAP=qwen36_kv_c -- 8 8 $R
+ kv_gate qwen36 "kv qwen36 prefill in chunks of 3" 1 16 4 COLI_VK_CHAIN_ROWS=3 COLI_DENSE_I8=0 SNAP=qwen36_kv_c -- 8 8 $R
+ kv_gate qwen36 "kv qwen36 cap 1, blocks of 8" 1 24 8 COLI_DENSE_I8=0 SNAP=qwen36_kv_c -- 1 8 $R
+ kv_gate qwen36 "kv qwen36 tier off, tiled GEMM" 1 24 4 COLI_VK_TIER=0 COLI_VK_GEMM_MIN_S=2 COLI_DENSE_I8=0 SNAP=qwen36_kv_c -- 8 8 $R
+ CPUENV=COLI_DENSE_IDOT=0 kv_gate qwen36 "kv qwen36 int8 trunk" 1 16 4 SNAP=qwen36_kv_c -- 8 8 $R
+ kv_gate qwen36 "kv qwen36 dense model" 1 16 4 COLI_DENSE_I8=0 SNAP=qwen38_27b_kv_c -- 8 8 qwen38_27b_kv/ref_full.json
+ CHAINMODE=2 kv_gate qwen36 "kv qwen36 prompts only" 1 16 4 COLI_DENSE_I8=0 SNAP=qwen36_kv_c -- 8 8 $R
+ # COLI_VK_KV_SPLIT=0: the whole mirrors, as before the split
+ chain_gate qwen36 "kv qwen36 split off" 1 COLI_VK_KV_SPLIT=0 COLI_VK_KV_DEVICE_ROWS=16 COLI_VK_KV_BLOCK=4 COLI_DENSE_I8=0 SNAP=qwen36_kv_c -- 8 8 $R
+ if grep -qa 'KV cache split' vk.log; then cat vk.log; fail "kv qwen36 split off: the split ran"; fi
+ # the device lost mid-decode with the split on: the DeltaNet state rebuilt on the CPU
+ lost_late qwen36 "kv qwen36 device lost" 20 rebuild COLI_VK_KV_DEVICE_ROWS=16 COLI_VK_KV_BLOCK=4 COLI_DENSE_I8=0 SNAP=qwen36_kv_c -- 8 8 $R
+ # serve sessions (pins, prompt-cache extensions, a divergent prompt, logprobs) and the
+ # prefix-reuse contract across the split
+ QWEN36_TINY=$PWD/qwen36_tiny_c COLI_VULKAN=1 COLI_VK_CHAIN=1 COLI_VK_KV_DEVICE_ROWS=8 COLI_VK_KV_BLOCK=4 \
+ COLI_USAGE=$PWD/chain.usage $PY tests/test_qwen36_prefix_serve.py
+ CHAIN_SERVE_EXPECT='KV split: [0-9]+ layer steps, [1-9][0-9]* with a host part' \
+ $PY tests/vulkan_chain_serve.py ./qwen36 qwen36_tiny_c COLI_DENSE_I8=0 COLI_VK_KV_DEVICE_ROWS=8 COLI_VK_KV_BLOCK=4
+}
+kv_split_qwen36_sanitize() {
+ kv_qwen36_fixtures
+ kv_san qwen36 "asan kv qwen36" COLI_VK_KV_DEVICE_ROWS=8 COLI_VK_KV_BLOCK=4 COLI_DENSE_I8=0 SNAP=qwen36_kv_c ./qwen36 8 8 qwen36_kv/ref_full.json
+ kv_san qwen36 "asan kv qwen36 chunks of 3" COLI_VK_KV_DEVICE_ROWS=16 COLI_VK_KV_BLOCK=4 COLI_VK_CHAIN_ROWS=3 SNAP=qwen36_kv_c \
+ ./qwen36 8 8 qwen36_kv/ref_full.json
+ kv_san qwen36 "asan kv qwen36 dense model" COLI_VK_KV_DEVICE_ROWS=16 COLI_VK_KV_BLOCK=4 COLI_DENSE_I8=0 SNAP=qwen38_27b_kv_c \
+ ./qwen36 8 8 qwen38_27b_kv/ref_full.json
+ $PY -c "import sys; sys.path.insert(0, 'tests'); from prefix_serve_harness import ensure_byte_tokenizer as t; from pathlib import Path; t(Path('qwen36_tiny_c'))"
+ CHAIN_SERVE_EXPECT='KV split: [0-9]+ layer steps, [1-9][0-9]* with a host part' \
+ $PY tests/vulkan_chain_serve.py ./qwen36 qwen36_tiny_c COLI_DENSE_I8=0 COLI_VK_KV_DEVICE_ROWS=8 COLI_VK_KV_BLOCK=4 > san.log 2>&1 || { cat san.log; fail "asan kv qwen36 serve"; }
+ if grep -qE "ERROR: AddressSanitizer|runtime error:" san.log; then cat san.log; fail "asan kv qwen36 serve: sanitizer diagnostic"; fi
+ echo "OK asan kv qwen36 serve: $(tail -1 san.log)"
+}
+
+# qwen38: a 40-token prompt and 40 new tokens (the generation the transformers reference
+# and the CPU agree on), with and without the MTP head; the serve fixture with a tokenizer
+kv_qwen38_fixtures() {
+ local P; P=$($PY -c "print(','.join(str(3 + (i * 9 + i * i * 2) % 60) for i in range(40)))")
+ $PY tools/make_qwen38_tiny.py --out qwen38_kv --prompt-ids "$P" --max-new 40
+ $PY tools/make_qwen38_tiny.py --out qwen38_kv_mtp --prompt-ids "$P" --max-new 40 --mtp
+ $PY tools/make_qwen38_tiny.py --out qwen38_tiny_mtp --mtp
+ $PY tools/make_edge_tiny_tokenizer.py --vocab-size 64 ./qwen38_tiny_mtp # serve speaks text
+}
+kv_split_qwen38() {
+ kv_qwen38_fixtures
+ local R=qwen38_kv/ref.json M=qwen38_kv_mtp/ref.json batch f
+ for batch in 0 1; do
+ kv_gate qwen38 "kv qwen38 16 rows on the device, batch=$batch" 1 16 4 Q38_PREFILL_BATCH=$batch SNAP=qwen38_kv -- 4 8 $R
+ done
+ kv_gate qwen38 "kv qwen38 two blocks on the device" 1 8 4 SNAP=qwen38_kv -- 4 8 $R
+ kv_gate qwen38 "kv qwen38 prefill in chunks of 3" 1 16 4 COLI_VK_CHAIN_ROWS=3 SNAP=qwen38_kv -- 4 8 $R
+ # QSA's lists with blocks pinned by their reads (steps of 4 rows leave the window three
+ # of six slots): the device's part and the host's take the listed positions
+ COLI_VK_KV_PIN=1 kv_gate qwen38 "kv qwen38 blocks pinned by QSA's reads" 1 24 4 COLI_VK_CHAIN_ROWS=4 SNAP=qwen38_kv -- 4 8 $R
+ grep -qa 'KV split: .* [1-9][0-9]* blocks pinned by reads' vk.log || { cat vk.log; fail "kv qwen38: no block was pinned"; }
+ kv_gate qwen38 "kv qwen38 int8 trunk" 0 16 4 Q38_TRUNK_MIN_KB=0 SNAP=qwen38_kv -- 4 8 $R
+ kv_gate qwen38 "kv qwen38 tier off" 1 16 4 COLI_VK_TIER=0 SNAP=qwen38_kv -- 4 8 $R
+ # MTP verify (S = 2) and its rollback across the split: drafts as they come, every
+ # draft rejected, every one accepted, alternating
+ for f in "" reject accept mixed; do
+ kv_gate qwen38 "kv qwen38 MTP ${f:-drafting}" 1 16 4 Q38_MTP=1 Q38_MTP_FORCE=$f SNAP=qwen38_kv_mtp -- 2 8 $M
+ done
+ CHAINMODE=2 kv_gate qwen38 "kv qwen38 prompts only" 1 16 4 SNAP=qwen38_kv -- 4 8 $R
+ CHAINMODE=2 kv_gate qwen38 "kv qwen38 prompts only, MTP" 1 16 4 Q38_MTP=1 Q38_MTP_FORCE=mixed SNAP=qwen38_kv_mtp -- 2 8 $M
+ chain_gate qwen38 "kv qwen38 split off" 1 COLI_VK_KV_SPLIT=0 COLI_VK_KV_DEVICE_ROWS=16 COLI_VK_KV_BLOCK=4 SNAP=qwen38_kv -- 4 8 $R
+ if grep -qa 'KV cache split' vk.log; then cat vk.log; fail "kv qwen38 split off: the split ran"; fi
+ # the device lost with the split on, between steps and inside an MTP verify: the
+ # DeltaNet, conv and PLE states rebuilt on the CPU from the prefix record
+ lost_late qwen38 "kv qwen38 device lost" 10 rebuild COLI_VK_KV_DEVICE_ROWS=16 COLI_VK_KV_BLOCK=4 SNAP=qwen38_kv -- 4 8 $R
+ lost_late qwen38 "kv qwen38 device lost in a verify" 15 rebuild COLI_VK_KV_DEVICE_ROWS=16 COLI_VK_KV_BLOCK=4 Q38_MTP=1 \
+ Q38_MTP_FORCE=mixed SNAP=qwen38_kv_mtp -- 2 8 $M
+ # serve sessions (pins, prompt-cache extensions, a divergent prompt, logprobs) with
+ # and without MTP, and prompts only
+ local E='KV split: [0-9]+ layer steps, [1-9][0-9]* with a host part'
+ CHAIN_SERVE_EXPECT="$E" $PY tests/vulkan_chain_serve.py ./qwen38 qwen38_tiny_mtp Q38_MTP=1 COLI_VK_KV_DEVICE_ROWS=8 COLI_VK_KV_BLOCK=4
+ CHAIN_SERVE_EXPECT="$E" $PY tests/vulkan_chain_serve.py ./qwen38 qwen38_tiny_mtp COLI_VK_KV_DEVICE_ROWS=8 COLI_VK_KV_BLOCK=4
+ CHAIN_SERVE_EXPECT="$E" COLI_VK_CHAIN=2 $PY tests/vulkan_chain_serve.py ./qwen38 qwen38_tiny_mtp Q38_MTP=1 \
+ COLI_VK_KV_DEVICE_ROWS=8 COLI_VK_KV_BLOCK=4
+}
+kv_split_qwen38_sanitize() {
+ kv_qwen38_fixtures
+ kv_san qwen38 "asan kv qwen38" COLI_VK_KV_DEVICE_ROWS=8 COLI_VK_KV_BLOCK=4 SNAP=qwen38_kv ./qwen38 4 8 qwen38_kv/ref.json
+ COLI_VK_KV_PIN=1 kv_san qwen38 "asan kv qwen38 pinned blocks, chunks of 4" COLI_VK_KV_DEVICE_ROWS=24 COLI_VK_KV_BLOCK=4 COLI_VK_CHAIN_ROWS=4 \
+ SNAP=qwen38_kv ./qwen38 4 8 qwen38_kv/ref.json
+ kv_san qwen38 "asan kv qwen38 MTP mixed" COLI_VK_KV_DEVICE_ROWS=16 COLI_VK_KV_BLOCK=4 Q38_MTP=1 Q38_MTP_FORCE=mixed \
+ SNAP=qwen38_kv_mtp ./qwen38 2 8 qwen38_kv_mtp/ref.json
+ CHAIN_SERVE_EXPECT='KV split: [0-9]+ layer steps, [1-9][0-9]* with a host part' \
+ $PY tests/vulkan_chain_serve.py ./qwen38 qwen38_tiny_mtp Q38_MTP=1 COLI_VK_KV_DEVICE_ROWS=8 COLI_VK_KV_BLOCK=4 \
+ > san.log 2>&1 || { cat san.log; fail "asan kv qwen38 serve"; }
+ if grep -qE "ERROR: AddressSanitizer|runtime error:" san.log; then cat san.log; fail "asan kv qwen38 serve: sanitizer diagnostic"; fi
+ echo "OK asan kv qwen38 serve: $(tail -1 san.log)"
+}
+
+# olmoe: the tiny checkpoint and, from it, a 40-token prompt's 40-token generation by
+# transformers (the oracle the engine's run checks)
+kv_olmoe_fixtures() {
+ $PY tools/make_olmoe_tiny.py --output olmoe_tiny --force
+ $PY tools/convert_olmoe_merged.py --model olmoe_tiny --out olmoe_tiny_c
+ $PY tools/make_edge_tiny_tokenizer.py --vocab-size 128 olmoe_tiny_c
+ $PY - <<'PY'
+import json, torch
+from transformers import OlmoeForCausalLM
+torch.set_num_threads(1)
+m = OlmoeForCausalLM.from_pretrained("olmoe_tiny").eval().float()
+prompt = [(17 + i * 21 + (i * i) % 13) % 125 + 2 for i in range(40)]
+with torch.no_grad():
+ g = m.generate(torch.tensor([prompt]), max_new_tokens=40, do_sample=False, use_cache=True)[0].tolist()
+json.dump({"prompt_ids": prompt, "full_ids": g}, open("olmoe_tiny/ref_olmoe_long.json", "w"))
+PY
+}
+kv_split_olmoe() {
+ kv_olmoe_fixtures
+ local R=olmoe_tiny/ref_olmoe_long.json
+ kv_gate olmoe "kv olmoe 16 rows on the device" 1 16 4 SNAP=olmoe_tiny_c -- 8 8 $R
+ kv_gate olmoe "kv olmoe two blocks on the device" 1 8 4 SNAP=olmoe_tiny_c -- 8 8 $R
+ kv_gate olmoe "kv olmoe prefill in chunks of 3" 1 16 4 COLI_VK_CHAIN_ROWS=3 SNAP=olmoe_tiny_c -- 8 8 $R
+ kv_gate olmoe "kv olmoe cap 1, blocks of 8" 1 24 8 SNAP=olmoe_tiny_c -- 1 8 $R
+ kv_gate olmoe "kv olmoe 4-bit experts" 1 16 4 SNAP=olmoe_tiny_c -- 8 4 $R
+ kv_gate olmoe "kv olmoe PILOT cap 2" 1 16 4 PILOT=1 WIDE=2 SNAP=olmoe_tiny_c -- 2 8 $R
+ kv_gate olmoe "kv olmoe tier off, tiled GEMM" 1 24 4 COLI_VK_TIER=0 COLI_VK_GEMM_MIN_S=2 SNAP=olmoe_tiny_c -- 8 8 $R
+ CHAINMODE=2 kv_gate olmoe "kv olmoe prompts only" 1 16 4 SNAP=olmoe_tiny_c -- 8 8 $R
+ chain_gate olmoe "kv olmoe split off" 1 COLI_VK_KV_SPLIT=0 COLI_VK_KV_DEVICE_ROWS=16 COLI_VK_KV_BLOCK=4 SNAP=olmoe_tiny_c -- 8 8 $R
+ if grep -qa 'KV cache split' vk.log; then cat vk.log; fail "kv olmoe split off: the split ran"; fi
+ # the device lost mid-decode with the split on: the CPU redoes the step (nothing to rebuild)
+ lost_late olmoe "kv olmoe device lost" 7 redo COLI_VK_KV_DEVICE_ROWS=16 COLI_VK_KV_BLOCK=4 SNAP=olmoe_tiny_c -- 8 8 $R
+ # serve sessions and the prefix-reuse and dashboard contracts across the split
+ CHAIN_SERVE_EXPECT='KV split: [0-9]+ layer steps, [1-9][0-9]* with a host part' \
+ $PY tests/vulkan_chain_serve.py ./olmoe olmoe_tiny_c COLI_VK_KV_DEVICE_ROWS=8 COLI_VK_KV_BLOCK=4
+ CHAIN_SERVE_EXPECT='KV split: [0-9]+ layer steps, [1-9][0-9]* with a host part' \
+ $PY tests/vulkan_chain_serve.py ./olmoe olmoe_tiny_c PILOT=1 WIDE=2 COLI_VK_KV_DEVICE_ROWS=8 COLI_VK_KV_BLOCK=4
+ COLI_VULKAN=1 COLI_VK_CHAIN=1 COLI_VK_TIER_SYNC=1 COLI_VK_TIER_BALANCE=0 COLI_VK_KV_DEVICE_ROWS=8 COLI_VK_KV_BLOCK=4 \
+ OLMOE_TINY=olmoe_tiny_c $PY -m unittest tests.test_olmoe_prefix_serve tests.test_olmoe_dashboard_hits
+}
+kv_split_olmoe_sanitize() {
+ kv_olmoe_fixtures
+ local R=olmoe_tiny/ref_olmoe_long.json
+ kv_san olmoe "asan kv olmoe" COLI_VK_KV_DEVICE_ROWS=8 COLI_VK_KV_BLOCK=4 SNAP=olmoe_tiny_c ./olmoe 8 8 $R
+ kv_san olmoe "asan kv olmoe chunks of 3, PILOT" COLI_VK_KV_DEVICE_ROWS=16 COLI_VK_KV_BLOCK=4 COLI_VK_CHAIN_ROWS=3 PILOT=1 WIDE=2 \
+ SNAP=olmoe_tiny_c ./olmoe 2 8 $R
+ CHAIN_SERVE_EXPECT='KV split: [0-9]+ layer steps, [1-9][0-9]* with a host part' \
+ $PY tests/vulkan_chain_serve.py ./olmoe olmoe_tiny_c COLI_VK_KV_DEVICE_ROWS=8 COLI_VK_KV_BLOCK=4 > san.log 2>&1 || { cat san.log; fail "asan kv olmoe serve"; }
+ if grep -qE "ERROR: AddressSanitizer|runtime error:" san.log; then cat san.log; fail "asan kv olmoe serve: sanitizer diagnostic"; fi
+ echo "OK asan kv olmoe serve: $(tail -1 san.log)"
+}
+
+# inkling: the tiny model with a 40-token prompt and 40 new tokens (its global layer
+# attends over all 80 positions, the bias bank reaching back 32), its expert containers,
+# the dense-int4g64 container and the model at D = 6144
+kv_inkling_fixtures() {
+ $PY tools/make_tiny_inkling.py tiny_inkling
+ $PY - tiny_inkling <<'PY'
+import json, sys, torch
+from transformers import InklingForCausalLM
+d = sys.argv[1]
+m = InklingForCausalLM.from_pretrained(d).eval().float()
+prompt = [(3 + i * 7) % 250 for i in range(40)]
+with torch.no_grad():
+ full = m.generate(torch.tensor([prompt]), max_new_tokens=40, do_sample=False, use_cache=True)[0].tolist()
+ tf = m(torch.tensor([full])).logits[0].argmax(-1).tolist()
+json.dump({"prompt_ids": prompt, "full_ids": full, "tf_pred": tf}, open(f"{d}/ref_long.json", "w"))
+PY
+ $PY - <<'PY'
+import importlib.util, os, shutil
+from safetensors.torch import load_file, save_file
+spec = importlib.util.spec_from_file_location("conv", "tools/convert_inkling_int4.py")
+conv = importlib.util.module_from_spec(spec); spec.loader.exec_module(conv)
+t = load_file("tiny_inkling/model.safetensors")
+if "model.embed_tokens.embed_norm.weight" in t:
+ t["model.embed_norm.weight"] = t.pop("model.embed_tokens.embed_norm.weight")
+for d in ("tiny_inkling_hf", "tiny_inkling_x-tml", "tiny_inkling_x-pass", "tiny_inkling_x-i4", "tiny_inkling_x-i8"):
+ shutil.rmtree(d, ignore_errors=True)
+os.makedirs("tiny_inkling_hf")
+save_file(t, "tiny_inkling_hf/model.safetensors")
+shutil.copy("tiny_inkling/config.json", "tiny_inkling_hf/")
+conv.selftest_e2e("tiny_inkling_hf", "tiny_inkling_x") # tiny_inkling_x-i4: int4 experts
+conv.convert_dir("tiny_inkling_x-tml", "tiny_inkling_x-i8", 8) # int8 experts
+PY
+ rm -rf tiny_inkling_q && cp -r tiny_inkling tiny_inkling_q
+ $PY - tools/convert_inkling_dense_int4.py tiny_inkling_q <<'PY'
+import importlib.util, sys
+spec = importlib.util.spec_from_file_location("conv", sys.argv[1])
+conv = importlib.util.module_from_spec(spec); spec.loader.exec_module(conv)
+conv.MIN_ELEMS = 0
+conv.ATTN_BITS = 4
+base = conv.classify
+conv.classify = lambda n, s, d: "int8" if n.endswith(".mlp.down_proj.weight") else base(n, s, d)
+sys.argv = ["convert_inkling_dense_int4.py", "--dir", sys.argv[2]]
+conv.main()
+PY
+ mkdir -p tiny_inkling_q/dense-int4g64
+ mv tiny_inkling_q/dense-int4g64.safetensors tiny_inkling_q/dense-int4g64/dense.safetensors
+ $PY tools/make_tiny_inkling.py tiny_inkling_wide --wide
+ $PY -c 'import sys; from pathlib import Path; sys.path.insert(0, "."); from tests.test_inkling_prefix_serve import ensure_tokenizer; ensure_tokenizer(Path("tiny_inkling"))'
+}
+kv_split_inkling() {
+ kv_inkling_fixtures
+ local R=tiny_inkling/ref_long.json fx
+ kv_gate inkling "kv inkling 16 rows on the device" 1 16 4 SNAP=tiny_inkling -- 8 0 $R
+ kv_gate inkling "kv inkling two blocks on the device" 1 8 4 SNAP=tiny_inkling -- 8 0 $R
+ kv_gate inkling "kv inkling cap 1, blocks of 8" 1 24 8 SNAP=tiny_inkling -- 1 0 $R
+ for fx in tiny_inkling_q tiny_inkling_x-i4 tiny_inkling_x-i8; do
+ kv_gate inkling "kv inkling $fx" 1 16 4 SNAP=$fx -- 8 0 $R
+ done
+ kv_gate inkling "kv inkling runtime int4" 1 16 4 SNAP=tiny_inkling -- 2 4 $R
+ kv_gate inkling "kv inkling TOPP" 1 16 4 TOPP=0.3 SNAP=tiny_inkling_x-i4 -- 2 0 $R
+ kv_gate inkling "kv inkling tier off" 1 16 4 COLI_VK_TIER=0 SNAP=tiny_inkling -- 8 0 $R
+ kv_gate inkling "kv inkling prefill in chunks of 3" 1 16 4 COLI_VK_CHAIN_ROWS=3 SNAP=tiny_inkling -- 8 0 $R
+ kv_gate inkling "kv inkling tiled GEMM" 1 24 4 COLI_VK_GEMM_MIN_S=2 SNAP=tiny_inkling -- 8 0 $R
+ CHAINMODE=2 kv_gate inkling "kv inkling prompts only" 1 16 4 SNAP=tiny_inkling -- 8 0 $R
+ kv_gate inkling "kv inkling D=6144" 1 8 4 SNAP=tiny_inkling_wide -- 8 0 tiny_inkling_wide/ref_inkling.json
+ # COLI_VK_KV_SPLIT=0: the whole mirrors, as before the split
+ chain_gate inkling "kv inkling split off" 1 COLI_VK_KV_SPLIT=0 COLI_VK_KV_DEVICE_ROWS=16 COLI_VK_KV_BLOCK=4 SNAP=tiny_inkling -- 8 0 $R
+ if grep -qa 'KV cache split' vk.log; then cat vk.log; fail "kv inkling split off: the split ran"; fi
+ # the device lost mid-decode with the split on: K/V and convolution states rebuilt on the CPU
+ lost_late inkling "kv inkling device lost" 20 rebuild COLI_VK_KV_DEVICE_ROWS=16 COLI_VK_KV_BLOCK=4 SNAP=tiny_inkling -- 8 0 $R
+ # serve sessions (pins, prompt-cache extensions, a divergent prompt, logprobs), prompts
+ # only, and the prefix-reuse and dashboard contracts, across the split
+ CHAIN_SERVE_EXPECT='KV split: [0-9]+ layer steps, [1-9][0-9]* with a host part' \
+ $PY tests/vulkan_chain_serve.py ./inkling tiny_inkling INK_PREFIX_LOG=1 COLI_VK_KV_DEVICE_ROWS=8 COLI_VK_KV_BLOCK=4
+ COLI_VK_CHAIN=2 CHAIN_SERVE_EXPECT='KV split: [0-9]+ layer steps, [1-9][0-9]* with a host part' \
+ $PY tests/vulkan_chain_serve.py ./inkling tiny_inkling INK_PREFIX_LOG=1 COLI_VK_KV_DEVICE_ROWS=8 COLI_VK_KV_BLOCK=4
+ COLI_VULKAN=1 COLI_VK_CHAIN=1 COLI_VK_TIER_SYNC=1 COLI_VK_TIER_BALANCE=0 COLI_VK_KV_DEVICE_ROWS=8 COLI_VK_KV_BLOCK=4 \
+ INKLING_TINY=tiny_inkling $PY -m unittest tests.test_inkling_prefix_serve tests.test_inkling_dashboard_hits
+}
+kv_split_inkling_sanitize() {
+ kv_inkling_fixtures
+ local R=tiny_inkling/ref_long.json
+ kv_san inkling "asan kv inkling" COLI_VK_KV_DEVICE_ROWS=8 COLI_VK_KV_BLOCK=4 SNAP=tiny_inkling ./inkling 8 0 $R
+ kv_san inkling "asan kv inkling chunks of 3" COLI_VK_KV_DEVICE_ROWS=16 COLI_VK_KV_BLOCK=4 COLI_VK_CHAIN_ROWS=3 SNAP=tiny_inkling \
+ ./inkling 8 0 $R
+ kv_san inkling "asan kv inkling int4 experts" COLI_VK_KV_DEVICE_ROWS=16 COLI_VK_KV_BLOCK=4 SNAP=tiny_inkling_x-i4 ./inkling 2 0 $R
+ CHAIN_SERVE_EXPECT='KV split: [0-9]+ layer steps, [1-9][0-9]* with a host part' \
+ $PY tests/vulkan_chain_serve.py ./inkling tiny_inkling INK_PREFIX_LOG=1 COLI_VK_KV_DEVICE_ROWS=8 COLI_VK_KV_BLOCK=4 \
+ > san.log 2>&1 || { cat san.log; fail "asan kv inkling serve"; }
+ if grep -qE "ERROR: AddressSanitizer|runtime error:" san.log; then cat san.log; fail "asan kv inkling serve: sanitizer diagnostic"; fi
+ echo "OK asan kv inkling serve: $(tail -1 san.log)"
+}
+
+# mimo: its full-attention layers split (the windowed ones keep their rings); the tiny
+# fixture's long case (a 72-token prompt) and the picture's, each with 40 new tokens
+kv_mimo_fixtures() {
+ $PY tools/make_mimo_tiny.py --output ./mimo_tiny --force --vision
+ MGRID=$($PY -c "import json;i=json.load(open('mimo_tiny/ref.json'))['image'];print(i['grid_h'],i['grid_w'])")
+ mimo_served_fixture
+}
+# mimo_kv_gate <tag> <case> <rows> <block> <env...>: the case's prompt and 40 new tokens,
+# the CPU's tokens, then every position's logits over the CPU's whole sequence within
+# 1e-4 of the largest (CHAINMODE: COLI_VK_CHAIN's value); steps with a host part
+mimo_kv_gate() {
+ local tag=$1 c=$2 rows=$3 blk=$4 extra=() P lg; shift 4
+ [ "$c" = image ] && extra=(--image mimo_tiny/patches.f32 --grid $MGRID)
+ P=$(mimo_ids $c prompt_ids)
+ local kv=(COLI_VULKAN=1 COLI_VK_CHAIN=${CHAINMODE:-1} COLI_VK_TIER_SYNC=1 COLI_VK_KV_DEVICE_ROWS=$rows COLI_VK_KV_BLOCK=$blk)
+ env "$@" COLI_TEMP=0 ./mimo mimo_tiny --ids "$P" --ngen 40 "${extra[@]}" > mimo-cpu.txt 2>/dev/null
+ env "$@" "${kv[@]}" COLI_TEMP=0 ./mimo mimo_tiny --ids "$P" --ngen 40 "${extra[@]}" > mimo-vk.txt 2> mimo-vk.err
+ [ -s mimo-cpu.txt ] && cmp -s mimo-cpu.txt mimo-vk.txt || { cat mimo-cpu.txt mimo-vk.txt mimo-vk.err; fail "$tag $c: the chain's tokens differ from the CPU"; }
+ [ "$(kv_hostparts mimo mimo-vk.err)" -gt 0 ] || { cat mimo-vk.err; fail "$tag $c: the split never ran a host part"; }
+ local hp; hp=$(grep -a "KV split:" mimo-vk.err | tail -1 | sed 's/.*KV split: //')
+ rm -f cpu.f32 vk.f32
+ env "$@" MIMO_LOGITS=cpu.f32 ./mimo mimo_tiny --ids "$P $(cat mimo-cpu.txt)" --ngen 0 "${extra[@]}" > /dev/null 2>&1
+ env "$@" "${kv[@]}" MIMO_LOGITS=vk.f32 ./mimo mimo_tiny --ids "$P $(cat mimo-cpu.txt)" --ngen 0 "${extra[@]}" > /dev/null 2> mimo-vk.err
+ lg=$(logits_close cpu.f32 vk.f32) || { echo "$lg"; fail "$tag $c: logits"; }
+ echo "OK $tag $c: tokens = CPU, $lg; $hp"
+}
+kv_split_mimo() {
+ kv_mimo_fixtures
+ local bits c
+ for bits in 32 0 8; do mimo_kv_gate "kv mimo bits=$bits, 16 rows on the device" long 16 4 MIMO_DENSE_BITS=$bits; done
+ mimo_kv_gate "kv mimo two blocks on the device" long 8 4 MIMO_DENSE_BITS=32
+ mimo_kv_gate "kv mimo a picture" image 16 4 MIMO_DENSE_BITS=0
+ mimo_kv_gate "kv mimo one token at a time" long 16 4 MIMO_DENSE_BITS=32 MIMO_CHUNK=1
+ mimo_kv_gate "kv mimo blocks of 3" long 16 4 MIMO_DENSE_BITS=32 MIMO_CHUNK=3
+ mimo_kv_gate "kv mimo chunks of 3 rows" long 24 8 MIMO_DENSE_BITS=32 COLI_VK_CHAIN_ROWS=3
+ mimo_kv_gate "kv mimo tier off, tiled GEMM" long 16 4 MIMO_DENSE_BITS=0 COLI_VK_TIER=0 COLI_VK_GEMM_MIN_S=2
+ CHAINMODE=2 mimo_kv_gate "kv mimo prompts only" long 16 4 MIMO_DENSE_BITS=32
+ # COLI_VK_KV_SPLIT=0: the whole mirrors, as before the split
+ mimo_chain_gate "kv mimo split off" long MIMO_DENSE_BITS=32 COLI_VK_KV_SPLIT=0 COLI_VK_KV_DEVICE_ROWS=16 COLI_VK_KV_BLOCK=4
+ if grep -qa 'KV cache split' mimo-vk.err; then cat mimo-vk.err; fail "kv mimo split off: the split ran"; fi
+ # the device lost with the split on (mid-decode; inside a prompt in chunks of 3): the
+ # host's caches hold whole steps, the CPU runs on from where they end
+ mimo_lost_gate "kv mimo device lost mid-decode" long 3 MIMO_DENSE_BITS=32 COLI_VK_KV_DEVICE_ROWS=16 COLI_VK_KV_BLOCK=4
+ mimo_lost_gate "kv mimo device lost inside a chunked prompt" long 100 MIMO_DENSE_BITS=32 COLI_VK_CHAIN_ROWS=3 \
+ COLI_VK_KV_DEVICE_ROWS=16 COLI_VK_KV_BLOCK=4
+ # serve sessions (photos restored, prompt-cache extensions, a divergent prompt,
+ # logprobs) frame for frame within the family's 1e-3, both modes and a lost device
+ local k
+ for k in 1 2; do
+ CHAIN_SERVE_EXPECT='KV split: [0-9]+ layer steps, [1-9][0-9]* with a host part' CHAIN_SERVE_TOL=1e-3 COLI_VK_CHAIN=$k \
+ $PY tests/vulkan_chain_serve.py ./mimo mimo_tiny_served OMP_NUM_THREADS=2 COLI_VK_KV_DEVICE_ROWS=8 COLI_VK_KV_BLOCK=4
+ done
+ CHAIN_SERVE_TOL=1e-3 $PY tests/vulkan_chain_serve.py ./mimo mimo_tiny_served OMP_NUM_THREADS=2 COLI_VK_CHAIN_FAULT=60 \
+ COLI_VK_KV_DEVICE_ROWS=8 COLI_VK_KV_BLOCK=4
+}
+kv_split_mimo_sanitize() {
+ kv_mimo_fixtures
+ local P; P=$(mimo_ids long prompt_ids)
+ kv_san mimo "asan kv mimo" COLI_VK_KV_DEVICE_ROWS=8 COLI_VK_KV_BLOCK=4 MIMO_DENSE_BITS=32 COLI_TEMP=0 ./mimo mimo_tiny --ids "$P" --ngen 20
+ kv_san mimo "asan kv mimo blocks of 3, the FP8 trunk" COLI_VK_KV_DEVICE_ROWS=16 COLI_VK_KV_BLOCK=4 MIMO_DENSE_BITS=0 MIMO_CHUNK=3 \
+ COLI_TEMP=0 ./mimo mimo_tiny --ids "$P" --ngen 10
+ CHAIN_SERVE_EXPECT='KV split: [0-9]+ layer steps, [1-9][0-9]* with a host part' CHAIN_SERVE_TOL=1e-3 \
+ $PY tests/vulkan_chain_serve.py ./mimo mimo_tiny_served OMP_NUM_THREADS=2 COLI_VK_KV_DEVICE_ROWS=8 COLI_VK_KV_BLOCK=4 \
+ > san.log 2>&1 || { cat san.log; fail "asan kv mimo serve"; }
+ if grep -qE "ERROR: AddressSanitizer|runtime error:" san.log; then cat san.log; fail "asan kv mimo serve: sanitizer diagnostic"; fi
+ echo "OK asan kv mimo serve: $(tail -1 san.log)"
+}
+
+# kv_mla_gate <engine> <tag> <tol 0|1> <rows> <block> <env...> -- <argv...>: mla_gate (the
+# CPU's tokens, logits within 1e-4 with tol 1; FAULT_BACK, REBUILD as there) with the
+# split on, which must have run steps with a host part (PINNED=1: and pinned blocks)
+kv_mla_gate() {
+ local eng=$1 tag=$2 tol=$3 rows=$4 blk=$5; shift 5
+ mla_gate "$eng" "$tag" "$tol" COLI_VK_KV_DEVICE_ROWS=$rows COLI_VK_KV_BLOCK=$blk "$@"
+ [ "$(kv_hostparts "$eng" vk.log)" -gt 0 ] || { cat vk.log; fail "$tag: the split never ran a host part"; }
+ if [ "${PINNED:-0}" = 1 ]; then grep -qa 'KV split: .* [1-9][0-9]* blocks pinned by reads' vk.log || { cat vk.log; fail "$tag: no block was pinned"; }; fi
+ echo " $tag: $(grep -a "KV split:" vk.log | tail -1 | sed 's/.*KV split: //')"
+}
+# colibri (GLM-5.2): the 32-token oracle over 8 or 16 rows on the device; DSA's lists
+kv_split_colibri() {
+ [ -f glm_tiny_serve/tokenizer.json ] || glm_chain_fixtures
+ export CAP_RAISE=0
+ local G="SNAP=glm_tiny REF=ref_glm.json"
+ kv_mla_gate colibri "kv colibri decode, 8 rows on the device" 1 8 4 $G -- 64 16 16
+ kv_mla_gate colibri "kv colibri cap 1, blocks of 2" 1 8 2 $G -- 1 16 16
+ kv_mla_gate colibri "kv colibri prefill in chunks of 3" 1 8 4 $G TF=1 COLI_VK_CHAIN_ROWS=3 -- 64 16 16
+ kv_mla_gate colibri "kv colibri 4-bit trunk and experts" 1 8 4 $G IDOT=0 -- 2 4 4
+ kv_mla_gate colibri "kv colibri i4 container" 1 8 4 SNAP=glm_tiny_i4 REF=glm_tiny_i4/ref_glm.json IDOT=0 -- 1 4 4
+ # DSA's lists: the device skips the positions it does not hold, the host takes them,
+ # and the blocks the lists read most are pinned (chunks of 3 leave room for them)
+ COLI_VK_KV_PIN=1 PINNED=1 kv_mla_gate colibri "kv colibri DSA top-4, pinned blocks" 1 16 4 $G DSA_TOPK=4 COLI_VK_CHAIN_ROWS=3 -- 64 16 16
+ kv_mla_gate colibri "kv colibri DSA top-4, prefill in chunks of 5" 1 8 4 $G DSA_TOPK=4 TF=1 COLI_VK_CHAIN_ROWS=5 -- 64 16 16
+ kv_mla_gate colibri "kv colibri DSA_FORCE" 1 8 4 $G DSA_FORCE=1 -- 64 16 16
+ # drafts: the verify rows over the split, rejected rows rewritten
+ kv_mla_gate colibri "kv colibri n-gram drafts" 1 8 4 $G DRAFT=3 -- 64 16 16
+ kv_mla_gate colibri "kv colibri MTP depth 2" 1 8 4 SNAP=glm_tiny_mtp REF=glm_tiny_mtp/ref_glm.json DRAFT=2 -- 64 16 16
+ kv_mla_gate colibri "kv colibri tier off" 1 8 4 $G COLI_VK_TIER=0 -- 64 16 16
+ # prompts only: the prompt and the drafts' verify rows on the device, decode on the CPU
+ CHAINMODE=2 kv_mla_gate colibri "kv colibri prompts only, drafts" 1 8 4 $G DRAFT=3 -- 64 16 16
+ # COLI_VK_KV_SPLIT=0: the whole mirrors, as before the split
+ mla_gate colibri "kv colibri split off" 1 COLI_VK_KV_SPLIT=0 COLI_VK_KV_DEVICE_ROWS=8 COLI_VK_KV_BLOCK=4 $G -- 64 16 16
+ if grep -qa 'KV cache split' vk.log; then cat vk.log; fail "kv colibri split off: the split ran"; fi
+ FAULT_BACK=12 kv_mla_gate colibri "kv colibri device lost mid-decode" 1 8 4 $G -- 64 16 16
+ # serve sessions frame for frame: pins, the prompt cache, the prefill read-out, two KV
+ # slots taking the device in turn, DSA's lists with drafts
+ local E='KV split: [0-9]+ layer steps, [1-9][0-9]* with a host part' K="COLI_VK_KV_DEVICE_ROWS=8 COLI_VK_KV_BLOCK=4"
+ # shellcheck disable=SC2086
+ {
+ CHAIN_SERVE_EXPECT="$E" CHAIN_SERVE_DIALECT=colibri $PY tests/vulkan_chain_serve.py ./colibri glm_tiny_serve SERVE_BATCH=1 IDOT=0 $K
+ CHAIN_SERVE_EXPECT="$E" CHAIN_SERVE_DIALECT=colibri $PY tests/vulkan_chain_serve.py ./colibri glm_tiny_serve SERVE_BATCH=1 IDOT=0 DSA_TOPK=4 DRAFT=3 $K
+ CHAIN_SERVE_EXPECT="$E" CHAIN_SERVE_SLOTS=2 CHAIN_SERVE_DIALECT=colibri $PY tests/vulkan_chain_serve.py ./colibri glm_tiny_serve SERVE_BATCH=1 IDOT=0 KV_SLOTS=2 $K
+ }
+ unset CAP_RAISE
+}
+kv_split_colibri_sanitize() {
+ [ -f glm_tiny_serve/tokenizer.json ] || glm_chain_fixtures
+ export CAP_RAISE=0
+ local K="COLI_VK_KV_DEVICE_ROWS=8 COLI_VK_KV_BLOCK=4 USAGE_SAVE=0"
+ # shellcheck disable=SC2086
+ {
+ kv_san colibri "asan kv colibri decode" $K SNAP=glm_tiny REF=ref_glm.json ./colibri 64 16 16
+ COLI_VK_KV_PIN=1 kv_san colibri "asan kv colibri DSA top-4, pinned, chunks of 3" COLI_VK_KV_DEVICE_ROWS=16 COLI_VK_KV_BLOCK=4 USAGE_SAVE=0 SNAP=glm_tiny \
+ REF=ref_glm.json DSA_TOPK=4 COLI_VK_CHAIN_ROWS=3 ./colibri 64 16 16
+ kv_san colibri "asan kv colibri MTP depth 2" $K SNAP=glm_tiny_mtp REF=glm_tiny_mtp/ref_glm.json DRAFT=2 ./colibri 64 16 16
+ CHAIN_SERVE_EXPECT='KV split: [0-9]+ layer steps, [1-9][0-9]* with a host part' CHAIN_SERVE_SLOTS=2 CHAIN_SERVE_DIALECT=colibri \
+ $PY tests/vulkan_chain_serve.py ./colibri glm_tiny_serve SERVE_BATCH=1 IDOT=0 KV_SLOTS=2 $K > san.log 2>&1 \
+ || { cat san.log; fail "asan kv colibri serve"; }
+ }
+ if grep -qE "ERROR: AddressSanitizer|runtime error:" san.log; then cat san.log; fail "asan kv colibri serve: sanitizer diagnostic"; fi
+ echo "OK asan kv colibri serve: $(tail -1 san.log)"
+ unset CAP_RAISE
+}
+
+# glm53 (GLM-5.3 Flash): its MLA layer's latent rows split, the k-pooled lists through
+# the split, the KDA layers untouched; the 100-token prompt and 8-token decodes
+kv_split_glm53() {
+ [ -f glm53_serve/tokenizer.json ] || glm_chain_fixtures
+ local ids bits M="--model glm53_stream-i4"
+ ids=$($PY -c "print(','.join(str((i*37+5)%120+2) for i in range(100)))")
+ kv_mla_gate glm53 "kv glm53 decode, 8 rows on the device" 1 8 4 GLM53_BITS=32 -- $M --ids 5,7,9,11,13,17,19,23 --greedy 8
+ for bits in 32 8 4; do
+ kv_mla_gate glm53 "kv glm53 100-token prompt, ${bits}-bit trunk" 1 16 4 GLM53_BITS=$bits -- $M --ids $ids --greedy 8
+ done
+ COLI_VK_KV_PIN=1 PINNED=1 kv_mla_gate glm53 "kv glm53 chain chunks of 3, pinned blocks" 1 16 4 GLM53_BITS=32 COLI_VK_CHAIN_ROWS=3 -- $M --ids $ids --greedy 8
+ kv_mla_gate glm53 "kv glm53 prefill chunks of 7, one cache slot" 1 16 4 GLM53_BITS=4 GLM53_PREFILL_CHUNK=7 GLM53_EXPERT_GB=0.000001 \
+ -- $M --ids $ids --greedy 4
+ kv_mla_gate glm53 "kv glm53 blocks of 2" 1 6 2 GLM53_BITS=32 -- $M --ids $ids --greedy 6
+ kv_mla_gate glm53 "kv glm53 resident experts" 1 16 4 GLM53_BITS=32 -- --model glm53_tiny --ids $ids --greedy 6
+ kv_mla_gate glm53 "kv glm53 tier off" 1 16 4 GLM53_BITS=32 COLI_VK_TIER=0 -- $M --ids $ids --greedy 6
+ kv_mla_gate glm53 "kv glm53 an image" 1 4 2 GLM53_BITS=32 -- --model glm53_mm_tiny --ids 103,117,268,268,268,268,120,121 \
+ --patches glm53_mm_tiny/patches.f32 --grid 4x4 --greedy 4
+ CHAINMODE=2 kv_mla_gate glm53 "kv glm53 prompts only" 1 16 4 GLM53_BITS=32 GLM53_PREFILL_CHUNK=16 -- $M --ids $ids --greedy 6
+ # COLI_VK_KV_SPLIT=0: the whole mirrors, as before the split
+ mla_gate glm53 "kv glm53 split off" 1 COLI_VK_KV_SPLIT=0 COLI_VK_KV_DEVICE_ROWS=16 COLI_VK_KV_BLOCK=4 GLM53_BITS=32 -- $M --ids $ids --greedy 4
+ if grep -qa 'KV cache split' vk.log; then cat vk.log; fail "kv glm53 split off: the split ran"; fi
+ FAULT_BACK=7 REBUILD=1 kv_mla_gate glm53 "kv glm53 device lost, the KDA state rebuilt" 1 16 4 GLM53_BITS=32 GLM53_PREFILL_CHUNK=16 \
+ -- $M --ids $ids --greedy 6
+ # serve sessions frame for frame (pins, the prompt cache, read-outs), two sessions
+ # taking the device in turn; a pin restored over rows another branch rewrote
+ local E='KV split: [0-9]+ layer steps, [1-9][0-9]* with a host part' K="COLI_VK_KV_DEVICE_ROWS=8 COLI_VK_KV_BLOCK=4"
+ # shellcheck disable=SC2086
+ {
+ CHAIN_SERVE_EXPECT="$E" CHAIN_SERVE_DIALECT=numeric $PY tests/vulkan_chain_serve.py ./glm53 glm53_serve GLM53_BITS=32 $K
+ CHAIN_SERVE_EXPECT="$E" CHAIN_SERVE_DIALECT=numeric $PY tests/vulkan_chain_serve.py ./glm53 glm53_serve GLM53_BITS=4 COLI_VK_CHAIN_ROWS=3 $K
+ CHAIN_SERVE_EXPECT="$E" CHAIN_SERVE_SLOTS=2 CHAIN_SERVE_DIALECT=numeric $PY tests/vulkan_chain_serve.py ./glm53 glm53_serve GLM53_BITS=32 KV_SLOTS=2 $K
+ env $K COLI_VULKAN=1 COLI_VK_CHAIN=1 COLI_USAGE=$PWD/chain.usage $PY tests/glm53_pin_branch_harness.py --binary ./glm53 --fixture glm53_mm_tiny
+ }
+}
+kv_split_glm53_sanitize() {
+ [ -f glm53_serve/tokenizer.json ] || glm_chain_fixtures
+ local ids K="COLI_VK_KV_DEVICE_ROWS=16 COLI_VK_KV_BLOCK=4 USAGE_SAVE=0"
+ ids=$($PY -c "print(','.join(str((i*37+5)%120+2) for i in range(100)))")
+ # shellcheck disable=SC2086
+ {
+ kv_san glm53 "asan kv glm53 100-token prompt" $K GLM53_BITS=32 ./glm53 --model glm53_stream-i4 --ids $ids --greedy 6
+ COLI_VK_KV_PIN=1 kv_san glm53 "asan kv glm53 int4 trunk, chunks of 3, pinned" $K GLM53_BITS=4 COLI_VK_CHAIN_ROWS=3 ./glm53 --model glm53_stream-i4 --ids $ids --greedy 6
+ kv_san glm53 "asan kv glm53 device lost, rebuilt" $K GLM53_BITS=32 GLM53_PREFILL_CHUNK=16 COLI_VK_CHAIN_FAULT=20 ./glm53 \
+ --model glm53_stream-i4 --ids $ids --greedy 6
+ CHAIN_SERVE_EXPECT='KV split: [0-9]+ layer steps, [1-9][0-9]* with a host part' CHAIN_SERVE_SLOTS=2 CHAIN_SERVE_DIALECT=numeric \
+ $PY tests/vulkan_chain_serve.py ./glm53 glm53_serve GLM53_BITS=32 KV_SLOTS=2 COLI_VK_KV_DEVICE_ROWS=8 COLI_VK_KV_BLOCK=4 > san.log 2>&1 \
+ || { cat san.log; fail "asan kv glm53 serve"; }
+ }
+ if grep -qE "ERROR: AddressSanitizer|runtime error:" san.log; then cat san.log; fail "asan kv glm53 serve: sanitizer diagnostic"; fi
+ echo "OK asan kv glm53 serve: $(tail -1 san.log)"
+}
+
+# kv_k3c_gate <tag> <rows> <block> <env...>: k3c_gate (kimi's three cases, logits within
+# 2e-3 of the largest) with the split on; the last case run (long, 72 prompt tokens)
+# must have had steps with a host part
+kv_k3c_gate() {
+ local tag=$1 rows=$2 blk=$3; shift 3
+ k3c_gate "$tag" "$@" COLI_VK_KV_DEVICE_ROWS=$rows COLI_VK_KV_BLOCK=$blk
+ [ "$(kv_hostparts kimi_k3 vk.log)" -gt 0 ] || { cat vk.log; fail "$tag: the split never ran a host part"; }
+ echo " $tag: $(grep -a "KV split:" vk.log | tail -1 | sed 's/.*KV split: //')"
+}
+# kimi_k3: its gated MLA layers (NoPE) split, the KDA layers untouched
+kv_split_kimi() {
+ [ -f kimi_k3_tiny/model.safetensors ] || $PY tools/make_kimi_k3_tiny.py --output ./kimi_k3_tiny --force
+ k3c_serve_fixture
+ local O="K3_BITS=32 K3_MLA_BITS=32 K3_HEAD_BITS=32 K3_IDOT=0 COLI_TEMP=0" b
+ # shellcheck disable=SC2086
+ {
+ kv_k3c_gate "kv kimi_k3 f32, 16 rows on the device" 16 4 $O
+ kv_k3c_gate "kv kimi_k3 f32, blocks of 2" 8 2 $O
+ kv_k3c_gate "kv kimi_k3 f32, tier off" 16 4 $O COLI_VK_TIER=0
+ for b in 8 4; do kv_k3c_gate "kv kimi_k3 ${b}-bit trunk" 16 4 K3_BITS=$b K3_MLA_BITS=$b K3_HEAD_BITS=$b K3_IDOT=0 COLI_TEMP=0; done
+ kv_k3c_gate "kv kimi_k3 prefill one token at a time" 16 4 $O K3_CHUNK=1
+ kv_k3c_gate "kv kimi_k3 chain chunks of 3" 16 4 $O COLI_VK_CHAIN_ROWS=3
+ CHAINMODE=2 kv_k3c_gate "kv kimi_k3 prompts only" 16 4 $O
+ # COLI_VK_KV_SPLIT=0: the whole mirrors, as before the split
+ K3C_CASES=long k3c_gate "kv kimi_k3 split off" $O COLI_VK_KV_SPLIT=0 COLI_VK_KV_DEVICE_ROWS=16 COLI_VK_KV_BLOCK=4
+ if grep -qa 'KV cache split' vk.log; then cat vk.log; fail "kv kimi_k3 split off: the split ran"; fi
+ # the device lost: mid-decode (the KDA state rebuilt), and inside the long prompt
+ FAULT_BACK=3 REBUILD=1 kv_k3c_gate "kv kimi_k3 device lost mid-decode" 16 4 $O
+ K3C_CASES=long FAULT_BACK=150 kv_k3c_gate "kv kimi_k3 device lost in the prompt" 16 4 $O
+ }
+ # serve sessions frame for frame with recurrent-state checkpoints, and the checkpoint
+ # tests, across the split
+ local S="K3_BITS=32 K3_MLA_BITS=32 K3_HEAD_BITS=32 K3_IDOT=0 K3_PREFIX_LOG=1 USAGE_SAVE=0" K="COLI_VK_KV_DEVICE_ROWS=8 COLI_VK_KV_BLOCK=4"
+ local E='KV split: [0-9]+ layer steps, [1-9][0-9]* with a host part'
+ # shellcheck disable=SC2086
+ {
+ CHAIN_SERVE_TOL=2e-2 CHAIN_SERVE_EXPECT="$E" $PY tests/vulkan_chain_serve.py ./kimi_k3 kimi_k3_serve $S $K
+ CHAIN_SERVE_TOL=2e-2 CHAIN_SERVE_EXPECT="$E" $PY tests/vulkan_chain_serve.py ./kimi_k3 kimi_k3_serve $S COLI_K3_CKPT=4 $K
+ CHAIN_SERVE_TOL=2e-2 CHAIN_SERVE_EXPECT="$E" $PY tests/vulkan_chain_serve.py ./kimi_k3 kimi_k3_serve K3_BITS=4 K3_IDOT=0 K3_PREFIX_LOG=1 \
+ USAGE_SAVE=0 COLI_VK_CHAIN_ROWS=3 $K
+ env $K COLI_USAGE=$PWD/k3c.usage USAGE_SAVE=0 COLI_VULKAN=1 COLI_VK_CHAIN=1 COLI_VK_TIER_SYNC=1 \
+ $PY tests/test_kimi_k3_ckpt.py --binary ./kimi_k3 --fixture ./kimi_k3_tiny
+ }
+}
+kv_split_kimi_sanitize() {
+ [ -f kimi_k3_tiny/model.safetensors ] || $PY tools/make_kimi_k3_tiny.py --output ./kimi_k3_tiny --force
+ k3c_serve_fixture
+ local O="K3_BITS=32 K3_MLA_BITS=32 K3_HEAD_BITS=32 K3_IDOT=0 COLI_TEMP=0 USAGE_SAVE=0" K="COLI_VK_KV_DEVICE_ROWS=16 COLI_VK_KV_BLOCK=4" ids
+ ids=$(k3c_ids long)
+ # shellcheck disable=SC2086
+ {
+ kv_san kimi_k3 "asan kv kimi_k3 f32" $K $O ./kimi_k3 kimi_k3_tiny --ids "$ids" --ngen 8
+ kv_san kimi_k3 "asan kv kimi_k3 int4 trunk, chunks of 3" $K K3_BITS=4 K3_IDOT=0 USAGE_SAVE=0 COLI_VK_CHAIN_ROWS=3 ./kimi_k3 kimi_k3_tiny \
+ --ids "$ids" --ngen 8
+ kv_san kimi_k3 "asan kv kimi_k3 prompts only" $K $O COLI_VK_CHAIN=2 ./kimi_k3 kimi_k3_tiny --ids "$ids" --ngen 8
+ CHAIN_SERVE_TOL=2e-2 CHAIN_SERVE_EXPECT='KV split: [0-9]+ layer steps, [1-9][0-9]* with a host part' \
+ $PY tests/vulkan_chain_serve.py ./kimi_k3 kimi_k3_serve K3_BITS=32 K3_MLA_BITS=32 K3_HEAD_BITS=32 K3_IDOT=0 K3_PREFIX_LOG=1 \
+ USAGE_SAVE=0 COLI_K3_CKPT=4 COLI_VK_KV_DEVICE_ROWS=8 COLI_VK_KV_BLOCK=4 > san.log 2>&1 || { cat san.log; fail "asan kv kimi_k3 serve"; }
+ }
+ if grep -qE "ERROR: AddressSanitizer|runtime error:" san.log; then cat san.log; fail "asan kv kimi_k3 serve: sanitizer diagnostic"; fi
+ echo "OK asan kv kimi_k3 serve: $(tail -1 san.log)"
+}
+
+# deepseek_v41: the compressed rows of its kv_source layers split (ratios 2 and 1 in the
+# fixtures; the window ring and the index keys stay whole on the device). kv_v41_gate:
+# v41_gate (the CPU's tokens, every forward's logits within 1e-4 of the largest) under an
+# emulated small device, which must have run attention steps with a host part.
+kv_v41_gate() { # <tag> <rows> <block> <env...> -- <argv...>
+ local tag=$1 rows=$2 blk=$3; shift 3
+ v41_gate "$tag" COLI_VK_KV_DEVICE_ROWS=$rows COLI_VK_KV_BLOCK=$blk "$@"
+ if [ -z "${FAULT_BACK:-}" ]; then
+ [ "$(kv_hostparts deepseek_v41 vk.log)" -gt 0 ] || { cat vk.log; fail "$tag: the split never ran a host part"; }
+ fi
+ echo " $tag: $(grep -a "KV split:" vk.log | tail -1 | sed 's/.*KV split: //')"
+}
+kv_split_deepseek_v41() {
+ v41_chain_fixtures
+ local cap f K=(COLI_VK_KV_DEVICE_ROWS=8 COLI_VK_KV_BLOCK=2)
+ for cap in 2 8; do kv_v41_gate "kv deepseek_v41 40-token prompt cap=$cap" 16 4 SNAP=dsv41_long -- $cap dsv41_long/ref.json; done
+ kv_v41_gate "kv deepseek_v41 four blocks of 2" 8 2 SNAP=dsv41_long -- 8 dsv41_long/ref.json
+ COLI_VK_KV_PIN=1 kv_v41_gate "kv deepseek_v41 eight blocks of 4, four pinned" 32 4 SNAP=dsv41_long -- 8 dsv41_long/ref.json
+ kv_v41_gate "kv deepseek_v41 12-token prompt" 8 2 SNAP=dsv41_tiny -- 8 dsv41_tiny/ref.json
+ # DSpark: a verify is one chunk, and a chunk's new compressed rows fit the window (four
+ # blocks of 4 take a verify of 4 rows, eight blocks one of 12)
+ for f in 1 3; do kv_v41_gate "kv deepseek_v41 DSpark spec=$f" 16 4 SNAP=dsv41_long V41_DSPARK=1 V41_SPEC_FORCE=$f -- 8 dsv41_long/ref.json; done
+ kv_v41_gate "kv deepseek_v41 DSpark spec=5" 32 4 SNAP=dsv41_long V41_DSPARK=1 V41_SPEC_FORCE=5 -- 8 dsv41_long/ref.json
+ kv_v41_gate "kv deepseek_v41 prompt in chunks of 3" 16 4 SNAP=dsv41_long COLI_VK_CHAIN_ROWS=3 -- 8 dsv41_long/ref.json
+ CHAINMODE=2 kv_v41_gate "kv deepseek_v41 prompts only" 8 2 SNAP=dsv41_long -- 8 dsv41_long/ref.json
+ kv_v41_gate "kv deepseek_v41 tier off" 16 4 SNAP=dsv41_long COLI_VK_TIER=0 -- 8 dsv41_long/ref.json
+ kv_v41_gate "kv deepseek_v41 V41_INDEX_OWNER=1" 16 4 SNAP=dsv41_long V41_INDEX_OWNER=1 -- 8 dsv41_long/ref.json
+ # COLI_VK_KV_SPLIT=0: the whole mirrors, as before the split
+ v41_gate "kv deepseek_v41 split off" SNAP=dsv41_long COLI_VK_KV_SPLIT=0 "${K[@]}" -- 8 dsv41_long/ref.json
+ if grep -qa 'KV cache split' vk.log; then cat vk.log; fail "kv deepseek_v41 split off: the split ran"; fi
+ FAULT_BACK=5 kv_v41_gate "kv deepseek_v41 device lost mid-decode" 8 2 SNAP=dsv41_long -- 8 dsv41_long/ref.json
+ FAULT_BACK=60 kv_v41_gate "kv deepseek_v41 device lost in the prompt" 8 2 SNAP=dsv41_long -- 8 dsv41_long/ref.json
+ CHAIN_SERVE_EXPECT='KV split: [0-9]+ layer steps, [1-9][0-9]* with a host part' \
+ $PY tests/vulkan_chain_serve.py ./deepseek_v41 dsv41_tiny V41_DSPARK=0 "${K[@]}"
+ CHAIN_SERVE_EXPECT='KV split: [0-9]+ layer steps, [1-9][0-9]* with a host part' \
+ $PY tests/vulkan_chain_serve.py ./deepseek_v41 dsv41_tiny V41_DSPARK=1 "${K[@]}"
+ env "${K[@]}" COLI_VULKAN=1 COLI_VK_CHAIN=1 COLI_VK_TIER_SYNC=1 COLI_USAGE=$PWD/chain.usage \
+ COLI_DSV41_FIXTURE=$PWD/dsv41_tiny $PY -m unittest tests.test_dsv41_prefix_serve tests.test_dsv41_dspark_serve
+ rm -f chain.usage chain-serve.usage
+}
+kv_split_deepseek_v41_sanitize() {
+ v41_chain_fixtures
+ local K=(COLI_VK_KV_DEVICE_ROWS=8 COLI_VK_KV_BLOCK=2)
+ kv_san deepseek_v41 "asan kv deepseek_v41 40-token prompt" "${K[@]}" SNAP=dsv41_long ./deepseek_v41 8 dsv41_long/ref.json
+ COLI_VK_KV_PIN=1 kv_san deepseek_v41 "asan kv deepseek_v41 pinned blocks, chunks of 3" COLI_VK_KV_DEVICE_ROWS=32 COLI_VK_KV_BLOCK=4 \
+ COLI_VK_CHAIN_ROWS=3 SNAP=dsv41_long ./deepseek_v41 2 dsv41_long/ref.json
+ kv_san deepseek_v41 "asan kv deepseek_v41 DSpark spec=5" COLI_VK_KV_DEVICE_ROWS=32 COLI_VK_KV_BLOCK=4 SNAP=dsv41_long \
+ V41_DSPARK=1 V41_SPEC_FORCE=5 ./deepseek_v41 8 dsv41_long/ref.json
+ CHAIN_SERVE_EXPECT='KV split: [0-9]+ layer steps, [1-9][0-9]* with a host part' \
+ $PY tests/vulkan_chain_serve.py ./deepseek_v41 dsv41_tiny V41_DSPARK=1 "${K[@]}" > san.log 2>&1 || { cat san.log; fail "asan kv deepseek_v41 serve"; }
+ echo "OK asan kv deepseek_v41 serve: $(tail -1 san.log)"
+ rm -f chain.usage chain-serve.usage
+}
+
+# DeepSeek's split keeps sparse attention in its original order on the device:
+# only selected missing compressed rows are staged from RAM. The exact chunk and
+# decode-versus-teacher-forced checks therefore apply unchanged, including V4 bf16.
+kv_v4_gate() { # <tag> <rows> <block> <fixture> <case> <env...>
+ local tag=$1 rows=$2 blk=$3 fx=$4 c=$5; shift 5
+ v4_chain_gate "$tag" "$fx" "$c" COLI_VK_KV_DEVICE_ROWS=$rows COLI_VK_KV_BLOCK=$blk "$@"
+ if [ -z "${FAULT_BACK:-}" ]; then
+ [ "$(kv_hostparts deepseek_v4 v4-vk.err)" -gt 0 ] || { cat v4-vk.err; fail "$tag: the split never ran a host part"; }
+ fi
+ echo " $tag: $(grep -a "KV split:" v4-vk.err | tail -1 | sed 's/.*KV split: //')"
+}
+kv_split_deepseek_v4() {
+ v4_chain_fixtures
+ local c fx
+ # the compressed case makes 6 rows at ratio 4 (four blocks of 1 on the device: a split
+ # holds three blocks at least), the long one 19 (four blocks of 2)
+ for fx in deepseek_v4_tiny_t deepseek_v4_tiny_e8; do
+ kv_v4_gate "kv deepseek_v4 $fx compressed" 4 1 $fx compressed
+ kv_v4_gate "kv deepseek_v4 $fx long" 8 2 $fx long
+ done
+ COLI_VK_KV_PIN=1 kv_v4_gate "kv deepseek_v4 eight blocks of 2, four pinned" 16 2 deepseek_v4_tiny_t long
+ kv_v4_gate "kv deepseek_v4 2 output groups" 8 2 deepseek_v4_tiny_g2 long
+ kv_v4_gate "kv deepseek_v4 a window of 4, ratios 4 and 2" 8 2 deepseek_v4_tiny_w4 long
+ kv_v4_gate "kv deepseek_v4 an indexer of 64 heads of 128" 8 2 deepseek_v4_tiny_ix long
+ kv_v4_gate "kv deepseek_v4 tier off" 8 2 deepseek_v4_tiny_t long COLI_VK_TIER=0 COLI_VK_DENSE=0
+ kv_v4_gate "kv deepseek_v4 chain chunks of 3" 16 2 deepseek_v4_tiny_e8 long COLI_VK_CHAIN_ROWS=3
+ kv_v4_gate "kv deepseek_v4 prefill chunks of 7" 16 2 deepseek_v4_tiny_t long V4_PREFILL_CHUNK=7
+ CHAINMODE=2 kv_v4_gate "kv deepseek_v4 prompts only" 8 2 deepseek_v4_tiny_t long
+ # n-gram drafts (V4_DRAFT): accepted and rejected, every one rejected, a mix
+ local A=ids:20,21,22,23,24,25,26,27,28,29,20,21,22:12 R=ids:20,21,22,23,50,51,52,20,21,22:12
+ local M=ids:30,31,32,33,34,35,60,61,30,31,32,33,34,35,36,37,38,39,40,41,30,31,32:12
+ kv_v4_gate "kv deepseek_v4 n-gram drafts accepted and rejected" 4 1 deepseek_v4_tiny_t $A V4_DRAFT=4 V4_NGRAM_PARTIAL_KEEP=1
+ kv_v4_gate "kv deepseek_v4 an n-gram draft rejected" 4 1 deepseek_v4_tiny_t $R V4_DRAFT=4 V4_NGRAM_PARTIAL_KEEP=1
+ kv_v4_gate "kv deepseek_v4 n-gram drafts, a window of 4" 4 2 deepseek_v4_tiny_w4 $M V4_DRAFT=4 V4_NGRAM_PARTIAL_KEEP=1
+ # The same model arithmetic with full mirrors, a split, different chunking and
+ # read-based pins. These comparisons are bit-exact and have no logit tolerance.
+ local K=(COLI_VK_KV_DEVICE_ROWS=8 COLI_VK_KV_BLOCK=2)
+ v4_chain_same "kv deepseek_v4 split = full mirror" deepseek_v4_tiny_t long "COLI_VK_KV_SPLIT=0" "COLI_VK_KV_SPLIT=1" "${K[@]}"
+ v4_chain_same "kv deepseek_v4 chunks 1 = 3" deepseek_v4_tiny_t long "COLI_VK_CHAIN_ROWS=1" "COLI_VK_CHAIN_ROWS=3" "${K[@]}"
+ v4_chain_same "kv deepseek_v4 pinned residency" deepseek_v4_tiny_e8 long "COLI_VK_KV_PIN=0" "COLI_VK_KV_PIN=1" "${K[@]}"
+ v4_chain_same "kv deepseek_v4 draft chunks" deepseek_v4_tiny_w4 $M "COLI_VK_CHAIN_ROWS=1" "COLI_VK_CHAIN_ROWS=3" "${K[@]}" V4_DRAFT=4 V4_NGRAM_PARTIAL_KEEP=1
+ # COLI_VK_KV_SPLIT=0: the whole mirrors, as before the split
+ v4_chain_gate "kv deepseek_v4 split off" deepseek_v4_tiny_t long COLI_VK_KV_SPLIT=0 COLI_VK_KV_DEVICE_ROWS=8 COLI_VK_KV_BLOCK=2
+ if grep -qa 'KV cache split' v4-vk.err; then cat v4-vk.err; fail "kv deepseek_v4 split off: the split ran"; fi
+ FAULT_BACK=3 kv_v4_gate "kv deepseek_v4 device lost mid-decode" 8 2 deepseek_v4_tiny_t long
+ FAULT_BACK=40 kv_v4_gate "kv deepseek_v4 device lost in the prompt" 8 2 deepseek_v4_tiny_t long
+ # serve: a pin, its extension, a read-only prompt, a shorter one (texts and reuse =
+ # CPU, logprobs within 0.5)
+ COLI_VK_KV_DEVICE_ROWS=8 COLI_VK_KV_BLOCK=2 v4_chain_serve deepseek_v4_tiny_t
+ rm -rf deepseek_v4_tiny_t deepseek_v4_tiny_e8 deepseek_v4_tiny_g2 deepseek_v4_tiny_w4 deepseek_v4_tiny_ix chain.usage
+ rm -f cpu.f32 vk.f32 v4-*.f32 v4-*.json v4-*.err v4-tf.txt
+}
+kv_split_deepseek_v4_sanitize() {
+ # the V4 sanitized build as its chain family makes it (no LTO, the sanitizers at the link)
+ local SAN="-fsanitize=address,undefined -fno-omit-frame-pointer -g"
+ make deepseek-v4-clean >/dev/null 2>&1 || true
+ make deepseek-v4 VK=1 LTO=0 EXTRA_CFLAGS="$SAN" EXTRA_LDFLAGS="$SAN"
+ v4_chain_fixtures
+ local p K=(COLI_VK_KV_DEVICE_ROWS=8 COLI_VK_KV_BLOCK=2)
+ p=$(v4_chain_prompt deepseek_v4_tiny_t long)
+ kv_v4_san() { # <tag> <env and argv...>: v4_chain_san, and steps with a host part
+ v4_chain_san "$@"
+ [ -n "${FAULT_BACK:-}" ] || [ "$(kv_hostparts deepseek_v4 san.log)" -gt 0 ] || { cat san.log; fail "$1: the split never ran a host part"; }
+ }
+ kv_v4_san "asan kv deepseek_v4 long, 4 experts" "${K[@]}" ./deepseek_v4 ./deepseek_v4_tiny_t "$p" --raw-prompt --max-tokens 4
+ kv_v4_san "asan kv deepseek_v4 chunks of 3, a window of 4" "${K[@]}" COLI_VK_CHAIN_ROWS=3 ./deepseek_v4 ./deepseek_v4_tiny_w4 \
+ "$(v4_chain_prompt deepseek_v4_tiny_w4 long)" --raw-prompt --max-tokens 4
+ COLI_VK_KV_PIN=1 kv_v4_san "asan kv deepseek_v4 pinned blocks, 64 index heads" COLI_VK_KV_DEVICE_ROWS=16 COLI_VK_KV_BLOCK=2 ./deepseek_v4 \
+ ./deepseek_v4_tiny_ix "$(v4_chain_prompt deepseek_v4_tiny_ix long)" --raw-prompt --max-tokens 4
+ kv_v4_san "asan kv deepseek_v4 drafts" COLI_VK_KV_DEVICE_ROWS=4 COLI_VK_KV_BLOCK=1 V4_DRAFT=4 V4_NGRAM_PARTIAL_KEEP=1 ./deepseek_v4 \
+ ./deepseek_v4_tiny_w4 "$(v4_chain_prompt x ids:30,31,32,33,34,35,60,61,30,31,32,33,34,35,36,37,38,39,40,41,30,31,32)" --raw-prompt --max-tokens 12
+ FAULT_BACK=10 kv_v4_san "asan kv deepseek_v4 device lost" "${K[@]}" ./deepseek_v4 ./deepseek_v4_tiny_t "$p" --raw-prompt --max-tokens 4
+ COLI_VK_KV_DEVICE_ROWS=8 COLI_VK_KV_BLOCK=2 v4_chain_serve deepseek_v4_tiny_t > san.log 2>&1 || { cat san.log; fail "asan kv deepseek_v4 served"; }
+ echo "OK asan kv deepseek_v4 served: $(tail -1 san.log | cut -c1-120)"
+ rm -rf deepseek_v4_tiny_t deepseek_v4_tiny_e8 deepseek_v4_tiny_g2 deepseek_v4_tiny_w4 deepseek_v4_tiny_ix san.json san.log
+ make deepseek-v4-clean >/dev/null 2>&1 || true
+}
+
+# kv-split: every chain engine but DeepSeek's (whose fixtures want another transformers)
+family_kv_split() {
+ export OMP_NUM_THREADS=2
+ make qwen36 qwen38 olmoe inkling mimo colibri glm53 kimi_k3 tests/test_vk_chain VK=1
+ ./tests/test_vk_chain shaders/qmatmul.spv | tee vk_chain.log
+ tail -1 vk_chain.log | grep -qx PASS || fail "the chain's ops"
+ grep -a 'kvs ' vk_chain.log
+ kv_split_qwen36; kv_split_qwen38; kv_split_olmoe; kv_split_inkling; kv_split_mimo
+ kv_split_colibri; kv_split_glm53; kv_split_kimi
+ unset OMP_NUM_THREADS
+}
+family_kv_split_sanitize() {
+ make clean >/dev/null 2>&1 || true
+ make qwen36 qwen38 olmoe inkling mimo colibri glm53 kimi_k3 tests/test_vk_chain VK=1 EXTRA_CFLAGS="-fsanitize=address,undefined -fno-omit-frame-pointer -g"
+ export ASAN_OPTIONS=detect_leaks=0:detect_stack_use_after_return=0 UBSAN_OPTIONS=print_stacktrace=1
+ ./tests/test_vk_chain shaders/qmatmul.spv > san.log 2>&1 || true
+ if grep -qE "ERROR: AddressSanitizer|runtime error:" san.log || ! tail -1 san.log | grep -qx PASS; then cat san.log; fail "asan: the chain's ops"; fi
+ echo "OK asan: the chain's ops, the split KV cache's among them"
+ kv_split_qwen36_sanitize; kv_split_qwen38_sanitize; kv_split_olmoe_sanitize; kv_split_inkling_sanitize
+ kv_split_mimo_sanitize; kv_split_colibri_sanitize; kv_split_glm53_sanitize; kv_split_kimi_sanitize
+ make clean >/dev/null 2>&1 || true
+}
+family_kv_split_deepseek() {
+ export OMP_NUM_THREADS=2
+ make deepseek_v41 deepseek-v4 VK=1
+ kv_split_deepseek_v41; kv_split_deepseek_v4
+ unset OMP_NUM_THREADS
+}
+family_kv_split_deepseek_sanitize() {
+ make clean >/dev/null 2>&1 || true
+ # deepseek_v4's sanitized build is its own (no LTO, the sanitizers at the link too):
+ # kv_split_deepseek_v4_sanitize makes it, as v4_chain_family_sanitize does
+ make deepseek_v41 VK=1 EXTRA_CFLAGS="-fsanitize=address,undefined -fno-omit-frame-pointer -g"
+ export ASAN_OPTIONS=detect_leaks=0:detect_stack_use_after_return=0 UBSAN_OPTIONS=print_stacktrace=1
+ kv_split_deepseek_v41_sanitize; kv_split_deepseek_v4_sanitize
+ make clean >/dev/null 2>&1 || true
+}
+
+# The decision engines (docs/vulkan.md, "The decision engines"): laya and gliner_decide
+# with their forward on the device (COLI_VK_CHAIN=1, chain_enc.comp) and matrix by matrix
+# (COLI_VK_CHAIN=0), Clef's DECIDE on qwen36's per-matrix path and its chain, in f32, int8
+# and f16 (fmt 14), the rows read in place and copied. Each configuration against the CPU
+# (tests/vulkan_decide_compare.py: every case of the fixture, the same decisions, the
+# probabilities within 1e-5 and the logits within 1e-4 of the largest, matmuls on the
+# device); the tiny oracles through the device; the device opened with every path off,
+# byte for byte the CPU's; a device lost mid-run (the rest of the forwards on the CPU);
+# the serve tests and TypeSafe's SDKs (where installed) through the device, checked by
+# the engines' own lines. SAN=1 (decide-sanitize): a sanitized build, the comparisons
+# and the serve tests, UBSan halting on its first report.
+decide_vk_lines() { # <log> <tag>: every decision engine's forwards ran on the device
+ local e
+ for e in laya gliner_decide; do
+ grep -qE "^\[VK\] $e chain: [1-9][0-9]* forwards on the device" "$1" || { tail -50 "$1"; fail "$2: $e never ran on the device"; }
+ done
+ grep -qE "^\[VK\] qwen36 chain: [1-9][0-9]* forwards" "$1" || { tail -50 "$1"; fail "$2: Clef never ran on the device"; }
+}
+family_decide() {
+ local SANF=""
+ if [ -n "${SAN:-}" ]; then
+ SANF="-fsanitize=address,undefined -fno-omit-frame-pointer -g"
+ make clean >/dev/null 2>&1 || true
+ export ASAN_OPTIONS=detect_leaks=0:detect_stack_use_after_return=0 UBSAN_OPTIONS=print_stacktrace=1:halt_on_error=1
+ fi
+ make laya gliner_decide qwen36 tests/test_qwen36_weight_alloc VK=1 EXTRA_CFLAGS="$SANF"
+ COLI_VULKAN=0 ./tests/test_qwen36_weight_alloc
+ COLI_VULKAN=1 ./tests/test_qwen36_weight_alloc
+ make laya-tiny-generate gliner-decide-tiny-generate clef-tiny-generate
+ export OMP_NUM_THREADS=2
+ local c e
+ local -a cmp=(
+ "laya COLI_VK_CHAIN=1" "laya COLI_VK_CHAIN=0" "gliner COLI_VK_CHAIN=1" "gliner COLI_VK_CHAIN=0"
+ "laya COLI_VK_CHAIN=1 COLI_VK_STAGED=1" "gliner COLI_VK_CHAIN=1 COLI_VK_STAGED=1"
+ "laya COLI_VK_CHAIN=1 COLI_VK_CHAIN_FAULT=10" "gliner COLI_VK_CHAIN=1 COLI_VK_CHAIN_FAULT=10"
+ "clef" "clef COLI_VK_CHAIN=1"
+ "clef COLI_VK_CHAIN=1 COLI_VK_CHAIN_ROWS=1 COLI_VK_CHAIN_FAULT=10 COLI_VK_DENSE_HOST=0"
+ "clef COLI_DENSE_I8=1 COLI_DENSE_BITS=16 COLI_VK_CHAIN=1 COLI_VK_CHAIN_ROWS=1 COLI_VK_CHAIN_FAULT=10 COLI_VK_DENSE_HOST=0 COLI_VK_IMPORT=0"
+ "clef COLI_DENSE_I8=1 COLI_DENSE_IDOT=0 COLI_VK_CHAIN=1 COLI_VK_DENSE_HOST=0"
+ "clef COLI_DENSE_I8=1 COLI_DENSE_BITS=16 COLI_VK_CHAIN=1 COLI_VK_DENSE_HOST=0"
+ "clef COLI_DENSE_I8=1 COLI_DENSE_IDOT=0" "clef COLI_DENSE_I8=1 COLI_DENSE_IDOT=0 COLI_VK_CHAIN=1"
+ "clef COLI_DENSE_I8=1 COLI_DENSE_IDOT=0 COLI_VK_IMPORT=0"
+ "clef COLI_DENSE_I8=1 COLI_DENSE_BITS=16" "clef COLI_DENSE_I8=1 COLI_DENSE_BITS=16 COLI_VK_CHAIN=1"
+ "clef COLI_DENSE_I8=1 COLI_DENSE_BITS=16 COLI_VK_IMPORT=0")
+ for c in "${cmp[@]}"; do
+ # shellcheck disable=SC2086
+ $PY tests/vulkan_decide_compare.py $c > cmp.log 2>&1 || { cat cmp.log; fail "decide: $c"; }
+ tail -1 cmp.log
+ done
+ for e in laya gliner clef; do
+ $PY tests/vulkan_decide_compare.py --off $e COLI_VK_CHAIN=0 COLI_VK_DENSE=0 > cmp.log 2>&1 || { cat cmp.log; fail "decide: $e with every path off"; }
+ tail -1 cmp.log
+ done
+ [ -n "${SAN:-}" ] || for c in 0 1; do
+ COLI_VULKAN=1 COLI_VK_CHAIN=$c LAYA_TINY_REQUIRED=1 GLINER_DECIDE_TINY_REQUIRED=1 CLEF_TINY_REQUIRED=1 \
+ $PY -m unittest tests.test_laya_tiny tests.test_gliner_decide_tiny tests.test_clef_tiny > oracle.log 2>&1 \
+ || { cat oracle.log; fail "decide: the tiny oracles through the device, COLI_VK_CHAIN=$c"; }
+ echo "OK decide: the tiny oracles (Laya, GLiNER2.5-Decide, Clef) through the device, COLI_VK_CHAIN=$c"
+ done
+ COLI_VULKAN=1 COLI_VK_CHAIN=1 LAYA_TINY_REQUIRED=1 GLINER_DECIDE_TINY_REQUIRED=1 CLEF_TINY_REQUIRED=1 \
+ $PY -m unittest tests.test_decision_serve > serve.log 2>&1 || { tail -80 serve.log; fail "decide: the serve tests through the device"; }
+ decide_vk_lines serve.log "decide serve"
+ echo "OK decide: the serve tests (tests.test_decision_serve) through the device: $(tail -1 serve.log)"
+ if [ -z "${SAN:-}" ] && $PY -c "import typesafe_sdk" 2>/dev/null && [ -d "${JEV_TS_SDK_DIR:-tools/jev-sdk}/node_modules" ]; then
+ make mimo VK=1 mimo-tiny-generate
+ rm -f sdk-serve.log
+ COLI_VULKAN=1 COLI_VK_CHAIN=1 JEV_SDK_REQUIRED=1 JEV_SDK_SERVE_LOG=sdk-serve.log \
+ $PY -m unittest tests.test_jev_sdk > sdk.log 2>&1 || { tail -80 sdk.log; fail "decide: TypeSafe's SDKs through the device"; }
+ decide_vk_lines sdk-serve.log "decide SDKs"
+ echo "OK decide: TypeSafe's Python and TypeScript SDKs (tests.test_jev_sdk) through the device"
+ elif [ -z "${SAN:-}" ]; then
+ echo "SKIP decide: TypeSafe's SDKs are not installed (tools/requirements-jev-sdk.txt, npm ci in tools/jev-sdk)"
+ fi
+ rm -f cmp.log oracle.log serve.log sdk.log sdk-serve.log
+ unset OMP_NUM_THREADS
+ [ -z "${SAN:-}" ] || make clean >/dev/null 2>&1 || true
+}
+
case "${1:-}" in
shader) shader_formats ;;
qwen) family_qwen ;;
@@ -2685,6 +3961,8 @@ case "${1:-}" in
glm-sanitize) family_glm_sanitize ;;
qwen-chain) family_qwen_chain ;;
qwen-chain-sanitize) family_qwen_chain_sanitize ;;
+ qwen-spec) family_qwen_spec ;;
+ qwen-spec-sanitize) family_qwen_spec_sanitize ;;
inkling-olmoe-chain) family_inkling_olmoe_chain ;;
inkling-olmoe-chain-sanitize) family_inkling_olmoe_chain_sanitize ;;
glm-chain) family_glm_chain ;;
@@ -2693,6 +3971,16 @@ case "${1:-}" in
kimi-chain-sanitize) family_kimi_chain_sanitize ;;
deepseek-chain) family_deepseek_chain ;;
deepseek-chain-sanitize) family_deepseek_chain_sanitize ;;
+ prefill-qwen) family_prefill_qwen ;;
+ prefill-qwen-sanitize) family_prefill_qwen_sanitize ;;
+ prefill-inkling-olmoe) family_prefill_inkling_olmoe ;;
+ prefill-mimo-kimi) family_prefill_mimo_kimi ;;
+ prefill-glm) family_prefill_glm ;;
+ prefill-deepseek) family_prefill_deepseek ;;
+ kv-split) family_kv_split ;;
+ kv-split-sanitize) family_kv_split_sanitize ;;
+ kv-split-deepseek) family_kv_split_deepseek ;;
+ kv-split-deepseek-sanitize) family_kv_split_deepseek_sanitize ;;
staged) family_staged ;;
*-staged) # COLI_VK_STAGED unset for the window's run, then the whole family staged
if [ "$1" = qwen-staged ]; then env -u COLI_VK_STAGED bash tests/vulkan_engines.sh staged-window; fi
@@ -2700,5 +3988,10 @@ case "${1:-}" in
staged-window) staged_window ;;
staged-faults) family_staged_faults ;;
staged-faults-sanitize) SAN=1 family_staged_faults ;;
- *) echo "usage: $0 staged|<family>-staged|shader|qwen|qwen-sanitize|inkling-olmoe|inkling-olmoe-sanitize|mimo-qwenimage|deepseek|deepseek-sanitize|kimi|kimi-mimo-sanitize|glm|glm-sanitize|qwen-chain|qwen-chain-sanitize|mimo-chain|mimo-chain-sanitize|inkling-olmoe-chain|inkling-olmoe-chain-sanitize|glm-chain|glm-chain-sanitize|kimi-chain|kimi-chain-sanitize|deepseek-chain|deepseek-chain-sanitize" >&2; exit 2 ;;
+ decide) family_decide ;;
+ decide-sanitize) SAN=1 family_decide ;;
+ dense-only-*) g=${1#dense-only-}; fn=dho_family_${g//-/_}
+ declare -F "$fn" >/dev/null || { echo "no dense-only group ${g}" >&2; exit 2; }
+ "$fn" ;;
+ *) echo "usage: $0 decide|decide-sanitize|staged|<family>-staged|shader|qwen|qwen-sanitize|inkling-olmoe|inkling-olmoe-sanitize|mimo-qwenimage|deepseek|deepseek-sanitize|kimi|kimi-mimo-sanitize|glm|glm-sanitize|qwen-chain|qwen-chain-sanitize|qwen-spec|qwen-spec-sanitize|mimo-chain|mimo-chain-sanitize|inkling-olmoe-chain|inkling-olmoe-chain-sanitize|glm-chain|glm-chain-sanitize|kimi-chain|kimi-chain-sanitize|deepseek-chain|deepseek-chain-sanitize|dense-only-<group>[-sanitize]|prefill-qwen|prefill-qwen-sanitize|prefill-inkling-olmoe|prefill-mimo-kimi|prefill-glm|prefill-deepseek|kv-split|kv-split-sanitize|kv-split-deepseek|kv-split-deepseek-sanitize" >&2; exit 2 ;;
esac
diff --git a/c/tools/make_qwen36_tiny.py b/c/tools/make_qwen36_tiny.py
index 234efdd2..699ee05d 100644
--- a/c/tools/make_qwen36_tiny.py
+++ b/c/tools/make_qwen36_tiny.py
@@ -325,8 +325,11 @@ def main():
# --hidden/--inter override the preset's toy widths (qwen36-35b: 64/32). The
# shared expert kernel (expert_ffn.h) needs hidden and inter multiples of
# 64: --inter 64 exercises it (the CI's A/B step does exactly that).
+ # --vocab 128: every id is an ASCII byte of the byte tokenizer tests/prefix_serve_harness.py
+ # gives a fixture without one, so a serve test can resend a reply as text and the
+ # engine reads back the very ids it generated (tests/spec_drafts_harness.py)
for name in ("layers", "experts", "topk", "q-heads", "kv-heads", "hidden", "inter",
- "dn-key-heads", "dn-value-heads"):
+ "dn-key-heads", "dn-value-heads", "vocab"):
ap.add_argument(f"--{name}", type=int, default=None,
help=f"override the preset's {name.replace('-', '_')}")
ap.add_argument("--fused-experts", action="store_true", default=None,
@@ -338,7 +341,7 @@ def main():
for arg, key in (("layers", "n_layers"), ("experts", "n_experts"), ("topk", "topk"),
("q_heads", "q_heads"), ("kv_heads", "kv_heads"), ("hidden", "hidden"),
("inter", "inter"), ("dn_key_heads", "dn_key_heads"),
- ("dn_value_heads", "dn_value_heads"), ("fused_experts", "fused_experts"),
+ ("dn_value_heads", "dn_value_heads"), ("vocab", "vocab"), ("fused_experts", "fused_experts"),
("mtp", "mtp")):
if getattr(args, arg) is not None:
geo[key] = getattr(args, arg)
diff --git a/c/vk_chain.c b/c/vk_chain.c
index ca6b9eaa..805b9b32 100644
--- a/c/vk_chain.c
+++ b/c/vk_chain.c
@@ -19,6 +19,7 @@
#include <stdlib.h>
#include <string.h>
#include <time.h>
+#include "compat.h"
static double vkc_now_ms(void) { struct timespec t; clock_gettime(CLOCK_MONOTONIC, &t); return t.tv_sec * 1e3 + t.tv_nsec / 1e6; }
@@ -249,6 +250,7 @@ static VkPipeline make_pipe(VkShaderModule m, const VkSpecializationInfo *si) {
static void dsv4_init(void); /* chain_dsv4.comp (DeepSeek V4.1 / V4), below */
static void dsv4_shutdown(void);
+static void kvs_shutdown(void); /* chain_kvs.comp (the split KV cache), below */
int vkc_init(void) {
if (K.ready) return !K.lost;
if (!coli_vk_core(&K.core)) return 0;
@@ -662,7 +664,7 @@ static int matmul_aligned(const ColiVkTensorInfo *ti, VkcBuf *x, size_t xb, VkcB
struct VkcPC pc = {ti->fmt, S, ti->I, ti->O, ti->rowWords, ti->gs};
int ok;
/* the vectorized GEMV where the row is whole 16-byte steps and fits the staging */
- int per = ti->fmt == 1 ? 16 : ti->fmt == 4 ? 32 : ti->fmt == 11 ? 8 : ti->fmt == 10 ? 4 : 0;
+ int per = ti->fmt == 1 ? 16 : ti->fmt == 4 ? 32 : ti->fmt == 11 || ti->fmt == 14 ? 8 : ti->fmt == 10 ? 4 : 0;
int v4 = path < 0 && K.gemv4 && per && ti->rowWords % 4 == 0 && ti->I % 4 == 0 &&
(ti->fmt != 4 || (ti->gs > 0 && ti->gs % 32 == 0)) && (ti->rowWords / 4) * per / 4 <= K.gemv4_xs;
K.kind = path >= 0 ? PK_GEMM : ti->fmt == 1 ? PK_GEMV8 : PK_GEMV;
@@ -725,6 +727,69 @@ int vkc_attn(VkcBuf *q, VkcBuf *kc, VkcBuf *vc, VkcBuf *o, VkcBuf *gate, VkcBuf
w.a = *p;
return vkc_attn_w(q, kc, vc, o, gate, sel, NULL, &w);
}
+/* ---- blocked attention for prompt chunks (chain_attnb.comp), made on first use ----
+ * chain_attn reads a row's positions once per (head, row): a chunk of S rows reads its
+ * KV cache S * H times, which held a 2048-row chunk of Qwen3.6 at 1.2 s a layer on a
+ * Radeon 780M. From COLI_VK_ATTN_BLOCK rows (16; 0 = never) a call takes chain_attnb,
+ * one workgroup per KV head and block of rows: each K and V row read once per block.
+ * Below it, and for decode and an MTP verify, chain_attn's bits as before. */
+static struct { VkShaderModule mod; VkPipeline pipe; int tried, bc; } KAB;
+static VkPipeline kab_pipe(void) {
+ if (KAB.pipe || KAB.tried || !vkc_ready()) return KAB.pipe;
+ KAB.tried = 1;
+ /* the tile of positions as the device's shared memory allows: K and V rows of 256
+ * floats, the partial dots and the scores, about 3.2 KiB a position */
+ VkPhysicalDeviceProperties pp; vkGetPhysicalDeviceProperties((VkPhysicalDevice)K.core.phys, &pp);
+ int bc = 16;
+ while (bc >= 4 && (size_t)bc * (2 * 256 + 32 * 8 + 32 + 1) * 4 > pp.limits.maxComputeSharedMemorySize) bc /= 2;
+ if (bc < 4) return VK_NULL_HANDLE;
+ if ((KAB.mod = load_module(K.core.spv_path, "chain_attnb.spv"))) {
+ int32_t v = bc;
+ VkSpecializationMapEntry me = {0, 0, 4};
+ VkSpecializationInfo si = {1, &me, 4, &v};
+ if ((KAB.pipe = make_pipe(KAB.mod, &si))) KAB.bc = bc;
+ }
+ return KAB.pipe;
+}
+static void kab_shutdown(void) {
+ if (KAB.pipe) vkDestroyPipeline(K.dev, KAB.pipe, NULL);
+ if (KAB.mod) vkDestroyShaderModule(K.dev, KAB.mod, NULL);
+ memset(&KAB, 0, sizeof KAB);
+}
+/* ---- attention over a prompt chunk, cut over several submissions ----------------
+ * The attention ops' work grows with rows x positions: a chunk of thousands of rows makes
+ * one dispatch of seconds (8192 rows of Qwen3.6 at its full context: about 5 s on a Radeon
+ * 780M), past what a driver lets one submission run (amdgpu resets a ring after 10 s;
+ * Windows' TDR after 2 s). An attention whose rows x positions x heads x head dim passes
+ * COLI_VK_ATTN_SLICE (2^32; 0 = never) is recorded in slices of rows, each ending the
+ * frame it is in (submitted, not waited for) and the next opening a frame of its own: the
+ * next frame's first barrier orders it after the slice. A slice starts at a multiple of
+ * `align` (the blocked attention's rows per workgroup), so its blocks are the ones the
+ * whole dispatch would have made, and no row's arithmetic changes. */
+static long long attn_slice_budget(void) {
+ const char *e = getenv("COLI_VK_ATTN_SLICE");
+ long long v = e && *e ? atoll(e) : (1LL << 32);
+ return v < 0 ? 0 : v;
+}
+static int attn_slice_rows(int S, int pos_base, double width, int align) {
+ long long b = attn_slice_budget();
+ if (align < 1) align = 1;
+ if (!b || S <= align) return S;
+ double per_row = (double)(pos_base + S) * (width > 1 ? width : 1);
+ long long rr = (long long)((double)b / per_row);
+ rr = rr / align * align;
+ if (rr < align) rr = align;
+ return rr >= S ? S : (int)rr;
+}
+static int attn_slice_next(void) { /* the open frame out, a new one in */
+ K.st.attn_slices++;
+ return vkc_submit(0) && vkc_begin();
+}
+int vkc_attn_block_rows(void) {
+ const char *e = getenv("COLI_VK_ATTN_BLOCK");
+ int v = e && *e ? atoi(e) : 16;
+ return v < 0 ? 0 : v;
+}
int vkc_attn_w(VkcBuf *q, VkcBuf *kc, VkcBuf *vc, VkcBuf *o, VkcBuf *gate, VkcBuf *sel, VkcBuf *snk,
const VkcAttnW *p) {
K.kind = PK_ATTN;
@@ -732,7 +797,24 @@ int vkc_attn_w(VkcBuf *q, VkcBuf *kc, VkcBuf *vc, VkcBuf *o, VkcBuf *gate, VkcBu
if (w.vd <= 0) w.vd = w.a.hd;
if (w.a.hd > 256 || w.vd > 256 || w.a.H % w.a.KVH || w.win < 0 || w.ring < 0 || (w.sink && !snk)) return 0;
VkcBind bd[7] = {B(q, 0), B(kc, 0), B(vc, 0), B(o, 1), B(gate, 0), B(sel, 0), B(snk, 0)};
- return record(K.pipe[P_ATTN], bd, 7, &w, sizeof w, (uint32_t)w.a.H, (uint32_t)w.a.S, 1);
+ int G = w.a.H / w.a.KVH, blk = vkc_attn_block_rows();
+ int blocked = blk > 0 && w.a.S >= blk && G <= 32 && kab_pipe(), br = w.a.sel_row > 0 ? 1 : 32 / G;
+ int S = w.a.S, rr = attn_slice_rows(S, w.a.pos_base, (double)w.a.H * (w.a.hd > w.vd ? w.a.hd : w.vd), blocked ? br : 1);
+ int ok = 1;
+ for (int r0 = 0; ok && r0 < S; r0 += rr) {
+ VkcAttnW x = w;
+ int n = S - r0 < rr ? S - r0 : rr;
+ x.a.S = n; x.a.pos_base = w.a.pos_base + r0;
+ x.a.q_off += r0 * w.a.q_row; x.a.g_off += r0 * w.a.g_row; x.a.o_off += r0 * w.a.o_row;
+ if (w.a.sel_row > 0) x.a.sel_off += r0 * w.a.sel_row;
+ if (r0 && !attn_slice_next()) return 0;
+ if (blocked) {
+ struct { VkcAttnW w; int br; } pc = {x, br};
+ ok = record(KAB.pipe, bd, 7, &pc, sizeof pc, (uint32_t)x.a.KVH, (uint32_t)((x.a.S + br - 1) / br), 1);
+ K.st.attn_blocked += ok;
+ } else ok = record(K.pipe[P_ATTN], bd, 7, &x, sizeof x, (uint32_t)x.a.H, (uint32_t)x.a.S, 1);
+ }
+ return ok;
}
int vkc_dnconv(VkcBuf *in, VkcBuf *w, VkcBuf *ring, VkcBuf *out, VkcBuf *snap, const VkcDnConv *p) {
K.kind = PK_DNCONV;
@@ -804,11 +886,22 @@ int vkc_mla_core(VkcBuf *qabs, VkcBuf *qr, VkcBuf *lat, VkcBuf *rope, VkcBuf *se
K.kind = PK_MLA;
if (!K.mla_ok || p->S < 1 || p->H < 1 || p->K < 1 || p->K > 1024 || p->R < 0 || p->R > 128 ||
(p->R > 0 && !rope) || p->pos_base < 0 || p->kv_start < 0 || p->kv_start > p->pos_base) return 0;
- struct { int mode; VkcMlaCore b; } pc = {0, *p};
+ typedef struct { int mode; VkcMlaCore b; } CorePC;
+ CorePC pc = {0, *p};
if (!sel) pc.b.sel_row = 0;
VkcBind bd[6] = {B(qabs, 0), B(qr, 0), B(lat, 0), B(p->R > 0 ? rope : NULL, 0), B(pc.b.sel_row > 0 ? sel : NULL, 0),
B(clat, 1)};
- return record(K.mpipe[PM_MLA], bd, 6, &pc, sizeof pc, (uint32_t)p->H, (uint32_t)p->S, 1);
+ /* a long prompt chunk in slices of rows, a submission each (attn_slice_rows) */
+ int S = p->S, rr = attn_slice_rows(S, p->pos_base, (double)p->H * (p->K + p->R), 1), ok = 1;
+ for (int r0 = 0; ok && r0 < S; r0 += rr) {
+ CorePC x = pc;
+ x.b.S = S - r0 < rr ? S - r0 : rr; x.b.pos_base = p->pos_base + r0;
+ x.b.qa_off += r0 * p->qa_row; x.b.qr_off += r0 * p->qr_row; x.b.o_off += r0 * p->o_row;
+ if (pc.b.sel_row > 0) x.b.sel_off += r0 * pc.b.sel_row;
+ if (r0 && !attn_slice_next()) return 0;
+ ok = record(K.mpipe[PM_MLA], bd, 6, &x, sizeof x, (uint32_t)p->H, (uint32_t)x.b.S, 1);
+ }
+ return ok;
}
static int mla_rowop(int mode, VkcBuf *x, VkcBuf *aux, VkcBuf *y, const VkcMlaRow *p) {
if (!K.mla_ok || !open_frame() || K.lost || p->nseg < 0) return 0;
@@ -1058,10 +1151,136 @@ int vkc_relattn(VkcBuf *q, VkcBuf *kc, VkcBuf *vc, VkcBuf *o, VkcBuf *kvs, VkcBu
if (!pipe || p->hd > 256 || p->d_rel > 64 || p->d_rel < 0 || p->KVH < 1 || p->H % p->KVH || p->cap < 1) return 0;
K.kind = PK_ATTN;
VkcBind bd[8] = {B(q, 0), B(kc, 0), B(vc, 0), B(o, 1), B(kvs, 0), B(r, 0), B(relp, 0), B(tau, 0)};
- return record(pipe, bd, 8, p, sizeof *p, (uint32_t)p->H, (uint32_t)p->S, 1);
+ /* a long prompt chunk in slices of rows, a submission each (attn_slice_rows): the
+ * shader's s0 is the slice's first row, pos_base stays the step's */
+ int S = p->S, rr = attn_slice_rows(S, p->pos_base, (double)p->H * p->hd, 1), ok = 1;
+ for (int r0 = 0; ok && r0 < S; r0 += rr) {
+ struct { VkcRelAttn a; int s0; } x = {*p, r0};
+ x.a.S = S - r0 < rr ? S - r0 : rr;
+ x.a.q_off += r0 * p->q_row; x.a.o_off += r0 * p->o_row; x.a.r_off += r0 * p->r_row; x.a.tau_off += r0;
+ if (r0 && !attn_slice_next()) return 0;
+ ok = record(pipe, bd, 8, &x, sizeof x, (uint32_t)p->H, (uint32_t)x.a.S, 1);
+ }
+ return ok;
+}
+
+/* ---- the decision engines' encoders: chain_enc.comp, made on first use ---- */
+static struct { VkShaderModule mod; VkPipeline pipe; int tried; } KE;
+static VkPipeline ke_pipe(void) {
+ if (KE.pipe || KE.tried || !vkc_ready()) return KE.pipe;
+ KE.tried = 1;
+ if ((KE.mod = load_module(K.core.spv_path, "chain_enc.spv"))) KE.pipe = make_pipe(KE.mod, NULL);
+ return KE.pipe;
+}
+static void ke_shutdown(void) {
+ if (KE.pipe) vkDestroyPipeline(K.dev, KE.pipe, NULL);
+ if (KE.mod) vkDestroyShaderModule(K.dev, KE.mod, NULL);
+ memset(&KE, 0, sizeof KE);
+}
+int vkc_enc_ready(void) { return ke_pipe() != VK_NULL_HANDLE; }
+static int ke_ew(int kind, int mode, void *pc, size_t bytes, const VkcBind *bd, int nb, uint64_t n) {
+ VkPipeline pipe = ke_pipe();
+ if (!pipe) return 0;
+ *(int *)pc = mode;
+ K.kind = kind;
+ uint32_t gx, gy; grid((n + 255) / 256, &gx, &gy);
+ return record(pipe, bd, nb, pc, bytes, gx, gy, 1);
+}
+int vkc_enc_norm(VkcBuf *a, VkcBuf *b, VkcBuf *prm, VkcBuf *y, const VkcEncNorm *p) {
+ VkPipeline pipe = ke_pipe();
+ if (!pipe || p->D < 1 || p->D > 4096) return 0;
+ VkcEncNorm c = *p;
+ c.mode = 0;
+ K.kind = PK_NORM;
+ VkcBind bd[4] = {B(a, (p->flags & VKC_ENC_SUM) != 0), B((p->flags & VKC_ENC_ADD) ? b : NULL, 0), B(prm, 0), B(y, 1)};
+ uint32_t gx, gy; grid((uint64_t)p->rows, &gx, &gy);
+ return record(pipe, bd, 4, &c, sizeof c, gx, gy, 1);
+}
+int vkc_enc_bias(VkcBuf *y, VkcBuf *prm, const VkcEncBias *p) {
+ VkcEncBias c = *p;
+ VkcBind bd[4] = {B(NULL, 0), B(NULL, 0), B(prm, 0), B(y, 1)};
+ return ke_ew(PK_EW, 1, &c, sizeof c, bd, 4, (uint64_t)p->n);
+}
+int vkc_enc_geglu(VkcBuf *a, VkcBuf *prm, VkcBuf *y, const VkcEncGeglu *p) {
+ VkcEncGeglu c = *p;
+ VkcBind bd[4] = {B(a, 0), B(NULL, 0), B(prm, 0), B(y, 1)};
+ return ke_ew(PK_EW, 2, &c, sizeof c, bd, 4, (uint64_t)p->n);
+}
+int vkc_enc_rope(VkcBuf *x, VkcBuf *tab, VkcBuf *pos, const VkcEncRope *p) {
+ if (p->hd < 2 || p->hd % 2) return 0;
+ VkcEncRope c = *p;
+ VkcBind bd[5] = {B(x, 1), B(tab, 0), B(NULL, 0), B(NULL, 0), B(pos, 0)};
+ return ke_ew(PK_ROPE, 3, &c, sizeof c, bd, 5, (uint64_t)p->n);
+}
+int vkc_enc_addrow(VkcBuf *y, VkcBuf *tab, VkcBuf *idx, const VkcEncAddRow *p) {
+ VkcEncAddRow c = *p;
+ VkcBind bd[5] = {B(NULL, 0), B(NULL, 0), B(tab, 0), B(y, 1), B(idx, 0)};
+ return ke_ew(PK_EW, 4, &c, sizeof c, bd, 5, (uint64_t)p->n);
+}
+int vkc_enc_attn(VkcBuf *qkv, VkcBuf *c2p, VkcBuf *p2c, VkcBuf *o, VkcBuf *rng, VkcBuf *ridx, VkcBuf *pos,
+ const VkcEncAttn *p) {
+ VkPipeline pipe = ke_pipe();
+ if (!pipe || p->hd < 1 || p->hd > 128 || p->S < 1 || p->H < 1) return 0;
+ VkcEncAttn c = *p;
+ c.mode = 5;
+ K.kind = PK_ATTN;
+ VkcBind bd[7] = {B(qkv, 0), B(c2p, 0), B(p2c, 0), B(o, 1), B(rng, 0), B(ridx, 0), B(pos, 0)};
+ return record(pipe, bd, 7, &c, sizeof c, (uint32_t)p->H, (uint32_t)((p->S + 15) / 16), 1);
+}
+int vkc_enc_rel(VkcBuf *x, VkcBuf *pt, VkcBuf *y, const VkcEncRel *p) {
+ VkPipeline pipe = ke_pipe();
+ if (!pipe || p->hd < 1 || p->S < 1 || p->H < 1 || p->nr < 1) return 0;
+ VkcEncRel c = *p;
+ c.mode = 6;
+ K.kind = PK_ATTN;
+ VkcBind bd[4] = {B(x, 0), B(pt, 0), B(NULL, 0), B(y, 1)};
+ return record(pipe, bd, 4, &c, sizeof c, (uint32_t)((p->nr + 63) / 64), (uint32_t)((p->S + 15) / 16), (uint32_t)p->H);
}
void vkc_stats(VkcStats *st) { *st = K.st; }
+
+/* ---- big prefill chunks ------------------------------------------------------------
+ * The rows of a prompt chunk (vk_chain.h, vkc_chunk_rows). The free device memory is
+ * read once, at an engine's first call: a later reading would count the chain's own
+ * scratch as used and shrink the chunk from one forward to the next. */
+int vkc_chunk_auto(void) {
+ const char *e = getenv("COLI_VK_CHAIN_ROWS");
+ return !(e && *e) || !strcmp(e, "auto");
+}
+int vkc_chunk_rows(const char *engine, size_t row_bytes) {
+ const char *e = getenv("COLI_VK_CHAIN_ROWS");
+ if (e && *e && strcmp(e, "auto") != 0) {
+ int v = atoi(e);
+ return v < 1 ? 1 : v > 65535 ? 65535 : v;
+ }
+ static char said[16][24]; static int nsaid, rows_said[16];
+ for (int i = 0; i < nsaid; i++) if (!strcmp(said[i], engine ? engine : "")) return rows_said[i];
+ const char *mx = getenv("COLI_VK_CHAIN_ROWS_MAX");
+ int cap = mx && *mx ? atoi(mx) : 8192;
+ if (cap < 1) cap = 1;
+ if (cap > 65535) cap = 65535;
+ double used = 0, bud = 0, free_b;
+ int have = coli_vk_mem_budget(&used, &bud);
+ free_b = have ? (bud - used) * 1e9 : (double)coli_vk_device_local_bytes() / 4;
+ /* Expert outputs and CPU fallback scratch also grow with the chunk on a
+ * discrete device. Bound both memories, including Windows commit headroom. */
+ double avail = compat_mem_available_gb() * 1e9;
+ if (avail > 0 && avail < free_b) free_b = avail;
+ if (free_b < 0) free_b = 0;
+ double fit = row_bytes ? free_b / 2 / (double)row_bytes : cap;
+ int rows = fit >= cap ? cap : (int)fit;
+ if (rows < cap && rows >= 256) rows = rows / 256 * 256;
+ if (rows < 1) rows = 1;
+ if (nsaid < 16) {
+ snprintf(said[nsaid], sizeof said[nsaid], "%s", engine ? engine : "");
+ rows_said[nsaid++] = rows;
+ fprintf(stderr, "[VK] %s chain: prompt chunks of up to %d rows (%.0f KiB a row; %.2f GiB free on the device%s, "
+ "half of it at most; COLI_VK_CHAIN_ROWS sets it, COLI_VK_CHAIN_ROWS_MAX caps it at %d)\n",
+ engine ? engine : "engine", rows, row_bytes / 1024.0, free_b / 1073741824.0,
+ avail > 0 ? " or in RAM" : "", cap);
+ }
+ return rows;
+}
void vkc_prof_print(void) {
if (!K.prof) return;
double tot = 0; for (int k = 0; k < PK_N; k++) tot += K.prof_ms[k];
@@ -1097,6 +1316,8 @@ void vkc_shutdown(void) {
}
if (K.cpool) vkDestroyCommandPool(K.dev, K.cpool, NULL);
kx_shutdown();
+ kab_shutdown();
+ ke_shutdown();
for (int i = 0; i < P_NPIPE; i++) {
if (K.pipe[i]) vkDestroyPipeline(K.dev, K.pipe[i], NULL);
if (K.mod[i]) vkDestroyShaderModule(K.dev, K.mod[i], NULL);
@@ -1115,6 +1336,7 @@ void vkc_shutdown(void) {
if (K.pl) vkDestroyPipelineLayout(K.dev, K.pl, NULL);
if (K.dsl) vkDestroyDescriptorSetLayout(K.dev, K.dsl, NULL);
dsv4_shutdown();
+ kvs_shutdown();
memset(&K, 0, sizeof K);
K.cur = -1; K.gemm_rows = -1;
}
@@ -1222,3 +1444,104 @@ int vkc_dsv4_swiglu(VkcBuf *a, VkcBuf *b, VkcBuf *y, const VkcDsSwiglu *p) {
VkcBind bd[6] = {B(a, 0), B(b, 0), B(NULL, 0), B(NULL, 0), B(NULL, 0), B(y, 1)};
return dsv4_rec(PK_EW, 8, p, sizeof *p, bd, 6, gx, gy);
}
+
+/* ---- a KV cache split between the device and the host (chain_kvs.comp) ------------
+ * Its own optional pipeline, as DeepSeek's: without the shader only these ops decline
+ * (and an engine's split with them: past the budget its chain declines as before). */
+static struct { VkShaderModule mod; VkPipeline pipe; int tried; } KS;
+static VkPipeline kvs_pipe(void) {
+ if (KS.pipe || KS.tried || !vkc_ready()) return KS.pipe;
+ KS.tried = 1;
+ char path[1200];
+ const char *sl = strrchr(K.core.spv_path, '/');
+ size_t pre = sl ? (size_t)(sl - K.core.spv_path) + 1 : 0;
+ const char *file = "chain_kvs.spv";
+ if (pre + strlen(file) + 1 >= sizeof path) return VK_NULL_HANDLE;
+ memcpy(path, K.core.spv_path, pre); strcpy(path + pre, file);
+ FILE *f = fopen(path, "rb");
+ if (!f) return VK_NULL_HANDLE;
+ fclose(f);
+ if ((KS.mod = load_module(K.core.spv_path, file))) KS.pipe = make_pipe(KS.mod, NULL);
+ return KS.pipe;
+}
+static void kvs_shutdown(void) {
+ if (KS.pipe) vkDestroyPipeline(K.dev, KS.pipe, NULL);
+ if (KS.mod) vkDestroyShaderModule(K.dev, KS.mod, NULL);
+ memset(&KS, 0, sizeof KS);
+}
+int vkc_kvs_ready(void) { return kvs_pipe() != VK_NULL_HANDLE; }
+int vkc_kvs_attn(VkcBuf *q, VkcBuf *kc, VkcBuf *vc, VkcBuf *out, VkcBuf *gate, VkcBuf *sel, VkcBuf *snk, VkcBuf *tab,
+ const VkcKvsAttn *p) {
+ VkPipeline pipe = kvs_pipe();
+ if (!pipe || !q || !kc || !vc || !out || !tab || p->S < 1 || p->S > 65535 || p->H < 1 || p->H > 65535 ||
+ p->KVH < 1 || p->H % p->KVH || p->hd < 1 || p->hd > 256 || p->vd < 1 || p->vd > 256 || p->B < 1 || p->ns < 1 ||
+ p->nblk < 0 || p->anchor < 0 || p->win < 0 || (p->sink && !snk) || (p->sel_row > 0 && !sel) ||
+ ((p->flags & VKC_KVS_GATE) && (!gate || !(p->flags & VKC_KVS_FIN)))) return 0;
+ struct { int mode; VkcKvsAttn b; } pc = {0, *p};
+ K.kind = PK_ATTN;
+ VkcBind bd[8] = {B(q, 0), B(kc, 0), B(vc, 0), B(out, 1), B(p->flags & VKC_KVS_GATE ? gate : NULL, 0),
+ B(p->sel_row > 0 ? sel : NULL, 0), B(p->sink ? snk : NULL, 0), B(tab, 0)};
+ return record(pipe, bd, 8, &pc, sizeof pc, (uint32_t)p->H, (uint32_t)p->S, 1);
+}
+int vkc_kvs_mla(VkcBuf *qa, VkcBuf *qr, VkcBuf *lat, VkcBuf *rope, VkcBuf *sel, VkcBuf *out, VkcBuf *tab, const VkcKvsMla *p) {
+ VkPipeline pipe = kvs_pipe();
+ if (!pipe || !qa || !lat || !out || !tab || p->S < 1 || p->S > 65535 || p->H < 1 || p->H > 65535 || p->K < 1 ||
+ p->K > 1024 || p->R < 0 || p->R > 128 || (p->R > 0 && (!rope || !qr)) || p->B < 1 || p->ns < 1 || p->nblk < 0 ||
+ p->anchor < 0 ||
+ p->kv_start < 0 || (p->sel_row > 0 && !sel)) return 0;
+ struct { int mode; VkcKvsMla b; } pc = {1, *p};
+ K.kind = PK_MLA;
+ VkcBind bd[8] = {B(qa, 0), B(p->R > 0 ? qr : NULL, 0), B(lat, 0), B(p->R > 0 ? rope : NULL, 0), B(NULL, 0),
+ B(p->sel_row > 0 ? sel : NULL, 0), B(out, 1), B(tab, 0)};
+ return record(pipe, bd, 8, &pc, sizeof pc, (uint32_t)p->H, (uint32_t)p->S, 1);
+}
+int vkc_kvs_merge(VkcBuf *dev, VkcBuf *cpu, VkcBuf *gate, VkcBuf *out, const VkcKvsMerge *p) {
+ VkPipeline pipe = kvs_pipe();
+ if (!pipe || !dev || !cpu || !out || p->n < 0 || p->H < 1 || p->d < 1 || ((p->flags & VKC_KVS_GATE) && !gate)) return 0;
+ if (p->n == 0) return open_frame() && !K.lost;
+ struct { int mode; VkcKvsMerge b; } pc = {2, *p};
+ K.kind = PK_ATTN;
+ VkcBind bd[4] = {B(dev, 0), B(cpu, 0), B(p->flags & VKC_KVS_GATE ? gate : NULL, 0), B(out, 1)};
+ uint32_t gx, gy; grid((uint64_t)p->n, &gx, &gy);
+ return record(pipe, bd, 4, &pc, sizeof pc, gx, gy, 1);
+}
+int vkc_kvs_rel(VkcBuf *q, VkcBuf *kc, VkcBuf *vc, VkcBuf *out, VkcBuf *r, VkcBuf *tau, VkcBuf *relp, VkcBuf *tab,
+ const VkcKvsRel *p) {
+ VkPipeline pipe = kvs_pipe();
+ if (!pipe || !q || !kc || !vc || !out || !tau || !tab || p->S < 1 || p->S > 65535 || p->H < 1 || p->H > 65535 ||
+ p->KVH < 1 || p->H % p->KVH || p->hd < 1 || p->hd > 256 || p->d_rel < 0 || p->d_rel > 64 ||
+ (p->d_rel > 0 && (!r || !relp)) || p->B < 1 || p->nblk < 0 || p->anchor < 0) return 0;
+ struct { int mode; VkcKvsRel b; } pc = {3, *p};
+ K.kind = PK_ATTN;
+ VkcBind bd[8] = {B(q, 0), B(kc, 0), B(vc, 0), B(out, 1), B(p->d_rel > 0 ? r : NULL, 0), B(tau, 0),
+ B(p->d_rel > 0 ? relp : NULL, 0), B(tab, 0)};
+ return record(pipe, bd, 8, &pc, sizeof pc, (uint32_t)p->H, (uint32_t)p->S, 1);
+}
+int vkc_kvs_ds_cold(VkcBuf *q, VkcBuf *win, VkcBuf *cmp, VkcBuf *cold, VkcBuf *out, VkcBuf *prm, VkcBuf *list, VkcBuf *tab, const VkcKvsDs *p) {
+ VkPipeline pipe = kvs_pipe();
+ if (!pipe || !q || !win || !out || !prm || !list || !tab || p->S < 1 || p->S > 65535 || p->H < 1 || p->H > 65535 ||
+ p->hd < 1 || p->hd > 1024 || p->cnt < 0 || p->B < 1 || p->ns < 1 || p->nblk < 0 ||
+ ((p->flags & VKC_KVS_DSCOLD) && (!cold || !(p->flags & VKC_KVS_DSFIN))) ||
+ ((p->flags & VKC_KVS_V4) && !(p->flags & VKC_KVS_DSFIN))) return 0;
+ struct { int mode; VkcKvsDs b; } pc = {4, *p};
+ K.kind = PK_ATTN;
+ VkcBind bd[8] = {B(q, 0), B(win, 0), B(cmp ? cmp : win, 0), B(out, 1), B(prm, 0), B(list, 0), B(cold, 0), B(tab, 0)};
+ return record(pipe, bd, 8, &pc, sizeof pc, (uint32_t)p->H, (uint32_t)p->S, 1);
+}
+int vkc_kvs_ds(VkcBuf *q, VkcBuf *win, VkcBuf *cmp, VkcBuf *out, VkcBuf *prm, VkcBuf *list, VkcBuf *tab, const VkcKvsDs *p) {
+ return vkc_kvs_ds_cold(q, win, cmp, NULL, out, prm, list, tab, p);
+}
+unsigned long long vkc_serial(void) {
+ unsigned long long s = 0;
+ for (int i = 0; i < VKC_FRAMES; i++)
+ if (!K.fr[i].open && K.fr[i].serial > s) s = K.fr[i].serial;
+ return s;
+}
+int vkc_wait_serial(unsigned long long serial) {
+ if (!vkc_ready()) return 0;
+ for (int i = 0; i < VKC_FRAMES; i++) {
+ VkcFrame *f = &K.fr[i];
+ if (f->serial == serial && !f->open) return frame_wait(f) && !K.lost;
+ }
+ return !K.lost; /* reused since: it completed long ago */
+}
diff --git a/c/vk_chain.h b/c/vk_chain.h
index c30ab8ba..b3749e1e 100644
--- a/c/vk_chain.h
+++ b/c/vk_chain.h
@@ -29,6 +29,9 @@
* chain_hgemv, chain_dsa, its k-pooled modes included), Kimi Delta Attention
* (chain_kda), manifold-constrained hyper-connections (chain_mhc) and Kimi K3's
* attention residuals and SiTU-GLU (chain_ares): loaded beside the others but optional.
+ * - a KV cache past the device's budget (chain_kvs, vkc_kvs_*; vk_kvsplit.h drives it):
+ * the attention over the part of the cache on the device, merged with the host's
+ * part; optional too.
*
* Threading: the engine thread only (the main queue is the backend's, used from the
* same thread by coli_vk_matmul; the expert tier submits on its own queue).
@@ -105,6 +108,15 @@ int vkc_attn(VkcBuf *q, VkcBuf *kc, VkcBuf *vc, VkcBuf *o, VkcBuf *gate, VkcBuf
typedef struct { VkcAttn a; int win, ring, vd, kv_pm, sink, sink_off; } VkcAttnW;
int vkc_attn_w(VkcBuf *q, VkcBuf *kc, VkcBuf *vc, VkcBuf *o, VkcBuf *gate, VkcBuf *sel, VkcBuf *snk,
const VkcAttnW *p);
+/* From this many rows (COLI_VK_ATTN_BLOCK, 16; 0 = never) vkc_attn and vkc_attn_w take
+ * chain_attnb.comp: a workgroup per KV head and block of rows, each K and V row read once
+ * per block instead of once per (head, row); below it chain_attn as before. */
+int vkc_attn_block_rows(void);
+/* The attention ops (vkc_attn, vkc_attn_w, vkc_mla_core, vkc_relattn) whose rows x positions
+ * x heads x head dim pass COLI_VK_ATTN_SLICE (2^32; 0 = never) record their rows in slices,
+ * each ending its frame (submitted, not waited for) and the next in a new one: no single
+ * submission of a big prompt chunk runs for seconds (a driver resets a job past its
+ * timeout: amdgpu 10 s, Windows 2 s). The rows' arithmetic does not change. */
/* chain_dnconv.comp */
typedef struct { int S, CD, CK, in_off, in_row, out_off, out_row, snap_row, order, w_off, ring_off, snap_off; } VkcDnConv;
int vkc_dnconv(VkcBuf *in, VkcBuf *w, VkcBuf *ring, VkcBuf *out, VkcBuf *snap, const VkcDnConv *p);
@@ -413,11 +425,160 @@ int vkc_dsv4_round(VkcBuf *x, VkcBuf *y, const VkcDsRound *p);
typedef struct { int n, a_off, b_off, y_off; float lim; } VkcDsSwiglu;
int vkc_dsv4_swiglu(VkcBuf *a, VkcBuf *b, VkcBuf *y, const VkcDsSwiglu *p);
+/* Big prefill chunks: the prompt rows an engine's chain runs per chunk. COLI_VK_CHAIN_ROWS=n
+ * takes n (1..65535). Unset (or "auto"): the most rows, up to COLI_VK_CHAIN_ROWS_MAX
+ * (8192), whose buffers take at most half of the device memory free at the engine's first
+ * call (VK_EXT_memory_budget, else a quarter of the device-local heap; capped by
+ * available host RAM), row_bytes a row (the engine's chain
+ * scratch plus the routed experts' outputs for that row); a multiple of 256 rows below
+ * the cap; below 256 keep the rows that fit, at least one. Decided once per engine
+ * and printed as a [VK] line. */
+int vkc_chunk_rows(const char *engine, size_t row_bytes);
+/* 1 when the chunk is the budget's (COLI_VK_CHAIN_ROWS unset or "auto"): an engine that
+ * blocks a prompt on the host before the chain (MiMo's MIMO_CHUNK, DeepSeek V4's
+ * V4_PREFILL_CHUNK) then hands the chain blocks of the chain's chunk; with
+ * COLI_VK_CHAIN_ROWS set it keeps its own blocks, as before. */
+int vkc_chunk_auto(void);
+/* ---- a KV cache split between the device and the host (chain_kvs.comp) ----------------
+ * Past the device's budget a layer's cache keeps `ns` blocks of B positions on the
+ * device (the recent window, and where an engine selects positions, the blocks read
+ * most); the rest stays in the host's RAM, which holds the whole cache anyway. The
+ * device attends over its blocks and writes a partial result (the unnormalized value
+ * sum and the softmax statistics m, l), the host attends over the rest (vk_kvsplit.h),
+ * and vkc_kvs_merge joins the two through their statistics: the full attention up to
+ * the order of the sums. The residency is an int buffer `tab`: bt[nblk] at bt_off
+ * (block -> slot, -1 in the host's RAM only; bit 30 set: a block pinned by the reads of
+ * a selection), position t in device row slot*B + t % B. The partition of a row depends
+ * on its position only (the same bits however a forward is cut into steps): the device
+ * takes the blocks pos/B - anchor .. pos/B (all of them on the device while a step
+ * holds the row) and, from a list, the pinned blocks; the host the rest. Optional like
+ * the MLA ops: vkc_kvs_ready() says whether the shader is there.
+ * vkc_kvs_attn grouped-query attention (chain_attn's) over the device's rows: K/V
+ * head-major (kvh*rows + r)*hd or position-major (kv_pm), a window, a
+ * sink (it joins the device's part), a selection list (entries outside
+ * the device skipped). VKC_KVS_FIN: every visible position is on the
+ * device, the normalized output (VKC_KVS_GATE: times sigmoid(gate)) at
+ * o_off + s*o_row + h*vd as vkc_attn writes it; else the value sum there
+ * and (m, l) at st_off + (s*H + h)*2. hd, vd <= 256.
+ * vkc_kvs_mla the MLA core (vkc_mla_core's) over the device's rows: the latent at
+ * lat_off + r*lat_row, the rope key at rope_off + r*rope_row; outputs as
+ * above with o_seg per head. K <= 1024, R <= 128.
+ * vkc_kvs_rel Inkling's attention (vkc_relattn's) over a global layer's device rows.
+ * vkc_kvs_ds DeepSeek's sparse attention (vkc_dsv4_attn's) over its window rows and
+ * the compressed rows on the device.
+ * vkc_kvs_merge segment g = (s = g / H, h) of d floats: the device's part (dev at
+ * a_off + s*a_row + h*a_seg, (m, l) at sa_off + 2g) and the host's
+ * (cpu, packed at c_off + g*d, (m, l) at sc_off + 2g) into out at o_off
+ * + s*o_row + h*o_seg, VKC_KVS_GATE: times sigmoid(gate[g_off + s*g_row
+ * + h*g_seg + i]). */
+int vkc_kvs_ready(void);
+#define VKC_KVS_FIN 1
+#define VKC_KVS_GATE 2
+typedef struct { int S, H, KVH, hd, vd, pos_base, rows, q_off, q_row, q_seg, sel_off, sel_row; float scale;
+ int k_off, v_off, win, kv_pm, sink, sink_off, B, ns, bt_off, nblk, anchor, o_off, o_row, st_off, flags,
+ g_off, g_row, g_seg; } VkcKvsAttn;
+int vkc_kvs_attn(VkcBuf *q, VkcBuf *kc, VkcBuf *vc, VkcBuf *out, VkcBuf *gate, VkcBuf *sel, VkcBuf *snk, VkcBuf *tab,
+ const VkcKvsAttn *p);
+typedef struct { int S, H, K, R, pos_base, kv_start, qa_off, qa_row, qa_seg, qr_off, qr_row, qr_seg, lat_off, lat_row,
+ rope_off, rope_row, sel_off, sel_row, o_off, o_row, o_seg; float scale;
+ int B, ns, bt_off, nblk, anchor, flags, st_off; } VkcKvsMla;
+int vkc_kvs_mla(VkcBuf *qa, VkcBuf *qr, VkcBuf *lat, VkcBuf *rope, VkcBuf *sel, VkcBuf *out, VkcBuf *tab, const VkcKvsMla *p);
+typedef struct { int n, H, d, a_off, a_row, a_seg, sa_off, c_off, sc_off, o_off, o_row, o_seg, flags, g_off, g_row, g_seg; } VkcKvsMerge;
+#define VKC_KVS_BF16 4 /* vkc_kvs_merge: the result rounded to bf16 (DeepSeek V4) */
+int vkc_kvs_merge(VkcBuf *dev, VkcBuf *cpu, VkcBuf *gate, VkcBuf *out, const VkcKvsMerge *p);
+/* Inkling's attention over the device's rows of a global layer (chain_relattn's
+ * arithmetic, causal): score = tau[s] * (q . k * scale + bias(pos - t)), the bias mixed
+ * from r (r_off + s*r_row + h*d_rel) and relp (relp_off, [d_rel][ext]) for distances
+ * below ext, tau[s] at tau_off + s; K/V head-major (kvh*rows + r)*hd; outputs as
+ * vkc_kvs_attn's (VKC_KVS_FIN: normalized at o_off + s*o_row + h*hd). hd <= 256,
+ * d_rel <= 64. */
+typedef struct { int S, H, KVH, hd, pos_base, rows, q_off, q_row; float scale; int k_off, v_off, ext, d_rel, r_off, r_row,
+ relp_off, tau_off, B, bt_off, nblk, anchor, o_off, o_row, st_off, flags; } VkcKvsRel;
+int vkc_kvs_rel(VkcBuf *q, VkcBuf *kc, VkcBuf *vc, VkcBuf *out, VkcBuf *r, VkcBuf *tau, VkcBuf *relp, VkcBuf *tab,
+ const VkcKvsRel *p);
+/* DeepSeek's sparse attention with a sink over the device's rows (vkc_dsv4_attn's
+ * arithmetic): row s's list (l_off + s*l_row, cnt entries) of window rows (e < nwin,
+ * win at w_off + e*hd) and compressed rows c = e - nwin (in a slot: cmp at c_off +
+ * (slot*B + c % B)*hd; a compressed row in the host's RAM only is skipped), the sink at
+ * sink_off + h of prm. VKC_KVS_DSFIN: every listed row is on the device, out = the
+ * attention at o_off + s*o_row + h*hd with vkc_dsv4_attn's bits; else the part (its max
+ * over the sink too). VKC_KVS_V4: the weights and the output rounded to bf16 as
+ * DeepSeek V4. hd <= 1024. */
+#define VKC_KVS_V4 1
+#define VKC_KVS_DSFIN 8
+typedef struct { int S, H, hd, cnt, l_off, l_row, nwin, w_off, c_off, q_off, q_row, sink_off, flags; float scale;
+ int B, ns, bt_off, nblk, o_off, o_row, st_off; } VkcKvsDs;
+int vkc_kvs_ds(VkcBuf *q, VkcBuf *win, VkcBuf *cmp, VkcBuf *out, VkcBuf *prm, VkcBuf *list, VkcBuf *tab, const VkcKvsDs *p);
+/* Complete sparse attention with selected cold rows staged from canonical RAM.
+ * List entries -2-j address cold row j; -1 still means skipped. This keeps the
+ * original list order and, for V4, one maximum before rounding weights to bf16.
+ * V4 requires DSFIN: independently rounded partials cannot be merged equivalently. */
+#define VKC_KVS_DSCOLD 16
+int vkc_kvs_ds_cold(VkcBuf *q, VkcBuf *win, VkcBuf *cmp, VkcBuf *cold, VkcBuf *out,
+ VkcBuf *prm, VkcBuf *list, VkcBuf *tab, const VkcKvsDs *p);
+/* Frames by serial: vkc_serial() is the serial of the frame vkc_submit sent last;
+ * vkc_wait_serial(s) waits until that frame has completed while later ones run on
+ * (1; 0 on a lost device). The split's host part starts once the frame holding the
+ * queries is through, while the device runs its own part in the next frame. */
+unsigned long long vkc_serial(void);
+int vkc_wait_serial(unsigned long long serial);
+/* ---- the decision engines' encoders (chain_enc.comp) ---------------------------------
+ * Laya's ModernBERT and decision head (laya.c) and GLiNER2.5-Decide's DeBERTa-v3
+ * (gliner_decide.c) as one recorded forward: no cache, no state, every row of a request
+ * at once, so the attention reads the step's own q, k and v and a row sees a range of
+ * rows (its sequence, a window inside it) instead of a causal prefix. Made on first use
+ * like inkling's ops: vkc_enc_ready() says whether the shader is there. Offsets and
+ * strides in floats; rng, ridx, pos and idx are int32 buffers. The shader's header has
+ * each op's arithmetic.
+ * vkc_enc_norm LayerNorm of a (+ b with VKC_ENC_ADD; the sum written back to a with
+ * VKC_ENC_SUM) by w at w_off and the bias at bias_off (-1 none) in prm,
+ * into y; D <= 4096
+ * vkc_enc_bias y = act(y + bias) in place over n = rows*O (b_off -1: no bias)
+ * vkc_enc_geglu y = act(u + bu) * (g + bg), u and g the halves of a 2I-wide row
+ * vkc_enc_rope rotate-half RoPE in place on the query and key heads (part_off apart),
+ * cos and sin from a table at row pos[r]
+ * vkc_enc_addrow y += a table row picked per row by idx[r]
+ * vkc_enc_attn attention over each row's range rng[2r]..rng[2r+1] (inclusive), with
+ * DeBERTa's two relative terms (VKC_ENC_C2P, VKC_ENC_P2C: r = ridx[pos[i]
+ * - pos[j] + rc] - r0, c2p[h][i][r] at c_off in c2p, p2c[h][j][r] at p_off
+ * in p2c, nr buckets a row); hd <= 128
+ * vkc_enc_rel those products: y[y_off + (h*S + i)*nr + t] = x_i,h . p[r0 + t],h for
+ * every head, row and bucket t < nr (x at x_off + i*x_row + h*hd, p at
+ * p_off + r*p_row + h*hd: the relative embeddings through the layer's key
+ * or query projection) */
+int vkc_enc_ready(void);
+#define VKC_ENC_ADD 1
+#define VKC_ENC_SUM 2
+#define VKC_ENC_ACT_NONE 0
+#define VKC_ENC_ACT_GELU 1 /* erf */
+#define VKC_ENC_ACT_GELU_TANH 2
+#define VKC_ENC_ACT_RELU 3
+#define VKC_ENC_C2P 1
+#define VKC_ENC_P2C 2
+typedef struct { int mode, rows, D, a_off, a_row, b_off, b_row, y_off, y_row, w_off, bias_off, flags; float eps; } VkcEncNorm;
+int vkc_enc_norm(VkcBuf *a, VkcBuf *b, VkcBuf *prm, VkcBuf *y, const VkcEncNorm *p);
+typedef struct { int mode, n, O, y_off, y_row, b_off, act; } VkcEncBias;
+int vkc_enc_bias(VkcBuf *y, VkcBuf *prm, const VkcEncBias *p);
+typedef struct { int mode, n, I, a_off, a_row, y_off, y_row, b_off, act; } VkcEncGeglu;
+int vkc_enc_geglu(VkcBuf *a, VkcBuf *prm, VkcBuf *y, const VkcEncGeglu *p);
+typedef struct { int mode, n, H, hd, x_off, x_row, part_off, cos_off, sin_off; } VkcEncRope;
+int vkc_enc_rope(VkcBuf *x, VkcBuf *tab, VkcBuf *pos, const VkcEncRope *p);
+typedef struct { int mode, n, D, y_off, y_row, t_off; } VkcEncAddRow;
+int vkc_enc_addrow(VkcBuf *y, VkcBuf *tab, VkcBuf *idx, const VkcEncAddRow *p);
+typedef struct { int mode, S, H, hd, q_off, k_off, v_off, kv_row, o_off, o_row, flags, c_off, p_off, nr, rc;
+ float scale; int r0; } VkcEncAttn;
+int vkc_enc_attn(VkcBuf *qkv, VkcBuf *c2p, VkcBuf *p2c, VkcBuf *o, VkcBuf *rng, VkcBuf *ridx, VkcBuf *pos,
+ const VkcEncAttn *p);
+typedef struct { int mode, S, H, hd, x_off, x_row, p_off, p_row, y_off, nr, r0; } VkcEncRel;
+int vkc_enc_rel(VkcBuf *x, VkcBuf *pt, VkcBuf *y, const VkcEncRel *p);
+
/* counters, for the engines' [VK] lines */
typedef struct {
unsigned long long frames, waits, ops, matmuls, gemms, barriers, bytes_up, bytes_down;
double wait_ms; /* host time blocked in fence waits */
size_t dev_bytes; /* live chain buffers */
+ unsigned long long attn_blocked; /* attention calls through chain_attnb (blocks of rows) */
+ unsigned long long attn_slices; /* extra submissions that cut a long attention (COLI_VK_ATTN_SLICE) */
} VkcStats;
void vkc_stats(VkcStats *st);
/* COLI_VK_CHAIN_PROF=1: one stderr line of device time per kind of op */
diff --git a/c/vk_kvsplit.h b/c/vk_kvsplit.h
new file mode 100644
index 00000000..a3d2afe9
--- /dev/null
+++ b/c/vk_kvsplit.h
@@ -0,0 +1,1020 @@
+/* vk_kvsplit.h -- a chain engine's KV cache past the device's budget: the part read
+ * most on the device, the rest in the host's RAM (header-only; a *_chain.h includes it
+ * after vk_chain.h).
+ *
+ * The chain keeps the host's KV cache canonical and mirrors each attention layer's
+ * cache on the device behind a watermark (docs/vulkan.md, "The dense chain"). A long
+ * context makes those mirrors larger than the device can hold. Then each split layer
+ * keeps `rows` = ns * B rows on the device: ns slots of B positions, nw of them a window
+ * over the newest blocks (the step's own rows always land there) and np pinned to the
+ * blocks the engine's selection (QSA, DSA) reads most. The rest of the positions are
+ * only in the host's cache, which holds every row anyway. A step's attention then runs
+ * in two parts at once: the device over its share (vkc_kvs_attn / vkc_kvs_mla, frame
+ * F2) and the CPU over the rest (vkc_kv_host_attn, from the queries frame F1 brought
+ * down), and vkc_kvs_merge joins them through their softmax statistics (frame F3): the
+ * full attention up to the order of the sums. A step whose rows see nothing outside the
+ * device's share takes one frame as before (VKC_KVS_FIN).
+ *
+ * The device's share of a row depends on the row's position only: the `anchor` + 1
+ * newest blocks up to the row's (all of them in the window while a step holds the row,
+ * since a step writes at most `chunk` rows), plus, from a list, the pinned blocks. The
+ * host takes the rest, in chunks fixed by position. So a row's bits do not depend on
+ * how a forward is cut into steps (a prompt in one chunk or many, a decode step or a
+ * verify, a resumed prefix or a cold one), except through the pins, which follow the
+ * reads: pins are off by default, COLI_VK_KV_PIN=1 opts in. DeepSeek stages only
+ * selected cold rows to preserve its complete sparse attention on the device.
+ *
+ * The plan (vkc_kv_plan): whole mirrors when they fit the device's budget (the chain
+ * as it was, bit for bit), else the split, sized from the budget and resized as the
+ * host's cache grows. Knobs:
+ * COLI_VK_KV_SPLIT=0 never split: past the budget the chain declines as before
+ * COLI_VK_KV_DEVICE_ROWS=N keep N rows per layer on the device whenever the cache is
+ * larger (the tests' small device; a measurement's split)
+ * COLI_VK_KV_BLOCK=B positions per block (default 64)
+ * COLI_VK_KV_PIN=1 opt in to read-based pins (GQA/MLA rounding can then
+ * depend on read history; the default is deterministic)
+ *
+ * The residency (VkcKvTab, per layer): bt (block -> slot or -1) and rb (slot -> block
+ * or -1), uploaded to `tab` when they change; `valid`, the watermark: every resident
+ * row below it equals the host's. A step from pos_base first uploads the resident rows
+ * in [valid, pos_base) and the whole of a block that just got its slot (vkc_kv_push),
+ * then writes its new rows into their slots (vkc_kv_store); a CPU step lowers valid
+ * (vkc_kv_lower); a new host cache resets the tables (vkc_kv_reset). */
+#ifndef COLI_VK_KVSPLIT_H
+#define COLI_VK_KVSPLIT_H
+#include "vk_chain.h"
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <math.h>
+#include <time.h>
+#ifdef _OPENMP
+#include <omp.h>
+#endif
+
+typedef struct {
+ int nblk, wb0, wb1; /* blocks the table covers; the window's first and last block */
+ int *bt, *rb; /* block -> slot (-1), slot -> block (-1) */
+ unsigned char *pin, *fresh; /* per slot: pinned by reads; given its block since the last push */
+ unsigned *hits; /* per block: reads seen in the selection lists */
+ unsigned long long counted; /* lists counted (the hits halve every 128) */
+ int valid; /* resident rows below it equal the host's */
+ int dirty; /* bt / rb changed since their upload */
+} VkcKvTab;
+
+/* One array of a layer's cache: per position nseg segments of seglen floats (the KV heads
+ * of a head-major cache; 1 segment for a position-major one). The host's: segment h of
+ * position t at host + h*host_seg + t*seglen. The device's: segment h of row r at
+ * dev_off + (h*rows + r)*seglen in dev. */
+typedef struct {
+ int nseg, seglen;
+ const float *host; size_t host_seg;
+ VkcBuf *dev; size_t dev_off;
+} VkcKvPart;
+
+typedef struct {
+ int on; /* split: every split layer holds `rows` rows on the device */
+ int B, ns, nw, np, rows, nl; /* block size, slots (window, pinned), rows = ns*B, layers */
+ int cap; /* host positions the tables cover */
+ int chunk; /* the most rows a step may write */
+ int anchor; /* a row's device share: blocks pos/B - anchor .. pos/B */
+ VkcKvTab *t;
+ VkcBuf *tab; size_t tab_stride; /* per layer: bt[cap/B+1] then rb[ns], ints */
+ VkcBuf *dpart, *cpart; /* the device's part (F2) and the host's (written in F3) */
+ VkcBuf *qd, *sd; /* DOWN: the queries and the lists the host's part reads */
+ VkcBuf *cold_rows, *cold_list; /* selected sparse rows from RAM, reused one query at a time */
+ float *hpart; size_t hpart_n;
+ int *cold, *lo, *cnt, *causal; size_t cold_n, lo_n;
+ unsigned long long steps, splits, host_pos, pins, staged_rows; double host_ms, wait_ms;
+ const char *engine;
+} VkcKvSplit;
+
+static double vkc_kv_now(void) { struct timespec t; clock_gettime(CLOCK_MONOTONIC, &t); return t.tv_sec * 1e3 + t.tv_nsec / 1e6; }
+
+static void vkc_kv_free(VkcKvSplit *ks) {
+ if (ks->t) for (int i = 0; i < ks->nl; i++) {
+ VkcKvTab *t = &ks->t[i];
+ free(t->bt); free(t->rb); free(t->pin); free(t->fresh); free(t->hits);
+ }
+ free(ks->t); ks->t = NULL;
+ vkc_free(ks->tab); vkc_free(ks->dpart); vkc_free(ks->cpart); vkc_free(ks->qd); vkc_free(ks->sd);
+ vkc_free(ks->cold_rows); vkc_free(ks->cold_list); ks->cold_rows = ks->cold_list = NULL;
+ ks->tab = ks->dpart = ks->cpart = ks->qd = ks->sd = NULL;
+ free(ks->hpart); free(ks->cold); free(ks->lo); free(ks->cnt); free(ks->causal);
+ ks->hpart = NULL; ks->cold = ks->lo = ks->cnt = ks->causal = NULL; ks->hpart_n = ks->cold_n = ks->lo_n = 0;
+ ks->on = 0;
+}
+
+/* Every table empty: nothing resident (a new host cache, or a reset). */
+static void vkc_kv_reset(VkcKvSplit *ks) {
+ if (!ks->on) return;
+ for (int i = 0; i < ks->nl; i++) {
+ VkcKvTab *t = &ks->t[i];
+ for (int b = 0; b < t->nblk; b++) t->bt[b] = -1;
+ for (int j = 0; j < ks->ns; j++) { t->rb[j] = -1; t->pin[j] = 0; t->fresh[j] = 0; }
+ memset(t->hits, 0, (size_t)t->nblk * sizeof *t->hits);
+ t->valid = 0; t->dirty = 1; t->wb0 = t->wb1 = 0; t->counted = 0;
+ }
+}
+
+static long vkc_kv_env(const char *name, long def) {
+ const char *e = getenv(name);
+ return e && *e ? atol(e) : def;
+}
+
+/* Decide the device's share of `nl` layers' caches of `cap` positions, `row_bytes` a
+ * position and layer on the device (every array), steps of up to `chunk` rows; `selects`:
+ * the engine picks positions (pinned slots pay off); `held`: the bytes of the mirrors the
+ * engine holds now (freed before the new ones). Returns 1 for whole mirrors (ks->on = 0:
+ * the chain as before), 2 for the split (ks->on = 1, tables ready, every block in the
+ * host's RAM until a step places it), 0 when not even the split can run. */
+/* vkc_kv_plan_need: the same with the whole mirrors' bytes given (need_bytes; 0: nl * cap
+ * * row_bytes), for layers whose caches differ (DeepSeek's compressed rows, one ratio a
+ * layer): the decision compares those bytes with the budget, the tables cover cap
+ * positions and the split keeps the same rows in every layer. */
+static int vkc_kv_plan_need(VkcKvSplit *ks, const char *engine, int nl, size_t row_bytes, int cap, int chunk, int selects,
+ size_t held, size_t need_bytes) {
+ vkc_kv_free(ks);
+ ks->engine = engine;
+ if (nl < 1 || cap < 1 || !row_bytes) return 1;
+ size_t need = need_bytes ? need_bytes : (size_t)nl * (size_t)cap * row_bytes;
+ long forced = vkc_kv_env("COLI_VK_KV_DEVICE_ROWS", 0);
+ int split_ok = vkc_kv_env("COLI_VK_KV_SPLIT", 1) != 0 && vkc_kvs_ready();
+ int B = (int)vkc_kv_env("COLI_VK_KV_BLOCK", 64);
+ if (B < 1) B = 1;
+ long rows;
+ if (forced > 0) {
+ if (forced >= cap) return 1;
+ rows = forced;
+ } else {
+ double used = 0, bud = 0;
+ size_t avail;
+ if (coli_vk_mem_budget(&used, &bud)) avail = bud > used ? (size_t)((bud - used) * 1e9) : 0;
+ else {
+ size_t dev = coli_vk_device_local_bytes(), w = 0, n = 0;
+ VkcStats st; vkc_stats(&st);
+ coli_vk_mem_info(&w, &n);
+ avail = dev > w + st.dev_bytes ? dev - w - st.dev_bytes : 0;
+ }
+ avail += held;
+ size_t room = avail / 10 * 8; /* a fifth for scratch, frames and the rest */
+ if (need <= room) return 1;
+ rows = (long)(room / ((size_t)nl * row_bytes));
+ }
+ if (!split_ok) return 1; /* as before: the whole mirrors, or the decline */
+ (void)chunk; /* the step's rows follow the split (ks->chunk), so that a row's share is the same in every step */
+ int ns = (int)(rows / B);
+ if (ns < 3) ns = 3;
+ int nw = ns, np = 0;
+ if (selects && vkc_kv_env("COLI_VK_KV_PIN", 0) != 0) { nw = ns / 2 > 3 ? ns / 2 : 3; np = ns - nw; }
+ if ((long)ns * B >= cap) return 1;
+ ks->B = B; ks->ns = ns; ks->nw = nw; ks->np = np; ks->rows = ns * B; ks->nl = nl; ks->cap = cap;
+ /* A step writes at most cb blocks' worth of rows (an eighth of the window, a block at
+ * least), so its rows span at most cb blocks past its first. A row's share reaches
+ * back `anchor` blocks: every block of it stays in the window while a step holds the
+ * row (anchor <= nw - 1 - cb), and it covers the step's earlier rows (anchor >= cb),
+ * which reach the host's cache only after the step. */
+ int cb = nw / 8 > 1 ? nw / 8 : 1;
+ if (nw - 1 - cb < cb) cb = (nw - 1) / 2;
+ ks->chunk = cb * B;
+ ks->anchor = nw - 1 - cb;
+ int nblk = (cap + B - 1) / B + 1;
+ ks->tab_stride = (size_t)nblk + (size_t)ns;
+ ks->t = (VkcKvTab *)calloc((size_t)nl, sizeof *ks->t);
+ if (!ks->t) return 0;
+ for (int i = 0; i < nl; i++) {
+ VkcKvTab *t = &ks->t[i];
+ t->nblk = nblk;
+ t->bt = (int *)malloc((size_t)nblk * sizeof(int)); t->rb = (int *)malloc((size_t)ns * sizeof(int));
+ t->pin = (unsigned char *)calloc((size_t)ns, 1); t->fresh = (unsigned char *)calloc((size_t)ns, 1);
+ t->hits = (unsigned *)calloc((size_t)nblk, sizeof(unsigned));
+ if (!t->bt || !t->rb || !t->pin || !t->fresh || !t->hits) { vkc_kv_free(ks); return 0; }
+ }
+ ks->tab = vkc_buf((size_t)nl * ks->tab_stride * sizeof(int), VKC_DEV);
+ if (!ks->tab) { vkc_kv_free(ks); return 0; }
+ ks->on = 1;
+ vkc_kv_reset(ks);
+ fprintf(stderr, "[VK] %s chain: the KV cache split past the device's budget: %d of %d positions a layer on the device "
+ "(%d blocks of %d, %d for pins by reads; a row's newest %d positions on the device), the rest in RAM "
+ "(%.1f MiB on the device instead of %.1f)\n",
+ engine, ks->rows, cap, ns, B, np, (ks->anchor + 1) * B, (double)nl * ks->rows * row_bytes / 1048576.0,
+ need / 1048576.0);
+ return 2;
+}
+static int vkc_kv_plan(VkcKvSplit *ks, const char *engine, int nl, size_t row_bytes, int cap, int chunk, int selects,
+ size_t held) {
+ return vkc_kv_plan_need(ks, engine, nl, row_bytes, cap, chunk, selects, held, 0);
+}
+
+/* A slot for a block of layer li: a free one, else the oldest block behind the window
+ * that nothing pinned, else the pinned block behind the window read least. */
+static int vkc_kv_take(VkcKvSplit *ks, VkcKvTab *t) {
+ int best = -1;
+ for (int j = 0; j < ks->ns; j++) if (t->rb[j] < 0) return j;
+ for (int j = 0; j < ks->ns; j++)
+ if (!t->pin[j] && t->rb[j] < t->wb0 && (best < 0 || t->rb[j] < t->rb[best])) best = j;
+ if (best >= 0) return best;
+ for (int j = 0; j < ks->ns; j++)
+ if (t->pin[j] && t->rb[j] < t->wb0 && (best < 0 || t->hits[t->rb[j]] < t->hits[t->rb[best]])) best = j;
+ return best;
+}
+static void vkc_kv_assign(VkcKvTab *t, int j, int b, int pinned) {
+ if (t->rb[j] >= 0) t->bt[t->rb[j]] = -1;
+ t->rb[j] = b; t->bt[b] = j; t->pin[j] = (unsigned char)pinned; t->fresh[j] = 1; t->dirty = 1;
+}
+
+/* Place the window of a step writing [pos_base, pos_base + S), S <= ks->chunk: its
+ * blocks and the ones its rows' shares reach back to get slots. */
+static void vkc_kv_place(VkcKvSplit *ks, int li, int pos_base, int S) {
+ VkcKvTab *t = &ks->t[li];
+ if (pos_base + S <= 0) { t->wb0 = t->wb1 = 0; return; }
+ int B = ks->B, wb1 = (pos_base + S - 1) / B, wb0 = wb1 - ks->nw + 1;
+ if (wb0 < 0) wb0 = 0;
+ if (wb1 >= t->nblk) wb1 = t->nblk - 1;
+ t->wb0 = wb0; t->wb1 = wb1;
+ for (int j = 0; j < ks->ns; j++) /* blocks past the step (a rewind): nothing reads them */
+ if (t->rb[j] > wb1) { t->bt[t->rb[j]] = -1; t->rb[j] = -1; t->pin[j] = 0; t->dirty = 1; }
+ for (int b = wb0; b <= wb1; b++) {
+ if (t->bt[b] >= 0) continue;
+ int j = vkc_kv_take(ks, t);
+ if (j >= 0) vkc_kv_assign(t, j, b, 0);
+ }
+}
+
+/* Selection engines, after vkc_kv_run (the step's host part has used the tables): count
+ * the step's reads and pin up to two of the blocks read most behind the newest row's
+ * share (anchored: the GQA and MLA lists; else DeepSeek's, whose share is the
+ * residency, so only blocks in the host's RAM), ahead of pinned blocks read less.
+ * vkc_kv_count: lists as the GQA/MLA ops read them (row s: sel[s*sel_row] = count, the
+ * positions after; a negative count reads every position); vkc_kv_count_fixed:
+ * DeepSeek's (cnt entries a row, no count; entries below base are the window's, the
+ * others index e - base). */
+static void vkc_kv_promote(VkcKvSplit *ks, VkcKvTab *t, int anchored) {
+ if (++t->counted % 128 == 0) for (int b = 0; b < t->nblk; b++) t->hits[b] >>= 1;
+ int lim = anchored ? t->wb1 - ks->anchor : t->wb0;
+ for (int k = 0; k < 2; k++) {
+ int best = -1;
+ for (int b = 0; b < lim && b < t->nblk; b++) {
+ int j = t->bt[b];
+ if (!t->hits[b] || (j >= 0 && (t->pin[j] || !anchored))) continue;
+ if (best < 0 || t->hits[b] > t->hits[best]) best = b;
+ }
+ if (best < 0) return;
+ int npin = 0, low = -1;
+ for (int j = 0; j < ks->ns; j++) if (t->pin[j]) { npin++; if (low < 0 || t->hits[t->rb[j]] < t->hits[t->rb[low]]) low = j; }
+ if (npin >= ks->np) {
+ if (low < 0 || t->hits[t->rb[low]] >= t->hits[best]) return;
+ t->pin[low] = 0; t->dirty = 1; /* behind the window it may now be evicted; in it, it stays */
+ }
+ int j = t->bt[best];
+ if (j >= 0) { t->pin[j] = 1; t->dirty = 1; ks->pins++; continue; } /* on the device already */
+ for (int q = 0; q < ks->ns && j < 0; q++) if (t->rb[q] < 0) j = q;
+ for (int q = 0; q < ks->ns; q++)
+ if (!t->pin[q] && t->rb[q] >= 0 && t->rb[q] < t->wb0 && (j < 0 || (t->rb[j] >= 0 && t->rb[q] < t->rb[j]))) j = q;
+ if (j < 0) return;
+ vkc_kv_assign(t, j, best, 1);
+ ks->pins++;
+ }
+}
+static void vkc_kv_count(VkcKvSplit *ks, int li, const int *sel, int S, int sel_row) {
+ if (!ks->on || ks->np < 1 || !sel) return;
+ VkcKvTab *t = &ks->t[li];
+ for (int s = 0; s < S; s++) {
+ const int *r = sel + (size_t)s * sel_row;
+ for (int j = 0; j < r[0]; j++) if (r[1 + j] >= 0 && r[1 + j] / ks->B < t->nblk) t->hits[r[1 + j] / ks->B]++;
+ }
+ vkc_kv_promote(ks, t, 1);
+}
+static void vkc_kv_count_fixed(VkcKvSplit *ks, int li, const int *sel, int S, int sel_row, int cnt, int base) {
+ if (!ks->on || ks->np < 1 || !sel) return;
+ VkcKvTab *t = &ks->t[li];
+ for (int s = 0; s < S; s++) {
+ const int *r = sel + (size_t)s * sel_row;
+ for (int j = 0; j < cnt; j++) if (r[j] >= base && (r[j] - base) / ks->B < t->nblk) t->hits[(r[j] - base) / ks->B]++;
+ }
+ vkc_kv_promote(ks, t, 0);
+}
+
+/* Record the uploads that make layer li's resident rows below pos_base the host's, and
+ * its table when it changed. Into the open frame. */
+static int vkc_kv_push(VkcKvSplit *ks, int li, const VkcKvPart *pt, int npart, int pos_base) {
+ VkcKvTab *t = &ks->t[li];
+ int B = ks->B, ok = 1;
+ for (int j = 0; j < ks->ns && ok; j++) {
+ int b = t->rb[j];
+ if (b < 0) continue;
+ int t0 = t->fresh[j] ? b * B : (t->valid > b * B ? t->valid : b * B), t1 = b * B + B < pos_base ? b * B + B : pos_base;
+ t->fresh[j] = 0;
+ if (t1 <= t0) continue;
+ for (int k = 0; k < npart && ok; k++)
+ for (int h = 0; h < pt[k].nseg && ok; h++)
+ ok = vkc_write(pt[k].dev, pt[k].dev_off + ((size_t)h * ks->rows + (size_t)j * B + (t0 - b * B)) * pt[k].seglen,
+ pt[k].host + h * pt[k].host_seg + (size_t)t0 * pt[k].seglen,
+ (size_t)(t1 - t0) * pt[k].seglen * sizeof(float));
+ }
+ t->valid = pos_base;
+ if (ok && t->dirty) { /* bt with bit 30 on the pinned blocks, then rb */
+ size_t o = (size_t)li * ks->tab_stride;
+ int *bt = (int *)malloc((size_t)t->nblk * sizeof(int));
+ if (!bt) return 0;
+ for (int b = 0; b < t->nblk; b++) bt[b] = t->bt[b] >= 0 && t->pin[t->bt[b]] ? (t->bt[b] | 0x40000000) : t->bt[b];
+ ok = vkc_write(ks->tab, o, bt, (size_t)t->nblk * sizeof(int)) &&
+ vkc_write(ks->tab, o + (size_t)t->nblk, t->rb, (size_t)ks->ns * sizeof(int));
+ free(bt);
+ t->dirty = !ok;
+ }
+ return ok;
+}
+
+/* Record the copies of a step's new rows into their slots: row s, segment h at src_off
+ * + s*src_row + h*src_seg in src (seglen floats each). */
+static int vkc_kv_store(VkcKvSplit *ks, int li, const VkcKvPart *pt, VkcBuf *src, size_t src_off, size_t src_row,
+ size_t src_seg, int pos_base, int S) {
+ VkcKvTab *t = &ks->t[li];
+ VkcRegion *rg = (VkcRegion *)malloc(sizeof *rg * (size_t)S * pt->nseg);
+ if (!rg) return 0;
+ int n = 0;
+ for (int s = 0; s < S; s++) {
+ int p = pos_base + s, j = t->bt[p / ks->B];
+ if (j < 0) { free(rg); return 0; } /* the window holds every row of a step */
+ for (int h = 0; h < pt->nseg; h++)
+ rg[n++] = (VkcRegion){pt->dev_off + ((size_t)h * ks->rows + (size_t)j * ks->B + p % ks->B) * pt->seglen,
+ src_off + (size_t)s * src_row + (size_t)h * src_seg, (size_t)pt->seglen};
+ }
+ int ok = vkc_copy_regions(pt->dev, src, rg, n);
+ free(rg);
+ return ok;
+}
+static void vkc_kv_done(VkcKvSplit *ks, int li, int end) { if (ks->on) ks->t[li].valid = end; }
+static void vkc_kv_lower(VkcKvSplit *ks, int li, int pos_base) { if (ks->on && ks->t[li].valid > pos_base) ks->t[li].valid = pos_base; }
+
+/* 1 when a row of the step may read a position outside the device's share: causal
+ * rows see [first, pos] (first = pos - win + 1 with a window, else kv_start), the share
+ * starts at block pos/B - anchor; with lists (lists 1) the same, conservatively. */
+static int vkc_kv_cold(const VkcKvSplit *ks, int li, int pos_base, int S, int win, int kv_start, int lists) {
+ (void)li; (void)lists;
+ for (int s = 0; s < S; s++) {
+ int pos = pos_base + s, first = win > 0 ? pos - win + 1 : kv_start;
+ if (first < kv_start) first = kv_start;
+ if (first < 0) first = 0;
+ if ((pos / ks->B - ks->anchor) * ks->B > first) return 1;
+ }
+ return 0;
+}
+
+/* 1 when a block below index n is not on the device (DeepSeek's compressed rows: any of
+ * them may be listed). */
+static int vkc_kv_cold_upto(const VkcKvSplit *ks, int li, int n) {
+ const VkcKvTab *t = &ks->t[li];
+ for (int b = 0; b * ks->B < n && b < t->nblk; b++) if (t->bt[b] < 0) return 1;
+ return 0;
+}
+static int vkc_kv_rows_alloc(VkcKvSplit *ks, int S, size_t need) {
+ if (ks->lo_n < (size_t)S) {
+ int *a = (int *)realloc(ks->lo, (size_t)S * sizeof(int));
+ if (a) ks->lo = a;
+ int *b = a ? (int *)realloc(ks->cnt, (size_t)S * sizeof(int)) : NULL;
+ if (b) ks->cnt = b;
+ int *c = b ? (int *)realloc(ks->causal, (size_t)S * sizeof(int)) : NULL;
+ if (!c) return 0;
+ ks->causal = c; ks->lo_n = (size_t)S;
+ }
+ if (ks->cold_n < need) {
+ int *c = (int *)realloc(ks->cold, (need ? need : 1) * sizeof(int));
+ if (!c) return 0;
+ ks->cold = c; ks->cold_n = need;
+ }
+ return 1;
+}
+/* The host's entries of DeepSeek's lists: row s reads ks->cold[ks->lo[s] + j], j <
+ * ks->cnt[s]: its entries e >= base (sel[s*sel_row + j], j < cnt) whose index e - base
+ * is in a block the device does not hold, as e - base, in list order. */
+static int vkc_kv_cold_fixed(VkcKvSplit *ks, int li, int S, const int *sel, int sel_row, int cnt, int base) {
+ const VkcKvTab *t = &ks->t[li];
+ if (!vkc_kv_rows_alloc(ks, S, (size_t)S * (cnt > 0 ? cnt : 1))) return 0;
+ size_t nc = 0;
+ for (int s = 0; s < S; s++) {
+ const int *r = sel + (size_t)s * sel_row;
+ ks->lo[s] = (int)nc; ks->cnt[s] = 0; ks->causal[s] = 0;
+ for (int j = 0; j < cnt; j++) {
+ int e = r[j] - base;
+ if (r[j] >= base && (e / ks->B >= t->nblk || t->bt[e / ks->B] < 0)) { ks->cold[nc++] = e; ks->cnt[s]++; }
+ }
+ }
+ return 1;
+}
+
+/* The host's positions for each row of the step: row s reads ks->cold[ks->lo[s] + j],
+ * j < ks->cnt[s]. Causal (sel NULL, or a row whose count is negative): [first, the
+ * start of the row's share) in order (first as in vkc_kv_cold); listed: the row's
+ * entries outside its share (older than its anchored blocks and not pinned), in list
+ * order. */
+static int vkc_kv_cold_rows(VkcKvSplit *ks, int li, int pos_base, int S, int win, int kv_start, const int *sel, int sel_row) {
+ const VkcKvTab *t = &ks->t[li];
+ int B = ks->B, end = pos_base + S, start = kv_start > 0 ? kv_start : 0;
+ if (!vkc_kv_rows_alloc(ks, S, (size_t)end + (sel ? (size_t)S * sel_row : 0))) return 0;
+ for (int p = start; p < end; p++) ks->cold[p - start] = p; /* the causal list: every position, ascending */
+ size_t nc = (size_t)(end > start ? end - start : 0);
+ for (int s = 0; s < S; s++) {
+ int pos = pos_base + s, first = win > 0 ? pos - win + 1 : start, a = pos / B - ks->anchor;
+ if (first < start) first = start;
+ const int *r = sel ? sel + (size_t)s * sel_row : NULL;
+ if (r && r[0] >= 0) {
+ ks->lo[s] = (int)nc; ks->cnt[s] = 0; ks->causal[s] = 0;
+ for (int j = 0; j < r[0]; j++) {
+ int p = r[1 + j], b = p / B;
+ if (p < 0 || p > pos || b >= a) continue;
+ if (b < t->nblk && t->bt[b] >= 0 && t->pin[t->bt[b]]) continue;
+ ks->cold[nc++] = p; ks->cnt[s]++;
+ }
+ continue;
+ }
+ int lo = a * B > first ? a * B : first;
+ ks->lo[s] = first - start; ks->cnt[s] = lo - first; ks->causal[s] = 1;
+ }
+ return 1;
+}
+
+/* The host's part: rows s < S, heads h < H in groups of G sharing one key row (G = H /
+ * KV heads; MLA: H). Head h of row s queries q1 (d1 floats at q1 + s*q1_row + h*q1_seg)
+ * and q2 (d2, may be 0); position t of group g has keys k1 (k1 + g*k1_grp + t*k1_pos), k2
+ * and the value v (dv floats). Positions: ks->cold / lo / cnt (vkc_kv_cold_rows).
+ * Writes acc [S][H][dv] (unnormalized) and (m, l) [S][H][2]: the device's layout. */
+typedef struct {
+ int S, H, G, d1, d2, dv; float scale;
+ const float *q1; size_t q1_row, q1_seg;
+ const float *q2; size_t q2_row, q2_seg;
+ const float *k1; size_t k1_grp, k1_pos;
+ const float *k2; size_t k2_grp, k2_pos;
+ const float *v; size_t v_grp, v_pos;
+ /* Inkling: score = tau[s] * (q . k * scale + bias), bias = sum_j r[s*r_row + h*r_seg + j]
+ * * relp[j*ext + d] for d = pos_base + s - t < ext (NULL tau / r: none) */
+ const float *tau, *r; size_t r_row, r_seg; const float *relp; int d_rel, ext, pos_base;
+ int bf16w; /* DeepSeek V4: the value sum's weights rounded to bf16 (two passes a row) */
+} VkcKvHostAttn;
+static float vkc_kv_bf16(float f) { /* nearest even, as coli_bf16_round */
+ uint32_t u; memcpy(&u, &f, 4);
+ if ((u & 0x7f800000u) != 0x7f800000u) u += 0x7fffu + ((u >> 16) & 1u);
+ u &= 0xffff0000u; memcpy(&f, &u, 4);
+ return f;
+}
+/* a dot product in eight running sums (vectorizable without reassociating a single sum) */
+static inline float vkc_kv_dot(const float *a, const float *b, int n) {
+ float s[8] = {0.f, 0.f, 0.f, 0.f, 0.f, 0.f, 0.f, 0.f};
+ int d = 0;
+ for (; d + 8 <= n; d += 8) for (int k = 0; k < 8; k++) s[k] += a[d + k] * b[d + k];
+ float t = ((s[0] + s[4]) + (s[1] + s[5])) + ((s[2] + s[6]) + (s[3] + s[7]));
+ for (; d < n; d++) t += a[d] * b[d];
+ return t;
+}
+/* one position's score for head h of row s */
+static inline float vkc_kv_score(const VkcKvHostAttn *p, int s, int h, int grp, int t, const float *qa, const float *qb) {
+ float a = vkc_kv_dot(qa, p->k1 + (size_t)grp * p->k1_grp + (size_t)t * p->k1_pos, p->d1);
+ if (qb) a += vkc_kv_dot(qb, p->k2 + (size_t)grp * p->k2_grp + (size_t)t * p->k2_pos, p->d2);
+ float v = a * p->scale;
+ if (p->r) {
+ int dist = p->pos_base + s - t;
+ float bias = 0.f;
+ if (dist < p->ext) {
+ const float *rr = p->r + (size_t)s * p->r_row + (size_t)h * p->r_seg;
+ for (int j = 0; j < p->d_rel; j++) bias += rr[j] * p->relp[(size_t)j * p->ext + dist];
+ }
+ v = v + bias;
+ }
+ return p->tau ? p->tau[s] * v : v;
+}
+#define VKC_KV_TILE 64
+#define VKC_KV_HCH 2048 /* the host's chunks: positions [k*HCH, (k+1)*HCH) of a causal row, HCH entries of a list */
+/* o = o (+) c: two partials (m, l, acc[dv]) joined through their statistics */
+static void vkc_kv_join(float *o, const float *c, int dv) {
+ if (c[0] <= -1.0e38f) return;
+ float mn = o[0] > c[0] ? o[0] : c[0];
+ float ea = o[0] > -1.0e38f ? expf(o[0] - mn) : 0.f, eb = expf(c[0] - mn);
+ for (int d = 0; d < dv; d++) o[2 + d] = o[2 + d] * ea + c[2 + d] * eb;
+ o[1] = o[1] * ea + c[1] * eb; o[0] = mn;
+}
+/* row s's chunks and the entries [*j0, *j1) of chunk c */
+static int vkc_kv_nchunks(const VkcKvSplit *ks, int s, int bf16w) {
+ int n = ks->cnt[s];
+ if (bf16w || n < 1) return 1;
+ if (!ks->causal[s]) return (n + VKC_KV_HCH - 1) / VKC_KV_HCH;
+ int p0 = ks->cold[ks->lo[s]];
+ return (p0 + n - 1) / VKC_KV_HCH - p0 / VKC_KV_HCH + 1;
+}
+static void vkc_kv_chunk(const VkcKvSplit *ks, int s, int bf16w, int c, long *j0, long *j1) {
+ int n = ks->cnt[s];
+ if (bf16w || n < 1) { *j0 = 0; *j1 = n; return; }
+ if (!ks->causal[s]) { *j0 = (long)c * VKC_KV_HCH; *j1 = *j0 + VKC_KV_HCH < n ? *j0 + VKC_KV_HCH : n; return; }
+ long p0 = ks->cold[ks->lo[s]], a = (p0 / VKC_KV_HCH + c) * VKC_KV_HCH;
+ *j0 = a - p0 > 0 ? a - p0 : 0;
+ *j1 = a + VKC_KV_HCH - p0 < n ? a + VKC_KV_HCH - p0 : n;
+}
+/* One chunk's partial for the G heads of group grp of row s, from nothing, into o. */
+static void vkc_kv_host_chunk(const VkcKvSplit *ks, const VkcKvHostAttn *p, int s, int grp, long j0, long j1, float *o) {
+ int G = p->G, dv = p->dv;
+ const int *pos = ks->cold + ks->lo[s];
+ float sc[VKC_KV_TILE];
+ for (int g = 0; g < G; g++) {
+ float *og = o + (size_t)g * (dv + 2);
+ og[0] = -3.0e38f; og[1] = 0.f;
+ memset(og + 2, 0, (size_t)dv * sizeof(float));
+ }
+ if (j1 <= j0) return;
+ if (p->bf16w) { /* two passes: the row's max, then the weights rounded against it */
+ for (int g = 0; g < G; g++) {
+ int h = grp * G + g;
+ const float *qa = p->q1 + (size_t)s * p->q1_row + (size_t)h * p->q1_seg;
+ const float *qb = p->d2 ? p->q2 + (size_t)s * p->q2_row + (size_t)h * p->q2_seg : NULL;
+ float *og = o + (size_t)g * (dv + 2), *ac = og + 2, mx = -3.0e38f, l = 0.f;
+ for (long j = j0; j < j1; j++) { float v = vkc_kv_score(p, s, h, grp, pos[j], qa, qb); if (v > mx) mx = v; }
+ for (long j = j0; j < j1; j++) {
+ float e = expf(vkc_kv_score(p, s, h, grp, pos[j], qa, qb) - mx), w = vkc_kv_bf16(e);
+ const float *vr = p->v + (size_t)grp * p->v_grp + (size_t)pos[j] * p->v_pos;
+ l += e;
+ for (int d = 0; d < dv; d++) ac[d] += w * vr[d];
+ }
+ og[0] = mx; og[1] = l;
+ }
+ return;
+ }
+ for (long t0 = j0; t0 < j1; t0 += VKC_KV_TILE) {
+ int n = (int)(j1 - t0 < VKC_KV_TILE ? j1 - t0 : VKC_KV_TILE);
+ for (int g = 0; g < G; g++) {
+ int h = grp * G + g;
+ const float *qa = p->q1 + (size_t)s * p->q1_row + (size_t)h * p->q1_seg;
+ const float *qb = p->d2 ? p->q2 + (size_t)s * p->q2_row + (size_t)h * p->q2_seg : NULL;
+ float mt = -3.0e38f;
+ for (int i = 0; i < n; i++) {
+ sc[i] = vkc_kv_score(p, s, h, grp, pos[t0 + i], qa, qb);
+ if (sc[i] > mt) mt = sc[i];
+ }
+ float *og = o + (size_t)g * (dv + 2), *ac = og + 2;
+ float mn = og[0] > mt ? og[0] : mt;
+ float corr = og[0] > -1.0e38f ? expf(og[0] - mn) : 0.f;
+ float l = og[1] * corr;
+ if (corr != 1.f) for (int d = 0; d < dv; d++) ac[d] *= corr;
+ for (int i = 0; i < n; i++) {
+ float e = expf(sc[i] - mn);
+ const float *vr = p->v + (size_t)grp * p->v_grp + (size_t)pos[t0 + i] * p->v_pos;
+ l += e;
+ for (int d = 0; d < dv; d++) ac[d] += e * vr[d];
+ }
+ og[0] = mn; og[1] = l;
+ }
+ }
+}
+/* Each (row, head)'s chunks are joined in chunk order from nothing, whatever runs them:
+ * a task per (row, group) when there are enough of them to fill the threads, else a task
+ * per chunk and the joins after: the same bits either way, and for a causal row the
+ * same chunks whatever the step around it. */
+static void vkc_kv_host_attn(VkcKvSplit *ks, const VkcKvHostAttn *p, float *acc, float *st) {
+ int S = p->S, H = p->H, G = p->G, NG = H / G, dv = p->dv;
+ long total = 0;
+ for (int s = 0; s < S; s++) total += ks->cnt[s];
+ total *= NG;
+ ks->host_pos += (unsigned long long)total;
+ int nth = 1;
+#ifdef _OPENMP
+ nth = omp_get_max_threads();
+#endif
+ size_t per = (size_t)G * (dv + 2);
+ long *tstart = (long *)malloc((size_t)(S * NG + 1) * sizeof(long));
+ if (!tstart) { fprintf(stderr, "[VK] %s chain: out of memory for the host's attention\n", ks->engine); exit(1); }
+ long k = 0;
+ for (int s = 0; s < S; s++) for (int g = 0; g < NG; g++) { tstart[s * NG + g] = k; k += vkc_kv_nchunks(ks, s, p->bf16w); }
+ tstart[S * NG] = k;
+ int by_row = S * NG >= 2 * nth;
+ long ntask = by_row ? (long)S * NG : k;
+ float *tm = (float *)malloc((size_t)ntask * per * sizeof(float));
+ if (!tm) { fprintf(stderr, "[VK] %s chain: out of memory for the host's attention\n", ks->engine); exit(1); }
+ double work = (double)total * G * (p->d1 + p->d2 + p->dv); /* a team only for work worth waking it */
+ #pragma omp parallel for schedule(dynamic, 1) if (ntask > 1 && work > 4e6)
+ for (long task = 0; task < ntask; task++) {
+ float *out = tm + (size_t)task * per;
+ if (by_row) {
+ int s = (int)(task / NG), grp = (int)(task % NG), nch = vkc_kv_nchunks(ks, s, p->bf16w);
+ float *cbuf = (float *)malloc(per * sizeof(float));
+ if (!cbuf) { fprintf(stderr, "[VK] %s chain: out of memory for the host's attention\n", ks->engine); exit(1); }
+ for (int g = 0; g < G; g++) { float *og = out + (size_t)g * (dv + 2); og[0] = -3.0e38f; og[1] = 0.f; memset(og + 2, 0, (size_t)dv * sizeof(float)); }
+ for (int c = 0; c < nch; c++) {
+ long j0, j1;
+ vkc_kv_chunk(ks, s, p->bf16w, c, &j0, &j1);
+ vkc_kv_host_chunk(ks, p, s, grp, j0, j1, cbuf);
+ for (int g = 0; g < G; g++) vkc_kv_join(out + (size_t)g * (dv + 2), cbuf + (size_t)g * (dv + 2), dv);
+ }
+ free(cbuf);
+ continue;
+ }
+ int sg = 0;
+ { int a = 0, z = S * NG; while (a + 1 < z) { int m = (a + z) / 2; if (tstart[m] <= task) a = m; else z = m; } sg = a; }
+ int s = sg / NG, grp = sg % NG;
+ long j0, j1;
+ vkc_kv_chunk(ks, s, p->bf16w, (int)(task - tstart[sg]), &j0, &j1);
+ vkc_kv_host_chunk(ks, p, s, grp, j0, j1, out);
+ }
+ for (int s = 0; s < S; s++) for (int grp = 0; grp < NG; grp++) for (int g = 0; g < G; g++) {
+ int h = grp * G + g;
+ float *o = (float *)malloc((size_t)(dv + 2) * sizeof(float));
+ if (!o) { fprintf(stderr, "[VK] %s chain: out of memory for the host's attention\n", ks->engine); exit(1); }
+ o[0] = -3.0e38f; o[1] = 0.f; memset(o + 2, 0, (size_t)dv * sizeof(float));
+ if (by_row) vkc_kv_join(o, tm + (size_t)(s * NG + grp) * per + (size_t)g * (dv + 2), dv);
+ else for (long task = tstart[s * NG + grp]; task < tstart[s * NG + grp + 1]; task++)
+ vkc_kv_join(o, tm + (size_t)task * per + (size_t)g * (dv + 2), dv);
+ memcpy(acc + ((size_t)s * H + h) * dv, o + 2, (size_t)dv * sizeof(float));
+ st[((size_t)s * H + h) * 2] = o[0]; st[((size_t)s * H + h) * 2 + 1] = o[1];
+ free(o);
+ }
+ free(tstart); free(tm);
+}
+
+/* Partials of `part_n` floats: the device's (F2) in ks->dpart, the host's in ks->cpart.
+ * Reserve them with the scratch, before a frame records anything that binds them. */
+static int vkc_kv_parts(VkcKvSplit *ks, size_t part_n) {
+ if (!vkc_reserve(&ks->dpart, part_n * sizeof(float), VKC_DEV) || !vkc_reserve(&ks->cpart, part_n * sizeof(float), VKC_DEV))
+ return 0;
+ if (ks->hpart_n < part_n) {
+ float *h = (float *)realloc(ks->hpart, part_n * sizeof(float));
+ if (!h) return 0;
+ ks->hpart = h; ks->hpart_n = part_n;
+ }
+ return 1;
+}
+
+/* The split step of one layer. The open frame F1 holds everything up to the attention.
+ * Copies the host part's inputs down (n_q floats of q from q_off, then n_q2 of q2 from
+ * q2_off when q2, into ks->qd; n_sel ints of sel from sel_off, when sel, into ks->sd),
+ * submits F1, records dev(u) in a frame F2 and submits it, waits for F1, runs host(u, q,
+ * sel) while the device runs F2 (it fills ks->hpart; q is ks->qd's floats, q2's after
+ * the first n_q), and opens F3 with ks->hpart uploaded to ks->cpart (part_n floats,
+ * vkc_kv_parts first). Returns 1 with F3 open, 0 when the device failed. */
+static int vkc_kv_run2(VkcKvSplit *ks, VkcBuf *q, size_t q_off, size_t n_q, VkcBuf *q2, size_t q2_off, size_t n_q2,
+ VkcBuf *sel, size_t sel_off, size_t n_sel,
+ int (*dev)(void *u), int (*host)(void *u, const float *q, const int *sel), void *u, size_t part_n) {
+ if (!q2) n_q2 = 0;
+ if (!vkc_reserve(&ks->qd, (n_q + n_q2 ? n_q + n_q2 : 1) * sizeof(float), VKC_DOWN) ||
+ (sel && !vkc_reserve(&ks->sd, (n_sel ? n_sel : 1) * sizeof(int), VKC_DOWN))) return 0;
+ if (!vkc_copy(ks->qd, 0, q, q_off, n_q) || (n_q2 && !vkc_copy(ks->qd, n_q, q2, q2_off, n_q2)) ||
+ (sel && !vkc_copy(ks->sd, 0, sel, sel_off, n_sel)) || !vkc_submit(0)) return 0;
+ unsigned long long f1 = vkc_serial();
+ if (!vkc_begin() || !dev(u) || !vkc_submit(0)) return 0;
+ double t0 = vkc_kv_now();
+ if (!vkc_wait_serial(f1)) return 0;
+ double t1 = vkc_kv_now();
+ int ok = host(u, (const float *)vkc_ptr(ks->qd), sel ? (const int *)vkc_ptr(ks->sd) : NULL);
+ ks->wait_ms += t1 - t0; ks->host_ms += vkc_kv_now() - t1; ks->splits++;
+ return ok && vkc_begin() && vkc_write(ks->cpart, 0, ks->hpart, part_n * sizeof(float));
+}
+static int vkc_kv_run(VkcKvSplit *ks, VkcBuf *q, size_t q_off, size_t n_q, VkcBuf *sel, size_t sel_off, size_t n_sel,
+ int (*dev)(void *u), int (*host)(void *u, const float *q, const int *sel), void *u, size_t part_n) {
+ return vkc_kv_run2(ks, q, q_off, n_q, NULL, 0, 0, sel, sel_off, n_sel, dev, host, u, part_n);
+}
+
+/* ---- one layer's attention over the split cache, the forms the engines use ----------- */
+/* The device's attention of S rows in slices of rows, each slice in a submission of its
+ * own (the rows are independent: the same bits), so that no one submission runs long
+ * enough for a GPU watchdog (amdgpu resets a ring after 10 s; one 512-row prompt chunk
+ * over a few thousand positions per row came near that on a Radeon 780M). A slice has at
+ * most COLI_VK_KV_SLICE (default 2^30) score and value multiply-adds; per row: H heads,
+ * up to n positions, w floats a position. rec(u, s0, ns) records rows [s0, s0 + ns);
+ * between slices the open frame is submitted and a new one opened, so the caller's frame
+ * is open on return as it was on entry. */
+static int vkc_kv_sliced(int S, int H, long n, int w, int (*rec)(void *u, int s0, int ns), void *u) {
+ double per = (double)H * (n > 0 ? n : 1) * (w > 0 ? w : 1);
+ double lim = (double)vkc_kv_env("COLI_VK_KV_SLICE", 1L << 30);
+ int rows = per >= lim ? 1 : (int)(lim / per);
+ if (rows >= S) return rec(u, 0, S);
+ for (int s0 = 0; s0 < S; s0 += rows) {
+ if (s0 && (!vkc_submit(0) || !vkc_begin())) return 0;
+ if (!rec(u, s0, S - s0 < rows ? S - s0 : rows)) return 0;
+ }
+ return 1;
+}
+/* GQA (vkc_attn_w's form). q at s*q_row + h*q_seg of q (q_n floats a step's rows bring
+ * down); the output at s*o_row + h*vd of out; the gate (gated) at g_off + s*g_row +
+ * h*g_seg; lists at sel_off + s*sel_row of sel (sel_row 0: causal). The host's cache:
+ * K of group g, position t at Kh + g*k_grp + t*k_pos (V likewise). */
+typedef struct {
+ VkcKvSplit *ks; int li;
+ VkcBuf *q, *kc, *vc, *gate, *sel, *snk, *out;
+ int S, H, KVH, hd, vd, pos_base, win, kv_pm, sink, sink_off;
+ int q_row, q_seg, gated, g_off, g_row, g_seg, o_row, sel_off, sel_row;
+ float scale;
+ const float *Kh, *Vh; size_t k_grp, k_pos, v_grp, v_pos;
+} VkcKvGqa;
+/* rows [s0, s0 + ns) of the device's part (fin: of the output) */
+static int vkc_kv_gqa_rec(VkcKvGqa *a, int s0, int ns, int fin) {
+ VkcKvSplit *ks = a->ks;
+ int bt = (int)((size_t)a->li * ks->tab_stride), nb = ks->t[a->li].nblk;
+ VkcKvsAttn p = {ns, a->H, a->KVH, a->hd, a->vd, a->pos_base + s0, ks->rows, s0 * a->q_row, a->q_row, a->q_seg,
+ a->sel_off + s0 * a->sel_row, a->sel_row, a->scale, 0, 0, a->win, a->kv_pm, a->sink, a->sink_off, ks->B, ks->ns,
+ bt, nb, ks->anchor, 0, 0, 0, 0, a->g_off + s0 * a->g_row, a->g_row, a->g_seg};
+ if (fin) { p.o_off = s0 * a->o_row; p.o_row = a->o_row; p.flags = VKC_KVS_FIN | (a->gated ? VKC_KVS_GATE : 0); }
+ else { p.o_off = s0 * a->H * a->vd; p.o_row = a->H * a->vd; p.st_off = a->S * a->H * a->vd + s0 * a->H * 2; }
+ return vkc_kvs_attn(a->q, a->kc, a->vc, fin ? a->out : ks->dpart, fin ? a->gate : NULL, a->sel, a->snk, ks->tab, &p);
+}
+static int vkc_kv_gqa_part(void *u, int s0, int ns) { return vkc_kv_gqa_rec((VkcKvGqa *)u, s0, ns, 0); }
+static int vkc_kv_gqa_fin(void *u, int s0, int ns) { return vkc_kv_gqa_rec((VkcKvGqa *)u, s0, ns, 1); }
+static long vkc_kv_span(const VkcKvSplit *ks, int win, int sel_row) { /* the most positions a row's device part reads */
+ long n = (long)(ks->anchor + 1) * ks->B;
+ if (win > 0 && win < n) n = win;
+ if (sel_row > 0 && sel_row > n) n = sel_row;
+ return n;
+}
+static int vkc_kv_gqa_dev(void *u) {
+ VkcKvGqa *a = (VkcKvGqa *)u;
+ return vkc_kv_sliced(a->S, a->H, vkc_kv_span(a->ks, a->win, a->sel_row), a->hd + a->vd, vkc_kv_gqa_part, a);
+}
+static int vkc_kv_gqa_host(void *u, const float *q, const int *sel) {
+ VkcKvGqa *a = (VkcKvGqa *)u; VkcKvSplit *ks = a->ks;
+ if (!vkc_kv_cold_rows(ks, a->li, a->pos_base, a->S, a->win, 0, sel, a->sel_row)) return 0;
+ VkcKvHostAttn h = {a->S, a->H, a->H / a->KVH, a->hd, 0, a->vd, a->scale, q, (size_t)a->q_row, (size_t)a->q_seg,
+ NULL, 0, 0, a->Kh, a->k_grp, a->k_pos, NULL, 0, 0, a->Vh, a->v_grp, a->v_pos,
+ NULL, NULL, 0, 0, NULL, 0, 0, 0, 0};
+ vkc_kv_host_attn(ks, &h, ks->hpart, ks->hpart + (size_t)a->S * a->H * a->vd);
+ return 1;
+}
+/* Recorded where the engine's attention op would be, into the open frame; returns with
+ * a frame open (the same one, or F3 after a split). */
+static int vkc_kv_gqa(VkcKvGqa *a) {
+ VkcKvSplit *ks = a->ks;
+ if (a->S > ks->chunk) { /* a row's share would reach past the window: the engine must clamp its steps */
+ fprintf(stderr, "[VK] %s chain: a step of %d rows past the split's %d\n", ks->engine, a->S, ks->chunk);
+ return 0;
+ }
+ ks->steps++;
+ if (!vkc_kv_cold(ks, a->li, a->pos_base, a->S, a->win, 0, a->sel_row > 0))
+ return vkc_kv_sliced(a->S, a->H, vkc_kv_span(ks, a->win, a->sel_row), a->hd + a->vd, vkc_kv_gqa_fin, a);
+ size_t part = (size_t)a->S * a->H * (a->vd + 2);
+ if (!vkc_kv_parts(ks, part) ||
+ !vkc_kv_run(ks, a->q, 0, (size_t)a->S * a->q_row, a->sel_row > 0 ? a->sel : NULL, (size_t)a->sel_off,
+ (size_t)a->S * a->sel_row, vkc_kv_gqa_dev, vkc_kv_gqa_host, a, part)) return 0;
+ VkcKvsMerge mg = {a->S * a->H, a->H, a->vd, 0, a->H * a->vd, a->vd, a->S * a->H * a->vd, 0, a->S * a->H * a->vd,
+ 0, a->o_row, a->vd, a->gated ? VKC_KVS_GATE : 0, a->g_off, a->g_row, a->g_seg};
+ if (!vkc_kvs_merge(ks->dpart, ks->cpart, a->gate, a->out, &mg)) return 0;
+ if (a->sel_row > 0) vkc_kv_count(ks, a->li, (const int *)vkc_ptr(ks->sd), a->S, a->sel_row);
+ return 1;
+}
+
+/* The MLA core (vkc_mla_core's form): qa at qa_off + s*qa_row + h*qa_seg of qa (K
+ * floats), qr at qr_off + s*qr_row + h*qr_seg of qr (R floats; may be qa's buffer), the
+ * latent context at o_off + s*o_row + h*o_seg of out; lists at sel_off + s*sel_row of
+ * sel. The host's cache: the latent of position t at Lh + t*K, the rope key at Rh + t*R. */
+typedef struct {
+ VkcKvSplit *ks; int li;
+ VkcBuf *qa, *qr, *lat, *rope, *sel, *out;
+ int S, H, K, R, pos_base, kv_start;
+ int qa_off, qa_row, qa_seg, qr_off, qr_row, qr_seg, o_off, o_row, o_seg, sel_off, sel_row;
+ float scale;
+ const float *Lh, *Rh;
+ size_t nqa; /* (set by vkc_kv_mla) the qa floats F1 brings down */
+} VkcKvMla;
+static int vkc_kv_mla_rec(VkcKvMla *a, int s0, int ns, int fin) {
+ VkcKvSplit *ks = a->ks;
+ int bt = (int)((size_t)a->li * ks->tab_stride), nb = ks->t[a->li].nblk;
+ VkcKvsMla p = {ns, a->H, a->K, a->R, a->pos_base + s0, a->kv_start, a->qa_off + s0 * a->qa_row, a->qa_row, a->qa_seg,
+ a->qr_off + s0 * a->qr_row, a->qr_row, a->qr_seg, 0, a->K, 0, a->R, a->sel_off + s0 * a->sel_row, a->sel_row,
+ 0, 0, a->K, a->scale, ks->B, ks->ns, bt, nb, ks->anchor, 0, 0};
+ if (fin) { p.o_off = a->o_off + s0 * a->o_row; p.o_row = a->o_row; p.o_seg = a->o_seg; p.flags = VKC_KVS_FIN; }
+ else { p.o_off = s0 * a->H * a->K; p.o_row = a->H * a->K; p.st_off = a->S * a->H * a->K + s0 * a->H * 2; }
+ return vkc_kvs_mla(a->qa, a->qr, a->lat, a->rope, a->sel, fin ? a->out : ks->dpart, ks->tab, &p);
+}
+static int vkc_kv_mla_part(void *u, int s0, int ns) { return vkc_kv_mla_rec((VkcKvMla *)u, s0, ns, 0); }
+static int vkc_kv_mla_fin(void *u, int s0, int ns) { return vkc_kv_mla_rec((VkcKvMla *)u, s0, ns, 1); }
+static int vkc_kv_mla_dev(void *u) {
+ VkcKvMla *a = (VkcKvMla *)u;
+ return vkc_kv_sliced(a->S, a->H, vkc_kv_span(a->ks, 0, a->sel_row), 2 * a->K + a->R, vkc_kv_mla_part, a);
+}
+static int vkc_kv_mla_host(void *u, const float *q, const int *sel) {
+ VkcKvMla *a = (VkcKvMla *)u; VkcKvSplit *ks = a->ks;
+ if (!vkc_kv_cold_rows(ks, a->li, a->pos_base, a->S, 0, a->kv_start, sel, a->sel_row)) return 0;
+ VkcKvHostAttn h = {a->S, a->H, a->H, a->K, a->R, a->K, a->scale, q, (size_t)a->qa_row, (size_t)a->qa_seg,
+ q + a->nqa, (size_t)a->qr_row, (size_t)a->qr_seg, a->Lh, 0, (size_t)a->K, a->Rh, 0, (size_t)a->R,
+ a->Lh, 0, (size_t)a->K, NULL, NULL, 0, 0, NULL, 0, 0, 0, 0};
+ vkc_kv_host_attn(ks, &h, ks->hpart, ks->hpart + (size_t)a->S * a->H * a->K);
+ return 1;
+}
+static int vkc_kv_mla(VkcKvMla *a) {
+ VkcKvSplit *ks = a->ks;
+ if (a->S > ks->chunk) { /* a row's share would reach past the window: the engine must clamp its steps */
+ fprintf(stderr, "[VK] %s chain: a step of %d rows past the split's %d\n", ks->engine, a->S, ks->chunk);
+ return 0;
+ }
+ ks->steps++;
+ if (!vkc_kv_cold(ks, a->li, a->pos_base, a->S, 0, a->kv_start, a->sel_row > 0))
+ return vkc_kv_sliced(a->S, a->H, vkc_kv_span(ks, 0, a->sel_row), 2 * a->K + a->R, vkc_kv_mla_fin, a);
+ size_t part = (size_t)a->S * a->H * (a->K + 2);
+ a->nqa = (size_t)(a->S - 1) * a->qa_row + (size_t)(a->H - 1) * a->qa_seg + a->K;
+ size_t nqr = a->R > 0 ? (size_t)(a->S - 1) * a->qr_row + (size_t)(a->H - 1) * a->qr_seg + a->R : 0;
+ if (!vkc_kv_parts(ks, part) ||
+ !vkc_kv_run2(ks, a->qa, (size_t)a->qa_off, a->nqa, a->R > 0 ? a->qr : NULL, (size_t)a->qr_off, nqr,
+ a->sel_row > 0 ? a->sel : NULL, (size_t)a->sel_off, (size_t)a->S * a->sel_row,
+ vkc_kv_mla_dev, vkc_kv_mla_host, a, part)) return 0;
+ VkcKvsMerge mg = {a->S * a->H, a->H, a->K, 0, a->H * a->K, a->K, a->S * a->H * a->K, 0, a->S * a->H * a->K,
+ a->o_off, a->o_row, a->o_seg, 0, 0, 0, 0};
+ if (!vkc_kvs_merge(ks->dpart, ks->cpart, NULL, a->out, &mg)) return 0;
+ if (a->sel_row > 0) vkc_kv_count(ks, a->li, (const int *)vkc_ptr(ks->sd), a->S, a->sel_row);
+ return 1;
+}
+
+/* Inkling's global layers (vkc_kvs_rel's form): q at s*q_row + h*hd of q, r at s*r_row +
+ * h*d_rel of r, tau[s] at tau_off + s of tau (floats; the host's copy in tau_h), relp at
+ * relp_off of relp (the host's [d_rel][ext] in relp_h); the output at s*o_row + h*hd of
+ * out. The host's cache as VkcKvGqa's. */
+typedef struct {
+ VkcKvSplit *ks; int li;
+ VkcBuf *q, *kc, *vc, *r, *tau, *relp, *out;
+ int S, H, KVH, hd, pos_base, ext, d_rel, q_row, r_row, relp_off, tau_off, o_row;
+ float scale;
+ const float *Kh, *Vh; size_t k_grp, k_pos, v_grp, v_pos;
+ const float *tau_h, *relp_h;
+ size_t nq; /* (set by vkc_kv_rel) */
+} VkcKvRel;
+static int vkc_kv_rel_rec(VkcKvRel *a, int s0, int ns, int fin) {
+ VkcKvSplit *ks = a->ks;
+ int bt = (int)((size_t)a->li * ks->tab_stride), nb = ks->t[a->li].nblk;
+ VkcKvsRel p = {ns, a->H, a->KVH, a->hd, a->pos_base + s0, ks->rows, s0 * a->q_row, a->q_row, a->scale, 0, 0, a->ext, a->d_rel,
+ s0 * a->r_row, a->r_row, a->relp_off, a->tau_off + s0, ks->B, bt, nb, ks->anchor, 0, 0, 0, 0};
+ if (fin) { p.o_off = s0 * a->o_row; p.o_row = a->o_row; p.flags = VKC_KVS_FIN; }
+ else { p.o_off = s0 * a->H * a->hd; p.o_row = a->H * a->hd; p.st_off = a->S * a->H * a->hd + s0 * a->H * 2; }
+ return vkc_kvs_rel(a->q, a->kc, a->vc, fin ? a->out : ks->dpart, a->r, a->tau, a->relp, ks->tab, &p);
+}
+static int vkc_kv_rel_part(void *u, int s0, int ns) { return vkc_kv_rel_rec((VkcKvRel *)u, s0, ns, 0); }
+static int vkc_kv_rel_fin(void *u, int s0, int ns) { return vkc_kv_rel_rec((VkcKvRel *)u, s0, ns, 1); }
+static int vkc_kv_rel_dev(void *u) {
+ VkcKvRel *a = (VkcKvRel *)u;
+ return vkc_kv_sliced(a->S, a->H, vkc_kv_span(a->ks, 0, 0), 2 * a->hd + a->d_rel, vkc_kv_rel_part, a);
+}
+static int vkc_kv_rel_host(void *u, const float *q, const int *sel) {
+ VkcKvRel *a = (VkcKvRel *)u; VkcKvSplit *ks = a->ks;
+ (void)sel;
+ if (!vkc_kv_cold_rows(ks, a->li, a->pos_base, a->S, 0, 0, NULL, 0)) return 0;
+ VkcKvHostAttn h = {a->S, a->H, a->H / a->KVH, a->hd, 0, a->hd, a->scale, q, (size_t)a->q_row, (size_t)a->hd,
+ NULL, 0, 0, a->Kh, a->k_grp, a->k_pos, NULL, 0, 0, a->Vh, a->v_grp, a->v_pos,
+ a->tau_h, a->d_rel > 0 ? q + a->nq : NULL, (size_t)a->r_row, (size_t)a->d_rel, a->relp_h, a->d_rel,
+ a->ext, a->pos_base, 0};
+ vkc_kv_host_attn(ks, &h, ks->hpart, ks->hpart + (size_t)a->S * a->H * a->hd);
+ return 1;
+}
+static int vkc_kv_rel(VkcKvRel *a) {
+ VkcKvSplit *ks = a->ks;
+ if (a->S > ks->chunk) { /* a row's share would reach past the window: the engine must clamp its steps */
+ fprintf(stderr, "[VK] %s chain: a step of %d rows past the split's %d\n", ks->engine, a->S, ks->chunk);
+ return 0;
+ }
+ ks->steps++;
+ if (!vkc_kv_cold(ks, a->li, a->pos_base, a->S, 0, 0, 0))
+ return vkc_kv_sliced(a->S, a->H, vkc_kv_span(ks, 0, 0), 2 * a->hd + a->d_rel, vkc_kv_rel_fin, a);
+ size_t part = (size_t)a->S * a->H * (a->hd + 2);
+ a->nq = (size_t)a->S * a->q_row;
+ if (!vkc_kv_parts(ks, part) ||
+ !vkc_kv_run2(ks, a->q, 0, a->nq, a->d_rel > 0 ? a->r : NULL, 0, (size_t)a->S * a->r_row, NULL, 0, 0,
+ vkc_kv_rel_dev, vkc_kv_rel_host, a, part)) return 0;
+ VkcKvsMerge mg = {a->S * a->H, a->H, a->hd, 0, a->H * a->hd, a->hd, a->S * a->H * a->hd, 0, a->S * a->H * a->hd,
+ 0, a->o_row, a->hd, 0, 0, 0, 0};
+ return vkc_kvs_merge(ks->dpart, ks->cpart, NULL, a->out, &mg);
+}
+
+/* DeepSeek's sparse attention (vkc_kvs_ds's form): q at q_off + s*q_row + h*hd, the list
+ * at l_off + s*l_row (cnt entries), the window ring win, the split compressed rows cmp
+ * (ks->rows of hd), the sinks at sink_off of prm; the output at o_off + s*o_row + h*hd of
+ * out. The host's compressed rows of the layer at Ch + c*hd; n_cmp of them exist. v4:
+ * DeepSeek V4's bf16 weights and output. */
+typedef struct {
+ VkcKvSplit *ks; int li;
+ VkcBuf *q, *win, *cmp, *prm, *list, *out;
+ int S, H, hd, cnt, l_off, l_row, nwin, w_off, q_off, q_row, sink_off, v4, o_off, o_row, n_cmp;
+ float scale;
+ const float *Ch;
+} VkcKvDs;
+/* Sparse scratch grows exactly, without vkc_reserve's geometric over-allocation.
+ * Include the existing allocation when checking the driver budget before replacement. */
+static int vkc_kv_sparse_reserve(VkcBuf **b, size_t bytes) {
+ size_t old = vkc_bytes(*b);
+ if (old >= bytes) return 1;
+ double used = 0, budget = 0;
+ size_t free_bytes;
+ if (coli_vk_mem_budget(&used, &budget)) free_bytes = budget > used ? (size_t)((budget - used) * 1e9) : 0;
+ else {
+ size_t total = coli_vk_device_local_bytes(), weights = 0, count = 0;
+ VkcStats st; vkc_stats(&st); coli_vk_mem_info(&weights, &count);
+ free_bytes = total > weights + st.dev_bytes ? total - weights - st.dev_bytes : 0;
+ }
+ if (bytes - old > free_bytes) return 0;
+ vkc_free(*b); *b = vkc_buf(bytes, VKC_DEV);
+ return *b != NULL;
+}
+/* Sparse attention must retain its original list order, independently of residency.
+ * In particular, V4 rounds weights against the maximum of the complete list: partial
+ * softmax results rounded against separate maxima cannot be merged equivalently.
+ * Download the lists, stage only their missing compressed rows and let the original
+ * device arithmetic consume resident and staged rows in that order. Scratch is reused
+ * after each query and capped at 32 MiB; a larger requirement declines safely to the
+ * engine's CPU fallback instead of allocating another whole KV mirror. The current
+ * chunk's new compressed rows stay on the device and are never read from stale RAM. */
+static int vkc_kv_ds_stage(VkcKvDs *a) {
+ VkcKvSplit *ks = a->ks;
+ VkcKvTab *t = &ks->t[a->li];
+ size_t nl = (size_t)(a->S - 1) * a->l_row + a->cnt;
+ if (!vkc_reserve(&ks->sd, (nl ? nl : 1) * sizeof(int), VKC_DOWN) ||
+ !vkc_copy(ks->sd, 0, a->list, (size_t)a->l_off, nl)) return 0;
+ double before = vkc_kv_now();
+ if (!vkc_submit(1) || !vkc_begin()) return 0;
+ ks->wait_ms += vkc_kv_now() - before;
+ const int *list = (const int *)vkc_ptr(ks->sd);
+ size_t limit = (size_t)32 * 1024 * 1024 / ((size_t)a->hd * sizeof(float));
+ if (limit > (size_t)a->cnt) limit = (size_t)a->cnt;
+ int *mapped = (int *)malloc((size_t)(a->cnt ? a->cnt : 1) * sizeof(int));
+ float *cold = (float *)malloc((limit ? limit : 1) * (size_t)a->hd * sizeof(float));
+ int ok = mapped && cold;
+ for (int s = 0; ok && s < a->S; s++) {
+ double start = vkc_kv_now();
+ size_t n = 0;
+ for (int j = 0; j < a->cnt; j++) {
+ int e = list[(size_t)s * a->l_row + j];
+ mapped[j] = e < 0 ? -1 : e; /* all original negative entries mean skipped */
+ if (e < a->nwin) continue;
+ int r = e - a->nwin, b = r / ks->B;
+ if (r >= a->n_cmp || b >= t->nblk) { ok = 0; break; }
+ if (t->bt[b] >= 0) continue;
+ if (n == limit) { ok = 0; break; }
+ memcpy(cold + n * a->hd, a->Ch + (size_t)r * a->hd, (size_t)a->hd * sizeof(float));
+ mapped[j] = -2 - (int)n++;
+ }
+ ks->host_ms += vkc_kv_now() - start;
+ if (!ok) break;
+ if (!vkc_kv_sparse_reserve(&ks->cold_rows, (n ? n : 1) * (size_t)a->hd * sizeof(float)) ||
+ !vkc_kv_sparse_reserve(&ks->cold_list, (size_t)(a->cnt ? a->cnt : 1) * sizeof(int)) ||
+ (n && !vkc_write(ks->cold_rows, 0, cold, n * (size_t)a->hd * sizeof(float))) ||
+ !vkc_write(ks->cold_list, 0, mapped, (size_t)a->cnt * sizeof(int))) { ok = 0; break; }
+ VkcKvsDs p = {1, a->H, a->hd, a->cnt, 0, a->cnt, a->nwin, a->w_off, 0,
+ a->q_off + s * a->q_row, a->q_row, a->sink_off,
+ VKC_KVS_DSFIN | VKC_KVS_DSCOLD | (a->v4 ? VKC_KVS_V4 : 0), a->scale,
+ ks->B, ks->ns, (int)((size_t)a->li * ks->tab_stride), t->nblk,
+ a->o_off + s * a->o_row, a->o_row, 0};
+ ok = vkc_kvs_ds_cold(a->q, a->win, a->cmp, ks->cold_rows, a->out,
+ a->prm, ks->cold_list, ks->tab, &p);
+ ks->staged_rows += n;
+ /* Bound the frame's upload ring as well as the reusable device scratch. */
+ if (ok && s + 1 < a->S) ok = vkc_submit(0) && vkc_begin();
+ }
+ if (ok) { ks->splits++; vkc_kv_count_fixed(ks, a->li, list, a->S, a->l_row, a->cnt, a->nwin); }
+ free(mapped); free(cold);
+ return ok;
+}
+static int vkc_kv_ds(VkcKvDs *a) {
+ VkcKvSplit *ks = a->ks;
+ int bt = (int)((size_t)a->li * ks->tab_stride), nb = ks->t[a->li].nblk;
+ ks->steps++;
+ if (!vkc_kv_cold_upto(ks, a->li, a->n_cmp)) {
+ VkcKvsDs p = {a->S, a->H, a->hd, a->cnt, a->l_off, a->l_row, a->nwin, a->w_off, 0, a->q_off, a->q_row, a->sink_off,
+ VKC_KVS_DSFIN | (a->v4 ? VKC_KVS_V4 : 0), a->scale, ks->B, ks->ns, bt, nb, a->o_off, a->o_row, 0};
+ return vkc_kvs_ds(a->q, a->win, a->cmp, a->out, a->prm, a->list, ks->tab, &p);
+ }
+ return vkc_kv_ds_stage(a);
+}
+
+/* ---- the MLA layer op over the split cache (vk_chain.h's vkc_mla_qkv / vkc_mla_attn) --
+ * vkc_kv_mla_qkv: vkc_mla_qkv with the step's new rows written into `tmp` (a cache of at
+ * least S rows: the latent and rope rows of the step at rows 0..S-1, copied to down for
+ * the host as vkc_mla_qkv copies them), then stored into their slots of the split cache
+ * c (ks->rows rows). vkc_kv_mla_attn: vkc_mla_attn with the core in two parts
+ * (vkc_kv_mla: the device's rows and the host's, Lh / Rh the host's latent and rope rows
+ * of the layer, position-major) and the rest as vkc_mla_attn records it. */
+static int vkc_kv_mla_qkv(VkcKvSplit *ks, int li, const VkcMla *m, VkcMlaScratch *s, VkcBuf *x, size_t x_off, int S,
+ int pos_base, VkcBuf *cs, VkcMlaCache *c, VkcMlaCache *tmp, VkcBuf *down, size_t down_off) {
+ VkcMlaCache t = {tmp->lat, tmp->rope, S};
+ if (tmp->cap < S || !vkc_mla_qkv(m, s, x, x_off, S, 0, cs, &t, down, down_off)) return 0;
+ VkcKvPart pl = {1, m->K, NULL, 0, c->lat, 0}, pr = {1, m->R, NULL, 0, c->rope, 0};
+ return vkc_kv_store(ks, li, &pl, tmp->lat, 0, (size_t)m->K, 0, pos_base, S) &&
+ (m->R == 0 || vkc_kv_store(ks, li, &pr, tmp->rope, 0, (size_t)m->R, 0, pos_base, S));
+}
+static int vkc_kv_mla_attn(VkcKvSplit *ks, int li, const VkcMla *m, VkcMlaScratch *s, int S, int pos_base, int kv_start,
+ VkcMlaCache *c, VkcBuf *sel, size_t sel_off, int sel_row, VkcBuf *gate, size_t gate_off,
+ VkcBuf *out, size_t out_off, const float *Lh, const float *Rh) {
+ int H = m->H, QR = m->Q + m->R, HK = H * m->K, HV = H * m->V;
+ if (S < 1 || S > s->rows) return 0;
+ int ok;
+ if (m->kv_b) { /* qa[s][h][i] = sum_d kv_b[h*(Q+V) + d][i] q[s][h][d], d < Q */
+ VkcHgemv a = {1, S, H, m->Q, m->Q + m->V, 0, 0, H * QR, QR, 0, HK, m->K, 0, 0, 0};
+ ok = vkc_mla_hgemv(m->kv_b, s->q, s->qabs, NULL, &a);
+ } else { /* qa[s][h][i] = k_abs[h*K + i] . q[s][h][0..Q) */
+ VkcHgemv a = {0, S, H, m->K, m->K, 0, 0, H * QR, QR, 0, HK, m->K, 0, 0, 0};
+ ok = vkc_mla_hgemv(m->k_abs, s->q, s->qabs, NULL, &a);
+ }
+ VkcKvMla a = {ks, li, s->qabs, s->q, c->lat, c->rope, sel, s->clat, S, H, m->K, m->R, pos_base, kv_start,
+ 0, HK, m->K, m->Q, H * QR, QR, 0, HK, m->K, (int)sel_off, sel ? sel_row : 0, m->scale, Lh, Rh, 0};
+ ok = ok && vkc_kv_mla(&a);
+ VkcHgemv v = {0, S, H, m->V, m->kv_b ? m->Q + m->V : m->V, m->kv_b ? m->Q : 0, 0, HK, m->K, 0, HV, m->V,
+ (int)gate_off, HV, gate != NULL};
+ ok = ok && vkc_mla_hgemv(m->kv_b ? m->kv_b : m->v_abs, s->clat, s->ctx, gate, &v);
+ if (ok && m->o && out) ok = vkc_matmul(m->o, s->ctx, 0, out, out_off, S);
+ return ok;
+}
+
+static void vkc_kv_report(const VkcKvSplit *ks) {
+ if (!ks->on || !ks->steps) return;
+ fprintf(stderr, "[VK] %s chain: KV split: %llu layer steps, %llu with a host part (%.1f ms waiting for the queries, "
+ "%.1f ms of host work over %llu positions, %llu sparse rows staged), %llu blocks pinned by reads\n",
+ ks->engine, ks->steps, ks->splits, ks->wait_ms, ks->host_ms, ks->host_pos, ks->staged_rows, ks->pins);
+}
+#endif
diff --git a/c/vk_tier.c b/c/vk_tier.c
index d2c1ecae..4624de22 100644
--- a/c/vk_tier.c
+++ b/c/vk_tier.c
@@ -16,6 +16,7 @@
#include <string.h>
#include <pthread.h>
#include <time.h>
+#include "compat.h"
#include <errno.h>
#include "vk_tier.h"
#include "backend_vulkan.h"
@@ -34,6 +35,11 @@ typedef struct {
} VSlot;
typedef struct { int layer, eid; uint8_t *buf; } VQ;
typedef struct { int layer, eid, ok, copy_failed; ColiVkTensor *g, *u, *d; } VDone;
+/* streaming: a staging slot (an expert's three tensors on the device, filled again for
+ * every expert it carries) and a sub-batch in flight */
+enum { SS_FREE = 0, SS_FILLED, SS_BUSY };
+typedef struct { ColiVkTensor *t[3]; ColiVkExpert *ex; int layer, eid, state, pending, pref; } VStage;
+typedef struct { int n, nslot, cap, *slot; float **yout; } VSub;
static struct {
int on;
@@ -76,6 +82,18 @@ static struct {
unsigned long long routed, served, steps, uploads, upload_bytes, evictions, qfull, rated, refused, failed, warm;
double dev_ms, cpu_ms, wait_ms;
unsigned long long rep_routed, rep_served;
+ /* streaming (big prefill steps; see "streaming" below) */
+ int st_ok, st_slots, st_n, st_per, st_half, st_rows_env, gemm_rows, st_par;
+ VStage *st;
+ double bw_up, cpu_row_ms; /* measured: bytes per ms into a slot; CPU ms per (row, expert) */
+ int big, big_fail, big_n, big_taken, said_fwd;
+ unsigned sk; /* sub-batches issued in this step */
+ VSub hf[2]; /* the two in flight */
+ float *sy; size_t csy; /* the step's device rows, S*K*hidden */
+ int *bcnt, *bofs, *blist, *bcls, cblist; /* per expert of the step: rows, first row, class; the rows grouped */
+ uint32_t *pred; uint8_t *pred_ok; /* per layer: the last big step's rows per expert */
+ unsigned long long st_steps, st_experts, st_rows, st_bytes, st_subs, st_kept, st_kept_rows, pf_n, pf_used;
+ double st_up_ms;
} T;
/* "2.31 GiB", "35.0 MiB", "9.5 KiB": the budget lines read on a 24 GB card and a tiny fixture alike */
@@ -132,11 +150,20 @@ static int dev_fmt(VktFmt f, int *fmt, int *gs) {
/* One matrix [O x I] from its RAM form into a device tensor's mapping: rows at
* `stride` bytes (already zeroed past the row), scales as the backend reads them.
* Pure byte work; the values are never requantized. */
+static void convert_par(VktFmt f, int I, int O, const uint8_t *codes, const void *scales,
+ uint8_t *rows, size_t stride, float *sc, int par);
static void convert(VktFmt f, int I, int O, const uint8_t *codes, const void *scales,
uint8_t *rows, size_t stride, float *sc) {
+ convert_par(f, I, O, codes, scales, rows, stride, sc, 0);
+}
+/* par: the rows split over the OpenMP threads (streaming, on the engine thread) */
+static void convert_par(VktFmt f, int I, int O, const uint8_t *codes, const void *scales,
+ uint8_t *rows, size_t stride, float *sc, int par) {
size_t rb = src_row_bytes(f.kind, I);
+ par = par && O >= 64;
switch (f.kind) {
case VKT_SRC_I8_AS_I4_ROW: case VKT_SRC_I8_AS_I4_GS: /* int8 in [-8,7] -> v+8 nibble pairs */
+ #pragma omp parallel for schedule(static) if (par)
for (int o = 0; o < O; o++) {
const int8_t *src = (const int8_t *)codes + (size_t)o * I;
uint8_t *dst = rows + (size_t)o * stride;
@@ -146,6 +173,7 @@ static void convert(VktFmt f, int I, int O, const uint8_t *codes, const void *sc
}
break;
case VKT_SRC_I4S_PAIRS_ROW: case VKT_SRC_I4S_PAIRS_GS: /* two's complement nibble -> v+8: xor 8 */
+ #pragma omp parallel for schedule(static) if (par)
for (int o = 0; o < O; o++) {
const uint8_t *src = codes + (size_t)o * rb;
uint8_t *dst = rows + (size_t)o * stride;
@@ -153,6 +181,7 @@ static void convert(VktFmt f, int I, int O, const uint8_t *codes, const void *sc
}
break;
case VKT_SRC_I4U_PLANAR64: /* planar blocks (byte k = elements k, k+32) -> pairs (2j, 2j+1) */
+ #pragma omp parallel for schedule(static) if (par)
for (int o = 0; o < O; o++) {
const uint8_t *src = codes + (size_t)o * rb;
uint8_t *dst = rows + (size_t)o * stride;
@@ -170,6 +199,7 @@ static void convert(VktFmt f, int I, int O, const uint8_t *codes, const void *sc
}
break;
default: /* the same bytes */
+ #pragma omp parallel for schedule(static) if (par)
for (int o = 0; o < O; o++) memcpy(rows + (size_t)o * stride, codes + (size_t)o * rb, rb);
break;
}
@@ -380,7 +410,7 @@ static int peek_victim(uint64_t hs) {
* blocks of rows, or rows one by one, is the same forward). The tick counts tokens:
* the rows of the forward's first layer, every block of them. */
static void new_forward(void) {
- T.promos = 0; T.promo_cap = 0;
+ T.promos = 0; T.promo_cap = 0; T.said_fwd = 0;
if (T.tick >= T.decay_at) {
size_t n = (size_t)T.c.layers * T.c.experts;
for (size_t i = 0; i < n; i++) T.s[i].heat = tier_decay_value(T.s[i].heat);
@@ -522,6 +552,460 @@ static int grow_rows(int need) {
return 1;
}
+/* ---- streaming: a big prefill step's cold experts on the device -------------------------
+ * A prompt chunk of thousands of rows routes most of a layer's experts, each to dozens of
+ * rows or more. The resident ones run on the device as always; a cold one (not resident)
+ * the CPU would read from its RAM cache or the disk and multiply for every one of its
+ * rows. Streaming hands it to the device instead: the engine loads it as its CPU path
+ * would (VktConfig.load), the tier copies it into one of a few staging slots (st_n of
+ * them, carved out of the budget) and runs its rows there with the tiled GEMM, then the
+ * slot is free for the next one. The resident set and its LFRU never see a streamed
+ * expert: it is not offered for promotion (a prompt's whole working set passes through
+ * the slots and would flush the residents), and its routings count as heat, as every
+ * routing does.
+ *
+ * Such a step runs as sub-batches (backend_vulkan.c's coli_vk_xb_sub_*): the resident
+ * experts first, then the streamed ones, at most st_per streamed experts and st_half
+ * rows a sub-batch; sub-batch k goes to half k % 2 of the scratch, so while the device
+ * computes one the engine thread loads and uploads the experts of the next (double
+ * buffering). An expert with more rows than a sub-batch holds is cut into parts.
+ *
+ * The rule, per cold expert of a step: stream it when it has at least R rows,
+ * R = max(G, ceil(upload / cpu_row)),
+ * G the rows from which the device takes the tiled GEMM (COLI_VK_TIER_GEMM_ROWS, 16:
+ * fewer rows would run the per-row GEMV), upload the time one expert takes to reach a
+ * slot (its bytes at the bandwidth measured over the uploads so far), cpu_row the time
+ * the CPU spent per (row, expert) pair it computed in the prefill steps so far (its share
+ * of a step between issue and join, loads included). Below R the CPU keeps the expert:
+ * its rows cost the CPU less than the upload. Until both are measured R = G.
+ * COLI_VK_TIER_STREAM_ROWS=n fixes R at n (tests). A [VK] line per forward says what the
+ * rule gave, and the run's line how much streamed. */
+static double vkt_now_ms(void);
+static int stream_min_rows(void) {
+ if (T.st_rows_env > 0) return T.st_rows_env;
+ int r = T.gemm_rows;
+ if (T.bw_up > 0 && T.cpu_row_ms > 0) {
+ double up = (double)T.exp_bytes / T.bw_up, need = up / T.cpu_row_ms;
+ if (need > 1e6) need = 1e6;
+ int n = (int)need + (need > (int)need);
+ if (n > r) r = n;
+ }
+ return r;
+}
+int vkt_step_rows(int S, int block) {
+ if (T.on && T.st_ok && S >= T.gemm_rows) return S;
+ return S < block ? S : block;
+}
+
+/* the slots, made at the first step that streams */
+static int st_alloc(void) {
+ if (T.st) return T.st_n >= 2;
+ T.st = calloc((size_t)T.st_slots, sizeof(VStage));
+ if (!T.st) { T.st_ok = 0; return 0; }
+ int H = T.c.hidden, F = T.c.inter, n = 0;
+ for (int i = 0; i < T.st_slots; i++) {
+ VStage *v = &T.st[n];
+ int ok = 1;
+ for (int k = 0; k < 3 && ok; k++) {
+ int fmt = k < 2 ? T.gu_fmt : T.dn_fmt, gs = k < 2 ? T.gu_gs : T.dn_gs;
+ uint8_t *rows; size_t stride; float *sc;
+ ok = coli_vk_tier_tensor(&v->t[k], fmt, k < 2 ? H : F, k < 2 ? F : H, gs, &rows, &stride, &sc);
+ if (!ok) v->t[k] = NULL;
+ }
+ if (ok) v->ex = coli_vk_xb_expert(v->t[0], v->t[1], v->t[2]);
+ if (!ok || !v->ex) {
+ for (int k = 0; k < 3; k++) if (v->t[k]) coli_vk_tensor_free(v->t[k]);
+ memset(v, 0, sizeof *v);
+ break;
+ }
+ n++;
+ }
+ T.st_n = n;
+ if (n < 2) {
+ fprintf(stderr, "[VK] tier %s: no device memory for the streaming slots, cold experts stay on the CPU\n", T.engine);
+ T.st_ok = 0;
+ return 0;
+ }
+ if (n < T.st_slots) fprintf(stderr, "[VK] tier %s: %d of %d streaming slots\n", T.engine, n, T.st_slots);
+ T.st_per = n / 2;
+ return 1;
+}
+/* Experts' bytes into slots: eids[i] into T.st[slots[i]], ok[i] = 1 when it is there
+ * (else the CPU computes it). The engine loads them on this thread, several at once
+ * when it can (load_batch: its parallel reads); the conversions into the slots run in
+ * parallel, each expert into its own slot. */
+static int st_put(int layer, int eid, VStage *v, const VktExpertSrc *src, int par) {
+ const int H = T.c.hidden, F = T.c.inter;
+ if (!src->g || !src->u || !src->d || (T.gu_scales && (!src->gs || !src->us || !src->ds))) return 0;
+ const uint8_t *codes[3] = {src->g, src->u, src->d}; const void *scs[3] = {src->gs, src->us, src->ds};
+ for (int k = 0; k < 3; k++) {
+ uint8_t *rows; size_t stride; float *sc;
+ if (!coli_vk_tensor_refill(v->t[k], &rows, &stride, &sc)) return 0;
+ convert_par(k < 2 ? T.c.gate_up : T.c.down, k < 2 ? H : F, k < 2 ? F : H, codes[k], scs[k], rows, stride, sc, par);
+ }
+ (void)layer; (void)eid;
+ return 1;
+}
+static void st_fill_group(int layer, const int *eids, const int *slots, int n, uint8_t *ok) {
+ VktExpertSrc srcs[64]; void *hs[64];
+ if (n > 64) n = 64;
+ double ms = 0;
+ /* A device upload group can exceed the RAM cache. Consume each cache-sized
+ * batch completely before loading the next, which may reuse those RAM slots.
+ * A partial batch is capacity, not a reason to serialize all remaining reads. */
+ for (int base = 0; base < n;) {
+ int left = n - base;
+ int got = T.c.load_batch && left >= 2
+ ? T.c.load_batch(T.c.load_ctx, layer, eids + base, left, srcs, hs) : 0;
+ if (got < 0 || got > left) got = 0;
+ if (got) {
+ double t0 = vkt_now_ms();
+ #pragma omp parallel for schedule(dynamic, 1) if (T.st_par)
+ for (int j = 0; j < got; j++)
+ ok[base + j] = (uint8_t)st_put(layer, eids[base + j], &T.st[slots[base + j]], &srcs[j], 0);
+ ms += vkt_now_ms() - t0;
+ for (int j = 0; j < got; j++) T.c.release(T.c.load_ctx, hs[j]);
+ base += got;
+ continue;
+ }
+ /* A lone expert, or an engine without a usable batch hook. Its rows can
+ * still convert in parallel (COLI_VK_TIER_STREAM_PAR=0 keeps one thread). */
+ void *h = NULL; VktExpertSrc src;
+ ok[base] = 0;
+ if (T.c.load(T.c.load_ctx, layer, eids[base], &src, &h)) {
+ double t0 = vkt_now_ms();
+ ok[base] = (uint8_t)st_put(layer, eids[base], &T.st[slots[base]], &src, T.st_par);
+ ms += vkt_now_ms() - t0;
+ T.c.release(T.c.load_ctx, h);
+ }
+ base++;
+ }
+ double t0 = vkt_now_ms();
+ /* Fill the uploader's bounded staging window across the complete group.
+ * Committing each expert separately drains that window and waits for a
+ * fence after just three matrices, serializing transfers on discrete GPUs.
+ * A failed group is wholly left to the CPU; no partial expert is published. */
+ ColiVkTensor *tensors[3 * 64]; int nt = 0;
+ for (int i = 0; i < n; i++) if (ok[i])
+ for (int k = 0; k < 3; k++) tensors[nt++] = T.st[slots[i]].t[k];
+ if (nt && !coli_vk_tensor_commit(tensors, nt)) memset(ok, 0, (size_t)n);
+ int done = 0;
+ for (int i = 0; i < n; i++) {
+ VStage *v = &T.st[slots[i]];
+ if (!ok[i]) { v->state = SS_FREE; continue; }
+ v->layer = layer; v->eid = eids[i]; v->state = SS_FILLED; v->pending = 0; v->pref = 0;
+ done++;
+ }
+ ms += vkt_now_ms() - t0;
+ T.st_up_ms += ms; T.st_bytes += (unsigned long long)done * T.exp_bytes;
+ if (done && ms > 0) { double bw = (double)done * T.exp_bytes / ms; T.bw_up = T.bw_up > 0 ? 0.8 * T.bw_up + 0.2 * bw : bw; }
+}
+/* a sub-batch in flight joined: its rows copied to their places, its slots free */
+static int st_join_half(int h) {
+ VSub *b = &T.hf[h];
+ if (!coli_vk_xb_sub_busy(h)) return 1;
+ double dms = 0, t0 = vkt_now_ms();
+ int ok = coli_vk_xb_sub_join(h, b->yout, &dms);
+ T.wait_ms += vkt_now_ms() - t0;
+ T.dev_ms += dms;
+ for (int i = 0; i < b->nslot; i++) {
+ VStage *v = &T.st[b->slot[i]];
+ if (--v->pending <= 0) { v->pending = 0; v->state = SS_FREE; }
+ }
+ b->n = b->nslot = 0;
+ if (!ok) T.big_fail = 1;
+ return ok;
+}
+static int grow_int(int **p, int *cap, int need) {
+ if (need <= *cap) return 1;
+ int nc = *cap ? *cap : 64; while (nc < need) nc *= 2;
+ int *n = realloc(*p, (size_t)nc * sizeof(int));
+ if (!n) return 0;
+ *p = n; *cap = nc;
+ return 1;
+}
+
+/* One big step: every resident expert and every cold one the rule streams, as
+ * sub-batches; the CPU gets the rest (taken stays 0). */
+enum { BC_NONE = 0, BC_DEV, BC_STREAM, BC_CPU };
+static int issue_big(int layer, const float *x, int S, int K, const int *idx, const float *w, uint8_t *taken) {
+ const int E = T.c.experts, H = T.c.hidden, n = S * K;
+ int *cnt = T.bcnt, *ofs = T.bofs, *cls = T.bcls;
+ memset(cnt, 0, (size_t)E * sizeof(int));
+ for (int i = 0; i < n; i++) {
+ int e = idx[i];
+ if (e < 0 || e >= E) continue;
+ T.routed++;
+ VSlot *v = slot(layer, e);
+ if (v->heat < 0xFFFFFFFFu) v->heat++;
+ v->last = T.tick;
+ cnt[e]++;
+ }
+ /* the rows grouped by expert, each expert's in routing order; its class */
+ if (!grow_int(&T.blist, &T.cblist, n) || !grow_map(n)) return 0;
+ int at = 0, R = stream_min_rows();
+ for (int e = 0; e < E; e++) {
+ ofs[e] = at; at += cnt[e];
+ VSlot *v = slot(layer, e);
+ cls[e] = !cnt[e] ? BC_NONE : v->state == VS_RESIDENT && v->ex ? BC_DEV
+ : T.st_ok && cnt[e] >= R ? BC_STREAM : BC_CPU;
+ }
+ int *fill = malloc((size_t)E * sizeof(int));
+ if (!fill) return 0;
+ memcpy(fill, ofs, (size_t)E * sizeof(int));
+ for (int i = 0; i < n; i++) { int e = idx[i]; if (e >= 0 && e < E) T.blist[fill[e]++] = i; }
+ free(fill);
+ if (T.pred) { /* the prediction for the next big step at this layer */
+ for (int e = 0; e < E; e++) T.pred[(size_t)layer * E + e] = (uint32_t)cnt[e];
+ T.pred_ok[layer] = 1;
+ }
+ int ndev = 0, nst = 0, nkept = 0, kept_rows = 0;
+ for (int e = 0; e < E; e++) {
+ ndev += cls[e] == BC_DEV; nst += cls[e] == BC_STREAM;
+ if (cls[e] == BC_CPU) { nkept++; kept_rows += cnt[e]; }
+ }
+ /* prefetched slots this step does not stream go back */
+ for (int i = 0; T.st && i < T.st_n; i++) {
+ VStage *v = &T.st[i];
+ if (v->state == SS_FILLED && (v->layer != layer || cls[v->eid] != BC_STREAM)) v->state = SS_FREE;
+ }
+ if (nst && !st_alloc()) {
+ for (int e = 0; e < E; e++) if (cls[e] == BC_STREAM) { cls[e] = BC_CPU; nkept++; kept_rows += cnt[e]; }
+ nst = 0;
+ }
+ if (!T.said_fwd) {
+ T.said_fwd = 1;
+ char hb[32];
+ fprintf(stderr, "[VK] tier %s stream: a step of %d rows; cold experts with %d rows or more go to the device (",
+ T.engine, S, R);
+ if (T.st_rows_env > 0) fprintf(stderr, "COLI_VK_TIER_STREAM_ROWS");
+ else if (T.bw_up > 0 && T.cpu_row_ms > 0)
+ fprintf(stderr, "an upload of %s at %.1f GB/s takes %.3f ms, the CPU %.4f ms a row; the GEMM from %d rows",
+ human((double)T.exp_bytes, hb, sizeof hb), T.bw_up / 1e6, (double)T.exp_bytes / T.bw_up,
+ T.cpu_row_ms, T.gemm_rows);
+ else fprintf(stderr, "the GEMM's %d rows: the upload and the CPU's rows not measured yet", T.gemm_rows);
+ fprintf(stderr, "); layer %d: %d resident, %d streamed, %d kept on the CPU (%d rows)\n", layer, ndev, nst, nkept, kept_rows);
+ }
+ T.st_kept += (unsigned long long)nkept; T.st_kept_rows += (unsigned long long)kept_rows;
+ if (!ndev && !nst) return 0;
+ /* The backend counts both halves, activation buffers, descriptor windows and
+ * growth before allocating. Leave half of available RAM for CPU work too. */
+ int half = T.st_half < n ? T.st_half : n;
+ size_t ram = (size_t)(compat_mem_available_gb() * 1e9);
+ half = coli_vk_xb_sub_fit(half, E + 1, ram ? ram / 2 : SIZE_MAX);
+ if (half < 1 || !coli_vk_xb_sub_reserve(half, E + 1)) {
+ fprintf(stderr, "[VK] tier %s: no scratch for a step of %d rows, its experts stay on the CPU\n", T.engine, S);
+ return 0;
+ }
+ if ((size_t)n * H > T.csy) {
+ float *ny = realloc(T.sy, (size_t)n * H * sizeof(float));
+ if (!ny) return 0;
+ T.sy = ny; T.csy = (size_t)n * H;
+ }
+ for (int i = 0; i < n; i++) T.map[i] = -1;
+ T.big = 1; T.big_fail = 0; T.big_n = n; T.big_taken = 0; T.sk = 0;
+ T.can_balance = 0;
+ /* the order: resident experts, then the streamed ones already in a slot, then the rest */
+ int *order = malloc((size_t)E * sizeof(int)), no = 0;
+ ColiVkExpert **bex = malloc((size_t)(E + 1) * sizeof(*bex));
+ int *brows = malloc((size_t)(E + 1) * sizeof(int));
+ const float **bx = malloc((size_t)half * sizeof(*bx)); float *bw = malloc((size_t)half * sizeof(float));
+ int *bidx = malloc((size_t)half * sizeof(int));
+ int *eslot = malloc((size_t)E * sizeof(int));
+ int ok = order && bex && brows && bx && bw && bidx && eslot;
+ for (int h = 0; h < 2 && ok; h++) { /* a sub-batch's slots and output places */
+ if (T.hf[h].cap < half) {
+ float **ny = realloc(T.hf[h].yout, (size_t)half * sizeof(float *));
+ if (ny) { T.hf[h].yout = ny; T.hf[h].cap = half; } else ok = 0;
+ }
+ if (ok && !T.hf[h].slot) ok = (T.hf[h].slot = malloc((size_t)(E + 1) * sizeof(int))) != NULL;
+ }
+ if (!ok) { free(order); free(bex); free(brows); free(bx); free(bw); free(bidx); free(eslot); T.big = 0; return 0; }
+ for (int e = 0; e < E; e++) { eslot[e] = -1; if (cls[e] == BC_DEV) order[no++] = e; }
+ for (int i = 0; T.st && i < T.st_n; i++) {
+ VStage *v = &T.st[i];
+ if (v->state == SS_FILLED && v->layer == layer && cls[v->eid] == BC_STREAM && eslot[v->eid] < 0) {
+ eslot[v->eid] = i; order[no++] = v->eid;
+ if (v->pref) T.pf_used++;
+ }
+ }
+ for (int e = 0; e < E; e++) if (cls[e] == BC_STREAM && eslot[e] < 0) order[no++] = e;
+ /* sub-batches */
+ int cnt_b = 0, rows_b = 0, nst_b = 0, total = 0;
+ VSub *cur = NULL;
+ int curh = 0;
+#define SUBMIT_CURRENT() do { \
+ if (cnt_b) { \
+ int okk = !T.big_fail && coli_vk_xb_sub_issue(curh, bex, brows, cnt_b, bx, bw); \
+ if (okk) { \
+ for (int q = 0; q < rows_b; q++) { taken[bidx[q]] = 1; T.map[bidx[q]] = bidx[q]; } \
+ total += rows_b; T.st_subs++; T.sk++; \
+ T.inflight = 1; \
+ } else { \
+ T.big_fail = 1; \
+ for (int q = 0; q < cur->nslot; q++) { VStage *vv = &T.st[cur->slot[q]]; if (--vv->pending <= 0) { vv->pending = 0; vv->state = SS_FREE; } } \
+ cur->nslot = 0; cur->n = 0; \
+ } \
+ } \
+ cnt_b = rows_b = nst_b = 0; cur = NULL; \
+ } while (0)
+ for (int o = 0; o < no && !T.big_fail; o++) {
+ int e = order[o];
+ /* An earlier group fill can have reassigned this expert to the CPU.
+ * Its entry remains in order[], but has no resident device tensor. */
+ if (cls[e] == BC_CPU) continue;
+ int sl = -1;
+ if (cls[e] == BC_STREAM) {
+ if (nst_b >= T.st_per) SUBMIT_CURRENT();
+ if (T.big_fail) break;
+ if (eslot[e] < 0) {
+ /* this expert and the next ones without a slot, as many as the sub-batch has
+ * room for, loaded and uploaded together; free slots first, joining the
+ * oldest sub-batch in flight when there are not enough */
+ int room = T.st_per - nst_b, gid[64], gsl[64], gn = 0;
+ if (room > 64) room = 64;
+ for (int o2 = o; o2 < no && gn < room; o2++)
+ if (cls[order[o2]] == BC_STREAM && eslot[order[o2]] < 0) gid[gn++] = order[o2];
+ int nfree = 0;
+ for (int tries = 0; tries < 3; tries++) {
+ nfree = 0;
+ for (int i = 0; i < T.st_n && nfree < gn; i++) if (T.st[i].state == SS_FREE) gsl[nfree++] = i;
+ if (nfree >= gn) break;
+ int old = coli_vk_xb_sub_busy(T.sk % 2) ? (int)(T.sk % 2) : (int)((T.sk + 1) % 2);
+ if (cur && old == curh && cnt_b) old = !curh; /* never the half being filled */
+ if (!coli_vk_xb_sub_busy(old)) break;
+ st_join_half(old);
+ }
+ if (T.big_fail) break;
+ if (nfree < gn) gn = nfree;
+ uint8_t gok[64];
+ st_fill_group(layer, gid, gsl, gn, gok);
+ for (int i = 0; i < gn; i++) {
+ if (gok[i]) eslot[gid[i]] = gsl[i];
+ else { cls[gid[i]] = BC_CPU; T.st_kept++; T.st_kept_rows += (unsigned long long)cnt[gid[i]]; }
+ }
+ if (cls[e] != BC_STREAM || eslot[e] < 0) { /* not loaded (or no slot): the CPU's */
+ if (cls[e] == BC_STREAM) { cls[e] = BC_CPU; T.st_kept++; T.st_kept_rows += (unsigned long long)cnt[e]; }
+ continue;
+ }
+ }
+ sl = eslot[e];
+ T.st_experts++; T.st_rows += (unsigned long long)cnt[e];
+ }
+ ColiVkExpert *ex = sl >= 0 ? T.st[sl].ex : slot(layer, e)->ex;
+ for (int r0 = 0; r0 < cnt[e] && !T.big_fail; ) {
+ if (rows_b && rows_b + (cnt[e] - r0 < half ? cnt[e] - r0 : half) > half) SUBMIT_CURRENT();
+ if (T.big_fail) break;
+ if (!cur) { /* a new sub-batch in half sk % 2: the one there before is joined first */
+ curh = (int)(T.sk % 2);
+ if (coli_vk_xb_sub_busy(curh) && !st_join_half(curh)) break;
+ cur = &T.hf[curh]; cur->n = cur->nslot = 0;
+ }
+ int take = cnt[e] - r0; if (take > half - rows_b) take = half - rows_b;
+ bex[cnt_b] = ex; brows[cnt_b] = take; cnt_b++;
+ for (int q = 0; q < take; q++) {
+ int i = T.blist[ofs[e] + r0 + q];
+ bx[rows_b + q] = x + (size_t)(i / K) * H;
+ bw[rows_b + q] = w ? w[i] : 1.0f;
+ bidx[rows_b + q] = i;
+ cur->yout[rows_b + q] = T.sy + (size_t)i * H;
+ }
+ rows_b += take; cur->n = rows_b;
+ if (sl >= 0) {
+ if (!T.st[sl].pending || T.st[sl].state != SS_BUSY) { T.st[sl].state = SS_BUSY; }
+ T.st[sl].pending++;
+ cur->slot[cur->nslot++] = sl;
+ if (r0 == 0) nst_b++;
+ }
+ r0 += take;
+ }
+ }
+ SUBMIT_CURRENT();
+#undef SUBMIT_CURRENT
+ free(order); free(bex); free(brows); free(bx); free(bw); free(bidx); free(eslot);
+ T.big_taken = total;
+ if (!total && !T.inflight) { /* nothing went (or everything failed before a submit) */
+ T.big = 0;
+ if (T.big_fail) {
+ fprintf(stderr, "[VK] tier %s: a batch failed, its experts are recomputed on the CPU and the tier stops\n", T.engine);
+ T.on = 0;
+ }
+ return 0;
+ }
+ T.steps++; T.served += (unsigned long long)total; T.st_steps++;
+ T.S = S; T.K = K;
+ T.t_issued = vkt_now_ms();
+ return total;
+}
+static int join_big(const float **rows) {
+ double t0 = vkt_now_ms(), tc = t0 - T.t_issued;
+ T.cpu_ms += tc;
+ /* the oldest first */
+ int a = (int)(T.sk % 2), b = !a;
+ st_join_half(a); st_join_half(b);
+ int cpu_pairs = T.big_n - T.big_taken;
+ if (cpu_pairs >= 32 && tc > 0) {
+ double r = tc / cpu_pairs;
+ T.cpu_row_ms = T.cpu_row_ms > 0 ? 0.8 * T.cpu_row_ms + 0.2 * r : r;
+ }
+ T.inflight = 0; T.big = 0;
+ int ok = !T.big_fail;
+ for (int i = 0; i < T.big_n; i++) rows[i] = ok && T.map[i] >= 0 ? T.sy + (size_t)i * T.c.hidden : NULL;
+ for (int i = 0; T.st && i < T.st_n; i++) if (T.st[i].state == SS_BUSY) { T.st[i].state = SS_FREE; T.st[i].pending = 0; }
+ if (!ok) {
+ fprintf(stderr, "[VK] tier %s: a batch failed, its experts are recomputed on the CPU and the tier stops\n", T.engine);
+ T.on = 0;
+ return 0;
+ }
+ quiesce();
+ return 1;
+}
+
+/* layer's likely streamed experts into the free slots, while the device runs its attention */
+int vkt_stream_prefetch(int layer, int S) {
+ if (!T.on || !T.st_ok || T.inflight || layer < 0 || layer >= T.c.layers || S < T.gemm_rows) return 0;
+ if (!coli_vk_xb_ready() || !st_alloc()) return 0;
+ const int E = T.c.experts, K = T.c.topk;
+ int R = stream_min_rows();
+ /* the slots a step before left filled are free again */
+ for (int i = 0; i < T.st_n; i++) if (T.st[i].state != SS_BUSY) T.st[i].state = SS_FREE;
+ double *est = malloc((size_t)E * sizeof(double));
+ int *cand = malloc((size_t)E * sizeof(int));
+ if (!est || !cand) { free(est); free(cand); return 0; }
+ double tot = 0;
+ if (T.pred && T.pred_ok[layer]) {
+ double sum = 0;
+ for (int e = 0; e < E; e++) sum += T.pred[(size_t)layer * E + e];
+ for (int e = 0; e < E; e++) est[e] = sum > 0 ? (double)T.pred[(size_t)layer * E + e] * S * K / sum : 0;
+ tot = sum;
+ } else {
+ for (int e = 0; e < E; e++) tot += slot(layer, e)->heat;
+ for (int e = 0; e < E; e++) est[e] = tot > 0 ? (double)slot(layer, e)->heat * S * K / tot : 0;
+ }
+ int nc = 0;
+ if (tot > 0)
+ for (int e = 0; e < E; e++) {
+ VSlot *v = slot(layer, e);
+ if (v->state == VS_RESIDENT || est[e] < R) continue;
+ int at = nc++;
+ while (at > 0 && est[cand[at - 1]] < est[e]) { cand[at] = cand[at - 1]; at--; }
+ cand[at] = e;
+ }
+ int got = 0, used = 0;
+ if (nc > T.st_n) nc = T.st_n;
+ while (used < nc) { /* groups of a sub-batch's worth, into the slots in order */
+ int gn = nc - used < T.st_per ? nc - used : T.st_per, gsl[64]; uint8_t gok[64];
+ if (gn > 64) gn = 64;
+ for (int i = 0; i < gn; i++) gsl[i] = used + i;
+ st_fill_group(layer, cand + used, gsl, gn, gok);
+ for (int i = 0; i < gn; i++) if (gok[i]) { T.st[gsl[i]].pref = 1; got++; }
+ used += gn;
+ }
+ T.pf_n += (unsigned long long)got;
+ free(est); free(cand);
+ return got;
+}
+
static int issue(int layer, const float *x, int S, int K, const int *idx, const float *w, uint8_t *taken) {
if (S > 0 && K > 0) memset(taken, 0, (size_t)S * K);
if (!T.on || T.inflight || layer < 0 || layer >= T.c.layers || S < 1 || K < 1) return 0;
@@ -537,6 +1021,7 @@ static int issue(int layer, const float *x, int S, int K, const int *idx, const
}
T.last_layer = layer;
quiesce();
+ if (T.st_ok && S >= T.gemm_rows) return issue_big(layer, x, S, K, idx, w, taken);
int E = T.c.experts, H = T.c.hidden, n = S * K;
if (!grow_map(n)) return 0;
/* heat for every routed expert, groups for the resident ones (first seen first) */
@@ -627,6 +1112,7 @@ int vkt_issue_w(int layer, const float *x, int S, int K, const int *idx, const f
int vkt_join(const float **rows) {
if (!T.inflight) return 0;
+ if (T.big) return join_big(rows);
double t0 = vkt_now_ms(), dms = 0, tc = t0 - T.t_issued;
T.cpu_ms += tc;
int ok = coli_vk_xb_join(T.by, &dms);
@@ -662,18 +1148,7 @@ int vkt_resident(int layer, int eid) {
int vkt_ready(void) { return T.on; }
/* ---- budget ---------------------------------------------------------------------- */
-static size_t mem_available(void) {
-#ifdef __linux__
- FILE *f = fopen("/proc/meminfo", "r");
- if (!f) return 0;
- char line[256]; size_t kb = 0;
- while (fgets(line, sizeof line, f)) if (sscanf(line, "MemAvailable: %zu kB", &kb) == 1) break;
- fclose(f);
- return kb * 1024;
-#else
- return 0;
-#endif
-}
+static size_t mem_available(void) { return (size_t)(compat_mem_available_gb() * 1e9); }
static double env_gb(const char *name, double def) {
const char *e = getenv(name);
return e && *e ? atof(e) : def;
@@ -743,7 +1218,7 @@ int vkt_init(const VktConfig *cfg, uint32_t *const *heat) {
size_t full = (size_t)want / blk, rest = (size_t)want - full * blk;
fit = (long long)full * (long long)(blk / T.exp_bytes) + (long long)(rest / T.exp_bytes);
}
- long long all = (long long)T.c.layers * T.c.experts;
+ long long all = (long long)T.c.layers * T.c.experts, fit_raw = fit;
if (fit > all) fit = all;
if (fit < 2) {
char hb[32], he[32];
@@ -752,13 +1227,48 @@ int vkt_init(const VktConfig *cfg, uint32_t *const *heat) {
return 0;
}
if (T.c.max_experts > 0 && fit > T.c.max_experts) fit = T.c.max_experts; /* the engine's count cap */
+ /* Streaming (big prefill steps, see "streaming" above): COLI_VK_TIER_STREAM=0 turns
+ * it off; it needs the engine's load hook and the GEMM route. Its staging slots come
+ * out of the budget: the residents give up what the budget cannot hold beside them. */
+ {
+ ColiVkXbStats xs; coli_vk_xb_stats(&xs);
+ T.gemm_rows = xs.gemm_rows;
+ const char *se = getenv("COLI_VK_TIER_STREAM"), *sn = getenv("COLI_VK_TIER_STREAM_SLOTS");
+ const char *sr = getenv("COLI_VK_TIER_STREAM_ROWS");
+ int want_st = !(se && *se == '0') && T.c.load && T.c.release && T.gemm_rows > 0;
+ int slots = sn && *sn ? atoi(sn) : 64;
+ if (slots > fit_raw / 2) slots = (int)(fit_raw / 2);
+ if (want_st && slots >= 2) {
+ if (fit > fit_raw - slots) fit = fit_raw - slots;
+ T.st_ok = fit >= 2;
+ if (!T.st_ok) fit = fit_raw > all ? all : fit_raw;
+ }
+ if (T.st_ok) {
+ T.st_slots = slots;
+ T.st_rows_env = sr && *sr ? atoi(sr) : 0;
+ /* a sub-batch's rows: its x rows within 32 MiB (a card without Resizable BAR
+ * keeps them in its host-visible window) */
+ long hr = (32L << 20) / ((long)T.c.hidden * 4);
+ T.st_half = hr < 1 ? 1 : hr > 65535 ? 65535 : (int)hr;
+ const char *sp = getenv("COLI_VK_TIER_STREAM_PAR");
+ T.st_par = !(sp && *sp == '0');
+ const char *sh = getenv("COLI_VK_TIER_STREAM_HALF"); /* tests: small sub-batches */
+ if (sh && *sh && atoi(sh) > 0) T.st_half = atoi(sh) > 65535 ? 65535 : atoi(sh);
+ T.bcnt = malloc((size_t)T.c.experts * sizeof(int)); T.bofs = malloc((size_t)T.c.experts * sizeof(int));
+ T.bcls = malloc((size_t)T.c.experts * sizeof(int));
+ T.pred = calloc((size_t)T.c.layers * T.c.experts, sizeof(uint32_t)); T.pred_ok = calloc((size_t)T.c.layers, 1);
+ if (!T.bcnt || !T.bofs || !T.bcls || !T.pred || !T.pred_ok) T.st_ok = 0;
+ } else if (want_st)
+ fprintf(stderr, "[VK] tier %s: no room for streaming slots beside the residents, cold experts stay on the CPU\n", eng);
+ }
T.max_resident = (int)fit;
T.budget = (size_t)want;
/* Every expert fits in one block's worth: the pool's limit (and so its one block)
* is what they take, not the budget, so a small model does not hold a 256 MB
* block for a few experts. */
- size_t lim = T.budget, need = (size_t)(all + 1) * T.exp_bytes;
- if (fit == all && need < lim && need <= ((size_t)256 << 20)) lim = need;
+ size_t lim = T.budget, need = (size_t)(all + 1 + (T.st_ok ? T.st_slots : 0)) * T.exp_bytes;
+ if (fit + (T.st_ok ? T.st_slots : 0) >= all && fit_raw >= all + (T.st_ok ? T.st_slots : 0) &&
+ need < lim && need <= ((size_t)256 << 20)) lim = need;
coli_vk_tier_pool_limit(lim);
const char *bal = getenv("COLI_VK_TIER_BALANCE");
T.balance = T.c.in_ram != NULL && !(bal && *bal == '0');
@@ -797,10 +1307,12 @@ int vkt_init(const VktConfig *cfg, uint32_t *const *heat) {
if (T.gu_gs) fprintf(stderr, " gs %d", T.gu_gs);
fprintf(stderr, ", down fmt %d", T.dn_fmt);
if (T.dn_gs) fprintf(stderr, " gs %d", T.dn_gs);
- fprintf(stderr, "), %s, %s queue, up to %d promotions per token%s%s\n",
+ fprintf(stderr, "), %s, %s queue, up to %d promotions per token%s%s",
T.uma ? (cap && *cap ? "shared RAM (COLI_VK_TIER_GB)" : "shared RAM: a quarter of what the expert cache leaves")
: "device memory", coli_vk_xb_queue_shared() ? "shared" : "own", T.rate,
T.balance ? ", balanced against the CPU" : "", T.sync ? ", uploads awaited (COLI_VK_TIER_SYNC)" : "");
+ if (T.st_ok) fprintf(stderr, ", streaming cold experts of prompt steps through %d slots", T.st_slots);
+ fprintf(stderr, "\n");
return 1;
}
@@ -821,6 +1333,17 @@ void vkt_report(const char *scope, unsigned long long ram_hits, unsigned long lo
human((double)T.upload_bytes, hl, sizeof hl), T.warm, T.evictions, T.qfull, T.rated, T.failed,
T.dev_ms, T.cpu_ms, T.wait_ms, T.dev_ms > 0 ? 100.0 * hidden / T.dev_ms : 0.0);
if (T.balance) fprintf(stderr, " | balance: device share %.2f, %llu rows handed to the CPU", T.share, T.handed);
+ if (T.st_ok && T.st_steps) {
+ char hs[32];
+ fprintf(stderr, " | stream: %llu steps, %llu cold experts (%s, %.2f GB/s) and %llu rows streamed in %llu sub-batches, "
+ "prefetched %llu (%llu used), %llu cold experts (%llu rows) kept on the CPU",
+ T.st_steps, T.st_experts, human((double)T.st_bytes, hs, sizeof hs),
+ T.st_up_ms > 0 ? (double)T.st_bytes / T.st_up_ms / 1e6 : 0.0, T.st_rows, T.st_subs, T.pf_n, T.pf_used,
+ T.st_kept, T.st_kept_rows);
+ }
+ ColiVkXbStats xs; coli_vk_xb_stats(&xs);
+ if (xs.cooperative_matmuls)
+ fprintf(stderr, " | cooperative matmuls: %llu", xs.cooperative_matmuls);
fprintf(stderr, "\n");
}
@@ -831,6 +1354,7 @@ void vkt_shutdown(void) {
pthread_join(T.th, NULL);
T.th_on = 0;
}
+ if (T.inflight && T.big) { st_join_half(0); st_join_half(1); T.inflight = T.big = 0; }
if (T.inflight) { const float *dummy[1]; (void)dummy; coli_vk_xb_join(T.by, NULL); T.inflight = 0; }
for (int i = 0; i < T.qn; i++) free(T.q[(T.qh + i) % VKT_QCAP].buf);
T.qn = 0;
@@ -838,10 +1362,13 @@ void vkt_shutdown(void) {
quiesce();
size_t n = (size_t)T.c.layers * T.c.experts;
for (size_t i = 0; i < n; i++) if (T.s[i].ex) { coli_vk_xb_expert_free(T.s[i].ex); T.s[i].ex = NULL; }
+ for (int i = 0; T.st && i < T.st_n; i++) if (T.st[i].ex) coli_vk_xb_expert_free(T.st[i].ex);
}
pthread_mutex_destroy(&T.mx); pthread_cond_destroy(&T.cv); pthread_cond_destroy(&T.cv_room); pthread_cond_destroy(&T.cv_done);
free(T.s); free(T.grp); free(T.map); free(T.touched); free(T.bex); free(T.brows); free(T.bfirst);
free(T.bx); free(T.by); free(T.bw); free(T.evict); free(T.done);
+ free(T.st); free(T.sy); free(T.bcnt); free(T.bofs); free(T.bcls); free(T.blist); free(T.pred); free(T.pred_ok);
+ for (int h = 0; h < 2; h++) { free(T.hf[h].slot); free(T.hf[h].yout); }
memset(&T, 0, sizeof T);
}
#endif /* COLI_VULKAN */
diff --git a/c/vk_tier.h b/c/vk_tier.h
index 37715488..4c3f5d06 100644
--- a/c/vk_tier.h
+++ b/c/vk_tier.h
@@ -91,6 +91,20 @@
* 4. Report: vkt_report("run" or "turn", ram_hits, disk_loads) where the engine
* prints its "[VK] <engine>" line; EMAP's tier bits: vkt_resident(l, e) -> 2.
*
+ * 5. Streaming (optional, for prompts; vk_tier.c, "streaming"): give .load and .release
+ * (and .load_batch when the engine reads several experts in parallel), which hand
+ * the tier an expert's bytes as the CPU path would get them. Then
+ * - block a prompt step with B = vkt_step_rows(S, block): the whole step when the
+ * tier streams, so each expert's rows meet in one GEMM;
+ * - vkt_issue takes, besides the resident experts, every cold expert the rule
+ * streams (it loads them through .load, uploads them into its staging slots and
+ * computes them as sub-batches); taken[] says which pairs, as before, and the
+ * join hands back their rows;
+ * - optionally, between submitting the frame that computes a layer's routing and
+ * waiting for it: vkt_stream_prefetch(layer, S) loads the experts that layer will
+ * likely stream while the device works.
+ * COLI_VK_TIER_STREAM=0 turns it off: the tier behaves as without the hooks.
+ *
* Source kinds (VktSrc) and what the device holds -- the conversion runs on the
* uploader thread, nothing changes in the engine's RAM:
* VKT_SRC_I8_ROW int8, f32 per row -> fmt 1
@@ -135,6 +149,14 @@ typedef struct { VktSrc kind; int gs; } VktFmt; /* gs: group (block) size, 0
#define VKT_ACT_SITU 1
#define VKT_ACT_SWIGLU_V4 2
+/* One expert as it sits in RAM: codes and scales of gate, up, down (float scales,
+ * or ue8m0 bytes for VKT_SRC_MXFP4_E8M0; NULL for bf16/f32). Read during the call
+ * only: the tier copies what it keeps. */
+typedef struct {
+ const void *g, *u, *d;
+ const void *gs, *us, *ds;
+} VktExpertSrc;
+
typedef struct {
const char *engine; /* names the [VK] tier lines */
int layers, experts, hidden, inter, topk;
@@ -152,15 +174,21 @@ typedef struct {
/* Optional: at most this many experts resident whatever the budget holds (0 = no
* cap); for an engine whose users already size its device tier in experts. */
int max_experts;
+ /* Optional, for streaming (a big prefill step's cold experts computed on the device,
+ * see "streaming" below): get an expert's bytes as the engine's CPU path would (its
+ * RAM cache, else the disk), valid until release(ctx, *h). 1 = *src filled; 0 = not
+ * to be had (the CPU computes it). Called on the engine thread, from vkt_issue and
+ * vkt_stream_prefetch, one expert at a time. Without it the tier never streams. */
+ int (*load)(void *ctx, int layer, int eid, VktExpertSrc *src, void **h);
+ void (*release)(void *ctx, void *h);
+ void *load_ctx;
+ /* Optional, with load: n experts at once (an engine whose reads run in parallel),
+ * srcs[i] and h[i] as load's, each released on its own; returns how many it loaded,
+ * the first ones in order. A partial batch is consumed and released before the
+ * next batch; 0 falls back to load for the next expert. */
+ int (*load_batch)(void *ctx, int layer, const int *eids, int n, VktExpertSrc *srcs, void **h);
} VktConfig;
-/* One expert as it sits in RAM: codes and scales of gate, up, down (float scales,
- * or ue8m0 bytes for VKT_SRC_MXFP4_E8M0; NULL for bf16/f32). Read during the call
- * only: the tier copies what it keeps. */
-typedef struct {
- const void *g, *u, *d;
- const void *gs, *us, *ds;
-} VktExpertSrc;
#ifdef COLI_VULKAN
/* 1 unless COLI_VK_TIER=0: whether an engine with routed experts will try the tier
@@ -192,6 +220,16 @@ void vkt_begin_forward(void);
void vkt_report(const char *scope, unsigned long long ram_hits, unsigned long long disk_loads);
/* Sizing helpers for engines: bytes one expert takes on the device in a source format. */
size_t vkt_expert_bytes(int hidden, int inter, VktFmt gate_up, VktFmt down);
+/* Streaming: how many rows an engine hands one vkt_issue for a step of S rows whose
+ * blocks are otherwise `block` rows: S when the tier streams steps of that size (the
+ * whole step, so each expert's rows meet in one GEMM), else min(S, block). */
+int vkt_step_rows(int S, int block);
+/* Streaming, optional: layer's experts the next step will likely stream, loaded and
+ * uploaded now, while the device runs the layer's attention (the engine calls this
+ * after submitting that frame and before waiting for it). The prediction is the
+ * layer's routing in the previous big step (else the history); an expert it misses
+ * is streamed after the routing, one it adds costs an upload. Returns how many. */
+int vkt_stream_prefetch(int layer, int S);
#else
static inline int vkt_wanted(void){return 0;}
static inline int vkt_init(const VktConfig *c, uint32_t *const *h){(void)c;(void)h;return 0;}
@@ -209,6 +247,8 @@ static inline int vkt_resident(int l,int e){(void)l;(void)e;return 0;}
static inline void vkt_begin_forward(void){}
static inline void vkt_report(const char *s,unsigned long long r,unsigned long long d){(void)s;(void)r;(void)d;}
static inline size_t vkt_expert_bytes(int h,int i,VktFmt a,VktFmt b){(void)h;(void)i;(void)a;(void)b;return 0;}
+static inline int vkt_step_rows(int S,int b){return S<b?S:b;}
+static inline int vkt_stream_prefetch(int l,int S){(void)l;(void)S;return 0;}
#endif
#endif
diff --git a/docs/ENVIRONMENT.md b/docs/ENVIRONMENT.md
index 94812ed3..29dc4bce 100644
--- a/docs/ENVIRONMENT.md
+++ b/docs/ENVIRONMENT.md
@@ -17,7 +17,7 @@ what follows, but the sister engines read their own:
| `kimi_k3` | `c/kimi_k3.c` | the `K3_*` family, and the Vulkan expert tier's `COLI_VK_TIER*` — see [Kimi K3 engine](#kimi-k3-engine-kimi_k3) |
| `inkling` | `c/inkling.c` | `INK_*`, plus `CTX_MAX`, `PIN_N`, `REP_PEN`, `GPU_DEV`, `NOGPU`, the Vulkan expert tier's `COLI_VK_TIER*` — see [Inkling engine](#inkling-engine-inkling) |
| `qwen36` | `c/qwen36.c` | `QWEN_*`, `Q36_*`, its dense/CUDA-tier controls, the `CACHE_ROUTE` family (VRAM tier over RAM cache), and the Vulkan expert tier's `COLI_VK_TIER*` — see [Qwen3.6 engine](#qwen36-engine-qwen36) |
-| `qwen38` | `c/qwen38.c` | `Q38_MAXT`, `Q38_EOS`, `Q38_NATIVE_FP8`, `Q38_NATIVE_BF16`, `Q38_EXPERT_INT4`, `Q38_MTP`, `Q38_PREFILL_BATCH`, `Q38_TRUNK_CPU_INT8`, `Q38_FP8_KERNEL`, `COLI_TIMERS`, the Vulkan expert tier's `COLI_VK_TIER*` — see [Qwen3.8 engine](#qwen38-engine-qwen38) |
+| `qwen38` | `c/qwen38.c` | `Q38_MAXT`, `Q38_EOS`, `Q38_NATIVE_FP8`, `Q38_NATIVE_BF16`, `Q38_EXPERT_INT4`, `Q38_MTP`, `Q38_MTP_DRAFTS`, `COLI_LOOKUP`, `Q38_PREFILL_BATCH`, `Q38_TRUNK_CPU_INT8`, `Q38_FP8_KERNEL`, `COLI_TIMERS`, the Vulkan expert tier's `COLI_VK_TIER*` — see [Qwen3.8 engine](#qwen38-engine-qwen38) |
| `olmoe` | `c/olmoe.c` | `HOT`, `WIDE`, `SMOOTH`, `CONF_LIMIT`, `MAX_NEW`, `CHAT`, `EXPERT_DROP`, `WARMUP`, the Vulkan expert tier's `COLI_VK_TIER*` — see [OLMoE engine](#olmoe-engine-olmoe) |
| `deepseek_v4` | `c/deepseek_v4.c` | `CTX`, the `V4_*` / `DSV4_*` families, the two `COLI_CUDA_*_BATCH` gates and the Vulkan expert tier's `COLI_VK_TIER*` — see [DeepSeek V4 engine](#deepseek-v4-engine-deepseek_v4); note that the CUDA section below describes `colibri.c` knobs (`COLI_CUDA`, `CUDA_DENSE`, ...) which the V4 engine does not read — its GPU switch is `DSV4_CUDA` |
| `deepseek_v41` | `c/deepseek_v41.c` | the `V41_*` family and the Vulkan expert tier's `COLI_VK_TIER*`: see [DeepSeek V4.1 engine](#deepseek-v41-engine-deepseek_v41) |
@@ -246,13 +246,30 @@ With `COLI_VULKAN=1` every MoE engine (qwen36, qwen38, inkling, olmoe, kimi_k3,
| `COLI_VK_TIER_SYNC` | `0` | `1`: every layer step waits for the uploads staged so far before it runs, so which experts are resident depends on the routing alone, not on when the uploader thread ran (the tests set it: a tiny fixture's run can end before that thread is first scheduled). With `COLI_VK_TIER_BALANCE=0` as well, a run with the tier is reproducible. A promotion that displaces a resident while a batch is in flight waits for the batch's join to free it. Costs the overlap of uploads with compute. |
| `COLI_VK_TIER_BALANCE` | on | `0`: the device takes every resident expert of a step. On, when a join keeps waiting for the device, the step's resident experts the CPU also holds in RAM go back to the CPU beyond the device's share, which moves with every join (down when the CPU waited more than a tenth of its own time, up when the device finished before 80% of it). |
| `COLI_VK_TIER_GEMM_ROWS` | `16` | Rows from which an expert of a step takes the tiled GEMM (prefill) instead of the per-row GEMV; `0` never. Below it a row's bits do not depend on how many rows share the dispatch. |
+| `COLI_VK_TIER_COOP` | on where supported | Cooperative-matrix GEMM for resident and streamed expert batches. The same format and group-size checks as the per-matrix path apply to each gate, up and down projection independently; other projections keep the FP32 GEMM. `0` disables it; `COLI_VK_COOP=0` disables all cooperative pipelines. |
+| `COLI_VK_TIER_COOP_ROWS` | `64` | Minimum rows of one expert for the cooperative path, also subject to `COLI_VK_TIER_GEMM_ROWS`. Small decode and draft-verification batches retain their GEMV arithmetic. |
| `COLI_VK_TIER_QUEUE` | a second queue | `0`: the tier's batches share the main queue with the dense matmuls (they then serialize). Unset: a second queue of the main family, else a compute-only family's (RADV), else shared (Lavapipe). |
-| `COLI_VK_DENSE` | on; off on a device sharing the CPU's RAM while the tier is on | Where the dense (non-expert) matrices of qwen36, qwen38, inkling, olmoe, kimi_k3 (its shared experts), mimo, deepseek_v41, deepseek_v4, glm53 and qwenimage run. `0`: on the CPU (with the tier on, the device takes the routed experts only); `1`: on the device whatever the device. Unset: on the device, except on an integrated GPU or a CPU device (Lavapipe) while the engine runs the expert tier (every one of them but qwenimage, which has no routed experts), where they stay on the CPU: there the dense matmuls cost more than the tier gains (measured on a Radeon 780M). The `[VK] <engine>: device ready, dense matrices on the ...` line says which and why. The GLM engine reads it through the same rule with its own default, off (see above). |
+| `COLI_VK_TIER_STREAM` | on | Streaming: a prompt step's cold (not resident) experts with enough rows are loaded as the CPU path would load them, uploaded into staging slots and computed on the device as GEMMs, the engine handing the tier the whole step (see [vulkan.md](vulkan.md#big-prompt-chunks-and-expert-streaming)). `0`: off, the tier as before. Needs an engine's load hook: every MoE engine has one. |
+| `COLI_VK_TIER_STREAM_SLOTS` | `64` | Staging slots (experts), out of the tier's budget. |
+| `COLI_VK_TIER_STREAM_ROWS` | the rule | Rows from which a cold expert streams. Unset: the larger of `COLI_VK_TIER_GEMM_ROWS` and the measured upload time over the measured CPU time per (row, expert); the `[VK] tier <engine> stream:` line says what it gave. Tests fix it. |
+| `COLI_VK_TIER_STREAM_HALF` | x rows within 32 MiB | Rows of a streaming sub-batch (tests: small ones cut experts into parts). |
+| `COLI_VK_TIER_STREAM_PAR` | on | `0`: a streamed expert's conversion into its slot runs on the engine thread alone. |
+| `COLI_VK_DENSE` | on; off on a device sharing the CPU's RAM while the tier is on | Where the dense (non-expert) matrices of qwen36, qwen38, inkling, olmoe, kimi_k3 (its shared experts), mimo, deepseek_v41, deepseek_v4, glm53, qwenimage, and laya and gliner_decide with `COLI_VK_CHAIN=0`, run. `0`: on the CPU (with the tier on, the device takes the routed experts only); `1`: on the device whatever the device. Unset: on the device, except on an integrated GPU or a CPU device (Lavapipe) while the engine runs the expert tier (every one of them but qwenimage, which has no routed experts), where they stay on the CPU: there the dense matmuls cost more than the tier gains (measured on a Radeon 780M). The `[VK] <engine>: device ready, dense matrices on the ...` line says which and why. The GLM engine reads it through the same rule with its own default, off (see above). |
| `COLI_VK_CHAIN` | on for a discrete GPU; qwen36 and olmoe on, qwen38 off, and mimo, inkling, colibri, glm53, kimi_k3, deepseek_v41 and deepseek_v4 off (not measured) on an integrated GPU with the tier; off on a CPU device | qwen36, qwen38, olmoe, inkling, mimo, colibri (GLM-5.2), glm53 (GLM-5.3 Flash), kimi_k3, deepseek_v41 (DeepSeek V4.1 Flash) and deepseek_v4: every layer's dense chain recorded into one submission per layer, the residual stream and the recurrent and KV state on the device (see [vulkan.md](vulkan.md#the-dense-chain-vk_chainc)). `0` off, `1` on, `2` on for prompts only (decode on the CPU). The `[VK] <engine>: dense chain ...` line says which and why. |
-| `COLI_VK_CHAIN_ROWS` | `512` | Prompt rows per chunk of the chain's prefill. |
+| `COLI_VK_DENSE_HOST` | auto | Whether the dense weights the device holds (the chain on, or `COLI_VK_DENSE` on) keep a copy in host RAM. `0`: the device only. Each resident dense matrix goes up at start and its host copy is given back; the steps the chain does not run take the device too; the CPU reads a matrix back from disk only when it needs one (a lost device), and the engine's expert cache no longer reserves the released host copies. `coli plan` still charges device allocations to physical RAM on integrated GPUs. `1`: the host copy stays as the CPU fallback, as before. Unset: the device only on an integrated GPU (its memory is the same RAM) and on a discrete GPU whose free memory, less 1 GiB, holds the dense weights; the host copy stays on a CPU device (Lavapipe), on a discrete GPU without room, and whenever the dense part runs on the CPU. The `[VK] <engine>: dense weights ...` line says which and why, a second one how much RAM was given back, and one at exit what was read back and the resident set. See [vulkan.md](vulkan.md#dense-weights-on-the-device-only). |
+| `COLI_VK_CHAIN_ROWS` | from the budget, up to 8192 | Prompt rows per chunk of the chain's prefill. Unset or `auto`: the most rows whose scratch fits half the device memory free at the engine's first forward (also bounded by available physical RAM and Windows commit headroom), up to `COLI_VK_CHAIN_ROWS_MAX`; MiMo, GLM-5.3, Kimi K3 and DeepSeek V4 then hand the chain blocks of that chunk instead of `MIMO_CHUNK`, `GLM53_PREFILL_CHUNK`, `K3_CHUNK` and the 128-row block, when those are unset. Set: that many rows, the engines' own blocks as before. See [vulkan.md](vulkan.md#big-prompt-chunks-and-expert-streaming). |
+| `COLI_VK_CHAIN_ROWS_MAX` | `8192` | The most rows the budget's chunk takes. |
+| `COLI_VK_ATTN_BLOCK` | `16` | Rows from which the chain's attention takes the blocked shader (`chain_attnb.comp`: K and V read once per block of rows); `0` never. Below it, and at decode, `chain_attn.comp` as before. |
+| `COLI_VK_ATTN_SLICE` | `4294967296` | Rows x positions x heads x head dim past which the chain's attention ops are recorded in slices of rows, a submission each, so no submission of a big prompt chunk runs past a driver's job timeout; `0` never. |
+| `COLI_VK_CHAIN` (laya, gliner_decide) | on for a GPU, discrete or integrated; off on a CPU device | The decision engines' encoder forward (and Laya's head layers) recorded as one submission on the device, the activations staying there (`decide_vk.h`, [vulkan.md](vulkan.md#the-decision-engines)). `0`: matrix by matrix through the device (`COLI_VK_DENSE` decides, as above), the rest on the CPU; `1`: on. The `[VK] <engine>: forward on the device ...` line says which and why. Clef's is qwen36's chain, as above. |
+| `COLI_VK_IMPORT` | on for a model without routed experts on a device sharing the CPU's RAM; off otherwise | qwen36: the device reads the int8 and f16 dense rows where the host keeps them (`VK_EXT_external_memory_host`, `coli_vk_tensor_import`) instead of holding a copy of each. `1` requests import, `0` copies. On an integrated GPU, imported rows remain shared even with `COLI_VK_DENSE_HOST=0`: they already have one physical copy, and their pages stay alive while Vulkan reads them. `COLI_VK_IMPORT=0` forces an independent device copy whose host pages can be released; only those released pages count as RAM given back. On other devices, `COLI_VK_DENSE_HOST=0` disables import. Unset, on for the dense 27B family (Qwen3.8-27B, Clef) on an integrated GPU or Lavapipe, whose trunk is the whole model and would be held twice (Clef's f16 trunk, 54 GB, twice past a 61 GB box); a copy elsewhere. Where the device cannot import (no extension, staged uploads) the rows are copied. See [vulkan.md](vulkan.md#the-decision-engines). |
| `COLI_VK_CHAIN_GEMV` | on | `0`: the chain's decode matrices take `qmatmul.comp`'s GEMV instead of `chain_gemv.comp`'s. |
| `COLI_VK_CHAIN_SPIN_US` | `2000` | How long a wait on a chain frame polls before blocking. |
| `COLI_VK_CHAIN_PROF` | off | `1`: device time per kind of chain op, one line per run. |
+| `COLI_VK_KV_SPLIT` | on | Past the device's budget a chain engine keeps part of each attention layer's KV cache on the device and attends over the rest on the CPU, the two parts merged (DeepSeek stages only selected missing rows to preserve sparse-attention arithmetic) (see [vulkan.md](vulkan.md#a-kv-cache-past-the-devices-budget)). `0`: never split: the whole mirrors as before, and past the budget the chain declines. |
+| `COLI_VK_KV_DEVICE_ROWS` | from the device's budget | The positions a split layer keeps on the device; set, the cache splits whenever it is longer (the tests' small device, a measurement's split). |
+| `COLI_VK_KV_BLOCK` | `64` | Positions per block of the split cache's block table. |
+| `COLI_VK_KV_PIN` | off | `1`: pin blocks by sparse-selection reads. For QSA/DSA/pooled MLA this can change rounding with read history; the default keeps the partition independent of chunking and prefix reuse. DeepSeek stages missing selected rows and preserves its arithmetic in either mode. |
| `COLI_VK_CHAIN_FAULT` | off | Tests: the n-th chain frame fails as a lost device would; the engine rebuilds the state on the CPU and runs there (colibri, mimo and the DeepSeek engines have none to rebuild: the CPU continues from its host caches). |
| `DUMP` | unset | colibri, glm53, deepseek_v41 and deepseek_v4 in a `VK=1` build: every logits row the forward computes, appended to this file as raw f32 (the Vulkan gates compare the device's with the CPU's). |
@@ -331,6 +348,7 @@ These are for testing, benchmarking, or internal use — not part of the everyda
| `GRAMMAR` | unset | Path to a GBNF grammar file to constrain generation. Takes precedence over `SCHEMA`. |
| `SCHEMA` | unset | Path to a JSON-Schema file compiled to GBNF to constrain generation (consulted only when `GRAMMAR` is empty). |
| `GRAMMAR_DRAFT` | unset | Max grammar-forced draft span length. |
+| `COLI_LOOKUP` | `0` (off) | colibri without an MTP head: `=1` takes the n-gram drafts from the prompt lookup of [speculative.md](speculative.md) (the longest suffix of 4 down to 2 tokens seen before, its most recent occurrence) instead of the last bigram, up to `DRAFT` per verify (5 when `DRAFT` is unset), gated by the measured acceptance and forward times (`COLI_SPEC_GATE=0` drafts every proposal). Unset: the bigram source as before. |
| `COLI_DRAFT_CORPUS` | unset | Path to a file of frozen token ids (whitespace-separated, `-1` separates spans) used as a speculative draft source: the engine proposes the continuation that followed the longest suffix of the live context found in the corpus. Off when unset. Build one from any run with `TOKENS=1`. See [corpus-draft.md](corpus-draft.md). |
| `COLI_CORPUS_K` | `8` (max 48) | Proposal depth for `COLI_DRAFT_CORPUS`. Deeper raises the forward multiplier and the per-forward cost. |
| `COLI_CORPUS_MINACC` | `50` | Acceptance floor (percent) for the corpus source. Below it over a 24-proposal window the source pauses for 256 tokens, then re-arms — rejected drafts cost real time. |
@@ -468,6 +486,10 @@ and the CPU/GPU execution split.
| `QWEN_EXPERT_KERNEL` | `1` (on) | Routed experts run through the shared `expert_ffn.h` kernel: the int4 stays packed in RAM (planar layout, half the expert-cache RSS of the int8 unpack), gate+up are one pass, and a layer is two OpenMP regions over (expert, row-chunk) items instead of 3 x top-k GEMV regions. Takes effect on an int4 gs=64 container whose hidden and expert widths are multiples of 64, and not under the CUDA expert tier. `=0` restores the unpack-to-int8 path; the two produce the same tokens (1024-token decode on the real container byte-identical; pinned on the tiny int4 fixture in CI), only the f32 accumulation order inside a dot differs. Measured at cap 256 on the real container: 12.8 -> 15.7 tok/s, peak RSS 29 -> 17 GB. |
| `QWEN_DENSE_BATCH` | `1` (on) | On AVX2/FMA, reuse each dense-int8 weight decode across two prompt rows. `=0` restores one GEMV call per row. Decode `S=1` is unchanged. |
| `QWEN_SHARED_BATCH` | bounded by 32 MiB scratch | Batch the CPU shared expert across prompt rows. `=0` restores scalar calls; a positive integer caps rows per chunk. The CUDA-tier overlap path is unchanged. |
+| `COLI_LOOKUP` | `0` (off) | `=1`: prompt-lookup drafts. When the recent tokens repeat an n-gram (4 down to 2 tokens) of the prompt or the output, the tokens that followed it are drafted and checked in one verify forward; the output is that of plain decoding, greedy or sampled. A gate drafts only where the measured acceptance and verify cost say it pays. Off under the CUDA tier, `CACHE_ROUTE` and qpack. See [speculative.md](speculative.md). |
+| `COLI_LOOKUP_DRAFTS` | `5` | Lookup drafts per verify at most (1..5); a verify of `k` drafts copies the DeltaNet state after `k` rows (63 MiB each on the 35B). |
+| `COLI_SPEC_GATE` | `1` (on) | `=0`: every proposal drafted in full, the gate's estimates ignored (tests). |
+| `COLI_LOOKUP_FORCE` | unset | Tests only, `ref.json` mode: the reference's tokens as the proposal; `accept`, `mixed`, `cycle`, `row1`..`row5` as for `Q38_MTP_FORCE`. |
| `Q36_MAXT` | conservative engine default (16384 for a Clef checkpoint, its own `max_length`) | Lower the served/context capacity; it cannot raise the model's compiled safety ceiling. |
| `COLI_VULKAN` | `0` | `VK=1` build: the dense trunk on the Vulkan device, and the routed experts on the shared expert tier (`COLI_VK_TIER*`, `COLI_VK_DENSE`, see [Vulkan](#vulkan-any-gpu-with-a-vulkan-12-driver)). With the tier on, the engine keeps the expert history `COLI_USAGE` (default `<snap>/.coli_usage`), saved at every run and serve turn end; it keeps none otherwise. |
| `Q36_DN_GPU` | `0` (off) | CUDA expert tier: a decode token runs every DeltaNet layer whose in_proj and out_proj sit on one card end to end on that card (conv, recurrence, gated norm; state resident in VRAM). Measured on the 35B, 3070, trunk in VRAM: DeltaNet 16.2 -> 10.8 ms/token, the token 39.4 -> 33.3 ms. See [qwen36-cuda-tier.md](qwen36-cuda-tier.md#the-deltanet-layer-on-the-card-q36_dn_gpu1). |
@@ -489,7 +511,9 @@ checkpoint layout and the text-only capability boundary.
| `Q38_MTP` | `0` (off) | `=1` loads the checkpoint's MTP head and decodes speculatively: the head drafts the next token and one forward over two rows verifies it; a rejected draft rolls the DeltaNet/PLE state back to a copy taken after the first row. The output is that of plain decoding, greedy or sampled. A checkpoint without an MTP head is refused. Acceptance and tokens/forward go to stderr at the end of a run and after every served turn. Measured on the release with the int4-g64 sidecar: 1.94 tokens/forward and 12-14% more tokens/s, the expert reads from disk being the same with and without it. See [qwen38.md](qwen38.md#speculative-decoding-with-the-mtp-head). |
| `Q38_MTP_WIRING` | `b` | How the MTP head normalizes the model's four streams before `fc_hidden`, which the tensors leave open: `b` one norm per stream (the reading the trained weights agree with: 94-96% of drafts accepted on the release), `a` one norm over all four streams as one vector (85-92%; a diagnostic). Only the acceptance rate depends on it, never the output. |
| `Q38_MTP_CAP` | the cap | Expert cache of the MTP head's layer. Its experts are the snapshot's FP8 even beside the int4-g64 sidecar; the startup line prints what the cache costs full. |
-| `Q38_MTP_FORCE`, `Q38_MTP_DUMP` | unset | Tests only: `reject` rolls back every draft, `accept` / `mixed` draft the reference's next token (always, every other one) in `ref.json` mode; `Q38_MTP_DUMP=<file>` writes every draft's row, token and logits. |
+| `Q38_MTP_DRAFTS` | `2` | With `Q38_MTP=1`: the MTP head's drafts per verify, `1` to `3` (the deeper ones read the head's own streams), or `0` / `auto` to let the gate pick 0 to 3 per verify from the measured acceptance and verify cost. A verify of `k` drafts copies the recurrent state after `k` rows (113 MiB each). The default 2 measured fastest on a Ryzen 7 PRO 8700GE, on the CPU and in the Vulkan dense chain; `1` is the one-draft verify that came before. See [speculative.md](speculative.md). |
+| `Q38_MTP_FORCE`, `Q38_MTP_DUMP` | unset | Tests only: `reject` rolls back every draft, `accept` / `mixed` draft the reference's next token (always, every other verify) in `ref.json` mode, `cycle` moves the wrong draft over the rows verify by verify, `row1`..`row3` makes that draft wrong; `Q38_MTP_DUMP=<file>` writes every first draft's row, token and logits. |
+| `COLI_LOOKUP`, `COLI_LOOKUP_DRAFTS`, `COLI_SPEC_GATE`, `COLI_LOOKUP_FORCE` | `0`, `5`, `1`, unset | Prompt-lookup drafts, as for qwen36 below; on qwen38 combined with the MTP head (a verify carries the proposal worth more per unit of time). |
| `Q38_PREFILL_BATCH` | `1` (on) | Route prompt rows in bounded expert-major chunks and batch resident shared-expert/DeltaNet projections. `=0` restores row-at-a-time prompt execution for A/B diagnosis; decode is unchanged. |
| `Q38_TRUNK_CPU_INT8` | `1` (on) | The dense trunk (DeltaNet and attention projections, hyper-connection mixers, shared expert, router, lm_head; every matrix of at least `Q38_TRUNK_MIN_KB`) is kept on the CPU as int8 rows with one scale per row and the BF16 copy is released; `q38_weight_matmul` quantizes the activation to int8 and uses the integer kernels of `idot.h` for decode and prefill. `=0` keeps the BF16 rows and the f32 kernel (the numeric reference). See [qwen38.md](qwen38.md#the-trunk-on-the-cpu-int8-rows). |
| `COLI_MAP_EXPERTS` | `0` | Point the native-FP8 routed-expert slots (or the int4-g64 sidecar's records) at a read-only mapping of their shard instead of copying 14 MB per miss into a slab. Same variable as in `glm53`. |
@@ -589,6 +613,8 @@ Read **only** by `c/laya.c`, the decision engine ([laya.md](laya.md)).
| `COLI_LAYA_MAX_LEN` | the checkpoint's `max_len` | Tokens per question sequence (512 on the English checkpoint, capped at the encoder's 8192 positions). |
| `COLI_LAYA_HEAD_MAX_LEN` | the checkpoint's `head_max_len` | Tokens shared by a question's instructions and options; raise it for questions with many options. |
+With a `VK=1` build, `COLI_VULKAN=1` runs the forward on a Vulkan device (`COLI_VK_CHAIN`, `COLI_VK_DENSE` above; [vulkan.md](vulkan.md#the-decision-engines)).
+
## GLiNER2.5-Decide engine (`gliner_decide`)
Read **only** by `c/gliner_decide.c`, the decision engine ([gliner_decide.md](gliner_decide.md)).
@@ -597,6 +623,8 @@ Read **only** by `c/gliner_decide.c`, the decision engine ([gliner_decide.md](gl
|---|---|---|
| `COLI_GLINER_MAX_LEN` | `4096` | Tokens of the one sequence a request becomes: every question with its options, then the state. The state is cut at the last whole word that fits; questions that alone exceed it are refused with a 422. |
+With a `VK=1` build, `COLI_VULKAN=1` runs the forward on a Vulkan device (`COLI_VK_CHAIN`, `COLI_VK_DENSE` above; [vulkan.md](vulkan.md#the-decision-engines)).
+
---
## Server / CLI (`openai_server.py`, `coli`)
diff --git a/docs/docs.json b/docs/docs.json
index ccd5bacb..427b82f3 100644
--- a/docs/docs.json
+++ b/docs/docs.json
@@ -27,7 +27,7 @@
},
{
"group": "Configure and tune",
- "pages": ["SETTINGS", "ENVIRONMENT", "tuning", "CACHE_ROUTE"]
+ "pages": ["SETTINGS", "ENVIRONMENT", "tuning", "speculative", "CACHE_ROUTE"]
},
{
"group": "Hardware backends",
diff --git a/docs/qwen36.md b/docs/qwen36.md
index 42fc06d4..3a31475b 100644
--- a/docs/qwen36.md
+++ b/docs/qwen36.md
@@ -107,6 +107,18 @@ measurements and what they leave out are in
9.94 tok/s against the CPU's 6.01 and the 512-token prompt 9.5 s
([vulkan.md](vulkan.md#the-chain-on-a-radeon-780m)).
+## Prompt-lookup drafts (`COLI_LOOKUP=1`, off by default)
+
+When the recent tokens repeat an n-gram of the prompt or the output (a code edit, a
+quote, repeated structure), `COLI_LOOKUP=1` drafts the up to 5 tokens that followed it
+and checks them in one verify forward. The DeltaNet state is copied after each verify
+row, so a rejected draft rolls back by swapping a copy in. A gate drafts only where the
+measured acceptance and verify cost say it pays. The output is that of plain decoding,
+greedy or sampled, on the CPU and in the Vulkan dense chain. Lookup stays off under the
+CUDA tier, `CACHE_ROUTE` and a qpack container, whose results depend on what is
+resident. On a code edit, Qwen3.6-35B-A3B decoded 7.90 tok/s with lookup against 7.42 without (1.56 tokens per forward); on a chat prompt the gate declined every proposal and the speed was unchanged. How it works, the gate, the settings and the tests:
+[speculative.md](speculative.md).
+
## The expert kernel
Routed experts run through `c/expert_ffn.h`, a header shared with the other
diff --git a/docs/qwen38.md b/docs/qwen38.md
index 1bdbd1d7..e377ef66 100644
--- a/docs/qwen38.md
+++ b/docs/qwen38.md
@@ -342,6 +342,18 @@ matrices, DeltaNet, attention and `lm_head`. Where the routed experts fit in
RAM, those disk reads go away and the dense part is a larger share of each
token, so the gain should be larger. That case has not been measured.
+### Deeper verifies and prompt lookup
+
+`Q38_MTP_DRAFTS` lets the head draft up to 3 tokens per verify: each draft past the
+first reads the head's own streams from the row before, with the draft just proposed.
+The verify copies the DeltaNet and PLE state after each of its rows but the last, so a
+rejection after row `k` restores the state after row `k`. `Q38_MTP_DRAFTS=0` lets a gate
+pick the depth per verify from the measured acceptance by position and the measured
+cost of a verify by its rows. `COLI_LOOKUP=1` adds prompt-lookup drafts (up to 5, from
+the context's n-grams), and a verify carries whichever proposal is worth more. The
+output stays that of plain decoding. Measured on the release with the int4-g64 sidecar: 4.94 tok/s at two drafts against 4.71 at one and 4.12 without MTP on the CPU (2.75 tokens per forward), 4.55 against 4.39 and 3.91 with the Vulkan tier and chain on a Radeon 780M, so two drafts is the default. The details, the settings and the
+tests are in [speculative.md](speculative.md).
+
`make -C c qwen38-tiny-mtp-check` adds a head with random weights under the
release's names to the three tiny fixtures and gates on the head's draft
logits against a float32 reference of both wirings
diff --git a/docs/speculative.md b/docs/speculative.md
new file mode 100644
index 00000000..cebcefbb
--- /dev/null
+++ b/docs/speculative.md
@@ -0,0 +1,294 @@
+# Speculative decoding: MTP drafts and prompt lookup
+
+Speculative decoding guesses the next few tokens cheaply, then checks them all in one
+forward of the model. Tokens are selected from the main model's verified logits, never
+accepted solely on the draft's prediction. With the same arithmetic and execution
+placement, the output matches plain decoding; the number of forwards changes. Dynamic
+CPU/GPU placement can change rounding and therefore occasionally a greedy token, as
+described below.
+
+What runs where:
+
+| Engine | Draft sources | Verify |
+|---|---|---|
+| `qwen38` (Qwen3.8 Flash Next) | the checkpoint's MTP head, up to 3 drafts (`Q38_MTP=1`, `Q38_MTP_DRAFTS`); prompt lookup, up to 5 (`COLI_LOOKUP=1`) | up to 6 rows, on the CPU or in the Vulkan dense chain |
+| `qwen36` (Qwen3.6, Qwen3-Coder, Qwen3.8-27B) | prompt lookup, up to 5 (`COLI_LOOKUP=1`) | up to 6 rows, on the CPU or in the Vulkan dense chain |
+| `colibri` (GLM-5.2) | its MTP head and n-gram drafts as before; with `COLI_LOOKUP=1` and no head, the n-gram drafts come from the prompt lookup below and its gate | as before |
+
+## How a verify works
+
+The caller picks a token and asks for the logits after it. A draft source proposes `k`
+tokens after that one, and one forward over the token and its `k` drafts (the verify,
+`k + 1` rows) returns the first row's logits, the ones a plain step would give, and
+keeps the others. The caller's next picks settle the drafts one at a time:
+
+- a pick that equals the next draft is answered from that draft's row, with no forward;
+- the first pick that differs ends the verify: the rows after the ones that stood are
+ undone, and the next verify starts from the token the caller picked.
+
+The picks come from the exact logits, greedy or sampled. A draft never reaches the
+sampler: a sampled token that happens to equal a draft is an accepted draft.
+
+**Undoing rows on a model with recurrent state.** Qwen3.6 and Qwen3.8 are hybrids: three
+of every four layers are Gated DeltaNet, which keeps a recurrent state rather than rows
+indexed by position, and Qwen3.8 adds the PLE convolution's ring. Such a state cannot be
+rewound. So a verify copies the DeltaNet states, the convolution rings and the PLE ring
+(with its n-gram history) after each of its rows but the last, one copy slot per row. A
+rejection after row `r` swaps slot `r` in (pointers, nothing recomputed). The attention
+and indexer rows past the standing rows stay where they are as a stale tail that the
+next forward overwrites, as every rewind in these engines treats them. `kv_len` and the
+`kv_prefix` record come down to the standing rows. In the Vulkan chain, the KV mirror's
+watermark and the pooled-key watermark come down too.
+
+The copies cost memory, allocated the first time a verify that deep runs:
+
+| Model | One copy slot | Slots for 3 MTP drafts | Slots for 5 lookup drafts |
+|---|---|---|---|
+| Qwen3.8 Flash Next (36 DeltaNet layers, 48 value heads of 128 x 128) | 113 MiB | 3 (338 MiB) | 5 (563 MiB) |
+| Qwen3.6-35B-A3B (30 DeltaNet layers, 32 value heads of 128 x 128) | 63 MiB | | 5 (314 MiB) |
+
+The chain keeps its own copies on the device, the same size.
+
+**Every verify row gets a decode step's bits.** The CPU kernels compute each row the
+same way whatever the row count. A device matrix runs a verify's rows one at a time
+(`q38_weight_matmul`, qwen36's `vk_dense_matmul`), and the chain takes the per-row GEMV
+for every matrix of a verify. The Vulkan expert tier sends at most 15 rows of an expert
+through its per-row route, so the 6 rows of a verify always take it. `lm_head` reads
+each row on its own. In the chain, one copy is written by the recurrent shaders as they
+pass its row. More copies split the convolution and the recurrence into one dispatch
+per copy, each ending on its row. The state crosses between them through memory in f32,
+so the rows get the bits of the single dispatch.
+
+Drafting stays off where a row's result depends on what is resident: qwen36 under the
+CUDA expert tier, `CACHE_ROUTE` or a qpack container (`[qwen36] COLI_LOOKUP=1: no
+drafts under ...`). One case stays inside the gate but changes the last bits: with the
+Vulkan expert tier on, a verify's rows, the rejected drafts' included, feed the tier's
+history, so which experts are on the device follows what was drafted. An expert on the
+device sums in a different order than on the CPU, so the logits of a run with drafts and
+one without can differ in their last bits, as two runs with different histories do.
+When two logits are close, this can also change a greedy token. The tiny tests compare
+tokens there, and drafts against no drafts byte for byte with the tier off; they do not
+establish universal token identity with dynamic tier placement on real checkpoints.
+A real FP8 Qwen3.8 check produced different text with the adaptive tier, but identical
+text with zero, two and three MTP drafts when the starting history, explicit tier budget
+and placement were fixed (`COLI_VK_TIER_GB=4`, `COLI_VK_TIER_RATE=0`,
+`COLI_VK_TIER_BALANCE=0`, `COLI_VK_TIER_STREAM=0`). This is a diagnostic comparison,
+not a recommendation to freeze the tier for normal use.
+
+## The MTP head's drafts (qwen38)
+
+`Q38_MTP=1` loads the head (see [qwen38.md](qwen38.md#speculative-decoding-with-the-mtp-head)).
+Its first draft reads the model's four streams at the last fed position with the picked
+token. Each next draft reads the head's own streams from the row before, in place of the
+model's (only the verify computes those), with the draft just proposed. Those deeper
+rows write the head's K/V and indexer rows past its settled ones, a tail that the
+settled pairs overwrite once the verify has run. After the verify the head takes the
+standing rows' streams, each with the token that followed it.
+
+`Q38_MTP_DRAFTS` sets the drafts per verify: `1`, `2` or `3` drafts every time, or `0` /
+`auto` to let the gate pick 0 to 3 per verify. The default is 2, the depth measured
+fastest (below). `Q38_MTP_DRAFTS=1` is the one-draft verify that came before.
+
+## Prompt lookup
+
+`COLI_LOOKUP=1` looks for the longest n-gram of 4 down to 2 tokens that ends the token
+history (prompt and output so far) and also occurs earlier in it, at its most recent
+earlier occurrence. The up to 5 tokens that followed it there are the proposal
+(`COLI_LOOKUP_DRAFTS` caps it, 1 to 5). Code edits, quotes and repeated structure give
+long accepted runs; free text gives few proposals, and those few are short. On qwen38
+with the MTP head, a verify carries whichever proposal has the higher expected tokens
+per unit of time, the longer one on a tie.
+
+## The gate
+
+`spec_draft.h` decides how many drafts a verify carries. Per source it keeps the running
+probability `a_j` that draft `j` is accepted when the drafts before it were (an average
+over about 16 verifies, from priors of 0.8, 0.6 and 0.5 for the MTP head and 0.5 for
+lookup), the measured seconds `T(n)` of a forward of `n` rows, and the measured seconds
+`d` a source spends per draft past its first (the MTP head's deeper drafts). The depth
+it picks maximizes
+
+```
+value(k) = E(k) / cost(k)
+E(k) = 1 + a_1 + a_1 a_2 + ... + a_1 ... a_k tokens a verify of k drafts yields
+cost(k) = (T(k+1) + (k-1) d) / T(1) in plain decode steps
+```
+
+and it drafts only when that beats a plain step (`value(k) > 1`). A row count not
+measured yet costs `T(1) (1 + (n-1) rho)`, with `rho` fitted from the multi-row forwards
+that ran (0.5 until one has). Every 16 picks it tries one draft deeper than its best,
+so an estimate it stopped feeding can recover. After 64 forwards without a plain step it
+takes one, so `T(1)` stays measured. Prompt lookup is always gated. The MTP head is
+gated under `Q38_MTP_DRAFTS=0`. `COLI_SPEC_GATE=0` drafts every proposal in full
+(the tests use it).
+
+When the gate decided anything, a run prints what it measured on a `[qwen38 spec gate]`
+or `[qwen36 spec gate]` line: `T(1)`, the cost of each row count it saw in units of
+`T(1)`, `rho`, the MTP head's cost per extra draft and the acceptance by position.
+
+## Report lines
+
+Every run, and every served turn, prints its counts on stderr (never on the wire):
+
+```
+[qwen38 MTP] run: 2.75 tokens/forward (36 forwards per 99 tokens) | acceptance 90.0% (63/70 drafts) | wiring b | fixed depth, by position: 1 97.1% (34/35) 2 82.9% (29/35) 3 0.0% (0/0)
+[qwen36 lookup] run: 1.56 tokens/forward (102 forwards per 159 tokens) | acceptance 85.1% (57/67 drafts in 64 verifies) | gate on, 32 declined, 5 probes
+```
+
+Tokens are the decode tokens fed after the prompt, and forwards are the forwards that
+fed them. The acceptance by position counts a draft when its verify carried it. `ended
+mid-verify` marks a turn that stopped with drafts unconsumed: they were undone, and the
+state is the one plain decoding leaves.
+
+## Settings
+
+| Variable | Default | Effect |
+|---|---|---|
+| `Q38_MTP_DRAFTS` | `2` | qwen38 with `Q38_MTP=1`: the MTP head's drafts per verify, `1` to `3`, or `0` / `auto` for the gate's pick. |
+| `COLI_LOOKUP` | `0` | `1`: prompt-lookup drafts (qwen38, qwen36), and colibri's n-gram drafts from the same lookup and gate when it has no MTP head. |
+| `COLI_LOOKUP_DRAFTS` | `5` | Lookup drafts per verify at most, `1` to `5`. |
+| `COLI_SPEC_GATE` | `1` | `0`: every proposal drafted in full, the gate's estimates ignored (tests). |
+| `Q38_MTP_FORCE`, `COLI_LOOKUP_FORCE` | unset | Tests: in `ref.json` mode, `accept` drafts the reference's tokens, `mixed` makes every other verify's first draft wrong, `cycle` moves the wrong draft over the rows verify by verify, `rowN` makes draft `N` wrong; `Q38_MTP_FORCE=reject` also refuses every draft. |
+
+## Measured
+
+On `ds`: Ryzen 7 PRO 8700GE (8 cores, 16 threads), 61 GiB, NVMe, Radeon 780M (RADV),
+`OMP_NUM_THREADS=8`. Every run started with the 1-minute load under 2 and no other
+engine running. Qwen3.8 runs started after the model files were dropped from the page
+cache. Qwen3.6's container (22 GB) stayed in the page cache, and its numbers are two
+separate rounds. The same binary ran every arm of a table. With drafts or without,
+every run printed the same text (the same md5 within each prompt). These are single
+runs. Two runs repeated later in the same conditions (Qwen3.8, CPU: no MTP 4.12 tok/s,
+one draft 4.69) put the run-to-run spread at about 0.5%.
+
+**Qwen3.8 Flash Next, int4-g64 sidecar, chat prompt with thinking, 100 tokens.**
+The CPU runs use cap 170. The tier-and-chain runs (`COLI_VULKAN=1 COLI_VK_CHAIN=1`) use
+cap 96, each from a copy of the same expert history (one unrelated 40-token run).
+
+| MTP | CPU tok/s | tier + chain tok/s | tokens/forward | acceptance by draft position |
+|---|---|---|---|---|
+| off | 4.12 | 3.91 | 1.00 | |
+| 1 draft | 4.71 (+14%) | 4.39 (+12%) | 1.94 | 94.1% |
+| **2 drafts** | **4.94 (+20%)** | **4.55 (+16%)** | 2.75 | 97.1%, 82.9% |
+| 3 drafts | 4.83 (+17%) | 4.42 (+13%) | 3.30 | 93.1%, 79.3%, 65.5% |
+| the gate's depth | 4.76 (+16%) | 4.41 (+13%) | 2.02 | 95.9%, 75.0% (4 verifies of 2) |
+
+The head's acceptance falls with depth: each draft past the first reads the head's own
+streams instead of the model's. Tokens per forward still rise to 3.30 at three drafts,
+but the speed peaks at two. A verify of `k` drafts reads the trunk once, but every row
+routes its own 10 experts per layer (the cache hit rate falls from 81% without MTP to
+76% at two drafts), and a rejected row's expert reads are spent all the same. At three
+drafts the third row stands 65% of the time and costs more than it returns. The
+default is therefore 2 (`Q38_MTP_DRAFTS=2`) on the CPU and in the chain alike.
+
+**Prompt lookup on a code edit.** The prompt asks to rename a variable in 23 lines of C
+and print the code again (`enable_thinking: false`). 160 tokens; Qwen3.8 at cap 170 on
+the CPU, Qwen3.6-35B-A3B (int4 gs64 container) at cap 256 on the CPU.
+
+| | Qwen3.8 tok/s | tokens/forward | Qwen3.6 tok/s (two rounds) | tokens/forward |
+|---|---|---|---|---|
+| no drafts | 2.91 | 1.00 | 7.40, 7.44 | 1.00 |
+| lookup (`COLI_LOOKUP=1`) | 3.11 (+7%) | 1.56, 83.8% of 68 drafts | 7.89, 7.90 (+6%) | 1.56, 85.1% of 67 drafts |
+| MTP, 1 draft | 3.20 (+10%) | 1.99, 100% | | |
+| MTP, 1 draft + lookup | 3.19 | 1.99 (lookup never chosen) | | |
+| MTP at the gate's depth + lookup | 3.30 (+13%) | 2.74, 100% | | |
+
+**Prompt lookup on a chat prompt** (the thinking chat above for Qwen3.8; a chat prompt
+without thinking for Qwen3.6). Free text gave a few short proposals, and the gate
+declined every one (11 on Qwen3.8, 12 on Qwen3.6), so no verify ran. Speed was the
+plain run's within the spread: Qwen3.8 4.16 against 4.12 tok/s, Qwen3.6 10.38 and 10.43
+against 10.37 and 10.47.
+
+So lookup pays where the output repeats the context (+6 to 7% on the edit) and costs
+nothing where it does not. With the MTP head attached, the head's drafts were worth more
+than lookup's on this edit (100% acceptance). `COLI_LOOKUP` stays off by default anyway:
+without `COLI_VULKAN` every engine's default must stay what it was, and qwen36's serve
+`PROF` frame counts the forwards a turn took, which drafts change. Turn it on for code
+editing and other copy-heavy work.
+
+**Vulkan follow-up, existing int4-g64 sidecar.** On the same 780M, with the code-edit
+prompt above (241 prompt tokens), 128 generated tokens, cap 170, eight threads and
+the same starting expert history, all eight runs below produced byte-identical text.
+Each configuration ran once, after advisory eviction of checkpoint file pages, with
+no other inference process running. These are the engine's `Speed` values: generated
+tokens divided by **prefill plus generation time**, excluding model load. They are
+not steady decode throughput.
+
+| Draft source | Expert tier, CPU dense work | Expert tier and Vulkan dense chain |
+|---|---:|---:|
+| off | 2.90 tok/s | 2.68 tok/s |
+| lookup, up to 5 drafts | 3.24 tok/s | 2.85 tok/s |
+| MTP, 3 drafts | 3.49 tok/s | 2.97 tok/s |
+| MTP 3 and lookup | 3.47 tok/s | 3.00 tok/s |
+
+MTP accepted all 95 judged drafts in each of its runs: 32 forwards for 127 decode
+tokens, or 3.97 tokens per forward. Lookup alone accepted 58/66 drafts on the tier
+and 77/102 in the chain; the adaptive gate chose different depths. With both sources
+enabled, every verify used MTP and lookup supplied no drafts. Their gains therefore
+do not add. The small differences between MTP alone and the combined mode are not
+evidence of a lookup benefit.
+
+The tier-only run took 44.2 seconds without drafts and 36.7 seconds with MTP 3;
+time to first token was 16.99 and 17.44 seconds respectively. Subtracting those rounded
+times gives approximately 4.67 and 6.59 tokens/s after the first token. The dense
+chain took 47.7 and 43.0 seconds, with time to first token of 13.35 and 14.08 seconds.
+It released 4.07 GiB of dense host copies without MTP, 4.16 GiB with MTP, and reloaded
+none for CPU fallback. Lower memory use and faster prefill need not imply faster
+decode on a shared-memory GPU.
+
+The MTP layer still follows the host's control and attention path. In the device-only
+dense configuration its dense matrices dispatch on the GPU, while its routed experts
+remain in their native FP8 format on the CPU; the model's existing int4-g64 sidecar
+does not include that layer. This is not a fully device-resident drafter.
+
+Not measured: a long context with speculation, a model whose routed experts all fit
+in RAM, and any discrete GPU (none was available).
+
+## Another engine with a verify path
+
+`spec_draft.h` holds no engine state. An engine that verifies drafts calls:
+
+- `spec_lookup(history, len, 2, 4, k, out)` for a proposal from the context;
+- `spec_gate_pick(&gate, source, proposed, &value)` for how many of them to carry;
+- `spec_gate_forward(&gate, rows, seconds)` after every forward (plain steps too);
+- `spec_gate_result(&gate, source, judged, accepted)` when a verify is settled, where
+ `judged` is the accepted count plus one after a rejection;
+- `spec_gate_draft_cost(&gate, source, seconds)` per extra draft, for a source whose
+ deeper drafts cost time.
+
+colibri's n-gram source does exactly this under `COLI_LOOKUP=1` (`spec_decode` in
+`c/colibri.c`). GLM-5.2 is attention only, so its verify needs no copies: a rejection
+rewinds `kv` and the next forward overwrites the rows. DeepSeek V4.1's DSpark drafter
+(`c/deepseek_v41.c`) has a verify path of its own, with the window rings' undo rows; its
+depth could come from `spec_gate_pick` with `SPEC_SRC_MTP` the same way, timing its
+verifies with `spec_gate_forward`. That is not wired: DSpark's three stages run at a
+fixed depth today, and choosing it per verify needs measurements on V4.1 that this work
+did not make.
+
+## Tests
+
+- `make -C c spec-drafts-check`: `tests/test_spec_draft.c` (the lookup and the gate on
+ their own), `tests/test_qwen38_spec_alloc.c` (every allocation failure while adding
+ deeper rollback slots releases the unfinished slot, preserves completed state and
+ permits a retry), then `tests/spec_drafts_harness.py` on qwen38's BF16, FP8 and int4-g64
+ fixtures with an MTP head and on qwen36 fixtures with an ASCII vocabulary (so a serve
+ session can resend a reply as text). Each run with drafts against the same run
+ without, tokens and last logits byte for byte: MTP depths 1, 2 and 3 with every draft
+ rejected, every one right, alternating, the rejection cycling over the rows and at
+ each row, with the forwards, tokens and accepted drafts the loop must count; the
+ gate's depth; lookup forced the same ways at 5 and 2 drafts, and natural lookup on a
+ repeating prompt; MTP and lookup together; caps 1 and 8, prefill batching off, the
+ int8 trunk, int4-g64 experts.
+- Serve sessions in the harness: a repeating prompt, its continuation by the reply
+ (prefix reuse of the state a speculating turn left, where turns end with a verify's
+ drafts unconsumed), a pinned prompt and a turn restoring the pin, a sampled turn and a
+ logprobs turn, every frame equal to the session without drafts.
+- `bash c/tests/vulkan_engines.sh qwen-spec` (CI leg `qwen-spec`, Lavapipe): the same
+ harness on the CPU and in the dense chain (tier off, byte for byte), the chain in
+ chunks of 3 rows, the per-matrix path and prompts only (`COLI_VK_CHAIN=2`); deep
+ verifies against the CPU with the tier on; the device lost while verifies of up to 4
+ rows run; colibri's lookup against its oracle on the CPU and in the chain.
+- `qwen-spec-sanitize`: the harness's subset under ASan and UBSan, on the CPU and in the
+ chain.
+- The existing gates of the S = 2 verify (`make -C c qwen38-tiny-mtp-check`, the
+ `qwen-chain` family) run unchanged.
diff --git a/docs/vulkan.md b/docs/vulkan.md
index 72a8c67b..71cb6e1e 100644
--- a/docs/vulkan.md
+++ b/docs/vulkan.md
@@ -113,7 +113,7 @@ the shared expert tier ([below](#the-routed-expert-tier-vk_tierc)).
| Engine | On the device | Weight formats | Stays on the CPU |
|---|---|---|---|
-| qwen36 (Qwen3.6, Qwen3-Coder, Qwen3.8-27B) | the dense trunk; routed experts on the expert tier | int8 rows; int4-g64 with `COLI_DENSE_BITS=4`; f32 with `COLI_DENSE_I8=0`; experts int4-g64, int4 per row, int8 per row or gs64 | DeltaNet `dn_a`/`dn_b`, vision tower, the experts the tier does not hold |
+| qwen36 (Qwen3.6, Qwen3-Coder, Qwen3.8-27B, Clef) | the dense trunk; routed experts on the expert tier | int8 rows; int4-g64 with `COLI_DENSE_BITS=4`; f16 (fmt 14) with `COLI_DENSE_BITS=16`; f32 with `COLI_DENSE_I8=0`; experts int4-g64, int4 per row, int8 per row or gs64 | DeltaNet `dn_a`/`dn_b`, vision tower, Clef's joint head, the experts the tier does not hold |
| qwen38 (Qwen3.8 Flash Next) | the trunk; routed experts on the expert tier | int8 trunk rows, bf16, f32 (`Q38_NATIVE_BF16=0`); experts int4-g64 (sidecar), FP8 128x128 blocks, bf16 | the MTP head's experts, the experts the tier does not hold |
| inkling | dense and shared-expert matrices; routed experts on the expert tier | int8 and int4-g64 (dense-int4g64 container), f32, bf16; experts int4 or int8 per row (container or runtime quantization), f32 | embedding and audio lookups, CUDA residents (with CUDA or Metal on, the experts too); bf16 on CPUs with the AVX512-BF16 dot (see below); the experts the tier does not hold |
| olmoe | attention q/k/v/o, router, lm_head; routed experts on the expert tier | f32; experts int8 per row | embedding, the experts the tier does not hold |
@@ -123,6 +123,8 @@ the shared expert tier ([below](#the-routed-expert-tier-vk_tierc)).
| deepseek_v4 | resident dense layers, head, router, compressors; routed experts on the expert tier | fp8 in 128x128 blocks, bf16; experts MXFP4 (fp4, a ue8m0 scale per 32) with [an activation of its own](#deepseek-v4s-activation) | the indexer's `weights_proj`, DSpark stages, the `--oracle` path's dense layers, the experts the tier does not hold |
| glm53 (GLM-5.3 Flash) | the resident matrices (an f32 checkpoint's experts among them); the streaming container's routed experts on the expert tier | int8 and int4-g64 (`GLM53_BITS`); experts int4-gs64 | f32 matrices (`GLM53_BITS=32`), the streamed experts the tier does not hold, all of them when `swiglu_limit` is 0 |
| qwenimage | the DiT's matrices | int8, bf16, f32 (`COLI_IMG_BITS`) | text encoder, VAE, attention |
+| laya (Laya) | the whole forward: encoder and decision head layers ([below](#the-decision-engines)); with `COLI_VK_CHAIN=0` their matrices | f16 (fmt 14, the release's F16 values), f32 | the scorer and the act head on the marker rows, the tokenizer |
+| gliner_decide (GLiNER2.5-Decide) | the whole encoder ([below](#the-decision-engines)); with `COLI_VK_CHAIN=0` its matrices | f32 | the classifier on the `[L]` rows, the tokenizer |
Each engine ends a run, and each serve turn, with
`[VK] <engine>: N matmuls on the GPU`. That count is how you tell a path that ran
@@ -265,6 +267,116 @@ The CPU gains too where the block lets a matrix stay in cache across rows (Qwen3
DeltaNet: 7.1 to 2.5 s). Qwen3.8's block saves its 0.9 s of PLE GEMVs, inside the
run-to-run spread of its expert reads (cold page cache, about 2 s).
+## The decision engines
+
+Laya (`laya.c`), GLiNER2.5-Decide (`gliner_decide.c`) and Clef (`qwen36.c` with
+`clef_head.h`) answer a decision with one forward over the request's rows and
+generate no text. Their encoder work runs as a batch of matrix products.
+Build them with `VK=1` (`make laya gliner_decide qwen36 VK=1`) and set `COLI_VULKAN=1`.
+
+**Laya and GLiNER2.5-Decide** (`decide_vk.h`). The device opens once the weights are
+loaded, and every matrix of the encoder and the head goes up then, so the first request
+pays nothing for it: Laya's as f16 (fmt 14), since the release stores F16 and every value
+is one (the same values in half the bytes), GLiNER2.5-Decide's as f32 (fmt 10). A forward
+then runs one of two ways:
+
+- **On the device** (the default on a GPU, discrete or integrated; `COLI_VK_CHAIN=1`
+ anywhere). The engine records its whole forward as frames of the dense chain, one
+ submission per forward: the embedding rows go up, the encoder (and Laya's two head
+ layers) run in the CPU path's order, and only the rows the scorer reads come back
+ (Laya's markers and each sequence's first row, GLiNER2.5-Decide's `[L]` rows). The
+ scorer, the act head and the classifier, a few rows each, stay on the CPU.
+- **Matrix by matrix** (`COLI_VK_CHAIN=0`): every dense matrix through `coli_vk_matmul`
+ (the tiled GEMM from the backend's threshold), the norms, the attention and the rest
+ on the CPU. `COLI_VK_DENSE=0` as well keeps everything on the CPU.
+
+What runs between the matrices is `chain_enc.comp` (`vkc_enc_*` in `vk_chain.h`):
+
+| Op | What it does |
+|---|---|
+| `vkc_enc_norm` | LayerNorm with weight and bias, the residual add fused (`x += h; h = LN(x)` for ModernBERT's pre-norm layers, `x = LN(h + x)` for DeBERTa's post-norm ones) |
+| `vkc_enc_bias` | a matrix's bias, then GELU (erf or tanh), ReLU or nothing, in place |
+| `vkc_enc_geglu` | ModernBERT's GeGLU: the first half of `Wi`'s row through GELU, times the second |
+| `vkc_enc_rope` | rotate-half RoPE on the query and key heads, positions from each sequence's start, cos and sin from the host's own table (global and local theta) |
+| `vkc_enc_addrow` | a table row per row: Laya's type embedding by the row's question type |
+| `vkc_enc_attn` | bidirectional attention over a range of rows per row: its sequence (a request's questions are separate sequences in one batch), or Laya's +-64 window inside it; DeBERTa's two relative terms added by bucket, an online softmax over tiles of 32 rows, 16 query rows a workgroup |
+| `vkc_enc_rel` | DeBERTa's relative terms as its CPU attention computes them: each row's query against the key projection of the relative embeddings (content to position) and its key against their query projection (position to content), per head, for the buckets the forward can reach |
+
+The arithmetic is the CPU path's in f32: the two differ in the order of the sums, GELU's
+`erf` comes from a fit of `erfc` with a relative error under 1.2e-7, and LayerNorm sums in
+float where the engines sum in double. A device lost in the middle of a forward leaves
+that forward to the CPU, which recomputes it from the ids: a decision has no state to
+rebuild. Each request ends with `[VK] <engine>: N matmuls on the GPU` and
+`[VK] <engine> chain: F forwards on the device ...`.
+
+**Clef** runs through qwen36's own Vulkan paths, which DECIDE now uses: the backbone's
+prefill over the record matrix by matrix (the default on an integrated GPU, where a
+model without routed experts keeps its dense matrices on the device and the chain off)
+or as the dense chain (`COLI_VK_CHAIN=1`; the default on a discrete GPU), whose read-out
+brings back every row's hidden state for the head. The joint head stays on the CPU. A `[clef]` line reports backbone and head
+time separately so an end-to-end measurement can identify the limiting stage. Two things were missing for it:
+
+- **f16 rows.** `COLI_DENSE_BITS=16` keeps the trunk in the container's f16, and
+ qwen36's device path knew the int4, int8 and f32 copies only. The backend now takes f16
+ rows as fmt 14 (no scales, low half the even column, like bf16's 11) in the GEMV, the
+ fp32 tiled GEMM and the chain's decode GEMV; the cooperative-matrix GEMM keeps its
+ formats.
+- **The rows read in place** (`COLI_VK_IMPORT`). A resident copy on a device that shares
+ the CPU's RAM is the matrix held twice: Clef's trunk is 27 GB in int8 and 54 GB in
+ f16, and the f16 one does not fit twice in 61 GB. `coli_vk_tensor_import` hands the
+ device the host's own pages (`VK_EXT_external_memory_host`): qwen36 allocates its int8
+ and f16 rows page-aligned when `COLI_VULKAN` is set, and the device reads them where
+ they are; their scales, a float a row, are copied. On by default for a model without
+ routed experts on an integrated GPU or Lavapipe; every other model copies as before.
+ The harness checks imported rows bit for bit against a copy (int8, f32, f16, the GEMV
+ and the GEMM).
+
+The in-place import falls back to an ordinary resident copy if the extension, alignment
+or buffer requirements cannot be met. It is disabled with staged uploads. The host
+allocation remains alive while the device reads it. This saves the duplicate weight
+storage on a shared-memory GPU; it is not a claim of faster matrix multiplication.
+On an integrated GPU this also applies with `COLI_VK_DENSE_HOST=0`: imported rows
+already have one physical copy. Keeping them in place avoids moving the entire
+trunk into a potentially smaller device-local heap and exhausting its scratch budget.
+`COLI_VK_IMPORT=0` forces independent device copies and permits their host pages to
+be released. Only pages actually released contribute to the host-memory drop counter.
+
+The dense Qwen/Clef backbone submits each layer separately, retaining the residual
+stream on the device. Without routed experts there is no intermediate host readback
+to divide the work; recording all layers of a large prompt in one submission can
+exceed the driver's watchdog. The regression compares CPU answers across repeated
+requests and checks that the submission count grows with the number of layers.
+
+On the real Clef 27B int8 checkpoint, Radeon 780M/RADV completed six requests
+(254–755 tokens) with 401 independent device matrices, 23.86 GiB of host copies
+released and no CPU reloads. The shared-page configuration completed the same six
+requests with 24,412.5 MiB read in place; its maximum absolute logit difference from
+the independent copies was 3.5e-7. Against two CPU references with the same float
+activation arithmetic (`COLI_DENSE_IDOT=0`), the maximum difference was 3.42e-6.
+The shared-page budget selected 3,328 prompt rows automatically. These are correctness
+and memory checks on one integrated GPU; they do not establish throughput on a
+discrete GPU or under Windows.
+
+### Validation and performance
+
+`bash c/tests/vulkan_engines.sh decide` checks all three engines against their CPU
+outputs: identical decisions and refusals, probabilities within 1e-5 and logits within
+1e-4 relative to the largest magnitude (at least 1). It also verifies that work was
+actually dispatched. The matrix and chain paths, f16 and int8 Clef rows, import and
+copy, staged uploads, CPU fallback after a device loss, serve sessions and SDK requests
+are covered. `decide-sanitize` runs the comparison and serve tests under ASan and UBSan.
+With both `COLI_VK_CHAIN=0` and `COLI_VK_DENSE=0`, answers must equal the CPU's byte for
+byte apart from elapsed times. These gates have passed on Lavapipe; they establish
+correctness, not GPU speed.
+
+For latency, measure complete requests after a warm-up in the same running engine,
+using identical records, weight precision and CPU thread count. Compare the CPU with
+`COLI_VULKAN=1 COLI_VK_CHAIN=0` and `COLI_VULKAN=1 COLI_VK_CHAIN=1`, and report the median
+`engine_ms`, input lengths, GPU, available RAM and competing workload. Clef's
+backbone/head timing identifies how much of the request Vulkan can accelerate. No
+end-to-end speedup for all three real checkpoints is established by the fixture gates,
+and no discrete GPU measurement is included here.
+
## The routed-expert tier (`vk_tier.c`)
The engines above keep their dense matrices on the device (on a discrete GPU; on
@@ -301,8 +413,9 @@ of them on the device the way a GPU-equipped PC should use its card:
row in routing (rank) order, the device's and the CPU's alike: the same order as a
CPU-only run, so the device's experts differ from the CPU's only by their own
summation order. A device row's bits do not depend on how many rows share its
- dispatch either (up to 15 rows an expert takes the per-row GEMV route), so an MTP
- verify's two rows get a decode step's bits. From 16 rows (prefill) an expert takes
+ dispatch either (up to 15 rows an expert takes the per-row GEMV route), so a
+ speculative verify's rows (two to six, [speculative.md](speculative.md)) get a decode
+ step's bits. From 16 rows (prefill) an expert takes
the tiled GEMM for gate, up and down.
- **Without `COLI_VULKAN`, nothing changes.** In a `VK=1` build with `COLI_VULKAN`
unset, and in a build without `VK=1`, the stdout and the last logits of every
@@ -353,6 +466,7 @@ the device, since no Vulkan tier runs.
| `COLI_VK_TIER_SYNC` | `0` | `1`: each layer step first waits for the uploads staged so far: residency then follows the routing alone (with `COLI_VK_TIER_BALANCE=0`, the run is reproducible). A promotion that displaces a resident while a batch is in flight waits for the join to free it. For tests and debugging. |
| `COLI_VK_TIER_GEMM_ROWS` | `16` | Rows from which an expert of a step takes the tiled GEMM instead of the per-row GEMV; `0` never. |
| `COLI_VK_TIER_QUEUE` | a second queue | `0`: the tier shares the main queue (its batches and the dense matmuls then serialize). |
+| `COLI_VK_TIER_STREAM` | on | A prompt step's cold experts streamed to the device ([below](#big-prompt-chunks-and-expert-streaming)); `0` off. `_SLOTS`, `_ROWS`, `_HALF`, `_PAR` there. |
| `COLI_VK_DENSE` | on, but off on a device sharing the CPU's RAM while the tier is on | `0`: the dense trunk stays on the CPU and the device takes the routed experts only; `1`: the trunk on the device whatever the device. Unset: on a discrete GPU, or with the tier off, on the device; on an integrated GPU or Lavapipe with the tier on, on the CPU. The startup line says which and why. (The GLM engine reads it through the same rule with its own default, off.) |
| `COLI_USAGE` | `<snap>/.coli_usage` | The history the warm start reads; each engine's is in the table above. qwen36 and deepseek_v41 keep it only while the tier is on, and save it at the end of every run and serve turn. olmoe reads one only when this is set, inkling also takes `PIN=<path>`, deepseek_v4 reads its expert store's own (`<model>/.coli_usage`, always kept) and MiMo none. |
@@ -684,7 +798,7 @@ deepseek_v41 and deepseek_v4 with DeepSeek's own ([below](#deepseek-v41-flash-an
| device, frame A1 | the previous layer's MoE output joining the residual (in the CPU's order: routed experts in rank order, then the gated shared expert, then the add); the input RMSNorm; the gated attention (q/k/v, per-head q/k norm, RoPE, the new K/V rows into the device cache, attention with the output gate, o_proj) or the Gated DeltaNet (qkv/z/b/a, the causal convolution with its ring, the recurrence with its state, the gated norm, out_proj); the residual add; the post-attention norm; the router logits | the four hyper-connection streams: each block's gated-residual read (per-stream norm, the low-rank pair, the stream mix, the inject weights) and its write-back; the PLE layer's projections, gate and dilated convolution (the n-gram table rows come up from the host); QSA with its indexer: each block's pooled key computed once on the device when a step completes it, and the top-k selection per query row |
| host | the router's softmax and top-k; the routed experts (the expert tier's device batch and the CPU's share, joined in rank order); the new K/V rows copied into the host's cache | the new index-key rows too |
| device, frame A2 (not waited for) | the shared expert and its gate, while the host computes the routed experts | |
-| last frame | the final norm and lm_head on the last row | the final mixer; lm_head on the last one or two rows (an MTP verify) |
+| last frame | the final norm and lm_head on the last row, or on every row of a speculative verify | the final mixer; lm_head on the last row, or on every row of a verify |
Per layer the host gets the normalized rows the routed experts read (D floats a row)
and the router logits (E floats a row), and the new K/V (and index-key) rows of an
@@ -697,7 +811,9 @@ routed experts (Qwen3.8-27B) has no host step: its whole forward is one frame.
as with the GLM engine's KV mirror. The device holds a mirror per layer with a
watermark: rows below it equal the host's. A step from `pos_base` first uploads
the rows between the watermark and `pos_base`; a step that runs on the CPU lowers
- the watermark to its `pos_base`; a cache that grows is mirrored again. Qwen3.8's
+ the watermark to its `pos_base`; a cache that grows is mirrored again. Past the
+ device's budget the mirror keeps part of the cache and the CPU attends over the
+ rest ([below](#a-kv-cache-past-the-devices-budget)). Qwen3.8's
pooled block keys have a watermark of their own, lowered to the first block a step
rewrites (a rejected draft's block is recomputed when a step completes it again).
- The DeltaNet recurrent state and conv rings, and Qwen3.8's PLE ring: on the device
@@ -705,10 +821,16 @@ routed experts (Qwen3.8-27B) has no host step: its whole forward is one frame.
copy is brought back before anything reads it there (a pinned snapshot, the prompt
cache, a CPU step) and pushed up after anything writes it there (a reset: a fill
with zeros on the device; a restored snapshot: an upload).
-- An MTP verify (S = 2) snapshots the DeltaNet states, the conv rings and the PLE ring
- after its first row on the device (the shaders write the snapshot as they pass
- that row); a rejected draft swaps the device buffers, as the CPU swaps its own.
- Its matrices take the per-row GEMV, so its rows get a decode step's bits.
+- A speculative verify (S = 2 to 6: the token and its MTP or prompt-lookup drafts,
+ [speculative.md](speculative.md)) copies the DeltaNet states, the conv rings and the
+ PLE ring on the device after each of its rows but the last, one slot per row. One
+ copy the shaders write as they pass its row; more split the convolution and the
+ recurrence into one dispatch per copy, each ending on its row (the same bits). A draft
+ rejected after row r swaps slot r's device buffers in, as the CPU swaps its own, and
+ the KV and pooled-key watermarks come down to the standing rows. Its matrices take
+ the per-row GEMV, so its rows get a decode step's bits. Slots past the first are
+ allocated the first time a verify that deep runs (113 MiB each on Qwen3.8 Flash Next,
+ 63 MiB on Qwen3.6-35B).
Prompt-cache and prefix reuse need nothing else: a reused prefix is rows below the
watermark and a recurrent state that already sits where the next step expects it.
@@ -716,7 +838,7 @@ The serve and prefix tests run with the chain on (below).
**What stays on the CPU.** The routed experts the tier does not hold, the router's
top-k, the embedding gather and the vision tower's rows, Qwen3.8's n-gram table reads
-and the MTP head (its experts are FP8 beside the int4 sidecar, two rows per draft).
+and the MTP head (its experts are FP8 beside the int4 sidecar, a few rows per draft).
The chain declines, and the per-matrix path runs with the state synced first, under
the CUDA expert tier (CUDA keeps its priority), a qpack container, PILOT prefetch, or
a geometry outside its shaders (head dim above 256, a DeltaNet value head above 128 or
@@ -742,11 +864,15 @@ f32 throughout, as the CPU's f32 path.
| Variable | Default | Effect |
|---|---|---|
-| `COLI_VK_CHAIN` | on for a discrete GPU; on an integrated GPU with the expert tier, what the engine measured (qwen36 and olmoe on, qwen38 off; mimo, inkling, colibri, glm53, kimi_k3, deepseek_v41 and deepseek_v4 off: not measured); off on a CPU device | `1`: every layer's dense chain on the device; `2`: prompts only (forwards of more than two rows; decode and MTP verifies on the per-matrix path, the state moving between the two); `0`: the per-matrix path. The `[VK] <engine>: dense chain ...` line says which and why. |
-| `COLI_VK_CHAIN_ROWS` | `512` | Prompt rows per chunk: a longer prompt runs every layer chunk by chunk (the device's scratch is sized for one chunk). |
+| `COLI_VK_CHAIN` | on for a discrete GPU; on an integrated GPU with the expert tier, what the engine measured (qwen36 and olmoe on, qwen38 off; mimo, inkling, colibri, glm53, kimi_k3, deepseek_v41 and deepseek_v4 off: not measured); off on a CPU device | `1`: every layer's dense chain on the device; `2`: prompts only (forwards of more than two rows that are not a speculative verify; decode and verifies on the per-matrix path, the state moving between the two); `0`: the per-matrix path. The `[VK] <engine>: dense chain ...` line says which and why. |
+| `COLI_VK_CHAIN_ROWS` | from the budget, up to 8192 | Prompt rows per chunk: a longer prompt runs every layer chunk by chunk (the device's scratch is sized for one chunk). Unset or `auto`: the most rows whose buffers fit half the free device memory, up to `COLI_VK_CHAIN_ROWS_MAX` (8192); see [big prompt chunks](#big-prompt-chunks-and-expert-streaming). It was 512 before. |
| `COLI_VK_CHAIN_GEMV` | on | `0`: the decode matrices take `qmatmul.comp`'s GEMV instead of `chain_gemv.comp`'s. |
| `COLI_VK_CHAIN_SPIN_US` | `2000` | How long a wait on a chain frame polls the fence before blocking. |
| `COLI_VK_CHAIN_PROF` | off | `1`: one `[VK] chain profile` line of device time per kind of op (timestamps). |
+| `COLI_VK_KV_SPLIT` | on | `0`: never split the KV cache past the device's budget ([below](#a-kv-cache-past-the-devices-budget)); the whole mirrors, and past the budget the chain declines. |
+| `COLI_VK_KV_DEVICE_ROWS` | from the budget | The positions a split layer keeps on the device; set, the cache splits whenever it is longer. |
+| `COLI_VK_KV_BLOCK` | `64` | Positions per block of the split's block table. |
+| `COLI_VK_KV_PIN` | off | `1`: enable read-based block pins; QSA/DSA/pooled MLA rounding can then depend on read history. DeepSeek preserves its sparse arithmetic in either mode. |
Each run and serve turn prints `[VK] <engine> chain: N forwards, F frames (ops,
matmuls, tiled GEMM), the time spent waiting for the device, the routed experts' host
@@ -1414,6 +1540,383 @@ carried (ratios 1 to 4, the overlapping form with its bias, within 2e-6), the sc
the candidate mask and the top-k list slot for slot on scores exact in float on both
sides, ties included, with and without a mask, and the engram gate (within 2e-6).
+## Big prompt chunks and expert streaming
+
+A long prompt used to cross the chain in chunks of 512 rows, and a chunk's MoE step reached
+the tier in blocks of 32 to 128 rows (the engines' own prefill blocks): each block re-read
+the experts it routed, and the tier could take only the experts it already held. The CPU
+computed the rest, on a long prompt most of them, for every block. Two changes together:
+
+- **Big chunks.** The chain's chunk now comes from the device's free memory, up to 8192
+ rows, and the engines hand the tier a chunk's whole MoE step at once. One pass over a
+ layer's experts then serves every row of the chunk.
+- **Expert streaming.** In a step that big, a cold expert (not resident) is routed to
+ dozens or hundreds of rows. The tier uploads it into a staging slot on the device and
+ runs its rows as a GEMM there, instead of leaving it to the CPU. The slots are reused,
+ and the resident set and its LFRU are not touched.
+
+### The chunk (`vkc_chunk_rows`)
+
+`COLI_VK_CHAIN_ROWS=n` sets the chunk as before. Unset (or `auto`), each engine counts
+its chain's scratch per row from its own reservations (a counting pass over the same
+`vkc_reserve` calls), adds the simultaneous tier and CPU expert outputs and the
+engines' CPU gather/activation workspaces, and takes the most rows, up to
+`COLI_VK_CHAIN_ROWS_MAX` (8192), whose buffers fit half of the device memory free
+at its first forward. Free memory is
+`VK_EXT_memory_budget`'s budget less its use (without the extension, a quarter of the
+device-local heap). The CPU buffers must fit too: available physical RAM (and
+Windows commit headroom) also caps the budget. Qwen3.8 includes its simultaneous
+CPU expert input, gate, up and output buffers even when all experts run on the
+device. Chunks of at least 256 rows round down to a multiple of 256; tighter
+budgets keep the rows that fit, down to one. The decision
+is taken once per engine, after the tier has filled, so on a discrete card the room the
+tier leaves (`COLI_VK_TIER_RESERVE_GB`) is what the chunk gets. One line says what it
+took:
+
+```
+[VK] qwen36 chain: prompt chunks of up to 8192 rows (325 KiB a row; 7.30 GiB free on the device or in RAM, half of it at most; COLI_VK_CHAIN_ROWS sets it, COLI_VK_CHAIN_ROWS_MAX caps it at 8192)
+```
+
+The engines whose prefill blocks the prompt before the chain hand it the chain's chunk
+when the chunk comes from the budget: MiMo instead of `MIMO_CHUNK`'s 64 rows, GLM-5.3
+instead of `GLM53_PREFILL_CHUNK`'s 128, Kimi K3 instead of `K3_CHUNK`'s 32, DeepSeek V4
+instead of its 128-row CPU block (its chained MoE step, the expert union, takes any
+batch). Each of those variables, when set, keeps its blocks, as before. DeepSeek V4.1
+and V4 grow their window rings to the window plus the chunk. The indexers' score
+scratch keeps its cap (32M floats a forward on DeepSeek V4.1 and GLM-5.2).
+
+**What did not scale, and the fixes.** Every chain op takes up to 65535 rows; the
+dispatches over rows stay within the device limits at 8192. Two things did not scale:
+
+- **Attention reads.** `chain_attn.comp` reads a row's positions once per (head, row), so
+ a chunk of S rows read its KV cache S x H times: 1.2 s a layer for 2048 rows of
+ Qwen3.6 on the 780M. From `COLI_VK_ATTN_BLOCK` rows (16) `vkc_attn` takes
+ `chain_attnb.comp` instead. A workgroup takes the query heads of one KV head for a
+ block of rows (up to 32 (head, row) pairs). It walks the positions in tiles, each
+ tile's K and V rows staged in shared memory once for every pair: tiles of 4 to 16
+ positions, as the device's shared memory allows. It takes every form `chain_attn`
+ takes (a window, a ring, position-major rows, a sink, selection lists). Below 16 rows,
+ so for decode and MTP verifies, `chain_attn` runs as before with the same bits.
+- **Submission length.** The attention's work grows with rows x positions, and one
+ submission of a big chunk ran for seconds. On the 780M a frame of a 5632-row chunk at
+ 8192 positions was reset by the driver (`ring comp_1.1.0 timeout` in the kernel log,
+ the device lost). Past `COLI_VK_ATTN_SLICE` (2^32 rows x positions x heads x head
+ dim), `vkc_attn`, `vkc_attn_w`, `vkc_mla_core` and `vkc_relattn` record their rows in
+ slices. Each slice ends its frame, submitted and not waited for, and the next starts a
+ new one, which its first barrier orders after the slice. A slice starts on a multiple
+ of the blocked attention's rows, so no row's arithmetic changes.
+
+### Streaming (`vk_tier.c`)
+
+An engine that gives the tier `VktConfig.load` and `.release` (and `.load_batch` when it
+reads experts in parallel) streams. Those hooks hand the tier an expert's bytes as the
+CPU path would get them: through the engine's RAM cache, else from disk. Every MoE
+engine has them. For a step of at least `COLI_VK_TIER_GEMM_ROWS` rows (16),
+`vkt_step_rows` gives the engine the whole step instead of its usual block, and
+`vkt_issue` runs it as follows:
+
+- **The rule**, per cold expert of the step. Stream it when it has at least R rows:
+ R = max(G, ceil(upload / cpu_row)). G is the rows from which the device takes the
+ tiled GEMM (16; fewer rows would run the per-row GEMV). `upload` is the time one expert
+ takes to reach a slot, its bytes at the bandwidth the uploads so far measured.
+ `cpu_row` is the time the CPU spent per (row, expert) pair it computed in the prefill
+ steps so far, its share of a step between issue and join, loads included. Below R the
+ CPU keeps the expert: its rows cost the CPU less than the upload. Until both are
+ measured, R = G. `COLI_VK_TIER_STREAM_ROWS=n` fixes R at n. One line per forward says
+ what the rule gave:
+
+ ```
+ [VK] tier qwen36 stream: a step of 2560 rows; cold experts with 16 rows or more go to the device (an upload of 1.7 MiB at 8.2 GB/s takes 0.217 ms, the CPU 0.2280 ms a row; the GEMM from 16 rows); layer 0: 245 resident, 11 streamed, 0 kept on the CPU (0 rows)
+ ```
+
+- **Sub-batches** (backend: `coli_vk_xb_sub_*`). The step's device work runs as a
+ sequence of bounded batches: the resident experts first, then the streamed ones, with
+ at most half the slots and `COLI_VK_TIER_STREAM_HALF` rows a batch. That cap keeps a
+ batch's x rows within 32 MiB. Before allocating, the backend can reduce the batch
+ further to fit half the free budget of every affected memory heap and available
+ RAM. This calculation includes both halves, gate/up/hidden/output buffers, the
+ doubled descriptor windows and allocation growth; existing buffers count only
+ their additional bytes. A tight budget therefore reduces rows before an allocation
+ failure could disable the tier. An expert with more rows than a batch holds is cut
+ into parts.
+ Batch k goes to half k % 2 of the scratch, so while the device computes one, the engine
+ thread loads and uploads the experts of the next (double buffering), and a batch's
+ outputs are copied out when it is joined. The recording is the single batch's,
+ through the same `xb_record_submit`.
+- **The staging slots.** `COLI_VK_TIER_STREAM_SLOTS` (64) experts' worth of tensors, made
+ at the first step that streams and filled again for each expert they carry
+ (`coli_vk_tensor_refill`). On mapped memory that is the tensor's own mapping. With
+ staged uploads it is a host image that `coli_vk_tensor_commit` copies over. The slots
+ come out of the tier's budget: the residents give up what the budget cannot hold
+ beside them (64 slots of Qwen3.6's 7200 residents). The conversion into a slot is the
+ tier's own byte work (`convert`). A group the engine loaded together converts in
+ parallel, one expert per thread; a lone one converts with its rows split over the
+ threads (`COLI_VK_TIER_STREAM_PAR=0` keeps it on the engine thread).
+- **Nothing pollutes the resident set.** A streamed expert is not offered for promotion:
+ a prompt's whole working set passes through the slots and would otherwise flush the
+ residents. Its routings count as heat, as every routing does, and the slots are free
+ again when their batch is joined.
+- **Prefetch while attention runs.** The routing of a layer is known only after that
+ layer's attention and router, in every engine here: the router reads the post-attention
+ residual, so neither layer L's nor layer L+1's experts are known while L's attention
+ runs. What is known is a prediction. The chain engines submit the frame that computes
+ a layer's routing without waiting, call `vkt_stream_prefetch(layer, rows)`, then wait.
+ The tier fills its free slots with that layer's likely streamed experts while the
+ device works: the routing of the previous big step at the same layer when there was
+ one (a long prompt's earlier chunk), else the history's heat. A prediction that misses
+ costs an upload; an expert it does not cover is streamed after the routing as usual.
+- **Device work.** Streamed and resident experts use the tiled GEMM. From 64 rows
+ of one expert, devices supporting `VK_KHR_cooperative_matrix` also use its
+ cooperative path for int8, int4, int3, MXFP4 and FP8 projections with compatible
+ group sizes. Eligibility is checked separately for gate, up and down; other
+ formats retain the FP32 GEMM. Activation scales are computed on the device,
+ including the intermediate hidden rows, without a CPU readback. The hi/lo
+ activation split and FP32 accumulators follow the per-matrix path. Decode and
+ small draft-verification batches retain the GEMV. Integer weights use the
+ narrower tile; FP8 and MXFP4 retain the FP32 GEMM when a wider cooperative
+ tile would process extra padded rows. `COLI_VK_TIER_COOP=0` disables
+ this path and `COLI_VK_TIER_COOP_ROWS` changes its row threshold. The tier's
+ `cooperative matmuls` counter counts projections actually submitted.
+
+An isolated Radeon 780M/RADV expert benchmark (D=2560, intermediate=640,
+64 rows, two warmups and the median of nine alternating issue/join measurements)
+measured 1.410→1.153 ms for int8, 0.793→0.644 ms for int4-g64, and
+0.965→0.833 ms for FP8. These are expert-batch times, not model tokens/second.
+The correctness harness exercises mixed projection formats, incomplete tiles,
+zero and widely scaled activations, streaming and DeepSeek V4's rounding path.
+Formats or shapes that do not qualify retain the existing kernels.
+
+The run's line counts what streamed:
+
+```
+[VK] tier qwen36 run: ... | stream: 40 steps, 2188 cold experts (3.61 GiB, 10.72 GB/s) and 796154 rows streamed in 742 sub-batches, prefetched 0 (0 used), 929 cold experts (5548 rows) kept on the CPU
+```
+
+| Engine | Its loads (`load` / `load_batch`) | The step the tier sees |
+|---|---|---|
+| qwen36 | `expert_hold` (and the int8 copy for the int8-slot kernel), one at a time | `moe_vk_run`'s block becomes the chunk |
+| qwen38 | the layer cache, `q38_expert_get_batch` for a group | `q38_moe_prefill`'s rows the chunk |
+| olmoe | `expert_get` / `expert_put` | `moe_vk_run`'s block the chunk |
+| inkling | the slot cache, a group's misses filled in parallel | `ink_vk_moe`'s block the chunk |
+| mimo | `experts_ensure` for a group (parallel reads) | the chain's chunk instead of `MIMO_CHUNK` |
+| colibri | pins, the LRU, the misses read into the working set in parallel and promoted after the copy | `moe_vk`'s block the chunk |
+| glm53 | `expert_block_read` for a group | the chain's chunk instead of `GLM53_PREFILL_CHUNK` |
+| kimi_k3 | the layer cache, the misses read into the working set in parallel and promoted after the copy | the chain's chunk instead of `K3_CHUNK` |
+| deepseek_v41 | `expert_slots_at` for a group | the backbone's MoE takes the chunk (its arrays on the heap) |
+| deepseek_v4 | the store's lease (pinned rows16 experts unpacked as for a promotion) | the chain's chunk instead of the 128-row block |
+
+| Variable | Default | Effect |
+|---|---|---|
+| `COLI_VK_CHAIN_ROWS` | from the budget, up to 8192 | Prompt rows per chain chunk; `auto` as unset. Set, it also keeps the engines' own prompt blocks (above). |
+| `COLI_VK_CHAIN_ROWS_MAX` | `8192` | The most rows the budget's chunk takes. |
+| `COLI_VK_ATTN_BLOCK` | `16` | Rows from which the chain's attention takes `chain_attnb.comp`; `0` never. |
+| `COLI_VK_ATTN_SLICE` | `4294967296` | Rows x positions x heads x head dim past which an attention is cut over several submissions; `0` never. |
+| `COLI_VK_TIER_STREAM` | on | `0`: no streaming; the tier takes a prompt step in the engine's usual blocks and leaves the cold experts to the CPU, as before. |
+| `COLI_VK_TIER_STREAM_SLOTS` | `64` | Staging slots (experts) for streaming, out of the tier's budget. |
+| `COLI_VK_TIER_STREAM_ROWS` | the rule | Fix R, the rows from which a cold expert streams (tests). |
+| `COLI_VK_TIER_STREAM_HALF` | x rows within 32 MiB | Rows of a sub-batch (tests: small ones cut experts into parts). |
+| `COLI_VK_TIER_STREAM_PAR` | on | `0`: a streamed expert's conversion on the engine thread alone. |
+
+### Validation and timing
+
+The `prefill-*` families in `c/tests/vulkan_engines.sh` compare all ten MoE engines
+against their CPU path. They exercise the default chunk, forced small streaming
+slots and sub-batches, staged uploads, prefetch across chunks, and streaming off.
+The streaming cases require device work and streamed experts; a silent CPU fallback
+does not pass. Sanitized cases check the host allocations and lifetime handling.
+Lavapipe checks correctness and cannot establish hardware throughput.
+
+For a timing comparison, use the same model, quantization, prompt IDs, output length,
+RAM cache capacity and usage-history file in each run. Compare the former 512-row
+chain (`COLI_VK_CHAIN_ROWS=512 COLI_VK_ATTN_BLOCK=0 COLI_VK_TIER_STREAM=0`) with the
+default chunk and streaming. Also run the default chunk with only
+`COLI_VK_TIER_STREAM=0` to separate larger chunks from streaming. Record the selected
+chunk, resident and streamed experts, CPU rows, upload bytes, device wait time, peak
+RAM and device memory, time to first token and decode tokens per second. Repeat
+with both cold and warm caches on an otherwise idle machine. A shorter prefill does
+not imply faster decode; staging slots also consume room in the expert tier.
+
+A paired real-model check on Radeon 780M/RADV used Qwen3.8 Flash Next's int4-g64
+expert sidecar, a 2,048-token prompt, one generated token, eight CPU threads,
+16 RAM slots per layer and the same starting usage history. Each checkpoint was
+advised out of the filesystem cache before its run; no other inference process ran.
+
+| Configuration | Prompt plus one token, excluding load | Whole process | Expert assignments on GPU |
+|---|---:|---:|---:|
+| Former 512-row chunks, streaming off, host dense copies retained | 217.9 s | 229.1 s | 27.6% |
+| Automatic chunks, streaming, device-only dense weights; FP32 expert GEMM | 44.8 s | 56.3 s | 96.0% |
+| Same, cooperative expert GEMM enabled | 43.6 s | 55.3 s | 96.0% |
+
+The new budget selected 2,304 rows, released 4.07 GiB of dense host copies, and
+streamed about 21.8 GiB of cold experts. All three runs generated the same token.
+This is one paired prefill measurement per configuration. It measures the combined
+chunk, streaming and memory changes; the small difference between the last two
+rows does not establish an end-to-end cooperative-kernel gain by itself.
+
+For native FP8 decode on the same machine, a separate short prompt with 16
+generated tokens, CAP=16 and three runs per mode (one cold, two warm) produced identical text.
+Median decode speeds were 1.10 tok/s on the CPU, 1.42 with the expert tier and
+CPU dense work, and 1.14 with the full dense chain. The tier-only configuration
+won that workload. These measurements do not predict discrete-GPU or Windows
+performance, and the prefill result uses a different expert quantization.
+
+Performance depends on the device, memory bandwidth, model dimensions, expert
+coverage and prompt length. FP32 tiled GEMM does not imply that the expert batch
+uses matrix acceleration hardware. Measure each model and target GPU before
+choosing nondefault chunk or streaming settings.
+
+Additional real-model checks used the same Radeon 780M, eight threads, 64 generated
+tokens, identical starting expert histories within each comparison, and advisory
+eviction of checkpoint file pages before each process. Each configuration ran once
+with no concurrent inference. The values below are generated tokens divided by
+prompt plus generation time, excluding model load.
+
+| Model and starting history | CPU | Expert tier, CPU dense work | Expert tier and dense chain |
+|---|---:|---:|---:|
+| Qwen3.6-35B-A3B, existing int4-g64 container, cap 64 | 5.37 tok/s | 7.70 tok/s | 10.22 tok/s |
+| OlMoE-1B-7B, existing 8-bit container, cap 64, empty history | 13.30 tok/s | 10.97 tok/s | 9.43 tok/s |
+| Same OlMoE, common history learned from the CPU run | 13.19 tok/s | 11.77 tok/s | 16.35 tok/s |
+
+Qwen3.6's CPU and dense-chain output matched byte for byte; the tier-only output
+differed. Dynamic CPU/GPU expert placement changes reduction order and can change
+a greedy choice between close logits. All OlMoE configurations produced the same
+64 token IDs. Without history, the OlMoE tier uploaded 687 experts (4.03 GiB) during
+generation; with history it could warm the selected experts before the timed prompt.
+That startup distinction reverses which configuration wins. These are workload
+checks, not evidence that enabling every GPU path always improves latency.
+
+The [speculative decoding measurements](speculative.md#measured) also compare
+Qwen3.8's existing int4-g64 sidecar with three MTP drafts, prompt lookup and both
+together, and distinguish prompt time from generation time.
+
+### A KV cache past the device's budget
+
+Every chain engine keeps the host's KV cache canonical and mirrors each attention
+layer's cache on the device. A long context makes those mirrors large (Qwen3.6-35B-A3B's
+ten attention layers hold 40 KiB a position, 2.5 GiB at 64K), and past the device's
+budget the chain could not hold them. Then each split layer keeps only part of its cache
+on the device, the host's RAM holds the rest (it holds every row anyway), and a step's
+attention runs on both at once, the two results merged through their softmax
+statistics. Below the budget nothing changes: the whole mirrors, the same bits as
+before. The pieces are shared (`vk_kvsplit.h`, the `vkc_kvs_*` ops of `vk_chain.h`,
+`shaders/chain_kvs.comp`), so every attention form takes the same path.
+
+**What sits where.** A split layer has `ns` slots of `B` positions on the device (a
+block table, `bt`, maps a block to its slot). `nw` slots are a window over the newest
+blocks: a step's own rows always land there. Where the engine selects positions (Qwen3.8's
+QSA, GLM-5.2's DSA, GLM-5.3's pooled lists, DeepSeek's indexer), with `COLI_VK_KV_PIN=1` the other half of the
+slots holds the blocks the lists read most, counted per step and pinned two at a time
+ahead of pinned blocks read less (hits halve every 128 steps). A step from `pos_base`
+uploads the resident rows below it that the host wrote since (a watermark, as for the
+whole mirrors), and the whole of a block that just got its slot.
+
+**The partition, and why it is the row's own.** A row at position `pos` attends on the
+device over the blocks `pos/B - anchor .. pos/B` (and, from a list, the pinned blocks),
+and on the CPU over the rest. A step writes at most `chunk` rows (an eighth of the
+window), so every block of a row's share is in the window while any step holds the row,
+and the share covers the step's earlier rows, which reach the host's cache only after the
+step. Because the share depends on the row's position only, and the host's part is cut
+into chunks fixed by position and joined in order, a row gets the same bits however a
+forward is cut into steps: a prompt in one chunk or many, a decode step, a verify, a
+resumed prefix or a cold one. Pins follow the history of reads, so a listed row's bits
+can depend on it; pins are therefore off by default and enabled explicitly with
+`COLI_VK_KV_PIN=1`. DeepSeek stages only the missing compressed rows named by its
+selection, then runs the complete sparse attention on the device in the original list
+order. Its arithmetic remains independent of cache residency and pins.
+
+**A step.** One that sees nothing outside the device's share records the attention as
+before, in the layer's frame (`VKC_KVS_FIN`: the device's arithmetic gives the same
+bits as a merge with an empty host part). Otherwise the frame (F1) is submitted with the
+queries (and lists) copied down; the device's part is recorded in a frame of its own
+(F2) and submitted; the host waits for F1 only and computes its part over the host's
+rows while F2 runs (OpenMP over rows, heads and chunks of 2048 positions); a third frame
+(F3) uploads the host's part, merges (`vkc_kvs_merge`: `M = max(m_d, m_c)`, `out =
+(e^(m_d-M) acc_d + e^(m_c-M) acc_c) / (e^(m_d-M) l_d + e^(m_c-M) l_c)`, the gate after,
+DeepSeek V4's bf16 rounding after) and carries the rest of the layer. The extra host
+round trip is paid only on layers with a host part.
+
+| Op | Shader | What it does |
+|---|---|---|
+| `vkc_kvs_attn` | `chain_kvs` (mode 0) | grouped-query attention (chain_attn's arithmetic) over the device's share: head-major or position-major rows, V's own head dim, a window, a sink (the device's part holds it), lists |
+| `vkc_kvs_mla` | `chain_kvs` (mode 1) | the MLA core (chain_mla's) over the device's share of the latent and rope rows, causal or listed |
+| `vkc_kvs_rel` | `chain_kvs` (mode 3) | Inkling's attention (chain_relattn's): the relative-position bias and tau |
+| `vkc_kvs_ds` | `chain_kvs` (mode 4) | DeepSeek's sparse attention with its sink over the window rows and the resident compressed rows; with every listed row resident, `vkc_dsv4_attn`'s bits |
+| `vkc_kvs_merge` | `chain_kvs` (mode 2) | the two parts joined, a gate or V4's bf16 rounding after |
+
+`vk_kvsplit.h` carries the rest: the plan, the tables, the uploads and the stores of a
+step's rows, the host's part (`vkc_kv_host_attn`, with the same bias, tau and V4
+rounding), and one call per form for an engine (`vkc_kv_gqa`, `vkc_kv_mla`,
+`vkc_kv_mla_qkv` and `vkc_kv_mla_attn`, `vkc_kv_rel`, `vkc_kv_ds`).
+
+**The plan** (`vkc_kv_plan`, when an engine makes or grows its mirrors): the whole
+mirrors when they fit four fifths of the device's free budget (`VK_EXT_memory_budget`,
+or the largest device-local heap less what is allocated), else the rows that do, at
+least three blocks. `COLI_VK_KV_DEVICE_ROWS=N` sets the rows itself (the tests' small
+device, a measurement's split), `COLI_VK_KV_BLOCK` the block (64), `COLI_VK_KV_SPLIT=0`
+turns the split off (past the budget the chain declines as before), `COLI_VK_KV_PIN=1`
+enables read-based pins. The line says what was decided:
+
+```
+[VK] qwen36 chain: the KV cache split past the device's budget: 4096 of 65568 positions a layer on the device (64 blocks of 64, 0 for pins by reads; a row's newest 3584 positions on the device), the rest in RAM (160.0 MiB on the device instead of 2561.2)
+```
+
+and each run reports the split's steps: `[VK] <engine> chain: KV split: N layer steps, M
+with a host part (time waiting for the queries, host work over P positions, Q sparse rows staged), K
+blocks pinned by reads`. Sparse staging reuses at most 32 MiB of device scratch per
+query, accounted for by the chain allocator. Allocation or scratch-cap failure takes
+the existing CPU fallback; it never allocates another complete compressed cache.
+
+**What splits, per engine.**
+
+| Engine | Split | Stays whole on the device |
+|---|---|---|
+| qwen36 | the K/V of the attention layers | the DeltaNet state |
+| qwen38 | the K/V of the attention layers; QSA's lists reach both parts, pins follow them | the index keys and pooled block keys (every row scores them every step), the DeltaNet and PLE state |
+| olmoe | every layer's K/V | |
+| mimo | the full-attention layers' K/V (position-major, the sink in the device's part) | the sliding layers' rings (bounded by the window) |
+| inkling | the global layers' K/V (the bias and tau on both sides) | the sliding layers' rings, the convolution states |
+| colibri | the latent and rope rows; DSA's lists reach both parts, pins follow them | the DSA index keys |
+| glm53 | the MLA latent rows; the pooled lists reach both parts, pins follow them | the index keys, pool gates and pooled keys, the KDA state |
+| kimi_k3 | the gated MLA layers' latent rows | the KDA state |
+| deepseek_v41, deepseek_v4 | the compressed rows the sparse attention reads; the indexer's picks pin blocks; each chunk's new compressed rows reach the host's array as the chunk ends | the window ring, the index keys, the compressors' rings |
+
+Every engine's host cache already holds a step's rows before the next step runs, so the
+host's part reads only rows the host has; DeepSeek's chains, which wrote their state
+back at the end of a forward, now copy each chunk's new compressed rows as the chunk
+ends.
+
+**Arithmetic.** f32 throughout: the device and the host sum in other orders than the
+CPU's attention and than each other, so a split row agrees with the CPU's to rounding.
+DeepSeek V4 rounds its attention weights to bf16 against the maximum of the complete
+selection. The split preserves that maximum and the original accumulation order by
+staging missing selected rows. Independent partial bf16 softmaxes are not equivalent
+and are not used. Both DeepSeek variants retain the full-device result bit for bit;
+V4's chunk, decode versus teacher-forced, and draft-rejection checks remain enabled.
+
+**A lost device** is handled as each engine handles it without the split: the KV rows
+are the host's, so the CPU redoes the step (and an engine with a recurrent state
+rebuilds it from the prefix record, as before).
+
+**Tests.** `make vk-chain-check VK=1` runs the ops through the engines' sequence of
+steps (the window placed, the rows below uploaded, the step's rows stored, the parts
+merged), decode and prefill, against a double-precision reference of the full attention:
+GQA with a gate, a sink, position-major rows, a window and lists with pins; MLA (NoPE, a
+nonzero start, lists, latents up to 1024); Inkling's bias and tau; DeepSeek's lists
+(V4's rounding too, both split and whole caches equal `vkc_dsv4_attn` bit for bit); the MLA
+layer test of every geometry over a split cache; the host's cache filled as an engine's
+is (a step's rows only after the step, NaN before); and the same bits for every
+position in steps of 1, 2, 3 and 4. `tests/vulkan_engines.sh kv-split` gates every chain
+engine but DeepSeek's under an emulated small device (`COLI_VK_KV_DEVICE_ROWS` of 8 to 32
+positions in blocks of 2 to 8, contexts of 80 positions and more, most steps with a host
+part): the CPU's tokens, logits within each family's tolerance, chunks of 3, prompts
+only, the split off, a lost device, pins, MTP and n-gram drafts, serve sessions with
+pins, prompt-cache extensions and divergent prompts, and the prefix-reuse tests;
+`kv-split-deepseek` the same for deepseek_v41 and deepseek_v4; `kv-split-sanitize` and
+`kv-split-deepseek-sanitize` a set of each under ASan and UBSan.
+
+Validation on the local Lavapipe device covers numerical correctness, not hardware
+throughput. A discrete GPU has not been measured for this change.
+
## Adding an engine to the tier
Every MoE engine here is on the tier ([the table above](#the-routed-expert-tier-vk_tierc));
@@ -1582,6 +2085,88 @@ Resizable BAR**, the case staging is for (none is available); there the copies c
once per upload and the device then reads VRAM instead of the window or system RAM. The
contributor who reported the RTX 3070 offered to run it.
+## Dense weights on the device only
+
+Without this mode, an engine whose dense part runs on the device (the chain, or the
+per-matrix path with `COLI_VK_DENSE`) keeps the host copy of every resident dense
+matrix as the CPU's fallback. That costs the dense part's RAM twice on an integrated
+GPU (the device's memory is the same RAM), and on any machine it takes from the
+experts' RAM cache: the cap or the plan counts the dense weights in RAM. On a 32 GB
+machine with a 16 GB card the difference decides the plan. DeepSeek V4 Flash there
+(#1852) planned its dense layers as not resident (reloaded from disk every forward),
+so its chain declined and the CPU did the dense work while the GPU sat at 9%.
+
+`COLI_VK_DENSE_HOST=0` puts the dense weights on the device only:
+
+- **At start**, once the device and the chain are decided and before the expert tier
+ sizes its budget, every resident dense matrix goes up (staged or mapped, as
+ [above](#memory-placement-without-resizable-bar)) and its host copy is given back.
+ The per-matrix path and the chain share those tensors. The tier's budget sees them
+ placed.
+- **Every step the chain does not run** (prompts only, a declined step, a head or a
+ draft the engine runs per matrix) takes the device too: the CPU has no copy. In this
+ mode the engine forces the per-matrix path on, whatever `COLI_VK_DENSE` says.
+- **The CPU fallback still works.** When the CPU needs a matrix the device holds alone
+ (a lost device, `COLI_VK_CHAIN_FAULT`, a staged upload that failed), the engine reads
+ it back from the checkpoint, matrix by matrix as each is first needed, with the bytes
+ the load made (the same conversion: Qwen3.8's int8 rows, GLM's quantization, ...),
+ and keeps it from there on. The tokens are the CPU's.
+- **What stays on the host** is what the CPU reads directly every step: the embedding
+ tables (their rows are gathered on the CPU), norms and small vectors, and per engine
+ the pieces listed below. Qwen/Clef's imported weight pages on an integrated GPU
+ also stay alive: those rows are shared with the device and already occupy RAM once.
+- **The RAM goes to the experts.** Each engine that sizes its own expert cache from
+ the RAM it sees counts the dense weights as not in RAM in this mode, and so does
+ `coli plan` (`resource_plan.py`) credits only recognized matrices and formats
+ that the engine can release. It keeps the embedding, CPU-readable vectors,
+ norms and unrecognized components in the RAM reservation. Packed formats whose
+ runtime layout cannot be established from the checkpoint, GLM's CLI-selected
+ QT formats, and Inkling's hardware-dependent BF16 path receive no advance RAM
+ credit; the runtime may release more after successful uploads. Quantized matrices
+ receive a conservative credit for their loaded size, not the larger source
+ tensor. Qwen3.8's CPU-int8 estimate follows `Q38_TRUNK_SKIP` and
+ `Q38_TRUNK_MIN_KB` and includes the float scales per row. On an integrated
+ GPU the device copy still occupies physical RAM: the planner charges it once in
+ `shared_device_dense_bytes`, inside `runtime_bytes`. Dropping the host copy frees
+ one copy of the weights; it does not remove the device's memory from the budget.
+ On a discrete GPU the fit decision uses the device budget minus its current heap
+ usage and a 1 GiB reserve. The shared-device reservation and fit estimate remain
+ conservative when a component's device representation is unknown. The resulting
+ expert capacity depends on that device, context size and available memory.
+
+For GLM-5.3 the RAM estimate applies `GLM53_BITS` to recognized dense matrices
+individually. Int4 requires columns divisible by 64; narrower matrices use int8
+rows with a float scale per row, as in the engine. Embeddings, norms, routers,
+vision weights and unknown floating components retain an F32 reservation. The
+absorbed `kv_b` matrices also retain a conservative F32 reservation. Packed source
+weights and their scales keep their stored size. Cached checkpoint scans do not
+cache environment settings: changing `GLM53_BITS` updates the next plan without
+requiring a rescan.
+
+**The default** (`coli_vk_dense_host_decide`, beside `coli_vk_dense_decide` and
+`coli_vk_chain_decide`): with the dense part on the device, an integrated GPU drops
+the host copies (its memory is the same RAM: the copy would hold the dense part
+twice); a discrete GPU drops them when its free memory (`VK_EXT_memory_budget`), less
+1 GiB, holds them, else keeps them as the fallback; a CPU device (Lavapipe) keeps them.
+`COLI_VK_DENSE_HOST=1` keeps them anywhere. With the dense part on the CPU there is
+nothing to drop. The lines it prints:
+
+```
+[VK] qwen38: dense weights on the device only (an integrated GPU: its memory is the CPU's RAM, the host copy would hold them twice; COLI_VK_DENSE_HOST=1 keeps it)
+[VK] qwen38: 1007 dense matrices on the device only, 4.21 GiB of host RAM given back; kept on the host: the embedding (its rows are gathered on the CPU), the vision tower, norms; RSS now 2.10 GiB
+...
+[VK] qwen38: dense weights at exit: 1007 matrices on the device only (4.21 GiB of host copies dropped), 0 read back from disk for the CPU (0.0 MiB); RSS 24.60 GiB
+```
+
+The `dense-only-*` test families cover Qwen3.6, Qwen3.8, OLMoE, Inkling, MiMo,
+Kimi K3, GLM-5.2, GLM-5.3, DeepSeek V4 and V4.1, including staged uploads,
+device loss and sanitizer variants. On Kimi K3 the generated steps' logits must
+match the run retaining its host copies byte for byte. Earlier prompt logprobs
+use a CPU head with host copies and a GPU head without them, so they retain the
+existing CPU-reference numerical bound; they are not an identical-arithmetic
+comparison. A trace that declines the chain must still exercise the per-matrix
+GPU fallback.
+
## Correctness
- `gcc -O3 -DVK_TEST backend_vulkan.c -o test_vk -lvulkan -lm && ./test_vk
@@ -1652,6 +2237,24 @@ contributor who reported the RTX 3070 offered to run it.
and mid-decode (the KDA state rebuilt), serve sessions with prefix reuse and
recurrent-state checkpoints in RAM and on disk; `kimi-chain-sanitize` under ASan and
UBSan.
+- Big prompt chunks and expert streaming: `make vk-tier-check VK=1` (`test_vk_tier`'s
+ `stream` section) runs big steps through the sub-batches against its CPU reference in
+ six source formats and all three activations (DeepSeek V4's roundings bit for bit):
+ experts cut into parts, the staging slots filled through both hooks, prefetch, cold
+ experts below the rule's rows left to the CPU, and the resident set the warm start's
+ before and after. `vk-chain-check` runs the blocked attention against the same cases
+ as `chain_attn` (windows, rings, position-major rows, sinks, lists), and every
+ attention op cut in slices. `tests/vulkan_engines.sh prefill-qwen`,
+ `prefill-inkling-olmoe`, `prefill-mimo-kimi`, `prefill-glm` and `prefill-deepseek`
+ run each chain engine on a prompt past its usual block (110 to 300 tokens), four ways:
+ the chunk from the budget with the rule's streaming; a forced chunk of 40 to 100 rows
+ with a forced threshold of 3 rows, four slots and sub-batches of 24 rows; the same
+ staged; the feature off (`COLI_VK_CHAIN_ROWS=512 COLI_VK_TIER_STREAM=0
+ COLI_VK_ATTN_BLOCK=0`). Each one gives the CPU's tokens and logits, the streamed runs
+ show streamed experts, and a run over several chunks shows prefetched experts used.
+ Also qwen36's device lost while a prompt's sub-batches are in flight (the state
+ rebuilt on the CPU). The forced runs run again under ASan and UBSan, staged and
+ mapped (`prefill-qwen-sanitize` and the families' own sanitized halves).
- int4 weights decode as offset-binary (nibble−8), byte-identical layout to
the CPU path — no repacking.
- Khronos validation layers: the backend never enables them, so the loader
@@ -1710,6 +2313,9 @@ hit-rate line is the tier-effectiveness number.
expert tier, the KV mirror) goes through staged uploads
([above](#memory-placement-without-resizable-bar)); that path is tested by forcing
it and by emulating the small window, and not yet measured on such a card.
+- Streaming copies each assignment's input row into the scratch and each output row
+ out of it (the expert batch's layout). A gather of the chunk's rows by index in the
+ GEMM shaders would save both copies; it is not written.
- Without the dense chain, DSA top-k selection, ragged multi-slot serving, and
quantized-KV caches fall back to the CPU attention path; with it, the DSA selection
runs on the device.