Skip to content

Run frontier MoE models on hardware you already own — pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦

C

vulkan: a chain default for engines not measured on an integrated GPU

JustVugg2ae408ed4444
diff --git a/c/backend_vulkan.c b/c/backend_vulkan.c
index 7f931a6c..443f04b7 100644
--- a/c/backend_vulkan.c
+++ b/c/backend_vulkan.c
@@ -1227,7 +1227,8 @@ int coli_vk_dense(void) { return g_dense_on; }
* COLI_VK_CHAIN set and non-empty: 0 off, 2 prefill only, any other number on.
* Unset, a discrete GPU: on.
* Unset, an integrated GPU: `igpu` when the engine runs the expert tier (tier_on),
- * else off. `igpu` is what the engine measured on one (a Radeon 780M, docs/vulkan.md,
+ * else off; an engine never timed on one passes COLI_VK_CHAIN_UNMEASURED, which is
+ * off and says so. `igpu` is what the engine measured on one (a Radeon 780M, docs/vulkan.md,
* "The dense chain"): qwen36 passes ON (its chain won decode, 9.9 against 8.0 tok/s,
* and prefill, 9.5 against 12.2 s); qwen38 OFF (its chain won prefill, 30.1 against
* 38.7 s, and lost decode, 3.2 against 3.8 tok/s: the device's GEMV at the GPU's
@@ -1235,8 +1236,7 @@ int coli_vk_dense(void) { return g_dense_on; }
* still lost 5% of decode, the trunk's device copy taking from the tier's budget).
* olmoe ON (OLMoE-1B-7B: the chain decoded 17.3 against 12.8 tok/s and prefilled 512
* tokens in 5.5 against 6.4 s; the CPU alone decodes 23.1 tok/s, its trunk being f32).
- * An engine that has no measurement there passes COLI_VK_CHAIN_UNMEASURED: off, and
- * the line says it was not measured (inkling: no checkpoint of it runs on the box).
+ * inkling passes COLI_VK_CHAIN_UNMEASURED (no checkpoint of it runs on the box).
* Unset, a CPU device (Lavapipe): off.
* Printed as a [VK] line with an engine name (NULL: silent). */
int coli_vk_chain_decide(const char *engine, int tier_on, int igpu) {
@@ -1248,12 +1248,13 @@ int coli_vk_chain_decide(const char *engine, int tier_on, int igpu) {
on = v == 0 ? COLI_VK_CHAIN_OFF : v == 2 ? COLI_VK_CHAIN_PREFILL : COLI_VK_CHAIN_ON;
snprintf(why, sizeof why, "COLI_VK_CHAIN=%s", e);
} else if (coli_vk_device_integrated()) {
- on = tier_on && igpu != COLI_VK_CHAIN_UNMEASURED ? igpu : COLI_VK_CHAIN_OFF;
+ int measured = igpu != COLI_VK_CHAIN_UNMEASURED;
+ on = tier_on && measured ? igpu : COLI_VK_CHAIN_OFF;
snprintf(why, sizeof why, "an integrated GPU%s: %s; COLI_VK_CHAIN=0 off, 1 on, 2 prompts only",
tier_on ? " with the expert tier" : " without the expert tier",
on == COLI_VK_CHAIN_ON ? "measured faster on decode and prefill"
: on == COLI_VK_CHAIN_PREFILL ? "measured faster on prefill, slower on decode"
- : tier_on && igpu != COLI_VK_CHAIN_UNMEASURED ? "measured slower on decode" : "not measured");
+ : tier_on && measured ? "measured slower on decode" : "not measured");
} else if (coli_vk_device_shares_ram()) {
on = COLI_VK_CHAIN_OFF;
snprintf(why, sizeof why, "a CPU device; COLI_VK_CHAIN=1 turns it on");
diff --git a/c/backend_vulkan.h b/c/backend_vulkan.h
index 64391814..a4f73e5a 100644
--- a/c/backend_vulkan.h
+++ b/c/backend_vulkan.h
@@ -249,9 +249,7 @@ unsigned long long coli_vk_matmul_calls(void);
#define COLI_VK_CHAIN_OFF 0
#define COLI_VK_CHAIN_ON 1
#define COLI_VK_CHAIN_PREFILL 2 /* forwards of more than two rows only */
-/* `igpu` only: the engine was not measured on an integrated GPU; off there, and the
- * [VK] line says so */
-#define COLI_VK_CHAIN_UNMEASURED 3
+#define COLI_VK_CHAIN_UNMEASURED 3 /* as `igpu`: not measured on an integrated GPU, so off there */
int coli_vk_chain_decide(const char *engine, int tier_on, int igpu);
/* The device as the chain sees it: Vulkan handles as void * (VkInstance,
* VkPhysicalDevice, VkDevice, VkQueue), the memory types the backend picked, the
diff --git a/docs/vulkan.md b/docs/vulkan.md
index c8f5d388..cd894397 100644
--- a/docs/vulkan.md
+++ b/docs/vulkan.md
@@ -810,7 +810,9 @@ What the numbers say:
GPU the chain is on. On an integrated GPU with the expert tier on, each engine passes
what it measured here: qwen36 on, qwen38 off (decode, a chat's steady state, is slower
in both modes; `COLI_VK_CHAIN=2` is the choice for long prompts). Without the tier, and
-on a CPU device such as Lavapipe, it is off. The startup line says which and why:
+on a CPU device such as Lavapipe, it is off. An engine not timed on an integrated GPU
+passes `COLI_VK_CHAIN_UNMEASURED`: off there, and the line says "not measured". The
+startup line says which and why:
```
[VK] qwen36: dense chain on (an integrated GPU with the expert tier: measured faster on decode and prefill; COLI_VK_CHAIN=0 off, 1 on, 2 prompts only)