Skip to content

Run frontier MoE models on hardware you already own — pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦

C

glm: COLI_VK_EXPERTS=0 disables the VK expert tier (experts stay on the parallel CPU loop)

Steve Markgraf936d13e081fa
diff --git a/c/colibri.c b/c/colibri.c
index 6f1d0c81..712296d5 100644
--- a/c/colibri.c
+++ b/c/colibri.c
@@ -3297,7 +3297,8 @@ static void moe(Model *m, Layer *l, int layer, float *x, int S, float *out, int
#endif
int vk_active = 0; (void)vk_active;
#ifdef COLI_VULKAN
- vk_active = g_vulkan && !omp_in_parallel() && S<=4;
+ vk_active = g_vulkan && g_vk_budget>0 && !omp_in_parallel() && S<=4; /* budget 0 = tier off,
+ * experts stay on the normal (parallel) CPU loop */
float *vk_xh = vk_active?falloc((int64_t)S*K*D):NULL;
float *vk_yh = vk_active?falloc((int64_t)S*K*D):NULL;
#endif