Run frontier MoE models on hardware you already own — pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦
C
cuda tier: size the VRAM prefix against the measured headroom, not the requested budget (#1405)
JustVugg6cbd5aaf4c8d
diff --git a/c/colibri.c b/c/colibri.cindex 879237e4..ddedb990 100644--- a/c/colibri.c+++ b/c/colibri.c@@ -9815,7 +9815,16 @@ static void pin_load(Model *m, const char *statspath, double gb, int trusted){#ifdef COLI_ANSraw_n=g_cuda_raw_experts;#endif- prefix_est=pin_prefix_for_budget(m,r,n,budget,raw_n)+g_cuda_ndev;+ /* Size the prefix against what the card can actually take, not the+ * number on the command line. An explicit CUDA_EXPERT_GB above the+ * measured headroom is honoured by the upload loop (#491: it degrades+ * per expert), but a prefix estimated from it lands its excess in the+ * RAM pin: 5090 + CUDA_DENSE=1 + CUDA_EXPERT_GB=28, headroom ~18 GB,+ * 864 uploaded and the other ~450 of a 1,314 prefix pinned in RAM on+ * top of PIN_GB, 9.7 GB the user never asked for (#1405). */+ double prefix_budget=budget;+ if(safe_total>0 && safe_total<prefix_budget) prefix_budget=safe_total;+ prefix_est=pin_prefix_for_budget(m,r,n,prefix_budget,raw_n)+g_cuda_ndev;if(prefix_est>n) prefix_est=n;cpu_from=prefix_est; /* prefix RAM is returned after upload */}