Skip to content

Run frontier MoE models on hardware you already own — pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦

C

fix(qwen36): skip RAM-derived cap for dense checkpoints (num_experts 0)

Jaapjan Tinbergena67674f7fe88
diff --git a/c/qwen36.c b/c/qwen36.c
index 80673482..1313c051 100644
--- a/c/qwen36.c
+++ b/c/qwen36.c
@@ -2242,7 +2242,15 @@ static void model_init_range(Model *m, const char *snap, int cap, int bits,
if (quantize_dense)
fprintf(stderr, "[dense-i8] %d matrices %s during load, %.1f GB f32 freed\n", qcount,
dense_bits() == 16 ? "kept in f16" : "quantized", qfreed/1073741824.0);
- if (cap <= 0) {
+ if (cap <= 0 && c->n_experts <= 0) {
+ /* Dense checkpoint (#1757): nothing is ever routed, so the per-layer
+ * expert cache is never touched by moe()/expert_get(). Sizing it
+ * from RAM would be meaningless -- n_experts==0 has nothing to clamp
+ * qwen36_cap_for_ram's derived value against, so an unclamped RAM
+ * budget could otherwise calloc an absurd slot count for a cache
+ * that will sit empty. cap=1 is a harmless placeholder. */
+ cap = 1;
+ } else if (cap <= 0) {
/* cap<=0 sentinel: derive from host RAM, same "0 = auto" convention as
* colibri.c/olmoe.c. rss_gb() here reflects all dense weights resident
* (this loop just finished) but not yet DN_rec/DN_conv/m->cache