Run frontier MoE models on hardware you already own — pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦
C
serve: COLI_SERVE_ALL_STOPS re-arms the full stop set (#549)
JustVugga391eeefa0b4
diff --git a/c/sample.h b/c/sample.hindex 414c6210..1df00ad4 100644--- a/c/sample.h+++ b/c/sample.h@@ -119,8 +119,14 @@ static void stops_arm_tok(const Cfg *c, int tok_eos, Tok *T){/* #401: in serve mode keep ONLY <|endoftext|>. Role markers <|user|>/<|observation|>* (config stops + tokenizer special set) are boundaries the Python server owns; as* hard stops they cut generation the moment the model opens a <tool_call> block,- * because int4 argmax noise picks a stop-token ID over the correct '<' token. */- if (getenv("SERVE") && tok_eos >= 0) {+ * because int4 argmax noise picks a stop-token ID over the correct '<' token.+ *+ * #549: on some quantized containers (notably int4-gs64) an end-of-turn special+ * OTHER than <|endoftext|> wins the final-token margin, so serve mode never stops and+ * runs into hallucinated turns. COLI_SERVE_ALL_STOPS=1 re-arms the full stop set for+ * users who are NOT doing tool calls (or whose client owns the tool boundary), at the+ * cost of the #401 tool-call safety. Default off — filter unchanged. */+ if (getenv("SERVE") && tok_eos >= 0 && !getenv("COLI_SERVE_ALL_STOPS")) {int kept = 0;for (int i = 0; i < g_nstop; i++) if (g_stop[i] == tok_eos) g_stop[kept++] = g_stop[i];if (kept < g_nstop) fprintf(stderr, "[stop] serve mode: filtered %d non-EOS stop tokens (tool-call safety, #401)\n", g_nstop - kept);