Skip to content

Run frontier MoE models on hardware you already own β€” pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦

C

Merge remote-tracking branch 'origin/dev' into pr259

JustVugg3fdc6d394edd
diff --cc README.md
index e34508ce,4521c830..62ba5c78
--- a/README.md
+++ b/README.md
@@@ -468,7 -468,7 +468,7 @@@ works against the colibrì OpenAI-compa
compatible endpoint. Nothing leaves the endpoint you configure. The terminal
`coli chat` remains the first-class interface.
- Useful knobs (env or flags): `--temp T` token sampling temperature (default 0.7 + nucleus 0.90 β€” tuned for int4; 0 = greedy), `--topp 0.7` adaptive expert top-p (30–40% less disk), `--ngen N` max tokens per answer (`:more` in chat continues a truncated one), `--repin N` adapt RAM/VRAM hot experts every N emitted tokens, `AUTOPIN=0` disable the learning cache's auto-pin, `THINK=1` enable GLM-5.2's reasoning block, `DRAFT=n` MTP draft depth, `GRAMMAR=g.gbnf` grammar-forced drafts for constrained JSON/NDJSON output (`GRAMMAR_DRAFT=n` caps the forced span), `TF=1` teacher-forcing validation, `PILOT=1` router-lookahead disk prefetch (experimental β€” see below), `URING=1` Linux-only batched expert I/O (implies `PIPE=1`; also batches `PILOT_REAL`), `CAP_RAISE=0` don't auto-grow the expert cache.
-Useful knobs (env or flags): `--temp T` token sampling temperature (default 0.7 + nucleus 0.90 β€” tuned for int4; 0 = greedy), `--topp 0.7` adaptive expert top-p (30–40% less disk), `--ngen N` max tokens per answer (`:more` in chat continues a truncated one), `--repin N` adapt RAM/VRAM hot experts every N emitted tokens, `AUTOPIN=0` disable the learning cache's auto-pin, `THINK=1` enable GLM-5.2's reasoning block, `DRAFT=n` MTP draft depth, `GRAMMAR=g.gbnf` grammar-forced drafts for constrained JSON/NDJSON output (`GRAMMAR_DRAFT=n` caps the forced span), `TF=1` teacher-forcing validation, `PILOT=1` router-lookahead disk prefetch (experimental β€” see below), `CAP_RAISE=0` don't auto-grow the expert cache, `PIPE=0` disable the async expert-load pool (**default ON on Windows** since `windows-optimizations` β€” overlaps expert `pread` with the matmul so the CPU isn't idle waiting on the SSD; measured βˆ’18% disk service time), `RAM_GB=<n>` claim more RAM for the expert cache than the conservative auto-detect (e.g. `RAM_GB=31` on a 32 GB host raises the cache cap and hit rate measurably).
++Useful knobs (env or flags): `--temp T` token sampling temperature (default 0.7 + nucleus 0.90 β€” tuned for int4; 0 = greedy), `--topp 0.7` adaptive expert top-p (30–40% less disk), `--ngen N` max tokens per answer (`:more` in chat continues a truncated one), `--repin N` adapt RAM/VRAM hot experts every N emitted tokens, `AUTOPIN=0` disable the learning cache's auto-pin, `THINK=1` enable GLM-5.2's reasoning block, `DRAFT=n` MTP draft depth, `GRAMMAR=g.gbnf` grammar-forced drafts for constrained JSON/NDJSON output (`GRAMMAR_DRAFT=n` caps the forced span), `TF=1` teacher-forcing validation, `PILOT=1` router-lookahead disk prefetch (experimental β€” see below), `URING=1` Linux-only batched expert I/O (implies `PIPE=1`; also batches `PILOT_REAL`), `PIPE=0` disable the async expert-load pool (**default ON on Windows** β€” overlaps expert `pread` with the matmul so the CPU isn't idle waiting on the SSD; measured βˆ’18% disk service time), `RAM_GB=<n>` claim more RAM for the expert cache than the conservative auto-detect (e.g. `RAM_GB=31` on a 32 GB host raises the cache cap and hit rate measurably), `CAP_RAISE=0` don't auto-grow the expert cache.
### Resource policy
diff --cc c/glm.c
index 81978a81,8c1c112c..80952d8c
--- a/c/glm.c
+++ b/c/glm.c
@@@ -35,11 -35,9 +35,12 @@@
#include <sys/resource.h>
#include <sys/mman.h> /* mlock: inchioda le pagine in RAM / wire pages into RAM */
#include <sys/stat.h> /* fstat per mmap degli shard (COLI_MMAP) */
+ #include <signal.h> /* SIGINT = stop morbido del turno in serve mode */
#endif
#include "st.h"
+#ifdef __linux__
+#include "uring.h"
+#endif
#include "tok.h"
#include "tier.h"
#include "grammar.h" /* metodo F: draft grammaticali (#48) */
@@@ -1801,9 -1654,11 +1841,12 @@@ static int uring_wait_all(UringBatch *b
* condvar exist ONLY to park/wake idle workers, never for correctness. Gated
* behind PIPE=1; OFF => the original blocking-load + serial-matmul path runs
* byte-identically. */
- static int g_pipe=0; /* PIPE=1: async expert-load pipeline (default OFF) */
+ static int g_pipe=0; /* PIPE=1: async expert-load pipeline. Default ON for Windows
+ * (parsed in main: getenv("PIPE")?:1 on _WIN32, :0 elsewhere).
+ * Keeps expert pread off the forward-pass thread so loads overlap
+ * the matmul. PIPE=0 opts back into the blocking serial path. */
static int g_pipe_nw=8; /* PIPE_WORKERS=n: I/O worker threads (disk-parallel reads) */
+static int g_uring=0; /* URING=1: Linux io_uring load/completion backend; implies PIPE */
typedef struct {
_Atomic uint64_t cur; /* (gen<<8)|index; gen main-only, index 0..njobs (≀64) */
_Atomic int njobs; /* current batch job count */