Run frontier MoE models on hardware you already own β pure C, zero deps, experts streamed from disk. Tiny engine, immense model. π¦
C
Merge remote-tracking branch 'origin/dev' into pr259
JustVugg3fdc6d394edd
diff --cc README.mdindex e34508ce,4521c830..62ba5c78--- a/README.md+++ b/README.md@@@ -468,7 -468,7 +468,7 @@@ works against the colibrΓ¬ OpenAI-compacompatible endpoint. Nothing leaves the endpoint you configure. The terminal`coli chat` remains the first-class interface.- Useful knobs (env or flags): `--temp T` token sampling temperature (default 0.7 + nucleus 0.90 β tuned for int4; 0 = greedy), `--topp 0.7` adaptive expert top-p (30β40% less disk), `--ngen N` max tokens per answer (`:more` in chat continues a truncated one), `--repin N` adapt RAM/VRAM hot experts every N emitted tokens, `AUTOPIN=0` disable the learning cache's auto-pin, `THINK=1` enable GLM-5.2's reasoning block, `DRAFT=n` MTP draft depth, `GRAMMAR=g.gbnf` grammar-forced drafts for constrained JSON/NDJSON output (`GRAMMAR_DRAFT=n` caps the forced span), `TF=1` teacher-forcing validation, `PILOT=1` router-lookahead disk prefetch (experimental β see below), `URING=1` Linux-only batched expert I/O (implies `PIPE=1`; also batches `PILOT_REAL`), `CAP_RAISE=0` don't auto-grow the expert cache.-Useful knobs (env or flags): `--temp T` token sampling temperature (default 0.7 + nucleus 0.90 β tuned for int4; 0 = greedy), `--topp 0.7` adaptive expert top-p (30β40% less disk), `--ngen N` max tokens per answer (`:more` in chat continues a truncated one), `--repin N` adapt RAM/VRAM hot experts every N emitted tokens, `AUTOPIN=0` disable the learning cache's auto-pin, `THINK=1` enable GLM-5.2's reasoning block, `DRAFT=n` MTP draft depth, `GRAMMAR=g.gbnf` grammar-forced drafts for constrained JSON/NDJSON output (`GRAMMAR_DRAFT=n` caps the forced span), `TF=1` teacher-forcing validation, `PILOT=1` router-lookahead disk prefetch (experimental β see below), `CAP_RAISE=0` don't auto-grow the expert cache, `PIPE=0` disable the async expert-load pool (**default ON on Windows** since `windows-optimizations` β overlaps expert `pread` with the matmul so the CPU isn't idle waiting on the SSD; measured β18% disk service time), `RAM_GB=<n>` claim more RAM for the expert cache than the conservative auto-detect (e.g. `RAM_GB=31` on a 32 GB host raises the cache cap and hit rate measurably).++Useful knobs (env or flags): `--temp T` token sampling temperature (default 0.7 + nucleus 0.90 β tuned for int4; 0 = greedy), `--topp 0.7` adaptive expert top-p (30β40% less disk), `--ngen N` max tokens per answer (`:more` in chat continues a truncated one), `--repin N` adapt RAM/VRAM hot experts every N emitted tokens, `AUTOPIN=0` disable the learning cache's auto-pin, `THINK=1` enable GLM-5.2's reasoning block, `DRAFT=n` MTP draft depth, `GRAMMAR=g.gbnf` grammar-forced drafts for constrained JSON/NDJSON output (`GRAMMAR_DRAFT=n` caps the forced span), `TF=1` teacher-forcing validation, `PILOT=1` router-lookahead disk prefetch (experimental β see below), `URING=1` Linux-only batched expert I/O (implies `PIPE=1`; also batches `PILOT_REAL`), `PIPE=0` disable the async expert-load pool (**default ON on Windows** β overlaps expert `pread` with the matmul so the CPU isn't idle waiting on the SSD; measured β18% disk service time), `RAM_GB=<n>` claim more RAM for the expert cache than the conservative auto-detect (e.g. `RAM_GB=31` on a 32 GB host raises the cache cap and hit rate measurably), `CAP_RAISE=0` don't auto-grow the expert cache.### Resource policydiff --cc c/glm.cindex 81978a81,8c1c112c..80952d8c--- a/c/glm.c+++ b/c/glm.c@@@ -35,11 -35,9 +35,12 @@@#include <sys/resource.h>#include <sys/mman.h> /* mlock: inchioda le pagine in RAM / wire pages into RAM */#include <sys/stat.h> /* fstat per mmap degli shard (COLI_MMAP) */+ #include <signal.h> /* SIGINT = stop morbido del turno in serve mode */#endif#include "st.h"+#ifdef __linux__+#include "uring.h"+#endif#include "tok.h"#include "tier.h"#include "grammar.h" /* metodo F: draft grammaticali (#48) */@@@ -1801,9 -1654,11 +1841,12 @@@ static int uring_wait_all(UringBatch *b* condvar exist ONLY to park/wake idle workers, never for correctness. Gated* behind PIPE=1; OFF => the original blocking-load + serial-matmul path runs* byte-identically. */- static int g_pipe=0; /* PIPE=1: async expert-load pipeline (default OFF) */+ static int g_pipe=0; /* PIPE=1: async expert-load pipeline. Default ON for Windows+ * (parsed in main: getenv("PIPE")?:1 on _WIN32, :0 elsewhere).+ * Keeps expert pread off the forward-pass thread so loads overlap+ * the matmul. PIPE=0 opts back into the blocking serial path. */static int g_pipe_nw=8; /* PIPE_WORKERS=n: I/O worker threads (disk-parallel reads) */+static int g_uring=0; /* URING=1: Linux io_uring load/completion backend; implies PIPE */typedef struct {_Atomic uint64_t cur; /* (gen<<8)|index; gen main-only, index 0..njobs (β€64) */_Atomic int njobs; /* current batch job count */