Run frontier MoE models on hardware you already own β pure C, zero deps, experts streamed from disk. Tiny engine, immense model. π¦
C
docs: explain hybrid CUDA OpenMP override (#1293)
ZacharyZcRe8e4cc0d203e
diff --git a/docs/ENVIRONMENT.md b/docs/ENVIRONMENT.mdindex 5c09b518..66edc822 100644--- a/docs/ENVIRONMENT.md+++ b/docs/ENVIRONMENT.md@@ -84,7 +84,7 @@ Format: `VAR` β default β effect.| `COLI_DISKCLASS_WINDOW` | see source | Recency window (in ticks) for the DISK-CLASS heat statistic. || `URING` | `0` (off) | Linux-only queued expert I/O. `URING=1` implies `PIPE=1`, forces cold reads through io-wq (`IOSQE_ASYNC`), replaces blocking loader pthreads and spin waits with batched SQEs/CQEs, and batches `PILOT_REAL` loads on a separate ring. Use `DIRECT=1` for cold NVMe to avoid page-cache copy/readahead limits. Fails clearly if the kernel denies io_uring; incompatible with `COLI_MMAP=1`. || `DIRECT` | `0` (off) | Use `O_DIRECT`/unbuffered reads for expert slabs. **Drive-dependent β measure it on your hardware.** On real NVMe with DRAM cache and headroom it is often a large win (measured +34% decode with `PIPE=1` on a Blackwell/Windows box, and 4.25β9.69 GB/s in iobench on a GB10); on QLC/DRAM-less drives or slow/virtualised disks it can be neutral to negative. Helps sustained NVMe; keeps the zero-copy GPU path. |-| `COLI_NO_OMP_TUNE` | off | **Kill-switch** for the OpenMP hot-thread tuning (`OMP_WAIT_POLICY=active` spin + proc-bind). Set `=1` when the CPU is mostly waiting on the GPU (Metal) so spin doesn't steal the shared power budget. |+| `COLI_NO_OMP_TUNE` | off | **Kill-switch** for the OpenMP hot-thread tuning (`OMP_WAIT_POLICY=active` spin + proc-bind). Set `=1` when the CPU is mostly waiting on the GPU (Metal) so spin doesn't steal the shared power budget. Hybrid CUDA/CPU hosts may test an explicit user-owned policy only with controlled profiling; see [tuning.md](tuning.md#hybrid-cudacpu-openmp-override). || `COLI_NUMA` | auto in generated plans on multi-socket Linux; otherwise off | `COLI_NUMA=1` selectively interleaves large expert and dense slabs across NUMA nodes via `mbind` (raw syscall, no libnuma). Helps multi-socket hosts (+7β40% expert matmul); silent no-op on single-node or non-Linux. Explicit `COLI_NUMA=0` overrides the generated plan. || `MLOCK` | `-1` (auto: on for macOS) | Wire the streamed expert cache into physical RAM (`mlock`) to dodge the memory compressor. `0` off, `1` force. || `CAP` | unset | Expert-cache cap (slots/layer) when no CLI positional was given. Precedence: explicit `--cap`/positional > `CAP` > platform default > historic default (#379). Mainly for direct `./glm` use β `coli` users should prefer `--cap`. |diff --git a/docs/tuning.md b/docs/tuning.mdindex 19e6bf94..4d221ced 100644--- a/docs/tuning.md+++ b/docs/tuning.md@@ -59,6 +59,38 @@ physical-core cap when launched directly; explicit `OMP_NUM_THREADS` and the> right before the re-exec, so explicit `OMP_*` pinning works as documented.> `COLI_OMP_TUNED=1` remains the escape hatch that skips the re-exec entirely.+### Hybrid CUDA/CPU OpenMP override++The main GLM engine intentionally skips its active-wait OpenMP tuning when+`COLI_CUDA` is set. Active worker teams have measured severe regressions by+contending with CUDA dispatch and synchronization, so a CUDA model that still+routes some experts through RAM is not, by itself, a reason to change that+default.++On a specific hybrid host where profiling shows the CPU expert window on the+critical path, the supported experiment is an explicit user-owned OpenMP+policy:++```bash+OMP_WAIT_POLICY=active GOMP_SPINCOUNT=200000 KMP_BLOCKTIME=200 \+OMP_PROC_BIND=close OMP_DYNAMIC=FALSE \+COLI_CUDA=1 ./coli run --model /models/glm52_i4 "Benchmark prompt"+```++The OpenMP runtime reads these variables before `main()`, so they must be set+on the engine invocation rather than exported after startup. The engine uses+`overwrite=0`; explicit values remain authoritative. `GOMP_SPINCOUNT` applies+to libgomp and `KMP_BLOCKTIME` to Intel/LLVM OpenMP runtimes, so carrying both+keeps the command portable across common builds.++Treat this as a measured per-host override, not a recommended CUDA default.+Compare it against the unchanged command with stable page-cache state and an+interleaved run order; report CPU expert time, GPU critical time, disk wait,+and end-to-end throughput. Reject it if GPU time grows or the CPU window was+already hidden behind the GPU. Do not use active waiting on Apple Silicon: CPU+spin has measured slower there by stealing the shared CPU/GPU power budget.+`COLI_NO_OMP_TUNE=1` remains the explicit passive-policy kill switch.+```bashcoli plan --model /models/glm52_i4 --policy qualitycoli run --auto-tier --policy quality "Explain MoE offloading"