Skip to content

Run frontier MoE models on hardware you already own β€” pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦

C

docs: explain hybrid CUDA OpenMP override (#1293)

ZacharyZcRe8e4cc0d203e
diff --git a/docs/ENVIRONMENT.md b/docs/ENVIRONMENT.md
index 5c09b518..66edc822 100644
--- a/docs/ENVIRONMENT.md
+++ b/docs/ENVIRONMENT.md
@@ -84,7 +84,7 @@ Format: `VAR` β€” default β€” effect.
| `COLI_DISKCLASS_WINDOW` | see source | Recency window (in ticks) for the DISK-CLASS heat statistic. |
| `URING` | `0` (off) | Linux-only queued expert I/O. `URING=1` implies `PIPE=1`, forces cold reads through io-wq (`IOSQE_ASYNC`), replaces blocking loader pthreads and spin waits with batched SQEs/CQEs, and batches `PILOT_REAL` loads on a separate ring. Use `DIRECT=1` for cold NVMe to avoid page-cache copy/readahead limits. Fails clearly if the kernel denies io_uring; incompatible with `COLI_MMAP=1`. |
| `DIRECT` | `0` (off) | Use `O_DIRECT`/unbuffered reads for expert slabs. **Drive-dependent β€” measure it on your hardware.** On real NVMe with DRAM cache and headroom it is often a large win (measured +34% decode with `PIPE=1` on a Blackwell/Windows box, and 4.25β†’9.69 GB/s in iobench on a GB10); on QLC/DRAM-less drives or slow/virtualised disks it can be neutral to negative. Helps sustained NVMe; keeps the zero-copy GPU path. |
-| `COLI_NO_OMP_TUNE` | off | **Kill-switch** for the OpenMP hot-thread tuning (`OMP_WAIT_POLICY=active` spin + proc-bind). Set `=1` when the CPU is mostly waiting on the GPU (Metal) so spin doesn't steal the shared power budget. |
+| `COLI_NO_OMP_TUNE` | off | **Kill-switch** for the OpenMP hot-thread tuning (`OMP_WAIT_POLICY=active` spin + proc-bind). Set `=1` when the CPU is mostly waiting on the GPU (Metal) so spin doesn't steal the shared power budget. Hybrid CUDA/CPU hosts may test an explicit user-owned policy only with controlled profiling; see [tuning.md](tuning.md#hybrid-cudacpu-openmp-override). |
| `COLI_NUMA` | auto in generated plans on multi-socket Linux; otherwise off | `COLI_NUMA=1` selectively interleaves large expert and dense slabs across NUMA nodes via `mbind` (raw syscall, no libnuma). Helps multi-socket hosts (+7–40% expert matmul); silent no-op on single-node or non-Linux. Explicit `COLI_NUMA=0` overrides the generated plan. |
| `MLOCK` | `-1` (auto: on for macOS) | Wire the streamed expert cache into physical RAM (`mlock`) to dodge the memory compressor. `0` off, `1` force. |
| `CAP` | unset | Expert-cache cap (slots/layer) when no CLI positional was given. Precedence: explicit `--cap`/positional > `CAP` > platform default > historic default (#379). Mainly for direct `./glm` use β€” `coli` users should prefer `--cap`. |
diff --git a/docs/tuning.md b/docs/tuning.md
index 19e6bf94..4d221ced 100644
--- a/docs/tuning.md
+++ b/docs/tuning.md
@@ -59,6 +59,38 @@ physical-core cap when launched directly; explicit `OMP_NUM_THREADS` and the
> right before the re-exec, so explicit `OMP_*` pinning works as documented.
> `COLI_OMP_TUNED=1` remains the escape hatch that skips the re-exec entirely.
+### Hybrid CUDA/CPU OpenMP override
+
+The main GLM engine intentionally skips its active-wait OpenMP tuning when
+`COLI_CUDA` is set. Active worker teams have measured severe regressions by
+contending with CUDA dispatch and synchronization, so a CUDA model that still
+routes some experts through RAM is not, by itself, a reason to change that
+default.
+
+On a specific hybrid host where profiling shows the CPU expert window on the
+critical path, the supported experiment is an explicit user-owned OpenMP
+policy:
+
+```bash
+OMP_WAIT_POLICY=active GOMP_SPINCOUNT=200000 KMP_BLOCKTIME=200 \
+OMP_PROC_BIND=close OMP_DYNAMIC=FALSE \
+COLI_CUDA=1 ./coli run --model /models/glm52_i4 "Benchmark prompt"
+```
+
+The OpenMP runtime reads these variables before `main()`, so they must be set
+on the engine invocation rather than exported after startup. The engine uses
+`overwrite=0`; explicit values remain authoritative. `GOMP_SPINCOUNT` applies
+to libgomp and `KMP_BLOCKTIME` to Intel/LLVM OpenMP runtimes, so carrying both
+keeps the command portable across common builds.
+
+Treat this as a measured per-host override, not a recommended CUDA default.
+Compare it against the unchanged command with stable page-cache state and an
+interleaved run order; report CPU expert time, GPU critical time, disk wait,
+and end-to-end throughput. Reject it if GPU time grows or the CPU window was
+already hidden behind the GPU. Do not use active waiting on Apple Silicon: CPU
+spin has measured slower there by stealing the shared CPU/GPU power budget.
+`COLI_NO_OMP_TUNE=1` remains the explicit passive-policy kill switch.
+
```bash
coli plan --model /models/glm52_i4 --policy quality
coli run --auto-tier --policy quality "Explain MoE offloading"