Run frontier MoE models on hardware you already own β pure C, zero deps, experts streamed from disk. Tiny engine, immense model. π¦
C
fix(coli): honour explicit COLI_CUDA=0 over the Windows bare-run auto-enable
Mohamed Mastourif9391914044a
diff --git a/README.md b/README.mdindex aaf74637..02d77177 100644--- a/README.md+++ b/README.md@@ -148,6 +148,13 @@ on-device across layers so the CPU expert loop runs uninterrupted; on AppleSilicon an experimental [Metal backend](docs/metal.md) does the batched expertmath on the unified-memory GPU.+> **On real NVMe, measure `DIRECT=1`.** O_DIRECT bypasses the page cache and is+> often a large win on drives with DRAM cache and bandwidth headroom (+34%+> decode measured with `PIPE=1` on a Blackwell/Windows box; 4.25β9.69 GB/s in+> iobench on a GB10) β but it is drive-dependent: QLC/DRAM-less or virtualised+> disks can be neutral to negative. Try it first; keep what your hardware+> rewards.+### Faithful model, compressed stateThe forward pass is validated **token-exact against a `transformers` oracle**diff --git a/c/coli b/c/coliindex 5cef2fc9..133d21e6 100755--- a/c/coli+++ b/c/coli@@ -248,7 +248,14 @@ def env_for(a):# has working detection + the explicit-flag UX, and changing bare-chat# semantics there is out of scope. Falls back to CPU with a warning if# nvidia-smi is missing (discover_gpus can't size VRAM without it).- if (sys.platform == "win32" and a.gpu is None and not a.vram):+ # An explicit COLI_CUDA=0 in the environment must win over the implicit+ # auto-enable: before this check, a Windows user setting COLI_CUDA=0 for+ # a CPU baseline silently got a ~12.6 GB VRAM expert tier anyway (the+ # engine's "CPU" rows were GPU-assisted). --gpu none remains the+ # canonical hard off-switch (works on every platform, also clears the+ # CUDA_* sizing vars).+ if (sys.platform == "win32" and a.gpu is None and not a.vram+ and e.get("COLI_CUDA") != "0"):if cuda_binary():from resource_plan import discover_gpus, build_plan, environment_for_plan, format_bytesgpus = discover_gpus()@@ -277,6 +284,11 @@ def env_for(a):f"(cannot size VRAM); running CPU-only. Add nvidia-smi to PATH or pass "f"--vram N to enable CUDA.{C.r}", file=sys.stderr)# else: CPU build (no coli_cuda.dll) β stay silent, CPU is correct.+ elif e.get("COLI_CUDA") == "0":+ # honoured off-switch: also drop stale device/sizing vars so the+ # engine can't be re-enabled by leftovers (same as --gpu none).+ e.pop("COLI_GPU",None); e.pop("COLI_GPUS",None)+ e.pop("CUDA_EXPERT_GB",None); e.pop("CUDA_DENSE",None)# --gpu/--vram SENZA --auto-tier: prima venivano ignorati in silenzio e il run# partiva CPU-only senza alcun avviso β benchmark "GPU" pubblicati per errore (#121).if a.gpu is not None:diff --git a/docs/ENVIRONMENT.md b/docs/ENVIRONMENT.mdindex ce44fcea..d7abcc13 100644--- a/docs/ENVIRONMENT.md+++ b/docs/ENVIRONMENT.md@@ -41,7 +41,7 @@ Format: `VAR` β default β effect.| `PIPE` | `0` (off) | Overlap expert disk-load with matmul via I/O worker threads. Byte-identical output; reorders I/O. `PIPE=1` opts in. || `PIPE_WORKERS` | `8` | Number of pthread loaders when `PIPE=1`, or the io-wq worker maximum per ring when `URING=1` (capped at 64). Tune to SSD queue depth and available cores. || `URING` | `0` (off) | Linux-only queued expert I/O. `URING=1` implies `PIPE=1`, forces cold reads through io-wq (`IOSQE_ASYNC`), replaces blocking loader pthreads and spin waits with batched SQEs/CQEs, and batches `PILOT_REAL` loads on a separate ring. Use `DIRECT=1` for cold NVMe to avoid page-cache copy/readahead limits. Fails clearly if the kernel denies io_uring; incompatible with `COLI_MMAP=1`. |-| `DIRECT` | `0` (off) | Use `O_DIRECT`/unbuffered reads for expert slabs. Helps sustained NVMe; keeps the zero-copy GPU path. |+| `DIRECT` | `0` (off) | Use `O_DIRECT`/unbuffered reads for expert slabs. **Drive-dependent β measure it on your hardware.** On real NVMe with DRAM cache and headroom it is often a large win (measured +34% decode with `PIPE=1` on a Blackwell/Windows box, and 4.25β9.69 GB/s in iobench on a GB10); on QLC/DRAM-less drives or slow/virtualised disks it can be neutral to negative. Helps sustained NVMe; keeps the zero-copy GPU path. || `COLI_NO_OMP_TUNE` | off | **Kill-switch** for the OpenMP hot-thread tuning (`OMP_WAIT_POLICY=active` spin + proc-bind). Set `=1` when the CPU is mostly waiting on the GPU (Metal) so spin doesn't steal the shared power budget. || `COLI_NUMA` | auto in generated plans on multi-socket Linux; otherwise off | `COLI_NUMA=1` selectively interleaves large expert and dense slabs across NUMA nodes via `mbind` (raw syscall, no libnuma). Helps multi-socket hosts (+7β40% expert matmul); silent no-op on single-node or non-Linux. Explicit `COLI_NUMA=0` overrides the generated plan. || `MLOCK` | `-1` (auto: on for macOS) | Wire the streamed expert cache into physical RAM (`mlock`) to dodge the memory compressor. `0` off, `1` force. |@@ -91,7 +91,7 @@ Per-drive byte counts are reported in a `MIRROR:` stats line. Combine with `DIRE| Variable | Default | Effect ||---|---|---|-| `COLI_CUDA` | off | Enable the CUDA backend. Requires a CUDA build. |+| `COLI_CUDA` | off | Enable the CUDA backend. Requires a CUDA build. An explicit `COLI_CUDA=0` disables it **and suppresses the Windows bare-run auto-enable** (before this, Windows "CPU" runs with `COLI_CUDA=0` silently got a VRAM expert tier). The CLI flag `--gpu none` is the canonical hard off-switch on every platform. || `COLI_GPU` / `COLI_GPUS` | unset | Device selection (`auto`, `none`, or a list like `0,1`). Requires `COLI_CUDA=1`. || `CUDA_DENSE` | `0` | Place dense (non-expert) matmuls on the GPU. || `CUDA_EXPERT_GB` | `0` | VRAM budget (GB) for caching experts on the GPU. |