Run frontier MoE models on hardware you already own β pure C, zero deps, experts streamed from disk. Tiny engine, immense model. π¦
C
docs(env): refresh ENVIRONMENT.md β document all 34 drifted env vars
woolcoxm099e0c13c4ef
diff --git a/docs/ENVIRONMENT.md b/docs/ENVIRONMENT.mdindex 4bf2d55c..535288a9 100644--- a/docs/ENVIRONMENT.md+++ b/docs/ENVIRONMENT.md@@ -2,7 +2,7 @@Reference for the environment variables read by the colibrΓ¬ engine.-**Generated from `upstream/dev @ 6d3ed7e`** by scanning every `getenv()` site in `c/glm.c`. Defaults and behavior are taken from the source; see [MAINTAINING-DOCS.md](MAINTAINING-DOCS.md) to regenerate this after the code changes.+**Generated from `dev @ d5327e2`** by scanning every `getenv()` site in `c/glm.c` and the other C sources (`c/olmoe.c`, `c/backend_cuda.cu`, `c/backend_metal.mm`). Defaults and behavior are taken from the source; see [MAINTAINING-DOCS.md](MAINTAINING-DOCS.md) to regenerate this after the code changes.## Which program reads these?@@ -43,6 +43,7 @@ Format: `VAR` β default β effect.| `URING` | `0` (off) | Linux-only queued expert I/O. `URING=1` implies `PIPE=1`, forces cold reads through io-wq (`IOSQE_ASYNC`), replaces blocking loader pthreads and spin waits with batched SQEs/CQEs, and batches `PILOT_REAL` loads on a separate ring. Use `DIRECT=1` for cold NVMe to avoid page-cache copy/readahead limits. Fails clearly if the kernel denies io_uring; incompatible with `COLI_MMAP=1`. || `DIRECT` | `0` (off) | Use `O_DIRECT`/unbuffered reads for expert slabs. Helps sustained NVMe; keeps the zero-copy GPU path. || `COLI_NO_OMP_TUNE` | off | **Kill-switch** for the OpenMP hot-thread tuning (`OMP_WAIT_POLICY=active` spin + proc-bind). Set `=1` when the CPU is mostly waiting on the GPU (Metal) so spin doesn't steal the shared power budget. |+| `COLI_NUMA` | off (Linux only) | `COLI_NUMA=1` interleaves expert slabs across NUMA nodes via `mbind` (raw syscall, no libnuma). Helps multi-socket hosts (+7β40% expert matmul); silent no-op on single-node or non-Linux. || `MLOCK` | `-1` (auto: on for macOS) | Wire the streamed expert cache into physical RAM (`mlock`) to dodge the memory compressor. `0` off, `1` force. || `CAP_RAISE` | `1` (on) | Let the engine raise the expert-cache cap above `topk` when RAM allows (bigger batches). `0` fixes the cap. || `PREFETCH` | `0` | Prefetch depth for streamed experts. |@@ -54,16 +55,26 @@ Format: `VAR` β default β effect.| `PILOT` | `0` (off) | Router-piloted cross-layer expert prefetch. || `PILOT_REAL` | `0` (off) | Value-preserving real cross-layer prefetch loads (`PILOT_REAL=1` opts in). || `PILOT_K` | `6` if `PILOT_REAL` else `8` | Number of experts the pilot prefetches per step. |+| `PILOT_TWO` | `0` (off) | Two-step shared-expert-corrected router prediction for the pilot. |+| `COUPLE` | unset | Path to a coupling-score file driving cross-layer expert prefetch (#176). When set, `couple_load` reads it. |+| `COUPLE_K` | `8` | Top-K coupled experts per layer when `COUPLE` is set. |+| `COUPLE_D` | `1` | Coupling lookahead depth (`1` or `2`) when `COUPLE` is set. || `CACHE_ROUTE` | `0` (off) | Opt-in max-rank cache-aware MoE routing (pinβͺLRU prefer within top-M). See [CACHE_ROUTE.md](CACHE_ROUTE.md). || `ROUTE_J` | `2` | Sacred top ranks always taken when `CACHE_ROUTE=1`. || `ROUTE_M` | `12` | Max-rank window for resident preference when `CACHE_ROUTE=1`. || `ROUTE_P` | `0` | Cumulative mass window for CACHE_ROUTE (`0` = fixed M). || `ROUTE_ALPHA` | `1` | Scale gate mass of substituted experts before renorm (`1` = off). || `ROUTE_AGREE` | auto | Overlap% + KL vs true top-K; auto-on when `CACHE_ROUTE=1`. |+| `ROUTE_TRACE` | unset | If set to a path, logs every routing decision there (testing/analysis). || `ABSORB` | `-1` (auto: absorbed for Sβ€4) | MLA attention absorption mode. || `IDOT` | `1` | Integer dot-product kernel. `IDOT=0` uses exact f32 kernels (for A/B numerical checks). || `COLI_POLICY` | `quality` | Resource policy: `quality`, `balanced`, or `experimental-fast`. || `PROF` | `0` (off) | Performance profile: a startup header (machine + effective config), then per run β or per turn in serve mode, on stderr β forward-latency percentiles (p50/p90/p99/max), expert-I/O totals and cache-tier fill, phase shares of wall time, and a verdict naming the knob most likely to help on this machine. Output is additive; `PROF` unset changes nothing. |+| `COLI_NO_FUSED_PAIR` | `0` (off) | `=1` disables the fused-pair matmul kernel. |+| `DISK_SPLIT` | `0` (off) | `=1` splits the reported disk-load time across the draft/absorb/forward phases in stats. |+| `I4S` | unset | Engage the int4 `IDOT` kernel only for batch `S>=<n>` (testing). |+| `SPEC_PIN` | `1` (on) | Speculation gate mode. `0` reverts to the legacy S-dependent speculation gates (#163). |+| `COLI_RAM_OVERCOMMIT` | off | `=1` overrides the "projected peak > MemAvailable β exit(2)" guard so a run that risks kernel OOM-kill is allowed to proceed. |---@@ -77,7 +88,22 @@ Format: `VAR` β default β effect.| `CUDA_EXPERT_GB` | `0` | VRAM budget (GB) for caching experts on the GPU. || `CUDA_RELEASE_HOST` | auto (`1` if >1 device) | Release host-side copies after upload. || `COLI_CUDA_ATTN` | off | Run Sβ€4 attention on the GPU. |+| `COLI_CUDA_ATTN_SHARD` | off | `=1` splits KV-b heads across devices during attention load (multi-GPU). || `COLI_CUDA_PROFILE` | off | Emit CUDA timing. |+| `COLI_CUDA_PIPE` | `0` (off) | `1` engages the multi-step attention pipeline; `2` enables the pipe2 path. |+| `COLI_CUDA_PIPE_SHARD` | off | `=1` runs the multi-device P2P head-shard attention path (opt-in for NVLink topologies; serializes ~95 MB/layer over a star PCIe topology). |+| `COLI_CUDA_PIPE_S_MIN` | `1` single-GPU, `8` multi-GPU | Minimum prefill batch S to engage the pipe2 CUDA path. |+| `COLI_CUDA_MTP` | `0` (off) | `=1` opts into MTP speculation under CUDA (off by default: cold streaming experts run on CPU where the fused-pair/IDOT kernels diverge in FP order, collapsing draft acceptance, #163/#292). |+| `COLI_CUDA_ASYNC` | on | `=0` forces synchronous `cudaMemcpy` instead of async + pinned host staging. |+| `COLI_CUDA_DUAL_PROJ` | on | `=0` issues gate+up as two separate launches instead of one fused `grouped_hidden_w4_dual`. |+| `COLI_CUDA_W4_PACKED` | on | `=0` disables the grouped packed-int4 path. |+| `COLI_CUDA_TC_INT4` | off | `=1` uses the W4A4 WMMA Tensor Core path (when all expert tensors are int4 and dims divide). |+| `COLI_CUDA_TC_MIN_ROWS` | `8` | Min rows-per-expert to engage the W4A4 Tensor Core path. |+| `COLI_CUDA_TC_W4A16` | off | `=1` uses the lossless W4A16 Tensor Core path (compute capability β₯7). |+| `COLI_CUDA_TC_W4A16_MIN` | `16` | Per-expert row threshold above which W4A16 TC tiles dispatch (smaller batches fall back to the naive kernel). |+| `COLI_CUDA_SHARED_W4A16` | off | `=1` uploads shared-expert weights and runs the shared-MLP W4A16 Tensor Core kernel. |+| `COLI_CUDA_SHARED_W4A16_MIN_ROWS` | `32` | Min row count to engage the shared-MLP W4A16 kernel. |+| `COLI_METAL_UNTRACKED` | off (Metal only) | `=1` sets `MTLResourceHazardTrackingModeUntracked` on Metal buffers (reduces hazard-tracking overhead). |---@@ -89,8 +115,11 @@ These are for testing, benchmarking, or internal use β not part of the everyda|---|---|---|| `SPEC` | `1` | Speculative decoding on/off. || `DRAFT` | `-1` (auto: 3 with MTP, else 0) | Number of speculative draft tokens per step. |-| `GRAMMAR` | unset | Path to a GBNF grammar file to constrain generation. |+| `GRAMMAR` | unset | Path to a GBNF grammar file to constrain generation. Takes precedence over `SCHEMA`. |+| `SCHEMA` | unset | Path to a JSON-Schema file compiled to GBNF to constrain generation (consulted only when `GRAMMAR` is empty). || `GRAMMAR_DRAFT` | unset | Max grammar-forced draft span length. |+| `EXPERT_BUDGET` | `0` (off) | Cap experts loaded per layer (MoE-Spec). **Quarantined:** silently forced to `0` unless `EXPERT_BUDGET_EXPERIMENTAL` is set β every tested value is either no faster or incoherent (issue #303). |+| `EXPERT_BUDGET_EXPERIMENTAL` | unset | Setting it (any value) allows `EXPERT_BUDGET>0` to actually take effect (expect garbage, #294). || `DSA` | on | Dynamic Sparse Attention indexer. `DSA=0` disables. || `DSA_FORCE` | `0` | Force the DSA path on. || `DSA_TOPK` | model value | Override the DSA index top-k (testing). |@@ -101,11 +130,15 @@ These are for testing, benchmarking, or internal use β not part of the everyda| `PIN_FILL` | `0` | Fill the pinned store even without usage data. || `MTP_DEBUG` / `MTP_PRENORM` / `MTP_SWAP` | off | MTP head debugging / ablations. || `STATS` | unset | Write an expert-usage histogram to `STATS=<file>` at end of run. |+| `TOKENS` | unset | If set, dumps generated token ids to stderr for A/B comparison. || `SCORE` | unset | Scoring/eval mode over `SCORE=<file>`. |+| `SCORE_PREFIX` | on | If unset or `β 0`, prepends `[gMASK]<sop>` to scoring contexts (GLM-family only). |+| `REPIN_VERBOSE` | off | If set, prints per-swap `[REPIN]` diagnostics during VRAM repin. || `REF` / `REF_FORCE` | `ref_glm.json` | Reference-output comparison mode. || `REPLAY` | unset | Replay mode. || `TF` | unset | Teacher-forcing mode. || `CHAT_TEMPLATE` | `1` | Apply the GLM chat template (`0` = raw prompt). |+| `PPL` | off (`olmoe.c` only) | `PPL=1` enters teacher-forced NLL/perplexity meter mode in the OLMoE sister engine. |---@@ -136,7 +169,7 @@ These are read by the Python programs (not the `glm` engine), so they don't appe- `SNAP` β model snapshot directory (required by `glm`; set from `--model`).- `SERVE`, `SERVE_BATCH` β select serve / batched-serve mode.-- `PROMPT` β one-shot text mode.+- `PROMPT` β one-shot text mode (the engine also honors `COLI_PROMPT`, preferred cross-platform; `PROMPT` is ignored on Windows if it contains cmd.exe `$`-metacharacters).- `COLI_OMP_TUNED` β internal sentinel guarding the OMP re-exec (see `COLI_NO_OMP_TUNE`); not user-facing.---