Run frontier MoE models on hardware you already own β pure C, zero deps, experts streamed from disk. Tiny engine, immense model. π¦
C
docs(env): document COLI_PREFILL_CHUNK, and what CUDA_DENSE is worth
ThefloorMinera25539ea9266
diff --git a/docs/ENVIRONMENT.md b/docs/ENVIRONMENT.mdindex 8e87e1d8..1d1484f2 100644--- a/docs/ENVIRONMENT.md+++ b/docs/ENVIRONMENT.md@@ -18,6 +18,7 @@ Format: `VAR` β default β effect.|---|---|---|| `RAM_GB` | `0` (auto β 88% of free RAM) | RAM budget in GB for the resident/streamed expert working set. Higher β more experts stay hot β higher cache hit rate. || `CTX` | `4096` | Maximum context length (tokens) the KV cache is sized for. |+| `COLI_PREFILL_CHUNK` | `0` (off) | Run a long prompt through the layers in N-token slices instead of one pass. Every S-scaled activation buffer shrinks from prompt-sized to chunk-sized, which is the remedy when a long prompt exhausts CUDA scratch. Byte-identical output (verified at N=256). Skipped under an active MTP draft. **Cost:** a slice of 512 tokens already routes to essentially every expert of every layer (`P(miss) = (1-topk/n_experts)^N`), so each slice re-reads the whole non-resident expert set -- prefer the largest N that still fits your scratch. || `NGEN` | `256` (engine) | Max tokens to generate before stopping (stop tokens can end sooner). `coli --ngen` defaults to `1024`. || `COLI_TEMP` | `-1` (auto: `1.0` for chat/text, greedy elsewhere) | Sampling temperature. **`COLI_TEMP=0` = greedy/argmax = deterministic.** `TEMP` still works as a deprecated alias, but only if fully numeric: `$TEMP` is the temp-*directory* path on Windows and for the ROCm runtime (#509), so prefer `COLI_TEMP`. || `NUCLEUS` | `0.90` | Nucleus (top-p) mass kept when sampling. Slightly tighter than the official 0.95 because the int4 tail is noisy. |@@ -154,7 +155,7 @@ See [docs/vulkan.md](vulkan.md). On multi-core boxes also set `COLI_NO_OMP_TUNE=|---|---|---|| `COLI_CUDA` | off | Enable the CUDA backend. Requires a CUDA build. An explicit `COLI_CUDA=0` disables it **and suppresses the Windows bare-run auto-enable** (before this, Windows "CPU" runs with `COLI_CUDA=0` silently got a VRAM expert tier). The CLI flag `--gpu none` is the canonical hard off-switch on every platform. || `COLI_GPU` / `COLI_GPUS` | unset | Device selection (`auto`, `none`, or a list like `0,1`). Requires `COLI_CUDA=1`. |-| `CUDA_DENSE` | `0` | Place dense (non-expert) matmuls on the GPU. |+| `CUDA_DENSE` | `0` | Place dense (non-expert) matmuls on the GPU. Off by default the engine reports `routed experts only (resident dense on CPU)`: on a host where the CPU is the limiter this leaves the dense path of every layer on the CPU while the VRAM tier serves experts only. Measured x2.8 on a 4x A6000 / 24-core host (1.53 -> 4.26 tok/s). || `CUDA_EXPERT_GB` | `0` | VRAM budget (GB) for caching experts on the GPU. || `CUDA_RELEASE_HOST` | auto (`1` if >1 device) | Release host-side copies after upload. || `COLI_CUDA_ATTN` | off | Run Sβ€4 attention on the GPU. |