Run frontier MoE models on hardware you already own — pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦
C
docs: document compressed CUDA expert tier
ZacharyZcR3d586d7a79d6
diff --git a/docs/ENVIRONMENT.md b/docs/ENVIRONMENT.mdindex 8d06ef0a..47bda964 100644--- a/docs/ENVIRONMENT.md+++ b/docs/ENVIRONMENT.md@@ -126,6 +126,9 @@ See [docs/vulkan.md](vulkan.md). On multi-core boxes also set `COLI_NO_OMP_TUNE=| `COLI_CUDA_TC_W4A16_MIN` | `16` | Per-expert row threshold above which W4A16 TC tiles dispatch (smaller batches fall back to the naive kernel). || `COLI_CUDA_SHARED_W4A16` | off | `=1` uploads shared-expert weights and runs the shared-MLP W4A16 Tensor Core kernel. || `COLI_CUDA_SHARED_W4A16_MIN_ROWS` | `32` | Min row count to engage the shared-MLP W4A16 kernel. |+| `CUDA_RAW_EXPERTS` | unset | Experimental ANS build only: keep this many hottest experts raw, then store subsequent VRAM experts losslessly compressed. Requires `COLI_ANS_SIDECAR`. |+| `COLI_ANS_SIDECAR` | unset | Experimental ANS build only: path to the sequential compressed-expert sidecar. |+| `COLI_ANS_PACK` | `0` | Experimental ANS build only: `=1` creates `COLI_ANS_SIDECAR` during pinning and exits before inference. || `COLI_METAL_UNTRACKED` | off (Metal only) | `=1` sets `MTLResourceHazardTrackingModeUntracked` on Metal buffers (reduces hazard-tracking overhead). |> **Windows note.** On Windows, a bare `coli chat` / `coli run` / `coli serve`diff --git a/docs/cuda.md b/docs/cuda.mdindex 835ba971..f99e7f3f 100644--- a/docs/cuda.md+++ b/docs/cuda.md@@ -66,6 +66,46 @@ dispatch (`COLI_CUDA_TC_W4A16=1`), 96-token greedy decode measuredFull experiment log: [experiments/glm52-6x5090-2026-07-12.md](experiments/glm52-6x5090-2026-07-12.md).These are host-specific capacity results, not portable defaults.+## Experimental lossless compressed expert tier++An optional [DietGPU](https://github.com/facebookresearch/dietgpu) ANS tier keeps+warm int4 experts compressed in VRAM and decodes the selected tensors directly+to reusable device scratch before grouped expert matmul. It does not quantize,+prune, substitute, or move expert weights over PCIe during decode.++This path is disabled in normal builds. DietGPU must first be built as shared+libraries, then supplied explicitly:++```bash+make CUDA=1 COLI_ANS=1 DIETGPU_ROOT=/opt/dietgpu+```++Create the sidecar once using the same model, hot-expert profile, device set,+and placement budget intended for inference:++```bash+COLI_CUDA=1 COLI_GPUS=0,1,2,3 CUDA_EXPERT_GB=auto \+PIN=stats.txt PIN_GB=all CUDA_RAW_EXPERTS=2500 \+COLI_ANS_SIDECAR=/models/glm52-experts.ans COLI_ANS_PACK=1 \+SNAP=/models/glm52-int4 ./colibri 1 4 4+```++Remove `COLI_ANS_PACK` to load the sidecar. Records are checked against the+expected format, dimensions, raw size, codec bound, and file length. A sidecar+is tied to its exact expert placement order; changing the model, profile,+budgets, raw-prefix size, or device set requires repacking it.++On a 6× RTX 5090 host with GLM-5.2 int4, a 2,500-expert raw tier plus 8,128+compressed experts increased VRAM capacity from 9,335 to 10,628 experts+(+13.9%). Greedy decode was byte-identical to the raw-tier baseline and measured+6.19→7.12 tok/s over 32 tokens (+15.0%), and 6.75→7.31 tok/s over 128 tokens+(+8.3%). The 110 GB sidecar took about 185 seconds to load. These results are a+capacity trade, not a general compression guarantee: incompressible records+are rejected, startup is currently serial, and the archived DietGPU dependency+makes this an experimental build-time option.+Live `REPIN` is disabled because it would invalidate the sidecar's fixed expert+order.+## The GPU-resident pipeline (`COLI_CUDA_PIPE`)`COLI_CUDA_PIPE=2` keeps the residual stream on-device across layers: rmsnorms,