Skip to content

Run frontier MoE models on hardware you already own — pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦

C

docs: document compressed CUDA expert tier

ZacharyZcR3d586d7a79d6
diff --git a/docs/ENVIRONMENT.md b/docs/ENVIRONMENT.md
index 8d06ef0a..47bda964 100644
--- a/docs/ENVIRONMENT.md
+++ b/docs/ENVIRONMENT.md
@@ -126,6 +126,9 @@ See [docs/vulkan.md](vulkan.md). On multi-core boxes also set `COLI_NO_OMP_TUNE=
| `COLI_CUDA_TC_W4A16_MIN` | `16` | Per-expert row threshold above which W4A16 TC tiles dispatch (smaller batches fall back to the naive kernel). |
| `COLI_CUDA_SHARED_W4A16` | off | `=1` uploads shared-expert weights and runs the shared-MLP W4A16 Tensor Core kernel. |
| `COLI_CUDA_SHARED_W4A16_MIN_ROWS` | `32` | Min row count to engage the shared-MLP W4A16 kernel. |
+| `CUDA_RAW_EXPERTS` | unset | Experimental ANS build only: keep this many hottest experts raw, then store subsequent VRAM experts losslessly compressed. Requires `COLI_ANS_SIDECAR`. |
+| `COLI_ANS_SIDECAR` | unset | Experimental ANS build only: path to the sequential compressed-expert sidecar. |
+| `COLI_ANS_PACK` | `0` | Experimental ANS build only: `=1` creates `COLI_ANS_SIDECAR` during pinning and exits before inference. |
| `COLI_METAL_UNTRACKED` | off (Metal only) | `=1` sets `MTLResourceHazardTrackingModeUntracked` on Metal buffers (reduces hazard-tracking overhead). |
> **Windows note.** On Windows, a bare `coli chat` / `coli run` / `coli serve`
diff --git a/docs/cuda.md b/docs/cuda.md
index 835ba971..f99e7f3f 100644
--- a/docs/cuda.md
+++ b/docs/cuda.md
@@ -66,6 +66,46 @@ dispatch (`COLI_CUDA_TC_W4A16=1`), 96-token greedy decode measured
Full experiment log: [experiments/glm52-6x5090-2026-07-12.md](experiments/glm52-6x5090-2026-07-12.md).
These are host-specific capacity results, not portable defaults.
+## Experimental lossless compressed expert tier
+
+An optional [DietGPU](https://github.com/facebookresearch/dietgpu) ANS tier keeps
+warm int4 experts compressed in VRAM and decodes the selected tensors directly
+to reusable device scratch before grouped expert matmul. It does not quantize,
+prune, substitute, or move expert weights over PCIe during decode.
+
+This path is disabled in normal builds. DietGPU must first be built as shared
+libraries, then supplied explicitly:
+
+```bash
+make CUDA=1 COLI_ANS=1 DIETGPU_ROOT=/opt/dietgpu
+```
+
+Create the sidecar once using the same model, hot-expert profile, device set,
+and placement budget intended for inference:
+
+```bash
+COLI_CUDA=1 COLI_GPUS=0,1,2,3 CUDA_EXPERT_GB=auto \
+PIN=stats.txt PIN_GB=all CUDA_RAW_EXPERTS=2500 \
+COLI_ANS_SIDECAR=/models/glm52-experts.ans COLI_ANS_PACK=1 \
+SNAP=/models/glm52-int4 ./colibri 1 4 4
+```
+
+Remove `COLI_ANS_PACK` to load the sidecar. Records are checked against the
+expected format, dimensions, raw size, codec bound, and file length. A sidecar
+is tied to its exact expert placement order; changing the model, profile,
+budgets, raw-prefix size, or device set requires repacking it.
+
+On a 6× RTX 5090 host with GLM-5.2 int4, a 2,500-expert raw tier plus 8,128
+compressed experts increased VRAM capacity from 9,335 to 10,628 experts
+(+13.9%). Greedy decode was byte-identical to the raw-tier baseline and measured
+6.19→7.12 tok/s over 32 tokens (+15.0%), and 6.75→7.31 tok/s over 128 tokens
+(+8.3%). The 110 GB sidecar took about 185 seconds to load. These results are a
+capacity trade, not a general compression guarantee: incompressible records
+are rejected, startup is currently serial, and the archived DietGPU dependency
+makes this an experimental build-time option.
+Live `REPIN` is disabled because it would invalidate the sidecar's fixed expert
+order.
+
## The GPU-resident pipeline (`COLI_CUDA_PIPE`)
`COLI_CUDA_PIPE=2` keeps the residual stream on-device across layers: rmsnorms,