Run frontier MoE models on hardware you already own β pure C, zero deps, experts streamed from disk. Tiny engine, immense model. π¦
C
docs: vulkan ReBAR requirement, shader resolution, cross-backend benchmarking notes
Steve Markgrafca89f11ea6d4
diff --git a/docs/ENVIRONMENT.md b/docs/ENVIRONMENT.mdindex 16fb3240..49c87ca8 100644--- a/docs/ENVIRONMENT.md+++ b/docs/ENVIRONMENT.md@@ -93,7 +93,7 @@ Per-drive byte counts are reported in a `MIRROR:` stats line. Combine with `DIRE| Variable | Default | Effect ||---|---|---|| `COLI_VULKAN` | off | Enable the Vulkan backend. Requires a `make VK=1` build; fails at startup (no silent fallback) if libvulkan or the compiled shaders are missing. |-| `COLI_VK_SHADERS` | `shaders/qmatmul.spv` | Path to the compiled main shader; the gate_up and attention shaders are found next to it. |+| `COLI_VK_SHADERS` | auto | Path to the compiled `qmatmul.spv` **or** the directory holding the `.spv` set; the other shaders are found next to it. Unset: `shaders/` next to the binary, then CWD-relative `shaders/qmatmul.spv`. || `COLI_VK_EXPERTS` | `320` | Pinned VRAM expert tier size: top-N experts by `.coli_usage` heat uploaded once at startup and served from VRAM with no RAM slot or disk read. `0` disables the tier (experts stay on the CPU path). ~19 MB VRAM per int4 expert. || `COLI_VK_DENSE` | `0` | Run the resident dense matmuls (attention projections, shared expert) on the GPU. || `COLI_VK_ATTN` | `0` | Run the Sβ€4 MLA absorb attention core (+ fused o-projection) on the GPU, with a persistent device-side KV mirror. |diff --git a/docs/vulkan.md b/docs/vulkan.mdindex 01d1f52f..67c872f1 100644--- a/docs/vulkan.md+++ b/docs/vulkan.md@@ -26,6 +26,20 @@ Set `COLI_NO_OMP_TUNE=1` on multi-core boxes: the engine's OMP self-tuneVulkan, and spinning worker threads starve the async I/O pool (measuredCPU expert bandwidth 28 β 5 GB/s without it).+**Discrete cards need Resizable BAR.** The weight tiers allocate+HOST_VISIBLE|DEVICE_LOCAL memory; with ReBAR disabled that combination only+exists in a ~256 MB BAR window, and the driver silently places everything+beyond it in system RAM β the tier then *reports* resident experts while every+access crosses PCIe, slower than the CPU path (measured 0.11 vs 0.24 tok/s+either side of the BIOS toggle on an RX 9070 XT). The engine now warns at init+when the host-visible slice of VRAM is small; if you see that warning, enable+Resizable BAR / Smart Access Memory in the BIOS. Unified-memory APUs are+unaffected.++The compiled shaders are found via `COLI_VK_SHADERS` (either the+`qmatmul.spv` file or the directory holding the `.spv` set); unset, the+engine looks in `shaders/` next to the binary, then relative to the CWD.+## What runs on the GPU| Piece | Env | Mechanism |@@ -65,6 +79,27 @@ The two write-combined-memory rules that make this possible: buffers the CPUreads back must be HOST_CACHED (ReBAR VRAM reads at ~40 MB/s otherwise), andeverything else lives HOST_VISIBLE|DEVICE_LOCAL.+## Benchmarking against other backends++Two defaults will silently skew any Vulkan-vs-CUDA/HIP comparison:++- **MTP speculation**: CUDA/HIP builds disable model drafts by default+ (`DRAFT` auto-resolves to 0 under `COLI_CUDA=1`, see #163), while CPU and+ Vulkan runs keep `DRAFT=3`. The arms then execute different decode loops β+ the speculative arm routes ~2Γ the expert positions per emitted token+ (rejected draft positions still pay their expert I/O), which dominates on+ storage-bound boxes. Output is identical either way (greedy verify is+ lossless), so nothing looks wrong. Pin `DRAFT=0` (or `DRAFT=3+ COLI_CUDA_MTP=1`) explicitly on **both** arms.+- **GPU clocks**: decode dispatches are microsecond bursts that never ramp+ DPM on their own; the memory clock can sit parked through an entire run.+ Pin `power_dpm_force_performance_level=high` (both arms) or disclose it.++Also note `experts loaded/token` in the run stats counts *routed positions*+(including rejected speculative ones) before any cache/tier is consulted β+it does not fall when the VK tier serves a hit; the `vk` bucket in the+hit-rate line is the tier-effectiveness number.+## Limits and future work- Decode-focused: the expert tier and attention core serve `S<=4`; prefill