Skip to content

Run frontier MoE models on hardware you already own β€” pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦

C

docs: vulkan ReBAR requirement, shader resolution, cross-backend benchmarking notes

Steve Markgrafca89f11ea6d4
diff --git a/docs/ENVIRONMENT.md b/docs/ENVIRONMENT.md
index 16fb3240..49c87ca8 100644
--- a/docs/ENVIRONMENT.md
+++ b/docs/ENVIRONMENT.md
@@ -93,7 +93,7 @@ Per-drive byte counts are reported in a `MIRROR:` stats line. Combine with `DIRE
| Variable | Default | Effect |
|---|---|---|
| `COLI_VULKAN` | off | Enable the Vulkan backend. Requires a `make VK=1` build; fails at startup (no silent fallback) if libvulkan or the compiled shaders are missing. |
-| `COLI_VK_SHADERS` | `shaders/qmatmul.spv` | Path to the compiled main shader; the gate_up and attention shaders are found next to it. |
+| `COLI_VK_SHADERS` | auto | Path to the compiled `qmatmul.spv` **or** the directory holding the `.spv` set; the other shaders are found next to it. Unset: `shaders/` next to the binary, then CWD-relative `shaders/qmatmul.spv`. |
| `COLI_VK_EXPERTS` | `320` | Pinned VRAM expert tier size: top-N experts by `.coli_usage` heat uploaded once at startup and served from VRAM with no RAM slot or disk read. `0` disables the tier (experts stay on the CPU path). ~19 MB VRAM per int4 expert. |
| `COLI_VK_DENSE` | `0` | Run the resident dense matmuls (attention projections, shared expert) on the GPU. |
| `COLI_VK_ATTN` | `0` | Run the S≀4 MLA absorb attention core (+ fused o-projection) on the GPU, with a persistent device-side KV mirror. |
diff --git a/docs/vulkan.md b/docs/vulkan.md
index 01d1f52f..67c872f1 100644
--- a/docs/vulkan.md
+++ b/docs/vulkan.md
@@ -26,6 +26,20 @@ Set `COLI_NO_OMP_TUNE=1` on multi-core boxes: the engine's OMP self-tune
Vulkan, and spinning worker threads starve the async I/O pool (measured
CPU expert bandwidth 28 β†’ 5 GB/s without it).
+**Discrete cards need Resizable BAR.** The weight tiers allocate
+HOST_VISIBLE|DEVICE_LOCAL memory; with ReBAR disabled that combination only
+exists in a ~256 MB BAR window, and the driver silently places everything
+beyond it in system RAM β€” the tier then *reports* resident experts while every
+access crosses PCIe, slower than the CPU path (measured 0.11 vs 0.24 tok/s
+either side of the BIOS toggle on an RX 9070 XT). The engine now warns at init
+when the host-visible slice of VRAM is small; if you see that warning, enable
+Resizable BAR / Smart Access Memory in the BIOS. Unified-memory APUs are
+unaffected.
+
+The compiled shaders are found via `COLI_VK_SHADERS` (either the
+`qmatmul.spv` file or the directory holding the `.spv` set); unset, the
+engine looks in `shaders/` next to the binary, then relative to the CWD.
+
## What runs on the GPU
| Piece | Env | Mechanism |
@@ -65,6 +79,27 @@ The two write-combined-memory rules that make this possible: buffers the CPU
reads back must be HOST_CACHED (ReBAR VRAM reads at ~40 MB/s otherwise), and
everything else lives HOST_VISIBLE|DEVICE_LOCAL.
+## Benchmarking against other backends
+
+Two defaults will silently skew any Vulkan-vs-CUDA/HIP comparison:
+
+- **MTP speculation**: CUDA/HIP builds disable model drafts by default
+ (`DRAFT` auto-resolves to 0 under `COLI_CUDA=1`, see #163), while CPU and
+ Vulkan runs keep `DRAFT=3`. The arms then execute different decode loops β€”
+ the speculative arm routes ~2Γ— the expert positions per emitted token
+ (rejected draft positions still pay their expert I/O), which dominates on
+ storage-bound boxes. Output is identical either way (greedy verify is
+ lossless), so nothing looks wrong. Pin `DRAFT=0` (or `DRAFT=3
+ COLI_CUDA_MTP=1`) explicitly on **both** arms.
+- **GPU clocks**: decode dispatches are microsecond bursts that never ramp
+ DPM on their own; the memory clock can sit parked through an entire run.
+ Pin `power_dpm_force_performance_level=high` (both arms) or disclose it.
+
+Also note `experts loaded/token` in the run stats counts *routed positions*
+(including rejected speculative ones) before any cache/tier is consulted β€”
+it does not fall when the VK tier serves a hit; the `vk` bucket in the
+hit-rate line is the tier-effectiveness number.
+
## Limits and future work
- Decode-focused: the expert tier and attention core serve `S<=4`; prefill