Run frontier MoE models on hardware you already own — pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦
C
docs: Vulkan backend (vulkan.md, ENVIRONMENT.md table, README pointers)
Steve Markgraf1b0a89cb3fd1
diff --git a/README.md b/README.mdindex 713a8fe8..ac59876d 100644--- a/README.md+++ b/README.md@@ -213,7 +213,10 @@ layer's experts — routing is measurably **71.6% predictable one layer ahead**.On GPUs, the resident pipeline (`COLI_CUDA_PIPE=2`) keeps the residual streamon-device across layers so the CPU expert loop runs uninterrupted; on AppleSilicon an experimental [Metal backend](docs/metal.md) does the batched expert-math on the unified-memory GPU.+math on the unified-memory GPU; and a [Vulkan backend](docs/vulkan.md) brings+the expert tier, dense projections, and the MLA attention core to any GPU with+a Vulkan 1.2 driver — including AMD cards via Mesa/RADV (measured faster than+ROCm on RDNA4) and cards the vendor stacks no longer support.> **On real NVMe, measure `DIRECT=1`.** O_DIRECT bypasses the page cache and is> often a large win on drives with DRAM cache and bandwidth headroom (+34%@@ -369,6 +372,7 @@ and the optional API gateway.| Tuning knobs, policies, the learning cache, prefetch | [docs/tuning.md](docs/tuning.md) || Windows 11 native build (+ CUDA DLL) | [docs/windows.md](docs/windows.md) || CUDA backend, VRAM expert tier, full residency | [docs/cuda.md](docs/cuda.md) |+| Vulkan backend (any GPU: AMD via RADV, incl. cards ROCm dropped) | [docs/vulkan.md](docs/vulkan.md) || Apple Silicon Metal backend | [docs/metal.md](docs/metal.md) || OpenAI-compatible API, KV slots, web dashboard | [docs/api.md](docs/api.md) || Grammar-forced drafts (structured output) | [docs/grammar-draft.md](docs/grammar-draft.md) |diff --git a/docs/ENVIRONMENT.md b/docs/ENVIRONMENT.mdindex e9f8069e..9e97d303 100644--- a/docs/ENVIRONMENT.md+++ b/docs/ENVIRONMENT.md@@ -88,6 +88,18 @@ Format: `VAR` — default — effect.Per-drive byte counts are reported in a `MIRROR:` stats line. Combine with `DIRECT=1` so the two copies never compete for page cache.+## Vulkan (any GPU with a Vulkan 1.2 driver)++| Variable | Default | Effect |+|---|---|---|+| `COLI_VULKAN` | off | Enable the Vulkan backend. Requires a `make VK=1` build; fails at startup (no silent fallback) if libvulkan or the compiled shaders are missing. |+| `COLI_VK_SHADERS` | `shaders/qmatmul.spv` | Path to the compiled main shader; the gate_up and attention shaders are found next to it. |+| `COLI_VK_EXPERTS` | `320` | Pinned VRAM expert tier size: top-N experts by `.coli_usage` heat uploaded once at startup and served from VRAM with no RAM slot or disk read. `0` disables the tier (experts stay on the CPU path). ~19 MB VRAM per int4 expert. |+| `COLI_VK_DENSE` | `0` | Run the resident dense matmuls (attention projections, shared expert) on the GPU. |+| `COLI_VK_ATTN` | `0` | Run the S≤4 MLA absorb attention core (+ fused o-projection) on the GPU, with a persistent device-side KV mirror. |++See [docs/vulkan.md](vulkan.md). On multi-core boxes also set `COLI_NO_OMP_TUNE=1` (see that doc for why).+## CUDA (NVIDIA)| Variable | Default | Effect |diff --git a/docs/vulkan.md b/docs/vulkan.mdnew file mode 100644index 00000000..01d1f52f--- /dev/null+++ b/docs/vulkan.md@@ -0,0 +1,76 @@+# Vulkan backend (any GPU with a Vulkan 1.2 driver)++colibrì includes an opt-in Vulkan compute backend that runs the whole GLM+decode compute path on any GPU a Vulkan driver can see — no CUDA, no ROCm.+That includes cards the vendor stacks have dropped (ROCm 7 removed Polaris:+an RX 580 runs here via RADV) and, measured on an RX 9070 (RDNA4), it is+*faster* than the ROCm/HIP backend on the same card.++```bash+cd c+make glm VK=1 # needs libvulkan + glslc (shaderc) for the shaders+COLI_VULKAN=1 COLI_VK_DENSE=1 COLI_VK_ATTN=1 \+PIN=<model>/.coli_usage PIN_GB=0 COLI_NO_OMP_TUNE=1 \+./coli run "Hello" --topp 0.7+```++Requirements: `libvulkan` and a Vulkan **1.2** ICD with+`GL_KHR_shader_subgroup_arithmetic` (any Mesa RADV, AMDVLK, NVIDIA or Intel+ANV driver from the last several years), plus `glslc` at build time. The+backend picks the most capable physical device (discrete > integrated) and+degrades to the CPU path on any failure — a wedged GPU can slow a run, never+corrupt it.++Set `COLI_NO_OMP_TUNE=1` on multi-core boxes: the engine's OMP self-tune+(active spin-wait) is skipped under `COLI_CUDA`/`COLI_METAL` but not under+Vulkan, and spinning worker threads starve the async I/O pool (measured+CPU expert bandwidth 28 → 5 GB/s without it).++## What runs on the GPU++| Piece | Env | Mechanism |+|---|---|---|+| Routed experts (hot set) | `COLI_VK_EXPERTS=N` (default 320) | Top-N experts by `.coli_usage` heat uploaded **once at startup** into a VRAM registry; at decode they are served from VRAM with **no RAM slot, no disk read, no prefetch**, as one async fused batch (gate+up+silu→down, hidden on-device) overlapped with the CPU computing the remaining experts. Shown as the `vk` bucket in the hit-rate line. |+| Dense projections | `COLI_VK_DENSE=1` | q_a+kv_a fused into one submit, q_b, o; shared expert as a single fused expert-group submit. Resident int4/int8 weights upload once. |+| MLA attention core | `COLI_VK_ATTN=1` | One dispatch per layer: absorbed query, scores over the KV window, softmax, weighted latent, value rows, **fused with the o-projection** (the context vector never leaves the GPU). The latent/rope KV lives in a persistent per-layer device mirror, appended ~2.3 KB/token/layer with the same invalidation points as the CUDA KV shadow. |++The `PIN_GB=0` (with `PIN` still set) in the example is deliberate: the VRAM+registry holds the same hot experts a RAM pin would, so the pin's RAM is+better spent on the adaptive LRU cache. Keep `PIN` set so AUTOPIN does not+re-pin from history.++## Correctness++- `gcc -O3 -DVK_TEST backend_vulkan.c -o test_vk -lvulkan -lm && ./test_vk+ shaders/qmatmul.spv` runs a CPU-reference exactness harness over every+ primitive (GEMV int4/int8 across shapes incl. the long-row o-projection,+ fused gate+up, the full expert group sync and async, the matmul pair, and+ the absorb attention core incl. causal S=2, kv_start windows, int8, and+ long-context cases). Typical maxrel ~1e-5..2e-3 (fp32 reduction order).+- Engine-level: greedy decode with the full stack matches the pure-CPU+ engine token-for-token on the validation prompt.+- int4 weights decode as offset-binary (nibble−8), byte-identical layout to+ the CPU path — no repacking.++## Measured performance (AMD RX 9070, RDNA4, RADV/Mesa 26.1)++Expert-MLP primitive (K experts, int4 6144→2048→6144, per-call incl. readback):+Vulkan **0.11–0.13 ms/expert** vs the production ROCm/HIP expert group+**0.179 ms/expert** — ~35% faster. The decode MLA attention core runs 3.7×+faster than the HIP kernel on the same card. End-to-end GLM-5.2 (744B int4,+NVMe-streamed) decode on a 12-core Zen2 + RX 9070 box: Vulkan+**1.7–1.8 tok/s** (64-token) / **1.6** (256-token) / **1.58 sustained**+(512-token) vs the HIP backend at 1.5–1.55 on identical settings.+The two write-combined-memory rules that make this possible: buffers the CPU+reads back must be HOST_CACHED (ReBAR VRAM reads at ~40 MB/s otherwise), and+everything else lives HOST_VISIBLE|DEVICE_LOCAL.++## Limits and future work++- Decode-focused: the expert tier and attention core serve `S<=4`; prefill+ uses the CPU/batched paths (dense projections do run on VK at prefill).+- DSA top-k selection, ragged multi-slot serving, and quantized-KV caches+ fall back to the CPU attention path.+- Not yet done: cooperative-matrix (coopmat) prefill kernels, a fully+ resident-layer pipeline, Polaris/gfx803 validation on real hardware (the+ shaders use dynamic subgroup sizes and are wave64-safe by construction).