Skip to content

Run frontier MoE models on hardware you already own — pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦

C

docs: Vulkan backend (vulkan.md, ENVIRONMENT.md table, README pointers)

Steve Markgraf1b0a89cb3fd1
diff --git a/README.md b/README.md
index 713a8fe8..ac59876d 100644
--- a/README.md
+++ b/README.md
@@ -213,7 +213,10 @@ layer's experts — routing is measurably **71.6% predictable one layer ahead**.
On GPUs, the resident pipeline (`COLI_CUDA_PIPE=2`) keeps the residual stream
on-device across layers so the CPU expert loop runs uninterrupted; on Apple
Silicon an experimental [Metal backend](docs/metal.md) does the batched expert
-math on the unified-memory GPU.
+math on the unified-memory GPU; and a [Vulkan backend](docs/vulkan.md) brings
+the expert tier, dense projections, and the MLA attention core to any GPU with
+a Vulkan 1.2 driver — including AMD cards via Mesa/RADV (measured faster than
+ROCm on RDNA4) and cards the vendor stacks no longer support.
> **On real NVMe, measure `DIRECT=1`.** O_DIRECT bypasses the page cache and is
> often a large win on drives with DRAM cache and bandwidth headroom (+34%
@@ -369,6 +372,7 @@ and the optional API gateway.
| Tuning knobs, policies, the learning cache, prefetch | [docs/tuning.md](docs/tuning.md) |
| Windows 11 native build (+ CUDA DLL) | [docs/windows.md](docs/windows.md) |
| CUDA backend, VRAM expert tier, full residency | [docs/cuda.md](docs/cuda.md) |
+| Vulkan backend (any GPU: AMD via RADV, incl. cards ROCm dropped) | [docs/vulkan.md](docs/vulkan.md) |
| Apple Silicon Metal backend | [docs/metal.md](docs/metal.md) |
| OpenAI-compatible API, KV slots, web dashboard | [docs/api.md](docs/api.md) |
| Grammar-forced drafts (structured output) | [docs/grammar-draft.md](docs/grammar-draft.md) |
diff --git a/docs/ENVIRONMENT.md b/docs/ENVIRONMENT.md
index e9f8069e..9e97d303 100644
--- a/docs/ENVIRONMENT.md
+++ b/docs/ENVIRONMENT.md
@@ -88,6 +88,18 @@ Format: `VAR` — default — effect.
Per-drive byte counts are reported in a `MIRROR:` stats line. Combine with `DIRECT=1` so the two copies never compete for page cache.
+## Vulkan (any GPU with a Vulkan 1.2 driver)
+
+| Variable | Default | Effect |
+|---|---|---|
+| `COLI_VULKAN` | off | Enable the Vulkan backend. Requires a `make VK=1` build; fails at startup (no silent fallback) if libvulkan or the compiled shaders are missing. |
+| `COLI_VK_SHADERS` | `shaders/qmatmul.spv` | Path to the compiled main shader; the gate_up and attention shaders are found next to it. |
+| `COLI_VK_EXPERTS` | `320` | Pinned VRAM expert tier size: top-N experts by `.coli_usage` heat uploaded once at startup and served from VRAM with no RAM slot or disk read. `0` disables the tier (experts stay on the CPU path). ~19 MB VRAM per int4 expert. |
+| `COLI_VK_DENSE` | `0` | Run the resident dense matmuls (attention projections, shared expert) on the GPU. |
+| `COLI_VK_ATTN` | `0` | Run the S≤4 MLA absorb attention core (+ fused o-projection) on the GPU, with a persistent device-side KV mirror. |
+
+See [docs/vulkan.md](vulkan.md). On multi-core boxes also set `COLI_NO_OMP_TUNE=1` (see that doc for why).
+
## CUDA (NVIDIA)
| Variable | Default | Effect |
diff --git a/docs/vulkan.md b/docs/vulkan.md
new file mode 100644
index 00000000..01d1f52f
--- /dev/null
+++ b/docs/vulkan.md
@@ -0,0 +1,76 @@
+# Vulkan backend (any GPU with a Vulkan 1.2 driver)
+
+colibrì includes an opt-in Vulkan compute backend that runs the whole GLM
+decode compute path on any GPU a Vulkan driver can see — no CUDA, no ROCm.
+That includes cards the vendor stacks have dropped (ROCm 7 removed Polaris:
+an RX 580 runs here via RADV) and, measured on an RX 9070 (RDNA4), it is
+*faster* than the ROCm/HIP backend on the same card.
+
+```bash
+cd c
+make glm VK=1 # needs libvulkan + glslc (shaderc) for the shaders
+COLI_VULKAN=1 COLI_VK_DENSE=1 COLI_VK_ATTN=1 \
+PIN=<model>/.coli_usage PIN_GB=0 COLI_NO_OMP_TUNE=1 \
+./coli run "Hello" --topp 0.7
+```
+
+Requirements: `libvulkan` and a Vulkan **1.2** ICD with
+`GL_KHR_shader_subgroup_arithmetic` (any Mesa RADV, AMDVLK, NVIDIA or Intel
+ANV driver from the last several years), plus `glslc` at build time. The
+backend picks the most capable physical device (discrete > integrated) and
+degrades to the CPU path on any failure — a wedged GPU can slow a run, never
+corrupt it.
+
+Set `COLI_NO_OMP_TUNE=1` on multi-core boxes: the engine's OMP self-tune
+(active spin-wait) is skipped under `COLI_CUDA`/`COLI_METAL` but not under
+Vulkan, and spinning worker threads starve the async I/O pool (measured
+CPU expert bandwidth 28 → 5 GB/s without it).
+
+## What runs on the GPU
+
+| Piece | Env | Mechanism |
+|---|---|---|
+| Routed experts (hot set) | `COLI_VK_EXPERTS=N` (default 320) | Top-N experts by `.coli_usage` heat uploaded **once at startup** into a VRAM registry; at decode they are served from VRAM with **no RAM slot, no disk read, no prefetch**, as one async fused batch (gate+up+silu→down, hidden on-device) overlapped with the CPU computing the remaining experts. Shown as the `vk` bucket in the hit-rate line. |
+| Dense projections | `COLI_VK_DENSE=1` | q_a+kv_a fused into one submit, q_b, o; shared expert as a single fused expert-group submit. Resident int4/int8 weights upload once. |
+| MLA attention core | `COLI_VK_ATTN=1` | One dispatch per layer: absorbed query, scores over the KV window, softmax, weighted latent, value rows, **fused with the o-projection** (the context vector never leaves the GPU). The latent/rope KV lives in a persistent per-layer device mirror, appended ~2.3 KB/token/layer with the same invalidation points as the CUDA KV shadow. |
+
+The `PIN_GB=0` (with `PIN` still set) in the example is deliberate: the VRAM
+registry holds the same hot experts a RAM pin would, so the pin's RAM is
+better spent on the adaptive LRU cache. Keep `PIN` set so AUTOPIN does not
+re-pin from history.
+
+## Correctness
+
+- `gcc -O3 -DVK_TEST backend_vulkan.c -o test_vk -lvulkan -lm && ./test_vk
+ shaders/qmatmul.spv` runs a CPU-reference exactness harness over every
+ primitive (GEMV int4/int8 across shapes incl. the long-row o-projection,
+ fused gate+up, the full expert group sync and async, the matmul pair, and
+ the absorb attention core incl. causal S=2, kv_start windows, int8, and
+ long-context cases). Typical maxrel ~1e-5..2e-3 (fp32 reduction order).
+- Engine-level: greedy decode with the full stack matches the pure-CPU
+ engine token-for-token on the validation prompt.
+- int4 weights decode as offset-binary (nibble−8), byte-identical layout to
+ the CPU path — no repacking.
+
+## Measured performance (AMD RX 9070, RDNA4, RADV/Mesa 26.1)
+
+Expert-MLP primitive (K experts, int4 6144→2048→6144, per-call incl. readback):
+Vulkan **0.11–0.13 ms/expert** vs the production ROCm/HIP expert group
+**0.179 ms/expert** — ~35% faster. The decode MLA attention core runs 3.7×
+faster than the HIP kernel on the same card. End-to-end GLM-5.2 (744B int4,
+NVMe-streamed) decode on a 12-core Zen2 + RX 9070 box: Vulkan
+**1.7–1.8 tok/s** (64-token) / **1.6** (256-token) / **1.58 sustained**
+(512-token) vs the HIP backend at 1.5–1.55 on identical settings.
+The two write-combined-memory rules that make this possible: buffers the CPU
+reads back must be HOST_CACHED (ReBAR VRAM reads at ~40 MB/s otherwise), and
+everything else lives HOST_VISIBLE|DEVICE_LOCAL.
+
+## Limits and future work
+
+- Decode-focused: the expert tier and attention core serve `S<=4`; prefill
+ uses the CPU/batched paths (dense projections do run on VK at prefill).
+- DSA top-k selection, ragged multi-slot serving, and quantized-KV caches
+ fall back to the CPU attention path.
+- Not yet done: cooperative-matrix (coopmat) prefill kernels, a fully
+ resident-layer pipeline, Polaris/gfx803 validation on real hardware (the
+ shaders use dynamic subgroup sizes and are wave64-safe by construction).