Run frontier MoE models on hardware you already own — pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦
C
Merge pull request #1834 from JustVugg/feat/vulkan-gemm
Vincenzo Fornaro437612fd19fa
diff --cc docs/vulkan.mdindex 60a786c0,1c9db21d..31b5307c--- a/docs/vulkan.md+++ b/docs/vulkan.md@@@ -134,26 -134,77 +134,94 @@@ memory, which is also why MiMo's exper**Status.** CI checks every engine above on Lavapipe (`tests/vulkan_engines.sh`, the`vulkan-engines` job): each configuration gives the CPU run's tokens, and its matmul-count is above zero. That proves correctness, not speed. None of these engines has-been measured on a real GPU yet.+count is above zero. That proves correctness, not speed.++The first real GPU measured is an integrated Radeon 780M (RADV) in a Ryzen 7 PRO+8700GE (16 threads, 61 GiB DDR5, NVMe): same binaries, cold page cache, load under 2.+The shader harness (`tests/vulkan_engines.sh shader`) passes every format case there.+The engines are correct on it and slower than the CPU today:++| Workload | CPU | Vulkan, 780M |+|---|---|---|+| Qwen3.8 Flash Next int4, decode 100 tokens | 3.55 tok/s | 1.53 tok/s |+| Qwen3.8 Flash Next, prefill 512 tokens | 51 s | 109 s |+| Qwen3.6-35B-A3B, decode | 5.97 tok/s | 3.06 tok/s (identical output) |+| Qwen3.6-35B-A3B, prefill 512 tokens | 39.8 s | 60.7 s |++Why, measured:+- **Decode**: every matmul is a synchronous submit, about 726 per token, and an+ integrated GPU reads the same RAM as the CPU.+- **Prefill**: the shader is a per-row GEMV, so each weight is read once per prompt+ row.+ ## Prefill: the tiled GEMMs++ `qmatmul.comp` is a GEMV per activation row: at S rows every weight is fetched and+ decoded S times, which holds a Radeon 780M at 80 GFLOP/s whatever S is. From S = 2,+ `coli_vk_matmul` (the resident-matrix path of every engine) takes a tiled GEMM+ instead, so the API and the callers are unchanged:++ - `qmatmul_gemm.comp`, fp32, every format: a workgroup owns a BM-output x BN-row tile+ of y, decodes its weight rows into shared memory once per 32-input step and runs+ them against BN activation rows staged beside them, a TM x TN block per thread.+ Group scales fold into the decoded weight; each step sums into a fresh partial+ (blocked summation), which keeps the error at the GEMV's level.+ - `qmatmul_coop.comp`, where the device has `VK_KHR_cooperative_matrix` with a+ 16x16x16 fp16 x fp16 -> fp32 subgroup shape and a settable subgroup size: the+ formats whose weights decode exactly to fp16 (int8, int4, int3-g64, MXFP4, fp8;+ grouped ones with gs % 32 == 0). Nothing is rounded to fp16: each activation row is+ scaled by a power of two the host computes during the upload, split into an fp16+ high part (top 11 bits) and an fp16 low part, and the MMA runs on both; every+ 16-input product then joins fp32 accumulators, times the group scale. The harness+ holds it to the fp32 GEMM's bound.+ - Each shader is built at a few tile widths (BN 32 for a 32-row prefill chunk, 64,+ 128); a call takes the narrowest that covers its S. The threshold, measured on the+ 780M: S >= 2 and S*O >= 4096 (a 48-output matrix stays on the GEMV up to S = 32).+ `COLI_VK_GEMM_MIN_S` overrides it, `COLI_VK_COOP=0` keeps the fp32 GEMM.++ Measured on a Radeon 780M (RDNA3, RADV, Mesa 26.0) sharing DDR5 with a Ryzen 7 PRO+ 8700GE, `OMP_NUM_THREADS=8`. One matrix, I = 2560, O = 6144, S = 512, back to back+ (the harness, `COLI_VK_TEST_GEMM_BENCH=1`; the CPU column is the kernel an engine+ runs for that storage, `-march=native`):++ | fmt | GEMV | tiled GEMM (fp32) | cooperative matrix | CPU, 8 threads |+ |---|---|---|---|---|+ | 1 int8 | 82 GFLOP/s | 1431 | 1909 | 841 (int8 activations, VNNI) |+ | 2 int4 | 104 | 1486 | 2164 | 224 |+ | 4 int4-g64 | 86 | 1406 | 2026 | 488 (int8 activations, VNNI) |+ | 5 int3-g64 | 86 | 1441 | 2039 | 125 |+ | 7 MXFP4 | 64 | 1390 | 1928 | 199 |+ | 10 f32 | 29 | 948 | | 224 |+ | 11 bf16 | 51 | 1222 | | 25 |+ | 12 fp8 | 43 | 1259 | 1735 | 33 |++ End to end, prefill of 512 tokens (`N_NEW=1`, cold page cache, same binaries):++ | | CPU | Vulkan, GEMV | Vulkan, tiled GEMM |+ |---|---|---|---|+ | Qwen3.8 Flash Next (int8 trunk, bf16) | 50.7 s | 111.1 s | 53.8 s |+ | same, `Q38_PREFILL_BATCH_ROWS=512` | 44.7 s | | 44.3 s |+ | Qwen3.6-35B-A3B (int8 dense) | 40.9 s | 62.0 s | 56.3 s |++ On this APU the GEMM brings Qwen3.8's Vulkan prefill from 111 s to the CPU's+ level: level with it with 512-row prefill chunks (44.3 against 44.7 s), 6% behind+ at the default 32-row chunks; the generated token is the CPU's on both models.+ `VK_PROF=1` shows where the time goes. Qwen3.8 at the default chunks spends 8.4 s in+ resident matmuls on the device against the CPU's 6.5 s: 6.8 s in 5,052+ cooperative-matrix GEMMs (the DeltaNet projections at S = 32, 3.1 s; shared experts+ and router, 1.7 s; attention and gated residuals at S = 512, 2.0 s) and 1.6 s in+ 2,273 GEMVs (1,024 one-token PLE projections, 0.86 s; matrices too narrow for the+ GEMM, 0.7 s). Two things measured there hold it at parity:+ - A clock stuck at 800 MHz. The engines call the device synchronously, one matrix+ at a time between CPU phases, and with `power_dpm_force_performance_level=auto`+ the GPU stays at its 800 MHz floor through those millisecond bursts (over 97% of+ the samples during the Qwen3.8 runs): a GEMM that takes 1.1 ms back to back takes+ 2.3 ms after a 3 ms CPU gap. Fewer, larger calls (512-row chunks) are what lift+ it to parity.+ - Per-token calls. Qwen3.6 without the CUDA tier projects its DeltaNet inputs one+ token at a time (`deltanet()` in qwen36.c): 46,081 of its 46,281 device matmuls+ are S = 1 and stay on the GEMV (20 s of its 56).+## Correctness- `gcc -O3 -DVK_TEST backend_vulkan.c -o test_vk -lvulkan -lm && ./test_vk