Run frontier MoE models on hardware you already own — pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦
C
docs: Qwen-Image's attention on the matrix units, opt-in
JustVugg95bf0f32d3cc
diff --git a/docs/qwen-image.md b/docs/qwen-image.mdindex 20256f08..d06bf50a 100644--- a/docs/qwen-image.md+++ b/docs/qwen-image.md@@ -288,6 +288,13 @@ every block costs 0.3 s on a 512x512 step (5.64 s against 5.35); a discrete cardmoves them over PCIe, about 6.5 GB a step. With blocks streamed, the VAE decodes onthe CPU.+`COLI_VK_QI_ATTN=coop` runs the attention on the device's matrix units (tensor+cores, WMMA, XMX; cooperative matrices at subgroups of 32 or 64) with int8 weights:+f16 operands, the result the f32 attention's to about one part in a thousand. It is+off by default because on the 780M it is slower (a 1024x1024 step's attention 32.5 s+against 17.9): RDNA3's matrix units are only twice its f32 rate. A card whose matrix+units outrun its f32 by more may gain; that has not been measured.+Measured on a Radeon 780M (an integrated GPU sharing the RAM of the 8-core Zen 4above), 8 steps: