Run frontier MoE models on hardware you already own — pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦
C
docs/qwen36: Qwen3-Coder-30B-A3B, measured against the bf16 release
JustVugg31ffe8c06c1f
diff --git a/docs/media/qwen3-coder-web.png b/docs/media/qwen3-coder-web.pngnew file mode 100644index 00000000..f71a1adcBinary files /dev/null and b/docs/media/qwen3-coder-web.png differdiff --git a/docs/qwen36.md b/docs/qwen36.mdindex 15fe8e6c..41f7be62 100644--- a/docs/qwen36.md+++ b/docs/qwen36.md@@ -199,14 +199,17 @@ tier takes one format per expert and refuses a mixed container with a line## Which checkpoints, and what the banner calls them-Three Qwen checkpoints resolve to this engine: two MoE ones that declare-`model_type: qwen3_5_moe_text`, and a dense one that declares `qwen3_5`:+These Qwen checkpoints resolve to this engine: two hybrid MoE ones that declare+`model_type: qwen3_5_moe_text`, a dense one that declares `qwen3_5`, and the+all-attention Qwen3 MoE (`qwen3_moe`) of Qwen3-Coder:| checkpoint | layers | experts | hidden | banner ||---|---|---|---|---|| Qwen/Qwen3.6-35B-A3B | 40 (10 attention) | 256, top-8 | 2048 | `Qwen3.6-35B-A3B · 35B MoE` || Qwen/Qwen3.8-2.4T-A95B | 92 (23 attention) | 512, top-10 | 8192 | `Qwen3.8-2.4T-A95B · 2.4T MoE` || Qwen/Qwen3.8-27B | 64 (16 attention) | none: one MLP of 17408 per layer | 5120 | `Qwen3.8-27B · 27B` |+| Qwen/Qwen3-Coder-30B-A3B-Instruct | 48 (all attention) | 128, top-8 | 2048 | `Qwen3-Coder-30B-A3B · 30B MoE` |+| cerebras/Qwen3-Coder-REAP-25B-A3B | 48 (all attention) | 103, top-8 | 2048 | `Qwen3-Coder-REAP-25B-A3B · 25B MoE` (named, not run here) |The registry names a checkpoint by its geometry (`display_variants` on the`qwen36` descriptor), so the banner says what is on disk. A config that@@ -294,6 +297,63 @@ been run with real pictures.Not yet: the CUDA tier (a dense checkpoint runs on the CPU), the MTP head(skipped by the converter), video, and an int4 container on disk.+### Qwen3-Coder-30B-A3B++Qwen/Qwen3-Coder-30B-A3B-Instruct (Apache-2.0) is `Qwen3MoeForCausalLM`: 48+attention layers and no DeltaNet, no attention output gate, no shared expert,+rotary over the whole head (`rope_theta` 1e7), plain RMSNorm weights, and 128+experts top-8 renormalized (`norm_topk_prob`). The converter recognises+`model_type: qwen3_moe` and writes exactly that into `qwen36_meta.json`+(all-attention `layer_types`, `shared_inter: 0`, `partial_rotary_factor: 1.0`,+`zero_centered_norms: false`); the engine then skips the shared expert and+passes the attention output ungated. The REAP prunes are the same+architecture with fewer experts and resolve to the same family under their+own name. The chat template is its own: tools as XML, calls as+`<tool_call><function=...><parameter=...>`, and no thinking at all. The gateway+recognises it from the template, renders it byte for byte+(`tests/test_qwen3_coder_chat_template.py` holds it to the release's+`chat_template.jinja`), and keeps `enable_thinking` off. The API model id is+`qwen3-coder-30b-a3b-colibri`.++```bash+hf download Qwen/Qwen3-Coder-30B-A3B-Instruct --local-dir qwen3-coder # 61.1 GB, bf16+python3 tools/convert_qwen36.py --model qwen3-coder --out qwen3-coder-i4 --ebits 4 --gs 64+coli serve --model qwen3-coder-i4 --cap 128 # every expert in RAM+coli serve --model qwen3-coder-i4 --cap 32 # 6.5 GB resident+```++The int4 gs64 container is 19 GB and converts in under a minute; `--ebits 8`+gives a 30 GB int8 one. As everywhere on this engine, `--cap` sizes the+expert cache and `--ram` does not (see below).++Against the bf16 release, on a 325-token code question and answer, teacher+forced (every position's logits; the reference reads the release one layer at+a time in f32):++| experts | dense trunk | top-1 = bf16 | top-5 overlap | mean \|Δ log p\| | max \|Δ log p\| |+|---|---|---|---|---|---|+| int4 gs64 | int8 (the default) | 96.9% | 91.1% | 0.176 | 3.74 |+| int4 gs64 | f32 (`COLI_DENSE_I8=0`) | 96.0% | 91.7% | 0.134 | 2.59 |+| int8 | int8 | 95.1% | 95.2% | 0.098 | 2.13 |+| int8 | f32 | 98.1% | 97.8% | 0.022 | 0.60 |++Decode on a Ryzen 7 PRO 8700GE (8 cores, 64 GB, NVMe RAID), the CLI from a cold+cache, 128 tokens, dense trunk int8:++| container | experts cached per layer | decode | resident |+|---|---|---|---|+| int4 gs64 | 128 (all) | 8.5 and 9.6 tok/s (two runs) | 15.2 GB |+| int8 | 128 (all) | 6.4 tok/s | 25.1 GB |+| int4 gs64 | 32 | 5.1 tok/s | 6.5 GB |+| int8 | 32 | 3.8 tok/s | 9.6 GB |++Through `coli chat` with `--cap 128` and the cache warm, a 577-token answer+streams at 13 tok/s. Prefill is the slow part on the CPU: a request whose tool+block makes the prompt about 500 tokens takes around two minutes from a cold+cache.+++### The converter's tensor contractBoth checkpoints ship experts **fused** per layer (`mlp.experts.gate_up_proj`,