Skip to content

Run frontier MoE models on hardware you already own — pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦

C

docs/qwen36: Qwen3-Coder-30B-A3B, measured against the bf16 release

JustVugg31ffe8c06c1f
diff --git a/docs/media/qwen3-coder-web.png b/docs/media/qwen3-coder-web.png
new file mode 100644
index 00000000..f71a1adc
Binary files /dev/null and b/docs/media/qwen3-coder-web.png differ
diff --git a/docs/qwen36.md b/docs/qwen36.md
index 15fe8e6c..41f7be62 100644
--- a/docs/qwen36.md
+++ b/docs/qwen36.md
@@ -199,14 +199,17 @@ tier takes one format per expert and refuses a mixed container with a line
## Which checkpoints, and what the banner calls them
-Three Qwen checkpoints resolve to this engine: two MoE ones that declare
-`model_type: qwen3_5_moe_text`, and a dense one that declares `qwen3_5`:
+These Qwen checkpoints resolve to this engine: two hybrid MoE ones that declare
+`model_type: qwen3_5_moe_text`, a dense one that declares `qwen3_5`, and the
+all-attention Qwen3 MoE (`qwen3_moe`) of Qwen3-Coder:
| checkpoint | layers | experts | hidden | banner |
|---|---|---|---|---|
| Qwen/Qwen3.6-35B-A3B | 40 (10 attention) | 256, top-8 | 2048 | `Qwen3.6-35B-A3B · 35B MoE` |
| Qwen/Qwen3.8-2.4T-A95B | 92 (23 attention) | 512, top-10 | 8192 | `Qwen3.8-2.4T-A95B · 2.4T MoE` |
| Qwen/Qwen3.8-27B | 64 (16 attention) | none: one MLP of 17408 per layer | 5120 | `Qwen3.8-27B · 27B` |
+| Qwen/Qwen3-Coder-30B-A3B-Instruct | 48 (all attention) | 128, top-8 | 2048 | `Qwen3-Coder-30B-A3B · 30B MoE` |
+| cerebras/Qwen3-Coder-REAP-25B-A3B | 48 (all attention) | 103, top-8 | 2048 | `Qwen3-Coder-REAP-25B-A3B · 25B MoE` (named, not run here) |
The registry names a checkpoint by its geometry (`display_variants` on the
`qwen36` descriptor), so the banner says what is on disk. A config that
@@ -294,6 +297,63 @@ been run with real pictures.
Not yet: the CUDA tier (a dense checkpoint runs on the CPU), the MTP head
(skipped by the converter), video, and an int4 container on disk.
+### Qwen3-Coder-30B-A3B
+
+Qwen/Qwen3-Coder-30B-A3B-Instruct (Apache-2.0) is `Qwen3MoeForCausalLM`: 48
+attention layers and no DeltaNet, no attention output gate, no shared expert,
+rotary over the whole head (`rope_theta` 1e7), plain RMSNorm weights, and 128
+experts top-8 renormalized (`norm_topk_prob`). The converter recognises
+`model_type: qwen3_moe` and writes exactly that into `qwen36_meta.json`
+(all-attention `layer_types`, `shared_inter: 0`, `partial_rotary_factor: 1.0`,
+`zero_centered_norms: false`); the engine then skips the shared expert and
+passes the attention output ungated. The REAP prunes are the same
+architecture with fewer experts and resolve to the same family under their
+own name. The chat template is its own: tools as XML, calls as
+`<tool_call><function=...><parameter=...>`, and no thinking at all. The gateway
+recognises it from the template, renders it byte for byte
+(`tests/test_qwen3_coder_chat_template.py` holds it to the release's
+`chat_template.jinja`), and keeps `enable_thinking` off. The API model id is
+`qwen3-coder-30b-a3b-colibri`.
+
+```bash
+hf download Qwen/Qwen3-Coder-30B-A3B-Instruct --local-dir qwen3-coder # 61.1 GB, bf16
+python3 tools/convert_qwen36.py --model qwen3-coder --out qwen3-coder-i4 --ebits 4 --gs 64
+coli serve --model qwen3-coder-i4 --cap 128 # every expert in RAM
+coli serve --model qwen3-coder-i4 --cap 32 # 6.5 GB resident
+```
+
+The int4 gs64 container is 19 GB and converts in under a minute; `--ebits 8`
+gives a 30 GB int8 one. As everywhere on this engine, `--cap` sizes the
+expert cache and `--ram` does not (see below).
+
+Against the bf16 release, on a 325-token code question and answer, teacher
+forced (every position's logits; the reference reads the release one layer at
+a time in f32):
+
+| experts | dense trunk | top-1 = bf16 | top-5 overlap | mean \|Δ log p\| | max \|Δ log p\| |
+|---|---|---|---|---|---|
+| int4 gs64 | int8 (the default) | 96.9% | 91.1% | 0.176 | 3.74 |
+| int4 gs64 | f32 (`COLI_DENSE_I8=0`) | 96.0% | 91.7% | 0.134 | 2.59 |
+| int8 | int8 | 95.1% | 95.2% | 0.098 | 2.13 |
+| int8 | f32 | 98.1% | 97.8% | 0.022 | 0.60 |
+
+Decode on a Ryzen 7 PRO 8700GE (8 cores, 64 GB, NVMe RAID), the CLI from a cold
+cache, 128 tokens, dense trunk int8:
+
+| container | experts cached per layer | decode | resident |
+|---|---|---|---|
+| int4 gs64 | 128 (all) | 8.5 and 9.6 tok/s (two runs) | 15.2 GB |
+| int8 | 128 (all) | 6.4 tok/s | 25.1 GB |
+| int4 gs64 | 32 | 5.1 tok/s | 6.5 GB |
+| int8 | 32 | 3.8 tok/s | 9.6 GB |
+
+Through `coli chat` with `--cap 128` and the cache warm, a 577-token answer
+streams at 13 tok/s. Prefill is the slow part on the CPU: a request whose tool
+block makes the prompt about 500 tokens takes around two minutes from a cold
+cache.
+
+![Qwen3-Coder-30B-A3B in coli web](media/qwen3-coder-web.png)
+
### The converter's tensor contract
Both checkpoints ship experts **fused** per layer (`mlp.experts.gate_up_proj`,