Skip to content

Run frontier MoE models on hardware you already own — pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦

C

docs(qwen36): QT_HOME=layer, the pipeline measured against the index split

kreuzzelg6aca38bd5ef1
diff --git a/docs/ENVIRONMENT.md b/docs/ENVIRONMENT.md
index ca33d5f0..9a718911 100644
--- a/docs/ENVIRONMENT.md
+++ b/docs/ENVIRONMENT.md
@@ -467,6 +467,7 @@ and the CPU/GPU execution split.
| `Q36_MAXT` | conservative engine default | Lower the served/context capacity; it cannot raise the model's compiled safety ceiling. |
| `COLI_VULKAN` | `0` | `VK=1` build: the dense trunk on the Vulkan device, and the routed experts on the shared expert tier (`COLI_VK_TIER*`, `COLI_VK_DENSE`, see [Vulkan](#vulkan-any-gpu-with-a-vulkan-12-driver)). With the tier on, the engine keeps the expert history `COLI_USAGE` (default `<snap>/.coli_usage`), saved at every run and serve turn end; it keeps none otherwise. |
| `Q36_DN_GPU` | `0` (off) | CUDA expert tier: a decode token runs every DeltaNet layer whose in_proj and out_proj sit on one card end to end on that card (conv, recurrence, gated norm; state resident in VRAM). Measured on the 35B, 3070, trunk in VRAM: DeltaNet 16.2 -> 10.8 ms/token, the token 39.4 -> 33.3 ms. See [qwen36-cuda-tier.md](qwen36-cuda-tier.md#the-deltanet-layer-on-the-card-q36_dn_gpu1). |
+| `QT_HOME` | `expert` | CUDA expert tier on two or more cards: `layer` homes every expert of a layer on one card (layer ranges in proportion to the allowances, `QT_LAYER_SPLIT=<n>` layers on the first card), with the layer's trunk and, for the last layer, lm_head alongside -- a pipeline instead of a per-layer join. See [qwen36-cuda-tier.md](qwen36-cuda-tier.md#two-cards-as-a-pipeline-qt_homelayer). |
## Qwen3.8 engine (`qwen38`)
diff --git a/docs/qwen36-cuda-tier.md b/docs/qwen36-cuda-tier.md
index 079857b6..629db423 100644
--- a/docs/qwen36-cuda-tier.md
+++ b/docs/qwen36-cuda-tier.md
@@ -264,6 +264,31 @@ ms -- the expert groups are bound by GPU time and by the slower card's
`COLI_GPUS` card even when a hand-written list reserves some of them for the
trunk (the default hands a reserved card's experts back to the others, #1361).
+### Two cards as a pipeline (`QT_HOME=layer`)
+
+By default expert `eid` is homed on device `eid % n_gpus` in *every* layer,
+so each layer's group is issued to both cards and `qt_take` waits for the
+slower one -- measured on the 3070 + Quadro RTX 4000 pair, `take` 5.2 ms per
+token against 0.2 on one card. `QT_HOME=layer` homes every expert of a layer
+on one device, chosen per layer range in proportion to the cards' allowances
+(or `QT_LAYER_SPLIT=<n>` layers on the first card), puts the layer's trunk
+components on the same device and lm_head on the last layer's, and issues a
+group to exactly one device: the token runs the first layers on one card and
+the rest on the other, like llama.cpp's layer split. Budgets, warmstart, LFRU
+and the swap path follow the homes; a hand-written list's reservation does
+not remove a card in this mode, its layers' experts live there.
+
+Measured (same box, `auto` without the probe, `Q36_DN_GPU=1`, 200 tokens):
+the join is gone (`take` 0.17 ms), but the pipeline stays behind the 3070
+alone -- 38.6 ms/token at a 28/12 split against 33.3 -- because lm_head and
+twelve layers now run on the slower card, and because the MoE phase does not
+fall below ~14 ms at 97-100 % residency: with eight experts on one card the
+group costs about 0.35 ms per layer whatever the hit rate (launch cost per
+expert, not bandwidth; with the experts split 4+4 it is 11.2 ms). Cold, two
+cards win (45.3 against 52.3 ms), since more experts are resident at once.
+So on two *equal* cards the index split remains the faster form; the layer
+split is for unequal cards and for the day the group is one launch.
+
## Measured (Threadripper 3945WX 12C, RTX 3070 8 GB + Quadro RTX 4000 8 GB, Qwen3.6-35B-A3B int4, 200-token decode)
| | 1 GPU (8 GB) | 2 GPUs (16 GB) |