Run frontier MoE models on hardware you already own — pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦
C
docs(qwen36): the DeltaNet layer on the card, experts=all, the split counts model layers
kreuzzelg111556429ef1
diff --git a/docs/ENVIRONMENT.md b/docs/ENVIRONMENT.mdindex 9b9865db..ca33d5f0 100644--- a/docs/ENVIRONMENT.md+++ b/docs/ENVIRONMENT.md@@ -466,6 +466,7 @@ and the CPU/GPU execution split.| `QWEN_SHARED_BATCH` | bounded by 32 MiB scratch | Batch the CPU shared expert across prompt rows. `=0` restores scalar calls; a positive integer caps rows per chunk. The CUDA-tier overlap path is unchanged. || `Q36_MAXT` | conservative engine default | Lower the served/context capacity; it cannot raise the model's compiled safety ceiling. || `COLI_VULKAN` | `0` | `VK=1` build: the dense trunk on the Vulkan device, and the routed experts on the shared expert tier (`COLI_VK_TIER*`, `COLI_VK_DENSE`, see [Vulkan](#vulkan-any-gpu-with-a-vulkan-12-driver)). With the tier on, the engine keeps the expert history `COLI_USAGE` (default `<snap>/.coli_usage`), saved at every run and serve turn end; it keeps none otherwise. |+| `Q36_DN_GPU` | `0` (off) | CUDA expert tier: a decode token runs every DeltaNet layer whose in_proj and out_proj sit on one card end to end on that card (conv, recurrence, gated norm; state resident in VRAM). Measured on the 35B, 3070, trunk in VRAM: DeltaNet 16.2 -> 10.8 ms/token, the token 39.4 -> 33.3 ms. See [qwen36-cuda-tier.md](qwen36-cuda-tier.md#the-deltanet-layer-on-the-card-q36_dn_gpu1). |## Qwen3.8 engine (`qwen38`)diff --git a/docs/qwen36-cuda-tier.md b/docs/qwen36-cuda-tier.mdindex d96bd5ae..079857b6 100644--- a/docs/qwen36-cuda-tier.md+++ b/docs/qwen36-cuda-tier.md@@ -203,6 +203,67 @@ overwrote the group's input and output mid-flight -- no CUDA error, onlywrong numbers. qwen36 never called the dense path inside that window, so itsoutputs were unaffected.+## The DeltaNet layer on the card (`Q36_DN_GPU=1`)++With the trunk in VRAM the CPU still spent about 8 of 39 ms per token on the+thirty DeltaNet layers' convolution, L2 norms, recurrence and gated norm+(`dn-sub: proj 7.9 | conv 3.0 | l2n+rec 2.4 | norm+out 2.8`). None of that is+arithmetic -- a token's convolution is 33k multiply-adds, the recurrence 32+outer products of 128 x 128 -- it is the round trip: the in_proj's result+came down to the host, the recurrence ran there, its output went back up for+the out_proj, four copies and their synchronisations per layer, thirty layers+per token.++`Q36_DN_GPU=1` keeps the layer on the card. For every DeltaNet layer whose+in_proj (`dnproj`) and out_proj (`dnout`) the placer put on the same device,+the conv ring, the recurrent state (32 x 128 x 128 f32, 2 MB per layer, 63 MB+for the 35B) and the conv/norm weights go there too, and a decode token runs+the layer end to end in one device chain (`coli_cuda_dn_step`): x up, in_proj+GEMV, causal conv + SiLU, per value head the L2 norms (double, eps inside the+square root as on the CPU), the decay, the delta against the decayed state,+the write-back, `out = q^T S` over the updated state, the gated RMSNorm with+`silu(z)`, out_proj GEMV, out down. The two gates (`a`, `b` projections, 32+outputs each) stay on the CPU and travel as kernel parameters. One upload and+one download of 8 KB per layer, the state never crosses the bus.++The host arrays remain the state's owner. The card's copy is a cache that is+either fresh (holds what the host holds) or ahead of it; a CPU step -- a+prompt (`S > 1` keeps the batched CPU path), a layer whose projections are+not on one card, a step after a failed GPU step -- pulls the state down+first, a snapshot for `--pin` pulls all layers, `reset_recurrent` and a pin+restore invalidate the card's copy. So `--pin`, prefix reuse and the+`QT_UPLOAD_SYNC`-style diagnostics keep working unchanged. The automatic+placer now puts a layer's `dnout` on the device its `dnproj` went to when that+device has the room, so under `auto` every DeltaNet layer qualifies; a+hand-written list must keep the two on one card itself (note that a split+such as `dnproj=0:20+1:20` counts *model* layers 0..39, not DeltaNet layers).++Measured on the 3070 alone (per-row int4 35B, 49-token prompt, 200 decoded+tokens, trunk in VRAM, `COLI_TIMERS=1`):++| | `Q36_DN_GPU=0` | `Q36_DN_GPU=1` |+|---|---|---|+| DeltaNet ms/token | 16.2 | **10.8** |+| token ms | 39.4 / 39.3 | **33.9 / 33.3** |+| tok/s (engine line, TTFT included) | 21.2 / 21.3 | **24.1 / 24.4** |+| VRAM hit rate | 82 % | 82 % |+| layers on the card | 0 | 30 |++Fourteen per cent per token; the 10.8 ms that remain are the two GEMVs (16+and 8 MB per layer) and the two copies. The output is not byte-identical to+the CPU path (the GEMVs sum in a different order; two GPU runs are identical+to each other), same as every other placed component. On two cards the layer+gains the same 6-7 ms, but this box's second card (Quadro RTX 4000) runs a+dense GEMV about 65 % slower than the 3070 (lm_head 7.3 against 4.4 ms), and+at 99-100 % expert residency the MoE phase only drops from 14.0 to 11.2-13.6+ms -- the expert groups are bound by GPU time and by the slower card's+`take()`, not by misses -- so the best two-card form (trunk on the 3070,+`experts=all`) lands at 32.9 ms, level with the 3070 alone.++`COLI_PLACE` gained `experts=all`: the routed experts stay on every+`COLI_GPUS` card even when a hand-written list reserves some of them for the+trunk (the default hands a reserved card's experts back to the others, #1361).+## Measured (Threadripper 3945WX 12C, RTX 3070 8 GB + Quadro RTX 4000 8 GB, Qwen3.6-35B-A3B int4, 200-token decode)| | 1 GPU (8 GB) | 2 GPUs (16 GB) |