Run frontier MoE models on hardware you already own — pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦
Qwen3.6-35B-A3B on colibri
c/qwen36.c runs Qwen/Qwen3.6-35B-A3B
(35B total / ~3B active, Apache 2.0) — a hybrid architecture: 25% Gated
Attention layers, 75% Gated DeltaNet (linear attention) layers, each followed
by a streamed MoE block (256 experts, top-8 + 1 shared). Dense weights stay
resident; routed experts stream from the container through an LRU + pinned
cache. Development notes live in docs/qwen36-phase01.md /
qwen36-phase02.md.
Architecture-identical checkpoints (same config geometry, e.g. KAT-Coder-V2.5-Dev) run on this engine unchanged.
Quickstart
Pre-converted containers (int4 experts, self-contained, ~23 GB):
# group-scaled int4 (gs64) — recommended, see "Which container" below
hf download Kreuzzelg/qwen36-35b-a3b-colibri-i4-gs64 --local-dir ~/Models/qwen36_i4_gs64
# per-row int4
hf download Kreuzzelg/qwen36-35b-a3b-colibri-i4 --local-dir ~/Models/qwen36_i4
or convert the original bf16 checkpoint yourself (~70 GB download):
python3 c/tools/convert_qwen36.py --repo Qwen/Qwen3.6-35B-A3B --out ~/Models/qwen36_i4_gs64 --gs 64
Build and chat:
make -C c qwen36
COLI_MODEL=~/Models/qwen36_i4_gs64 ./c/coli chat
coli reads the model's config.json and matches its model_type against the
family registry (qwen3_5_moe / qwen3_5_moe_text — an exact match, so other
Qwen architectures are not claimed by this engine), picks it, and drives it over the serve protocol — coli web and
coli serve (OpenAI-compatible API) work the same way.
Tool calling is supported through the HTTP gateway using Qwen3.6's native
<tool_call> / <tool_response> protocol. The gateway translates between
the native format and the supported API tool representations; see the
per-engine API matrix.
Direct invocation without the gateway:
SNAP=~/Models/qwen36_i4_gs64 TOK=~/Models/qwen36_i4_gs64/tokenizer.json \
N_NEW=200 ./c/qwen36 256 4 prompt.txt
CPU prefill batching
On AVX2/FMA CPUs, dense int8 projections process two prompt rows per weight
decode. The same kernel is used by attention, the router, DeltaNet projections,
and the CPU shared expert. S=1 decode keeps the original four-register GEMV,
and non-AVX2 builds retain the established per-row implementation. The shared
expert additionally batches gate/up/down calls across prompt rows with at most
32 MiB of temporary activations; the CUDA expert tier keeps its per-token shared
work so it can continue to overlap the in-flight GPU groups.
Both optimizations are bit-exact and on by default. For controlled A/Bs,
QWEN_DENSE_BATCH=0 restores per-row dense-int8 GEMVs and
QWEN_SHARED_BATCH=0 restores per-row shared-expert calls. A positive
QWEN_SHARED_BATCH=N limits each shared chunk to N rows.
Requirements: ~30 GB RAM for comfortable expert caching and NVMe storage for
the container. The default build is CPU-only; make -C c qwen36 CUDA=1 adds
the optional CUDA VRAM expert tier documented in
qwen36-cuda-tier.md. The same tier builds for AMD
through ROCm with make -C c qwen36 HIP=1 HIP_ARCH=<gfx> (for example
HIP_ARCH=gfx1151, with ROCM_HOME and HIPCC pointing at the toolchain):
measured on a Ryzen AI MAX+ 395, output bit-identical to the CPU path and 2.4x
faster than CPU-only (#1502).
Vulkan
In a VK=1 build (make -C c qwen36 VK=1), COLI_VULKAN=1 puts the dense trunk on
the Vulkan device and the routed experts on the shared Vulkan expert tier
(c/vk_tier.c, vulkan.md): a cache of
experts in device memory, filled at startup from the expert history and adapted
while you chat, computed by the device while the CPU computes the rest of the
layer step, every expert's output joining its row in rank order. It takes every
container this engine reads: the shared kernel's planar int4-g64 slots, the int8
copy of an int4 container (packed back to int4 on the device), int8 per row or gs64,
and the mixed int4 gate/up + int8 down layout. On an integrated GPU (or Lavapipe) the
trunk stays on the CPU by default while the tier is on, where it costs more on the
device than the tier gains; a discrete GPU takes both. COLI_VK_DENSE=0 keeps the
trunk on the CPU anywhere and COLI_VK_DENSE=1 puts it on the device anywhere;
COLI_VK_TIER=0 turns the tier off (the trunk then goes to the device).
This engine kept no expert history before; with the tier on it keeps route_trace.h's
.coli_usage beside the container (COLI_USAGE moves it), saved at the end of
every run and serve turn, and fills the tier from it at the next start. coli setup gives Qwen3.6-35B-A3B a starting one (vulkan.md). With
COLI_CUDA=1 as well, the CUDA tier wins.
On an integrated Radeon 780M, with the int4 gs64 container at cap 64 and the trunk on the CPU, the tier decoded at 8.03 tok/s against the CPU's 6.02 and reached the first token of a 512-token prompt in 12.3 s against 35.7 s, the same text; the measurements and what they leave out are in vulkan.md. With the dense chain on as well (the default for this engine on an integrated GPU with the tier), decode reached 9.94 tok/s against the CPU's 6.01 and the 512-token prompt 9.5 s (vulkan.md).
Prompt-lookup drafts (on by default, COLI_LOOKUP=0 turns them off)
When the recent tokens repeat an n-gram of the prompt or the output (a code edit, a
quote, repeated structure), the engine drafts the up to 5 tokens that followed it
and checks them in one verify forward. The DeltaNet state is copied after each verify
row, so a rejected draft rolls back by swapping a copy in. A gate drafts only where the
measured acceptance and verify cost say it pays. The output is that of plain decoding,
greedy or sampled, on the CPU and in the Vulkan dense chain. Lookup stays off under the
CUDA tier, CACHE_ROUTE and a qpack container, whose results depend on what is
resident. On a code edit, Qwen3.6-35B-A3B decoded 7.90 tok/s with lookup against 7.42 without (1.56 tokens per forward); on a chat prompt the gate declined every proposal and the speed was unchanged. How it works, the gate, the settings and the tests:
speculative.md.
The expert kernel
Routed experts run through c/expert_ffn.h, a header shared with the other
MoE engines rather than a set of GEMVs of this engine's own. Three things
changed with it, measured on the real gs64 container at full residency on an
8-core AVX-512 box (61 GB, DDR5-5600, 58 GB/s DRAM read):
- The int4 stays int4. The engine used to unpack every expert to int8 at
load; the kernel keeps the container's nibbles, repacked once into a planar
layout where
and 0x0Fyields 32 elements in order andsrli 4the other 32, so a block costs no unpack instruction. Half the bytes per token and half the expert-cache RSS: peak RSS at cap 256 went from 25 GB to 15 GB. - A layer is a unit of work. gate+up share one pass over the activation, and threads split (expert, row-chunk) items: two OpenMP regions per layer instead of 3 x top-k. A prompt row routed to an expert another row already used reads that expert from cache, not DRAM.
- Same tokens. Activations stay f32 (the kernel also has an int8
activation mode,
mode 1, not wired in here: same policy asIDOT). The only difference from the old path is the accumulation order inside a dot; a 1024-token greedy decode on the real container is byte-identical, and CI pins old vs new on a tiny int4 fixture at caps 1, 2, 8 and 16.
Over a 1024-token greedy decode at cap 256 (same prompt, byte-identical
text): 12.8 -> 15.7 tok/s, MoE per token 34 -> 20 ms on average and 30 -> 17
ms in the last windows, peak RSS 29 -> 17 GB. Of the 20 ms, 11 are the kernel
(the DRAM floor for the int4 bytes is 9) and 6 are the residual misses of a
97.6% hit rate, fetched one at a time; that fetch is the next thing to
overlap, not this kernel. tests/test_expert_ffn holds the numerics.
QWEN_EXPERT_KERNEL=0 restores the int8 path for A/Bs. The CUDA expert tier
keeps its own path: it uploads the pair-layout int4 and computes misses from
the int8 copy.
The dense trunk: integer dot products
Every dense GEMV of a token, the DeltaNet in and out projections, the attention q/k/v/o, the shared expert and lm_head, used to multiply int8 weights by f32 activations: each weight byte converted to f32 and fed to an FMA, eight weights per instruction. On a 16-core AVX-512 host lm_head (248320 x 2048 int8, 508 MB) ran at 29 GB/s on a memory bus that does 80: the kernel was the limit, and the dense part of the token is 1.9 GB of int8 on the 35B, three times what the routed experts read.
Since 1.12.1 the activation is quantized to int8 once per call (one scale, amax/127, the same contract the expert integer kernels use) and the products are integer: 32 weights per instruction on AVX2, 64 on AVX-512 VNNI, exact int32 sums scaled once per output. The routed experts take the same path for their activations. Measured on Qwen3.6-35B-A3B, 8 threads, 300 decoded tokens, every expert resident, perplexity on 4 x 512 tokens of English text:
| tok/s | ms/token: DeltaNet proj / out, attention, lm_head, expert compute | perplexity | |
|---|---|---|---|
| f32 activations (1.12.0) | 6.71 | 21.9 / 8.6, 9.5, 12.6, 22.7 | 13.79 |
int8 activations, dense trunk (COLI_DENSE_IDOT=1) |
7.35 | 17.4 / 6.3, 7.7, 10.2, 22.7 | 13.92 (+1.0%) |
int8 activations, routed experts (QWEN_EXPERT_ACT=i8) |
7.20 | 21.9 / 8.6, 9.5, 12.6, 15.9 | 13.80 (+0.1%) |
| both (the 1.12.1 default) | 8.23 (+22.6%) | 17.2 / 6.3, 7.5, 10.1, 15.6 | 13.97 (+1.3%) |
Both are the default; COLI_DENSE_IDOT=0 and QWEN_EXPERT_ACT=f32
restore the f32 kernels, which stay bit-identical to their references.
int4 for the trunk is opt-in, per component. COLI_DENSE_BITS=4 stores
the dense matrices as int4 in blocks of 64 with one scale per block (the
planar layout of the grouped expert kernel) and halves the bytes the token
reads, but the parts of the trunk pay 4 bits very differently, so
COLI_DENSE_INT4 picks which ones take it:
| int4 on | tok/s | perplexity |
|---|---|---|
| nothing (int8) | 7.35 | 13.92 |
lmhead |
8.15 (lm_head 10.1 to 8.4 ms; the total is within noise of the cache warming) | 14.13 (+2.4% vs f32) |
lmhead,dnproj,dnout |
14.56 (+5.6%) | |
lmhead,dnproj,dnout,shexp |
14.94 (+8.3%) | |
everything (attn included) |
8.09 | 15.17 (+10%) |
A least-squares refinement of the block scale was tried and changes nothing
(15.16 against 15.17): the loss is the matrices' sensitivity, not the
quantizer. If you take one, take lmhead: 254 MB less per token for the
smallest cost.
Cache-aware routing (CACHE_ROUTE, off by default)
The residual misses above are the lever's target. CACHE_ROUTE=1 ports the
GLM engine's max-rank re-routing (CACHE_ROUTE.md,
arXiv:2412.00099) to this engine with two residency levels: inside the top-M
window, a slot past the sacred top-J prefers an expert already in the VRAM
tier, then one in the RAM cache, then the plain ranking. It is lossy: it
changes which experts run, so the semantic contract is off while it is set and
the footer prints what it cost, route_agree (overlap with the true top-K)
and route_kl (mass KL), next to the swap and hit rates. Unset, the router
is the original loop and the token ids are byte-identical; ROUTE_AGREE=1
alone prints the meters at 100 % / 0 without touching routing.
Qwen3.6 routes top-8 (plus the shared expert), so the default ROUTE_J=2
leaves six substitutable slots per token; the tiny fixture routes top-2 and
needs ROUTE_J<2 to show any swap at all. A/B it the way the GLM doc does:
same prompt and seed, tok/s and hit rate against agreement and KL, and treat
PPL=1 on a teacher-forced reference as the quality bar.
Which container?
The gs64 container carries one scale per 64-weight group instead of one per
row. Against the per-row container it measured a cosine to the int8 anchor of
0.99313 instead of 0.98777 and a KL of 0.080 instead of 0.109, about 44% less
quantization error. On GLM, per-row int4 was the root cause of think-mode loops and
never-terminating generations (#455), and group scales fixed them in
controlled A/Bs — with moe_intermediate_size=512, Qwen's rows are short, so
per-row quantization error concentrates the same way. The gs64 container costs
~1.7 GB more on disk and a few percent on cold-start; warm decode speed is the
same or slightly better.
Mixed: int8 down, int4 gate/up. convert_qwen36.py --ebits 4 --gs 64 --down-bits 8 (--down-gs for grouped down scales, 0 = per row) writes one
slab per expert with down_proj in int8 and gate/up as above -- 5.7 bits per
weight against gs64's 4.5. It is the knob that produced the #1370 numbers on
wikitext-2 (16 x 512 tokens): gs64 7.325, mixed 7.281, all experts int8 7.153,
Ollama's Q4_K_M 7.147 -- down alone recovers a quarter of the gap to int8,
the rest sits in gate/up, and at equal bits Q4_K_M's asymmetric quantizer is
ahead. Keep it as a measurement tool and a middle step for boxes with RAM to
spare; it is not the answer to the gap. The engine tells the layout apart by
size and reads each matrix in its own format on the CPU path; the CUDA VRAM
tier takes one format per expert and refuses a mixed container with a line
(COLI_CUDA=1 ignored), so such a container runs CPU-only for now.
Which checkpoints, and what the banner calls them
These Qwen checkpoints resolve to this engine: two hybrid MoE ones that declare
model_type: qwen3_5_moe_text, a dense one that declares qwen3_5, and the
all-attention Qwen3 MoE (qwen3_moe) of Qwen3-Coder:
| checkpoint | layers | experts | hidden | banner |
|---|---|---|---|---|
| Qwen/Qwen3.6-35B-A3B | 40 (10 attention) | 256, top-8 | 2048 | Qwen3.6-35B-A3B · 35B MoE |
| Qwen/Qwen3.8-2.4T-A95B | 92 (23 attention) | 512, top-10 | 8192 | Qwen3.8-2.4T-A95B · 2.4T MoE |
| Qwen/Qwen3.8-27B | 64 (16 attention) | none: one MLP of 17408 per layer | 5120 | Qwen3.8-27B · 27B |
| Qwen/Qwen3-Coder-30B-A3B-Instruct | 48 (all attention) | 128, top-8 | 2048 | Qwen3-Coder-30B-A3B · 30B MoE |
| cerebras/Qwen3-Coder-REAP-25B-A3B | 48 (all attention) | 103, top-8 | 2048 | Qwen3-Coder-REAP-25B-A3B · 25B MoE (named, not run here) |
The registry names a checkpoint by its geometry (display_variants on the
qwen36 descriptor), so the banner says what is on disk. A config that
matches neither, a tiny fixture for instance, is named by its own
model_type and measured geometry rather than by a sibling's parameter
count (#1045).
The 2.4T checkpoint is architecture-identical to the 35B: same layer
pattern, every engine guard holds, and the registry's planner puts its KV
cache at 1.44 GiB for 8k context and 46 GiB at the 256k maximum, with a
context-free DeltaNet state of 0.55 GiB. What this engine cannot do for it
is hold the experts: the warmstart keeps every expert in RAM by design (see
--ram below), which is ~1.4 TB of int4 for 2.4T. Serving it needs the
disk-streaming design, not this one. The conversion and the geometry checks
are in place so that work starts from a verified shape, not from a guess.
The dense 27B
Qwen3.8-27B (#1757) is Qwen3_5ForConditionalGeneration: the same Gated
DeltaNet + gated attention layers, with one SwiGLU MLP per layer and no
router. The engine loads that MLP as the shared expert, ungated, and routes
nothing; the converter writes num_experts: 0 and the MLP width into
qwen36_meta.json. It ships Qwen3.8's chat_template.jinja, not Qwen3.6's
(the same file the qwen38 renderer is pinned to): the converter copies it
into the container, and the gateway recognises it and renders with the
Qwen3.8 rules (reasoning on by default at xhigh, the XML tool-call form,
history that keeps its thinking) while the engine stays qwen36. The API model
id is qwen3.8-27b-colibri.
python3 tools/convert_qwen36.py --model <Qwen3.8-27B download> --out q27_c
./coli chat --model q27_c --gpu none
The container keeps the weights in f16 (51 GB) and the engine quantizes them while loading. Every weight is read for every token, so speed is set by memory bandwidth, not by the disk. Measured on a 16-thread CPU server (8 OpenMP threads), perplexity over 1000 tokens of human-written text:
| dense weights | RSS | scoring | perplexity, English | perplexity, Italian |
|---|---|---|---|---|
| int8 (default) | 29.1 GB | 2.25 tok/s | 4.85 | 11.19 |
COLI_DENSE_BITS=4 COLI_DENSE_INT4=shexp,lmhead |
21.6 GB | 3.15 tok/s | 4.94 (+1.8%) | 11.69 (+4.5%) |
COLI_DENSE_BITS=4 (everything) |
18.8 GB | 3.72 tok/s | 5.02 (+3.5%) | 12.15 (+8.6%) |
Through the gateway (coli serve), greedy decode ran at 2.1 tok/s in int8 and
3.45 tok/s with everything in int4, after a load of 73 and 114 s; the answers
to the same English, Italian and Turkish questions were the same but for a word.
The MLP is 17 of the 27 billion parameters, so int4 on the MLP and the LM head
keeps most of the saving at half the loss. With a matrix in int4 the engine
no longer keeps its int8 copy (COLI_DENSE_KEEP_I8=1 does); that is what
brings the full int4 run from 42.4 to 18.8 GB.
Images
Qwen3.8-27B reads images. Its vision tower is the ViT of the whole family
(27 blocks, hidden 1152, patch 16, 2x2 merge; only the output width follows
the text model), so the engine runs it through the same qwen38_vision.h
the qwen38 engine uses, now split across OpenMP threads. The converter copies
it into model-vision.safetensors and writes its shape into
qwen36_meta.json, together with preprocessor_config.json.
What is specific to this family is where the image sits in the rope. The
attention layers use interleaved M-RoPE (mrope_section [11, 11, 10]): an
image token at merged (row, col) is rotated by (start, start + row,
start + col), the text after the image resumes at start + max(rows, cols),
and every later position carries that offset (HF's rope_deltas). The
engine follows Qwen3_5Model.get_rope_index exactly; with plain 1D
positions the tiny oracle below loses 10 of 16 tokens.
An image reaches it the way it reaches the other vision engines: a path in
a coli chat message, an attachment in coli web, or an image_url part
on /v1/chat/completions. One image per request. Q36_MAX_IMAGE_TOKENS
caps the tokens an image costs (the preprocessor's own ceiling is far above
what a CPU prefill wants); the image is shrunk, not cropped.
tools/make_qwen36_vl_tiny.py builds a toy Qwen3_5ForConditionalGeneration
and a reference from transformers with one 4 x 8-patch image; the engine
matches it token for token, and tests/test_qwen36_vision_serve.py holds the
IMAGE frame path to the same tokens. Qwen3.6-35B carries the same tower, so a
35B container converted with this converter gets images too; only the 27B has
been run with real pictures.
Not yet: the CUDA tier (a dense checkpoint runs on the CPU), the MTP head (skipped by the converter), video, and an int4 container on disk.
Qwen3-Coder-30B-A3B
Qwen/Qwen3-Coder-30B-A3B-Instruct (Apache-2.0) is Qwen3MoeForCausalLM: 48
attention layers and no DeltaNet, no attention output gate, no shared expert,
rotary over the whole head (rope_theta 1e7), plain RMSNorm weights, and 128
experts top-8 renormalized (norm_topk_prob). The converter recognises
model_type: qwen3_moe and writes exactly that into qwen36_meta.json
(all-attention layer_types, shared_inter: 0, partial_rotary_factor: 1.0,
zero_centered_norms: false); the engine then skips the shared expert and
passes the attention output ungated. The REAP prunes are the same
architecture with fewer experts and resolve to the same family under their
own name. The chat template is its own: tools as XML, calls as
<tool_call><function=...><parameter=...>, and no thinking at all. The gateway
recognises it from the template, renders it byte for byte
(tests/test_qwen3_coder_chat_template.py holds it to the release's
chat_template.jinja), and keeps enable_thinking off. The API model id is
qwen3-coder-30b-a3b-colibri.
hf download Qwen/Qwen3-Coder-30B-A3B-Instruct --local-dir qwen3-coder # 61.1 GB, bf16
python3 tools/convert_qwen36.py --model qwen3-coder --out qwen3-coder-i4 --ebits 4 --gs 64
coli serve --model qwen3-coder-i4 --cap 128 # every expert in RAM
coli serve --model qwen3-coder-i4 --cap 32 # 6.5 GB resident
The int4 gs64 container is 19 GB and converts in under a minute; --ebits 8
gives a 30 GB int8 one. As everywhere on this engine, --cap sizes the
expert cache and --ram does not (see below).
Against the bf16 release, on a 325-token code question and answer, teacher forced (every position's logits; the reference reads the release one layer at a time in f32):
| experts | dense trunk | top-1 = bf16 | top-5 overlap | mean |Δ log p| | max |Δ log p| |
|---|---|---|---|---|---|
| int4 gs64 | int8 (the default) | 96.9% | 91.1% | 0.176 | 3.74 |
| int4 gs64 | f32 (COLI_DENSE_I8=0) |
96.0% | 91.7% | 0.134 | 2.59 |
| int8 | int8 | 95.1% | 95.2% | 0.098 | 2.13 |
| int8 | f32 | 98.1% | 97.8% | 0.022 | 0.60 |
Decode on a Ryzen 7 PRO 8700GE (8 cores, 64 GB, NVMe RAID), the CLI from a cold cache, 128 tokens, dense trunk int8:
| container | experts cached per layer | decode | resident |
|---|---|---|---|
| int4 gs64 | 128 (all) | 8.5 and 9.6 tok/s (two runs) | 15.2 GB |
| int8 | 128 (all) | 6.4 tok/s | 25.1 GB |
| int4 gs64 | 32 | 5.1 tok/s | 6.5 GB |
| int8 | 32 | 3.8 tok/s | 9.6 GB |
Through coli chat with --cap 128 and the cache warm, a 577-token answer
streams at 13 tok/s. Prefill is the slow part on the CPU: a request whose tool
block makes the prompt about 500 tokens takes around two minutes from a cold
cache.

The converter's tensor contract
Both checkpoints ship experts fused per layer (mlp.experts.gate_up_proj,
mlp.experts.down_proj), a one-layer multi-token-prediction head (mtp.*,
mtp_num_hidden_layers: 1), and the 35B additionally a vision tower
(visual.*). tools/qwen36_tensor_kinds.py classifies every tensor name
before the first shard is read: layer tensors are converted, mtp.* and
visual.* are skipped on purpose and reported with a count, and a name
the contract does not know stops the conversion. A converter that silently
drops what it does not recognise produces a container that loads and is
quietly missing a tensor; this one refuses instead (the GLM-5.3 precedent).
tests/test_qwen36_tensor_kinds.py pins the contract to both real indexes.
Validating the 2.4T shape without a single weight
tools/make_qwen36_tiny.py --geometry qwen38-2p4t builds a fixture with the
2.4T's structural numbers at toy widths -- 92 layers, interval 4, 512 experts
top-10, 16:1 attention heads, 8:1 DeltaNet heads -- and rewrites the shard
into the real layout: fused experts plus an mtp.* head. The converter must
split the one and skip the other, and the engine must match the transformers
reference token for token. CI runs it at cache capacities 1, 2 and 512, and
under ASan/UBSan. Locally:
cd c && make qwen36
python3 tools/make_qwen36_tiny.py --geometry qwen38-2p4t --seed 3 \
--out q24 --ref-mode full --emit-ref q24/ref_full.json
python3 tools/convert_qwen36.py --model q24 --out q24_c --ebits 8
COLI_DENSE_I8=0 SNAP=q24_c ./qwen36 512 8 q24/ref_full.json
(--seed 3: the default seed collapses this geometry's reference to one
repeated token, which a shape error could still reproduce; seed 3 yields
twelve distinct tokens over sixteen.)
--ram is not honoured by this engine
The engine reads no RAM_GB: grep -c RAM_GB c/qwen36.c returns 0, and passing
--ram changes nothing. Said here rather than left to be discovered, because a
flag that appears to work and does not is worse than one documented as
unsupported.
What sizes the expert cache instead depends on where the experts live. On the
CPU path they stream from the container on demand, through the LRU described at
the top of this page, and --cap N is the direct lever on how many slots per
layer that cache holds. On the CUDA expert tier (#713) the hot set is resident
in VRAM and CUDA_EXPERT_GB decides its size. Neither path consults the RAM
budget.
(Earlier revisions of this section said qwen36 does not stream experts at all,
which contradicted the description at the top of the page and was wrong for the
CPU path: c/qwen36.c reads experts on demand with pread plus
posix_fadvise(DONTNEED) and caches them LRU. Reported in #1444.)