Run frontier MoE models on hardware you already own — pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦
C
qwen36: images for the family's vision checkpoints (Qwen3.8-27B)
JustVugg73fe2cb23a8d
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.ymlindex 70d21228..3791874e 100644--- a/.github/workflows/ci.yml+++ b/.github/workflows/ci.yml@@ -527,6 +527,23 @@ jobs:./qwen36 "$cap" 8 qwen38_27b_tiny/ref_full.jsonecho "::endgroup::"done+ # The same model with its vision tower (Qwen3_5ForConditionalGeneration):+ # a prompt with one 4x8-patch image, token-exact against transformers.+ # Every position after the image depends on the 3D M-RoPE positions+ # (1D positions give 6/16 here), and the serve test holds the IMAGE+ # frame path to the same tokens.+ python3 tools/make_qwen36_vl_tiny.py --out qwen38_27b_vl_tiny+ python3 tools/convert_qwen36.py --model qwen38_27b_vl_tiny \+ --out qwen38_27b_vl_tiny_c --ebits 8 | tee convert_vl.log+ grep -q "\[vision\] model-vision.safetensors" convert_vl.log+ for cap in 1 8; do+ echo "::group::27B vision cap=$cap"+ COLI_DENSE_I8=0 SNAP=qwen38_27b_vl_tiny_c \+ ./qwen36 "$cap" 8 qwen38_27b_vl_tiny/ref.json+ echo "::endgroup::"+ done+ QWEN36_VL_TINY=qwen38_27b_vl_tiny_c QWEN36_VL_REF=qwen38_27b_vl_tiny/ref.json \+ python3 -m unittest -v tests.test_qwen36_vision_serve- name: Same run under ASan + UBSanrun: |cd cdiff --git a/README.md b/README.mdindex f3e75e4a..0dc81719 100644--- a/README.md+++ b/README.md@@ -453,7 +453,7 @@ the model's `config.json`):> | **Kimi K3** | ~1.6 TB | 32 GB+ | not needed |> | **DeepSeek V4 Flash** | ~167 GB (REAP 150B: ~85 GB) | 16 GB min, 32 GB comfortable | optional; any NVIDIA card from the GTX 10 series up (Pascal/Turing via `CUDA_ARCH=portable-pre-ampere NO_TC=1`, best on RTX 50) makes prefill 5-10x and decode ~2.5x faster |> | **Qwen3.8-Flash-Next** | ~185.5 GB (official FP8 checkpoint) | 16 GB min, 24 GB comfortable at the default context | optional; the CUDA VRAM expert tier with dense trunk quantized to int8 in VRAM |-> | **Qwen3.8-27B** (dense) | ~51 GB converted (f16) | 20 GB with int4 dense weights, 30 GB in int8 | not used yet (CPU) |+> | **Qwen3.8-27B** (dense, text and images) | ~51 GB converted (f16) | 20 GB with int4 dense weights, 30 GB in int8 | not used yet (CPU) |> | **Qwen3.6-35B-A3B** | ~20 GB (int4-gs64 container) | 24 GB (needs full RAM residency) | optional; the CUDA VRAM expert tier measured **1.44 -> 10.05 tok/s (7.0x)** on two 8 GB cards, output bit-identical to CPU |>> A GPU only ever makes it faster. Speed is set by your disk, because the expertsdiff --git a/c/Makefile b/c/Makefileindex 495b2017..35ff646d 100644--- a/c/Makefile+++ b/c/Makefile@@ -1185,7 +1185,7 @@ qwen36: qwen36$(EXE)endif# Rebuild when CUDA_DLL changes; otherwise an existing CPU-only executable can# be reported as up to date despite selecting the Windows CUDA DLL tier.-qwen36$(EXE): qwen36.c sse41_kernels.h decode_batch.h serve_poll.h cli_args.h qwen36_tier.h expert_ffn.h idot.h st.h json.h compat.h omp_tune.h kv_prefix.h pin_pool.h edge_adapter_internal.h edge_adapters.h edge_runtime.h segment_adapter_internal.h segment_adapters.h segment_runtime.h gsgemv.h qgemv.h $(QWEN36_TIER_SRC) $(CUDA_OBJ) .build-config+qwen36$(EXE): qwen36.c qwen38_vision.h sse41_kernels.h decode_batch.h serve_poll.h cli_args.h qwen36_tier.h expert_ffn.h idot.h st.h json.h compat.h omp_tune.h kv_prefix.h pin_pool.h edge_adapter_internal.h edge_adapters.h edge_runtime.h segment_adapter_internal.h segment_adapters.h segment_runtime.h gsgemv.h qgemv.h $(QWEN36_TIER_SRC) $(CUDA_OBJ) .build-config$(CC) $(QWEN36_CFLAGS) qwen36.c $(QWEN36_TIER_SRC) $(CUDA_OBJ) -o qwen36$(EXE) $(QWEN36_LDFLAGS)# DeepSeek V4.1 Flash: one file, like every other portable engine. The fp4 experts@@ -1254,48 +1254,48 @@ qwen38-tier-engine-check: qwen38-tiny-fp8-generate tests/test_qwen38_tier_engine# Same tier sources as the engine: the test includes qwen36.c, so with CUDA=1# it needs qwen36_tier.c and the backend object too (without CUDA, the header's# inline stubs cover it and both are empty).-tests/test_qwen36_ctx$(EXE): tests/test_qwen36_ctx.c gsgemv.h qgemv.h sse41_kernels.h qwen36.c expert_ffn.h qwen36_tier.h st.h json.h compat.h $(QWEN36_TIER_SRC) $(CUDA_OBJ)+tests/test_qwen36_ctx$(EXE): tests/test_qwen36_ctx.c gsgemv.h qgemv.h sse41_kernels.h qwen36.c qwen38_vision.h expert_ffn.h qwen36_tier.h st.h json.h compat.h $(QWEN36_TIER_SRC) $(CUDA_OBJ)$(CC) $(CFLAGS) $< $(QWEN36_TIER_SRC) $(CUDA_OBJ) -o $@ $(LDFLAGS)# CACHE_ROUTE: route_select() against a residency table, no model needed.-tests/test_qwen36_cache_route$(EXE): tests/test_qwen36_cache_route.c gsgemv.h qgemv.h sse41_kernels.h qwen36.c expert_ffn.h qwen36_tier.h st.h json.h compat.h $(QWEN36_TIER_SRC) $(CUDA_OBJ)+tests/test_qwen36_cache_route$(EXE): tests/test_qwen36_cache_route.c gsgemv.h qgemv.h sse41_kernels.h qwen36.c qwen38_vision.h expert_ffn.h qwen36_tier.h st.h json.h compat.h $(QWEN36_TIER_SRC) $(CUDA_OBJ)$(CC) $(CFLAGS) $< $(QWEN36_TIER_SRC) $(CUDA_OBJ) -o $@ $(LDFLAGS)-tests/test_qwen36_dense_batch$(EXE): tests/test_qwen36_dense_batch.c gsgemv.h qgemv.h sse41_kernels.h qwen36.c expert_ffn.h qwen36_tier.h st.h json.h compat.h $(QWEN36_TIER_SRC) $(CUDA_OBJ)+tests/test_qwen36_dense_batch$(EXE): tests/test_qwen36_dense_batch.c gsgemv.h qgemv.h sse41_kernels.h qwen36.c qwen38_vision.h expert_ffn.h qwen36_tier.h st.h json.h compat.h $(QWEN36_TIER_SRC) $(CUDA_OBJ)$(CC) $(QWEN36_CFLAGS) $< $(QWEN36_TIER_SRC) $(CUDA_OBJ) -o $@ $(QWEN36_LDFLAGS)-tests/test_qwen36_json_escape$(EXE): tests/test_qwen36_json_escape.c gsgemv.h qgemv.h sse41_kernels.h qwen36.c expert_ffn.h qwen36_tier.h st.h json.h compat.h $(QWEN36_TIER_SRC) $(CUDA_OBJ)+tests/test_qwen36_json_escape$(EXE): tests/test_qwen36_json_escape.c gsgemv.h qgemv.h sse41_kernels.h qwen36.c qwen38_vision.h expert_ffn.h qwen36_tier.h st.h json.h compat.h $(QWEN36_TIER_SRC) $(CUDA_OBJ)$(CC) $(QWEN36_CFLAGS) $< $(QWEN36_TIER_SRC) $(CUDA_OBJ) -o $@ $(QWEN36_LDFLAGS)# Both tokenizer.json merge spellings ("a b" strings and ["a","b"] pairs)# must index the same merge table; the pair form is what Qwen3.6 ships.-tests/test_qwen36_tok_merges$(EXE): tests/test_qwen36_tok_merges.c gsgemv.h qgemv.h sse41_kernels.h qwen36.c expert_ffn.h qwen36_tier.h st.h json.h compat.h $(QWEN36_TIER_SRC) $(CUDA_OBJ)+tests/test_qwen36_tok_merges$(EXE): tests/test_qwen36_tok_merges.c gsgemv.h qgemv.h sse41_kernels.h qwen36.c qwen38_vision.h expert_ffn.h qwen36_tier.h st.h json.h compat.h $(QWEN36_TIER_SRC) $(CUDA_OBJ)$(CC) $(QWEN36_CFLAGS) $< $(QWEN36_TIER_SRC) $(CUDA_OBJ) -o $@ $(QWEN36_LDFLAGS)# A byte-counted serving payload may end mid-character; the pre-tokenizer must# not read past it. qwen38 already gates this (tests/test_qwen38_tokenizer.c).-tests/test_qwen36_tok_truncated$(EXE): tests/test_qwen36_tok_truncated.c gsgemv.h qgemv.h sse41_kernels.h qwen36.c expert_ffn.h qwen36_tier.h st.h json.h compat.h $(QWEN36_TIER_SRC) $(CUDA_OBJ)+tests/test_qwen36_tok_truncated$(EXE): tests/test_qwen36_tok_truncated.c gsgemv.h qgemv.h sse41_kernels.h qwen36.c qwen38_vision.h expert_ffn.h qwen36_tier.h st.h json.h compat.h $(QWEN36_TIER_SRC) $(CUDA_OBJ)$(CC) $(QWEN36_CFLAGS) $< $(QWEN36_TIER_SRC) $(CUDA_OBJ) -o $@ $(QWEN36_LDFLAGS)# push_id / bpe_piece must refuse loudly, not crash, on a failed realloc during growth.-tests/test_qwen36_encode_oom$(EXE): tests/test_qwen36_encode_oom.c gsgemv.h qgemv.h sse41_kernels.h qwen36.c expert_ffn.h qwen36_tier.h st.h json.h compat.h $(QWEN36_TIER_SRC) $(CUDA_OBJ)+tests/test_qwen36_encode_oom$(EXE): tests/test_qwen36_encode_oom.c gsgemv.h qgemv.h sse41_kernels.h qwen36.c qwen38_vision.h expert_ffn.h qwen36_tier.h st.h json.h compat.h $(QWEN36_TIER_SRC) $(CUDA_OBJ)$(CC) $(QWEN36_CFLAGS) $< $(QWEN36_TIER_SRC) $(CUDA_OBJ) -o $@ $(QWEN36_LDFLAGS)# slot_ensure_int8's wiring onto #1271's unpack_int4_to_int8: regression check# that its output matches the old separate scalar nibble-unpack loop it replaced.-tests/test_qwen36_slot_int8$(EXE): tests/test_qwen36_slot_int8.c qwen36.c qwen36_tier.h st.h json.h compat.h $(QWEN36_TIER_SRC) $(CUDA_OBJ)+tests/test_qwen36_slot_int8$(EXE): tests/test_qwen36_slot_int8.c qwen36.c qwen38_vision.h qwen36_tier.h st.h json.h compat.h $(QWEN36_TIER_SRC) $(CUDA_OBJ)$(CC) $(QWEN36_CFLAGS) $< $(QWEN36_TIER_SRC) $(CUDA_OBJ) -o $@ $(QWEN36_LDFLAGS)# the dense trunk's integer path: quantizer contract, int4 planar packing, dispatch.-tests/test_qwen36_dense_idot$(EXE): tests/test_qwen36_dense_idot.c gsgemv.h qgemv.h sse41_kernels.h qwen36.c expert_ffn.h qwen36_tier.h st.h json.h compat.h $(QWEN36_TIER_SRC) $(CUDA_OBJ)+tests/test_qwen36_dense_idot$(EXE): tests/test_qwen36_dense_idot.c gsgemv.h qgemv.h sse41_kernels.h qwen36.c qwen38_vision.h expert_ffn.h qwen36_tier.h st.h json.h compat.h $(QWEN36_TIER_SRC) $(CUDA_OBJ)$(CC) $(QWEN36_CFLAGS) $< $(QWEN36_TIER_SRC) $(CUDA_OBJ) -o $@ $(QWEN36_LDFLAGS)# #1653: added tokens are split out before the regex pre-tokenizer, HF-style.-tests/test_qwen36_tokenizer$(EXE): tests/test_qwen36_tokenizer.c gsgemv.h qgemv.h sse41_kernels.h qwen36.c expert_ffn.h qwen36_tier.h st.h json.h compat.h $(QWEN36_TIER_SRC) $(CUDA_OBJ)+tests/test_qwen36_tokenizer$(EXE): tests/test_qwen36_tokenizer.c gsgemv.h qgemv.h sse41_kernels.h qwen36.c qwen38_vision.h expert_ffn.h qwen36_tier.h st.h json.h compat.h $(QWEN36_TIER_SRC) $(CUDA_OBJ)$(CC) $(QWEN36_CFLAGS) $< $(QWEN36_TIER_SRC) $(CUDA_OBJ) -o $@ $(QWEN36_LDFLAGS)# Reproducible local timing evidence; intentionally not a noisy CI perf gate.-tests/bench_qwen36_dense_batch$(EXE): tests/bench_qwen36_dense_batch.c gsgemv.h qgemv.h sse41_kernels.h qwen36.c qwen36_tier.h st.h json.h compat.h $(QWEN36_TIER_SRC) $(CUDA_OBJ)+tests/bench_qwen36_dense_batch$(EXE): tests/bench_qwen36_dense_batch.c gsgemv.h qgemv.h sse41_kernels.h qwen36.c qwen38_vision.h qwen36_tier.h st.h json.h compat.h $(QWEN36_TIER_SRC) $(CUDA_OBJ)$(CC) $(QWEN36_CFLAGS) $< $(QWEN36_TIER_SRC) $(CUDA_OBJ) -o $@ $(QWEN36_LDFLAGS)inkling$(EXE): inkling.c cli_args.h st.h json.h tok.h tok_unicode.h tok_unicode_deepseek.h tok_unicode_o200k.h compat.h omp_tune.h route_trace.h kv_prefix.h pin_pool.h serve_codec.h serve_budget.h backend_cuda_ink.h backend_metal.h edge_adapters.h edge_runtime.h edge_tok_internal.h segment_adapter_internal.h segment_adapters.h segment_runtime.h $(INK_CUDA_OBJ) $(METAL_OBJ)@@ -1620,7 +1620,7 @@ tests/test_olmoe_serve_framing$(EXE): tests/test_olmoe_serve_framing.c serve_budtests/test_olmoe_cache_index$(EXE): tests/test_olmoe_cache_index.c sse41_kernels.h olmoe.c st.h json.h compat.h sample.h tok.h tok_unicode.h tok_unicode_deepseek.h tok_unicode_o200k.h omp_tune.h route_trace.h serve_codec.h$(CC) $(NOCUDA_CFLAGS) $< -o $@ $(NOCUDA_LDFLAGS)-tests/test_qwen36_cache_index$(EXE): tests/test_qwen36_cache_index.c gsgemv.h qgemv.h sse41_kernels.h qwen36.c expert_ffn.h st.h json.h compat.h sample.h tok.h tok_unicode.h tok_unicode_deepseek.h tok_unicode_o200k.h omp_tune.h route_trace.h serve_codec.h qwen36_tier.h $(QWEN36_TIER_SRC) $(CUDA_OBJ)+tests/test_qwen36_cache_index$(EXE): tests/test_qwen36_cache_index.c gsgemv.h qgemv.h sse41_kernels.h qwen36.c qwen38_vision.h expert_ffn.h st.h json.h compat.h sample.h tok.h tok_unicode.h tok_unicode_deepseek.h tok_unicode_o200k.h omp_tune.h route_trace.h serve_codec.h qwen36_tier.h $(QWEN36_TIER_SRC) $(CUDA_OBJ)$(CC) $(QWEN36_CFLAGS) $< $(QWEN36_TIER_SRC) $(CUDA_OBJ) -o $@ $(QWEN36_LDFLAGS)tests/test_idot$(EXE): tests/test_idot.c sse41_kernels.h colibri.c oracle.h st.h uring.h json.h tok.h tok_unicode.h tok_unicode_deepseek.h compat.h grammar.h tier.h quant.h fp8_format.h idot.h sample.h kv_persist.h telemetry.h route_trace.h $(VK_OBJ)@@ -2091,22 +2091,22 @@ tests/test_qwen36_tier_dense$(EXE): tests/test_qwen36_tier_dense.c tests/qwen36_$(CC) $(CFLAGS) -DCOLI_CUDA $< -o $@ $(LDFLAGS)# DeltaNet input projection batching, state parity and block failure fallback.-tests/test_qwen36_dnproj_batch$(EXE): tests/test_qwen36_dnproj_batch.c gsgemv.h qgemv.h sse41_kernels.h tests/qwen36_fake_cuda.h qwen36.c qwen36_tier.c qwen36_tier.h backend_cuda.h compat.h quant.h+tests/test_qwen36_dnproj_batch$(EXE): tests/test_qwen36_dnproj_batch.c gsgemv.h qgemv.h sse41_kernels.h tests/qwen36_fake_cuda.h qwen36.c qwen38_vision.h qwen36_tier.c qwen36_tier.h backend_cuda.h compat.h quant.h$(CC) $(CFLAGS) -DCOLI_CUDA $< -o $@ $(LDFLAGS)# the automatic trunk placement can be withdrawn after the startup probe (fake backend).tests/test_qwen36_tier_withdraw$(EXE): tests/test_qwen36_tier_withdraw.c tests/qwen36_fake_cuda.h qwen36_tier.c qwen36_tier.h backend_cuda.h$(CC) $(CFLAGS) -DCOLI_CUDA $< -o $@ $(LDFLAGS)-tests/test_qwen36_tier_int8_engine$(EXE): tests/test_qwen36_tier_int8_engine.c gsgemv.h qgemv.h sse41_kernels.h tests/qwen36_fake_cuda.h qwen36.c expert_ffn.h qwen36_tier.c qwen36_tier.h backend_cuda.h cli_args.h st.h json.h compat.h+tests/test_qwen36_tier_int8_engine$(EXE): tests/test_qwen36_tier_int8_engine.c gsgemv.h qgemv.h sse41_kernels.h tests/qwen36_fake_cuda.h qwen36.c qwen38_vision.h expert_ffn.h qwen36_tier.c qwen36_tier.h backend_cuda.h cli_args.h st.h json.h compat.h$(CC) $(CFLAGS) -DCOLI_CUDA $< -o $@ $(LDFLAGS)# dnout / attnproj / shexp offered, placed and served from VRAM (fake backend).-tests/test_qwen36_trunk_dense$(EXE): tests/test_qwen36_trunk_dense.c gsgemv.h qgemv.h sse41_kernels.h tests/qwen36_fake_cuda.h qwen36.c expert_ffn.h qwen36_tier.c qwen36_tier.h backend_cuda.h cli_args.h st.h json.h compat.h+tests/test_qwen36_trunk_dense$(EXE): tests/test_qwen36_trunk_dense.c gsgemv.h qgemv.h sse41_kernels.h tests/qwen36_fake_cuda.h qwen36.c qwen38_vision.h expert_ffn.h qwen36_tier.c qwen36_tier.h backend_cuda.h cli_args.h st.h json.h compat.h$(CC) $(CFLAGS) -DCOLI_CUDA $< -o $@ $(LDFLAGS)# #1391: il decode path deve offrire gli esperti int8 al tier, non solo il warmstart-tests/test_qwen36_tier_int8_decode$(EXE): tests/test_qwen36_tier_int8_decode.c gsgemv.h qgemv.h sse41_kernels.h tests/qwen36_fake_cuda.h qwen36.c expert_ffn.h qwen36_tier.c qwen36_tier.h backend_cuda.h cli_args.h st.h json.h compat.h+tests/test_qwen36_tier_int8_decode$(EXE): tests/test_qwen36_tier_int8_decode.c gsgemv.h qgemv.h sse41_kernels.h tests/qwen36_fake_cuda.h qwen36.c qwen38_vision.h expert_ffn.h qwen36_tier.c qwen36_tier.h backend_cuda.h cli_args.h st.h json.h compat.h$(CC) $(CFLAGS) -DCOLI_CUDA $< -o $@ $(LDFLAGS)# The qwen38 engine through its own main() on the fake backend: tier startdiff --git a/c/openai_server.py b/c/openai_server.pyindex c7928f8d..a7ccf782 100644--- a/c/openai_server.py+++ b/c/openai_server.py@@ -2063,6 +2063,19 @@ def expand_qwen38_images(messages, model_dir, max_tokens=None):return rewritten, images+def qwen36_has_vision(model_dir):+ """Whether a qwen36 container carries its vision tower (#1757). The converter+ writes the tower's shape into qwen36_meta.json only when it copied the weights;+ older containers, and text-only checkpoints, have none."""+ if not model_dir:+ return False+ try:+ meta = json.loads((Path(model_dir) / "qwen36_meta.json").read_text(encoding="utf-8"))+ except (OSError, ValueError):+ return False+ return isinstance(meta, dict) and isinstance(meta.get("vision"), dict)++def expand_glm53_images(messages, model_dir):"""Sostituisce le parti immagine coi loro segnaposto e ne estrae le patch.@@ -5303,8 +5316,12 @@ class APIHandler(BaseHTTPRequestHandler):raise APIError(400, "one image per request for now; the engine ""holds a single pending image.", "messages")image = images[0] if images else None- elif ARCH == "qwen38":- ceiling = os.environ.get("Q38_MAX_IMAGE_TOKENS")+ elif ARCH == "qwen38" or (ARCH == "qwen36" and qwen36_has_vision(+ getattr(self.server.engine, "model_dir", None))):+ # Qwen3.5/3.6/3.8 share the tower and the preprocessor, so qwen36+ # checkpoints converted with their tower take the same path (#1757).+ ceiling = os.environ.get("Q38_MAX_IMAGE_TOKENS" if ARCH == "qwen38"+ else "Q36_MAX_IMAGE_TOKENS")messages, images = expand_qwen38_images(messages, getattr(self.server.engine, "model_dir", None),int(ceiling) if ceiling else None)diff --git a/c/qwen36.c b/c/qwen36.cindex 7a4ffb2d..78d40495 100644--- a/c/qwen36.c+++ b/c/qwen36.c@@ -68,6 +68,7 @@ static int qwen36_max_ctx(void) {#include "qwen36_tier.h" /* optional CUDA VRAM expert tier */#include "expert_ffn.h" /* routed experts: planar int4 kernel + layer runner */#include "idot.h" /* integer dot kernels for the dense trunk (COLI_DENSE_IDOT, COLI_DENSE_BITS) */+#include "qwen38_vision.h" /* the ViT: the same tower in Qwen3.5/3.6/3.8 (#1757) */#ifdef COLI_SEGMENT_ADAPTER#include "segment_runtime.h"#include "segment_adapters.h"@@ -689,6 +690,16 @@ typedef struct {* per expert, [gate int4 packed | up int4 packed | down int8], 2*inter*hidden* bytes -- told apart from int4 (1.5x) and int8 (3x) by size, like today. */int expert_down_bits, expert_down_gs;+ /* Vision (#1757). The tower is the same in Qwen3.5, 3.6 and 3.8 -- only its+ * output width follows the text model -- so the qwen38 one is reused. 0 depth+ * = text only (a container converted without it, or a text-only checkpoint). */+ int vis_depth, vis_hidden, vis_heads, vis_inter, vis_patch, vis_merge;+ int vis_temporal, vis_in_ch, vis_out_hidden, vis_num_pos;+ int image_token;+ /* Interleaved M-RoPE: frequency j rotates by the height position when+ * j % 3 == 1 and j < 3*sec[1], by the width one when j % 3 == 2 and+ * j < 3*sec[2], else by the temporal one. Text has all three equal. */+ int mrope_section[3];} Cfg;/* ---- Dense int8: a dense matrix that is quantized to int8 during load@@ -782,6 +793,14 @@ typedef struct {int resident_mode; /* 0 off; 1 pin this-prompt experts (CPU no-evict -> GPU resident) */int resident_collecting; /* prefill in progress, collecting routed experts */int first_step; /* the first step() call is the prefill */+ /* Vision, for the turn that carries an image. vis_map maps an ABSOLUTE+ * position to a row of vis_rows or -1; mpos holds the (t, h, w) rope+ * positions of the prompt, and past it every axis is pos + rope_delta+ * (HF's mrope_position_deltas). All of it is dropped after the turn. */+ Q38Vision vis; int vis_ready;+ float *vis_rows; int vis_rows_n;+ int *vis_map, vis_map_len;+ int *mpos, mpos_len, rope_delta;} Model;static pthread_mutex_t g_pilot_mx = PTHREAD_MUTEX_INITIALIZER;@@ -1350,6 +1369,8 @@ static void load_cfg(Cfg *c, const char *snap) {c->n_experts = 256; c->topk = 8; c->inter = 512; c->shared_inter = 512;c->n_group = 1; c->topk_group = 1; c->norm_topk = 1; c->has_qk_norm = 1; c->has_bias = 0;c->attn_output_gate = 1; c->n_active = 0;+ c->vis_depth = 0; c->image_token = -1;+ c->mrope_section[0] = 11; c->mrope_section[1] = 11; c->mrope_section[2] = 10;if (c->n_layers <= 0 || c->n_layers > 512) { fprintf(stderr, "load_cfg: n_layers=%d out of range 1..512\n", c->n_layers); exit(1); }c->is_attn = calloc((size_t)c->n_layers, sizeof(uint8_t));for (int i = 0; i < c->n_layers; i++) c->is_attn[i] = (i % 4 == 3) ? 1 : 0;@@ -1401,6 +1422,25 @@ static void validate_cfg(const Cfg *c, int n_layers_from_config) {CFG_NEED(c->inter > 0 && c->shared_inter > 0,"moe_inter %d / shared_inter %d must be positive", c->inter, c->shared_inter);}+ if (c->vis_depth) {+ CFG_NEED(c->vis_depth > 0 && c->vis_depth <= 64, "vision depth %d out of range", c->vis_depth);+ CFG_NEED(c->vis_hidden > 0 && c->vis_hidden <= 8192 && c->vis_heads > 0 &&+ c->vis_hidden % c->vis_heads == 0, "vision hidden %d / heads %d", c->vis_hidden, c->vis_heads);+ CFG_NEED(c->vis_inter > 0 && c->vis_inter <= 65536, "vision inter %d", c->vis_inter);+ CFG_NEED(c->vis_patch > 0 && c->vis_patch <= 64 && c->vis_merge > 0 && c->vis_merge <= 8 &&+ c->vis_temporal > 0 && c->vis_temporal <= 8 && c->vis_in_ch > 0 && c->vis_in_ch <= 8,+ "vision patch %d merge %d temporal %d channels %d", c->vis_patch, c->vis_merge,+ c->vis_temporal, c->vis_in_ch);+ CFG_NEED(c->vis_num_pos > 0 && c->vis_num_pos <= 65536, "vision positions %d", c->vis_num_pos);+ CFG_NEED(c->vis_out_hidden == c->hidden, "vision tower writes %d wide rows into a %d-wide model",+ c->vis_out_hidden, c->hidden);+ CFG_NEED(c->image_token >= 0 && c->image_token < c->vocab, "image_token_id %d outside the vocabulary",+ c->image_token);+ CFG_NEED(c->mrope_section[0] >= 0 && c->mrope_section[1] >= 0 && c->mrope_section[2] >= 0 &&+ 2 * (c->mrope_section[0] + c->mrope_section[1] + c->mrope_section[2]) == c->rotary_dim,+ "mrope_section %d+%d+%d does not cover rotary_dim %d", c->mrope_section[0],+ c->mrope_section[1], c->mrope_section[2], c->rotary_dim);+ }CFG_NEED(c->q_heads > 0 && c->kv_heads > 0 && c->head_dim > 0,"attention dims q_heads=%d kv_heads=%d head_dim=%d must be positive",c->q_heads, c->kv_heads, c->head_dim);@@ -1473,6 +1513,21 @@ static void load_meta(Cfg *c, const char *snap) {if((v=json_get(r,"norm_topk_prob"))&&v->t==J_BOOL) c->norm_topk=v->boolean;if((v=json_get(r,"has_bias"))&&v->t==J_BOOL) c->has_bias=v->boolean;if((v=json_get(r,"has_qk_norm"))&&v->t==J_BOOL) c->has_qk_norm=v->boolean;+ if((v=json_get(r,"image_token_id"))&&v->t==J_NUM) c->image_token=(int)v->num;+ if((v=json_get(r,"mrope_section"))&&v->t==J_ARR&&v->len==3)+ for(int k=0;k<3;k++) if(v->kids[k]->t==J_NUM) c->mrope_section[k]=(int)v->kids[k]->num;+ { /* the tower, written by the converter only when it copied the weights */+ jval *vv = json_get(r,"vision");+ if (vv && vv->t==J_OBJ) {+ jval *w;+ #define GV(name,field) if((w=json_get(vv,name))&&w->t==J_NUM) c->field=(int)w->num+ GV("depth",vis_depth); GV("hidden",vis_hidden); GV("heads",vis_heads);+ GV("inter",vis_inter); GV("patch",vis_patch); GV("merge",vis_merge);+ GV("temporal",vis_temporal); GV("in_ch",vis_in_ch);+ GV("out_hidden",vis_out_hidden); GV("num_pos",vis_num_pos);+ #undef GV+ }+ }/* derive rotary_dim from head_dim * partial_rotary_factor (HF formula) */if (c->partial_rotary_factor > 0.f)c->rotary_dim = (int)(c->head_dim * c->partial_rotary_factor + 0.5f);@@ -1542,6 +1597,118 @@ static void load_tq(Model *m, const char *name, int I, int O, int quantize, consif (getenv("COLI_KEEP_F32")) out->w = p; else { free(p); out->w = NULL; }}+/* ---------- vision (#1757) ----------+ * The weights are the checkpoint's own model.visual.*, copied by the converter,+ * read as f32 like every other small tensor here. */+static void q36_vis_linear(Model *m, Q38Linear *l, const char *stem, int out, int in) {+ char nm[256];+ snprintf(nm, sizeof nm, "model.visual.%s.weight", stem);+ l->w = load_t_n(m, nm, (int64_t)out * in);+ snprintf(nm, sizeof nm, "model.visual.%s.bias", stem);+ l->b = st_has(&m->S, nm) ? load_t_n(m, nm, out) : NULL;+ l->out = out; l->in = in;+}+static void q36_vis_norm(Model *m, Q38Norm *n, const char *stem, int width) {+ char nm[256];+ snprintf(nm, sizeof nm, "model.visual.%s.weight", stem); n->w = load_t_n(m, nm, width);+ snprintf(nm, sizeof nm, "model.visual.%s.bias", stem); n->b = load_t_n(m, nm, width);+}+static void q36_load_vision(Model *m) {+ Cfg *c = &m->c;+ if (!c->vis_depth) return;+ if (!st_has(&m->S, "model.visual.pos_embed.weight")) {+ fprintf(stderr, "[qwen36] qwen36_meta.json describes a vision tower but the container has none; text only\n");+ c->vis_depth = 0; return;+ }+ Q38Vision *v = &m->vis;+ memset(v, 0, sizeof *v);+ v->depth = c->vis_depth; v->hidden = c->vis_hidden; v->heads = c->vis_heads;+ v->head_dim = c->vis_hidden / c->vis_heads; v->inter = c->vis_inter;+ v->patch = c->vis_patch; v->merge = c->vis_merge; v->temporal = c->vis_temporal;+ v->in_ch = c->vis_in_ch; v->out_hidden = c->vis_out_hidden;+ v->num_pos = c->vis_num_pos; v->side = (int)(sqrt((double)c->vis_num_pos) + 0.5);+ v->eps = 1e-6f;+ if (v->side * v->side != v->num_pos) {+ fprintf(stderr, "[qwen36] vision num_position_embeddings %d is not a square grid -- refusing\n", v->num_pos);+ exit(1);+ }+ q36_vis_linear(m, &v->patch_embed, "patch_embed.proj", v->hidden, v->in_ch * v->temporal * v->patch * v->patch);+ v->pos_embed = load_t_n(m, "model.visual.pos_embed.weight", (int64_t)v->num_pos * v->hidden);+ v->blocks = calloc((size_t)v->depth, sizeof(Q38VBlock));+ if (!v->blocks) { fprintf(stderr, "OOM vision blocks\n"); exit(1); }+ for (int i = 0; i < v->depth; i++) {+ char stem[96];+ snprintf(stem, sizeof stem, "blocks.%d.norm1", i); q36_vis_norm(m, &v->blocks[i].norm1, stem, v->hidden);+ snprintf(stem, sizeof stem, "blocks.%d.norm2", i); q36_vis_norm(m, &v->blocks[i].norm2, stem, v->hidden);+ snprintf(stem, sizeof stem, "blocks.%d.attn.qkv", i); q36_vis_linear(m, &v->blocks[i].qkv, stem, 3 * v->hidden, v->hidden);+ snprintf(stem, sizeof stem, "blocks.%d.attn.proj", i); q36_vis_linear(m, &v->blocks[i].proj, stem, v->hidden, v->hidden);+ snprintf(stem, sizeof stem, "blocks.%d.mlp.linear_fc1", i); q36_vis_linear(m, &v->blocks[i].fc1, stem, v->inter, v->hidden);+ snprintf(stem, sizeof stem, "blocks.%d.mlp.linear_fc2", i); q36_vis_linear(m, &v->blocks[i].fc2, stem, v->hidden, v->inter);+ }+ int wide = v->hidden * v->merge * v->merge;+ q36_vis_norm(m, &v->merger_norm, "merger.norm", v->hidden);+ q36_vis_linear(m, &v->merger_fc1, "merger.linear_fc1", wide, wide);+ q36_vis_linear(m, &v->merger_fc2, "merger.linear_fc2", v->out_hidden, wide);+ m->vis_ready = 1;+ fprintf(stderr, "[qwen36] vision tower: %d blocks, hidden %d, %d heads, patch %d, merge %d\n",+ v->depth, v->hidden, v->heads, v->patch, v->merge);+}++static void q36_vision_detach(Model *m) {+ free(m->vis_rows); free(m->vis_map); free(m->mpos);+ m->vis_rows = NULL; m->vis_map = NULL; m->mpos = NULL;+ m->vis_rows_n = m->vis_map_len = m->mpos_len = m->rope_delta = 0;+}++/* Run the tower on one image and lay its rows and rope positions over the prompt+ * `ids` (absolute positions 0..n-1). One image per prompt: its placeholders must+ * be one contiguous run of exactly as many tokens as the merged grid gives.+ * Positions follow HF Qwen3_5Model.get_rope_index: text counts up on all three+ * axes; the image's token (row, col) sits at (start, start+row, start+col); the+ * text after it resumes at start + max(rows, cols); past the prompt every axis+ * is pos + rope_delta, rope_delta = max position + 1 - n. */+static int q36_vision_attach(Model *m, const float *patches, int grid_h, int grid_w,+ const int *ids, int n) {+ Cfg *c = &m->c;+ if (!m->vis_ready || n <= 0) return -1;+ if (grid_h <= 0 || grid_w <= 0 || grid_h % c->vis_merge || grid_w % c->vis_merge) return -1;+ int lh = grid_h / c->vis_merge, lw = grid_w / c->vis_merge, tokens = lh * lw;+ int first = -1, slots = 0;+ for (int i = 0; i < n; i++) if (ids[i] == c->image_token) { if (first < 0) first = i; slots++; }+ if (slots != tokens || first < 0) {+ fprintf(stderr, "[qwen36] prompt has %d image placeholders but the grid gives %d tokens\n", slots, tokens);+ return -1;+ }+ for (int i = first; i < first + tokens; i++)+ if (ids[i] != c->image_token) {+ fprintf(stderr, "[qwen36] the image placeholders are not one contiguous run (one image per prompt)\n");+ return -1;+ }+ q36_vision_detach(m);+ m->vis_rows = calloc((size_t)tokens * c->hidden, sizeof(float));+ m->vis_map = malloc((size_t)n * sizeof(int));+ m->mpos = malloc((size_t)n * 3 * sizeof(int));+ if (!m->vis_rows || !m->vis_map || !m->mpos) { q36_vision_detach(m); return -1; }+ if (q38_vision_forward(&m->vis, patches, grid_h, grid_w, m->vis_rows) != tokens) { q36_vision_detach(m); return -1; }+ int cur = 0, maxpos = -1;+ for (int i = 0; i < n; i++) {+ int *pp = m->mpos + (size_t)i * 3;+ if (i >= first && i < first + tokens) {+ int k = i - first, start = cur;+ m->vis_map[i] = k;+ pp[0] = start; pp[1] = start + k / lw; pp[2] = start + k % lw;+ if (k == tokens - 1) cur = start + (lh > lw ? lh : lw);+ } else {+ m->vis_map[i] = -1;+ pp[0] = pp[1] = pp[2] = cur++;+ }+ for (int a = 0; a < 3; a++) if (pp[a] > maxpos) maxpos = pp[a];+ }+ m->vis_rows_n = tokens; m->vis_map_len = n; m->mpos_len = n;+ m->rope_delta = maxpos + 1 - n;+ return tokens;+}+static void model_init_range(Model *m, const char *snap, int cap, int bits,int layer_begin, int layer_end,int load_boundaries, int allocate_state) {@@ -1579,6 +1746,7 @@ static void model_init_range(Model *m, const char *snap, int cap, int bits,load_tq(m, "lm_head.weight", c->hidden, c->vocab, quantize_dense, "lmhead", &m->lm_head);if (m->lm_head.q || m->lm_head.q4) { qcount++; qfreed += (double)c->hidden * c->vocab * sizeof(float); }m->final_norm = load_t_n(m, "model.norm.weight", c->hidden);+ q36_load_vision(m);}m->L = calloc((size_t)c->n_layers, sizeof(Layer));/* Phase 2: the converter stores EVERY layer (Gated-Attention + Gated DeltaNet)@@ -2102,6 +2270,26 @@ static void rope_head_partial(float *x, int pos, int rope_dim, int head_dim, flo}}+/* The same rotation with a position per axis (interleaved M-RoPE, see Cfg).+ * With the three positions equal it is rope_head_partial, operation for+ * operation. */+static void rope_head_mrope(float *x, const int pos3[3], const int sec[3], int rope_dim, float theta) {+ int h = rope_dim / 2;+ for (int j = 0; j < h; j++) {+ int axis = (j % 3 == 1 && j < 3 * sec[1]) ? 1 : (j % 3 == 2 && j < 3 * sec[2]) ? 2 : 0;+ float inv = powf(theta, -2.0f * j / rope_dim);+ float ang = pos3[axis] * inv, cs = cosf(ang), sn = sinf(ang);+ float a = x[j], b = x[j+h];+ x[j] = a*cs - b*sn;+ x[j+h] = b*cs + a*sn;+ }+}+static inline void mrope_at(const Model *m, int p, int pos3[3]) {+ if (m->mpos && p >= 0 && p < m->mpos_len) {+ pos3[0] = m->mpos[(size_t)p * 3]; pos3[1] = m->mpos[(size_t)p * 3 + 1]; pos3[2] = m->mpos[(size_t)p * 3 + 2];+ } else pos3[0] = pos3[1] = pos3[2] = p + m->rope_delta;+}+/* Gated Attention (GQA) matching HF Qwen3_5MoeAttention:* - q_proj outputs query(head_dim) ++ attn_output_gate(head_dim); k/v are head_dim.* - per-head q/k RMSNorm (weight [head_dim], 1.0+weight).@@ -2143,12 +2331,14 @@ static void attention(Model *m, Layer *l, int layer, float *x, int S, int pos_bafor (int hh = 0; hh < H; hh++) {float *qh = query + ((int64_t)s*H + hh)*hd;if (l->qn) rmsnorm_row(qh, qh, l->qn, hd, c->eps);- rope_head_partial(qh, pos_base + s, rotary, hd, c->theta);+ if (m->mpos || m->rope_delta) { int p3[3]; mrope_at(m, pos_base + s, p3); rope_head_mrope(qh, p3, c->mrope_section, rotary, c->theta); }+ else rope_head_partial(qh, pos_base + s, rotary, hd, c->theta);}for (int kvh = 0; kvh < KV; kvh++) {float *kh = k + (int64_t)s*KV*kvd + kvh*kvd;if (l->kn) rmsnorm_row(kh, kh, l->kn, kvd, c->eps);- rope_head_partial(kh, pos_base + s, rotary, kvd, c->theta);+ if (m->mpos || m->rope_delta) { int p3[3]; mrope_at(m, pos_base + s, p3); rope_head_mrope(kh, p3, c->mrope_section, rotary, c->theta); }+ else rope_head_partial(kh, pos_base + s, rotary, kvd, c->theta);}}for (int s = 0; s < S; s++) for (int kvh = 0; kvh < KV; kvh++) {@@ -2957,13 +3147,19 @@ static float *step(Model *m, const int *ids, int S, int pos_base) {ids[s], c->vocab - 1);exit(1);}- memcpy(x + (int64_t)s*D, m->embed + (int64_t)ids[s]*D, D*sizeof(float));+ int vrow = (m->vis_map && pos_base + s < m->vis_map_len) ? m->vis_map[pos_base + s] : -1;+ if (vrow >= 0 && vrow < m->vis_rows_n) /* an image placeholder: the tower's row */+ memcpy(x + (int64_t)s*D, m->vis_rows + (int64_t)vrow*D, D*sizeof(float));+ else+ memcpy(x + (int64_t)s*D, m->embed + (int64_t)ids[s]*D, D*sizeof(float));}layers_forward_range(m, x, S, pos_base, 0, c->n_layers, 1, lf);/* Recorded HERE, where the tokens actually entered the state, rather than* derived from the caller's bookkeeping: the invariant that fed[0..len-1]* are the ids the state was built from is the whole safety argument. */kv_prefix_record(&m->kvp, ids, pos_base, S);+ /* the placeholders' ids do not say which picture they held */+ if (m->vis_map && m->vis_rows_n > 0) kv_prefix_taint(&m->kvp);m->token_count += S; m->freq_token_count += S;if (!m->hot_pinned && m->hot_n > 0 && m->freq_token_count >= m->warmup_tokens) pin_hot_experts(m);m->kv_len = pos_base + S;@@ -3394,10 +3590,39 @@ typedef struct { char id[64]; int max_tok; float temp, top_p; char *payload; intint pin; /* SUBMIT pin=1: fotografa lo stato dopo il prefill */} ServeReq;+/* An image waiting for the SUBMIT that uses it (IMAGE <id> <bytes> <grid_h>+ * <grid_w>, then the float32 patches and a newline -- the frame serve_codec.h+ * defines and the gateway already sends to qwen38). A second one before the+ * SUBMIT replaces the first, and says so. */+static struct { unsigned char *patches; unsigned long long bytes; int grid_h, grid_w, present; } g_pending_image;+static void q36_pending_image_clear(void){+ free(g_pending_image.patches);+ memset(&g_pending_image, 0, sizeof g_pending_image);+}+static int serve_read_req(ServeReq *q){char line[512], cmd[16], id[64];if(!fgets(line,sizeof(line),stdin)) return -1;if(sscanf(line,"%15s %63s",cmd,id)<2) return 0;+ if(!strcmp(cmd,"IMAGE")){+ /* The payload is consumed whatever happens next: left in the stream it+ * would be read as the following frame's header. A header that does not+ * say how long the payload is cannot be skipped, so it ends the session. */+ unsigned long long bytes; int gh, gw;+ if(sscanf(line,"%*s %*s %llu %d %d",&bytes,&gh,&gw)!=3 || bytes>(1ull<<30) || gh<1 || gw<1){+ printf("ERROR %s bad image header\n",id); fflush(stdout); return -1;+ }+ unsigned char *buf = malloc((size_t)bytes + 1);+ if(!buf){ printf("ERROR %s out of memory\n",id); fflush(stdout); return -1; }+ if(bytes && fread(buf,1,(size_t)bytes,stdin)!=(size_t)bytes){ free(buf); return -1; }+ int t = fgetc(stdin); if(t=='\r') t = fgetc(stdin);+ if(t!='\n'){ free(buf); return -1; }+ if(g_pending_image.present) fprintf(stderr,"[qwen36] a second image arrived before its SUBMIT; dropping the first\n");+ q36_pending_image_clear();+ g_pending_image.patches = buf; g_pending_image.bytes = bytes;+ g_pending_image.grid_h = gh; g_pending_image.grid_w = gw; g_pending_image.present = 1;+ return 0;+ }if(!strcmp(cmd,"CANCEL")||!strcmp(cmd,"STOP")) return 0;if(strcmp(cmd,"SUBMIT")) return 0;int slot, plen, max_tok; float temp, top_p;@@ -3608,6 +3833,25 @@ static int qwen36_serve_budget(int np, int max_tok, int max_ctx, int read_only){static void serve_one(Model *m, ServeReq *q){int *ids=NULL, np=0;encode_text(q->payload, &ids, &np); /* payload is raw prompt text; qwen36 adds no BOS */+ if(g_pending_image.present){+ Cfg *vc = &m->c;+ if(!m->vis_ready){+ printf("ERROR %s this engine has no vision tower; images are not supported\n",q->id);+ fflush(stdout); q36_pending_image_clear(); free(ids); return;+ }+ unsigned long long want = (unsigned long long)g_pending_image.grid_h*g_pending_image.grid_w*+ vc->vis_in_ch*vc->vis_temporal*vc->vis_patch*vc->vis_patch*sizeof(float);+ if(g_pending_image.bytes!=want){+ printf("ERROR %s BAD_IMAGE bytes=%llu expected=%llu\n",q->id,g_pending_image.bytes,want);+ fflush(stdout); q36_pending_image_clear(); free(ids); return;+ }+ if(q36_vision_attach(m,(const float*)g_pending_image.patches,g_pending_image.grid_h,+ g_pending_image.grid_w,ids,np)<0){+ printf("ERROR %s BAD_IMAGE the prompt and the grid disagree\n",q->id);+ fflush(stdout); q36_pending_image_clear(); free(ids); return;+ }+ q36_pending_image_clear();+ }int max_ctx = qwen36_max_ctx();int budget = qwen36_serve_budget(np, q->max_tok, max_ctx, q->logprobs > 0);if(budget < 0){@@ -3748,7 +3992,7 @@ static void serve_loop(Model *m){* "routed now" flash worked while the residency colour never moved.* inkling.c, kimi_k3.c, qwen38.c, deepseek_v41.c and colibri.c* already do this. */- if(r==2){ serve_one(m,&q); free(q.payload); emap_emit(m); }+ if(r==2){ serve_one(m,&q); q36_vision_detach(m); free(q.payload); emap_emit(m); }}}@@ -3868,6 +4112,7 @@ int main(int argc, char **argv) {int is_ref = 0;+ jval *ref_image = NULL; /* {"grid_h", "grid_w", "patches": [...]}: one image for the oracle */int rplen = (int)strlen(refpath);if (rplen>=5 && strcmp(refpath+rplen-5, ".json")==0) is_ref = 1;@@ -3895,6 +4140,7 @@ int main(int argc, char **argv) {fseek(f,0,SEEK_END); long n=ftell(f); fseek(f,0,SEEK_SET);buf=malloc(n+1); if (fread(buf,1,n,f)!=(size_t)n) {} buf[n]=0; fclose(f);jval *ref = json_parse(buf, &arena);+ ref_image = ref ? json_get(ref, "image") : NULL;prompt = read_int_array(ref,"prompt_ids",&np);full = read_int_array(ref,"full_ids",&nfull);n_new = nfull - np;@@ -3922,6 +4168,27 @@ int main(int argc, char **argv) {g_expert_gs = m.c.expert_gs;if (g_expert_gs) fprintf(stderr, "[qwen36] group-scaled experts: gs=%d\n", g_expert_gs);fprintf(stderr, "resident weights loaded in %.1fs | RSS after load: %.2f GB\n", m.dense_load_s, rss_gb());+ if (ref_image && ref_image->t == J_OBJ) {+ jval *gh = json_get(ref_image, "grid_h"), *gw = json_get(ref_image, "grid_w");+ jval *pv = json_get(ref_image, "patches");+ if (!gh || !gw || !pv || gh->t != J_NUM || gw->t != J_NUM || pv->t != J_ARR) {+ fprintf(stderr, "ref.json image needs grid_h, grid_w and patches\n"); return 1;+ }+ float *px = malloc((size_t)pv->len * sizeof(float));+ if (!px) { fprintf(stderr, "OOM image patches\n"); return 1; }+ for (int i = 0; i < pv->len; i++) px[i] = pv->kids[i]->t == J_NUM ? (float)pv->kids[i]->num : 0.f;+ Cfg *vc = &m.c;+ long long want = (long long)gh->num * (long long)gw->num * vc->vis_in_ch * vc->vis_temporal * vc->vis_patch * vc->vis_patch;+ if (!m.vis_ready || pv->len != want ||+ q36_vision_attach(&m, px, (int)gh->num, (int)gw->num, prompt, np) < 0) {+ fprintf(stderr, "the ref.json image does not fit this model (%d values, %lld expected, tower %s)\n",+ pv->len, want, m.vis_ready ? "loaded" : "absent");+ return 1;+ }+ free(px);+ fprintf(stderr, "[qwen36] oracle image: %dx%d patches, %d tokens, rope delta %d\n",+ (int)gh->num, (int)gw->num, m.vis_rows_n, m.rope_delta);+ }/* dense matrices are quantized to int8 (+ int4 planar per COLI_DENSE_BITS/* COLI_DENSE_INT4, see dense_int4_wanted) during model_init above* (COLI_DENSE_I8=0 disables it) -- see load_tq/QW; model_init_rangediff --git a/c/qwen38_vision.h b/c/qwen38_vision.hindex e0a6a7b1..aece88d2 100644--- a/c/qwen38_vision.h+++ b/c/qwen38_vision.h@@ -69,14 +69,18 @@ typedef struct {static void q38v_linear(float *out, const float *in, const Q38Linear *l, int rows){+ /* Every output is one dot product in a fixed order, so splitting (row, output)+ * pairs across threads changes the time and not a bit of the result. A real+ * image is hundreds of rows through 27 blocks: single-threaded it was minutes+ * per picture (#1757). */+ #pragma omp parallel for collapse(2) schedule(static)for (int r = 0; r < rows; r++) {- const float *x = in + (size_t)r * l->in;- float *y = out + (size_t)r * l->out;for (int o = 0; o < l->out; o++) {+ const float *x = in + (size_t)r * l->in;const float *w = l->w + (size_t)o * l->in;float acc = l->b ? l->b[o] : 0.f;for (int i = 0; i < l->in; i++) acc += w[i] * x[i];- y[o] = acc;+ out[(size_t)r * l->out + o] = acc;}}}@@ -226,6 +230,11 @@ static int q38_vision_forward_dbg(const Q38Vision *v, const float *patches,const float scale = 1.f / sqrtf((float)D);memset(att, 0, (size_t)n * H * sizeof(float));+ /* each (head, query) pair owns its slice of att and its own score row */+ #pragma omp parallel+ {+ float *scores = (float *)malloc((size_t)n * sizeof(float));+ #pragma omp for collapse(2) schedule(static)for (int h = 0; h < nh; h++)for (int i = 0; i < n; i++) {const float *q = qkv + (size_t)i * 3 * H + (size_t)h * D;@@ -247,6 +256,8 @@ static int q38_vision_forward_dbg(const Q38Vision *v, const float *patches,for (int d = 0; d < D; d++) dst[d] += weight * val[d];}}+ free(scores);+ }q38v_linear(tmp, att, &blk->proj, n);for (size_t i = 0; i < (size_t)n * H; i++) x[i] += tmp[i];diff --git a/c/tests/test_openai_server.py b/c/tests/test_openai_server.pyindex 7260640e..300cef42 100644--- a/c/tests/test_openai_server.py+++ b/c/tests/test_openai_server.py@@ -29,7 +29,7 @@ from openai_server import (APIError, APIHandler, APIServer, ClientCancelled,render_chat_qwen38, render_chat_v4, render_chat_dsv41,_dsv4_tool_calls, serve,resolve_generation_prompt, split_thinking_reply,- detect_chat_flavor,+ detect_chat_flavor, qwen36_has_vision,starts_in_reasoning,stop_policy, tune_child_env)@@ -231,6 +231,22 @@ class TemplateTest(unittest.TestCase):self.assertIn("<|im_start|>assistant\nParis.<|im_end|>",render_chat_qwen(first + [past, follow], enable_thinking=True))+ def test_qwen36_images_follow_the_container(self):+ """#1757: a qwen36 container converted with its vision tower says so in+ qwen36_meta.json, and only then does the gateway turn image parts into+ patches for the engine; an older or text-only container keeps refusing them."""+ import tempfile+ with tempfile.TemporaryDirectory() as d:+ self.assertFalse(qwen36_has_vision(d))+ meta = Path(d) / "qwen36_meta.json"+ meta.write_text(json.dumps({"num_experts": 0}), encoding="utf-8")+ self.assertFalse(qwen36_has_vision(d))+ meta.write_text(json.dumps({"vision": {"depth": 27}}), encoding="utf-8")+ self.assertTrue(qwen36_has_vision(d))+ meta.write_text("not json", encoding="utf-8")+ self.assertFalse(qwen36_has_vision(d))+ self.assertFalse(qwen36_has_vision(None))+def test_qwen38_template_on_the_qwen36_engine(self):"""#1757: Qwen3.8-27B is a dense model of Qwen3.5's architecture, so the qwen36engine runs it, but it ships Qwen3.8's chat_template.jinja. The gateway rendersdiff --git a/c/tests/test_qwen36_tensor_kinds.py b/c/tests/test_qwen36_tensor_kinds.pyindex eaa9276b..dca4ab0a 100644--- a/c/tests/test_qwen36_tensor_kinds.py+++ b/c/tests/test_qwen36_tensor_kinds.py@@ -217,10 +217,13 @@ class TensorKindsTest(unittest.TestCase):def test_every_27b_tensor_is_placed(self):"""The dense checkpoint: the MoE kinds are absent, the dense MLP is placed, the- mtp head and the vision tower are skipped."""+ vision tower is converted (#1757) and the mtp head is skipped."""prefix, placed = self._classify_all(_concrete(QWEN38_27B))self.assertEqual(prefix, "model.language_model.")- self.assertEqual({p[1] for p in placed if p[0] == "skip"}, {"mtp", "visual"})+ self.assertEqual({p[1] for p in placed if p[0] == "skip"}, {"mtp"})+ vision = [p[1] for p in placed if p[0] == "vision"]+ self.assertEqual(len(vision), 3) # the three listed kinds+ self.assertTrue(all(not v.startswith("visual.") for v in vision), vision)layer = {p[2] for p in placed if p[0] == "layer"}self.assertTrue(DENSE_MLP_KINDS <= layer)self.assertFalse({k for k in layer if k.startswith("mlp.") and k not in DENSE_MLP_KINDS})@@ -228,7 +231,8 @@ class TensorKindsTest(unittest.TestCase):def test_every_35b_tensor_is_placed(self):prefix, placed = self._classify_all(_concrete(QWEN36_35B))self.assertEqual(prefix, "model.language_model.")- self.assertEqual({p[1] for p in placed if p[0] == "skip"}, {"mtp", "visual"})+ self.assertEqual({p[1] for p in placed if p[0] == "skip"}, {"mtp"})+ self.assertTrue(any(p[0] == "vision" for p in placed))self.assertEqual(sum(p[0] == "layer" for p in placed),sum(p[0] == "layer" for p in self._classify_all(_concrete(QWEN38_2P4T))[1]))@@ -263,8 +267,7 @@ class TensorKindsTest(unittest.TestCase):classify(name, "model.")def test_skip_groups_have_a_stated_reason(self):- for group in ("mtp", "visual"):- self.assertTrue(skip_reason(group))+ self.assertTrue(skip_reason("mtp"))self.assertEqual(skip_reason("nope"), "")def test_layer_kinds_are_exact_suffixes(self):diff --git a/c/tests/test_qwen36_vision_serve.py b/c/tests/test_qwen36_vision_serve.pynew file mode 100644index 00000000..49c6d761--- /dev/null+++ b/c/tests/test_qwen36_vision_serve.py@@ -0,0 +1,123 @@+"""Images through qwen36's serve protocol (#1757): IMAGE frame, tower, generation.++The CLI oracle (tools/make_qwen36_vl_tiny.py, ./qwen36 cap 8 ref.json) holds the+engine token for token to transformers. This holds the SERVE path to that same+answer: the prompt goes in as text, the image as the IMAGE frame the gateway+sends, and the bytes that come back must be the reference's tokens decoded.+Then the refusals: patches that do not fit the grid, and placeholders that do+not match it. And a text-only turn still works after an image.++ QWEN36_VL_TINY=<converted container> QWEN36_VL_REF=<fixture>/ref.json \\+ python3 -m unittest tests.test_qwen36_vision_serve+"""+import json+import os+import struct+import subprocess+import unittest+from pathlib import Path++HERE = Path(__file__).resolve().parent.parent+ENGINE = HERE / "qwen36"+FIXTURE = os.environ.get("QWEN36_VL_TINY")+REF = os.environ.get("QWEN36_VL_REF")+IMAGE_PAD, START, END = b"<|image_pad|>", b"<|vision_start|>", b"<|vision_end|>"+++def expected_bytes(tokenizer, ids):+ """The bytes the engine sends for `ids` with this fixture's tokenizer."""+ by_id = {v: k for k, v in tokenizer["model"]["vocab"].items()}+ specials = {t["id"]: t["content"] for t in tokenizer["added_tokens"]}+ direct = set(range(33, 127)) | set(range(161, 173)) | set(range(174, 256))+ unmap, spare = {}, 0+ for b in range(256):+ if b in direct:+ unmap[chr(b)] = b+ else:+ unmap[chr(256 + spare)] = b+ spare += 1+ out = b""+ for i in ids:+ if i in specials:+ out += specials[i].encode()+ else:+ out += bytes(unmap[c] for c in by_id[i])+ return out+++class Qwen36VisionServe(unittest.TestCase):+ @classmethod+ def setUpClass(cls):+ if not (FIXTURE and REF and ENGINE.is_file()):+ raise unittest.SkipTest("QWEN36_VL_TINY / QWEN36_VL_REF not set, or qwen36 not built")+ cls.ref = json.loads(Path(REF).read_text())+ cls.tokenizer = json.loads((Path(FIXTURE) / "tokenizer.json").read_text())++ def start(self, snap):+ env = dict(os.environ, SNAP=str(snap), SERVE="1", COLI_DENSE_I8="0",+ OMP_NUM_THREADS="2", COLI_NO_OMP_TUNE="1")+ p = subprocess.Popen([str(ENGINE), "1", "8"], stdin=subprocess.PIPE, stdout=subprocess.PIPE,+ stderr=subprocess.DEVNULL, env=env)+ def stop():+ p.kill()+ p.wait()+ p.stdin.close()+ p.stdout.close()+ self.addCleanup(stop)+ while b"READY" not in p.stdout.readline():+ pass+ return p++ def turn(self, p, rid, prompt, patches=None, grid=(0, 0), max_new=16):+ if patches is not None:+ blob = struct.pack(f"<{len(patches)}f", *patches)+ p.stdin.write(f"IMAGE {rid} {len(blob)} {grid[0]} {grid[1]}\n".encode() + blob + b"\n")+ p.stdin.write(f"SUBMIT {rid} 0 {len(prompt)} {max_new} 0.0 1.0\n".encode() + prompt + b"\n")+ p.stdin.flush()+ got = b""+ while True:+ line = p.stdout.readline().decode("utf-8", "replace").strip("\r\n\x01")+ if not line:+ return "EOF", got+ if line.startswith(f"DATA {rid} "):+ n = int(line.split()[2])+ got += p.stdout.read(n + 1)[:n]+ elif line.startswith(f"DONE {rid}") or line.startswith(f"ERROR {rid}"):+ return line, got++ def prompt(self):+ ids = self.ref["prompt_ids"]+ tokens = ids.count(300)+ return bytes([1, 2, 3]) + START + IMAGE_PAD * tokens + END + bytes([4, 5, 6])++ def test_serve_gives_the_oracle_tokens(self):+ p = self.start(FIXTURE)+ image = self.ref["image"]+ verdict, got = self.turn(p, "a", self.prompt(), image["patches"], (image["grid_h"], image["grid_w"]))+ self.assertTrue(verdict.startswith("DONE"), verdict)+ want = expected_bytes(self.tokenizer, self.ref["full_ids"][len(self.ref["prompt_ids"]):])+ self.assertEqual(got, want)+ # a different picture must change the answer, or the tower's rows are lost+ verdict, other = self.turn(p, "b", self.prompt(), [-v for v in image["patches"]],+ (image["grid_h"], image["grid_w"]))+ self.assertTrue(verdict.startswith("DONE"), verdict)+ self.assertNotEqual(other, got)+ # and the turn after an image is text only again+ verdict, _ = self.turn(p, "c", bytes([3, 4, 5]))+ self.assertTrue(verdict.startswith("DONE"), verdict)++ def test_refusals(self):+ p = self.start(FIXTURE)+ image = self.ref["image"]+ grid = (image["grid_h"], image["grid_w"])+ verdict, _ = self.turn(p, "d", self.prompt(), image["patches"][:-8], grid)+ self.assertIn("BAD_IMAGE", verdict)+ short = bytes([1]) + START + IMAGE_PAD * 3 + END+ verdict, _ = self.turn(p, "e", short, image["patches"], grid)+ self.assertIn("BAD_IMAGE", verdict)+ verdict, _ = self.turn(p, "f", bytes([3, 4, 5]))+ self.assertTrue(verdict.startswith("DONE"), verdict)+++if __name__ == "__main__":+ unittest.main()diff --git a/c/tools/convert_qwen36.py b/c/tools/convert_qwen36.pyindex af2e5ce9..bc06b8fa 100644--- a/c/tools/convert_qwen36.py+++ b/c/tools/convert_qwen36.py@@ -333,7 +333,8 @@ def main():# The chat template travels with the container: the gateway renders what the# checkpoint was trained on, and a Qwen3.8 template on this engine (Qwen3.8-27B,# #1757) is recognised from this file.- for extra in ("chat_template.jinja", "tokenizer_config.json", "generation_config.json"):+ for extra in ("chat_template.jinja", "tokenizer_config.json", "generation_config.json",+ "preprocessor_config.json"):try:extra_path = (hf_hub_download(args.repo, extra, token=token) if args.repoelse str(src_dir / extra))@@ -363,6 +364,7 @@ def main():prefix = resolve_prefix(wm.keys())layer_map = {} # layer index -> [keys]global_map = {} # kind -> key+ vision_map = {} # name after "visual." -> keyskipped = {} # group -> countunknown = []for k in wm:@@ -375,6 +377,8 @@ def main():layer_map.setdefault(placed[1], []).append(k)elif placed[0] == "global":global_map[placed[1]] = k+ elif placed[0] == "vision":+ vision_map[placed[1]] = kelse:skipped[placed[1]] = skipped.get(placed[1], 0) + 1if unknown:@@ -511,6 +515,23 @@ def main():upload_local(gpath)gpath.unlink()+ # ---- vision tower as f16, under model.visual.* (#1757) ----+ # The same ViT in Qwen3.5/3.6/3.8; the engine runs it through qwen38_vision.h.+ vcfg = cfg_full.get("vision_config") or {}+ if vision_map and not vcfg:+ print(f"note: {len(vision_map)} visual.* tensor(s) but no vision_config; the tower is left out")+ if vision_map and vcfg:+ if "model-vision.safetensors" in done_files:+ print("[vision] already on HF, skip")+ else:+ v_out = {f"model.visual.{name}": get_tensor(k).half() for name, k in vision_map.items()}+ vpath = out / "model-vision.safetensors"+ save_file(v_out, str(vpath))+ print(f"[vision] {vpath.name} ({len(v_out)} tensors)")+ if args.stream_upload:+ upload_local(vpath)+ vpath.unlink()+# ---- per layer: Gated-Attention (full_attention) AND Gated DeltaNet (linear_attention) ----# Every layer also carries its MoE/MLP block. DeltaNet layers export linear_attn.* which# the generic f16 copy below handles automatically; the engine implements the recurrence.@@ -652,6 +673,16 @@ def main():h.__exit__(None, None, None)except Exception:pass+ if vision_map and vcfg:+ meta["image_token_id"] = int(cfg_full["image_token_id"])+ meta["vision"] = {+ "depth": int(vcfg["depth"]), "hidden": int(vcfg["hidden_size"]),+ "heads": int(vcfg["num_heads"]), "inter": int(vcfg["intermediate_size"]),+ "patch": int(vcfg["patch_size"]), "merge": int(vcfg["spatial_merge_size"]),+ "temporal": int(vcfg["temporal_patch_size"]), "in_ch": int(vcfg["in_channels"]),+ "out_hidden": int(vcfg["out_hidden_size"]),+ "num_pos": int(vcfg["num_position_embeddings"]),+ }(out / "qwen36_meta.json").write_text(json.dumps(meta, indent=2), encoding="utf-8")print(f"[meta] {out / 'qwen36_meta.json'}")diff --git a/c/tools/make_qwen36_vl_tiny.py b/c/tools/make_qwen36_vl_tiny.pynew file mode 100644index 00000000..5579f27c--- /dev/null+++ b/c/tools/make_qwen36_vl_tiny.py@@ -0,0 +1,124 @@+#!/usr/bin/env python3+"""Tiny Qwen3.8-27B-shaped vision-language fixture and its token-exact reference.++Qwen3_5ForConditionalGeneration at toy widths: the dense text model of the+qwen38-27b-dense geometry (make_qwen36_tiny.py) plus a small ViT of the tower+the whole family shares. transformers generates greedily from a prompt that+holds ONE image, and ref.json records the prompt ids, the generated ids and the+image patches, so `./qwen36 <cap> 8 ref.json` can be held to the same tokens+(#1757). A random model answers anything; token equality is the test.++The image grid is 4 x 8 patches on purpose: 2 x 4 tokens after the merge, so+the text after the image resumes at start + max(2, 4) = start + 4 while 8+positions went by. Every position after the image then depends on getting+Qwen3.5's 3D rope positions right, not just the first one.++ python3 tools/make_qwen36_vl_tiny.py --out qwen36_vl_tiny+"""+import argparse+import json+import sys+from pathlib import Path++import torch++sys.path.insert(0, str(Path(__file__).resolve().parent))+sys.path.insert(0, str(Path(__file__).resolve().parent.parent / "tests"))+import make_qwen36_tiny # noqa: E402++SEED = 20260927+IMAGE, VIDEO, START, END = 300, 301, 302, 303 # special ids inside the 320-token toy vocabulary+++SPECIALS = {"<|image_pad|>": IMAGE, "<|video_pad|>": VIDEO, "<|vision_start|>": START,+ "<|vision_end|>": END, "<|im_start|>": 304, "<|im_end|>": 305, "<|endoftext|>": 319}+++def write_tokenizer(out: Path):+ """One token per ASCII byte (the GPT-2 byte map tok.h builds) plus Qwen's+ special tokens as added tokens at their fixture ids, so a serve prompt written+ as text reaches the engine as exactly the oracle's ids."""+ from prefix_serve_harness import byte_level_vocab+ # ASCII bytes for ids 0..127 (the prompts only use a few of them); every+ # other id decodes to printable filler. A lone byte >= 0x80 is not valid UTF-8,+ # and the engine holds such bytes back from the stream, so ids that decode to+ # one would hide from a test that compares what the client receives.+ vocab = {k: v for k, v in byte_level_vocab(256).items() if v < 128}+ for i in range(128, 320):+ if i not in SPECIALS.values():+ vocab[f"t{i}"] = i+ added = [{"id": i, "content": t, "single_word": False, "lstrip": False, "rstrip": False,+ "normalized": False, "special": True} for t, i in sorted(SPECIALS.items(), key=lambda kv: kv[1])]+ (out / "tokenizer.json").write_text(json.dumps({+ "version": "1.0", "added_tokens": added,+ "model": {"type": "BPE", "vocab": vocab, "merges": []}}), encoding="utf-8")+++def build(out: Path, grid_h=4, grid_w=8, max_new=16, seed=SEED):+ from transformers import Qwen3_5Config, Qwen3_5ForConditionalGeneration++ geo = make_qwen36_tiny.GEOMETRIES["qwen38-27b-dense"]+ hidden, layers = geo["hidden"], geo["n_layers"]+ text = dict(+ vocab_size=320, hidden_size=hidden, intermediate_size=geo["inter"],+ num_hidden_layers=layers, num_attention_heads=geo["q_heads"],+ num_key_value_heads=geo["kv_heads"], head_dim=32,+ max_position_embeddings=512, rms_norm_eps=1e-6, tie_word_embeddings=False,+ linear_conv_kernel_dim=4, linear_key_head_dim=8, linear_value_head_dim=8,+ linear_num_key_heads=geo["dn_key_heads"], linear_num_value_heads=geo["dn_value_heads"],+ layer_types=["full_attention" if i % 4 == 3 else "linear_attention" for i in range(layers)],+ hidden_act="silu", attention_bias=False, attention_dropout=0.0, use_cache=True,+ # the real checkpoint's rope: interleaved M-RoPE over a quarter of head_dim.+ # head_dim 32 gives four frequencies, so [2, 1, 1] puts at least one on+ # every axis (the real [11, 11, 10] spreads 32 the same way)+ rope_parameters={"rope_type": "default", "rope_theta": 10000.0,+ "partial_rotary_factor": 0.25, "mrope_interleaved": True,+ "mrope_section": [2, 1, 1]},+ partial_rotary_factor=0.25, pad_token_id=0, bos_token_id=1, eos_token_id=319,+ )+ vision = dict(depth=2, hidden_size=32, num_heads=4, intermediate_size=64, patch_size=16,+ spatial_merge_size=2, temporal_patch_size=2, in_channels=3,+ out_hidden_size=hidden, num_position_embeddings=64,+ hidden_act="gelu_pytorch_tanh", deepstack_visual_indexes=[])+ config = Qwen3_5Config(text_config=text, vision_config=vision, image_token_id=IMAGE,+ video_token_id=VIDEO, vision_start_token_id=START,+ vision_end_token_id=END, tie_word_embeddings=False)+ torch.manual_seed(seed)+ model = Qwen3_5ForConditionalGeneration(config).eval()+ with torch.no_grad(): # a tower whose output is not noise-small+ for name, parameter in model.named_parameters():+ if "visual" in name:+ parameter.copy_(torch.randn_like(parameter) * 0.5)+ out.mkdir(parents=True, exist_ok=True)+ model.save_pretrained(str(out))+ write_tokenizer(out)++ tokens = (grid_h // 2) * (grid_w // 2)+ prompt = [1, 2, 3, START] + [IMAGE] * tokens + [END, 4, 5, 6]+ generator = torch.Generator().manual_seed(seed + 1)+ features = 3 * 2 * 16 * 16+ patches = torch.randn(grid_h * grid_w, features, generator=generator)+ input_ids = torch.tensor([prompt])+ with torch.no_grad():+ full = model.generate(+ input_ids, pixel_values=patches, image_grid_thw=torch.tensor([[1, grid_h, grid_w]]),+ mm_token_type_ids=(input_ids == IMAGE).int(), max_new_tokens=max_new,+ do_sample=False, use_cache=True)[0].tolist()+ reference = {"prompt_ids": prompt, "full_ids": full, "mode": "full", "model": "qwen36_vl_tiny",+ "image": {"grid_h": grid_h, "grid_w": grid_w,+ "patches": [round(float(v), 7) for v in patches.reshape(-1)]}}+ (out / "ref.json").write_text(json.dumps(reference))+ print(f"wrote {out}: {tokens} image tokens, prompt {len(prompt)}, full {full[len(prompt):]}")+++def main() -> int:+ parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])+ parser.add_argument("--out", type=Path, required=True)+ parser.add_argument("--max-new", type=int, default=16)+ args = parser.parse_args()+ build(args.out, max_new=args.max_new)+ return 0+++if __name__ == "__main__":+ sys.exit(main())diff --git a/c/tools/qwen36_tensor_kinds.py b/c/tools/qwen36_tensor_kinds.pyindex 3269184a..6d1393ed 100644--- a/c/tools/qwen36_tensor_kinds.py+++ b/c/tools/qwen36_tensor_kinds.py@@ -61,15 +61,17 @@ _EXPERT_SEPARATE = re.compile(r"^mlp\.experts\.(\d+)\.(gate_proj|up_proj|down_prGLOBAL_KINDS = frozenset(("embed_tokens.weight", "norm.weight", "lm_head.weight"))-# Whole subtrees the text engine does not implement. Skipped deliberately,+# Whole subtrees the engine does not implement. Skipped deliberately,# counted, and reported -- never silently.SKIP_PREFIXES = (("mtp.", "mtp", "multi-token-prediction head (mtp_num_hidden_layers); ""the engine predicts one token per step and never reads it"),- ("model.visual.", "visual", "vision tower; the engine is text-only"),- ("visual.", "visual", "vision tower; the engine is text-only"),)+# The vision tower, converted since #1757: the same ViT in Qwen3.5/3.6/3.8, run by+# the engine through qwen38_vision.h. Placed as ("vision", <name after visual.>).+VISION_PREFIXES = ("model.visual.", "visual.")+_LAYER = re.compile(r"^layers\.(\d+)\.(.+)$")@@ -89,7 +91,8 @@ def classify(name, prefix):"""Return one of("global", kind) embed / final norm / lm_head("layer", index, kind) a tensor of transformer layer <index>- ("skip", group) mtp / visual, deliberately not converted+ ("vision", name) the vision tower, name after "visual."+ ("skip", group) mtp, deliberately not convertedor raise UnknownTensor.``kind`` for a layer is the suffix after ``layers.<i>.``; for the@@ -99,6 +102,9 @@ def classify(name, prefix):for skip_prefix, group, _why in SKIP_PREFIXES:if name.startswith(skip_prefix):return ("skip", group)+ for vision_prefix in VISION_PREFIXES:+ if name.startswith(vision_prefix):+ return ("vision", name[len(vision_prefix):])if name == "lm_head.weight" or name == prefix + "lm_head.weight":return ("global", "lm_head.weight")if name.startswith(prefix):diff --git a/docs/ENVIRONMENT.md b/docs/ENVIRONMENT.mdindex 85ca9a23..7c0ab378 100644--- a/docs/ENVIRONMENT.md+++ b/docs/ENVIRONMENT.md@@ -423,6 +423,7 @@ and the CPU/GPU execution split.| `COLI_DENSE_IDOT` | `1` (on) | The dense trunk's GEMVs (DeltaNet projections and out_proj, attention q/k/v/o, shared expert, lm_head) quantize the activation to int8 once per call and run integer dot products (maddubs on AVX2, vpdpbusd on AVX-VNNI / AVX-512 VNNI) instead of converting every int8 weight to f32. Not bit-identical to the f32 path; measured +1.0% perplexity, lm_head 12.6 to 10.2 ms/token. `=0` restores the f32-activation kernel. || `QWEN_EXPERT_ACT` | `i8` | The routed experts' activation quantized to int8 once per row (expert_ffn.h mode 1). Measured +0.1% perplexity, expert compute 22.7 to 15.9 ms/token. `=f32` restores f32 activations and the bit-identical contract with the pair kernels. || `COLI_DENSE_BITS` | `8` | `=4` stores the dense trunk as int4 in blocks of 64 with one scale per block (the K1b planar layout, half the bytes), served by the grouped integer kernel; implies the integer dot. Opt-in: on the 35B it costs +10% perplexity on the whole trunk, +2.4% on lm_head alone (see `COLI_DENSE_INT4`). |+| `Q36_MAX_IMAGE_TOKENS` | the checkpoint's preprocessor ceiling | Gateway, qwen36 containers with a vision tower (Qwen3.8-27B): ceiling on the tokens one image costs. The image is shrunk, not cropped. Without one a 1080p photo is about 2000 tokens of tower and prefill on the CPU. || `COLI_DENSE_KEEP_I8` | `0` (off) | qwen36: a matrix that got its int4 copy (`COLI_DENSE_BITS=4`) frees its int8 copy, which only the CUDA placer reads; `=1` keeps both. With `COLI_CUDA=1` both are kept anyway. Measured on Qwen3.8-27B: 42.4 GB resident with both copies, 18.8 GB without. || `COLI_DENSE_INT4` | all components | With `COLI_DENSE_BITS=4`, a comma list of the components that take int4: `lmhead`, `dnproj`, `dnout`, `attn`, `shexp`, `router`. Measured on the 35B: `lmhead` +2.4% perplexity for 254 MB less per token; `lmhead,dnproj,dnout` +5.6%; everything +10%. || `QWEN_EXPERT_KERNEL` | `1` (on) | Routed experts run through the shared `expert_ffn.h` kernel: the int4 stays packed in RAM (planar layout, half the expert-cache RSS of the int8 unpack), gate+up are one pass, and a layer is two OpenMP regions over (expert, row-chunk) items instead of 3 x top-k GEMV regions. Takes effect on an int4 gs=64 container whose hidden and expert widths are multiples of 64, and not under the CUDA expert tier. `=0` restores the unpack-to-int8 path; the two produce the same tokens (1024-token decode on the real container byte-identical; pinned on the tiny int4 fixture in CI), only the f32 accumulation order inside a dot differs. Measured at cap 256 on the real container: 12.8 -> 15.7 tok/s, peak RSS 29 -> 17 GB. |diff --git a/docs/qwen36.md b/docs/qwen36.mdindex 4113f955..40824fab 100644--- a/docs/qwen36.md+++ b/docs/qwen36.md@@ -256,9 +256,38 @@ keeps most of the saving at half the loss. With a matrix in int4 the engineno longer keeps its int8 copy (`COLI_DENSE_KEEP_I8=1` does); that is whatbrings the full int4 run from 42.4 to 18.8 GB.-Not yet: the CUDA tier (a dense checkpoint runs on the CPU), the vision-tower and the MTP head (both skipped by the converter), and an int4-container on disk.+#### Images++Qwen3.8-27B reads images. Its vision tower is the ViT of the whole family+(27 blocks, hidden 1152, patch 16, 2x2 merge; only the output width follows+the text model), so the engine runs it through the same `qwen38_vision.h`+the qwen38 engine uses, now split across OpenMP threads. The converter copies+it into `model-vision.safetensors` and writes its shape into+`qwen36_meta.json`, together with `preprocessor_config.json`.++What is specific to this family is where the image sits in the rope. The+attention layers use interleaved M-RoPE (`mrope_section` [11, 11, 10]): an+image token at merged (row, col) is rotated by (start, start + row,+start + col), the text after the image resumes at start + max(rows, cols),+and every later position carries that offset (HF's `rope_deltas`). The+engine follows `Qwen3_5Model.get_rope_index` exactly; with plain 1D+positions the tiny oracle below loses 10 of 16 tokens.++An image reaches it the way it reaches the other vision engines: a path in+a `coli chat` message, an attachment in `coli web`, or an `image_url` part+on `/v1/chat/completions`. One image per request. `Q36_MAX_IMAGE_TOKENS`+caps the tokens an image costs (the preprocessor's own ceiling is far above+what a CPU prefill wants); the image is shrunk, not cropped.++`tools/make_qwen36_vl_tiny.py` builds a toy `Qwen3_5ForConditionalGeneration`+and a reference from transformers with one 4 x 8-patch image; the engine+matches it token for token, and `tests/test_qwen36_vision_serve.py` holds the+IMAGE frame path to the same tokens. Qwen3.6-35B carries the same tower, so a+35B container converted with this converter gets images too; only the 27B has+been run with real pictures.++Not yet: the CUDA tier (a dense checkpoint runs on the CPU), the MTP head+(skipped by the converter), video, and an int4 container on disk.### The converter's tensor contract