FreeToken brings datacenter-scale model serving to your desktop. Run massive models locally, fast and efficiently.
feat(mm): serve image input on the Qwen families (#454)
Xiaoze Fan08d728d5e385
diff --git a/docs/cli.md b/docs/cli.mdindex e1eaa8aa..1611214b 100644--- a/docs/cli.md+++ b/docs/cli.md@@ -99,6 +99,26 @@ See [models.md](models.md#moe-strategies) for what each strategy does.| `--reasoning-parser` | auto | Splits chain-of-thought into `reasoning_content`; auto-inferred; `off` disables || `--enable-cache-report` | off | Report prefix-cache hits in each response's usage block |+### Image input++Experimental. Needs a vision-capable checkpoint (Qwen3.6, Qwen3.8-Flash-Next, Qwen3-VL); a request carrying images is+rejected otherwise. Images are accepted on all three protocols (OpenAI `image_url`,+Anthropic `image` blocks, Responses `input_image`) as an http(s) URL or base64. Images inside Anthropic+`tool_result` blocks are rejected with a 400: chat templates render tool messages as plain text.+`GET /v1/stats` reports what the server accepts as `model.input_modalities` (`["text"]` or `["text", "image"]`),+so a client can gate its attachment controls without reading the checkpoint config.++| Flag | Default | Meaning |+| --- | --- | --- |+| `--text-model-only` | off | Serve a multimodal checkpoint text-only: no encoder tower is built (its VRAM goes to the KV/expert pools) and every multimodal input is rejected. Same as `--mm-disable` with every encoder kind |+| `--mm-disable` | none | Encoder towers to leave unbuilt (`vision`, `audio`); every input they would serve is rejected |+| `--mm-encoder-weights` | host | Where the encoder tower's block weights live. `host` streams them from pinned host banks two blocks at a time behind the compute (about 60 MiB of VRAM instead of the whole tower; small images encode slower, about 17 ms instead of 7 ms for 448x448); `gpu` keeps them resident |+| `--image-min-tokens`, `--image-max-tokens` | processor defaults | Per-image token budget: the image processor resizes every image to take between these many tokens, converted to the family's own units (Qwen VL: one token per 32x32 pixels of the resized image, defaults 64 to 16384). Only families with dynamic resolution honor them |+| `--mm-processor-kwargs` | none | JSON object of extra keyword arguments for the checkpoint's image processor call, for family-specific knobs (Qwen VL: `{"size": {"longest_edge": 1048576}}`); applied after the token budget |+| `--mm-embed-cache-device` | cpu | Where encoded image embeddings live between prefill chunks. `cpu` keeps them out of the VRAM budget; `cuda` skips the copy back |+| `--allowed-media-domains` | any | Comma-separated hostname allowlist for image URLs; requests for other domains are rejected with a 400. Empty allows any domain |+| `--allowed-local-media-path` | off | Directory `file://` image refs may be read from; unset rejects local files |+## ft shell```bash@@ -118,7 +138,7 @@ ft ctl [--base-url http://127.0.0.1:1919] [--timeout 10] [--json] <subcommand>| Subcommand | Endpoint | Purpose ||---|---|---|| `health` | `GET /health` | Server status, model, load progress |-| `stats` | `GET /v1/stats` | Throughput, latency, VRAM, pool occupancy |+| `stats` | `GET /v1/stats` | Throughput, latency, VRAM, pool occupancy, accepted input modalities || `generate [prompt] [--max-tokens N] [--ignore-eos]` | `POST /generate` | Raw completion smoke test (no chat template) || `cache` | `GET /v1/cache/status` | Cache pool table || `cache --moe N \| --kv N \| --mamba N \| --swa N [--wait 300]` | `POST /v1/cache/rebuild` | Live pool resizing without a restart (`k`/`m` suffixes; `--kv`/`--swa` in tokens) |diff --git a/docs/models.md b/docs/models.mdindex e4115356..0a9e214a 100644--- a/docs/models.md+++ b/docs/models.md@@ -13,11 +13,15 @@ for them; other checkpoints of the same architectures work too.| Qwen3.6 / Qwen3.5 MoE | [Qwen/Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B) ([-FP8](https://huggingface.co/Qwen/Qwen3.6-35B-A3B-FP8)), [nvidia/Qwen3.6-35B-A3B-NVFP4](https://huggingface.co/nvidia/Qwen3.6-35B-A3B-NVFP4), [Qwen/Qwen3.5-35B-A3B](https://huggingface.co/Qwen/Qwen3.5-35B-A3B) ([-FP8](https://huggingface.co/Qwen/Qwen3.5-35B-A3B-FP8)) || Qwen3.8 / Qwen3.6 dense | [Qwen/Qwen3.8-27B](https://huggingface.co/Qwen/Qwen3.8-27B) ([-FP8](https://huggingface.co/Qwen/Qwen3.8-27B-FP8)), [RadixArk/Qwen3.8-27B-NVFP4](https://huggingface.co/RadixArk/Qwen3.8-27B-NVFP4), [Qwen/Qwen3.6-27B](https://huggingface.co/Qwen/Qwen3.6-27B) ([-FP8](https://huggingface.co/Qwen/Qwen3.6-27B-FP8)), [nvidia/Qwen3.6-27B-NVFP4](https://huggingface.co/nvidia/Qwen3.6-27B-NVFP4) || Qwen3-MoE | [Qwen/Qwen3-30B-A3B](https://huggingface.co/Qwen/Qwen3-30B-A3B) |+| Qwen3-VL | [Qwen/Qwen3-VL-8B-Instruct](https://huggingface.co/Qwen/Qwen3-VL-8B-Instruct), [Qwen/Qwen3-VL-30B-A3B-Instruct](https://huggingface.co/Qwen/Qwen3-VL-30B-A3B-Instruct) || gpt-oss | [openai/gpt-oss-120b](https://huggingface.co/openai/gpt-oss-120b), [openai/gpt-oss-20b](https://huggingface.co/openai/gpt-oss-20b) || Gemma-4 | [google/gemma-4-26B-A4B-it](https://huggingface.co/google/gemma-4-26B-A4B-it), [nvidia/Gemma-4-26B-A4B-NVFP4](https://huggingface.co/nvidia/Gemma-4-26B-A4B-NVFP4), [google/gemma-4-12B-it](https://huggingface.co/google/gemma-4-12B-it), [nvidia/Gemma-4-31B-IT-NVFP4](https://huggingface.co/nvidia/Gemma-4-31B-IT-NVFP4) .. || MiniMax-M2.5 | [nvidia/MiniMax-M2.5-NVFP4](https://huggingface.co/nvidia/MiniMax-M2.5-NVFP4) || Muse-Glimmer | [meta-models/Muse-Glimmer-30B](https://huggingface.co/meta-models/Muse-Glimmer-30B), [RedHatAI/Muse-Glimmer-30B-NVFP4](https://huggingface.co/RedHatAI/Muse-Glimmer-30B-NVFP4) |+Qwen3.6 (both variants, every listed weight format), Qwen3.8-Flash-Next and Qwen3-VL also accept image input by default;+pass `--text-model-only` to skip the vision tower; see [CLI reference](cli.md#image-input).+## MoE strategies`ft serve --moe-strategy {auto,fused,offload,cpu,hybrid}` (`--moe-backend` is the deprecated old spelling):diff --git a/pyproject.toml b/pyproject.tomlindex 8bd653f8..228108f0 100644--- a/pyproject.toml+++ b/pyproject.toml@@ -47,6 +47,7 @@ dependencies = ["numpy>=2.0,<2.5","openai>=2.0,<3","partial-json-parser>=0.2,<1",+ "pillow>=10,<13","prompt_toolkit>=3.0,<4","pydantic>=2.9,<3","pyzmq>=27,<28",@@ -55,8 +56,9 @@ dependencies = [# PyPI's torch 2.11.0 wheel is itself the cu130 build, so plain pip resolves# correctly from PyPI alone; uv additionally pins the index below."torch>=2.11,<2.12",+ "torchvision>=0.26,<0.27","tqdm>=4.66,<5",- "transformers>=5.5,<6",+ "transformers>=5.15,<6","triton==3.6.0; platform_system == 'Linux'","uvicorn>=0.30,<1",]diff --git a/python/freetoken/attention/qsa_sparse.py b/python/freetoken/attention/qsa_sparse.pyindex 4a28dc85..b07c9523 100644--- a/python/freetoken/attention/qsa_sparse.py+++ b/python/freetoken/attention/qsa_sparse.py@@ -88,6 +88,10 @@ class QSASparseMetadata(BaseAttnMetadata):cmp_rows: torch.Tensor | None = None # [T] int32, compressed slab destinationring_rows: torch.Tensor | None = None # [T] int32, flat ring row or -1positions: torch.Tensor | None = None # [T] int32, logical query positions+ # mrope only, built once per forward: row r of the caches is token r's [cos | sin] (queries) or its group start's (keys)+ rope_rows: torch.Tensor | None = None # [T] int32 arange+ q_rope_cache: torch.Tensor | None = None # [T, rotary_dim] float32+ k_rope_cache: torch.Tensor | None = None # [T, rotary_dim] float32# fmt: ondef get_last_indices(self, bs: int) -> torch.Tensor:@@ -129,6 +133,13 @@ class QSASparseAttnBackend(BaseAttnBackend):self._idx_slot = {lid: i for i, lid in enumerate(group.layer_ids)}self.rotary_config = group.rotary_configself._index_cos_sin: torch.Tensor | None = None+ self._section_table: torch.Tensor | None = None+ if group.rotary_config.mrope_section is not None:+ from freetoken.layers.rotary import build_section_table++ self._section_table = build_section_table(+ tuple(group.rotary_config.mrope_section), group.rotary_config.mrope_layout+ ).to(self.device)self._block_topk_kernel = _resolve_block_topk()# decode staging (static buffers under CUDA graphs; eager decode snapshots per step)@@ -175,6 +186,12 @@ class QSASparseAttnBackend(BaseAttnBackend):self._index_cos_sin = rope._cos_sin_cache.to(self.device)return self._index_cos_sin+ def _index_rope_rows(self, positions: torch.Tensor) -> torch.Tensor:+ """Per-token indexer cos/sin rows for [3, n] mrope positions."""+ from freetoken.layers.rotary import mrope_cos_sin_rows++ return mrope_cos_sin_rows(self._index_rope_cache(), positions, self._section_table)+# ----- metadata -----------------------------------------------------------------------def prepare_metadata(self, batch: Batch) -> None:reqs = batch.padded_reqs if hasattr(batch, "padded_reqs") else batch.reqs@@ -298,6 +315,17 @@ class QSASparseAttnBackend(BaseAttnBackend):md.positions = batch.positionsout_loc = batch.out_loc.to(torch.int64)positions = batch.positions.to(torch.int64)+ if self._section_table is not None:+ rope_positions = batch.get_attn_positions()+ if rope_positions.dim() == 1:+ rope_positions = rope_positions.unsqueeze(0).expand(3, -1)+ rope_positions = rope_positions.to(torch.int32)+ self.kvcache.rope_positions.index_copy_(0, out_loc, rope_positions.t().contiguous())+ # groups are ratio-aligned within a page, so the group start is the slot rounded down+ first_pos = self.kvcache.rope_positions.index_select(0, out_loc - out_loc % self.ratio).t()+ md.rope_rows = torch.arange(out_loc.numel(), dtype=torch.int32, device=self.device)+ md.q_rope_cache = self._index_rope_rows(rope_positions)+ md.k_rope_cache = self._index_rope_rows(first_pos)rows = torch.arange(out_loc.numel(), device=self.device)req = md.token_to_req.to(torch.int64)slots = md.ring_slots.to(torch.int64).index_select(0, req)@@ -338,10 +366,14 @@ class QSASparseAttnBackend(BaseAttnBackend):pooled,first,)+ if md.k_rope_cache is None:+ rope_positions, rope_cache = first, self._index_rope_cache()+ else:+ rope_positions, rope_cache = md.rope_rows, md.k_rope_cacheqsa_index_norm_rope(pooled,- first,- self._index_rope_cache(),+ rope_positions,+ rope_cache,index.k_norm_weight,index.eps,self.kvcache.cmp_k_cache(slot),@@ -364,10 +396,14 @@ class QSASparseAttnBackend(BaseAttnBackend):q_index = self._scratch("q_index", rows, self.index_heads, self.index_head_dim, dtype=self.dtype)+ if md.q_rope_cache is None:+ rope_positions, rope_cache = positions, self._index_rope_cache()+ else:+ rope_positions, rope_cache = md.rope_rows, md.q_rope_cacheqsa_index_norm_rope(index.q.view(-1, self.index_head_dim),- positions,- self._index_rope_cache(),+ rope_positions,+ rope_cache,index.q_norm_weight,index.eps,q_index.view(-1, self.index_head_dim),diff --git a/python/freetoken/core.py b/python/freetoken/core.pyindex 281ff41a..269b94ff 100644--- a/python/freetoken/core.py+++ b/python/freetoken/core.py@@ -39,9 +39,10 @@ class Req:uid: intsampling_params: SamplingParamscache_handle: BaseCacheHandle- # Optional precomputed multimodal soft-token embeddings (GPU, [num_image_tokens,- # hidden]) scattered at image-token positions during this request's prefill.- mm_embeds: torch.Tensor | None = None+ # per-item processor outputs and the tokenizer's precomputed mrope rows and delta+ mm_items: list | None = None+ mrope_positions_full: torch.Tensor | None = None # [3, prompt_len] int32, CPU+ mrope_delta: int = 0# --- hybrid-radix (GDN linear-state) per-request slots; None for non-hybrid models or# until allocated from LinearStatePool. Set by the scheduler (P2). ---@@ -115,6 +116,8 @@ class Batch:# these fields should be set by schedulerinput_ids: torch.Tensor = field(init=False)positions: torch.Tensor = field(init=False)+ # [3, n] t/h/w rope positions on mrope models; positions keeps its sequence-index meaning for token_pool / page_table+ mrope_positions: torch.Tensor | None = field(default=None, init=False)out_loc: torch.Tensor | None = field(init=False)# Per-(padded-)request table_idx as a GPU int64 tensor, used by GatedDeltaNet# decode to gather/scatter recurrent+conv state without host-side loops (so the@@ -132,8 +135,12 @@ class Batch:active_table_idx: "torch.Tensor | None" = None# this field should be set by attention backendattn_metadata: BaseAttnMetadata = field(init=False)- # concatenated multimodal soft-token embeddings for a prefill batch (or None)+ # concatenated multimodal soft-token embeddings for a prefill batch (or None) and the batch rows they land onmm_embeds: torch.Tensor | None = field(default=None, init=False)+ mm_rows: torch.Tensor | None = field(default=None, init=False)+ # this chunk's cache-miss items to encode and the gather plan [(uid, hash, row_lo, row_hi, n, pos), ...] in scatter order+ mm_encoder_jobs: list | None = field(default=None, init=False)+ mm_gather_plan: list | None = field(default=None, init=False)# Prefill log stats snapshotted at schedule time (before forward's complete_one()# advances cached_len), so the prefill log reports the tokens actually forwarded and# the prefix-cache hit -- matching SGLang's #new-token / #cached-token. Set by the@@ -154,6 +161,9 @@ class Batch:def is_decode(self) -> bool:return self.phase == "decode"+ def get_attn_positions(self) -> torch.Tensor:+ return self.mrope_positions if self.mrope_positions is not None else self.positions+@propertydef size(self) -> int:return len(self.reqs)diff --git a/python/freetoken/engine/config.py b/python/freetoken/engine/config.pyindex 7ab792f9..44c5a38e 100644--- a/python/freetoken/engine/config.py+++ b/python/freetoken/engine/config.py@@ -1,5 +1,6 @@from __future__ import annotations+import copyfrom dataclasses import dataclass, field, replacefrom functools import cached_propertyfrom typing import TYPE_CHECKING, List@@ -7,7 +8,8 @@ from typing import TYPE_CHECKING, Listimport torchfrom freetoken.distributed import DistributedInfofrom freetoken.layers.quantization import set_quant_config-from freetoken.models.register import _load_attr, checkpoint_quant_config, get_model_spec+from freetoken.mm.config import ENCODER_SECTIONS, MultimodalConfig+from freetoken.models.register import EncoderSpec, ModelSpec, _load_attr, checkpoint_quant_config, get_model_specfrom freetoken.utils import cached_load_hf_config, init_loggerif TYPE_CHECKING:@@ -87,6 +89,8 @@ class EngineConfig:# KV capacity in tokens; resolved into num_page_override by _adjust_config once page_size# is final. Mutually exclusive with num_page_override.num_token_override: int | None = None+ # Runtime knobs of the multimodal path; the architecture side (vision_config, mrope) lives in ModelConfig.+ mm: MultimodalConfig = field(default_factory=MultimodalConfig)def __post_init__(self):if self.moe_backend is None:@@ -101,13 +105,38 @@ class EngineConfig:def hf_config(self):return cached_load_hf_config(self.model_path)+ @cached_property+ def model_spec(self) -> ModelSpec:+ return get_model_spec(self.hf_config.architectures[0])++ @cached_property+ def active_encoders(self) -> tuple[EncoderSpec, ...]:+ """The encoder towers this process builds: the family registers them, the checkpoint config carries their section, --mm-disable did not name them."""+ return tuple(+ e+ for e in self.model_spec.encoders+ if getattr(self.hf_config, e.config_key, None) is not None+ and e.kind not in self.mm.disabled_encoders+ )++ @cached_property+ def served_modalities(self) -> frozenset[str]:+ """Modalities this process accepts."""+ return frozenset(m for e in self.active_encoders for m in e.modalities)+@cached_propertydef model_config(self) -> ModelConfig:- spec = get_model_spec(self.hf_config.architectures[0])- quant = checkpoint_quant_config(self.model_path, self.hf_config, spec)+ # the parser sees no section for a tower this process does not build (for the vision tower that also means 1-D rope)+ hf_config = copy.copy(self.hf_config)+ built = {e.config_key for e in self.active_encoders}+ for key in set(ENCODER_SECTIONS) | {e.config_key for e in self.model_spec.encoders}:+ if key not in built:+ setattr(hf_config, key, None)+ spec = self.model_spec+ quant = checkpoint_quant_config(self.model_path, hf_config, spec)set_quant_config(quant)- parse_config = _load_attr(spec.module, spec.parse_config)- return replace(parse_config(self.hf_config), quant=quant)+ model_config = _load_attr(spec.module, spec.parse_config)(hf_config)+ return replace(model_config, quant=quant)@propertydef max_seq_len(self) -> int:diff --git a/python/freetoken/engine/engine.py b/python/freetoken/engine/engine.pyindex 006c1908..c92e1d88 100644--- a/python/freetoken/engine/engine.py+++ b/python/freetoken/engine/engine.py@@ -14,6 +14,7 @@ from freetoken.gpu_select import gpu_identityfrom freetoken.layers import set_rope_devicefrom freetoken.layers.quantization import LayerKind, QuantBackend, finalize_quant, set_quant_backendfrom freetoken.moe.offload_cache import iter_offload_moe_layers+from freetoken.mm.config import ENCODER_SECTIONSfrom freetoken.models import create_model, load_weightfrom freetoken.moe import is_offload_moe_strategyfrom freetoken.moe.expert_banks import load_expert_banks@@ -33,7 +34,6 @@ from freetoken.kvcache.linear_state_pool import (logger = init_logger(__name__)-def _require_offload_cache_size(cache_size: int, num_experts: int) -> None:"""The offload MoE cache needs at least one slot per expert per layer. A too-small size(e.g. a bare offload run with moe_cache_size unset and auto disabled) must fail loudly."""@@ -328,6 +328,16 @@ class Engine:self.model = create_model(config.model_config)self.model.load_state_dict(self._load_weight_state_dict(config))finalize_quant(self.model)+ if config.active_encoders:+ from freetoken.models.blocks import SupportsMultimodal++ if not isinstance(self.model, SupportsMultimodal):+ raise TypeError(+ f"{type(self.model).__name__} has encoders registered but lacks the SupportsMultimodal hooks; "+ "run with --text-model-only"+ )+ # before the residency snapshot, so streamed blocks are not charged as resident weights+ self.model.place_encoder_weights(config.mm.encoder_weights)post_weights_free = self._sync_get_memory()[0]self._weights_bytes = self._baseline_free - post_weights_free# Pool-budget baseline for the desktop cache sliders: free VRAM after the weights are@@ -348,6 +358,27 @@ class Engine:self._init_offload_moe_cache(config)if hasattr(self.model, "prepare_for_runtime"):self.model.prepare_for_runtime()+ self.encoder_cache = None+ self.mm_processor = None+ if config.active_encoders:+ from freetoken.mm.encoder_cache import EncoderCache+ from freetoken.mm.processor import get_mm_processor++ self.mm_processor = get_mm_processor(config.model_path, config.mm)+ self.encoder_cache = EncoderCache(storage=config.mm.embed_cache_device)+ logger.info_rank0(+ f"Multimodal enabled: {type(self.mm_processor).__name__}, encoders "+ f"{[e.kind for e in config.active_encoders]} on {config.mm.encoder_weights}, serving {sorted(config.served_modalities)}"+ )+ self._warmup_encoders()+ elif any(getattr(config.hf_config, key, None) is not None for key in ENCODER_SECTIONS):+ logger.info_rank0(+ "Multimodal disabled: --text-model-only"+ if config.mm.text_model_only+ else "Multimodal disabled: --mm-disable"+ if config.mm.disabled_encoders+ else "Multimodal disabled: no encoder registered for this architecture"+ )# ======================= KV cache initialization ========================new_free = self._sync_get_memory()[1]@@ -429,6 +460,7 @@ class Engine:vocab_size=config.model_config.vocab_size,dummy_req=self.dummy_req,moe_offload_cache=self.moe_offload_cache,+ mrope=config.model_config.model_is_mrope,)if config.attention_backend.split(",")[0] == "triton":# Prefill runs on the first comma part; warm its autotune cache.@@ -474,10 +506,43 @@ class Engine:config.model_path,self.device,include_moe_experts=not is_offload_moe_strategy(config.moe_strategy),+ include_vision=bool(config.active_encoders),),device=self.device,)+ @torch.inference_mode()+ def _warmup_encoders(self) -> None:+ for item in self.mm_processor.dummy_items(self.dtype, self.device):+ if item.modality in self.config.served_modalities:+ self.model.encode(item)+ torch.cuda.synchronize(self.device)++ @torch.inference_mode()+ def _run_mm_encoder(self, batch: Batch) -> None:+ """Encode the chunk's cache-miss items and gather its embedding rows into batch.mm_embeds, right before the LM forward."""+ cache = self.encoder_cache+ jobs = batch.mm_encoder_jobs or ()+ # a job whose rows are not gathered this chunk would be encoded and freed unread+ planned = {row[1] for row in batch.mm_gather_plan}+ orphans = [item.hash for item in jobs if item.hash not in planned]+ assert not orphans, f"encoder jobs without gather rows: {orphans}"+ for item in jobs:+ if not cache.has(item.hash):+ if item.precomputed_embeddings is not None:+ emb = item.precomputed_embeddings.to(self.device, non_blocking=True)+ else:+ emb = self.model.encode(item)+ cache.put(item.hash, emb)+ # cached now; free the ~MiB feature buffer+ item.feature = None+ item.precomputed_embeddings = None+ parts = []+ for uid, item_hash, lo, hi, _, _ in batch.mm_gather_plan:+ parts.append(cache.get_slice(item_hash, lo, hi, self.device))+ cache.consume(item_hash, uid, hi - lo)+ if parts:+ batch.mm_embeds = torch.cat([p.to(self.dtype) for p in parts], dim=0)def _resolve_auto_moe_cache_size(self, config: EngineConfig, banks, method=None) -> tuple[int, int, bool]:"""Resolve --moe-cache-auto into (moe_cache_size, num_pages, prefill_overlap).@@ -906,10 +971,13 @@ class Engine:vocab_size=config.model_config.vocab_size,dummy_req=self.dummy_req,moe_offload_cache=self.moe_offload_cache,+ mrope=config.model_config.model_is_mrope,)def forward_batch(self, batch: Batch, args: BatchSamplingArgs) -> ForwardOutput:assert torch.cuda.current_stream() == self.stream+ if batch.mm_gather_plan:+ self._run_mm_encoder(batch)use_graph = self.graph_runner.can_use_cuda_graph(batch)with self.ctx.forward_batch(batch), self.model.forward_host_ctx(batch, use_graph):logits = self.graph_runner.replay(batch) if use_graph else self.model.forward()@@ -970,6 +1038,10 @@ class Engine:batch.padded_reqs = batch.reqsbatch.input_ids = torch.zeros(length, dtype=torch.int32, device=self.device)batch.positions = torch.arange(length, dtype=torch.int32, device=self.device)+ if self.config.model_config.model_is_mrope:+ batch.mrope_positions = (+ batch.positions.unsqueeze(0).expand(3, -1).contiguous()+ )batch.out_loc = dummy_row[:length]self.attn_backend.prepare_metadata(batch)with self.ctx.forward_batch(batch):diff --git a/python/freetoken/engine/graph.py b/python/freetoken/engine/graph.pyindex 4f202502..eb9684f6 100644--- a/python/freetoken/engine/graph.py+++ b/python/freetoken/engine/graph.py@@ -24,17 +24,24 @@ class GraphCaptureBuffer:input_ids: torch.Tensorout_loc: torch.Tensorpositions: torch.Tensor+ # [3, bs] t/h/w rope positions; allocated only for mrope models (else None).+ mrope_positions: torch.Tensor | Nonelogits: torch.Tensortable_idx: torch.Tensor # per-request slot id for GatedDeltaNet state gather/scatter# Decode GDN query indptr = arange(bs+1); a constant per captured bs, filled once.fla_cu_seqlens: torch.Tensor@classmethod- def init(cls, bs: int, vocab_size: int, device: torch.device) -> GraphCaptureBuffer:+ def init(+ cls, bs: int, vocab_size: int, device: torch.device, mrope: bool = False+ ) -> GraphCaptureBuffer:return GraphCaptureBuffer(input_ids=torch.zeros(bs, dtype=torch.int32, device=device),out_loc=torch.zeros(bs, dtype=torch.int32, device=device),positions=torch.zeros(bs, dtype=torch.int32, device=device),+ mrope_positions=(+ torch.zeros(3, bs, dtype=torch.int32, device=device) if mrope else None+ ),logits=torch.empty(bs, vocab_size, dtype=torch.float32, device=device),table_idx=torch.zeros(bs, dtype=torch.int32, device=device),fla_cu_seqlens=torch.arange(bs + 1, dtype=torch.int32, device=device),@@ -48,6 +55,8 @@ class GraphCaptureBuffer:batch.input_ids = self.input_ids[_slice]batch.out_loc = self.out_loc[_slice]batch.positions = self.positions[_slice]+ if self.mrope_positions is not None:+ batch.mrope_positions = self.mrope_positions[:, _slice]batch.linear_table_idx = self.table_idx[_slice]# Decode GDN metadata reads the persistent cu_seqlens (constant arange) and the# persistent table_idx slot map, so the captured kernels see stable addresses.@@ -61,6 +70,8 @@ class GraphCaptureBuffer:if batch.out_loc is not None:self.out_loc[_slice] = batch.out_locself.positions[_slice] = batch.positions+ if self.mrope_positions is not None:+ self.mrope_positions[:, _slice] = batch.mrope_positionsif batch.linear_table_idx is not None:self.table_idx[_slice] = batch.linear_table_idx@@ -105,6 +116,7 @@ class GraphRunner:vocab_size: int,dummy_req: Req,moe_offload_cache: OffloadMoeCache | None = None,+ mrope: bool = False,) -> None:cuda_graph_bs = _determine_cuda_graph_bs(cuda_graph_bs=cuda_graph_bs,@@ -116,6 +128,7 @@ class GraphRunner:self.graph_bs_list = sorted(cuda_graph_bs)self.dummy_req = dummy_reqself.moe_offload_cache = moe_offload_cache+ self.mrope = mropeself.stream = streamself.device = deviceself._capture_graphs(max_seq_len, vocab_size, model)@@ -145,7 +158,9 @@ class GraphRunner:free_memory = get_free_memory(self.device)logger.info_rank0(f"Free GPU memory before capturing CUDA graphs: {mem_GB(free_memory)}")- self.buffer = GraphCaptureBuffer.init(self.max_graph_bs, vocab_size, self.device)+ self.buffer = GraphCaptureBuffer.init(+ self.max_graph_bs, vocab_size, self.device, mrope=self.mrope+ )self._reset_moe_offload_cache()pbar = tqdm(diff --git a/python/freetoken/kernel/aot_models.py b/python/freetoken/kernel/aot_models.pyindex 8d066d43..a8ef85d4 100644--- a/python/freetoken/kernel/aot_models.py+++ b/python/freetoken/kernel/aot_models.py@@ -145,6 +145,7 @@ SUPPORTED_MODELS: tuple[AotModel, ...] = (moe_intermediate_size=768,expert_formats=("bf16",),aliases=("Qwen/Qwen3-30B-A3B-Thinking-2507",),+ arch_aliases=("Qwen3VLMoeForConditionalGeneration",), # Qwen3-VL-30B-A3B: same text tower),AotModel(name="Qwen/Qwen3.5-35B-A3B",@@ -154,6 +155,7 @@ SUPPORTED_MODELS: tuple[AotModel, ...] = (top_k=8,moe_intermediate_size=512,expert_formats=("bf16",),+ arch_aliases=("Qwen3_5MoeForCausalLM",), # text-only release of the same tower),AotModel(name="Qwen/Qwen3.5-35B-A3B-FP8",@@ -337,6 +339,7 @@ SUPPORTED_MODELS: tuple[AotModel, ...] = (hidden_size=5120,kv_groups=((4, 256),),aliases=("Qwen/Qwen3.6-27B-FP8", "nvidia/Qwen3.6-27B-NVFP4"),+ arch_aliases=("Qwen3_5ForCausalLM",), # text-only release of the same tower),AotModel(name="google/gemma-4-12B-it",@@ -375,6 +378,7 @@ SUPPORTED_MODELS: tuple[AotModel, ...] = (architecture="Qwen3ForCausalLM",hidden_size=4096,kv_groups=((8, 128),),+ arch_aliases=("Qwen3VLForConditionalGeneration",), # Qwen3-VL-8B: same text tower),AotModel(name="mistralai/Mistral-7B-Instruct-v0.3",diff --git a/python/freetoken/kernel/triton/rope.py b/python/freetoken/kernel/triton/rope.pyindex 0284c82d..4cd25c91 100644--- a/python/freetoken/kernel/triton/rope.py+++ b/python/freetoken/kernel/triton/rope.py@@ -141,3 +141,130 @@ def apply_rope_with_cos_sin_cache_inplace(__all__ = ["apply_rope_with_cos_sin_cache_inplace"]++[email protected](do_not_specialize=["nnz"])+def _mrope_tiled(+ Q, K, POS, CACHE, SEC,+ stride_qbs, stride_qh, stride_qd,+ stride_kbs, stride_kh, stride_kd,+ nnz,+ HEAD_Q, HEAD_K, rotary_dim, half,+ BLOCK_SEQ: tl.constexpr,+ BLOCK_HEAD: tl.constexpr,+ BLOCK_DHALF: tl.constexpr,+):+ # _rope_tiled with 3-axis positions: POS is [3, nnz], SEC[half] picks the axis each frequency slot reads+ seq_pid = tl.program_id(0)+ head_pid = tl.program_id(1)++ seq_range = seq_pid * BLOCK_SEQ + tl.arange(0, BLOCK_SEQ)+ head_range = head_pid * BLOCK_HEAD + tl.arange(0, BLOCK_HEAD)+ d = tl.arange(0, BLOCK_DHALF)++ seq_mask = seq_range < nnz+ dmask = d < half++ sec = tl.load(SEC + d, mask=dmask, other=0).to(tl.int64) # (D,) in {0,1,2}+ pos_ptr = POS + sec[None, :] * nnz + seq_range[:, None]+ cs_mask = seq_mask[:, None] & dmask[None, :]+ pos = tl.load(pos_ptr, mask=cs_mask, other=0).to(tl.int64) # (S,D)+ cache_row = CACHE + pos * rotary_dim+ cos = tl.load(cache_row + d[None, :], mask=cs_mask, other=0.0)+ sin = tl.load(cache_row + half + d[None, :], mask=cs_mask, other=0.0)+ cos = cos[:, None, :]+ sin = sin[:, None, :]++ d0 = d[None, None, :]+ d1 = (half + d)[None, None, :]++ qmask = (seq_mask[:, None, None]+ & (head_range[None, :, None] < HEAD_Q)+ & dmask[None, None, :])+ base_q = seq_range[:, None, None] * stride_qbs + head_range[None, :, None] * stride_qh+ q0 = tl.load(Q + base_q + d0 * stride_qd, mask=qmask, other=0.0).to(tl.float32)+ q1 = tl.load(Q + base_q + d1 * stride_qd, mask=qmask, other=0.0).to(tl.float32)+ tl.store(Q + base_q + d0 * stride_qd, (q0 * cos - q1 * sin).to(Q.dtype.element_ty), mask=qmask)+ tl.store(Q + base_q + d1 * stride_qd, (q1 * cos + q0 * sin).to(Q.dtype.element_ty), mask=qmask)++ kmask = (seq_mask[:, None, None]+ & (head_range[None, :, None] < HEAD_K)+ & dmask[None, None, :])+ base_k = seq_range[:, None, None] * stride_kbs + head_range[None, :, None] * stride_kh+ k0 = tl.load(K + base_k + d0 * stride_kd, mask=kmask, other=0.0).to(tl.float32)+ k1 = tl.load(K + base_k + d1 * stride_kd, mask=kmask, other=0.0).to(tl.float32)+ tl.store(K + base_k + d0 * stride_kd, (k0 * cos - k1 * sin).to(K.dtype.element_ty), mask=kmask)+ tl.store(K + base_k + d1 * stride_kd, (k1 * cos + k0 * sin).to(K.dtype.element_ty), mask=kmask)+++def apply_mrope_with_cos_sin_cache_inplace(+ positions: torch.Tensor,+ query: torch.Tensor,+ key: torch.Tensor,+ head_size: int,+ cos_sin_cache: torch.Tensor,+ section_table: torch.Tensor,+) -> None:+ """3-axis rope: positions is [3, nnz], section_table [rotary_dim//2] maps each frequency slot to its axis row; NeoX layout only."""+ if str(cos_sin_cache.dtype) != "torch.float32":+ raise ValueError("cos_sin_cache should be float32")+ assert positions.dim() == 2 and positions.shape[0] == 3+ assert query.is_cuda and key.is_cuda and positions.is_cuda+ assert cos_sin_cache.is_contiguous() and positions.is_contiguous()++ nnz = query.shape[0]+ if nnz == 0:+ return+ rotary_dim = cos_sin_cache.shape[1]+ half = rotary_dim // 2++ head_q = query.shape[1] // head_size+ head_k = key.shape[1] // head_size+ qv = query.view(nnz, head_q, head_size)+ kv = key.view(nnz, head_k, head_size)+ max_head = max(head_q, head_k)++ grid = lambda META: (+ triton.cdiv(nnz, META["BLOCK_SEQ"]),+ triton.cdiv(max_head, META["BLOCK_HEAD"]),+ )+ _mrope_tiled[grid](+ qv, kv, positions, cos_sin_cache, section_table,+ qv.stride(0), qv.stride(1), qv.stride(2),+ kv.stride(0), kv.stride(1), kv.stride(2),+ nnz, head_q, head_k, rotary_dim, half,+ BLOCK_DHALF=triton.next_power_of_2(half),+ BLOCK_SEQ=16,+ BLOCK_HEAD=1,+ num_warps=4,+ num_stages=1,+ )+++def apply_mrope_torch_fallback(+ positions: torch.Tensor,+ query: torch.Tensor,+ key: torch.Tensor,+ head_size: int,+ cos_sin_cache: torch.Tensor,+ section_table: torch.Tensor,+) -> None:+ """Pure-torch mrope (Triton unavailable); same in-place contract as the kernel."""+ nnz = query.shape[0]+ if nnz == 0:+ return+ rotary_dim = cos_sin_cache.shape[1]+ half = rotary_dim // 2+ pos = positions[section_table.long(), :].transpose(0, 1) # [nnz, half]+ cs = cos_sin_cache[pos] # [nnz, half, 2*half] rows gathered per slot+ idx = torch.arange(half, device=query.device)+ cos = cs[:, idx, idx].unsqueeze(1).float()+ sin = cs[:, idx, half + idx].unsqueeze(1).float()+ for t, heads in ((query, query.shape[1] // head_size), (key, key.shape[1] // head_size)):+ v = t.view(nnz, heads, head_size)+ lo, hi = v[..., :half].float(), v[..., half:rotary_dim].float()+ v[..., :half] = (lo * cos - hi * sin).to(v.dtype)+ v[..., half:rotary_dim] = (hi * cos + lo * sin).to(v.dtype)+++__all__ = ["apply_rope_with_cos_sin_cache_inplace", "apply_mrope_with_cos_sin_cache_inplace"]diff --git a/python/freetoken/kvcache/__init__.py b/python/freetoken/kvcache/__init__.pyindex 41b82f1d..1792da2b 100644--- a/python/freetoken/kvcache/__init__.py+++ b/python/freetoken/kvcache/__init__.py@@ -204,6 +204,7 @@ def create_kvcache_pool(index_ratio=spec.index_ratio,num_req_slots=num_req_slots,layer_ids=spec.layer_ids,+ mrope=model_config.model_is_mrope,)if len(kv_specs) == 1 and kv_specs[0].mla:diff --git a/python/freetoken/kvcache/qsa_pool.py b/python/freetoken/kvcache/qsa_pool.pyindex fddcdbd3..7de70977 100644--- a/python/freetoken/kvcache/qsa_pool.py+++ b/python/freetoken/kvcache/qsa_pool.py@@ -31,6 +31,8 @@ from .mha_pool import MHAKVCache# The index tiers are always 2-byte (compute dtype); spec_kv_bytes_per_token budgets the same._INDEX_DTYPE_BYTES = 2+# t/h/w int32 rope position kept per KV slot on mrope models+_ROPE_POS_BYTES = 3 * 4class QSAKVCache(MHAKVCache):@@ -63,6 +65,7 @@ class QSAKVCache(MHAKVCache):num_req_slots: int,ring_capacity: int | None = None,layer_ids: Sequence[int] | None = None,+ mrope: bool = False,) -> None:if index_ratio < 1 or page_size % index_ratio != 0:# slot // index_ratio only names one group when a group never straddles a page.@@ -89,6 +92,7 @@ class QSAKVCache(MHAKVCache):self._ring_capacity = ring_capacityself._index_dtype = dtypeself._page_size = page_size+ self._mrope = mropesuper().__init__(num_kv_heads=num_kv_heads,num_layers=num_layers,@@ -128,6 +132,12 @@ class QSAKVCache(MHAKVCache):dtype=self._index_dtype,device=self._device,)+ # 3-axis rope position of every stored token: a compressed group ropes at its first token, which under mrope is not derivable from the logical position+ self._rope_positions = (+ torch.zeros(num_pages * self._page_size, 3, dtype=torch.int32, device=self._device)+ if self._mrope+ else None+ )def rebuild(self, num_pages: int) -> None:# Free the index tiers BEFORE the K/V realloc (super().rebuild frees + syncs +@@ -137,6 +147,7 @@ class QSAKVCache(MHAKVCache):# cannot drop a live request's pending members.self._cmp_k_buffer = Noneself._pending_ring = None+ self._rope_positions = Nonesuper().rebuild(num_pages)self._zero_kv_slabs()try:@@ -163,6 +174,8 @@ class QSAKVCache(MHAKVCache):# One index-key row = all index layers at one position.row = spec.index_head_dim * spec.num_index_layers * _INDEX_DTYPE_BYTESfixed += num_req_slots * row * (cls.ring_capacity_for(spec.index_ratio) + 1)+ if config.model_config.model_is_mrope:+ per_token += _ROPE_POS_BYTESreturn per_token * config.page_size, fixed, config.page_size, 0def unit_bytes(self) -> tuple[int, int]:@@ -176,7 +189,7 @@ class QSAKVCache(MHAKVCache):* self._index_head_dim* self._index_dtype.itemsize)- return kv + slab // tokens, swa+ return kv + slab // tokens + (_ROPE_POS_BYTES if self._mrope else 0), swadef cmp_k_cache(self, slot: int) -> torch.Tensor:"""Compressed index keys of one sparse layer: ``[rows, index_head_dim]``."""@@ -186,6 +199,12 @@ class QSAKVCache(MHAKVCache):"""One sparse layer's pending ring: ``[num_req_slots, ring_capacity, index_head_dim]``."""return self._pending_ring[:, slot]+ @property+ def rope_positions(self) -> torch.Tensor:+ """``[num_tokens, 3]`` int32 t/h/w rope position per KV slot (written by the QSA backend)."""+ assert self._rope_positions is not None, "rope positions are only kept on mrope models"+ return self._rope_positions+@propertydef cmp_scratch_base(self) -> int:"""First scratch row of ``cmp_k_cache``; row ``cmp_scratch_base + table_idx`` sinks adiff --git a/python/freetoken/layers/rotary.py b/python/freetoken/layers/rotary.pyindex 3756a299..0d04645d 100644--- a/python/freetoken/layers/rotary.py+++ b/python/freetoken/layers/rotary.py@@ -85,6 +85,114 @@ class RotaryEmbedding(StateLessOP):return query, key+def mrope_cos_sin_rows(+ cos_sin_cache: torch.Tensor, positions: torch.Tensor, section_table: torch.Tensor+) -> torch.Tensor:+ """Per-token [cos | sin] rows for [3, n] positions: frequency slot i reads the cache row of axis section_table[i]."""+ half = cos_sin_cache.shape[1] // 2+ pos = positions[section_table.long(), :].transpose(0, 1).long() # [n, half]+ idx = torch.arange(half, device=cos_sin_cache.device)+ return torch.cat((cos_sin_cache[pos, idx], cos_sin_cache[pos, half + idx]), dim=1)+++def _mrope_torch(+ positions: torch.Tensor,+ query: torch.Tensor,+ key: torch.Tensor,+ head_size: int,+ cos_sin_cache: torch.Tensor,+ section_table: torch.Tensor,+) -> None:+ """Pure-torch mrope for triton-less installs; same in-place NeoX contract as the kernel."""+ nnz = query.shape[0]+ if nnz == 0:+ return+ rotary_dim = cos_sin_cache.shape[1]+ half = rotary_dim // 2+ rows = mrope_cos_sin_rows(cos_sin_cache, positions, section_table)+ cos = rows[:, :half].unsqueeze(1).float()+ sin = rows[:, half:].unsqueeze(1).float()+ for t, heads in ((query, query.shape[1] // head_size), (key, key.shape[1] // head_size)):+ v = t.view(nnz, heads, head_size)+ lo, hi = v[..., :half].float(), v[..., half:rotary_dim].float()+ v[..., :half] = (lo * cos - hi * sin).to(v.dtype)+ v[..., half:rotary_dim] = (hi * cos + lo * sin).to(v.dtype)+++MROPE_LAYOUTS = ("contiguous", "interleaved", "interleaved_glm")+++def build_section_table(mrope_section: tuple[int, int, int], layout: str) -> torch.Tensor:+ """Axis id (0=t, 1=h, 2=w) per frequency slot for a [t, h, w] section split.++ contiguous: T|H|W blocks. interleaved: H at slots 1,4,.. and W at 2,5,.. until each share is used, T takes the rest.+ interleaved_glm: round-robin over the three axes, skipping an axis once its share is used up.+ """+ st, sh, sw = mrope_section+ half = st + sh + sw+ sec = torch.zeros(half, dtype=torch.int32)+ if layout == "contiguous":+ sec[st : st + sh] = 1+ sec[st + sh :] = 2+ elif layout == "interleaved":+ sec[1 : 3 * sh : 3] = 1+ sec[2 : 3 * sw : 3] = 2+ if int((sec == 1).sum()) != sh or int((sec == 2).sum()) != sw:+ raise ValueError(f"mrope_section {mrope_section} is not representable in the interleaved layout")+ elif layout == "interleaved_glm":+ counts = [0, 0, 0]+ for i in range(half):+ ax = i % 3+ while counts[ax] >= mrope_section[ax]:+ ax = (ax + 1) % 3+ sec[i] = ax+ counts[ax] += 1+ else:+ raise ValueError(f"unknown mrope layout {layout!r}; expected one of {MROPE_LAYOUTS}")+ return sec+++class MRotaryEmbedding(RotaryEmbedding):+ """3-axis (t/h/w) rope over the parent's cos_sin_cache; section_table picks the axis row per frequency slot. Consumes positions [3, n]."""++ def __init__(+ self, *args, mrope_section: tuple, layout: str = "interleaved", **kwargs+ ) -> None:+ super().__init__(*args, **kwargs)+ assert self.is_neox, "mrope is defined on the NeoX half-rotation layout"+ half = self.rotary_dim // 2+ assert sum(mrope_section) == half, (mrope_section, half)+ self._section_table = build_section_table(tuple(mrope_section), layout)+ try:+ from freetoken.kernel.triton.rope import apply_mrope_with_cos_sin_cache_inplace++ self._kernel = apply_mrope_with_cos_sin_cache_inplace+ except ImportError:+ self._kernel = None++ def forward(+ self,+ positions: torch.Tensor,+ query: torch.Tensor,+ key: torch.Tensor,+ ) -> Tuple[torch.Tensor, torch.Tensor]:+ assert positions.dim() == 2, "mrope models feed [3, n] positions"+ if self._section_table.device != query.device:+ self._section_table = self._section_table.to(query.device)+ positions = positions.contiguous()+ if self._kernel is not None:+ self._kernel(+ positions=positions, query=query, key=key, head_size=self.head_size,+ cos_sin_cache=self._cos_sin_cache, section_table=self._section_table,+ )+ else:+ _mrope_torch(+ positions, query, key, self.head_size,+ self._cos_sin_cache, self._section_table,+ )+ return query, key++def _get_rope(head_dim: int,rotary_dim: int,@@ -217,8 +325,21 @@ def get_rope(base: float,rope_scaling: Tuple[Tuple[str, Any], ...] | None = None,is_neox: bool = True,+ mrope_section: Tuple[int, ...] | None = None,+ mrope_layout: str = "interleaved",) -> RotaryEmbedding:rope_map = dict(rope_scaling) if rope_scaling is not None else None++ def build() -> RotaryEmbedding:+ if mrope_section is not None:+ assert rope_map is None or rope_map.get("rope_type", "default") == "default"+ return MRotaryEmbedding(+ head_dim, rotary_dim, max_position, base,+ is_neox=is_neox, mrope_section=tuple(mrope_section),+ layout=mrope_layout,+ )+ return _get_rope(head_dim, rotary_dim, max_position, base, rope_map, is_neox)+t = torch.tensor([])if t.device == torch.device("meta"):# we cannot use meta device for rope@@ -227,8 +348,15 @@ def get_rope("We cannot use meta device for rope. Please call set_rope_device() first.")with torch.device(_ROPE_DEVICE):- return _get_rope(head_dim, rotary_dim, max_position, base, rope_map, is_neox)- return _get_rope(head_dim, rotary_dim, max_position, base, rope_map, is_neox)---__all__ = ["get_rope", "RotaryEmbedding", "set_rope_device"]+ return build()+ return build()+++__all__ = [+ "MROPE_LAYOUTS",+ "MRotaryEmbedding",+ "RotaryEmbedding",+ "build_section_table",+ "get_rope",+ "set_rope_device",+]diff --git a/python/freetoken/llm/llm.py b/python/freetoken/llm/llm.pyindex 1d4aaef0..20ae0874 100644--- a/python/freetoken/llm/llm.py+++ b/python/freetoken/llm/llm.py@@ -37,23 +37,12 @@ class LLM(Scheduler):**kwargs,)super().__init__(config)- self.pending_requests: List[Tuple[List[int] | str, SamplingParams]] = []+ self.pending_requests: List[Tuple[List[int] | str, SamplingParams, List[bytes] | None]] = []self.status_map: Dict[int, RequestStatus] = {}- self.mm_embeds_map: Dict[int, torch.Tensor] = {}self.counter = 0+ from freetoken.mm.processor import get_mm_processor- @torch.inference_mode()- def encode_images(- self, pixel_values: torch.Tensor, image_position_ids: torch.Tensor- ) -> torch.Tensor:- """Run the vision tower + projector on processor outputs, returning the- ``[num_image_tokens, hidden]`` soft-token embeddings (on device)."""- model = self.engine.model- if not hasattr(model, "encode_images"):- raise RuntimeError(f"{type(model).__name__} does not support image inputs")- return model.encode_images(- pixel_values.to(self.device), image_position_ids.to(self.device)- )+ self._mm_processor = get_mm_processor(model_path, config.mm)def _tokenize_one(self, prompt: List[int] | str) -> torch.Tensor:if isinstance(prompt, str):@@ -66,25 +55,31 @@ class LLM(Scheduler):raise RequestAllFinished()results: List[BaseBackendMsg] = []added, sum_input_len = 0, 0- for tokens_or_prompt, sampling_params in self.pending_requests:+ for tokens_or_prompt, sampling_params, images in self.pending_requests:if sum_input_len >= self.prefill_budget:breakinput_ids = self._tokenize_one(tokens_or_prompt)- sum_input_len += len(input_ids)- uid, added = self.counter + added, added + 1- results.append(- UserMsg(- uid=uid,+ msg = UserMsg(uid=0, input_ids=input_ids, sampling_params=sampling_params)+ if images:+ if self._mm_processor is None:+ raise ValueError("image input is not supported for this model")+ r = self._mm_processor.apply(input_ids, images)+ input_ids = r.input_ids+ msg = UserMsg(+ uid=0,input_ids=input_ids,sampling_params=sampling_params,- mm_embeds=self.mm_embeds_map.get(uid),+ mm_items=r.mm_items,+ mrope_positions=r.mrope_positions,+ mrope_delta=r.mrope_delta,)- )+ sum_input_len += len(input_ids)+ uid, added = self.counter + added, added + 1+ msg.uid = uid+ results.append(msg)self.status_map[uid] = RequestStatus(uid=uid,- input_ids=(- input_ids.tolist() if isinstance(tokens_or_prompt, str) else tokens_or_prompt- ),+ input_ids=input_ids.tolist(),output_ids=[],)self.counter += added@@ -106,30 +101,18 @@ class LLM(Scheduler):self,prompts: List[str] | List[List[int]],sampling_params: List[SamplingParams] | SamplingParams,- mm_inputs: List[Dict[str, torch.Tensor] | None] | None = None,+ images: List[List[bytes] | None] | None = None,) -> List[Dict[str, str | List[int]]]:- """Offline generation.-- ``mm_inputs`` (optional) is aligned with ``prompts``; each entry is either- ``None`` (text-only) or a dict with ``pixel_values`` ``[N, P, 3*patch**2]`` and- ``image_position_ids`` ``[N, P, 2]`` from the HF processor. For multimodal- prompts pass token-id ``prompts`` containing ``image_token_id`` placeholders.- """+ """Offline generation; images is aligned with prompts: the raw image files of each prompt in placeholder order, or None."""self.pending_requests = []self.status_map = {}- self.mm_embeds_map = {}self.counter = 0if isinstance(sampling_params, SamplingParams):sampling_params = [sampling_params] * len(prompts)- for prompt, sp in zip(prompts, sampling_params):- self.pending_requests.append((prompt, sp))- if mm_inputs is not None:- for uid, mm in enumerate(mm_inputs):- if mm is not None:- self.mm_embeds_map[uid] = self.encode_images(- mm["pixel_values"], mm["image_position_ids"]- )- torch.cuda.synchronize(self.device)+ if images is None:+ images = [None] * len(prompts)+ for prompt, sp, imgs in zip(prompts, sampling_params, images, strict=True):+ self.pending_requests.append((prompt, sp, imgs))try:self.run_forever()except RequestAllFinished:diff --git a/python/freetoken/message/__init__.py b/python/freetoken/message/__init__.pyindex e9279f0b..c22cb70d 100644--- a/python/freetoken/message/__init__.py+++ b/python/freetoken/message/__init__.py@@ -4,6 +4,7 @@ from .backend import (BatchBackendMsg,CacheRebuildBackendMsg,ExitMsg,+ MMItem,UserMsg,)from .frontend import BaseFrontendMsg, BatchFrontendMsg, CacheRebuildReply, UserReply@@ -26,6 +27,7 @@ __all__ = ["BatchBackendMsg","CacheRebuildBackendMsg","ExitMsg",+ "MMItem","UserMsg","BaseTokenizerMsg","BatchTokenizerMsg",diff --git a/python/freetoken/message/backend.py b/python/freetoken/message/backend.pyindex c42ecc5a..ccc1d792 100644--- a/python/freetoken/message/backend.py+++ b/python/freetoken/message/backend.py@@ -1,7 +1,7 @@from __future__ import annotations-from dataclasses import dataclass-from typing import Dict, List+from dataclasses import dataclass, field+from typing import Any, Dict, Listimport torchfrom freetoken.core import SamplingParams@@ -29,14 +29,56 @@ class ExitMsg(BaseBackendMsg):pass+@dataclass+class MMItem:+ """One multimodal input of a request (an image, video or audio clip).++ Family extras from the processor live in model_specific_data and read back as attributes (item.grid_thw).+ offsets are half-open [start, end) spans in input_ids; an item may occupy several.+ Exactly one of feature / precomputed_embeddings is set.+ """++ modality: str+ hash: int+ pad_value: int+ offsets: List[List[int]]+ feature: torch.Tensor | None = None # raw processor output (CPU), encoded by the engine+ precomputed_embeddings: torch.Tensor | None = None # final encoder output, skips encoding+ model_specific_data: Dict[str, Any] = field(default_factory=dict)++ def __getattr__(self, name: str) -> Any:+ data = self.__dict__.get("model_specific_data")+ if data is not None and name in data:+ return data[name]+ raise AttributeError(f"'{type(self).__name__}' object has no attribute '{name}'")++ @property+ def num_tokens(self) -> int:+ return sum(end - start for start, end in self.offsets)++ def is_image(self) -> bool:+ return self.modality == "image"++ def validate(self) -> None:+ if not self.offsets or any(end <= start for start, end in self.offsets):+ raise ValueError(f"MMItem offsets must be non-empty half-open spans: {self.offsets}")+ if (self.feature is None) == (self.precomputed_embeddings is None):+ raise ValueError("MMItem needs exactly one of feature / precomputed_embeddings")++ def encoder(self) -> Dict:+ return serialize_type(self)++@dataclassclass UserMsg(BaseBackendMsg):uid: intinput_ids: torch.Tensor # CPU 1D int32 tensorsampling_params: SamplingParams- # Optional precomputed multimodal soft-token embeddings (GPU tensor). Only used by- # the in-process offline path; remains None for the (serialized) online path.- mm_embeds: torch.Tensor | None = None+ # per-image processor outputs, in prompt order+ mm_items: List[MMItem] | None = None+ # precomputed [3, len(input_ids)] mrope positions and decode delta; None for text-only requests and 1-D rope models+ mrope_positions: torch.Tensor | None = None+ mrope_delta: int = 0@dataclassdiff --git a/python/freetoken/message/tokenizer.py b/python/freetoken/message/tokenizer.pyindex 33b75c78..9442ee05 100644--- a/python/freetoken/message/tokenizer.py+++ b/python/freetoken/message/tokenizer.py@@ -72,6 +72,7 @@ class TokenizeMsg(BaseTokenizerMsg):sampling_params: SamplingParamschat_template_kwargs: Dict[str, Any] | None = Nonetools: List[Dict[str, Any]] | None = None+ images: List[bytes] | None = None@dataclassdiff --git a/python/freetoken/message/utils.py b/python/freetoken/message/utils.pyindex ee92adf5..bf2c260e 100644--- a/python/freetoken/message/utils.py+++ b/python/freetoken/message/utils.py@@ -12,7 +12,6 @@ _TYPE_KEY = "__type__"# reading it as a serialized class -- without this, a request could crash the tokenizer worker._RAW_DICT_KEY = "__raw_dict__"-def _serialize_any(value: Any) -> Any:if isinstance(value, dict):encoded = {k: _serialize_any(v) for k, v in value.items()}@@ -32,10 +31,16 @@ def serialize_type(self) -> Dict:serialized = {}if isinstance(self, torch.Tensor):- assert self.dim() == 1, "we can only serialize 1D tensor for now"+ assert not self.is_cuda, "wire tensors must live on CPU"+ t = self.contiguous()serialized["__type__"] = "Tensor"- serialized["buffer"] = self.numpy().tobytes()- serialized["dtype"] = str(self.dtype)+ serialized["dtype"] = str(t.dtype)+ # 1-D tensors omit the shape so the payload matches the legacy wire format.+ if t.dim() != 1:+ serialized["shape"] = list(t.shape)+ if t.dtype == torch.bfloat16:+ t = t.view(torch.uint16) # numpy has no bf16; ship the raw bytes+ serialized["buffer"] = t.numpy().tobytes()return serialized# normal type@@ -64,14 +69,17 @@ def _deserialize_any(cls_map: Dict[str, Type], data: Any) -> Any:def deserialize_type(cls_map: Dict[str, Type], data: Dict) -> Any:type_name = data["__type__"]- # we can only serialize 1D tensor for nowif type_name == "Tensor":buffer = data["buffer"]dtype_str = data["dtype"].replace("torch.", "")- np_dtype = getattr(np, dtype_str)assert isinstance(buffer, bytes)- np_tensor = np.frombuffer(buffer, dtype=np_dtype)- return torch.from_numpy(np_tensor.copy())+ is_bf16 = dtype_str == "bfloat16"+ np_tensor = np.frombuffer(buffer, dtype=getattr(np, "uint16" if is_bf16 else dtype_str))+ tensor = torch.from_numpy(np_tensor.copy())+ if is_bf16:+ tensor = tensor.view(torch.bfloat16)+ shape = data.get("shape")+ return tensor if shape is None else tensor.view(shape)cls = cls_map.get(type_name)if cls is None:diff --git a/python/freetoken/mm/__init__.py b/python/freetoken/mm/__init__.pynew file mode 100644index 00000000..4a7ebcd9--- /dev/null+++ b/python/freetoken/mm/__init__.py@@ -0,0 +1,33 @@+"""Multimodal input handling: pad-id scheme, per-family processors, media fetch, embedding cache."""++from __future__ import annotations++import torch++# image spans become content pseudo ids above every real token id: the radix cache keys them by content and the model finds image rows by id >= MM_PAD_SHIFT_VALUE+# they are clamped back into the vocab before the embedding lookup; those rows are overwritten anyway+MM_PAD_SHIFT_VALUE = 1_000_000+++def restore_placeholder(input_ids: torch.Tensor, token_id: int) -> torch.Tensor:+ """input_ids with every content pad id put back to the family's placeholder token."""+ return input_ids.masked_fill(input_ids >= MM_PAD_SHIFT_VALUE, token_id)+++def mm_pad_value(item_hash: int) -> int:+ return MM_PAD_SHIFT_VALUE + item_hash % (1 << 30)+++def check_mm_pad_shift(vocab_size: int) -> None:+ if vocab_size > MM_PAD_SHIFT_VALUE:+ raise ValueError(+ f"vocab_size ({vocab_size}) exceeds MM_PAD_SHIFT_VALUE ({MM_PAD_SHIFT_VALUE}); "+ "image pseudo-ids would alias real token ids"+ )+++__all__ = [+ "MM_PAD_SHIFT_VALUE",+ "check_mm_pad_shift",+ "mm_pad_value",+]diff --git a/python/freetoken/mm/config.py b/python/freetoken/mm/config.pynew file mode 100644index 00000000..e76322a7--- /dev/null+++ b/python/freetoken/mm/config.py@@ -0,0 +1,33 @@+"""Runtime knobs of the multimodal path. The architecture side (vision_config, mrope) lives in ModelConfig."""++from __future__ import annotations++from dataclasses import dataclass, field+from typing import Any, Literal++# encoder tower kinds a family can register; --mm-disable <kind> leaves that tower unbuilt and refuses its inputs+ENCODER_KINDS = ("vision", "audio")+# checkpoint config sections that describe encoder towers; the parser never sees the section of a tower this process does not build+ENCODER_SECTIONS = ("vision_config", "audio_config")+++@dataclass(frozen=True)+class MultimodalConfig:+ # encoder kinds (ENCODER_KINDS) whose tower this process does not build; --text-model-only names them all+ disabled_encoders: frozenset[str] = frozenset()+ # Where encoded items wait between prefill chunks. "cpu": pinned host memory, "cuda": the device.+ embed_cache_device: Literal["cpu", "cuda"] = "cpu"+ # Encoder tower block weights. "host": pinned host banks streamed two blocks at a time behind the compute, "gpu": resident.+ encoder_weights: Literal["gpu", "host"] = "host"+ # per-image token budget; the family's MMProcessor converts it to its image processor's own limits, None keeps the checkpoint defaults+ image_min_tokens: int | None = None+ image_max_tokens: int | None = None+ # extra keyword arguments the family's MMProcessor passes to the image processor call, after the token budget+ processor_kwargs: dict[str, Any] = field(default_factory=dict)++ @property+ def text_model_only(self) -> bool:+ return set(ENCODER_KINDS) <= self.disabled_encoders+++__all__ = ["ENCODER_KINDS", "ENCODER_SECTIONS", "MultimodalConfig"]diff --git a/python/freetoken/mm/encoder_cache.py b/python/freetoken/mm/encoder_cache.pynew file mode 100644index 00000000..dab51bf0--- /dev/null+++ b/python/freetoken/mm/encoder_cache.py@@ -0,0 +1,87 @@+"""Encoder embedding cache keyed by content hash; an entry lives while any request still has rows of it to gather."""++from __future__ import annotations++from dataclasses import dataclass, field+from typing import Dict++import torch++from freetoken.utils import init_logger++logger = init_logger(__name__)+++@dataclass+class _Entry:+ tensor: torch.Tensor | None = None+ # uid -> embedding rows the request still has to gather, summed over its occurrences of the image+ remaining: Dict[int, int] = field(default_factory=dict)+++class EncoderCache:+ def __init__(self, storage: str = "cpu") -> None:+ assert storage in ("cpu", "cuda")+ self._storage = storage+ self._entries: Dict[int, _Entry] = {}++ def register(self, item_hash: int, uid: int, rows: int) -> None:+ """Claim rows of the image for uid before any of its chunks run; a repeated image adds up."""+ if rows <= 0:+ return+ entry = self._entries.setdefault(item_hash, _Entry())+ entry.remaining[uid] = entry.remaining.get(uid, 0) + rows++ def has(self, item_hash: int) -> bool:+ entry = self._entries.get(item_hash)+ return entry is not None and entry.tensor is not None++ def put(self, item_hash: int, embedding: torch.Tensor) -> None:+ entry = self._entries.get(item_hash)+ assert entry is not None, f"image {item_hash} encoded before any request registered it"+ if entry.tensor is not None:+ return+ if self._storage == "cpu":+ stored = torch.empty(+ embedding.shape, dtype=embedding.dtype, device="cpu",+ pin_memory=torch.cuda.is_available(),+ )+ stored.copy_(embedding, non_blocking=True)+ else:+ stored = embedding+ entry.tensor = stored++ def get_slice(self, item_hash: int, lo: int, hi: int, device: torch.device) -> torch.Tensor:+ view = self._entries[item_hash].tensor[lo:hi]+ if view.device == device:+ return view+ return view.to(device, non_blocking=True)++ def consume(self, item_hash: int, uid: int, rows: int) -> None:+ """Account rows uid gathered; the last row of the last holder frees the entry."""+ entry = self._entries[item_hash]+ left = entry.remaining[uid] - rows+ assert left >= 0, f"request {uid} gathered more rows of image {item_hash} than it registered"+ if left:+ entry.remaining[uid] = left+ return+ del entry.remaining[uid]+ if not entry.remaining:+ del self._entries[item_hash]++ def release(self, uid: int, hashes: list[int]) -> None:+ """Drop uid's claims whatever it gathered (abort); entries nobody else holds are freed."""+ for h in hashes:+ entry = self._entries.get(h)+ if entry is None:+ continue+ entry.remaining.pop(uid, None)+ if not entry.remaining:+ del self._entries[h]++ def stats(self) -> tuple[int, int]:+ stored = [e.tensor for e in self._entries.values() if e.tensor is not None]+ return len(stored), sum(t.numel() * t.element_size() for t in stored)+++__all__ = ["EncoderCache"]diff --git a/python/freetoken/mm/media.py b/python/freetoken/mm/media.pynew file mode 100644index 00000000..5c3c4ef7--- /dev/null+++ b/python/freetoken/mm/media.py@@ -0,0 +1,112 @@+"""Client-supplied image handling: ref collection, the accept gate, and byte fetching.++Raises plain ValueError on bad input; the server layer maps it to its wire error type.+"""++from __future__ import annotations++import base64+from typing import TYPE_CHECKING, Any++if TYPE_CHECKING:+ from freetoken.server.args import ServerArgs++_MAX_IMAGE_BYTES = 32 << 20+++def collect_image_refs(messages: list[dict[str, Any]]) -> list[dict[str, Any]]:+ """Pop the image refs out of rendered messages in prompt order; the bare {"type": "image"} parts stay for the chat template."""+ refs: list[dict[str, Any]] = []+ for m in messages:+ content = m.get("content")+ if not isinstance(content, list):+ continue+ for part in content:+ if isinstance(part, dict) and part.get("type") == "image":+ ref = part.pop("freetoken_ref", None)+ if ref:+ refs.append(ref)+ return refs+++def image_reject_reason(config: ServerArgs) -> str | None:+ """None when this server can accept image inputs, else the client-facing reason."""+ if "image" in config.served_modalities:+ return None+ if config.mm.text_model_only:+ return "image input is disabled on this server (--text-model-only)"+ if "vision" in config.mm.disabled_encoders:+ return "image input is disabled on this server (--mm-disable)"+ return "the served model has no vision support"+++def _check_media_domain(url: str, config: ServerArgs) -> None:+ """Reject a URL whose hostname is not in --allowed-media-domains; empty allowlist admits any domain."""+ from urllib.parse import urlparse++ raw = config.allowed_media_domains+ allowed = {d.strip().lower().rstrip(".") for d in raw.split(",") if d.strip()}+ if not allowed:+ return+ host = (urlparse(url).hostname or "").rstrip(".")+ if host not in allowed:+ raise ValueError(+ f"the URL must be from one of the allowed domains: {sorted(allowed)}; "+ f"input URL domain: {host or '<none>'}"+ )+++def _load_local_media(url: str, config: ServerArgs) -> bytes:+ """Read a file:// ref; the resolved path must be a strict subpath of --allowed-local-media-path."""+ from pathlib import Path+ from urllib.parse import urlparse+ from urllib.request import url2pathname++ root = config.allowed_local_media_path+ if not root:+ raise ValueError("cannot load local files without --allowed-local-media-path")+ spec = urlparse(url)+ filepath = Path(url2pathname((spec.netloc or "") + (spec.path or "")))+ # resolve() follows symlinks, so a link inside the root escaping it is rejected too+ resolved = filepath.resolve()+ if Path(root).resolve() not in resolved.parents:+ raise ValueError(+ f"the file path {filepath} must be a subpath of "+ f"--allowed-local-media-path {root}"+ )+ return resolved.read_bytes()+++async def fetch_image_bytes(refs: list[dict[str, Any]], config: ServerArgs) -> list[bytes]:+ """Resolve collected image refs (URLs / base64) to raw bytes, in order."""+ out: list[bytes] = []+ for ref in refs:+ data = ref.get("data") or ""+ try:+ if ref.get("kind") == "b64":+ out.append(base64.b64decode(data))+ elif data.startswith("data:"):+ out.append(base64.b64decode(data.split(",", 1)[1]))+ elif data.startswith(("http://", "https://")):+ import httpx++ _check_media_domain(data, config)+ # redirect targets are not re-checked against the allowlist+ async with httpx.AsyncClient(timeout=30.0, follow_redirects=True) as client:+ resp = await client.get(data)+ resp.raise_for_status()+ if len(resp.content) > _MAX_IMAGE_BYTES:+ raise ValueError(f"image exceeds {_MAX_IMAGE_BYTES} bytes")+ out.append(resp.content)+ elif data.startswith("file://"):+ out.append(_load_local_media(data, config))+ else:+ raise ValueError(+ "unsupported image source (expect http(s)/file url or base64)"+ )+ except Exception as exc: # noqa: BLE001 -- input-driven, client-classifiable+ raise ValueError(f"could not load image: {exc}") from exc+ return out+++__all__ = ["collect_image_refs", "fetch_image_bytes", "image_reject_reason"]diff --git a/python/freetoken/mm/processor.py b/python/freetoken/mm/processor.pynew file mode 100644index 00000000..48211a5b--- /dev/null+++ b/python/freetoken/mm/processor.py@@ -0,0 +1,215 @@+"""Per-family multimodal processors and the architecture registry; MMProcessor.apply is the image half of tokenization."""++from __future__ import annotations++from abc import ABC, abstractmethod+import hashlib+import importlib+import io+import threading+from dataclasses import dataclass+from typing import Any++import torch++from freetoken.message import MMItem+from freetoken.mm.config import MultimodalConfig+from freetoken.mm import check_mm_pad_shift+++@dataclass+class PromptReplacement:+ """The token sequence that replaces one placeholder; is_embed marks the positions that take image embeddings (None = all of them)."""++ full: list[int]+ is_embed: list[bool] | None = None++ @classmethod+ def select_token_id(cls, full: list[int], embed_token_id: int) -> PromptReplacement:+ return cls(full, [t == embed_token_id for t in full])++ def embed_spans(self) -> list[list[int]]:+ """Half-open [lo, hi) runs of embedding positions, relative to the start of full."""+ mask = self.is_embed if self.is_embed is not None else [True] * len(self.full)+ spans: list[list[int]] = []+ lo: int | None = None+ for i, flag in enumerate([*mask, False]):+ if flag and lo is None:+ lo = i+ elif not flag and lo is not None:+ spans.append([lo, i])+ lo = None+ return spans+++@dataclass+class MMResult:+ input_ids: torch.Tensor+ mm_items: list[MMItem]+ mrope_positions: torch.Tensor | None+ mrope_delta: int+++def _find_all(ids: list[int], target: list[int]) -> list[int]:+ """Start index of every non-overlapping occurrence of target."""+ n = len(target)+ found: list[int] = []+ i = 0+ while i <= len(ids) - n:+ if ids[i : i + n] == target:+ found.append(i)+ i += n+ else:+ i += 1+ return found+++class MMProcessor(ABC):+ """How a checkpoint's images become items, tokens and positions; apply is the family-agnostic driver, subclasses fill the hooks.++ Built from the checkpoint config alone and import-light: tokenizer workers call apply without the model layer, the engine calls dummy_items.+ Image preprocessing must stay on the CPU with one fixed implementation: radix keys and the embedding cache are content hashes of pixel_values, and resize results differ across backends/devices.+ """++ # the token sequence the chat template renders once per image+ placeholder: list[int]++ def __init__(self, model_path: str, mm: MultimodalConfig) -> None:+ self.model_path = model_path+ self.mm = mm+ self._image_processor_instance: Any = None+ self._image_processor_lock = threading.Lock()++ def _image_processor(self) -> Any:+ """The checkpoint's image processor, loaded on first use (tokenizer workers share the instance across threads)."""+ with self._image_processor_lock:+ if self._image_processor_instance is None:+ from transformers import AutoImageProcessor++ self._image_processor_instance = AutoImageProcessor.from_pretrained(self.model_path)+ return self._image_processor_instance++ def get_mm_processor_kwargs(self, mm: MultimodalConfig) -> dict[str, Any]:+ """Keyword arguments for the checkpoint processor call: the family's reading of the mm knobs, then mm.processor_kwargs on top; families with a token budget override this."""+ return dict(mm.processor_kwargs)++ @abstractmethod+ def process(self, images: list[Any]) -> list[MMItem]:+ """One MMItem per image with feature and hash; offsets are assigned by apply."""++ @abstractmethod+ def prompt_replacement(self, item: MMItem) -> PromptReplacement:+ """The token sequence that stands in for this item's placeholder in the prompt."""++ def positions(self, length: int, items: list[MMItem]) -> tuple[torch.Tensor, int] | None:+ """[3, length] t/h/w rope positions and the decode delta; None for 1-D rope families."""+ return None++ @abstractmethod+ def dummy_items(self, dtype: torch.dtype, device: torch.device) -> list[MMItem]:+ """The smallest encodable item of each modality the family serves, for warming the encoders."""++ def apply(self, input_ids: torch.Tensor, images: list[bytes]) -> MMResult:+ """Swap each image placeholder for its replacement sequence and precompute rope positions."""+ from PIL import Image++ ids = input_ids.tolist()+ target = self.placeholder+ slots = _find_all(ids, target)+ if len(slots) != len(images):+ raise ValueError(+ f"prompt renders {len(slots)} image placeholders but the request "+ f"carries {len(images)} images"+ )++ pils = [Image.open(io.BytesIO(raw)).convert("RGB") for raw in images]+ items = self.process(pils)+ out: list[int] = []+ cursor = 0+ for slot, item in zip(slots, items):+ repl = self.prompt_replacement(item)+ out.extend(ids[cursor:slot])+ base = len(out)+ full = list(repl.full)+ spans = repl.embed_spans()+ # embedding slots carry the content pad id so radix keys and the model's scatter mask see the image+ for lo, hi in spans:+ full[lo:hi] = [item.pad_value] * (hi - lo)+ out.extend(full)+ item.offsets = [[base + lo, base + hi] for lo, hi in spans]+ item.validate()+ cursor = slot + len(target)+ out.extend(ids[cursor:])+ new_ids = torch.tensor(out, dtype=input_ids.dtype)++ positions, delta = self.positions(len(out), items) or (None, 0)+ return MMResult(new_ids, items, positions, delta)+++def image_positions(+ length: int, blocks: list[tuple[int, int, int, int]]+) -> tuple[torch.Tensor, int]:+ """t/h/w positions: text advances all three rows, an image freezes t and spreads h/w over its grid, then the running position jumps by max(h, w).++ blocks are (offset, n_tokens, grid_h, grid_w) in prompt order; delta shifts decode positions (sequence index + delta)."""+ pos = torch.empty((3, length), dtype=torch.int32)+ st = 0+ cursor = 0+ for offset, n_tokens, h_m, w_m in blocks:+ run = offset - cursor+ if run:+ text = torch.arange(st, st + run, dtype=torch.int32)+ pos[:, cursor:offset] = text+ st += run+ cursor = offset+ idx = torch.arange(n_tokens, dtype=torch.int32)+ pos[0, cursor : cursor + n_tokens] = st+ pos[1, cursor : cursor + n_tokens] = st + idx // w_m+ pos[2, cursor : cursor + n_tokens] = st + idx % w_m+ st += max(h_m, w_m)+ cursor += n_tokens+ run = length - cursor+ if run:+ pos[:, cursor:] = torch.arange(st, st + run, dtype=torch.int32)+ st += run+ delta = int(pos.max().item()) + 1 - length+ return pos, delta+++def content_hash(feature: torch.Tensor, extra: bytes = b"") -> int:+ """62-bit hash of the wire bytes plus family extras (e.g. the grid, so transposed images differ)."""+ buf = feature.contiguous()+ if buf.dtype == torch.bfloat16:+ buf = buf.view(torch.uint16)+ hasher = hashlib.sha256(memoryview(buf.numpy()))+ hasher.update(extra)+ return int.from_bytes(hasher.digest()[:8], "little") % (1 << 62)+++def get_mm_processor(model_path: str, mm: MultimodalConfig | None = None) -> MMProcessor | None:+ """A processor for the checkpoint under the mm knobs, or None when no encoder is left to serve: the family registers none, the checkpoint ships none, or mm.disabled_encoders names them all; callers keep the instance, it holds the loaded image processor."""+ from freetoken.models.register import get_model_spec+ from freetoken.utils import cached_load_hf_config++ try:+ config = cached_load_hf_config(model_path)+ spec = get_model_spec((config.architectures or [None])[0])+ except Exception: # noqa: BLE001 -- missing/foreign config or unknown architecture: no multimodal input+ return None+ mm = mm or MultimodalConfig()+ served = [e for e in spec.encoders if getattr(config, e.config_key, None) is not None and e.kind not in mm.disabled_encoders]+ if spec.mm_processor is None or not served:+ return None+ check_mm_pad_shift(config.text_config.vocab_size)+ module, _, cls = spec.mm_processor.partition(":")+ return getattr(importlib.import_module(module), cls)(config, model_path, mm)+++__all__ = [+ "MMProcessor",+ "MMResult",+ "PromptReplacement",+ "content_hash",+ "get_mm_processor",+ "image_positions",+]diff --git a/python/freetoken/mm/processors/qwen_vl.py b/python/freetoken/mm/processors/qwen_vl.pynew file mode 100644index 00000000..669f7163--- /dev/null+++ b/python/freetoken/mm/processors/qwen_vl.py@@ -0,0 +1,95 @@+"""Qwen VL family image processor (Qwen2-VL through Qwen3.8): patch-grid ViT with a spatial merger, 3-axis rope positions."""++from __future__ import annotations++import struct+from typing import Any++import torch++from freetoken.message import MMItem+from freetoken.mm import mm_pad_value+from freetoken.mm.config import MultimodalConfig+from freetoken.mm.processor import MMProcessor, PromptReplacement, content_hash, image_positions+++class QwenVLMMProcessor(MMProcessor):+ def __init__(self, hf_config: Any, model_path: str, mm: MultimodalConfig) -> None:+ super().__init__(model_path, mm)+ vc = hf_config.vision_config+ self.image_token_id = hf_config.image_token_id+ self.placeholder = [self.image_token_id]+ self.merge = vc.spatial_merge_size+ self.pixels_per_token = (vc.patch_size * vc.spatial_merge_size) ** 2+ self.patch_dim = vc.in_channels * vc.temporal_patch_size * vc.patch_size**2+ self.is_mrope = "mrope_section" in hf_config.text_config.rope_parameters++ def get_mm_processor_kwargs(self, mm: MultimodalConfig) -> dict[str, Any]:+ kwargs: dict[str, Any] = {"return_tensors": "pt"}+ if mm.image_min_tokens is not None or mm.image_max_tokens is not None:+ # the budget travels as pixel areas in size.shortest_edge / longest_edge; bare min/max_pixels kwargs are ignored+ size = dict(self._image_processor().size)+ if mm.image_min_tokens is not None:+ size["shortest_edge"] = mm.image_min_tokens * self.pixels_per_token+ if mm.image_max_tokens is not None:+ size["longest_edge"] = mm.image_max_tokens * self.pixels_per_token+ kwargs["size"] = size+ return {**kwargs, **mm.processor_kwargs}++ def process(self, images: list[Any]) -> list[MMItem]:+ processor = self._image_processor()+ kwargs = self.get_mm_processor_kwargs(self.mm)+ items: list[MMItem] = []+ for pil in images:+ out = processor(images=pil, **kwargs)+ pixel_values = out["pixel_values"].to(torch.bfloat16)+ grid = out["image_grid_thw"][0].tolist()+ t, h, w = int(grid[0]), int(grid[1]), int(grid[2])+ if t != 1:+ raise ValueError("video/temporal input is not supported")+ # hash the bf16 wire buffer: half the bytes, and the exact tensor the cache and radix key agree on+ item_hash = content_hash(pixel_values, struct.pack("<3i", t, h, w))+ items.append(+ MMItem(+ modality="image",+ hash=item_hash,+ pad_value=mm_pad_value(item_hash),+ offsets=[],+ feature=pixel_values,+ model_specific_data={"grid_thw": [t, h, w]},+ )+ )+ return items++ def prompt_replacement(self, item: MMItem) -> PromptReplacement:+ t, h, w = item.grid_thw+ n_tokens = (t * h * w) // (self.merge * self.merge)+ return PromptReplacement([self.image_token_id] * n_tokens)++ def image_grid(self, item: MMItem) -> tuple[int, int]:+ _, h, w = item.grid_thw+ return h // self.merge, w // self.merge++ def positions(self, length: int, items: list[MMItem]) -> tuple[torch.Tensor, int] | None:+ if not self.is_mrope:+ return None+ blocks = []+ for item in items:+ ((start, end),) = item.offsets+ rows, cols = self.image_grid(item)+ blocks.append((start, end - start, rows, cols))+ return image_positions(length, blocks)++ def dummy_items(self, dtype: torch.dtype, device: torch.device) -> list[MMItem]:+ merge = self.merge+ return [MMItem(+ modality="image",+ hash=0,+ pad_value=0,+ offsets=[[0, 1]],+ feature=torch.zeros(merge * merge, self.patch_dim, dtype=dtype, device=device),+ model_specific_data={"grid_thw": [1, merge, merge]},+ )]+++__all__ = ["QwenVLMMProcessor"]diff --git a/python/freetoken/models/blocks.py b/python/freetoken/models/blocks.pyindex 0fa5fe4d..4f039fd3 100644--- a/python/freetoken/models/blocks.py+++ b/python/freetoken/models/blocks.py@@ -2,7 +2,9 @@ from __future__ import annotationsfrom abc import ABC, abstractmethodfrom contextlib import contextmanager-from typing import TYPE_CHECKING+from typing import TYPE_CHECKING, Protocol, runtime_checkable++import torchfrom freetoken.layers import (BaseOP,@@ -33,6 +35,29 @@ class BaseLLMModel(ABC, BaseOP):yield+@runtime_checkable+class SupportsMultimodal(Protocol):+ """What the engine asks of a model that serves multimodal items; the text forward takes its token embeddings from embed_input_ids."""++ def encode(self, item: MMItem) -> torch.Tensor:+ """``[item.num_tokens, D]`` soft tokens for one item of any modality the family serves, on the model device."""+ ...++ def place_encoder_weights(self, mode: str) -> None:+ """Every encoder tower's weights: ``gpu`` resident or ``host`` streamed from pinned banks."""+ ...+++def embed_input_ids(embed_tokens, input_ids: torch.Tensor, batch: Batch) -> torch.Tensor:+ """Token embeddings of the batch; on a chunk with multimodal rows, batch.mm_embeds' leading columns replace the rows batch.mm_rows."""+ if batch.mm_embeds is None:+ return embed_tokens.forward(input_ids)+ # multimodal rows carry content pad ids above the vocab: clamp for the lookup, the copy overwrites those rows+ x = embed_tokens.forward(input_ids.clamp(max=embed_tokens.num_embeddings - 1))+ x.index_copy_(0, batch.mm_rows, batch.mm_embeds[:, : x.shape[1]].to(x.dtype))+ return x++class GatedMLP(BaseOP):def __init__(self, config: ModelConfig, *, quant_config: QuantConfig | None = None, prefix: str = ""):self.gate_up_proj = LinearColParallelMerged(diff --git a/python/freetoken/models/config.py b/python/freetoken/models/config.pyindex 07a16df5..273662a9 100644--- a/python/freetoken/models/config.py+++ b/python/freetoken/models/config.py@@ -1,21 +1,11 @@from __future__ import annotations-import osfrom dataclasses import dataclassfrom typing import Any, ClassVar, Dict, List, Literal, Tuple, TypeAliasfrom freetoken.attention.base import AttnType-# State-dict key prefixes for the (optional) vision stack. Used both to drop the vision-# config (so the tower is never built) and to skip the matching tensors in the FTW reader.-VISION_KEY_PREFIXES = ("vision_tower.", "embed_vision.")-_VISION_TRUE = {"1", "true", "yes", "on"}---def vision_load_enabled() -> bool:- """Vision is opt-in (default OFF). The vision tower + multimodal embedder are ~1 GiB of- resident, never-quantized (bf16) GPU weights that text-only serving never touches, so we- skip building and loading them unless ``FREETOKEN_LOAD_VISION=1`` is set."""- return os.getenv("FREETOKEN_LOAD_VISION", "0").strip().lower() in _VISION_TRUE+# State-dict key prefixes of the vision stack; load_weight drops them when the engine serves text-only.+VISION_KEY_PREFIXES = ("vision_tower.", "embed_vision.", "visual.")def detect_expert_quant(hf_config: Any) -> str:@@ -98,6 +88,18 @@ class RotaryConfig:max_position: intbase: floatscaling: Dict[str, Any] | None+ # 3-axis rope sections; None keeps the 1-D rope path, set only when the model serves vision+ mrope_section: list | None = None+ mrope_layout: str = "contiguous" # see freetoken.layers.rotary.build_section_table+++def mrope_layout_from_rope_params(rope_params: Any) -> str:+ """rope_parameters flags -> layout name: mrope_interleaved_glm, else mrope_interleaved, else contiguous."""+ if rope_params.get("mrope_interleaved_glm"):+ return "interleaved_glm"+ if rope_params.get("mrope_interleaved"):+ return "interleaved"+ return "contiguous"@dataclass(frozen=True)@@ -357,6 +359,14 @@ class ModelConfig:def is_multimodal(self) -> bool:return self.vision_config is not None+ @property+ def model_is_mrope(self) -> bool:+ """True when any full-attention layer uses 3-axis (t/h/w) rope positions."""+ return any(+ getattr(getattr(g, "rotary_config", None), "mrope_section", None) is not None+ for g in self.attention_groups+ )+@propertydef has_hybrid_attention(self) -> bool:return len(self.attention_groups) > 1diff --git a/python/freetoken/models/gemma4/config.py b/python/freetoken/models/gemma4/config.pyindex 057e0a55..94718603 100644--- a/python/freetoken/models/gemma4/config.py+++ b/python/freetoken/models/gemma4/config.py@@ -9,7 +9,6 @@ from freetoken.models.config import (RotaryConfig,SWAAttentionGroupConfig,detect_expert_quant,- vision_load_enabled,)@@ -44,15 +43,6 @@ def _parse_vision_config(top_cfg: Any, text_hidden_size: int) -> VisionConfig |vc = getattr(top_cfg, "vision_config", None)if vc is None:return None- # Vision is opt-in (default OFF). The vision tower + multimodal embedder are ~1 GiB of- # resident, never-quantized (bf16) GPU weights that text-only serving never touches:- # forward() ignores them and _merge_multimodal is a no-op without image embeds. Default- # to text-only so that VRAM goes to KV / the expert cache; set FREETOKEN_LOAD_VISION=1- # to load it. Returning None here is the single switch -- both the model build (model.py)- # and weight loading (weight.py `include_vision = config.is_multimodal`) flow through- # parse_config, so is_multimodal flips off consistently across both.- if not vision_load_enabled():- return Nonerope_params = getattr(vc, "rope_parameters", None) or {}act = getattr(vc, "hidden_activation", "gelu_pytorch_tanh")act = "gelu_tanh" if "tanh" in act else actdiff --git a/python/freetoken/models/gemma4/model.py b/python/freetoken/models/gemma4/model.pyindex 83e49c4d..d151540b 100644--- a/python/freetoken/models/gemma4/model.py+++ b/python/freetoken/models/gemma4/model.py@@ -123,16 +123,8 @@ class Gemma4ForCausalLM(BaseLLMModel):convert_gemma4_to_gguf(self, config)@torch.inference_mode()- def encode_images(- self, pixel_values: torch.Tensor, image_position_ids: torch.Tensor- ) -> torch.Tensor:- """Run the vision tower + projector. Returns ``[num_valid_soft_tokens, text_hidden]``.-- ``pixel_values``: ``[num_images, num_patches, 3*patch**2]``;- ``image_position_ids``: ``[num_images, num_patches, 2]`` with ``(-1, -1)`` padding.- """- features = self.vision_tower.forward(pixel_values, image_position_ids)- return self.embed_vision.forward(features)+ def encode(self, item) -> torch.Tensor:+ raise NotImplementedError("gemma4 image input needs an MMProcessor (freetoken.mm.processors)")def forward(self) -> torch.Tensor:output = self.model.forward(get_global_ctx().batch.input_ids)diff --git a/python/freetoken/models/gemma4/weight.py b/python/freetoken/models/gemma4/weight.pyindex 740f4660..4413b67f 100644--- a/python/freetoken/models/gemma4/weight.py+++ b/python/freetoken/models/gemma4/weight.py@@ -133,20 +133,19 @@ def iter_weights(include_moe_experts: bool,include_non_moe: bool,) -> Iterator[tuple[str, torch.Tensor]]:- def rename_key(raw_name: str, *, include_vision: bool) -> str | None:+ def rename_key(raw_name: str) -> str | None:prefix = "model.language_model."if raw_name.startswith(prefix):return _rename_language_key(raw_name)if raw_name.startswith("language_model."):return _rename_language_key(raw_name)- if include_vision:- if raw_name.startswith("model.vision_tower."):- return ("vision_tower." + raw_name[len("model.vision_tower.") :]).replace(- ".linear.",- ".",- )- if raw_name.startswith("model.embed_vision."):- return "embed_vision." + raw_name[len("model.embed_vision.") :]+ if raw_name.startswith("model.vision_tower."):+ return ("vision_tower." + raw_name[len("model.vision_tower.") :]).replace(+ ".linear.",+ ".",+ )+ if raw_name.startswith("model.embed_vision."):+ return "embed_vision." + raw_name[len("model.embed_vision.") :]return Nonedef merge_info(key: str) -> tuple[str, MergeRule] | None:@@ -160,7 +159,6 @@ def iter_weights(if tp_info.size > 1:raise NotImplementedError("Gemma 4 weight loading currently supports TP=1 only")- include_vision = config.is_multimodalk_eq_v_layers = {layer_idfor layer_id in range(config.num_layers)@@ -178,7 +176,7 @@ def iter_weights():with safetensors.safe_open(file, framework="pt", device=str(device)) as f:for raw_name in f.keys():- name = rename_key(raw_name, include_vision=include_vision)+ name = rename_key(raw_name)if name is None:continuediff --git a/python/freetoken/models/qwen3/attention.py b/python/freetoken/models/qwen3/attention.pyindex 039dc50d..5ec37013 100644--- a/python/freetoken/models/qwen3/attention.py+++ b/python/freetoken/models/qwen3/attention.py@@ -57,6 +57,12 @@ class Qwen3Attention(BaseOP):if config.rotary_config.scalingelse None),+ mrope_section=(+ tuple(config.rotary_config.mrope_section)+ if config.rotary_config.mrope_section is not None+ else None+ ),+ mrope_layout=config.rotary_config.mrope_layout,)self.o_proj = LinearOProj(head_dim * config.num_qo_heads,@@ -76,7 +82,7 @@ class Qwen3Attention(BaseOP):self.q_norm.forward_inplace(q.view(-1, self.num_qo_heads, self.head_dim))if self.k_norm is not None:self.k_norm.forward_inplace(k.view(-1, self.num_kv_heads, self.head_dim))- q, k = self.rotary.forward(ctx.batch.positions, q, k)+ q, k = self.rotary.forward(ctx.batch.get_attn_positions(), q, k)q = q.view(-1, self.num_qo_heads, self.head_dim)o = ctx.attn_backend.forward(q, k, v, self.layer_id, ctx.batch)return self.o_proj.forward(o.view(-1, self.qo_attn_dim))diff --git a/python/freetoken/models/qwen3/model.py b/python/freetoken/models/qwen3/model.pyindex dc35930d..6ac7d2fa 100644--- a/python/freetoken/models/qwen3/model.py+++ b/python/freetoken/models/qwen3/model.py@@ -67,8 +67,10 @@ class Qwen3Model(BaseOP):class Qwen3ForCausalLM(BaseLLMModel):+ model_cls = Qwen3Model+def __init__(self, config: ModelConfig):- self.model = Qwen3Model(config)+ self.model = self.model_cls(config)self.lm_head = ParallelLMHead(num_embeddings=config.vocab_size,embedding_dim=config.hidden_size,diff --git a/python/freetoken/models/qwen3_5_moe/__init__.py b/python/freetoken/models/qwen3_5_moe/__init__.pyindex 8620ce9c..9acc5aac 100644--- a/python/freetoken/models/qwen3_5_moe/__init__.py+++ b/python/freetoken/models/qwen3_5_moe/__init__.py@@ -1,9 +1,17 @@from .config import parse_config-from .model import Qwen3_5MoEForCausalLM+from .model import (+ Qwen3_5ForCausalLM,+ Qwen3_5ForConditionalGeneration,+ Qwen3_5MoeForCausalLM,+ Qwen3_5MoeForConditionalGeneration,+)from .weight import iter_expert_pieces, iter_weights, iter_weights_parallel, nvfp4_expert_spec__all__ = [- "Qwen3_5MoEForCausalLM",+ "Qwen3_5ForCausalLM",+ "Qwen3_5ForConditionalGeneration",+ "Qwen3_5MoeForCausalLM",+ "Qwen3_5MoeForConditionalGeneration","parse_config","iter_weights","iter_weights_parallel",diff --git a/python/freetoken/models/qwen3_5_moe/attention.py b/python/freetoken/models/qwen3_5_moe/attention.pyindex 12a91bb3..7a9e6a33 100644--- a/python/freetoken/models/qwen3_5_moe/attention.py+++ b/python/freetoken/models/qwen3_5_moe/attention.py@@ -55,6 +55,12 @@ class Qwen3_5Attention(BaseOP):if config.rotary_config.scalingelse None),+ mrope_section=(+ tuple(config.rotary_config.mrope_section)+ if config.rotary_config.mrope_section is not None+ else None+ ),+ mrope_layout=config.rotary_config.mrope_layout,)self.o_proj = LinearReplicated(self.qo_attn_dim, config.hidden_size, has_bias=False,@@ -64,7 +70,7 @@ class Qwen3_5Attention(BaseOP):def _project(self, x: torch.Tensor):"""Returns (q, k, v, gate): q [N, num_q, head_dim] post qk-norm+rope,k [N, num_kv*head_dim] post norm+rope, v [N, num_kv*head_dim], gate [N, num_q*head_dim]."""- positions = get_global_ctx().batch.positions+ positions = get_global_ctx().batch.get_attn_positions()qkv = self.qkv_proj.forward(x)qg, k, v = torch.split(qkv, self._qkv_split, dim=-1)qg = qg.view(-1, self.num_q, self.head_dim * 2)diff --git a/python/freetoken/models/qwen3_5_moe/config.py b/python/freetoken/models/qwen3_5_moe/config.pyindex cc5a62b9..70cfbcf2 100644--- a/python/freetoken/models/qwen3_5_moe/config.py+++ b/python/freetoken/models/qwen3_5_moe/config.py@@ -8,7 +8,9 @@ from freetoken.models.config import (LinearGatedDeltaGroupConfig,ModelConfig,RotaryConfig,+ mrope_layout_from_rope_params,)+from freetoken.models.qwen3_vl.config import parse_vision_configdef _expert_quant(hf_config: Any, text: Any) -> tuple[str, tuple[int, int] | None]:@@ -74,12 +76,20 @@ def parse_config(hf_config: Any) -> ModelConfig:full_ids = tuple(i for i, t in enumerate(layer_types) if t == "full_attention")linear_ids = tuple(i for i, t in enumerate(layer_types) if t == "linear_attention")+ # 3-axis rope only with vision; text-only serving keeps the 1-D partial rope and the decode-graph layout+ vision_config = parse_vision_config(hf_config)full_rotary = RotaryConfig(head_dim=head_dim,rotary_dim=rotary_dim,max_position=text.max_position_embeddings,base=rope_theta,scaling=rope_scaling,+ mrope_section=(+ list(rope_params["mrope_section"])+ if vision_config is not None and "mrope_section" in rope_params+ else None+ ),+ mrope_layout=mrope_layout_from_rope_params(rope_params),)full_group = FullAttentionGroupConfig(name="full",@@ -127,7 +137,7 @@ def parse_config(hf_config: Any) -> ModelConfig:use_qk_norm=True,model_type=getattr(hf_config, "model_type", "qwen3_5_moe"),architectures=getattr(hf_config, "architectures", ["Qwen3_5MoeForConditionalGeneration"]),- vision_config=None, # text-only milestone+ vision_config=vision_config,image_token_id=getattr(hf_config, "image_token_id", None),attention_groups=groups,expert_quant=expert_quant,diff --git a/python/freetoken/models/qwen3_5_moe/model.py b/python/freetoken/models/qwen3_5_moe/model.pyindex d34793ad..8cdd41ae 100644--- a/python/freetoken/models/qwen3_5_moe/model.py+++ b/python/freetoken/models/qwen3_5_moe/model.py@@ -12,6 +12,8 @@ from freetoken.layers import (VocabParallelEmbedding,)from freetoken.models.blocks import BaseLLMModel+from freetoken.models.blocks import embed_input_ids+from freetoken.models.qwen3_vl.vision import Qwen3VLVisionModel, QwenVLVisionMixinfrom freetoken.utils import nvtx_annotatefrom .attention import Qwen3_5Attention@@ -87,7 +89,7 @@ class Qwen3_5Model(BaseOP):self.norm = GemmaRMSNorm(config.hidden_size, eps=config.rms_norm_eps)def forward(self, input_ids: torch.Tensor) -> torch.Tensor:- x = self.embed_tokens.forward(input_ids)+ x = embed_input_ids(self.embed_tokens, input_ids, get_global_ctx().batch)residual: torch.Tensor | None = Nonefor layer in self.layers.op_list:x, residual = layer.forward(x, residual)@@ -95,7 +97,7 @@ class Qwen3_5Model(BaseOP):return x-class Qwen3_5MoEForCausalLM(BaseLLMModel):+class Qwen3_5ForCausalLM(BaseLLMModel):def __init__(self, config: ModelConfig):self.model = Qwen3_5Model(config)self.lm_head = ParallelLMHead(@@ -113,4 +115,25 @@ class Qwen3_5MoEForCausalLM(BaseLLMModel):return self.lm_head.forward(output)-__all__ = ["Qwen3_5MoEForCausalLM"]+class Qwen3_5MoeForCausalLM(Qwen3_5ForCausalLM):+ """The MoE releases share the dense code path: the decoder picks the routed or dense MLP from config.num_experts."""+++class Qwen3_5ForConditionalGeneration(QwenVLVisionMixin, Qwen3_5ForCausalLM):+ def __init__(self, config: ModelConfig):+ super().__init__(config)+ if config.is_multimodal:+ assert not config.vision_config.deepstack_visual_indexes, "Qwen3.5 consumes no DeepStack features"+ self.visual = Qwen3VLVisionModel(config.vision_config, quant_config=config.quant, prefix="visual")+++class Qwen3_5MoeForConditionalGeneration(Qwen3_5ForConditionalGeneration):+ """The MoE releases with the vision tower; see Qwen3_5MoeForCausalLM."""+++__all__ = [+ "Qwen3_5ForCausalLM",+ "Qwen3_5ForConditionalGeneration",+ "Qwen3_5MoeForCausalLM",+ "Qwen3_5MoeForConditionalGeneration",+]diff --git a/python/freetoken/models/qwen3_5_moe/weight.py b/python/freetoken/models/qwen3_5_moe/weight.pyindex 5fc12e74..c74117b3 100644--- a/python/freetoken/models/qwen3_5_moe/weight.py+++ b/python/freetoken/models/qwen3_5_moe/weight.py@@ -13,6 +13,7 @@ import torchfrom freetoken.distributed import get_tp_infofrom freetoken.kernel.triton.nvfp4_dequant import dequant_nvfp4from freetoken.layers.quantization import QuantConfig, QuantKind, QuantScheme, get_quant_config+from freetoken.models.config import VISION_KEY_PREFIXESfrom freetoken.models.loader import ShardReader, iter_weight_filesfrom freetoken.models.nvfp4_banks import Nvfp4ExpertSourceSpecfrom freetoken.models.register import ModelSpec, get_model_spec@@ -20,6 +21,7 @@ from freetoken.utils import cached_load_hf_configfrom tqdm import tqdmfrom .config import parse_config+from freetoken.models.qwen3_vl.weight import rename_vl_prefix# bf16 checkpoints store the routed experts pre-stacked per layer_STACKED_EXPERT_RE = re.compile(r"^model\.layers\.\d+\.mlp\.experts\.(gate_up_proj|down_proj)$")@@ -53,16 +55,12 @@ _QUANT_DTYPES = (torch.float8_e4m3fn, torch.float8_e5m2, torch.uint8, torch.int8def _rename(raw_name: str) -> str | None:"""Checkpoint key -> FreeToken state-dict key, or None to skip."""- if raw_name.startswith(("mtp.", "model.visual.", "visual.")):+ if raw_name.startswith("mtp."):return None# static KV-cache scales of the quantizers; the KV cache runs in the engine's dtypeif raw_name.endswith((".k_scale", ".v_scale", ".q_scale", ".prob_scale")):return None- if raw_name.startswith("model.language_model."):- return "model." + raw_name[len("model.language_model."):]- if raw_name.startswith("language_model."):- return "model." + raw_name[len("language_model."):]- return raw_name+ return rename_vl_prefix(raw_name)def _is_gemma_norm(name: str) -> bool:@@ -225,6 +223,7 @@ def iter_weights(*,include_moe_experts: bool,include_non_moe: bool,+ include_vision: bool = True,) -> Iterator[tuple[str, torch.Tensor]]:"""Yield the dense weights fused to the model's buffers, and the routed experts only where a resident path takes them from here: bf16 stacked experts as stored, block-fp8 experts restacked per layer.@@ -237,18 +236,20 @@ def iter_weights(stacked = include_moe_experts and config.is_moe and config.expert_quant == "none"if include_non_moe or stacked:reader = _DenseReader(get_quant_config(), get_model_spec(hf_config.architectures[0])) if include_non_moe else None- yield from _iter_shards(model_path, device, reader, stacked=stacked)+ yield from _iter_shards(model_path, device, reader, stacked=stacked, include_vision=include_vision)if include_moe_experts and config.is_moe and config.expert_quant == "fp8_block":yield from _resident_fp8_experts(model_path, config)-def _iter_shards(model_path: str, device: torch.device, reader: _DenseReader | None, *, stacked: bool):+def _iter_shards(model_path: str, device: torch.device, reader: _DenseReader | None, *, stacked: bool, include_vision: bool):for file in tqdm(iter_weight_files(model_path), desc="Loading weights", disable=not get_tp_info().is_primary()):with safetensors.safe_open(file, framework="pt", device=str(device)) as f:for raw_name in f.keys():name = _rename(raw_name)if name is None or _EXPERT_RE.search(name):continue+ if not include_vision and name.startswith(VISION_KEY_PREFIXES):+ continueif _STACKED_EXPERT_RE.match(name):if stacked:yield name, f.get_tensor(raw_name)diff --git a/python/freetoken/models/qwen3_moe/attention.py b/python/freetoken/models/qwen3_moe/attention.pyindex 8a737b27..2c2e84b4 100644--- a/python/freetoken/models/qwen3_moe/attention.py+++ b/python/freetoken/models/qwen3_moe/attention.py@@ -57,6 +57,12 @@ class Qwen3MoeAttention(BaseOP):if config.rotary_config.scalingelse None),+ mrope_section=(+ tuple(config.rotary_config.mrope_section)+ if config.rotary_config.mrope_section is not None+ else None+ ),+ mrope_layout=config.rotary_config.mrope_layout,)self.o_proj = LinearOProj(head_dim * config.num_qo_heads,@@ -76,7 +82,7 @@ class Qwen3MoeAttention(BaseOP):self.q_norm.forward_inplace(q.view(-1, self.num_qo_heads, self.head_dim))if self.k_norm is not None:self.k_norm.forward_inplace(k.view(-1, self.num_kv_heads, self.head_dim))- q, k = self.rotary.forward(ctx.batch.positions, q, k)+ q, k = self.rotary.forward(ctx.batch.get_attn_positions(), q, k)q = q.view(-1, self.num_qo_heads, self.head_dim)o = ctx.attn_backend.forward(q, k, v, self.layer_id, ctx.batch)return self.o_proj.forward(o.view(-1, self.qo_attn_dim))diff --git a/python/freetoken/models/qwen3_moe/model.py b/python/freetoken/models/qwen3_moe/model.pyindex 4918092e..5d89656b 100644--- a/python/freetoken/models/qwen3_moe/model.py+++ b/python/freetoken/models/qwen3_moe/model.py@@ -68,8 +68,10 @@ class Qwen3Model(BaseOP):class Qwen3MoeForCausalLM(BaseLLMModel):+ model_cls = Qwen3Model+def __init__(self, config: ModelConfig):- self.model = Qwen3Model(config)+ self.model = self.model_cls(config)self.lm_head = ParallelLMHead(num_embeddings=config.vocab_size,embedding_dim=config.hidden_size,diff --git a/python/freetoken/models/qwen3_vl/__init__.py b/python/freetoken/models/qwen3_vl/__init__.pynew file mode 100644index 00000000..d79c838f--- /dev/null+++ b/python/freetoken/models/qwen3_vl/__init__.py@@ -0,0 +1,21 @@+from .config import VisionConfig, parse_config, parse_vision_config+from .model import (+ Qwen3VLForConditionalGeneration,+ Qwen3VLMoeForConditionalGeneration,+ deepstack_add,+)+from .vision import Qwen3VLVisionModel+from .weight import iter_weights, iter_weights_parallel, rename_vl_prefix++__all__ = [+ "Qwen3VLForConditionalGeneration",+ "Qwen3VLMoeForConditionalGeneration",+ "Qwen3VLVisionModel",+ "VisionConfig",+ "deepstack_add",+ "iter_weights",+ "iter_weights_parallel",+ "parse_config",+ "parse_vision_config",+ "rename_vl_prefix",+]diff --git a/python/freetoken/models/qwen3_vl/config.py b/python/freetoken/models/qwen3_vl/config.pynew file mode 100644index 00000000..0a5c26b3--- /dev/null+++ b/python/freetoken/models/qwen3_vl/config.py@@ -0,0 +1,101 @@+from __future__ import annotations++import dataclasses+from dataclasses import dataclass+from typing import Any++from freetoken.models.config import (+ FullAttentionGroupConfig,+ ModelConfig,+ RotaryConfig,+ mrope_layout_from_rope_params,+)+from freetoken.models.qwen3.config import parse_config as parse_qwen3+from freetoken.models.qwen3_moe.config import parse_config as parse_qwen3_moe+++@dataclass+class VisionConfig:+ hidden_size: int+ depth: int+ num_heads: int+ intermediate_size: int+ patch_size: int+ temporal_patch_size: int+ spatial_merge_size: int+ num_position_embeddings: int+ out_hidden_size: int+ in_channels: int+ # ViT block indices whose merged features are added to the first decoder layers; () disables DeepStack+ deepstack_visual_indexes: tuple[int, ...] = ()+++def parse_vision_config(hf_config: Any) -> VisionConfig | None:+ """None when the config carries no vision section, which is how a text-only engine asks for no tower and 1-D rope."""+ vc = getattr(hf_config, "vision_config", None)+ if vc is None:+ return None+ return VisionConfig(+ hidden_size=vc.hidden_size,+ depth=vc.depth,+ num_heads=vc.num_heads,+ intermediate_size=vc.intermediate_size,+ patch_size=vc.patch_size,+ temporal_patch_size=vc.temporal_patch_size,+ spatial_merge_size=vc.spatial_merge_size,+ num_position_embeddings=vc.num_position_embeddings,+ out_hidden_size=vc.out_hidden_size,+ in_channels=vc.in_channels,+ deepstack_visual_indexes=tuple(getattr(vc, "deepstack_visual_indexes", None) or ()),+ )+++def parse_config(hf_config: Any) -> ModelConfig:+ """Qwen3-VL: a Qwen3 (dense or MoE) text tower under text_config plus the shared Qwen VL vision tower."""+ text = hf_config.text_config+ base = parse_qwen3_moe(text) if getattr(text, "num_experts", 0) else parse_qwen3(text)++ rope_params = text.rope_parameters+ rope_type = rope_params.get("rope_type", "default")+ # the default rope type needs no scaling dict, and the mrope_section list must stay out of get_rope's cache key+ rope_scaling = (+ None+ if rope_type in (None, "default")+ else {k: v for k, v in rope_params.items() if not isinstance(v, (list, dict))}+ )+ vision_config = parse_vision_config(hf_config)+ rotary = RotaryConfig(+ head_dim=base.head_dim,+ rotary_dim=base.head_dim,+ max_position=text.max_position_embeddings,+ base=rope_params["rope_theta"],+ scaling=rope_scaling,+ mrope_section=(+ list(rope_params["mrope_section"])+ if vision_config is not None and "mrope_section" in rope_params+ else None+ ),+ mrope_layout=mrope_layout_from_rope_params(rope_params),+ )+ # an explicit group carries the mrope rotary config, which model_is_mrope reads from the groups+ full_group = FullAttentionGroupConfig(+ name="full",+ layer_ids=tuple(range(base.num_layers)),+ num_kv_heads=base.num_kv_heads,+ head_dim=base.head_dim,+ rotary_config=rotary,+ )+ return dataclasses.replace(+ base,+ rotary_config=rotary,+ attention_groups=(full_group,),+ moe_enabled=bool(getattr(text, "num_experts", 0)),+ vision_config=vision_config,+ image_token_id=hf_config.image_token_id,+ tie_word_embeddings=bool(getattr(hf_config, "tie_word_embeddings", False)),+ model_type=hf_config.model_type,+ architectures=list(hf_config.architectures),+ )+++__all__ = ["VisionConfig", "parse_config", "parse_vision_config"]diff --git a/python/freetoken/models/qwen3_vl/model.py b/python/freetoken/models/qwen3_vl/model.pynew file mode 100644index 00000000..cd13945f--- /dev/null+++ b/python/freetoken/models/qwen3_vl/model.py@@ -0,0 +1,80 @@+from __future__ import annotations++from typing import TYPE_CHECKING++import torch+from freetoken.core import get_global_ctx++from freetoken.models.qwen3.model import Qwen3ForCausalLM, Qwen3Model+from freetoken.models.blocks import embed_input_ids+from freetoken.models.qwen3_moe.model import Qwen3MoeForCausalLM, Qwen3Model as Qwen3MoeModel++from .vision import Qwen3VLVisionModel, QwenVLVisionMixin++if TYPE_CHECKING:+ from freetoken.models.config import ModelConfig+++def deepstack_add(+ x: torch.Tensor, rows: torch.Tensor, mm_embeds: torch.Tensor, level: int, hidden_size: int+) -> None:+ """Add DeepStack level `level` (the column block after the main embedding) onto the image rows of x, in place."""+ lo = hidden_size * (level + 1)+ x.index_add_(0, rows, mm_embeds[:, lo : lo + hidden_size].to(x.dtype))+++class Qwen3VLTextModel(Qwen3Model):+ def __init__(self, config: ModelConfig):+ super().__init__(config)+ self._deepstack_levels = len(config.vision_config.deepstack_visual_indexes) if config.vision_config else 0++ def forward(self, input_ids: torch.Tensor) -> torch.Tensor:+ batch = get_global_ctx().batch+ x = embed_input_ids(self.embed_tokens, input_ids, batch)+ residual: torch.Tensor | None = None+ for level, layer in enumerate(self.layers.op_list):+ x, residual = layer.forward(x, residual)+ if batch.mm_embeds is not None and level < self._deepstack_levels:+ deepstack_add(x, batch.mm_rows, batch.mm_embeds, level, x.shape[1])+ return self.norm.forward(x, residual)[0]+++class Qwen3VLMoeTextModel(Qwen3MoeModel):+ def __init__(self, config: ModelConfig):+ super().__init__(config)+ self._deepstack_levels = len(config.vision_config.deepstack_visual_indexes) if config.vision_config else 0++ def forward(self, input_ids: torch.Tensor) -> torch.Tensor:+ batch = get_global_ctx().batch+ x = embed_input_ids(self.embed_tokens, input_ids, batch)+ residual: torch.Tensor | None = None+ for level, layer in enumerate(self.layers.op_list):+ x, residual = layer.forward(x, residual)+ if batch.mm_embeds is not None and level < self._deepstack_levels:+ deepstack_add(x, batch.mm_rows, batch.mm_embeds, level, x.shape[1])+ return self.norm.forward(x, residual)[0]+++class Qwen3VLForConditionalGeneration(QwenVLVisionMixin, Qwen3ForCausalLM):+ model_cls = Qwen3VLTextModel++ def __init__(self, config: ModelConfig):+ super().__init__(config)+ if config.is_multimodal:+ self.visual = Qwen3VLVisionModel(config.vision_config, quant_config=config.quant, prefix="visual")+++class Qwen3VLMoeForConditionalGeneration(QwenVLVisionMixin, Qwen3MoeForCausalLM):+ model_cls = Qwen3VLMoeTextModel++ def __init__(self, config: ModelConfig):+ super().__init__(config)+ if config.is_multimodal:+ self.visual = Qwen3VLVisionModel(config.vision_config, quant_config=config.quant, prefix="visual")+++__all__ = [+ "Qwen3VLForConditionalGeneration",+ "Qwen3VLMoeForConditionalGeneration",+ "deepstack_add",+]diff --git a/python/freetoken/models/qwen3_vl/vision.py b/python/freetoken/models/qwen3_vl/vision.pynew file mode 100644index 00000000..7c4e0df2--- /dev/null+++ b/python/freetoken/models/qwen3_vl/vision.py@@ -0,0 +1,375 @@+"""Qwen VL vision tower: full-attention ViT blocks and a 2x2 patch merger; DeepStack taps ride along as extra output columns."""++from __future__ import annotations++import os+from typing import TYPE_CHECKING, List++import torch+import torch.nn.functional as F+from freetoken.distributed import get_tp_info+from freetoken.layers import (+ BaseOP,+ LinearColParallelMerged,+ LinearOProj,+ LinearQKVMerged,+ LinearRowParallel,+ OPList,+)+from freetoken.utils import div_even++from freetoken.models.weight_stream import BlockWeightStreamer++if TYPE_CHECKING:+ from freetoken.layers.quantization import QuantConfig+ from freetoken.message import MMItem+ from .config import VisionConfig+++class VisionLayerNorm(BaseOP):+ def __init__(self, size: int, eps: float = 1e-6):+ self.weight = torch.empty(size)+ self.bias = torch.empty(size)+ self._size = size+ self._eps = eps++ def forward(self, x: torch.Tensor) -> torch.Tensor:+ return F.layer_norm(x, (self._size,), self.weight, self.bias, self._eps)+++def _rotate_half(x: torch.Tensor) -> torch.Tensor:+ half = x.shape[-1] // 2+ return torch.cat((-x[..., half:], x[..., :half]), dim=-1)+++_warned_rope_fallback = False+++def _apply_vision_rope(+ q: torch.Tensor, k: torch.Tensor, cache: torch.Tensor, positions: torch.Tensor, head_dim: int+) -> None:+ """In-place 2D rope through the NeoX text kernel: the [c,c]/[s,s] layout is the half-rotation layout and positions index per-token cache rows."""+ global _warned_rope_fallback+ S = q.shape[0]+ try:+ from freetoken.kernel.triton.rope import apply_rope_with_cos_sin_cache_inplace++ apply_rope_with_cos_sin_cache_inplace(+ positions, q.view(S, -1), k.view(S, -1), head_dim, cache, is_neox=True+ )+ return+ except ImportError:+ # only a missing triton falls back; a CUDA error must not re-run eager on a half-rotated q/k+ if not _warned_rope_fallback:+ _warned_rope_fallback = True+ from freetoken.utils import init_logger++ init_logger(__name__).warning("vision rope triton kernel unavailable; using eager")+ half = cache.shape[1] // 2+ cos = torch.cat((cache[:, :half], cache[:, :half]), dim=-1).unsqueeze(-2)+ sin = torch.cat((cache[:, half:], cache[:, half:]), dim=-1).unsqueeze(-2)+ q.copy_((q.float() * cos + _rotate_half(q.float()) * sin).to(q.dtype))+ k.copy_((k.float() * cos + _rotate_half(k.float()) * sin).to(k.dtype))+++class VisionAttention(BaseOP):+ def __init__(self, vc: VisionConfig, *, quant_config: QuantConfig | None = None, prefix: str = ""):+ self.num_heads = div_even(vc.num_heads, get_tp_info().size)+ self.head_dim = vc.hidden_size // vc.num_heads+ self.qkv = LinearQKVMerged(+ vc.hidden_size, self.head_dim, vc.num_heads, vc.num_heads, has_bias=True,+ quant_config=quant_config, prefix=f"{prefix}.qkv",+ )+ self.proj = LinearOProj(+ vc.hidden_size, vc.hidden_size, has_bias=True, quant_config=quant_config, prefix=f"{prefix}.proj"+ )++ def attend(+ self, qkv: torch.Tensor, cache: torch.Tensor, positions: torch.Tensor, lengths: List[int]+ ) -> torch.Tensor:+ """Rope + bidirectional attention within each image on the fused qkv projection; returns [S, heads * head_dim]."""+ S = qkv.shape[0]+ # q/k/v stay views into the fused projection: the rope kernel takes strides, SDPA needs only head_dim contiguous+ q, k, v = qkv.view(S, 3, self.num_heads, self.head_dim).unbind(1)+ _apply_vision_rope(q, k, cache, positions, self.head_dim)+ q = q.transpose(0, 1).unsqueeze(0)+ k = k.transpose(0, 1).unsqueeze(0)+ v = v.transpose(0, 1).unsqueeze(0)+ # bidirectional attention within each image; images never attend across+ outs = [+ F.scaled_dot_product_attention(qs, ks, vs)+ for qs, ks, vs in zip(+ torch.split(q, lengths, dim=2),+ torch.split(k, lengths, dim=2),+ torch.split(v, lengths, dim=2),+ )+ ]+ o = outs[0] if len(outs) == 1 else torch.cat(outs, dim=2)+ return o[0].transpose(0, 1).reshape(S, -1)+++class VisionMLP(BaseOP):+ def __init__(self, vc: VisionConfig, *, quant_config: QuantConfig | None = None, prefix: str = ""):+ self.linear_fc1 = LinearColParallelMerged(+ vc.hidden_size, [vc.intermediate_size], has_bias=True, quant_config=quant_config, prefix=f"{prefix}.linear_fc1"+ )+ self.linear_fc2 = LinearRowParallel(+ vc.intermediate_size, vc.hidden_size, has_bias=True, quant_config=quant_config, prefix=f"{prefix}.linear_fc2"+ )++ def forward(self, x: torch.Tensor) -> torch.Tensor:+ if x.shape[0] <= _MLP_ROWS:+ return self.linear_fc2.forward(F.gelu(self.linear_fc1.forward(x), approximate="tanh"))+ # per-token op: row chunks bound the two [rows, intermediate] temporaries while the GEMMs stay large+ out = torch.empty_like(x)+ for lo in range(0, x.shape[0], _MLP_ROWS):+ rows = x[lo : lo + _MLP_ROWS]+ out[lo : lo + _MLP_ROWS] = self.linear_fc2.forward(F.gelu(self.linear_fc1.forward(rows), approximate="tanh"))+ return out+++# 4096 rows x 4304 x bf16 = 34 MiB per temporary while the GEMM still fills the GPU+# FREETOKEN_VIT_MLP_ROWS=0 disables the chunking; cuBLAS may round a row chunk one bf16 ulp differently from the whole image+_MLP_ROWS = int(os.environ.get("FREETOKEN_VIT_MLP_ROWS", 4096)) or 1 << 62+# 16384 rows x 1152 x fp32 = 72 MiB per interpolation temporary+_POS_ROWS = 16384+++class VisionBlock(BaseOP):+ def __init__(self, vc: VisionConfig, *, quant_config: QuantConfig | None = None, prefix: str = ""):+ self.norm1 = VisionLayerNorm(vc.hidden_size)+ self.norm2 = VisionLayerNorm(vc.hidden_size)+ self.attn = VisionAttention(vc, quant_config=quant_config, prefix=f"{prefix}.attn")+ self.mlp = VisionMLP(vc, quant_config=quant_config, prefix=f"{prefix}.mlp")++ def forward(+ self, x: torch.Tensor, cache: torch.Tensor, positions: torch.Tensor, lengths: List[int]+ ) -> torch.Tensor:+ x = x + self._attention(x, cache, positions, lengths)+ return x + self.mlp.forward(self.norm2.forward(x))++ def _attention(+ self, x: torch.Tensor, cache: torch.Tensor, positions: torch.Tensor, lengths: List[int]+ ) -> torch.Tensor:+ # each temporary dies inside the next call: norm out in the qkv GEMM, qkv in attention, its output in proj+ o = self.attn.attend(self.attn.qkv.forward(self.norm1.forward(x)), cache, positions, lengths)+ return self.attn.proj.forward(o)+++class _Conv3dParams(BaseOP):+ def __init__(self, out_ch: int, shape: tuple):+ self.weight = torch.empty(out_ch, *shape)+ self.bias = torch.empty(out_ch)++ forward = None+++class _EmbeddingParams(BaseOP):+ def __init__(self, rows: int, dim: int):+ self.weight = torch.empty(rows, dim)++ forward = None+++class VisionConv3dPatchEmbed(BaseOP):+ """Kept as a real Conv3d (kernel == stride): an equivalent unfolded matmul rounds differently."""++ def __init__(self, vc: VisionConfig):+ self._shape = (vc.in_channels, vc.temporal_patch_size, vc.patch_size, vc.patch_size)+ self.proj = _Conv3dParams(vc.hidden_size, self._shape)++ def forward(self, pixel_values: torch.Tensor) -> torch.Tensor:+ x = pixel_values.view(-1, *self._shape).to(self.proj.weight.dtype)+ out = F.conv3d(x, self.proj.weight, self.proj.bias, stride=self._shape[1:])+ return out.view(x.shape[0], -1)+++class VisionPatchMerger(BaseOP):+ def __init__(+ self,+ vc: VisionConfig,+ use_postshuffle_norm: bool = False,+ *,+ quant_config: QuantConfig | None = None,+ prefix: str = "",+ ):+ merged = vc.hidden_size * vc.spatial_merge_size**2+ # the DeepStack mergers normalize the 2x2-merged vector, the final merger each patch+ self.norm = VisionLayerNorm(merged if use_postshuffle_norm else vc.hidden_size)+ self.linear_fc1 = LinearColParallelMerged(+ merged, [merged], has_bias=True, quant_config=quant_config, prefix=f"{prefix}.linear_fc1"+ )+ self.linear_fc2 = LinearRowParallel(+ merged, vc.out_hidden_size, has_bias=True, quant_config=quant_config, prefix=f"{prefix}.linear_fc2"+ )+ self._merged = merged+ self._postshuffle_norm = use_postshuffle_norm++ def forward(self, x: torch.Tensor) -> torch.Tensor:+ if self._postshuffle_norm:+ x = self.norm.forward(x.view(-1, self._merged))+ else:+ x = self.norm.forward(x).view(-1, self._merged)+ return self.linear_fc2.forward(F.gelu(self.linear_fc1.forward(x)))+++class Qwen3VLVisionModel(BaseOP):+ """Pixels -> per-image soft tokens [sum(h*w)/merge^2, out_hidden x (1 + DeepStack levels)].++ Columns past out_hidden are the DeepStack side features in deepstack_visual_indexes order.+ """++ def __init__(self, vc: VisionConfig, *, quant_config: QuantConfig | None = None, prefix: str = "visual"):+ self.patch_embed = VisionConv3dPatchEmbed(vc)+ self.pos_embed = _EmbeddingParams(vc.num_position_embeddings, vc.hidden_size)+ self.blocks = OPList(+ [VisionBlock(vc, quant_config=quant_config, prefix=f"{prefix}.blocks.{i}") for i in range(vc.depth)]+ )+ self.merger = VisionPatchMerger(vc, quant_config=quant_config, prefix=f"{prefix}.merger")+ self.deepstack_merger_list = OPList(+ [+ VisionPatchMerger(+ vc, use_postshuffle_norm=True, quant_config=quant_config, prefix=f"{prefix}.deepstack_merger_list.{k}"+ )+ for k in range(len(vc.deepstack_visual_indexes))+ ]+ )+ self._vc = vc+ self._num_grid_per_side = int(vc.num_position_embeddings**0.5)+ self._inv_dim = (vc.hidden_size // vc.num_heads) // 2+ self._inv_freq: torch.Tensor | None = None+ self._streamer: BlockWeightStreamer | None = None++ def place_weights(self, mode: str) -> None:+ """gpu: every tensor resident; host: block tensors in pinned banks streamed two blocks at a time (mergers and embeddings stay resident)."""+ if mode == "host" and self._streamer is None:+ self._streamer = BlockWeightStreamer(self.blocks.op_list, self.pos_embed.weight.device)+ elif mode == "gpu" and self._streamer is not None:+ self._streamer.unstream()+ self._streamer = None+ elif mode not in ("gpu", "host"):+ raise ValueError(f"unknown vision weight placement {mode!r}")++ def _blocks(self):+ if self._streamer is None:+ return enumerate(self.blocks.op_list)+ return self._streamer.blocks(self.blocks.op_list)++ def _add_pos_embed(self, x: torch.Tensor, grid: torch.Tensor) -> None:+ """x += bilinear resample of the position table, in row chunks so the fp32 temporaries stay small."""+ from transformers.vision_utils import get_vision_interpolation_indices_and_weights++ interp_idx, interp_w = get_vision_interpolation_indices_and_weights(+ grid,+ num_grid_per_side=self._num_grid_per_side,+ mode="bilinear",+ align_corners=True,+ spatial_merge_size=self._vc.spatial_merge_size,+ )+ table = self.pos_embed.weight+ for lo in range(0, x.shape[0], _POS_ROWS):+ idx, w = interp_idx[lo : lo + _POS_ROWS], interp_w[lo : lo + _POS_ROWS]+ # one corner at a time, multiply then add: the same per-element arithmetic as the fused [S, 4, hidden] .sum(1)+ acc = F.embedding(idx[:, 0], table).float() * w[:, 0, None]+ for k in range(1, idx.shape[1]):+ acc += F.embedding(idx[:, k], table).float() * w[:, k, None]+ x[lo : lo + _POS_ROWS].add_(acc.to(x.dtype))++ def _rope_table(self, grid: torch.Tensor, device: torch.device) -> tuple[torch.Tensor, torch.Tensor]:+ from transformers.vision_utils import get_vision_position_ids++ if self._inv_freq is None or self._inv_freq.device != device:+ # built lazily: the model itself is constructed on the meta device+ # kept in the model dtype (bf16) on purpose; fp32 here would change the rope numerics+ inv_dim = self._inv_dim+ self._inv_freq = (+ 1.0 / (10000.0 ** (torch.arange(0, inv_dim, 2, dtype=torch.float32, device=device) / inv_dim))+ ).to(self.pos_embed.weight.dtype)+ pos_ids = get_vision_position_ids(grid, self._vc.spatial_merge_size)+ # long * bf16 -> bf16 freqs on purpose; fp32 freqs would round differently+ freqs = (pos_ids.unsqueeze(-1) * self._inv_freq).flatten(1) # [S, head_dim/2]+ # per-token rope rows [cos | sin] consumed by the NeoX kernel via positions=arange+ cache = torch.cat((freqs.cos(), freqs.sin()), dim=-1).to(torch.float32).contiguous()+ positions = torch.arange(pos_ids.shape[0], dtype=torch.int32, device=device)+ return cache, positions++ def _embed(self, pixel_values: torch.Tensor, grid_thw: List[List[int]]) -> tuple[torch.Tensor, tuple]:+ """Patch embeddings with the position table added, and the attention inputs every block takes."""+ from transformers.vision_utils import get_vision_cu_seqlens++ device = self.pos_embed.weight.device+ grid = torch.tensor(grid_thw, dtype=torch.long, device=device)+ x = self.patch_embed.forward(pixel_values.to(device))+ self._add_pos_embed(x, grid)+ cache, positions = self._rope_table(grid, device)+ return x, (cache, positions, torch.diff(get_vision_cu_seqlens(grid)).tolist())++ def _encode(+ self, pixel_values: torch.Tensor, grid_thw: List[List[int]], groups: torch.Tensor | None = None+ ) -> torch.Tensor:+ """Pixels through the patch embedding and the block stack; with ``groups`` each DeepStack tap's merged feature is copied into groups[1 + k]."""+ taps = self._vc.deepstack_visual_indexes+ x, attn = self._embed(pixel_values, grid_thw)+ for i, blk in self._blocks():+ x = blk.forward(x, *attn)+ if groups is not None and i in taps:+ k = taps.index(i)+ groups[1 + k].copy_(self.deepstack_merger_list.op_list[k].forward(x), non_blocking=True)+ return x++ @staticmethod+ def _unpark(groups: torch.Tensor, device: torch.device) -> torch.Tensor:+ """[rows, n * width] on the device from host slabs [n, rows, width]; group k fills columns k*width:(k+1)*width."""+ n, rows, width = groups.shape+ out = torch.empty(rows, n * width, dtype=groups.dtype, device=device)+ for k in range(n):+ out[:, k * width : (k + 1) * width].copy_(groups[k], non_blocking=True)+ return out++ @torch.inference_mode()+ def forward(self, pixel_values: torch.Tensor, grid_thw: List[List[int]]) -> torch.Tensor:+ """Soft tokens [rows, width x (1 + taps)], rows = sum(h*w) / merge^2: the merger's columns first, then one DeepStack tap per range.++ Same numbers as ``forward_naive``.+ """+ vc = self._vc+ taps = vc.deepstack_visual_indexes+ if not taps:+ return self.merger.forward(self._encode(pixel_values, grid_thw))+ # the merger outputs leave the GPU as they are produced and come back after the last block, so only the block working set is resident while the blocks run+ rows = pixel_values.shape[0] // vc.spatial_merge_size**2+ groups = torch.empty(+ (1 + len(taps), rows, vc.out_hidden_size), dtype=self.pos_embed.weight.dtype, device="cpu",+ pin_memory=torch.cuda.is_available(),+ )+ x = self._encode(pixel_values, grid_thw, groups)+ groups[0].copy_(self.merger.forward(x), non_blocking=True)+ del x+ return self._unpark(groups, self.pos_embed.weight.device)++ @torch.inference_mode()+ def forward_naive(self, pixel_values: torch.Tensor, grid_thw: List[List[int]]) -> torch.Tensor:+ """Reference forward, written the plain way: the block stack, then the merger outputs side by side."""+ taps = self._vc.deepstack_visual_indexes+ x, attn = self._embed(pixel_values, grid_thw)+ features = []+ for i, blk in self._blocks():+ x = blk.forward(x, *attn)+ if i in taps:+ features.append(self.deepstack_merger_list.op_list[taps.index(i)].forward(x))+ return torch.cat([self.merger.forward(x)] + features, dim=1)+++class QwenVLVisionMixin:+ """The engine hooks of a wrapper that owns a Qwen VL tower as ``self.visual``."""++ visual: Qwen3VLVisionModel++ def place_encoder_weights(self, mode: str) -> None:+ self.visual.place_weights(mode)++ def encode(self, item: MMItem) -> torch.Tensor:+ return self.visual.forward(item.feature, [item.grid_thw])+++__all__ = ["Qwen3VLVisionModel", "QwenVLVisionMixin"]diff --git a/python/freetoken/models/qwen3_vl/weight.py b/python/freetoken/models/qwen3_vl/weight.pynew file mode 100644index 00000000..3a7df174--- /dev/null+++ b/python/freetoken/models/qwen3_vl/weight.py@@ -0,0 +1,117 @@+from __future__ import annotations++from typing import Iterator++import safetensors+import torch+from freetoken.distributed import get_tp_info+from freetoken.models.config import VISION_KEY_PREFIXES+from freetoken.models.loader import MergeRule, iter_merged_tensors, iter_weight_files, shard_tensor+from freetoken.utils import cached_load_hf_config+from tqdm import tqdm++from .config import parse_config++_MERGE_RULES = {+ ".q_proj": MergeRule(".qkv_proj", "q", ("q", "k", "v")),+ ".k_proj": MergeRule(".qkv_proj", "k", ("q", "k", "v")),+ ".v_proj": MergeRule(".qkv_proj", "v", ("q", "k", "v")),+ ".gate_proj": MergeRule(".gate_up_proj", "gate", ("gate", "up")),+ ".up_proj": MergeRule(".gate_up_proj", "up", ("gate", "up")),+}+_FUSED_EXPERT_KEYS = ("gate_up_proj", "down_proj")+++def rename_vl_prefix(raw_name: str) -> str:+ """Checkpoint wrapper prefixes -> state-dict keys: tower under visual., text under model."""+ if raw_name.startswith(("model.visual.", "visual.")):+ return "visual." + raw_name.removeprefix("model.").removeprefix("visual.")+ if raw_name.startswith("model.language_model."):+ return "model." + raw_name[len("model.language_model.") :]+ if raw_name.startswith("language_model."):+ return "model." + raw_name[len("language_model.") :]+ return raw_name+++def _is_fused_experts(name: str) -> bool:+ parts = name.split(".")+ return len(parts) > 2 and parts[-2] == "experts" and parts[-1] in _FUSED_EXPERT_KEYS+++def _linear_layout(tensor: torch.Tensor) -> torch.Tensor:+ # the checkpoint stacks experts as [E, in, out]; the banks and fused kernels take [E, out, in] and copy from the strided view+ return tensor.transpose(1, 2)+++def iter_weights(+ model_path: str,+ device: torch.device,+ *,+ include_moe_experts: bool,+ include_non_moe: bool,+ include_vision: bool = True,+) -> Iterator[tuple[str, torch.Tensor]]:+ config = parse_config(cached_load_hf_config(model_path))+ tp_info = get_tp_info()++ def tensors() -> Iterator[tuple[str, torch.Tensor]]:+ for file in tqdm(+ iter_weight_files(model_path),+ desc="Loading weights",+ disable=not tp_info.is_primary(),+ ):+ with safetensors.safe_open(file, framework="pt", device=str(device)) as f:+ for raw_name in f.keys():+ name = rename_vl_prefix(raw_name)+ if not include_vision and name.startswith(VISION_KEY_PREFIXES):+ continue+ is_expert = _is_fused_experts(name)+ if is_expert and not include_moe_experts:+ continue+ if not is_expert and not include_non_moe:+ continue+ raw = f.get_tensor(raw_name)+ if is_expert:+ if tp_info.size > 1:+ raise NotImplementedError("Qwen3-VL-MoE experts are not tensor-parallel sharded")+ yield name, _linear_layout(raw)+ continue+ if name.startswith("visual.") and tp_info.size > 1:+ raise NotImplementedError("Qwen VL vision tower weights are not tensor-parallel sharded")+ tensor = shard_tensor(+ name,+ raw,+ rank=tp_info.rank,+ world_size=tp_info.size,+ num_kv_heads=config.num_kv_heads,+ )+ del raw+ yield name, tensor++ yield from iter_merged_tensors(tensors(), _MERGE_RULES, model_name="qwen3_vl")+++def iter_weights_parallel(+ model_path: str,+ device: torch.device,+ *,+ include_moe_experts: bool,+ include_non_moe: bool,+ workers: int = 8,+ chunk: int = 8 << 20,+) -> Iterator[tuple[str, torch.Tensor]]:+ """experts-only iter_weights over the common chunked multi-threaded O_DIRECT reader."""+ assert include_moe_experts and not include_non_moe, (+ "qwen3_vl parallel reader is experts-only (used by load_moe_expert_sources)"+ )+ from freetoken.models.weight import iter_expert_tensors_parallel++ if get_tp_info().size > 1:+ raise NotImplementedError("Qwen3-VL-MoE experts are not tensor-parallel sharded")+ for raw_name, raw in iter_expert_tensors_parallel(+ model_path, _is_fused_experts, workers=workers, chunk=chunk+ ):+ yield rename_vl_prefix(raw_name), _linear_layout(raw)+++__all__ = ["iter_weights", "iter_weights_parallel", "rename_vl_prefix"]diff --git a/python/freetoken/models/qwen4_exp/__init__.py b/python/freetoken/models/qwen4_exp/__init__.pyindex 6715e67e..b063da6a 100644--- a/python/freetoken/models/qwen4_exp/__init__.py+++ b/python/freetoken/models/qwen4_exp/__init__.py@@ -1,4 +1,4 @@-"""Qwen3.8-Flash-Next (model_type qwen4_exp), served text-only.+"""Qwen3.8-Flash-Next (model_type qwen4_exp); image input rides on the shared Qwen VL tower.48 decoder layers on hc_count=4 hyper-connection residual streams R [T, 4*hidden]:embed -> repeat(1, 4) -> [PLE at zero-based layer 1] -> per layer attn_hc.mix -> (GDN | QSA) -> attn_hc.combine -> mlp_hc.mix -> MoE -> mlp_hc.combine -> top-level mixer.mix -> lm_head.@@ -10,7 +10,7 @@ Contracts shared across modules (do not rename):"""from .config import parse_config-from .model import Qwen4ExpForCausalLM+from .model import Qwen4ExpForCausalLM, Qwen4ExpForConditionalGenerationfrom .weight import (ftw_side_files,nvfp4_expert_spec,@@ -26,6 +26,7 @@ __all__ = ["ftw_side_files","nvfp4_expert_spec","Qwen4ExpForCausalLM",+ "Qwen4ExpForConditionalGeneration","iter_weights","load_ple_table","parse_config",diff --git a/python/freetoken/models/qwen4_exp/attention.py b/python/freetoken/models/qwen4_exp/attention.pyindex 66211c7c..8f6b9388 100644--- a/python/freetoken/models/qwen4_exp/attention.py+++ b/python/freetoken/models/qwen4_exp/attention.py@@ -141,6 +141,8 @@ class Qwen4ExpAttention(BaseOP):max_position=rotary.max_position,base=rotary.base,rope_scaling=tuple(rotary.scaling.items()) if rotary.scaling else None,+ mrope_section=tuple(rotary.mrope_section) if rotary.mrope_section is not None else None,+ mrope_layout=rotary.mrope_layout,)self.indexer = Qwen4ExpIndexer(config, layer_id, prefix=f"{prefix}.indexer")@@ -155,7 +157,7 @@ class Qwen4ExpAttention(BaseOP):self.q_norm.forward_inplace(q)self.k_norm.forward_inplace(k)q, k = self.rotary.forward(- batch.positions, q.view(-1, self.qo_attn_dim), k.view(-1, self.kv_attn_dim)+ batch.get_attn_positions(), q.view(-1, self.qo_attn_dim), k.view(-1, self.kv_attn_dim))index = self.indexer.forward(x)o = get_global_ctx().attn_backend.qsa_forward(diff --git a/python/freetoken/models/qwen4_exp/config.py b/python/freetoken/models/qwen4_exp/config.pyindex 4bcd28de..e4137862 100644--- a/python/freetoken/models/qwen4_exp/config.py+++ b/python/freetoken/models/qwen4_exp/config.py@@ -7,12 +7,14 @@ import torchfrom freetoken.layers.quantization import QuantConfigfrom freetoken.models.config import (+ mrope_layout_from_rope_params,FullAttentionGroupConfig,LinearGatedDeltaGroupConfig,ModelConfig,RotaryConfig,SlotStateSpec,)+from freetoken.models.qwen3_vl.config import parse_vision_config@dataclass(frozen=True)@@ -40,6 +42,7 @@ class Qwen4ExpArgs:index_head_dim: intindex_budget: intindex_ratio: int+ image_token_id: int | None = None@propertydef index_topk_blocks(self) -> int:@@ -152,12 +155,19 @@ def parse_config(hf_config: Any) -> ModelConfig:if layer_types[lid] != "linear_attention":raise ValueError(f"PLE must sit on a linear_attention layer, got layer {lid}")+ vision_config = parse_vision_config(hf_config)full_rotary = RotaryConfig(head_dim=head_dim,rotary_dim=rotary_dim,max_position=text.max_position_embeddings,base=rope_theta,scaling=rope_scaling,+ mrope_section=(+ list(rope_params["mrope_section"])+ if vision_config is not None and "mrope_section" in rope_params+ else None+ ),+ mrope_layout=mrope_layout_from_rope_params(rope_params),)full_group = FullAttentionGroupConfig(name="full",@@ -214,6 +224,7 @@ def parse_config(hf_config: Any) -> ModelConfig:index_head_dim=int(text.indexer_head_dim),index_budget=int(text.indexer_budget),index_ratio=int(text.indexer_compress_ratio),+ image_token_id=getattr(hf_config, "image_token_id", None),)return ModelConfig(@@ -241,7 +252,7 @@ def parse_config(hf_config: Any) -> ModelConfig:use_qk_norm=True,model_type=getattr(hf_config, "model_type", "qwen4_exp"),architectures=getattr(hf_config, "architectures", ["Qwen4ExpForConditionalGeneration"]),- vision_config=None, # served text-only+ vision_config=vision_config,image_token_id=getattr(hf_config, "image_token_id", None),attention_groups=groups,expert_quant=expert_quant,diff --git a/python/freetoken/models/qwen4_exp/model.py b/python/freetoken/models/qwen4_exp/model.pyindex a4b63362..258badac 100644--- a/python/freetoken/models/qwen4_exp/model.py+++ b/python/freetoken/models/qwen4_exp/model.py@@ -27,6 +27,8 @@ from .attention import Qwen4ExpAttentionfrom .hc import GatedResidualfrom .moe import Qwen4ExpMoEfrom .ple import PLELayer+from freetoken.models.blocks import embed_input_ids+from freetoken.models.qwen3_vl.vision import Qwen3VLVisionModel, QwenVLVisionMixinif TYPE_CHECKING:from freetoken.core import Batch@@ -107,7 +109,8 @@ class Qwen4ExpModel(BaseOP):return list(self._ple)def forward(self, input_ids: torch.Tensor, batch: Batch) -> torch.Tensor:- hidden = self.embed_tokens.forward(input_ids).repeat(1, self.hc_count)+ hidden = embed_input_ids(self.embed_tokens, input_ids, batch)+ hidden = hidden.repeat(1, self.hc_count)meta = Noneif self._ple:from .ple import build_ple_metadata, commit_ngram_context@@ -180,6 +183,7 @@ class Qwen4ExpForCausalLM(BaseLLMModel):"per_head_vocab_sizes": emb.ngram_heads_vocab_sizes.tolist(),"per_head_offsets": emb.ngram_heads_offsets.tolist(),"eos_token_id": args.ngram_boundary_token_id,+ "image_token_id": args.image_token_id,}disk_table = DiskRowTable(resolve_row_source(folder),@@ -209,4 +213,18 @@ class Qwen4ExpForCausalLM(BaseLLMModel):return self.lm_head.forward(self.model.forward(batch.input_ids, batch))-__all__ = ["Qwen4ExpDecoderLayer", "Qwen4ExpForCausalLM", "Qwen4ExpModel", "build_linear_mixer"]+class Qwen4ExpForConditionalGeneration(QwenVLVisionMixin, Qwen4ExpForCausalLM):+ def __init__(self, config: ModelConfig) -> None:+ super().__init__(config)+ if config.is_multimodal:+ assert not config.vision_config.deepstack_visual_indexes, "Qwen3.8 consumes no DeepStack features"+ self.visual = Qwen3VLVisionModel(config.vision_config, quant_config=config.quant, prefix="visual")+++__all__ = [+ "Qwen4ExpDecoderLayer",+ "Qwen4ExpForCausalLM",+ "Qwen4ExpForConditionalGeneration",+ "Qwen4ExpModel",+ "build_linear_mixer",+]diff --git a/python/freetoken/models/qwen4_exp/ple.py b/python/freetoken/models/qwen4_exp/ple.pyindex 13157cb8..c944c494 100644--- a/python/freetoken/models/qwen4_exp/ple.py+++ b/python/freetoken/models/qwen4_exp/ple.py@@ -26,6 +26,7 @@ import torchimport torch.nn.functional as Ffrom freetoken.core import get_global_ctxfrom freetoken.layers import BaseOP, LinearReplicated+from freetoken.mm import restore_placeholderfrom .config import PLE_CONV_STATE, PLE_NGRAM_STATEfrom .hc import GroupedPlusOneRMSNorm@@ -333,12 +334,16 @@ def build_ple_metadata()fla = getattr(batch, "fla_metadata", None)slots_dev = getattr(batch, "linear_table_idx", None)+ # image rows hash as the placeholder token, never as the content pad ids+ input_ids = batch.input_ids+ if args.image_token_id is not None:+ input_ids = restore_placeholder(input_ids, args.image_token_id)if batch.is_decode and slots_dev is not None:slots = slots_dev.long()bs = slots.numel()return PLEMetadata(- input_ids=batch.input_ids,+ input_ids=input_ids,cu_seqlens=torch.arange(bs + 1, dtype=torch.int32, device=device),seq_lens=(1,) * bs,ngram_context=context_pool.index_select(0, slots).long(),@@ -360,7 +365,7 @@ def build_ple_metadata(context = context_pool.index_select(0, slots).long()context = torch.where(fresh.unsqueeze(1), context.new_full((), eos), context)return PLEMetadata(- input_ids=batch.input_ids,+ input_ids=input_ids,cu_seqlens=cu,seq_lens=tuple(lens),ngram_context=context,diff --git a/python/freetoken/models/qwen4_exp/ple_disk.py b/python/freetoken/models/qwen4_exp/ple_disk.pyindex 799866a6..cea62693 100644--- a/python/freetoken/models/qwen4_exp/ple_disk.py+++ b/python/freetoken/models/qwen4_exp/ple_disk.py@@ -13,6 +13,8 @@ from typing import Sequenceimport safetensorsimport torch+from freetoken.mm import MM_PAD_SHIFT_VALUE, restore_placeholder+from freetoken.core import Batchfrom freetoken.kernel.pinned import alloc_pinned_tensorfrom freetoken.utils import init_logger@@ -118,6 +120,7 @@ class DiskRowTable:self.heads = int(hash_constants["num_ngram_heads"])self.scale = source.scaleself.eos_token_id = int(hash_constants["eos_token_id"])+ self.image_token_id = hash_constants.get("image_token_id")sizes = [int(x) for x in hash_constants["per_head_vocab_sizes"]]offsets = [int(x) for x in hash_constants["per_head_offsets"]]need = max(o + s for o, s in zip(offsets, sizes))@@ -191,6 +194,16 @@ class DiskRowTable:offset += run.numel() - 2self._store.flush(self._flag.data_ptr() if graph and self._wait_sync else 0)+ def _ple_ids(self, input_ids: torch.Tensor) -> torch.Tensor:+ if self.image_token_id is None:+ return input_ids+ return restore_placeholder(input_ids, self.image_token_id)++ def _ple_context(self, ids: torch.Tensor, position: int) -> list[int]:+ if self.image_token_id is None:+ return _context(ids, position, self.eos_token_id)+ return [self.image_token_id if t >= MM_PAD_SHIFT_VALUE else t for t in _context(ids, position, self.eos_token_id)]+def host_fill_batch(self, batch: Batch, use_graph: bool):"""Stage this batch's rows; returns the post-dispatch fill callable under flag-sync, else None."""eos = self.eos_token_id@@ -205,7 +218,7 @@ class DiskRowTable:try:self._readback_event.synchronize()tokens = self._token_readback[:bs].to(torch.int64).tolist()- runs = [torch.tensor([*_context(r.input_ids, r.device_len - 1, eos), t], dtype=torch.int64)+ runs = [torch.tensor([*self._ple_context(r.input_ids, r.device_len - 1), t], dtype=torch.int64)for r, t in zip(reqs, tokens)]self.fill(runs, graph=True)except BaseException:@@ -218,14 +231,14 @@ class DiskRowTable:return _complete# launch-gating: this D2H is the step's readback and orders the fill after samplingtokens = batch.input_ids.to("cpu").to(torch.int64).tolist()- runs = [torch.tensor([*_context(r.input_ids, r.device_len - 1, eos), t], dtype=torch.int64)+ runs = [torch.tensor([*self._ple_context(r.input_ids, r.device_len - 1), t], dtype=torch.int64)for r, t in zip(reqs, tokens)]self.fill(runs, graph=use_graph)return Noneruns = [torch.cat((- torch.tensor(_context(req.input_ids, req.cached_len, eos), dtype=torch.int64),- req.input_ids[req.cached_len : req.device_len].to(torch.int64),+ torch.tensor(self._ple_context(req.input_ids, req.cached_len), dtype=torch.int64),+ self._ple_ids(req.input_ids[req.cached_len : req.device_len]).to(torch.int64),))for req in batch.padded_reqs]diff --git a/python/freetoken/models/qwen4_exp/weight.py b/python/freetoken/models/qwen4_exp/weight.pyindex 4a2a5b61..0ad04011 100644--- a/python/freetoken/models/qwen4_exp/weight.py+++ b/python/freetoken/models/qwen4_exp/weight.py@@ -6,7 +6,7 @@ Three separate paths, because the checkpoint's three weight classes live in diff* :func:`load_ple_table` -- the 47.7 GiB FP8 n-gram table, 128 checkpoint shards concatenated into one pinned :class:`HostBank`.* :func:`nvfp4_expert_spec` -- how the routed NVFP4 experts are named, for the offload cache's expert reader.-Dropped: ``mtp.*`` (speculative head, including its stacked ``mtp.layers.0.mlp.experts.*``) and ``model.visual.*`` (served text-only).+Dropped: ``mtp.*`` (speculative head, including its stacked ``mtp.layers.0.mlp.experts.*``); ``model.visual.*`` is kept only when the model built the tower."""from __future__ import annotations@@ -21,6 +21,9 @@ from typing import Iteratorimport safetensorsimport torchfrom freetoken.distributed import get_tp_info+from freetoken.models.qwen3_vl.weight import rename_vl_prefix++from freetoken.models.config import VISION_KEY_PREFIXESfrom freetoken.models.loader import drop_page_cache, iter_weight_filesfrom freetoken.models.nvfp4_banks import (Nvfp4ExpertSourceSpec,@@ -83,7 +86,7 @@ _ELEM_DTYPES = {"e4m3": torch.float8_e4m3fn}def _rename(raw_name: str) -> str | None:"""Checkpoint key -> FreeToken state-dict key, or None to skip."""- if raw_name.startswith(("mtp.", "model.visual.", "visual.")):+ if raw_name.startswith("mtp."):return Noneif _PLE_TABLE_INFIX in raw_name:return None # n-gram table + its scale: load_ple_table@@ -91,11 +94,7 @@ def _rename(raw_name: str) -> str | None:return None # routed experts: offload source banksif raw_name.endswith(_SCALE_SUFFIXES):return None- if raw_name.startswith("model.language_model."):- return "model." + raw_name[len("model.language_model.") :]- if raw_name.startswith("language_model."):- return "model." + raw_name[len("language_model.") :]- return raw_name+ return rename_vl_prefix(raw_name)def _split_kind(name: str) -> tuple[str, str]:@@ -195,6 +194,7 @@ def iter_weights(*,include_moe_experts: bool,include_non_moe: bool,+ include_vision: bool = True,) -> Iterator[tuple[str, torch.Tensor]]:"""Yield the dense (non-expert) weights, prefix-stripped and fused to the model's buffers.@@ -221,6 +221,8 @@ def iter_weights(name = _rename(raw_name)if name is None:continue+ if not include_vision and name.startswith(VISION_KEY_PREFIXES):+ continuetensor = f.get_tensor(raw_name)fused = fuser.fuse(name, tensor)if fused is None:diff --git a/python/freetoken/models/register.py b/python/freetoken/models/register.pyindex 957c0fa9..d0666f9d 100644--- a/python/freetoken/models/register.py+++ b/python/freetoken/models/register.py@@ -8,6 +8,13 @@ if TYPE_CHECKING:from .config import ModelConfig+@dataclass(frozen=True)+class EncoderSpec:+ kind: str # "vision" | "audio", the name --mm-disable takes+ config_key: str # the checkpoint config section that builds the tower+ modalities: tuple[str, ...] # inputs it serves, as implemented++@dataclass(frozen=True)class ModelSpec:module: str@@ -22,10 +29,17 @@ class ModelSpec:packed_modules_mapping: tuple[tuple[str, tuple[str, ...]], ...] = ()# checkpoint-name globs the family serves in bf16 although the quantization_config covers themunquantized_modules: tuple[str, ...] = ()+ # "module:Class" turning the checkpoint's media into items; None: the family takes no multimodal input+ mm_processor: str | None = None+ encoders: tuple[EncoderSpec, ...] = ()# Multimodal wrappers store the text tower under model.language_model._LANGUAGE_MODEL_ROOT = (("model", "model.language_model"),)+_QWEN_VL_ROOTS = _LANGUAGE_MODEL_ROOT + (("visual", "model.visual"),)+_QWEN_VL_PROCESSOR = "freetoken.mm.processors.qwen_vl:QwenVLMMProcessor"+# TODO: video once the processor samples frames; the tower already takes grid_thw with t > 1+_QWEN_VL_ENCODERS = (EncoderSpec("vision", "vision_config", ("image",)),)# Fused projections split back into their HF leaves so the quantization_config lookup sees the stored names._DENSE_PACKED = (("qkv_proj", ("q_proj", "k_proj", "v_proj")),@@ -34,6 +48,8 @@ _DENSE_PACKED = (# per-expert checkpoints: probe the first expert's projections for the experts container_EXPERTS_PACKED = (("experts", ("experts.0.gate_proj", "experts.0.up_proj", "experts.0.down_proj")),)_EXPERTS_W123_PACKED = (("experts", ("experts.0.w1", "experts.0.w2", "experts.0.w3")),)+# pre-stacked expert checkpoints (Qwen3-VL-MoE): the container's own two tensors are the probe+_STACKED_EXPERTS_PACKED = (("experts", ("experts.gate_up_proj", "experts.down_proj")),)_QWEN3_5_PACKED = _DENSE_PACKED + (("in_proj_qkvz", ("in_proj_qkv", "in_proj_z")),("in_proj_ba", ("in_proj_b", "in_proj_a")),@@ -81,6 +97,23 @@ _MODEL_REGISTRY: dict[str, ModelSpec] = {"Qwen3MoeForCausalLM",packed_modules_mapping=_DENSE_PACKED + _EXPERTS_PACKED,),+ # Qwen3-VL: the Qwen3 text tower under model.language_model. plus the shared Qwen VL vision tower with DeepStack; the MoE variant ships its experts pre-stacked.+ "Qwen3VLForConditionalGeneration": ModelSpec(+ "freetoken.models.qwen3_vl",+ "Qwen3VLForConditionalGeneration",+ checkpoint_roots=_QWEN_VL_ROOTS,+ mm_processor=_QWEN_VL_PROCESSOR,+ encoders=_QWEN_VL_ENCODERS,+ packed_modules_mapping=_DENSE_PACKED,+ ),+ "Qwen3VLMoeForConditionalGeneration": ModelSpec(+ "freetoken.models.qwen3_vl",+ "Qwen3VLMoeForConditionalGeneration",+ checkpoint_roots=_QWEN_VL_ROOTS,+ mm_processor=_QWEN_VL_PROCESSOR,+ encoders=_QWEN_VL_ENCODERS,+ packed_modules_mapping=_DENSE_PACKED + _STACKED_EXPERTS_PACKED,+ ),"MiniMaxM2ForCausalLM": ModelSpec("freetoken.models.minimax_m2","MiniMaxM2ForCausalLM",@@ -113,19 +146,36 @@ _MODEL_REGISTRY: dict[str, ModelSpec] = {),"Qwen3_5MoeForConditionalGeneration": ModelSpec("freetoken.models.qwen3_5_moe",- "Qwen3_5MoEForCausalLM",- checkpoint_roots=_LANGUAGE_MODEL_ROOT,+ "Qwen3_5MoeForConditionalGeneration",+ checkpoint_roots=_QWEN_VL_ROOTS,+ mm_processor=_QWEN_VL_PROCESSOR,+ encoders=_QWEN_VL_ENCODERS,+ packed_modules_mapping=_QWEN3_5_PACKED,+ unquantized_modules=_QWEN3_5_UNQUANTIZED,+ ),+ # text-only Qwen3.5 releases: text_config is the top-level config and there is no tower+ "Qwen3_5MoeForCausalLM": ModelSpec(+ "freetoken.models.qwen3_5_moe",+ "Qwen3_5MoeForCausalLM",+ packed_modules_mapping=_QWEN3_5_PACKED,+ unquantized_modules=_QWEN3_5_UNQUANTIZED,+ ),+ "Qwen3_5ForCausalLM": ModelSpec(+ "freetoken.models.qwen3_5_moe",+ "Qwen3_5ForCausalLM",packed_modules_mapping=_QWEN3_5_PACKED,unquantized_modules=_QWEN3_5_UNQUANTIZED,),# Qwen3.8-Flash-Next (model_type qwen4_exp): multimodal wrapper config (text tower in- # text_config, weights under model.language_model.); served text-only. 36 GDN + 12 QSA+ # text_config, weights under model.language_model.). 36 GDN + 12 QSA# compressed-sparse attention layers on 4 hyper-connection residual streams, a PLE# n-gram embedding layer, 512 NVFP4 routed experts top-10 + a gated shared expert."Qwen4ExpForConditionalGeneration": ModelSpec("freetoken.models.qwen4_exp",- "Qwen4ExpForCausalLM",- checkpoint_roots=_LANGUAGE_MODEL_ROOT,+ "Qwen4ExpForConditionalGeneration",+ checkpoint_roots=_QWEN_VL_ROOTS,+ mm_processor=_QWEN_VL_PROCESSOR,+ encoders=_QWEN_VL_ENCODERS,packed_modules_mapping=_QWEN4_EXP_PACKED,),# Dense Qwen3.x (no "Moe" in the arch name, num_experts==0, e.g. Qwen3.6-27B). Shares the@@ -133,8 +183,10 @@ _MODEL_REGISTRY: dict[str, ModelSpec] = {# loader handles the compressed-tensors NVFP4 layout."Qwen3_5ForConditionalGeneration": ModelSpec("freetoken.models.qwen3_5_moe",- "Qwen3_5MoEForCausalLM",- checkpoint_roots=_LANGUAGE_MODEL_ROOT,+ "Qwen3_5ForConditionalGeneration",+ checkpoint_roots=_QWEN_VL_ROOTS,+ mm_processor=_QWEN_VL_PROCESSOR,+ encoders=_QWEN_VL_ENCODERS,packed_modules_mapping=_QWEN3_5_PACKED,unquantized_modules=_QWEN3_5_UNQUANTIZED,),diff --git a/python/freetoken/models/weight.py b/python/freetoken/models/weight.pyindex 0cff7eec..5165d149 100644--- a/python/freetoken/models/weight.py+++ b/python/freetoken/models/weight.py@@ -211,6 +211,7 @@ def load_weight(device: torch.device,*,include_moe_experts: bool = True,+ include_vision: bool = True,) -> Iterator[Tuple[str, torch.Tensor]]:# FTW checkpoint: dense weights are stored post-iter_weights, so we replay them# model-agnostically instead of re-running the per-model reader. Which tensors exist is@@ -218,28 +219,27 @@ def load_weight(# fails loudly in load_state_dict (strict missing/unexpected expert keys), so the reader# just yields the stored weight tensors regardless of the include_moe_experts flag.from freetoken.checkpoint.ftw import is_ftw_checkpoint, iter_ftw_weights- from freetoken.models.config import VISION_KEY_PREFIXES, vision_load_enabled+ from freetoken.models.config import VISION_KEY_PREFIXESif is_ftw_checkpoint(model_path):- # The FTW dense shard stores whatever existed at conversion, including the vision- # stack. Vision is opt-in (default OFF, see vision_load_enabled): when it is off the- # model never builds the tower, so replaying those tensors would trip load_state_dict's- # strict unexpected-key check. Skip them here to match the model the engine built.- skip_vision = not vision_load_enabled()- for name, tensor in iter_ftw_weights(model_path):- if skip_vision and name.startswith(VISION_KEY_PREFIXES):- continue- yield name, tensor- return-- _config, spec = _spec_for_model_path(model_path)- iter_weights = _load_attr(spec.module, spec.iter_weights)- yield from iter_weights(- model_path,- device,- include_moe_experts=include_moe_experts,- include_non_moe=True,- )+ weights = iter_ftw_weights(model_path)+ else:+ _config, spec = _spec_for_model_path(model_path)+ iter_weights = _load_attr(spec.module, spec.iter_weights)+ # only a family that registers an encoder is asked about the tower; the others never load one+ kwargs = {"include_vision": include_vision} if spec.encoders else {}+ weights = iter_weights(+ model_path,+ device,+ include_moe_experts=include_moe_experts,+ include_non_moe=True,+ **kwargs,+ )+ for name, tensor in weights:+ # the FTW replay has no reader to skip the tower; a text-only engine never built it+ if not include_vision and name.startswith(VISION_KEY_PREFIXES):+ continue+ yield name, tensordef load_q4_0_moe_expert_sources(diff --git a/python/freetoken/models/weight_stream.py b/python/freetoken/models/weight_stream.pynew file mode 100644index 00000000..ac529eca--- /dev/null+++ b/python/freetoken/models/weight_stream.py@@ -0,0 +1,122 @@+"""Block weights streamed from pinned host banks through two device staging buffers: block i computes from buffer i % 2 while the copy stream fills the other with block i + 1."""++from __future__ import annotations++from dataclasses import dataclass+from typing import Any, Iterator, List, Sequence++import torch++from freetoken.layers import BaseOP++_ALIGN = 256+++@dataclass(frozen=True)+class _Slot:+ owner: Any+ attr: str+ offset: int+ nbytes: int+ dtype: torch.dtype+ shape: torch.Size++ def view(self, row: torch.Tensor) -> torch.Tensor:+ return row[self.offset : self.offset + self.nbytes].view(self.dtype).view(self.shape)+++def _slots(block: BaseOP) -> tuple[List[_Slot], int]:+ """Every tensor attribute of the block (recursively), laid out back to back with 256-byte alignment."""+ slots: List[_Slot] = []+ offset = 0++ def walk(op: BaseOP) -> None:+ nonlocal offset+ for name, value in op.__dict__.items():+ if name.startswith("_"):+ continue+ if isinstance(value, torch.Tensor):+ nbytes = value.numel() * value.element_size()+ slots.append(_Slot(op, name, offset, nbytes, value.dtype, value.shape))+ offset += -(-nbytes // _ALIGN) * _ALIGN+ elif isinstance(value, BaseOP):+ walk(value)++ walk(block)+ return slots, offset+++class BlockWeightStreamer:+ """Owns the host banks and the two staging buffers of a block stack; drives one forward with `blocks()`."""++ def __init__(self, blocks: Sequence[BaseOP], device: torch.device):+ self.device = device+ self._layouts = [_slots(b) for b in blocks]+ row_bytes = self._layouts[0][1]+ assert all(nbytes == row_bytes for _, nbytes in self._layouts), "streamed blocks must share one layout"+ self.bank = torch.empty((len(blocks), row_bytes), dtype=torch.uint8, pin_memory=True)+ self.staging = torch.empty((2, row_bytes), dtype=torch.uint8, device=device)+ for b, (slots, _) in enumerate(self._layouts):+ for s in slots:+ s.view(self.bank[b]).copy_(getattr(s.owner, s.attr))+ torch.cuda.synchronize(device)+ self._bind_all_to_host()+ self.copy_stream = torch.cuda.Stream(device=device)+ self.begin_event = torch.cuda.Event()+ self.ready_events = [torch.cuda.Event() for _ in range(2)]+ self.release_events = [torch.cuda.Event() for _ in range(2)]+ self._holder: List[int | None] = [None, None]+ self._has_release: List[bool] = [False, False]++ @property+ def device_bytes(self) -> int:+ return self.staging.numel()++ def _bind_all_to_host(self) -> None:+ # between forwards the attributes point at the pinned bank, so state_dict() stays complete and correct+ for b, (slots, _) in enumerate(self._layouts):+ for s in slots:+ setattr(s.owner, s.attr, s.view(self.bank[b]))++ def unstream(self) -> None:+ """Put every block's tensors back on the device as plain resident copies."""+ for b, (slots, _) in enumerate(self._layouts):+ for s in slots:+ setattr(s.owner, s.attr, s.view(self.bank[b]).to(self.device))+ torch.cuda.synchronize(self.device)++ def _prefetch(self, i: int) -> None:+ if i >= len(self._layouts):+ return+ buf = i % 2+ if self._holder[buf] == i:+ return+ with torch.cuda.stream(self.copy_stream):+ if self._has_release[buf]:+ self.copy_stream.wait_event(self.release_events[buf])+ self.staging[buf].copy_(self.bank[i], non_blocking=True)+ self.ready_events[buf].record(self.copy_stream)+ self._holder[buf] = i++ def blocks(self, ops: Sequence[BaseOP]) -> Iterator[tuple[int, BaseOP]]:+ """Yield (i, block) with block i's tensors bound to a filled staging buffer while block i + 1 streams in."""+ compute = torch.cuda.current_stream(self.device)+ # the copy stream must not overwrite staging that a still-running previous forward reads+ self.begin_event.record(compute)+ self.copy_stream.wait_event(self.begin_event)+ for i, op in enumerate(ops):+ buf = i % 2+ self._prefetch(i)+ self._prefetch(i + 1)+ compute.wait_event(self.ready_events[buf])+ slots, _ = self._layouts[i]+ for s in slots:+ setattr(s.owner, s.attr, s.view(self.staging[buf]))+ yield i, op+ self.release_events[buf].record(compute)+ self._has_release[buf] = True+ for s in slots:+ setattr(s.owner, s.attr, s.view(self.bank[i]))+++__all__ = ["BlockWeightStreamer"]diff --git a/python/freetoken/scheduler/cache.py b/python/freetoken/scheduler/cache.pyindex 44adde42..8f77be93 100644--- a/python/freetoken/scheduler/cache.py+++ b/python/freetoken/scheduler/cache.py@@ -93,10 +93,7 @@ class CacheManager:def match_req(self, req: PendingReq) -> MatchResult:input_len = req.input_lenassert input_len > 0, "Input length must be greater than 0."- # Multimodal requests must not reuse a shared prefix: image-placeholder tokens- # have identical ids across images but carry different content (and KV), so a- # match would serve the wrong image's KV. Match against the empty prefix.- ids = req.input_ids[:0] if req.mm_embeds is not None else req.input_ids[: input_len - 1]+ ids = req.input_ids[: input_len - 1]if self.is_swa:from freetoken.kvcache.swa_radix_cache import SWACacheHandlem = self.prefix_cache.match_prefix(ids)@@ -299,17 +296,6 @@ class CacheManager:# We should free it if the request has finished.page_indices = self.page_table[req.table_idx, : req.cached_len]old_handle = req.cache_handle- # Multimodal requests are never inserted into the shared prefix cache (see- # ``match_req``). Their KV pages stay owned by the active request and are freed- # on completion; nothing is exposed for cross-request reuse.- if req.mm_embeds is not None:- self.unlock(old_handle)- if finished:- tail = self._padded_tail(req, old_handle.cached_len)- if self.swa_paged:- self._free_swa(tail)- self._free(tail)- returninsert_ids = req.input_ids[: req.cached_len]cached_len, new_handle = self.prefix_cache.insert_prefix(insert_ids, page_indices)# unlock until all operations on handle is done@@ -350,13 +336,6 @@ class CacheManager:old_handle = req.cache_handlepage_indices = self.page_table[req.table_idx, : req.cached_len]- if req.mm_embeds is not None:- self.unlock(old_handle)- if finished:- self._free(page_indices[old_handle.cached_len :])- self._free_req_slots(req)- return-if finished:# A pending freeze (the tool-call anchor, or a prefill ×64 track the request# finished too early to chunk-commit) is a strictly shorter prefix than the live@@ -444,14 +423,6 @@ class CacheManager:old_handle = req.cache_handlepage_indices = self.page_table[req.table_idx, : req.cached_len]- if req.mm_embeds is not None:- self.unlock(old_handle)- if finished:- tail = self._padded_tail(req, old_handle.cached_len)- self._free_swa(tail)- self._free(tail)- return-insert_len = align_down(req.cached_len, self.page_size)freed = page_indices[:0]if insert_len > 0:diff --git a/python/freetoken/scheduler/mm.py b/python/freetoken/scheduler/mm.pynew file mode 100644index 00000000..5d9e4d86--- /dev/null+++ b/python/freetoken/scheduler/mm.py@@ -0,0 +1,64 @@+"""Scheduler-side multimodal planning: the embedding rows a chunk gathers and the items it encodes."""++from __future__ import annotations++from typing import TYPE_CHECKING, List, Tuple++if TYPE_CHECKING:+ from freetoken.message import MMItem+ from freetoken.mm.encoder_cache import EncoderCache+++def mm_rows_after(item: MMItem, cached_len: int) -> int:+ """Embedding rows of the item that prefill will gather once the first cached_len tokens are skipped."""+ return sum(max(0, span_hi - max(span_lo, cached_len)) for span_lo, span_hi in item.offsets)+++def plan_mm_chunk(+ uid: int,+ items: List[MMItem],+ window_lo: int,+ window_hi: int,+ encoder_cache: EncoderCache | None,+) -> tuple[List[MMItem], List[Tuple[int, int, int, int, int, int]]]:+ """Encoder jobs and the embedding-row gather plan for the items overlapping the chunk window [window_lo, window_hi).++ Plan rows are (uid, hash, row_lo, row_hi, num_tokens, pos): embedding rows [row_lo, row_hi) land at chunk position pos. A repeated image is one job.+ """+ jobs: List[MMItem] = []+ queued: set[int] = set()+ plan: List[Tuple[int, int, int, int, int, int]] = []+ for item in items:+ num_tokens = item.num_tokens+ needs_encode = item.hash not in queued and (encoder_cache is None or not encoder_cache.has(item.hash))+ row_base = 0+ for span_lo, span_hi in item.offsets:+ lo, hi = max(span_lo, window_lo), min(span_hi, window_hi)+ if lo < hi:+ if needs_encode:+ jobs.append(item)+ queued.add(item.hash)+ needs_encode = False+ plan.append((uid, item.hash, row_base + lo - span_lo, row_base + hi - span_lo, num_tokens, lo - window_lo))+ row_base += span_hi - span_lo+ return jobs, plan+++def plan_mm_batch(reqs, encoder_cache: EncoderCache | None) -> tuple[List[MMItem], List[Tuple[int, int, int, int, int, int]], List[int]]:+ """Jobs, plan and the batch rows of every gathered embedding row, over the reqs in batch order (each spans [cached_len, device_len))."""+ jobs: List[MMItem] = []+ plan: List[Tuple[int, int, int, int, int, int]] = []+ rows: List[int] = []+ offset = 0+ for req in reqs:+ if req.mm_items:+ req_jobs, req_plan = plan_mm_chunk(req.uid, req.mm_items, req.cached_len, req.device_len, encoder_cache)+ jobs.extend(req_jobs)+ plan.extend(req_plan)+ for _, _, row_lo, row_hi, _, pos in req_plan:+ rows.extend(range(offset + pos, offset + pos + row_hi - row_lo))+ offset += req.extend_len+ return jobs, plan, rows+++__all__ = ["mm_rows_after", "plan_mm_batch", "plan_mm_chunk"]diff --git a/python/freetoken/scheduler/prefill.py b/python/freetoken/scheduler/prefill.pyindex 60becf0e..3d94b53a 100644--- a/python/freetoken/scheduler/prefill.py+++ b/python/freetoken/scheduler/prefill.py@@ -7,11 +7,13 @@ import torchfrom freetoken.core import Batch, Reqfrom freetoken.utils import align_down, div_ceil, init_logger+from .mm import mm_rows_afterfrom .utils import PendingReqif TYPE_CHECKING:from freetoken.kvcache import BaseCacheHandlefrom freetoken.message import UserMsg+ from freetoken.mm.encoder_cache import EncoderCachefrom .cache import CacheManagerfrom .decode import DecodeManager@@ -43,6 +45,7 @@ class PrefillAdder:reserved_size: intcache_manager: CacheManagertable_manager: TableManager+ encoder_cache: EncoderCache | None = None# SWA-pool tokens charged to reqs admitted so far this pass. Mirrors reserved_size: swa is# allocated only in allocate_paged (after the pass), so swa_available_size does not decrement# across the admission loop -- without this, successive admits all see the full pool.@@ -185,11 +188,6 @@ class PrefillAdder:_slice = slice(cached_len, cached_len + chunk_size)device_ids = self.table_manager.token_pool[table_idx, _slice]device_ids.copy_(_maybe_pinned(pending_req.input_ids[_slice]), non_blocking=True)- if is_chunked and pending_req.mm_embeds is not None:- raise NotImplementedError(- "Multimodal prompts must fit in a single prefill chunk; increase "- "--max-extend-tokens or shrink the prompt."- )req = CLS(input_ids=pending_req.input_ids[: cached_len + chunk_size],table_idx=table_idx,@@ -198,8 +196,10 @@ class PrefillAdder:uid=pending_req.uid,cache_handle=cache_handle,sampling_params=pending_req.sampling_params,- mm_embeds=pending_req.mm_embeds,)+ req.mm_items = pending_req.mm_items+ req.mrope_positions_full = pending_req.mrope_positions_full+ req.mrope_delta = pending_req.mrope_delta# Hybrid GDN per-request state slots (None for non-hybrid). On a fresh admit these are# freshly allocated; on a chunked continuation they are inherited from the prior chunk.req.linear_slot_idx = linear_slot_idx@@ -255,11 +255,19 @@ class PrefillManager:cache_manager: CacheManagertable_manager: TableManagerdecode_manager: DecodeManager+ encoder_cache: EncoderCache | None = Nonepending_list: List[PendingReq] = field(default_factory=list)def add_one_req(self, req: UserMsg) -> None:self.pending_list.append(- PendingReq(req.uid, req.input_ids, req.sampling_params, mm_embeds=req.mm_embeds)+ PendingReq(+ req.uid,+ req.input_ids,+ req.sampling_params,+ mm_items=req.mm_items,+ mrope_positions_full=req.mrope_positions,+ mrope_delta=req.mrope_delta,+ ))def schedule_next_batch(self, prefill_budget: int) -> Batch | None:@@ -272,6 +280,7 @@ class PrefillManager:reserved_size=self.decode_manager.inflight_tokens,cache_manager=self.cache_manager,table_manager=self.table_manager,+ encoder_cache=self.encoder_cache,)reqs: List[Req] = []chunked_list: List[PendingReq] = []@@ -296,6 +305,12 @@ class PrefillManager:prompt_admissions.append((req.uid, pending_req.input_len, req.cache_handle.cached_len))+ if pending_req.mm_items and self.encoder_cache is not None:+ # claim the rows every chunk of this request will gather; the entry outlives the chunks+ for item in pending_req.mm_items:+ self.encoder_cache.register(+ item.hash, req.uid, mm_rows_after(item, req.cache_handle.cached_len)+ )log_new_tokens += req.extend_lenif not is_continuation:log_cached_tokens += req.cache_handle.cached_lendiff --git a/python/freetoken/scheduler/scheduler.py b/python/freetoken/scheduler/scheduler.pyindex 48923e3b..28520e13 100644--- a/python/freetoken/scheduler/scheduler.py+++ b/python/freetoken/scheduler/scheduler.py@@ -1,5 +1,7 @@from __future__ import annotations+import os+from typing import TYPE_CHECKING, List, NamedTuple, NoReturn, Set, Tuple, TypeAliasimport torch@@ -30,6 +32,7 @@ from .cache import CacheManagerfrom .config import SchedulerConfigfrom .decode import DecodeManagerfrom .io import SchedulerIOMixin+from .mm import plan_mm_batchfrom .prefill import ChunkedReq, PrefillManagerfrom .status import SchedulerStatusReporterfrom .table import TableManager@@ -89,7 +92,10 @@ class Scheduler(SchedulerIOMixin):)self.decode_manager = DecodeManager(config.page_size)self.prefill_manager = PrefillManager(- self.cache_manager, self.table_manager, self.decode_manager+ self.cache_manager,+ self.table_manager,+ self.decode_manager,+ encoder_cache=self.engine.encoder_cache,)# some alias for easy access@@ -133,6 +139,7 @@ class Scheduler(SchedulerIOMixin):min(config.max_extend_tokens, _chunk_cap) if _chunk_cap else config.max_extend_tokens)self.config = config+ self._model_is_mrope = config.model_config.model_is_mropeself.status_reporter = SchedulerStatusReporter(log=logger.info_rank0,decode_log_interval=config.decode_log_interval,@@ -489,6 +496,17 @@ class Scheduler(SchedulerIOMixin):"Dropping request %d because its abort arrived before admission", msg.uid)return+ if msg.mm_items and self.engine.encoder_cache is None:+ # no encoder runtime: fail loudly instead of decoding unexpanded placeholders+ self.send_result(+ [+ ErrorReplyMsg(+ uid=msg.uid,+ error="image input is not supported by this server",+ )+ ]+ )+ returninput_len, max_seq_len = len(msg.input_ids), self.engine.max_seq_lenmax_output_len = max_seq_len - input_lenif max_output_len <= 0:@@ -534,6 +552,15 @@ class Scheduler(SchedulerIOMixin):tombstones.pop(next(iter(tombstones)))req_to_free = self.prefill_manager.abort_req(msg.uid)req_to_free = req_to_free or self.decode_manager.abort_req(msg.uid)+ if (+ req_to_free is not None+ and req_to_free.mm_items+ and self.engine.encoder_cache is not None+ ):+ # drop the aborted request's claims; entries it held alone die here+ self.engine.encoder_cache.release(+ msg.uid, [item.hash for item in req_to_free.mm_items]+ )if req_to_free is not None:# SGLang-style abort: never free resources under an in-flight forward. If the# request is in the launched-but-not-drained batch (overlap), only mark it;@@ -781,6 +808,8 @@ class Scheduler(SchedulerIOMixin):if batch.is_prefill:self._gather_multimodal(batch)batch.positions = _make_positions(batch, self.device)+ if self._model_is_mrope:+ batch.mrope_positions = _make_mrope_positions(batch, self.device)input_mapping = _make_input_tuple(batch, self.device)write_mapping = _make_write_tuple(batch, self.device)batch.out_loc = self.engine.page_table[input_mapping]@@ -819,14 +848,12 @@ class Scheduler(SchedulerIOMixin):)def _gather_multimodal(self, batch: Batch) -> None:- """Concatenate per-request vision soft tokens (in request order) for a prefill- batch so the model can scatter them at image-token positions. ``req.mm_embeds``- is kept (not cleared) so the cache manager can recognize multimodal requests and- keep them out of the shared prefix cache (image placeholders share a token id but- carry per-image content)."""- parts = [req.mm_embeds for req in batch.reqs if req.mm_embeds is not None]- if parts:- batch.mm_embeds = torch.cat(parts, dim=0)+ """Plan the chunk's encoder jobs, gather rows and scatter rows over the batch; the engine runs them before the LM forward."""+ jobs, plan, rows = plan_mm_batch(batch.padded_reqs, self.engine.encoder_cache)+ if plan:+ batch.mm_encoder_jobs = jobs+ batch.mm_gather_plan = plan+ batch.mm_rows = torch.tensor(rows, dtype=torch.int64, pin_memory=True).to(self.device, non_blocking=True)def _schedule_next_batch(self) -> ForwardInput | None:# TODO: support other policies: e.g. DECODE first@@ -874,6 +901,28 @@ class Scheduler(SchedulerIOMixin):return forward_output+def _make_mrope_positions(batch: Batch, device: torch.device) -> torch.Tensor:+ """[3, N] rope rows: an image request's prompt tokens use their precomputed columns, everything else is sequence index + per-request delta."""+ needed = sum(r.extend_len for r in batch.padded_reqs)+ host = torch.empty((3, needed), dtype=torch.int32, pin_memory=True)+ offset = 0+ for req in batch.padded_reqs:+ length = req.extend_len+ out = host[:, offset : offset + length]+ full = req.mrope_positions_full+ if full is not None and req.device_len <= full.shape[1]:+ out.copy_(full[:, req.cached_len : req.device_len])+ else:+ row = torch.arange(+ req.cached_len + req.mrope_delta,+ req.device_len + req.mrope_delta,+ dtype=torch.int32,+ )+ out.copy_(row.unsqueeze(0).expand(3, -1))+ offset += length+ return host.to(device, non_blocking=True)++def _make_positions(batch: Batch, device: torch.device) -> torch.Tensor:needed_size = sum(r.extend_len for r in batch.padded_reqs)indices_host = torch.empty(needed_size, dtype=torch.int32, pin_memory=True)diff --git a/python/freetoken/scheduler/utils.py b/python/freetoken/scheduler/utils.pyindex 0d6dd512..83b79f2a 100644--- a/python/freetoken/scheduler/utils.py+++ b/python/freetoken/scheduler/utils.py@@ -17,7 +17,9 @@ class PendingReq:input_ids: torch.Tensorsampling_params: SamplingParamschunked_req: ChunkedReq | None = None- mm_embeds: torch.Tensor | None = None+ mm_items: list | None = None+ mrope_positions_full: torch.Tensor | None = None+ mrope_delta: int = 0@propertydef input_len(self) -> int:diff --git a/python/freetoken/server/anthropic_api.py b/python/freetoken/server/anthropic_api.pyindex 18911673..e42e0805 100644--- a/python/freetoken/server/anthropic_api.py+++ b/python/freetoken/server/anthropic_api.py@@ -218,7 +218,21 @@ def convert_anthropic_prompt(# -> reasoning_content; redacted_thinking stays skipped (opaque payload).thinking_parts.append(block.thinking)elif block.type == "image":- # Text-only server: drop image blocks rather than failing the request.+ src = block.source or {}+ stype = src.get("type")+ data = src.get("data") if stype == "base64" else src.get("url")+ if not data:+ # an unsupported image source must fail the request, not degrade to a text-only answer+ raise ValueError(f"unsupported image source type: {stype!r}")+ content_parts.append(+ {+ "type": "image",+ "freetoken_ref": {+ "kind": "b64" if stype == "base64" else "url",+ "data": data,+ },+ }+ )continueelif block.type == "tool_use":tool_calls.append(@@ -347,6 +361,9 @@ def _tool_result_text(content) -> str:parts: list[str] = []for item in content:if isinstance(item, dict):+ if item.get("type") == "image":+ # chat templates render tool messages as plain text, so an image here has nowhere to go+ raise ValueError("images inside tool results are not supported")parts.append(item.get("text") or "")else:parts.append(str(item))diff --git a/python/freetoken/server/api_server.py b/python/freetoken/server/api_server.pyindex 3e2acc85..5592a2b0 100644--- a/python/freetoken/server/api_server.py+++ b/python/freetoken/server/api_server.py@@ -206,10 +206,13 @@ class FrontendManager:``_frontend_tokenizer`` field). Called from a worker thread — never on the event loop."""with self._frontend_tokenizer_lock:if self._frontend_tokenizer is None:+ from freetoken.mm.processor import get_mm_processorfrom freetoken.tokenizer.tokenize import TokenizeManagerfrom freetoken.utils import load_tokenizer- self._frontend_tokenizer = TokenizeManager(load_tokenizer(self.config.model_path))+ self._frontend_tokenizer = TokenizeManager(+ load_tokenizer(self.config.model_path), get_mm_processor(self.config.model_path, self.config.mm)+ )return self._frontend_tokenizerdef warm_frontend_tokenizer(self) -> None:diff --git a/python/freetoken/server/args.py b/python/freetoken/server/args.pyindex 32e8d126..ab2fb9b7 100644--- a/python/freetoken/server/args.py+++ b/python/freetoken/server/args.py@@ -1,11 +1,13 @@from __future__ import annotationsimport argparse+import jsonimport osfrom dataclasses import dataclassfrom typing import List, Tupleimport torch+from freetoken.mm.config import ENCODER_KINDS, MultimodalConfigfrom freetoken.distributed import DistributedInfofrom freetoken.scheduler import SchedulerConfigfrom freetoken.utils import init_logger@@ -57,6 +59,10 @@ class ServerArgs(SchedulerConfig):# prompt_tokens_details.cached_tokens, Anthropic cache_read_input_tokens, Responses# input_tokens_details.cached_tokens). Mirrors sglang's --enable-cache-report.enable_cache_report: bool = False+ # Comma-separated hostname allowlist for client-supplied image URLs; empty admits any domain.+ allowed_media_domains: str = ""+ # Directory file:// image refs may be read from; empty rejects local files.+ allowed_local_media_path: str = ""# Comma-separated CORS allow-list for browser/webview clients (e.g. the desktop# app). Empty string disables CORS headers entirely; "*" allows any origin.cors_origins: str = "tauri://localhost,http://tauri.localhost,http://localhost:1420"@@ -98,6 +104,16 @@ class ServerArgs(SchedulerConfig):return f"tcp://127.0.0.1:{self.server_port + 1}"+def _json_object(text: str) -> dict:+ try:+ value = json.loads(text)+ except json.JSONDecodeError as exc:+ raise argparse.ArgumentTypeError(f"not valid JSON: {exc}") from None+ if not isinstance(value, dict):+ raise argparse.ArgumentTypeError("expected a JSON object")+ return value++def parse_args(args: List[str],run_shell: bool = False,@@ -424,6 +440,77 @@ def parse_args("as a GDN-aware radix (cross-request GDN-state prefix reuse); pass 'naive' to opt out.",)+ parser.add_argument(+ "--text-model-only",+ action="store_true",+ default=False,+ help="Serve a multimodal checkpoint text-only: no encoder tower is built (its VRAM goes to "+ "the KV/expert pools) and every multimodal input is rejected. Same as --mm-disable with "+ "every encoder kind.",+ )+ parser.add_argument(+ "--mm-disable",+ nargs="+",+ choices=list(ENCODER_KINDS),+ default=[],+ metavar="{vision,audio}",+ help="Encoder towers to leave unbuilt; every input they would serve is rejected.",+ )++ parser.add_argument(+ "--image-min-tokens",+ type=_positive_int,+ default=MultimodalConfig.image_min_tokens,+ help="Fewest tokens an image may take: the image processor scales smaller images up to it, "+ "in the family's own units. Default: the processor's own limit.",+ )+ parser.add_argument(+ "--image-max-tokens",+ type=_positive_int,+ default=MultimodalConfig.image_max_tokens,+ help="Most tokens an image may take: the image processor scales larger images down to it, "+ "in the family's own units (Qwen VL: one token per 32x32 pixels). Default: the processor's own limit.",+ )+ parser.add_argument(+ "--mm-processor-kwargs",+ type=_json_object,+ default=None,+ metavar="JSON",+ help="JSON object of extra keyword arguments for the checkpoint's image processor call, "+ "for family-specific knobs; applied after the token budget.",+ )++ parser.add_argument(+ "--mm-embed-cache-device",+ choices=["cpu", "cuda"],+ default=MultimodalConfig.embed_cache_device,+ help="Storage for encoded image embeddings between prefill chunks.",+ )++ parser.add_argument(+ "--mm-encoder-weights",+ choices=["gpu", "host"],+ default=MultimodalConfig.encoder_weights,+ help="Encoder tower block weights: pinned host banks streamed two blocks at a time behind the "+ "compute (default, about 60 MiB of VRAM instead of the whole tower), or resident on the GPU.",+ )++ parser.add_argument(+ "--allowed-media-domains",+ type=str,+ default=ServerArgs.allowed_media_domains,+ help="Comma-separated hostname allowlist for client-supplied image URLs. "+ "Empty (default) allows any domain.",+ )++ parser.add_argument(+ "--allowed-local-media-path",+ type=str,+ default=ServerArgs.allowed_local_media_path,+ help="Directory that file:// image refs may be read from. "+ "Unset (default) rejects local files.",+ )+parser.add_argument("--enable-cache-report",action="store_true",@@ -724,6 +811,13 @@ def parse_args(if kwargs["model_path"].startswith("~"):kwargs["model_path"] = os.path.expanduser(kwargs["model_path"])+ # a bad media root is a deployment mistake; fail at startup, not per request+ if kwargs["allowed_local_media_path"]:+ media_root = os.path.realpath(os.path.expanduser(kwargs["allowed_local_media_path"]))+ if not os.path.isdir(media_root):+ parser.error(f"--allowed-local-media-path {media_root} is not a directory")+ kwargs["allowed_local_media_path"] = media_root+if kwargs["served_model_name"] is None:kwargs["served_model_name"] = (os.path.basename(os.path.normpath(kwargs["model_path"])) or kwargs["model_path"]@@ -785,6 +879,19 @@ def parse_args(kwargs["tp_info"] = DistributedInfo(0, kwargs["tensor_parallel_size"])del kwargs["tensor_parallel_size"]+ disabled = set(ENCODER_KINDS) if kwargs.pop("text_model_only") else set()+ disabled.update(kwargs.pop("mm_disable"))+ image_min_tokens, image_max_tokens = kwargs.pop("image_min_tokens"), kwargs.pop("image_max_tokens")+ if image_min_tokens is not None and image_max_tokens is not None and image_min_tokens > image_max_tokens:+ parser.error(f"--image-min-tokens {image_min_tokens} exceeds --image-max-tokens {image_max_tokens}")+ kwargs["mm"] = MultimodalConfig(+ disabled_encoders=frozenset(disabled),+ embed_cache_device=kwargs.pop("mm_embed_cache_device"),+ encoder_weights=kwargs.pop("mm_encoder_weights"),+ image_min_tokens=image_min_tokens,+ image_max_tokens=image_max_tokens,+ processor_kwargs=kwargs.pop("mm_processor_kwargs") or {},+ )result = ServerArgs(**kwargs)logger.info(f"Parsed arguments:\n{result}")return result, run_shelldiff --git a/python/freetoken/server/generation.py b/python/freetoken/server/generation.pyindex 28f85d87..91056e8d 100644--- a/python/freetoken/server/generation.py+++ b/python/freetoken/server/generation.py@@ -23,6 +23,7 @@ from typing import Anyfrom . import request_ringfrom freetoken.core import SamplingParamsfrom freetoken.message import TokenizeMsg+from freetoken.mm.media import collect_image_refs, fetch_image_bytes, image_reject_reasonfrom freetoken.tokenizer.tokenize import resolve_thinking_modetry:@@ -42,12 +43,14 @@ from .reasoning_parser import ()-class GenerationError(Exception):+class GenerationError(ValueError):"""A request failed before producing output (surfaced via ``UserReply.error`` — e.g. achat template the tokenizer cannot render, or a prompt that exceeds the KV budget). Eachadapter turns this into its own wire-level error instead of hanging on a reply that neverarrives. ``code`` carries the stable class (``UserReply.error_code``) when the failure has- one, so an adapter can emit it as OpenAI's error ``code`` and clients need not parse prose."""+ one, so an adapter can emit it as OpenAI's error ``code`` and clients need not parse prose.++ Subclasses ValueError so every adapter's invalid-request handler maps it to a 400."""def __init__(self, message: str, code: str | None = None):super().__init__(message)@@ -231,15 +234,41 @@ def _render_message(message: dict[str, Any]) -> dict[str, Any]:return m-def _flatten_text_parts(parts: list[Any]) -> str:- texts: list[str] = []+def _flatten_text_parts(parts: list[Any]) -> str | list[dict[str, Any]]:+ """Plain string for text-only parts; a template-ready part list with {"type": "image"} entries (source kept under freetoken_ref) when images are present."""+ out: list[dict[str, Any]] = []+ has_image = Falsefor part in parts:ptype = part.get("type") if isinstance(part, dict) else Noneif ptype == "text":- texts.append((part.get("text") if isinstance(part, dict) else None) or "")+ out.append({"type": "text", "text": part.get("text") or ""})+ elif ptype in ("image_url", "input_image"):+ url = part.get("image_url")+ if isinstance(url, dict):+ url = url.get("url")+ url = url or part.get("url")+ if not url:+ raise ValueError("image content part carries no url")+ out.append({"type": "image", "freetoken_ref": {"kind": "url", "data": url}})+ has_image = True+ elif ptype == "image" and isinstance(part.get("freetoken_ref"), dict):+ out.append(part)+ has_image = Trueelse:- raise ValueError(f"Unsupported content part type for text-only server: {ptype}")- return "".join(texts)+ raise ValueError(f"Unsupported content part type: {ptype}")+ if not has_image:+ return "".join(p["text"] for p in out)+ return out+++async def _resolve_images(refs: list[dict[str, Any]], state: Any) -> list[bytes]:+ """Gate and fetch a request's images; every failure is a GenerationError."""+ if reason := image_reject_reason(state.config):+ raise GenerationError(reason)+ try:+ return await fetch_image_bytes(refs, state.config)+ except ValueError as exc:+ raise GenerationError(str(exc)) from excdef split_tool_lists(@@ -262,6 +291,8 @@ def split_tool_lists(async def submit_generation(spec: GenSpec, state: Any) -> int:"""Enqueue one generation from a GenSpec; return its uid. Every protocol adaptercalls this — it takes the neutral spec, not a wire request type."""+ refs = collect_image_refs(spec.messages)+ images = await _resolve_images(refs, state) if refs else Noneuid = state.new_user()await state.send_one(TokenizeMsg(@@ -270,6 +301,7 @@ async def submit_generation(spec: GenSpec, state: Any) -> int:sampling_params=spec.sampling_params,chat_template_kwargs=spec.chat_template_kwargs,tools=spec.template_tools,+ images=images,))return uid@@ -293,19 +325,22 @@ async def count_prompt_tokens(class the generation path maps to a 400. A tokenizer *initialization* failure (missingtemplate, load error) propagates as its original exception, a server fault. Load + tokenizerun in a worker thread so the event loop is never blocked."""+ refs = collect_image_refs(messages)+ images = await _resolve_images(refs, state) if refs else Nonemsg = TokenizeMsg(uid=0,text=messages,sampling_params=SamplingParams(),chat_template_kwargs=chat_template_kwargs,tools=tools,+ images=images,)manager = await asyncio.to_thread(state.frontend_tokenizer) # init failure -> server faulttry:- input_ids = (await asyncio.to_thread(manager.tokenize, [msg]))[0]+ (user_msg,) = await asyncio.to_thread(manager.tokenize, [msg])except _TemplateError as exc:raise GenerationError(str(exc)) from exc- return int(input_ids.numel())+ return int(user_msg.input_ids.numel())async def prerender_error(spec: GenSpec, state: Any) -> GenerationError | None:diff --git a/python/freetoken/server/launch.py b/python/freetoken/server/launch.pyindex acef5517..bb0c256f 100644--- a/python/freetoken/server/launch.py+++ b/python/freetoken/server/launch.py@@ -183,6 +183,7 @@ def launch_server("backend_addr": server_args.zmq_backend_addr,"frontend_addr": server_args.zmq_frontend_addr,"local_bs": 1,+ "mm": server_args.mm,"create": server_args.tokenizer_create_addr,"tokenizer_id": num_tokenizers,"ack_queue": ack_queue,@@ -202,6 +203,7 @@ def launch_server("backend_addr": server_args.zmq_backend_addr,"frontend_addr": server_args.zmq_frontend_addr,"local_bs": 1,+ "mm": server_args.mm,"create": server_args.tokenizer_create_addr,"tokenizer_id": i,"ack_queue": ack_queue,diff --git a/python/freetoken/server/openai_api.py b/python/freetoken/server/openai_api.pyindex a76898ab..dc2f73a9 100644--- a/python/freetoken/server/openai_api.py+++ b/python/freetoken/server/openai_api.py@@ -196,7 +196,10 @@ async def handle_chat_completion(if err is not None:return create_error_response(str(err), code=err.code)- uid = await submit_generation(spec, state)+ try:+ uid = await submit_generation(spec, state)+ except GenerationError as exc:+ return create_error_response(str(exc), code=exc.code)if req.stream:chunks = stream_chat_completion_chunks(uid, req, state, spec)diff --git a/python/freetoken/server/responses_api.py b/python/freetoken/server/responses_api.pyindex 60a1442f..fc1cde9e 100644--- a/python/freetoken/server/responses_api.py+++ b/python/freetoken/server/responses_api.py@@ -157,6 +157,8 @@ async def handle_responses(reasoning_parser=getattr(state.config, "reasoning_parser", None),)uid = await submit_generation(spec, state)+ except GenerationError as exc:+ return _error_response(400, str(exc), exc.code)except ValueError as exc:return _error_response(400, str(exc))@@ -260,7 +262,7 @@ def _convert_input_item(item: dict[str, Any]) -> list[dict[str, Any]]:role = item.get("role", "user")if role == "developer":role = "system"- return [{"role": role, "content": _input_text(item.get("content"))}]+ return [{"role": role, "content": _input_content(item.get("content"))}]if itype == "function_call":return [{@@ -328,6 +330,29 @@ def _merge_assistant_run(messages: list[dict[str, Any]]) -> list[dict[str, Any]]return merged+def _input_content(content: Any) -> str | list[dict[str, Any]]:+ """Like _input_text, but keeps input_image parts as template-ready image parts."""+ if not isinstance(content, list):+ return _input_text(content)+ parts: list[dict[str, Any]] = []+ has_image = False+ for part in content:+ if isinstance(part, dict) and part.get("type") == "input_image":+ url = part.get("image_url") or part.get("url")+ if isinstance(url, dict):+ url = url.get("url")+ if not url:+ # an input_image without a url (e.g. a file_id) is not servable; fail rather than answer text-only+ raise ValueError("input_image without image_url is not supported")+ parts.append({"type": "image", "freetoken_ref": {"kind": "url", "data": url}})+ has_image = True+ continue+ parts.append({"type": "text", "text": _input_text([part])})+ if not has_image:+ return "".join(p["text"] for p in parts)+ return parts++def _input_text(content: Any) -> str:if content is None:return ""diff --git a/python/freetoken/server/stats.py b/python/freetoken/server/stats.pyindex 76c6c308..1857d3d7 100644--- a/python/freetoken/server/stats.py+++ b/python/freetoken/server/stats.py@@ -102,7 +102,7 @@ class StatsTracker:def derive_model_card(config: Any) -> dict:- """attn enum + moe bool + ctx from the model config."""+ """attn enum + moe bool + ctx from the model config; input_modalities is what the API accepts right now."""mc = config.model_configif getattr(mc, "has_linear_attention", False):attn = "hybrid_linear"@@ -115,6 +115,7 @@ def derive_model_card(config: Any) -> dict:"ctx": config.max_seq_len,"attn": attn,"moe": bool(getattr(mc, "is_moe", False)),+ "input_modalities": ["text", *sorted(config.served_modalities)],}diff --git a/python/freetoken/tokenizer/server.py b/python/freetoken/tokenizer/server.pyindex 530e862d..d3365bfe 100644--- a/python/freetoken/tokenizer/server.py+++ b/python/freetoken/tokenizer/server.py@@ -24,6 +24,7 @@ from freetoken.message import (UserMsg,UserReply,)+from freetoken.mm.config import MultimodalConfigfrom freetoken.utils import (ZmqPullQueue,ZmqPushQueue,@@ -84,18 +85,17 @@ def _tokenize_requests(tokenize_manager: Any,messages: List[TokenizeMsg],logger: Any,-) -> tuple[List[TokenizeMsg], List[torch.Tensor], List[UserReply]]:+) -> tuple[List[UserMsg], List[UserReply]]:"""Tokenize independently, returning backend work plus terminal frontend errors.Successful tokenization deliberately emits no prompt-token reply: accounting startsonly when the scheduler later confirms first-prefill admission."""- ok_msgs: List[TokenizeMsg] = []- ok_tensors: List[torch.Tensor] = []+ backend: List[UserMsg] = []errors: List[UserReply] = []for msg in messages:try:- tokens = tokenize_manager.tokenize([msg])[0]+ user_msg = tokenize_manager.tokenize([msg])[0]except Exception as exc: # noqa: BLE001 — isolate, never crash the workerlogger.warning(f"tokenization failed for request {msg.uid}: {exc!r}")errors.append(@@ -109,7 +109,7 @@ def _tokenize_requests(continue# A zero-token prompt would trip the scheduler's input_len > 0 invariant and# crash the worker; reject it here as a terminal error instead.- if tokens.numel() == 0:+ if user_msg.input_ids.numel() == 0:errors.append(UserReply(uid=msg.uid,@@ -119,9 +119,8 @@ def _tokenize_requests())continue- ok_msgs.append(msg)- ok_tensors.append(tokens)- return ok_msgs, ok_tensors, errors+ backend.append(user_msg)+ return backend, errors@torch.inference_mode()@@ -136,6 +135,7 @@ def tokenize_worker(tokenizer_id: int = -1,model_source: str = "huggingface",ack_queue: mp.Queue[str] | None = None,+ mm: MultimodalConfig | None = None,) -> None:send_backend = ZmqPushQueue(backend_addr, create=False, encoder=BaseBackendMsg.encoder)send_frontend = ZmqPushQueue(frontend_addr, create=False, encoder=BaseFrontendMsg.encoder)@@ -144,10 +144,12 @@ def tokenize_worker(tokenizer = load_tokenizer(tokenizer_path)logger = init_logger(__name__, f"tokenizer_{tokenizer_id}")+ from freetoken.mm.processor import get_mm_processor+from .detokenize import DetokenizeManagerfrom .tokenize import TokenizeManager- tokenize_manager = TokenizeManager(tokenizer)+ tokenize_manager = TokenizeManager(tokenizer, get_mm_processor(tokenizer_path, mm))detokenize_manager = DetokenizeManager(tokenizer, load_eos_token_ids(tokenizer_path, tokenizer))@@ -245,18 +247,12 @@ def tokenize_worker(# Tokenize per-message so a single un-renderable request (e.g. a chat template# that rejects the message layout) becomes a terminal error reply for THAT uid# instead of an uncaught exception that kills the worker and bricks the server.- ok_msgs, ok_tensors, errors = _tokenize_requests(- tokenize_manager, tokenize_msg, logger- )+ backend, errors = _tokenize_requests(tokenize_manager, tokenize_msg, logger)if errors:send_frontend.put(errors[0] if len(errors) == 1 else BatchFrontendMsg(data=errors))- if ok_msgs:- backend = [- UserMsg(uid=msg.uid, input_ids=t, sampling_params=msg.sampling_params)- for msg, t in zip(ok_msgs, ok_tensors, strict=True)- ]+ if backend:send_backend.put(backend[0] if len(backend) == 1 else BatchBackendMsg(data=backend))if len(abort_msg) > 0:batch_output = BatchBackendMsg(diff --git a/python/freetoken/tokenizer/tokenize.py b/python/freetoken/tokenizer/tokenize.pyindex 0636b342..758ffd4d 100644--- a/python/freetoken/tokenizer/tokenize.py+++ b/python/freetoken/tokenizer/tokenize.py@@ -5,13 +5,16 @@ import jsonimport osimport threadingfrom types import ModuleType-from typing import Any, List+from typing import TYPE_CHECKING, Any, Listimport torch-from freetoken.message import TokenizeMsg+from freetoken.message import TokenizeMsg, UserMsgfrom freetoken.utils import init_loggerfrom transformers import PreTrainedTokenizerBase+if TYPE_CHECKING:+ from freetoken.mm.processor import MMProcessor+from .effort import (EffortProfile,ThinkingProfile,@@ -47,16 +50,17 @@ _EFFORT_PROBE_MESSAGES = [{"role": "user", "content": "ping"}]class TokenizeManager:- def __init__(self, tokenizer: PreTrainedTokenizerBase) -> None:+ def __init__(self, tokenizer: PreTrainedTokenizerBase, mm_processor: MMProcessor | None = None) -> None:self.tokenizer = tokenizer+ self.mm_processor = mm_processor # None: the model takes no imagesself._dsv4_encoder = _load_dsv4_encoder_if_needed(tokenizer)self._effort_profile: EffortProfile | None = Noneself._thinking_profile: ThinkingProfile | None = Noneself._effort_lock = threading.Lock()self._logged_effort_maps: set[tuple[Any, str | None]] = set()- def tokenize(self, msgs: List[TokenizeMsg]) -> List[torch.Tensor]:- results: List[torch.Tensor] = []+ def tokenize(self, msgs: List[TokenizeMsg]) -> List[UserMsg]:+ results: List[UserMsg] = []# TODO: batch tokenizationfor msg in msgs:prompt = self.render_prompt(msg)@@ -71,7 +75,25 @@ class TokenizeManager:prompt, return_tensors="pt", add_special_tokens=not templated))- results.append(input_ids.view(-1).to(torch.int32))+ input_ids = input_ids.view(-1).to(torch.int32)+ if msg.images:+ if self.mm_processor is None:+ raise ValueError("image input is not supported for this model")+ mm = self.mm_processor.apply(input_ids, msg.images)+ results.append(+ UserMsg(+ uid=msg.uid,+ input_ids=mm.input_ids,+ sampling_params=msg.sampling_params,+ mm_items=mm.mm_items,+ mrope_positions=mm.mrope_positions,+ mrope_delta=mm.mrope_delta,+ )+ )+ else:+ results.append(+ UserMsg(uid=msg.uid, input_ids=input_ids, sampling_params=msg.sampling_params)+ )return resultsdef render_prompt(self, msg: TokenizeMsg) -> str:diff --git a/tests/README.md b/tests/README.mdindex 83cd79f2..33e5756c 100644--- a/tests/README.md+++ b/tests/README.md@@ -48,6 +48,8 @@ checkpoint is set:| `FREETOKEN_AIME_MAX_TOKENS`| `e2e/test_aime.py` — per-sample token budget (default `16384`) || `FREETOKEN_AIME_SAMPLES` | `e2e/test_aime.py` — pass@N sample count when sampling (default `3`) || `FREETOKEN_AIME_MIN_FREE_GIB` | `e2e/test_aime.py` — required free GPU memory (default `70`) |+| `FREETOKEN_QWEN36_MODEL` | `tokenizer/test_mm_tokenize.py` — local Qwen3.6 checkpoint (image processor + tokenizer) |+| `FREETOKEN_QWEN3VL_MODEL` | `tokenizer/test_mm_tokenize.py` — local Qwen3-VL checkpoint (same expansion, DeepStack family) || `FREETOKEN_TEST_MOE_CACHE_SIZE` | `e2e/test_aime.py` — >0 switches to the offload MoE backend with this cache size || `FREETOKEN_TEST_MEM_RATIO` | `e2e/test_aime.py` — offload-mode memory_ratio (default `0.9`) || `FREETOKEN_REBUILD_TEST_MODEL` | `e2e/test_cache_rebuild.py` — a SMALL local model dir; boots a real server (falls back to `FREETOKEN_TEST_MODEL`) |diff --git a/tests/engine/test_mm_encoder.py b/tests/engine/test_mm_encoder.pynew file mode 100644index 00000000..1c12de58--- /dev/null+++ b/tests/engine/test_mm_encoder.py@@ -0,0 +1,96 @@+"""Engine._run_mm_encoder against a fake model: chunked gathers, a shared image, precomputed embeddings, orphan jobs."""++from __future__ import annotations++from types import SimpleNamespace++import pytest+import torch++from freetoken.engine.engine import Engine+from freetoken.message.backend import MMItem+from freetoken.mm.encoder_cache import EncoderCache++H = 8+++class _FakeVision:+ def __init__(self):+ self.calls = 0++ def encode(self, item: MMItem) -> torch.Tensor:+ self.calls += 1+ return torch.full((item.num_tokens, H), float(item.hash))+++def _engine(cache: EncoderCache, model=None) -> SimpleNamespace:+ return SimpleNamespace(+ encoder_cache=cache, device=torch.device("cpu"), dtype=torch.float32,+ model=model or _FakeVision(),+ )+++def _item(h: int, n_tokens: int, precomputed: torch.Tensor | None = None) -> MMItem:+ feature = None if precomputed is not None else torch.zeros(1)+ return MMItem(+ modality="image", hash=h, pad_value=0, offsets=[[0, n_tokens]],+ feature=feature, precomputed_embeddings=precomputed,+ )+++def _batch(jobs, plan) -> SimpleNamespace:+ return SimpleNamespace(mm_encoder_jobs=jobs, mm_gather_plan=plan, mm_embeds=None, mm_rows=None)+++def test_entry_lives_until_its_consumer_gathers_the_last_row():+ cache = EncoderCache(storage="cpu")+ eng = _engine(cache)+ item = _item(h=3, n_tokens=4)+ cache.register(3, 7, 4) # admission claims the whole image+ # chunk 1 consumes rows [0, 2) of a 4-token image+ Engine._run_mm_encoder(eng, _batch([item], [(7, 3, 0, 2, 4, 0)]))+ assert cache._entries[3].remaining == {7: 2}+ assert item.feature is None+ # chunk 2 finishes the image; the entry dies with its last claim+ Engine._run_mm_encoder(eng, _batch([], [(7, 3, 2, 4, 4, 0)]))+ assert not cache.has(3)+ assert eng.model.calls == 1+++def test_shared_image_is_encoded_once_and_sliced_per_request():+ cache = EncoderCache(storage="cpu")+ eng = _engine(cache)+ a, b = _item(h=5, n_tokens=4), _item(h=5, n_tokens=4)+ cache.register(5, 1, 4)+ cache.register(5, 2, 4)+ batch = _batch([a, b], [(1, 5, 0, 4, 4, 0), (2, 5, 0, 2, 4, 4)])+ Engine._run_mm_encoder(eng, batch)+ assert eng.model.calls == 1+ assert batch.mm_embeds.shape == (6, H)+ assert torch.equal(batch.mm_embeds, torch.full((6, H), 5.0))+ # request 1 consumed its image; request 2 still has rows to gather in its next chunk+ assert cache._entries[5].remaining == {2: 2}+++def test_precomputed_embeddings_bypass_the_encoder():+ cache = EncoderCache(storage="cpu")++ class _NoVision:+ def encode(self, item):+ raise AssertionError("encoder must not run for precomputed embeddings")++ eng = _engine(cache, model=_NoVision())+ emb = torch.arange(2 * H, dtype=torch.float32).view(2, H)+ item = _item(h=4, n_tokens=2, precomputed=emb)+ cache.register(4, 1, 2)+ batch = _batch([item], [(1, 4, 0, 2, 2, 0)])+ Engine._run_mm_encoder(eng, batch)+ assert torch.equal(batch.mm_embeds, emb)+ assert item.precomputed_embeddings is None+ assert not cache.has(4)+++def test_job_without_gather_row_fails_loudly():+ eng = _engine(EncoderCache(storage="cpu"))+ with pytest.raises(AssertionError, match="encoder jobs without gather rows"):+ Engine._run_mm_encoder(eng, _batch([_item(h=1, n_tokens=2)], []))diff --git a/tests/engine/test_mm_repeated_image.py b/tests/engine/test_mm_repeated_image.pynew file mode 100644index 00000000..a87b0424--- /dev/null+++ b/tests/engine/test_mm_repeated_image.py@@ -0,0 +1,48 @@+"""A request that carries the same image twice, chunked between the two occurrences (a real checkpoint)."""++from __future__ import annotations++import os+import subprocess+import sys++import pytest++CHECKPOINT = os.environ.get("FREETOKEN_QWEN3VL_MODEL", "")++pytestmark = [+ pytest.mark.slow,+ pytest.mark.skipif(not os.path.exists(os.path.join(CHECKPOINT, "config.json")), reason="FREETOKEN_QWEN3VL_MODEL not set"),+]++# the engine wants a fresh CUDA context, so each run gets its own process+_SCRIPT = """+import io, sys, torch+from PIL import Image+from freetoken.core import SamplingParams+from freetoken.llm import LLM+from freetoken.mm.config import MultimodalConfig++checkpoint, storage = sys.argv[1], sys.argv[2]+llm = LLM(model_path=checkpoint, dtype=torch.bfloat16, attention_backend="auto", max_running_req=1, cuda_graph_max_bs=1,+ max_extend_tokens=128, max_seq_len_override=4096, mm=MultimodalConfig(embed_cache_device=storage))+messages = [{"role": "user", "content": [{"type": "image"}, {"type": "image"},+ {"type": "text", "text": "What color are these two images? Answer in one word."}]}]+prompt = llm.tokenizer.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)+ids = llm.tokenizer.encode(prompt, add_special_tokens=False)+buf = io.BytesIO(); Image.new("RGB", (256, 256), (255, 0, 0)).save(buf, format="PNG"); red = buf.getvalue()+text = llm.generate([ids], SamplingParams(max_tokens=8, temperature=0.0), images=[[red, red]])[0]["text"]+print("RESULT", repr(text), llm.engine.encoder_cache.stats())+"""++[email protected]("storage", ["cpu", "cuda"])+def test_repeated_image_survives_the_chunk_boundary(storage):+ run = subprocess.run(+ [sys.executable, "-c", _SCRIPT, CHECKPOINT, storage],+ capture_output=True, text=True, timeout=900, env={**os.environ, "PYTHONPATH": os.environ.get("PYTHONPATH", "")},+ )+ assert run.returncode == 0, run.stderr[-2000:]+ result = [line for line in run.stdout.splitlines() if line.startswith("RESULT")][-1]+ assert "red" in result.lower(), result+ assert result.endswith("(0, 0)"), result # every claim gathered, nothing left behinddiff --git a/tests/kernels/test_mrope.py b/tests/kernels/test_mrope.pynew file mode 100644index 00000000..f8402f87--- /dev/null+++ b/tests/kernels/test_mrope.py@@ -0,0 +1,114 @@+"""MRotaryEmbedding correctness: text degeneracy, section layouts, kernel-vs-fallback."""++from __future__ import annotations++import pytest+import torch++cuda = pytest.mark.skipif(not torch.cuda.is_available(), reason="needs CUDA")++HEAD = 256+ROT = 64+SECTION = (11, 11, 10)++# independent expectations for SECTION under each layout (half = 32 slots)+REFERENCE_TABLES = {+ "contiguous": [0] * 11 + [1] * 11 + [2] * 10,+ "interleaved": [1 if i % 3 == 1 and i < 33 else 2 if i % 3 == 2 and i < 30 else 0 for i in range(32)],+ "interleaved_glm": [0, 1, 2] * 10 + [0, 1],+}+++def test_section_tables():+ from freetoken.layers.rotary import build_section_table++ for layout, expected in REFERENCE_TABLES.items():+ assert build_section_table(SECTION, layout).tolist() == expected, layout+ # GLM-V's [8,12,12]: T runs out first and H/W fill the tail+ assert build_section_table((8, 12, 12), "interleaved_glm").tolist() == [0, 1, 2] * 8 + [1, 1, 2, 1, 1, 2, 2, 2]+ with pytest.raises(ValueError, match="not representable"):+ build_section_table((8, 12, 12), "interleaved")+ with pytest.raises(ValueError, match="unknown mrope layout"):+ build_section_table(SECTION, "diagonal")+++def _make(mrope: bool, layout: str = "interleaved"):+ from freetoken.layers.rotary import get_rope++ # the engine builds rope layers inside a cuda device context+ with torch.device("cuda"):+ return get_rope(+ head_dim=HEAD, rotary_dim=ROT, max_position=4096, base=1e7,+ mrope_section=SECTION if mrope else None, mrope_layout=layout,+ )+++def _qk(n, heads=4, kv=2, seed=0):+ g = torch.Generator(device="cuda").manual_seed(seed)+ q = torch.randn(n, heads * HEAD, generator=g, dtype=torch.bfloat16, device="cuda")+ k = torch.randn(n, kv * HEAD, generator=g, dtype=torch.bfloat16, device="cuda")+ return q, k+++@cuda+def test_text_positions_degenerate_to_1d_rope():+ n = 64+ pos1 = torch.arange(n, dtype=torch.int32, device="cuda")+ pos3 = pos1.unsqueeze(0).expand(3, -1).contiguous()+ q1, k1 = _qk(n)+ q3, k3 = q1.clone(), k1.clone()+ _make(False).forward(pos1, q1, k1)+ _make(True).forward(pos3, q3, k3)+ # 1-D path may use flashinfer while mrope uses the triton kernel: allclose, not bitwise+ assert torch.allclose(q1.float(), q3.float(), atol=2e-2, rtol=2e-2)+ assert torch.allclose(k1.float(), k3.float(), atol=2e-2, rtol=2e-2)+++def _torch_reference(pos3, q, k, table):+ """Per-axis freqs, per-slot axis selection, NeoX rotation, fp32 math."""+ half = ROT // 2+ inv = 1.0 / (1e7 ** (torch.arange(0, ROT, 2, dtype=torch.float32, device="cuda") / ROT))+ sec = torch.tensor(table, dtype=torch.long, device="cuda")+ pos = pos3[sec, :].transpose(0, 1).float() # [n, half]+ freqs = pos * inv.unsqueeze(0) # [n, half]+ cos, sin = freqs.cos().unsqueeze(1), freqs.sin().unsqueeze(1)+ for t in (q, k):+ v = t.view(t.shape[0], -1, HEAD)+ lo, hi = v[..., :half].float(), v[..., half:ROT].float()+ v[..., :half] = (lo * cos - hi * sin).to(v.dtype)+ v[..., half:ROT] = (hi * cos + lo * sin).to(v.dtype)+++@cuda[email protected]("layout", sorted(REFERENCE_TABLES))+def test_matches_reference_semantics(layout):+ n = 97+ g = torch.Generator(device="cuda").manual_seed(7)+ pos3 = torch.randint(0, 4000, (3, n), generator=g, dtype=torch.int32, device="cuda")+ q, k = _qk(n, seed=7)+ q_ref, k_ref = q.clone(), k.clone()+ _make(True, layout).forward(pos3, q, k)+ _torch_reference(pos3, q_ref, k_ref, REFERENCE_TABLES[layout])+ assert torch.allclose(q.float(), q_ref.float(), atol=2e-2, rtol=2e-2)+ assert torch.allclose(k.float(), k_ref.float(), atol=2e-2, rtol=2e-2)+++@cuda+def test_kernel_matches_torch_fallback():+ from freetoken.kernel.triton.rope import (+ apply_mrope_torch_fallback,+ apply_mrope_with_cos_sin_cache_inplace,+ )++ rope = _make(True)+ n = 33+ g = torch.Generator(device="cuda").manual_seed(3)+ pos3 = torch.randint(0, 4000, (3, n), generator=g, dtype=torch.int32, device="cuda")+ q, k = _qk(n, seed=3)+ q2, k2 = q.clone(), k.clone()+ cache = rope._cos_sin_cache+ sec = rope._section_table.cuda()+ apply_mrope_with_cos_sin_cache_inplace(pos3, q, k, HEAD, cache, sec)+ apply_mrope_torch_fallback(pos3, q2, k2, HEAD, cache, sec)+ assert torch.allclose(q.float(), q2.float(), atol=1e-2, rtol=1e-2)+ assert torch.allclose(k.float(), k2.float(), atol=1e-2, rtol=1e-2)diff --git a/tests/kvcache/test_qsa_pool.py b/tests/kvcache/test_qsa_pool.pyindex 9aec2bb6..dea722f6 100644--- a/tests/kvcache/test_qsa_pool.py+++ b/tests/kvcache/test_qsa_pool.py@@ -71,7 +71,7 @@ def _spec(*, index_ratio=4, attn_type=AttnType.QSA, num_kv_heads=2, head_dim=64,def _config(spec, *, page_size=64, max_running_req=3):- mc = SimpleNamespace(num_layers=8, has_swa_attention=False, has_linear_attention=True)+ mc = SimpleNamespace(num_layers=8, has_swa_attention=False, has_linear_attention=True, model_is_mrope=False)mc.kv_cache_group_specs = lambda: (spec,)return SimpleNamespace(model_config=mc,@@ -201,6 +201,7 @@ def test_resolve_pool_class_and_factory():spec = _spec()mc = SimpleNamespace(+ model_is_mrope=False,num_layers=8, has_swa_attention=False, has_linear_attention=True,num_kv_heads=2, head_dim=64, dsv4_args=None,)diff --git a/tests/mm/test_encoder_cache.py b/tests/mm/test_encoder_cache.pynew file mode 100644index 00000000..ee028d4c--- /dev/null+++ b/tests/mm/test_encoder_cache.py@@ -0,0 +1,46 @@+"""EncoderCache ownership: rows claimed at admission, freed when the last claim is gathered."""++from __future__ import annotations++import torch++from freetoken.mm.encoder_cache import EncoderCache++CPU = torch.device("cpu")+++def test_entry_lives_until_every_claim_is_gathered():+ cache = EncoderCache(storage="cpu")+ emb = torch.arange(12, dtype=torch.float32).reshape(6, 2)+ cache.register(7, 1, 6)+ cache.register(7, 2, 6)+ assert not cache.has(7) # claimed, not encoded yet+ cache.put(7, emb)+ assert cache.has(7)+ assert torch.equal(cache.get_slice(7, 2, 4, CPU), emb[2:4])+ cache.consume(7, 1, 4)+ cache.consume(7, 1, 2)+ assert cache.has(7) # request 2 still has rows to gather+ cache.consume(7, 2, 6)+ assert not cache.has(7) and cache.stats() == (0, 0)+++def test_repeated_image_in_one_request_adds_up():+ cache = EncoderCache(storage="cpu")+ cache.register(7, 1, 4)+ cache.register(7, 1, 4)+ cache.put(7, torch.ones(4, 2))+ cache.consume(7, 1, 4) # the first occurrence is done+ assert cache.has(7)+ cache.consume(7, 1, 4)+ assert not cache.has(7)+++def test_release_drops_a_claim_even_before_the_tensor_arrives():+ cache = EncoderCache(storage="cpu")+ cache.register(5, 1, 3)+ cache.register(5, 2, 3)+ cache.release(1, [5, 99]) # unknown hash: no-op+ assert 5 in cache._entries+ cache.release(2, [5])+ assert cache._entries == {} and cache.stats() == (0, 0)diff --git a/tests/mm/test_processor.py b/tests/mm/test_processor.pynew file mode 100644index 00000000..87559d29--- /dev/null+++ b/tests/mm/test_processor.py@@ -0,0 +1,186 @@+"""MMProcessor: placeholder expansion on synthetic families, the Qwen VL processor and its positions, the registry (no checkpoint)."""++from __future__ import annotations++import io+from types import SimpleNamespace++import pytest+import torch++from freetoken.message import MMItem+from freetoken.mm import MM_PAD_SHIFT_VALUE+from freetoken.mm.config import MultimodalConfig+from freetoken.mm.processor import MMProcessor, PromptReplacement, image_positions+from freetoken.mm.processors.qwen_vl import QwenVLMMProcessor++PLACEHOLDER = 7+BOI, SOFT, EOI = 90, 91, 92+ARCH = "Qwen3_5ForConditionalGeneration"+++def _png():+ from PIL import Image++ buf = io.BytesIO()+ Image.new("RGB", (8, 8)).save(buf, format="PNG")+ return buf.getvalue()+++class _Family(MMProcessor):+ """Fixed-size images wrapped in begin/end tokens; only the soft tokens take embeddings."""++ placeholder = [PLACEHOLDER]++ def __init__(self, n_soft):+ super().__init__("/fake", MultimodalConfig())+ self.n_soft = n_soft++ def process(self, images):+ return [+ MMItem(modality="image", hash=100 + i, pad_value=MM_PAD_SHIFT_VALUE + 100 + i, offsets=[], feature=torch.zeros(1))+ for i, _ in enumerate(images)+ ]++ def prompt_replacement(self, item):+ return PromptReplacement.select_token_id([BOI] + [SOFT] * self.n_soft + [EOI], SOFT)++ def dummy_items(self, dtype, device):+ raise NotImplementedError+++class _Tiled(_Family):+ """Rows of soft tokens separated by a break token, like a tiled image."""++ def prompt_replacement(self, item):+ full = ([SOFT] * 2 + [BOI]) * 3+ return PromptReplacement.select_token_id(full, SOFT)+++def test_replacement_keeps_wrapper_tokens_and_pads_only_embedding_slots():+ ids = torch.tensor([1, 2, PLACEHOLDER, 3, PLACEHOLDER, 4], dtype=torch.int32)+ r = _Family(n_soft=3).apply(ids, [_png(), _png()])+ a, b = r.mm_items+ assert r.input_ids.tolist() == [1, 2, BOI, a.pad_value, a.pad_value, a.pad_value, EOI, 3, BOI, b.pad_value, b.pad_value, b.pad_value, EOI, 4]+ assert a.offsets == [[3, 6]] and b.offsets == [[9, 12]]+ assert a.num_tokens == 3 and r.mrope_positions is None and r.mrope_delta == 0+++def test_replacement_with_several_embedding_runs_yields_several_spans():+ ids = torch.tensor([PLACEHOLDER, 5], dtype=torch.int32)+ r = _Tiled(n_soft=0).apply(ids, [_png()])+ (item,) = r.mm_items+ p = item.pad_value+ assert r.input_ids.tolist() == [p, p, BOI, p, p, BOI, p, p, BOI, 5]+ assert item.offsets == [[0, 2], [3, 5], [6, 8]] and item.num_tokens == 6+++def test_placeholder_count_is_checked_before_any_image_is_decoded():+ ids = torch.tensor([PLACEHOLDER, PLACEHOLDER], dtype=torch.int32)+ with pytest.raises(ValueError, match="2 image placeholders but the request carries 1"):+ _Family(n_soft=1).apply(ids, [b"not an image"])+++def test_image_positions_freeze_t_spread_hw_and_advance_by_the_longer_side():+ # [text x2][img 2x3 = 6][text x1][img 3x1 = 3][text x2] -> 14 tokens+ pos, delta = image_positions(14, [(2, 6, 2, 3), (9, 3, 3, 1)])+ assert pos[:, :2].tolist() == [[0, 1]] * 3+ assert pos[0, 2:8].tolist() == [2] * 6+ assert pos[1, 2:8].tolist() == [2, 2, 2, 3, 3, 3]+ assert pos[2, 2:8].tolist() == [2, 3, 4, 2, 3, 4]+ assert pos[:, 8].tolist() == [5, 5, 5] # 2 + max(2, 3)+ assert pos[0, 9:12].tolist() == [6] * 3+ assert pos[1, 9:12].tolist() == [6, 7, 8]+ assert pos[2, 9:12].tolist() == [6, 6, 6]+ assert pos[:, 12:].tolist() == [[9, 10]] * 3 # 6 + max(3, 1)+ assert delta == 10 + 1 - 14+++def _hf_config(vision=True, mrope=True, arch=ARCH):+ rope = {"rope_theta": 1e7}+ if mrope:+ rope.update(mrope_section=[24, 20, 20], mrope_interleaved=True)+ return SimpleNamespace(+ architectures=[arch],+ image_token_id=151655,+ vision_config=(+ SimpleNamespace(spatial_merge_size=2, patch_size=16, temporal_patch_size=2, in_channels=3)+ if vision+ else None+ ),+ text_config=SimpleNamespace(rope_parameters=rope, vocab_size=248320),+ )+++def _grid_item(h, w, offsets):+ return MMItem(+ modality="image", hash=1, pad_value=1, offsets=offsets,+ feature=torch.zeros(1), model_specific_data={"grid_thw": [1, h, w]},+ )+++def test_qwen_processor_reads_config():+ proc = QwenVLMMProcessor(_hf_config(), "/nonexistent", MultimodalConfig())+ assert proc.image_token_id == 151655 and proc.is_mrope and proc.merge == 2+ assert proc.image_grid(_grid_item(8, 6, [[0, 12]])) == (4, 3)+ repl = proc.prompt_replacement(_grid_item(8, 6, []))+ assert proc.placeholder == [151655] and repl.full == [151655] * 12 and repl.embed_spans() == [[0, 12]]+ (dummy,) = proc.dummy_items(torch.bfloat16, torch.device("cpu"))+ dummy.validate()+ assert dummy.feature.shape == (4, 3 * 2 * 16 * 16) and dummy.num_tokens == 1+ assert dummy.grid_thw == [1, 2, 2]+++def test_qwen_token_budget_and_kwargs_reach_the_image_processor():+ from freetoken.mm.config import MultimodalConfig++ calls = []++ class _FakeImageProcessor:+ size = {"shortest_edge": 65536, "longest_edge": 16777216}++ def __call__(self, images, **kwargs):+ calls.append(kwargs)+ return {"pixel_values": torch.zeros(4, 3 * 2 * 16 * 16), "image_grid_thw": torch.tensor([[1, 2, 2]])}++ mm = MultimodalConfig(image_min_tokens=100, image_max_tokens=1000, processor_kwargs={"do_convert_rgb": False})+ proc = QwenVLMMProcessor(_hf_config(), "/nonexistent", mm)+ proc._image_processor = lambda: _FakeImageProcessor()+ (item,) = proc.process([object()])+ assert item.grid_thw == [1, 2, 2]+ # one token is a 32x32 patch of the resized image; the budget travels as pixel areas in size, the extra kwarg rides along+ assert calls == [{"return_tensors": "pt", "size": {"shortest_edge": 100 * 1024, "longest_edge": 1000 * 1024}, "do_convert_rgb": False}]+ proc = QwenVLMMProcessor(_hf_config(), "/nonexistent", MultimodalConfig())+ proc._image_processor = lambda: _FakeImageProcessor()+ proc.process([object()])+ assert calls[-1] == {"return_tensors": "pt"} # no budget, no size override+++def test_qwen_processor_positions_follow_the_grid():+ proc = QwenVLMMProcessor(_hf_config(), "/nonexistent", MultimodalConfig())+ # 3 text tokens, one 8x6-patch image (4x3 = 12 llm tokens), 2 text tokens+ pos, delta = proc.positions(17, [_grid_item(8, 6, [[3, 15]])])+ assert pos.shape == (3, 17)+ assert pos[:, :3].tolist() == [[0, 1, 2]] * 3+ assert pos[0, 3:15].tolist() == [3] * 12+ assert pos[1, 3:15].tolist() == [3 + i // 3 for i in range(12)]+ assert pos[2, 3:15].tolist() == [3 + i % 3 for i in range(12)]+ assert pos[:, 15:].tolist() == [[7, 8]] * 3 # resumes at 3 + max(4, 3)+ assert delta == 8 + 1 - 17+++def test_registry_resolves_by_architecture(monkeypatch):+ import freetoken.utils+ from freetoken.mm.processor import get_mm_processor++ configs = {+ "/qwen": _hf_config(),+ "/text-only": _hf_config(vision=False),+ "/unknown-vlm": _hf_config(arch="SomeOtherForConditionalGeneration"),+ }+ monkeypatch.setattr(freetoken.utils, "cached_load_hf_config", lambda p: configs[p])+ assert isinstance(get_mm_processor("/qwen"), QwenVLMMProcessor)+ assert get_mm_processor("/qwen", MultimodalConfig(disabled_encoders=frozenset({"vision"}))) is None # --mm-disable vision+ assert get_mm_processor("/text-only") is None+ assert get_mm_processor("/unknown-vlm") is None+ assert get_mm_processor("/missing") is None # a config that fails to load means no visiondiff --git a/tests/models/qwen4_exp/common.py b/tests/models/qwen4_exp/common.pyindex aa6f2399..0a52fb0e 100644--- a/tests/models/qwen4_exp/common.py+++ b/tests/models/qwen4_exp/common.py@@ -231,6 +231,8 @@ class Fixture:is_prefill=phase == "prefill",is_decode=phase == "decode",positions=positions,+ get_attn_positions=lambda: positions,+ mm_embeds=None,out_loc=out_loc,attn_metadata=None,active_table_idx=torch.tensor(@@ -353,12 +355,14 @@ def meta_state_dict(model_path: str) -> dict[str, torch.Tensor]:from freetoken.engine.config import EngineConfigfrom freetoken.engine.engine import _decode_targetfrom freetoken.layers import rotary+ from freetoken.mm.config import ENCODER_KINDS, MultimodalConfigfrom freetoken.models import create_modelfrom freetoken.utils.torch_utils import torch_dtypeif try_get_tp_info() is None:set_tp_info(rank=0, size=1)- config = EngineConfig(model_path=model_path, tp_info=try_get_tp_info(), dtype=torch.bfloat16, moe_strategy="offload")+ config = EngineConfig(model_path=model_path, tp_info=try_get_tp_info(), dtype=torch.bfloat16, moe_strategy="offload",+ mm=MultimodalConfig(disabled_encoders=frozenset(ENCODER_KINDS)))object.__setattr__(config.model_config, "moe_strategy", "offload")object.__setattr__(config.model_config, "decode_target", _decode_target(config))saved = rotary._ROPE_DEVICEdiff --git a/tests/models/qwen4_exp/test_config.py b/tests/models/qwen4_exp/test_config.pyindex b81e02f4..be532eec 100644--- a/tests/models/qwen4_exp/test_config.py+++ b/tests/models/qwen4_exp/test_config.py@@ -5,7 +5,10 @@ from types import SimpleNamespaceimport pytestfrom freetoken.attention import AttnType-from freetoken.models.config import FullAttentionGroupConfig, LinearGatedDeltaGroupConfig+from freetoken.models.config import (+ FullAttentionGroupConfig,+ LinearGatedDeltaGroupConfig,+)from freetoken.models.qwen4_exp.config import parse_configfrom .common import LOVEDHEART_NVFP4_FP8, NVIDIA_NVFP4, QWEN_FP8, RADIXARK_NVFP4@@ -157,6 +160,28 @@ def test_eos_token_id_list_uses_the_first_entry():assert parse_config(hf).qwen4_args.ngram_boundary_token_id == base+def _vision_config():+ return SimpleNamespace(+ hidden_size=1152, depth=27, num_heads=16, intermediate_size=4304, patch_size=16, temporal_patch_size=2,+ spatial_merge_size=2, num_position_embeddings=2304, out_hidden_size=2560, in_channels=3,+ deepstack_visual_indexes=[],+ )+++def test_vision_turns_on_mrope_and_the_tower():+ hf = _hf_config()+ hf.vision_config = _vision_config()+ config = parse_config(hf)+ assert config.is_multimodal and config.model_is_mrope+ assert config.rotary_config.mrope_section == [11, 11, 10]+ assert config.rotary_config.mrope_layout == "interleaved" and config.rotary_config.rotary_dim == 64+ assert config.vision_config.out_hidden_size == 2560 and config.vision_config.deepstack_visual_indexes == ()+ assert config.qwen4_args.image_token_id == 248056+++def test_text_only_keeps_the_1d_rope():+ config = parse_config(_hf_config())+ assert not config.is_multimodal and not config.model_is_mrope and config.rotary_config.mrope_section is None# the merged-projection prefixes the model asks the QuantConfig about (attention.py / gdn.py)DENSE_PREFIXES = ("model.layers.3.self_attn.qkv_proj", "model.layers.3.self_attn.o_proj",diff --git a/tests/models/qwen4_exp/test_qsa_backend.py b/tests/models/qwen4_exp/test_qsa_backend.pyindex 1d3b944c..23661079 100644--- a/tests/models/qwen4_exp/test_qsa_backend.py+++ b/tests/models/qwen4_exp/test_qsa_backend.py@@ -174,6 +174,7 @@ def test_decode_graph_replay_matches_eager():capture_batch = SimpleNamespace(padded_reqs=[dummy] * bs, reqs=[dummy] * bs, phase="decode", size=bs, padded_size=bs,is_prefill=False, is_decode=True, positions=static["positions"],+ get_attn_positions=lambda: static["positions"],out_loc=static["out_loc"], attn_metadata=None, active_table_idx=None,)fixture.backend.prepare_for_capture(capture_batch)diff --git a/tests/models/qwen4_exp/test_skeleton.py b/tests/models/qwen4_exp/test_skeleton.pyindex fa75d6ee..ac598e65 100644--- a/tests/models/qwen4_exp/test_skeleton.py+++ b/tests/models/qwen4_exp/test_skeleton.py@@ -346,7 +346,9 @@ def test_qsa_layer_matches_hf_dense():x = (torch.randn(seq_len, config.hidden_size, device=device, dtype=dtype) * 0.5)positions = torch.arange(seq_len, device=device, dtype=torch.int64)req = SimpleNamespace(extend_len=seq_len, cached_len=0, table_idx=1)- batch = SimpleNamespace(padded_reqs=[req], reqs=[req], positions=positions)+ batch = SimpleNamespace(+ padded_reqs=[req], reqs=[req], positions=positions, get_attn_positions=lambda: positions+ )backend = TorchDenseQSAReference(config, num_slots=4, max_len=64, device=device, dtype=dtype)_fresh_ctx(attn_backend=backend)@@ -473,10 +475,11 @@ def test_decoder_stack_prefill_and_decode(monkeypatch):last = torch.tensor([sum(len(p) for p in prompts[: i + 1]) - 1 for i in range(len(prompts))], device=device)+ positions = torch.cat([torch.arange(len(p)) for p in prompts]).to(device)batch = SimpleNamespace(padded_reqs=reqs, reqs=reqs, size=len(reqs), is_prefill=True, is_decode=False,input_ids=torch.tensor(flat, dtype=torch.int64, device=device),- positions=torch.cat([torch.arange(len(p)) for p in prompts]).to(device),+ positions=positions, get_attn_positions=lambda: positions, mm_embeds=None,attn_metadata=SimpleNamespace(get_last_indices=lambda bs: last[:bs]),)with ctx.forward_batch(batch):@@ -488,10 +491,11 @@ def test_decoder_stack_prefill_and_decode(monkeypatch):r.cached_len = len(p)r.extend_len = 1r.input_ids = torch.cat([r.input_ids, torch.tensor([14], dtype=torch.int64)])+ decode_positions = torch.tensor([len(p) for p in prompts], dtype=torch.int64, device=device)decode = SimpleNamespace(padded_reqs=reqs, reqs=reqs, size=len(reqs), is_prefill=False, is_decode=True,input_ids=torch.tensor([14] * len(reqs), dtype=torch.int64, device=device),- positions=torch.tensor([len(p) for p in prompts], dtype=torch.int64, device=device),+ positions=decode_positions, get_attn_positions=lambda: decode_positions, mm_embeds=None,attn_metadata=None,)with ctx.forward_batch(decode):diff --git a/tests/models/qwen4_exp/test_weight.py b/tests/models/qwen4_exp/test_weight.pyindex 1ddbf425..1a9a100e 100644--- a/tests/models/qwen4_exp/test_weight.py+++ b/tests/models/qwen4_exp/test_weight.py@@ -193,12 +193,12 @@ def _write_checkpoint(folder, raw: dict[str, torch.Tensor], quantization_config)return str(folder), {**raw, **table}-def _load(folder: str) -> dict[str, torch.Tensor]:+def _load(folder: str, *, vision: bool = True) -> dict[str, torch.Tensor]:install_quant_config(folder)return {name: tensor.clone()for name, tensor in iter_weights(- folder, torch.device("cpu"), include_moe_experts=True, include_non_moe=True+ folder, torch.device("cpu"), include_moe_experts=True, include_non_moe=True, include_vision=vision)}@@ -227,38 +227,13 @@ def loaded_fp8(checkpoint_fp8) -> dict[str, torch.Tensor]:return _load(checkpoint_fp8[0])-def _expected_names() -> set[str]:- names = {"model.embed_tokens.weight", "lm_head.weight"}- names |= {f"model.hyper_connection_mixer.{leaf}" for leaf in- ("hc_norm.weight", "input_mix_weight_down.weight", "input_mix_weight_up.weight")}- for layer in (0, 1):- for hc in ("attn_hyper_connection", "mlp_hyper_connection"):- names |= {f"model.layers.{layer}.{hc}.{leaf}" for leaf in (- "hc_norm.weight", "input_mix_weight_down_block_inject.weight",- "input_mix_weight_up.weight")}- names |= {f"model.layers.{layer}.mlp.{leaf}" for leaf in (- "gate.weight", "shared_expert.gate_up_proj.weight",- "shared_expert.down_proj.weight", "shared_expert_gate.weight")}- names |= {f"model.layers.0.linear_attn.{leaf}" for leaf in (- "in_proj.weight", "conv1d.weight", "A_log", "dt_bias", "norm.weight", "out_proj.weight")}- names |= {f"model.layers.0.ple.{leaf}" for leaf in (- "key_proj.weight", "value_proj.weight", "norm_key.weight", "norm_query.weight",- "norm_conv.weight", "conv1d.weight", "ple_embedding.layer_multipliers",- "ple_embedding.ngram_heads_offsets", "ple_embedding.ngram_heads_vocab_sizes")}- names |= {f"model.layers.1.self_attn.{leaf}" for leaf in (- "qkv_proj.weight", "o_proj.weight", "q_norm.weight", "k_norm.weight",- "indexer.index_qk_proj.weight", "indexer.q_layernorm.weight",- "indexer.k_layernorm.weight")}- return names---def test_key_map_is_exactly_the_model_state_dict(loaded):- assert set(loaded) == _expected_names()---def test_mtp_visual_experts_and_table_never_loaded(loaded):+def test_tower_keys_come_out_under_the_prefix_load_weight_filters(loaded):+ assert {n for n in loaded if "visual" in n} == {"visual.blocks.0.attn.qkv.weight", "visual.merger.norm.weight"}+++def test_mtp_experts_and_table_never_loaded(loaded):for name in loaded:- assert not name.startswith(("mtp.", "model.visual."))+ assert not name.startswith("mtp.")assert ".mlp.experts." not in nameassert "ngram_embedding" not in nameassert not name.endswith((".weight_scale", ".weight_scale_2", ".input_scale", ".weight_scale_inv"))@@ -486,7 +461,7 @@ FP8_MODULES = (def test_emitted_keys_are_the_model_state_dict(fixture, request):"""The reader fills exactly the buffers the engine builds from the same config, block-fp8 ones with the stored dtypes."""folder, _raw = request.getfixturevalue(fixture)- loaded, state = _load(folder), meta_state_dict(folder)+ loaded, state = _load(folder, vision=False), meta_state_dict(folder)assert set(loaded) == set(state)if fixture != "checkpoint_fp8":assert loaded["model.layers.0.linear_attn.in_proj.weight"].dtype is torch.bfloat16diff --git a/tests/models/test_qwen3_5_moe_weight.py b/tests/models/test_qwen3_5_moe_weight.pyindex 1efec609..f5dc2c3c 100644--- a/tests/models/test_qwen3_5_moe_weight.py+++ b/tests/models/test_qwen3_5_moe_weight.py@@ -278,9 +278,9 @@ def _install(folder: str) -> None:set_quant_config(checkpoint_quant_config(folder, hf, get_model_spec(hf.architectures[0])))-def _load(folder: str, *, experts: bool = False) -> dict[str, torch.Tensor]:+def _load(folder: str, *, experts: bool = False, vision: bool = True) -> dict[str, torch.Tensor]:_install(folder)- return {n: t.clone() for n, t in iter_weights(folder, torch.device("cpu"), include_moe_experts=experts, include_non_moe=True)}+ return {n: t.clone() for n, t in iter_weights(folder, torch.device("cpu"), include_moe_experts=experts, include_non_moe=True, include_vision=vision)}def _meta_state_dict(folder: str) -> dict[str, torch.Tensor]:@@ -288,11 +288,13 @@ def _meta_state_dict(folder: str) -> dict[str, torch.Tensor]:from freetoken.engine.config import EngineConfigfrom freetoken.engine.engine import _decode_targetfrom freetoken.layers import rotary+ from freetoken.mm.config import ENCODER_KINDS, MultimodalConfigfrom freetoken.models import create_modelfrom freetoken.utils.torch_utils import torch_dtypestrategy = "offload" if cached_load_hf_config(folder).architectures[0].startswith("Qwen3_5Moe") else "auto"- config = EngineConfig(model_path=folder, tp_info=try_get_tp_info(), dtype=torch.bfloat16, moe_strategy=strategy)+ config = EngineConfig(model_path=folder, tp_info=try_get_tp_info(), dtype=torch.bfloat16, moe_strategy=strategy,+ mm=MultimodalConfig(disabled_encoders=frozenset(ENCODER_KINDS)))object.__setattr__(config.model_config, "moe_strategy", strategy)object.__setattr__(config.model_config, "decode_target", _decode_target(config))saved = rotary._ROPE_DEVICE@@ -319,14 +321,14 @@ def checkpoint(request, tmp_path_factory):def test_emitted_keys_are_the_model_state_dict(checkpoint):"""Every layout fills exactly the buffers the engine builds from the same config, with the buffers' shapes and (for the weights) dtypes."""_name, folder, _raw = checkpoint- loaded, state = _load(folder), _meta_state_dict(folder)+ loaded, state = _load(folder, vision=False), _meta_state_dict(folder)assert set(loaded) == set(state)for key, tensor in loaded.items():assert tensor.shape == state[key].shape, keyif key.endswith(".weight"):assert tensor.dtype is state[key].dtype, keyassert not any(k.endswith((".input_global_scale", ".weight_scale_2", ".weight_global_scale", ".k_scale")) for k in loaded)- assert not any(".mlp.experts." in k or k.startswith(("mtp.", "model.visual.")) for k in loaded)+ assert not any(".mlp.experts." in k or k.startswith("mtp.") for k in loaded)def test_expert_quant_tag_follows_the_config(checkpoint):diff --git a/tests/models/test_qwen3_vl.py b/tests/models/test_qwen3_vl.pynew file mode 100644index 00000000..c45a9f15--- /dev/null+++ b/tests/models/test_qwen3_vl.py@@ -0,0 +1,153 @@+"""Qwen VL on the CPU side: parse_config, the engine's encoder decision, the registry invariants, the text-side image scatter (no checkpoint)."""++from __future__ import annotations++from types import SimpleNamespace++import pytest+import torch++from freetoken.distributed import DistributedInfo+from freetoken.engine.config import EngineConfig+from freetoken.mm import MM_PAD_SHIFT_VALUE+from freetoken.mm.config import ENCODER_KINDS, MultimodalConfig+from freetoken.models.blocks import embed_input_ids+from freetoken.models.qwen3_vl import deepstack_add, parse_config++ROPE = {"rope_theta": 5_000_000, "rope_type": "default", "mrope_section": [24, 20, 20], "mrope_interleaved": True}+++def _hf_config(num_experts=0, arch="Qwen3VLForConditionalGeneration"):+ text = SimpleNamespace(+ hidden_size=64, num_hidden_layers=4, num_attention_heads=4, num_key_value_heads=2, head_dim=16,+ intermediate_size=128, rms_norm_eps=1e-6, hidden_act="silu", vocab_size=1000,+ max_position_embeddings=4096, rope_parameters=ROPE, rope_scaling=ROPE, model_type="qwen3_vl_text",+ num_experts=num_experts, num_experts_per_tok=2, moe_intermediate_size=32, norm_topk_prob=True,+ )+ vision = SimpleNamespace(+ hidden_size=32, depth=3, num_heads=4, intermediate_size=64, patch_size=16, temporal_patch_size=2,+ spatial_merge_size=2, num_position_embeddings=16, out_hidden_size=64, in_channels=3,+ deepstack_visual_indexes=[0, 1],+ )+ return SimpleNamespace(+ text_config=text, vision_config=vision, image_token_id=151655, tie_word_embeddings=False,+ model_type="qwen3_vl", architectures=[arch],+ )+++def _engine_config(monkeypatch, hf, processor, **overrides):+ import freetoken.engine.config as engine_config+ import freetoken.mm.processor as mm_processor++ monkeypatch.setattr(engine_config, "cached_load_hf_config", lambda path: hf)+ monkeypatch.setattr(engine_config, "checkpoint_quant_config", lambda *args: None)+ monkeypatch.setattr(mm_processor, "get_mm_processor", lambda path, mm=None: processor)+ return EngineConfig(+ model_path="/fake", tp_info=DistributedInfo(rank=0, size=1), dtype=torch.bfloat16, **overrides+ )+++def test_vision_carries_mrope_and_deepstack():+ c = parse_config(_hf_config())+ assert c.is_multimodal and c.model_is_mrope+ assert c.rotary_config.mrope_section == [24, 20, 20] and c.rotary_config.mrope_layout == "interleaved"+ assert c.rotary_config.base == 5_000_000 and c.rotary_config.scaling is None+ assert c.vision_config.deepstack_visual_indexes == (0, 1) and c.image_token_id == 151655+ assert c.num_layers == 4 and c.num_experts == 0 and c.architectures == ["Qwen3VLForConditionalGeneration"]+ assert [g.attn_type.name for g in c.kv_cache_group_specs()] == ["FULL"]+++def test_no_vision_section_is_a_plain_qwen3():+ hf = _hf_config()+ hf.vision_config = None+ c = parse_config(hf)+ assert not c.is_multimodal and not c.model_is_mrope and c.rotary_config.mrope_section is None+++def test_engine_builds_the_tower_for_a_registered_family(monkeypatch):+ c = _engine_config(monkeypatch, _hf_config(), object())+ assert [e.kind for e in c.active_encoders] == ["vision"] and c.served_modalities == {"image"}+ assert c.model_config.is_multimodal and c.model_config.model_is_mrope+++def test_text_model_only_hands_the_parser_a_config_without_vision(monkeypatch):+ hf = _hf_config()+ c = _engine_config(monkeypatch, hf, object(), mm=MultimodalConfig(disabled_encoders=frozenset(ENCODER_KINDS)))+ assert not c.served_modalities+ assert not c.model_config.is_multimodal and not c.model_config.model_is_mrope+ assert hf.vision_config is not None # the cached checkpoint config is left alone++[email protected]("kinds, served", [({"vision"}, set()), ({"audio"}, {"image"})])+def test_mm_disable_drops_only_the_named_tower(monkeypatch, kinds, served):+ c = _engine_config(monkeypatch, _hf_config(), object(), mm=MultimodalConfig(disabled_encoders=frozenset(kinds)))+ assert c.served_modalities == served and not c.mm.text_model_only+ assert c.model_config.is_multimodal == bool(served) and c.model_config.model_is_mrope == bool(served)+++def test_registered_encoders_come_with_a_processor_and_the_model_hooks():+ from freetoken.models.blocks import SupportsMultimodal+ from freetoken.models.register import _MODEL_REGISTRY, _load_attr++ for arch, spec in _MODEL_REGISTRY.items():+ assert (spec.mm_processor is None) == (not spec.encoders), arch+ for e in spec.encoders:+ assert e.kind in ENCODER_KINDS and e.modalities, arch+ if spec.encoders:+ assert issubclass(_load_attr(spec.module, spec.model_cls), SupportsMultimodal), arch+++def test_a_family_without_registered_encoders_is_served_text_only(monkeypatch):+ import freetoken.engine.config as engine_config+ from dataclasses import replace+ from freetoken.models.register import get_model_spec++ monkeypatch.setattr(engine_config, "get_model_spec", lambda arch: replace(get_model_spec(arch), mm_processor=None, encoders=()))+ c = _engine_config(monkeypatch, _hf_config(), None)+ assert not c.active_encoders and not c.served_modalities and not c.model_config.is_multimodal+++def test_tower_roots_make_the_fp8_ignore_list_match():+ from freetoken.layers.quantization import NameMap, QuantConfig+ from freetoken.models.register import get_model_spec++ spec = get_model_spec("Qwen3VLForConditionalGeneration")+ # quantized Qwen VL exports list every tower module under its stored name and keep it bf16+ hf = SimpleNamespace(+ architectures=["Qwen3VLForConditionalGeneration"],+ quantization_config={+ "quant_method": "fp8",+ "weight_block_size": [128, 128],+ "modules_to_not_convert": [+ "lm_head",+ "model.visual.blocks.0.attn.qkv",+ "model.visual.blocks.0.attn.proj",+ "model.visual.blocks.0.mlp.linear_fc1",+ "model.visual.blocks.0.mlp.linear_fc2",+ "model.visual.merger.linear_fc1",+ "model.visual.merger.linear_fc2",+ ],+ },+ )+ with_roots = QuantConfig.from_hf(hf, name_map=NameMap(roots=spec.checkpoint_roots, packed=spec.packed_modules_mapping))+ assert with_roots.scheme_for("model.layers.0.self_attn.qkv_proj") is not None+ assert with_roots.scheme_for("visual.blocks.0.attn.qkv") is None+ assert with_roots.scheme_for("visual.merger.linear_fc2") is None+ # without the visual -> model.visual root the tower would be built fp8 against bf16 tensors+ without = QuantConfig.from_hf(hf, name_map=NameMap(roots=(), packed=spec.packed_modules_mapping))+ assert without.scheme_for("visual.blocks.0.attn.qkv") is not None+++def test_image_rows_take_the_leading_columns_and_deepstack_adds_the_next_block():+ H = 4+ table = torch.arange(10 * H, dtype=torch.float32).view(10, H)+ embed = SimpleNamespace(forward=lambda ids: table[ids], num_embeddings=10)+ assert torch.equal(embed_input_ids(embed, torch.tensor([1, 2]), SimpleNamespace(mm_embeds=None)), table[[1, 2]])+ ids = torch.tensor([1, MM_PAD_SHIFT_VALUE + 7, 2, MM_PAD_SHIFT_VALUE + 7])+ mm = torch.arange(2 * 3 * H, dtype=torch.float32).view(2, 3 * H)+ rows = torch.tensor([1, 3])+ x = embed_input_ids(embed, ids, SimpleNamespace(mm_embeds=mm, mm_rows=rows))+ assert torch.equal(x[0], table[1]) and torch.equal(x[2], table[2])+ assert torch.equal(x[1], mm[0, :H]) and torch.equal(x[3], mm[1, :H])+ deepstack_add(x, rows, mm, level=1, hidden_size=H)+ assert torch.equal(x[1], mm[0, :H] + mm[0, 2 * H : 3 * H]) and torch.equal(x[0], table[1])diff --git a/tests/models/test_qwen3_vl_vision.py b/tests/models/test_qwen3_vl_vision.pynew file mode 100644index 00000000..f8a18d3d--- /dev/null+++ b/tests/models/test_qwen3_vl_vision.py@@ -0,0 +1,72 @@+"""Qwen3VLVisionModel DeepStack side outputs on a tiny random tower."""++from __future__ import annotations++import pytest+import torch++from freetoken.distributed import set_tp_info, try_get_tp_info+from freetoken.models.qwen3_vl import Qwen3VLVisionModel, VisionConfig++pytestmark = pytest.mark.skipif(not torch.cuda.is_available(), reason="needs CUDA")++[email protected](autouse=True)+def _single_rank():+ # the tower is built from the same TP-aware layers as the text tower+ if try_get_tp_info() is None:+ set_tp_info(rank=0, size=1)+++def _tower(deepstack):+ vc = VisionConfig(+ hidden_size=64, depth=3, num_heads=4, intermediate_size=128, patch_size=16, temporal_patch_size=2,+ spatial_merge_size=2, num_position_embeddings=16, out_hidden_size=32, in_channels=3,+ deepstack_visual_indexes=deepstack,+ )+ torch.set_default_dtype(torch.bfloat16)+ try:+ with torch.device("cuda"):+ tower = Qwen3VLVisionModel(vc)+ finally:+ torch.set_default_dtype(torch.float32)+ for p in tower.state_dict().values():+ p.normal_(0, 0.02)+ return tower, vc+++def test_deepstack_levels_ride_as_extra_columns():+ tower, vc = _tower((0, 1))+ feature = torch.zeros(16, 3 * 2 * 16 * 16, dtype=torch.bfloat16, device="cuda")+ out = tower.forward(feature, [[1, 4, 4]])+ assert out.shape == (4, vc.out_hidden_size * 3)+ keys = tower.state_dict()+ # the DeepStack mergers normalize the merged vector, the final merger each patch+ assert keys["deepstack_merger_list.0.norm.weight"].shape == (64 * 4,)+ assert keys["merger.norm.weight"].shape == (64,)+ assert "deepstack_merger_list.1.linear_fc2.weight" in keys and "deepstack_merger_list.2.norm.weight" not in keys++[email protected]("deepstack", [(), (0, 1)])+def test_forward_matches_the_naive_forward(deepstack):+ tower, vc = _tower(deepstack)+ feature = torch.randn(64, 3 * 2 * 16 * 16, dtype=torch.bfloat16, device="cuda")+ out = tower.forward(feature, [[1, 8, 8]])+ assert out.shape == (16, vc.out_hidden_size * (1 + len(deepstack)))+ assert torch.equal(out, tower.forward_naive(feature, [[1, 8, 8]]))+++def test_host_streamed_weights_compute_the_same_output():+ tower, vc = _tower((1,))+ feature = torch.randn(64, 3 * 2 * 16 * 16, dtype=torch.bfloat16, device="cuda")+ resident = tower.forward(feature, [[1, 8, 8]])+ keys = tower.state_dict()+ tower.place_weights("host")+ assert tower.state_dict().keys() == keys.keys()+ assert tower.state_dict()["blocks.0.attn.qkv.weight"].device.type == "cpu"+ assert tower._streamer.device_bytes < sum(t.numel() * 2 for k, t in keys.items() if k.startswith("blocks."))+ for _ in range(2): # a second forward reuses the staging buffers behind the release events+ assert torch.equal(tower.forward(feature, [[1, 8, 8]]), resident)+ tower.place_weights("gpu")+ assert tower.state_dict()["blocks.0.attn.qkv.weight"].device.type == "cuda"+ assert torch.equal(tower.forward(feature, [[1, 8, 8]]), resident)diff --git a/tests/scheduler/test_abort_inflight_prefill.py b/tests/scheduler/test_abort_inflight_prefill.pyindex a50d0d32..85c21785 100644--- a/tests/scheduler/test_abort_inflight_prefill.py+++ b/tests/scheduler/test_abort_inflight_prefill.py@@ -79,7 +79,7 @@ def _launch_req(pool, cm, tm, prompt, *, cls=Req, track_seqlen=None):"""A launched (forward in flight) hybrid req: handle locked, pages allocated,GDN slots held, cached_len advanced -- the state _process_last_data will drain."""mr = cm.match_req(SimpleNamespace(input_ids=prompt, input_len=len(prompt),- mm_embeds=None))+ ))req = cls(input_ids=prompt, table_idx=tm.allocate(), cached_len=0, output_len=4,uid=UID, sampling_params=SamplingParams(max_tokens=4),cache_handle=mr.cuda_handle)diff --git a/tests/scheduler/test_commit_repoints_page_table.py b/tests/scheduler/test_commit_repoints_page_table.pyindex 9e770b06..50db29dc 100644--- a/tests/scheduler/test_commit_repoints_page_table.py+++ b/tests/scheduler/test_commit_repoints_page_table.py@@ -18,7 +18,7 @@ PROMPT = [1, 2, 3, 4, 5, 6, 7, 8]def _pend(ids):t = torch.tensor(ids, dtype=torch.int32)- return SimpleNamespace(input_ids=t, input_len=len(ids), mm_embeds=None)+ return SimpleNamespace(input_ids=t, input_len=len(ids))def _admit(cm, page_table, table_idx, ids, handle):diff --git a/tests/scheduler/test_cost_accounting_core.py b/tests/scheduler/test_cost_accounting_core.pyindex 8942c4ca..045010c6 100644--- a/tests/scheduler/test_cost_accounting_core.py+++ b/tests/scheduler/test_cost_accounting_core.py@@ -51,27 +51,28 @@ def _tokenize_msg(uid: int) -> TokenizeMsg:def test_successful_tokenization_does_not_account_prompt_before_admission():class Tokenizer:def tokenize(self, messages):- return [torch.tensor([10, 11, 12], dtype=torch.int32)]+ (msg,) = messages+ return [UserMsg(uid=msg.uid, input_ids=torch.tensor([10, 11, 12], dtype=torch.int32), sampling_params=msg.sampling_params)]- ok, tensors, errors = _tokenize_requests(Tokenizer(), [_tokenize_msg(1)], _Logger())- assert [msg.uid for msg in ok] == [1]- assert tensors[0].tolist() == [10, 11, 12]+ backend, errors = _tokenize_requests(Tokenizer(), [_tokenize_msg(1)], _Logger())+ assert [msg.uid for msg in backend] == [1]+ assert backend[0].input_ids.tolist() == [10, 11, 12]assert errors == [] # in particular, no early prompt_tokens_delta UserReplydef test_tokenization_failure_and_empty_prompt_are_terminal_without_usage():class Tokenizer:def tokenize(self, messages):- uid = messages[0].uid- if uid == 2:+ (msg,) = messages+ if msg.uid == 2:raise ValueError("bad template")- return [torch.empty(0, dtype=torch.int32)]+ return [UserMsg(uid=msg.uid, input_ids=torch.empty(0, dtype=torch.int32), sampling_params=msg.sampling_params)]logger = _Logger()- ok, tensors, errors = _tokenize_requests(+ backend, errors = _tokenize_requests(Tokenizer(), [_tokenize_msg(2), _tokenize_msg(3)], logger)- assert ok == [] and tensors == []+ assert backend == []assert [reply.uid for reply in errors] == [2, 3]assert all(reply.finished and reply.prompt_tokens_delta == 0 for reply in errors)assert "could not encode request" in errors[0].errordiff --git a/tests/scheduler/test_hybrid_cache_manager.py b/tests/scheduler/test_hybrid_cache_manager.pyindex 080999ad..098f2766 100644--- a/tests/scheduler/test_hybrid_cache_manager.py+++ b/tests/scheduler/test_hybrid_cache_manager.py@@ -25,7 +25,7 @@ def _pool(num_slots=16):def _pend(ids):# int32 to match production Req.input_ids dtype (fast_compare_key needs consistent dtype)t = torch.tensor(ids, dtype=torch.int32)- return SimpleNamespace(input_ids=t, input_len=len(ids), mm_embeds=None)+ return SimpleNamespace(input_ids=t, input_len=len(ids))def test_hybrid_cache_manager_donate_then_hit():diff --git a/tests/scheduler/test_mm.py b/tests/scheduler/test_mm.pynew file mode 100644index 00000000..1910adf2--- /dev/null+++ b/tests/scheduler/test_mm.py@@ -0,0 +1,92 @@+"""Chunk-window planning over image spans: rows, jobs, cache hits, multi-span items, a repeated image across a cut."""++from __future__ import annotations++import torch++from freetoken.message import MMItem+from freetoken.mm.encoder_cache import EncoderCache+from freetoken.scheduler.mm import mm_rows_after, plan_mm_batch, plan_mm_chunk++CPU = torch.device("cpu")+++def _item(h, offsets):+ return MMItem(modality="image", hash=h, pad_value=h, offsets=offsets, feature=torch.zeros(1))+++class _Cache:+ def __init__(self, *hashes):+ self._h = set(hashes)++ def has(self, h):+ return h in self._h+++def test_single_span_split_across_two_chunks():+ item = _item(7, [[10, 20]])+ jobs, plan = plan_mm_chunk(1, [item], 0, 16, None)+ assert jobs == [item] and plan == [(1, 7, 0, 6, 10, 10)]+ jobs, plan = plan_mm_chunk(1, [item], 16, 32, None)+ assert jobs == [item] and plan == [(1, 7, 6, 10, 10, 0)] # row_hi == num_tokens: last rows+++def test_cached_prefix_and_cache_hit_produce_no_job():+ item = _item(7, [[10, 20]])+ assert plan_mm_chunk(1, [item], 20, 40, None) == ([], []) # span entirely in the cached prefix+ jobs, plan = plan_mm_chunk(1, [item], 0, 40, _Cache(7))+ assert jobs == [] and plan == [(1, 7, 0, 10, 10, 10)] # gather only+++def test_multi_span_item_maps_to_consecutive_rows():+ item = _item(9, [[2, 5], [8, 12]]) # 3 + 4 rows+ jobs, plan = plan_mm_chunk(3, [item], 0, 10, None)+ assert jobs == [item] # one job per item even when two spans overlap the window+ assert plan == [(3, 9, 0, 3, 7, 2), (3, 9, 3, 5, 7, 8)]+ jobs, plan = plan_mm_chunk(3, [item], 10, 20, None)+ assert jobs == [item] and plan == [(3, 9, 5, 7, 7, 0)]+++def test_rows_after_the_cached_prefix():+ item = _item(1, [[2, 5], [8, 12]])+ assert mm_rows_after(item, 0) == 7+ assert mm_rows_after(item, 4) == 1 + 4 # the hit ends inside the first span+ assert mm_rows_after(item, 12) == 0+++def test_repeated_image_survives_a_chunk_cut_between_its_occurrences():+ cache = EncoderCache(storage="cpu")+ items = [_item(7, [[0, 4]]), _item(7, [[5, 9]])]+ for item in items:+ cache.register(item.hash, 1, mm_rows_after(item, 0))+ encoded = []++ def run_chunk(lo, hi):+ jobs, plan = plan_mm_chunk(1, items, lo, hi, cache)+ for item in jobs:+ if not cache.has(item.hash):+ assert item.feature is not None+ encoded.append(item.hash)+ cache.put(item.hash, torch.ones(4, 8))+ item.feature = None+ for uid, h, row_lo, row_hi, _, _ in plan:+ cache.get_slice(h, row_lo, row_hi, CPU)+ cache.consume(h, uid, row_hi - row_lo)+ return jobs++ assert len(run_chunk(0, 6)) == 1 # both occurrences overlap the window: one job+ assert cache.has(7) # the second occurrence still has rows to gather+ assert run_chunk(6, 10) == [] # nothing to encode, both features already gone+ assert encoded == [7] and not cache.has(7)+++def test_batch_rows_follow_the_reqs_in_batch_order():+ from types import SimpleNamespace++ # req 1: 6 tokens, image on [2, 5); req 2: chunk [4, 10) of a prompt whose image spans [3, 8)+ a = SimpleNamespace(uid=1, mm_items=[_item(7, [[2, 5]])], cached_len=0, device_len=6, extend_len=6)+ b = SimpleNamespace(uid=2, mm_items=[_item(8, [[3, 8]])], cached_len=4, device_len=10, extend_len=6)+ jobs, plan, rows = plan_mm_batch([a, b], None)+ assert [j.hash for j in jobs] == [7, 8]+ assert plan == [(1, 7, 0, 3, 3, 2), (2, 8, 1, 5, 5, 0)]+ assert rows == [2, 3, 4, 6, 7, 8, 9] # req 2 starts at batch row 6; its image rows 1..5 land on its first four tokensdiff --git a/tests/scheduler/test_swa_pagesize.py b/tests/scheduler/test_swa_pagesize.pyindex 74c2982d..3f55c5e8 100644--- a/tests/scheduler/test_swa_pagesize.py+++ b/tests/scheduler/test_swa_pagesize.py@@ -135,7 +135,7 @@ def test_hybrid_chunk_donate_skips_unaligned_boundary(ps):req = _req(3 * ps + 3, 1)h = cm.match_req(SimpleNamespace(input_ids=req.input_ids, input_len=req.input_len,- mm_embeds=None)).cuda_handle+ )).cuda_handlereq.cache_handle = hreq.cached_len = h.cached_lencm.lock(h)diff --git a/tests/server/test_anthropic_api.py b/tests/server/test_anthropic_api.pyindex 8ea1d1d1..59ad45da 100644--- a/tests/server/test_anthropic_api.py+++ b/tests/server/test_anthropic_api.py@@ -17,11 +17,14 @@ import osimport sysfrom types import SimpleNamespace+import pytest+_ROOT = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))_PY = os.path.join(_ROOT, "python")if _PY not in sys.path:sys.path.insert(0, _PY)+from freetoken.message import UserMsg # noqa: E402from freetoken.message.frontend import UserReply # noqa: E402from freetoken.server import anthropic_api as A # noqa: E402from freetoken.server.anthropic_models import AnthropicMessagesRequest # noqa: E402@@ -421,10 +424,12 @@ class FakeState:self._outputs = outputsself.maintenance_state = "serving"self.config = SimpleNamespace(+ mm=SimpleNamespace(text_model_only=False, disabled_encoders=frozenset()),reasoning_parser=None,tool_call_parser="llama3",served_model_name="test-model",model_path="/test",+ served_modalities=frozenset(),)self._uid = 0self._count_manager = None@@ -605,23 +610,20 @@ def test_convert_native_thinking_toggle():# --------------------------------------------------------------------------- ## /v1/messages/count_tokens (route + neutral count_prompt_tokens primitive)# --------------------------------------------------------------------------- #-class _FakeIds:- def __init__(self, n):- self._n = n-- def numel(self):- return self._n--class _FakeTokenizeManager:- """Counts tokens as len(str(prompt)) so tests are deterministic without a model."""+ """Counts tokens as len(str(prompt)) so tests are deterministic without a model; returns the wire message like the real manager."""def __init__(self):self.msgs = []def tokenize(self, msgs):+ import torch+self.msgs.extend(msgs)- return [_FakeIds(len(str(m.text))) for m in msgs]+ return [+ UserMsg(uid=m.uid, input_ids=torch.zeros(len(str(m.text)), dtype=torch.int32), sampling_params=m.sampling_params)+ for m in msgs+ ]def _count_client(manager=None, maintenance_state="serving"):@@ -811,10 +813,10 @@ def test_frontend_tokenizer_concurrent_first_build_dedupes():orig_load, orig_tm = _utils.load_tokenizer, _tok.TokenizeManager_utils.load_tokenizer = _slow_load- _tok.TokenizeManager = lambda tok: tok+ _tok.TokenizeManager = lambda tok, mm=None: toktry:fm = FrontendManager(- config=SimpleNamespace(model_path="dedupe-test"),+ config=SimpleNamespace(model_path="dedupe-test", mm=None),send_tokenizer=None,recv_tokenizer=None,)@@ -865,3 +867,49 @@ def test_tool_result_carries_the_wire_tool_use_id():results = [m for m in spec.messages if m["role"] == "tool"]assert [(m["tool_call_id"], m["content"]) for m in results] == [("toolu_b", "60F"), ("toolu_a", "72F")]+++def test_count_tokens_reads_the_real_manager_return_contract():+ import torch+ from freetoken.tokenizer.tokenize import TokenizeManager++ class _Tokenizer:+ def apply_chat_template(self, messages, **kwargs):+ return "rendered"++ def encode(self, prompt, return_tensors=None, add_special_tokens=True):+ return torch.tensor([[7, 8, 9, 10, 11]], dtype=torch.long)++ client, _ = _count_client(TokenizeManager(_Tokenizer()))+ r = client.post(+ "/v1/messages/count_tokens",+ json={"model": "claude-x", "messages": [{"role": "user", "content": "hi"}]},+ )+ assert r.status_code == 200, r.text+ assert r.json() == {"input_tokens": 5}+++def test_tool_result_image_is_rejected_not_dropped():+ body = {+ "model": "claude-x",+ "max_tokens": 16,+ "messages": [+ {"role": "user", "content": "take a screenshot"},+ {"role": "assistant", "content": [{"type": "tool_use", "id": "toolu_1", "name": "shot", "input": {}}]},+ {+ "role": "user",+ "content": [+ {+ "type": "tool_result",+ "tool_use_id": "toolu_1",+ "content": [{"type": "image", "source": {"type": "base64", "media_type": "image/png", "data": "aGk="}}],+ }+ ],+ },+ ],+ }+ with pytest.raises(ValueError, match="tool results"):+ A.convert_anthropic_to_genspec(AnthropicMessagesRequest.model_validate(body), {})+ r = _client(FakeState([])).post("/v1/messages", json=body)+ assert r.status_code == 400, r.text+ assert "tool results" in r.json()["error"]["message"]diff --git a/tests/server/test_effort_dialect.py b/tests/server/test_effort_dialect.pyindex 3b04ec54..6a68a4ec 100644--- a/tests/server/test_effort_dialect.py+++ b/tests/server/test_effort_dialect.py@@ -31,6 +31,7 @@ def run(coro):class FakeState:def __init__(self, reasoning_parser: str | None = None) -> None:self.config = SimpleNamespace(+ mm=SimpleNamespace(text_model_only=False, disabled_encoders=frozenset()),model_path="/models/unit-model",served_model_name="unit-model",tool_call_parser="llama3",diff --git a/tests/server/test_generation_accounting.py b/tests/server/test_generation_accounting.pyindex 204c1f8c..8276163b 100644--- a/tests/server/test_generation_accounting.py+++ b/tests/server/test_generation_accounting.py@@ -46,6 +46,7 @@ class FakeState:def __init__(self, replies: list[UserReply]) -> None:self.config = SimpleNamespace(+ mm=SimpleNamespace(text_model_only=False, disabled_encoders=frozenset()),model_path="/m",served_model_name="unit-model",tool_call_parser="llama3",diff --git a/tests/server/test_message_wire.py b/tests/server/test_message_wire.pyindex 3bd7cc6b..e2286671 100644--- a/tests/server/test_message_wire.py+++ b/tests/server/test_message_wire.py@@ -7,9 +7,13 @@ wire with its fields intact; these pin the ones carrying state a later consumerfrom __future__ import annotations+import pytest+import torch+from freetoken.message import (BaseBackendMsg,DetokenizeMsg,+ MMItem,BaseFrontendMsg,BaseTokenizerMsg,CacheRebuildBackendMsg,@@ -18,6 +22,7 @@ from freetoken.message import (CacheRebuildResultMsg,PromptAdmittedMsg,TokenizeMsg,+ UserMsg,UserReply,)from freetoken.core import SamplingParams@@ -122,3 +127,78 @@ def test_client_dicts_with_the_wire_tag_key_survive_intact():assert isinstance(out, TokenizeMsg)assert out.chat_template_kwargs == payloadassert out.tools[0]["function"]["parameters"] == payload+++def test_tensor_wire_nd_and_dtype_roundtrip():+ import numpy as np+ import torch+ from freetoken.message.utils import deserialize_type, serialize_type++ cases = [+ torch.arange(6, dtype=torch.int32),+ torch.arange(12, dtype=torch.int64).reshape(3, 4),+ torch.randn(2, 3, 5, dtype=torch.float32),+ torch.randn(4, 1536, dtype=torch.bfloat16),+ torch.randn(3, 4)[:, :2], # non-contiguous input+ torch.tensor(7, dtype=torch.int32), # 0-d+ ]+ for t in cases:+ out = deserialize_type({}, serialize_type(t))+ assert out.dtype == t.dtype+ assert out.shape == t.shape+ assert torch.equal(out, t)++ # legacy payloads carry no shape and decode as 1-D+ legacy = {+ "__type__": "Tensor",+ "buffer": np.arange(4, dtype=np.int32).tobytes(),+ "dtype": "torch.int32",+ }+ out = deserialize_type({}, legacy)+ assert out.shape == (4,) and out.dtype == torch.int32++ # 1-D payloads still omit the shape field (old decoders can read them)+ assert "shape" not in serialize_type(torch.arange(3, dtype=torch.int32))+++def test_user_msg_with_mm_items_survives_the_wire():+ item = MMItem(+ modality="image",+ hash=0x1234ABCD,+ pad_value=1_000_000 + 0x1234ABCD,+ offsets=[[7, 11]],+ feature=torch.randn(16, 1536, dtype=torch.bfloat16),+ model_specific_data={"grid_thw": [1, 4, 4]},+ )+ msg = UserMsg(+ uid=9,+ input_ids=torch.arange(16, dtype=torch.int32),+ sampling_params=SamplingParams(),+ mm_items=[item],+ mrope_positions=torch.zeros(3, 16, dtype=torch.int32),+ mrope_delta=-3,+ )+ out = BaseBackendMsg.decoder(msg.encoder())+ assert isinstance(out, UserMsg)+ got = out.mm_items[0]+ assert (got.hash, got.pad_value, got.offsets) == (0x1234ABCD, 1_000_000 + 0x1234ABCD, [[7, 11]])+ assert got.num_tokens == 4 and got.grid_thw == [1, 4, 4] # model_specific_data reads as attributes+ assert got.precomputed_embeddings is None+ assert torch.equal(got.feature, item.feature)+ assert out.mrope_positions.shape == (3, 16) and out.mrope_delta == -3+++def test_mm_item_shape_rules():+ t = torch.zeros(2, 4)+ item = MMItem(modality="image", hash=1, pad_value=1, offsets=[[3, 5], [9, 12]], feature=t)+ item.validate()+ assert item.num_tokens == 5+ assert getattr(item, "grid_thw", None) is None # missing model-specific key -> AttributeError path+ with pytest.raises(ValueError, match="half-open"):+ MMItem(modality="image", hash=1, pad_value=1, offsets=[[5, 5]], feature=t).validate()+ with pytest.raises(ValueError, match="exactly one"):+ MMItem(modality="image", hash=1, pad_value=1, offsets=[[0, 2]]).validate()+ with pytest.raises(ValueError, match="exactly one"):+ MMItem(+ modality="image", hash=1, pad_value=1, offsets=[[0, 2]], feature=t, precomputed_embeddings=t+ ).validate()diff --git a/tests/server/test_mm_input.py b/tests/server/test_mm_input.pynew file mode 100644index 00000000..19d9f865--- /dev/null+++ b/tests/server/test_mm_input.py@@ -0,0 +1,142 @@+"""Image input plumbing: content-part rendering, ref collection, fetch, and the gate.++No GPU and no model checkpoint: everything here is pure frontend logic."""++from __future__ import annotations++import asyncio+import base64+from types import SimpleNamespace++import pytest++from freetoken.mm.media import collect_image_refs, fetch_image_bytes, image_reject_reason+from freetoken.server.generation import GenerationError, render_messages+from freetoken.server.stats import derive_model_card++PNG = base64.b64encode(b"fakepng").decode()+++def _config(**overrides):+ fields = dict(+ model_path="/nonexistent", allowed_media_domains="", allowed_local_media_path="",+ served_model_name="unit-model", max_seq_len=8192, model_config=SimpleNamespace(),+ )+ text_model_only = overrides.pop("text_model_only", False)+ serves_images = overrides.pop("vision_enabled", False)+ mm = SimpleNamespace(+ text_model_only=text_model_only,+ disabled_encoders=frozenset({"vision", "audio"}) if text_model_only else frozenset(),+ )+ return SimpleNamespace(mm=mm, served_modalities=frozenset({"image"}) if serves_images else frozenset(), **{**fields, **overrides})+++def test_image_url_part_becomes_template_image_part():+ msgs = render_messages(+ [{"role": "user", "content": [+ {"type": "text", "text": "hi"},+ {"type": "image_url", "image_url": {"url": "https://x/y.png"}},+ ]}]+ )+ content = msgs[0]["content"]+ assert isinstance(content, list)+ assert content[1]["type"] == "image"+ refs = collect_image_refs(msgs)+ assert refs == [{"kind": "url", "data": "https://x/y.png"}]+ # refs are popped; the template-facing part stays+ assert msgs[0]["content"][1] == {"type": "image"}+ assert collect_image_refs(msgs) == []+++def test_collect_refs_preserves_prompt_order():+ msgs = render_messages(+ [+ {"role": "user", "content": [{"type": "image_url", "image_url": {"url": "u1"}}]},+ {"role": "user", "content": [+ {"type": "image_url", "image_url": {"url": "u2"}},+ {"type": "image_url", "image_url": {"url": "u3"}},+ ]},+ ]+ )+ assert [r["data"] for r in collect_image_refs(msgs)] == ["u1", "u2", "u3"]+++def test_fetch_decodes_data_uri_and_raw_b64():+ refs = [+ {"kind": "url", "data": f"data:image/png;base64,{PNG}"},+ {"kind": "b64", "data": PNG},+ ]+ assert asyncio.run(fetch_image_bytes(refs, _config())) == [b"fakepng", b"fakepng"]+++def test_fetch_failures_surface_as_generation_error(monkeypatch):+ from freetoken.server import generation as gen++ # bypass the capability gate; the fetch failure itself must become a GenerationError+ monkeypatch.setattr(gen, "image_reject_reason", lambda config: None)+ state = SimpleNamespace(config=_config())+ with pytest.raises(GenerationError):+ asyncio.run(gen._resolve_images([{"kind": "url", "data": "ftp://nope"}], state))+++def test_image_gate_reasons():+ assert "text-model-only" in image_reject_reason(_config(text_model_only=True))+ assert "vision" in image_reject_reason(_config(model_path="/nonexistent"))+++def test_stats_model_card_lists_the_accepted_input_modalities():+ assert derive_model_card(_config())["input_modalities"] == ["text"]+ assert derive_model_card(_config(vision_enabled=True))["input_modalities"] == ["text", "image"]+++def test_media_domain_allowlist():+ from freetoken.mm.media import _check_media_domain++ config = _config(allowed_media_domains="cdn.example.com, Other.COM.")+ _check_media_domain("https://cdn.example.com/a.png", config) # allowed: no raise+ _check_media_domain("https://OTHER.com./b.png", config) # case/root-dot normalized+ with pytest.raises(ValueError, match="allowed domains"):+ _check_media_domain("https://evil.com/a.png", config)+ # empty allowlist admits any domain+ _check_media_domain("https://evil.com/a.png", _config())++ with pytest.raises(ValueError, match="allowed domains"):+ asyncio.run(+ fetch_image_bytes([{"kind": "url", "data": "https://evil.com/a.png"}], config)+ )+++def test_local_media_requires_allowlisted_root(tmp_path):+ img = tmp_path / "img.png"+ img.write_bytes(b"fakepng")+ url = f"file://{img}"++ # gate off (default): rejected+ with pytest.raises(ValueError, match="allowed-local-media-path"):+ asyncio.run(fetch_image_bytes([{"kind": "url", "data": url}], _config()))++ # gate on, file under the root: served+ config = _config(allowed_local_media_path=str(tmp_path))+ assert asyncio.run(fetch_image_bytes([{"kind": "url", "data": url}], config)) == [b"fakepng"]++ # a path outside the root is rejected even with the gate on+ with pytest.raises(ValueError, match="subpath"):+ asyncio.run(fetch_image_bytes([{"kind": "url", "data": "file:///etc/hostname"}], config))+++def test_image_token_budget_flags_land_in_the_multimodal_config():+ from unittest.mock import patch++ from freetoken.server.args import parse_args++ hf = SimpleNamespace(to_dict=lambda: {"architectures": ["Qwen3VLForConditionalGeneration"], "torch_dtype": "bfloat16"})+ with patch("freetoken.utils.cached_load_hf_config", lambda _path: hf):+ args, _ = parse_args([+ "--model", "/models/anon", "--image-min-tokens", "64", "--image-max-tokens", "1024",+ "--mm-processor-kwargs", '{"size": {"longest_edge": 4096}}',+ ])+ assert (args.mm.image_min_tokens, args.mm.image_max_tokens) == (64, 1024)+ assert args.mm.processor_kwargs == {"size": {"longest_edge": 4096}}+ assert parse_args(["--model", "/models/anon"])[0].mm.processor_kwargs == {}+ with pytest.raises(SystemExit): # argparse reports the bad pair and exits+ parse_args(["--model", "/models/anon", "--image-min-tokens", "2048", "--image-max-tokens", "1024"])diff --git a/tests/server/test_openai_api.py b/tests/server/test_openai_api.pyindex 58d15d7c..e33018e1 100644--- a/tests/server/test_openai_api.py+++ b/tests/server/test_openai_api.py@@ -30,6 +30,7 @@ class FakeState:reasoning_parser: str | None = None,) -> None:self.config = SimpleNamespace(+ mm=SimpleNamespace(text_model_only=False, disabled_encoders=frozenset()),model_path="/models/unit-model",served_model_name="unit-model",tool_call_parser=tool_call_parser,diff --git a/tests/server/test_responses_api.py b/tests/server/test_responses_api.pyindex d283de88..0da62172 100644--- a/tests/server/test_responses_api.py+++ b/tests/server/test_responses_api.py@@ -317,6 +317,7 @@ class FakeState:self._cached_tokens = cached_tokens # stamped on the first ack (admission reply)self.maintenance_state = "serving"self.config = SimpleNamespace(+ mm=SimpleNamespace(text_model_only=False, disabled_encoders=frozenset()),reasoning_parser=None, tool_call_parser="llama3",served_model_name="test-model", model_path="/test",)diff --git a/tests/tokenizer/test_mm_tokenize.py b/tests/tokenizer/test_mm_tokenize.pynew file mode 100644index 00000000..5dbf2822--- /dev/null+++ b/tests/tokenizer/test_mm_tokenize.py@@ -0,0 +1,105 @@+"""Image tokenization against real Qwen VL checkpoints (skipped when absent)."""++from __future__ import annotations++import io+import os++import pytest+import torch++from freetoken.mm import MM_PAD_SHIFT_VALUE, mm_pad_value++MODELS = [+ p+ for p in (os.environ.get("FREETOKEN_QWEN36_MODEL", ""), os.environ.get("FREETOKEN_QWEN3VL_MODEL", ""))+ if os.path.exists(os.path.join(p, "config.json"))+]++pytestmark = pytest.mark.skipif(+ not MODELS, reason="neither FREETOKEN_QWEN36_MODEL nor FREETOKEN_QWEN3VL_MODEL points at a checkpoint"+)+++def _png(w, h):+ from PIL import Image++ buf = io.BytesIO()+ Image.new("RGB", (w, h), (120, 30, 200)).save(buf, format="PNG")+ return buf.getvalue()++[email protected](scope="module", params=MODELS, ids=os.path.basename)+def manager(request):+ from freetoken.mm.processor import get_mm_processor+ from freetoken.tokenizer.tokenize import TokenizeManager+ from freetoken.utils.hf import load_tokenizer++ return TokenizeManager(load_tokenizer(request.param), get_mm_processor(request.param))+++def _msg(images, n_parts=1):+ from freetoken.core import SamplingParams+ from freetoken.message import TokenizeMsg++ content = [{"type": "text", "text": "描述"}] + [{"type": "image"}] * n_parts+ return TokenizeMsg(+ uid=1, text=[{"role": "user", "content": content}],+ sampling_params=SamplingParams(), images=images,+ )+++def test_expansion_mrope_and_radix_ids(manager):+ r = manager.tokenize([_msg([_png(512, 512)])])[0]+ item = r.mm_items[0]+ L = r.input_ids.numel()+ assert item.num_tokens == 256 and item.grid_thw == [1, 32, 32]+ assert r.mrope_positions.shape == (3, L)+ assert int(r.mrope_positions.max()) + 1 - L == r.mrope_delta+ # the image span carries one content pad id above the vocab; text is untouched+ (start, end), = item.offsets+ span = r.input_ids[start:end]+ assert span.unique().numel() == 1+ assert span[0].item() == item.pad_value == mm_pad_value(item.hash) >= MM_PAD_SHIFT_VALUE+ assert bool((r.input_ids[:start] < MM_PAD_SHIFT_VALUE).all())+++def test_same_image_same_hash_and_pad(manager):+ a = manager.tokenize([_msg([_png(512, 512)])])[0]+ b = manager.tokenize([_msg([_png(512, 512)])])[0]+ assert a.mm_items[0].hash == b.mm_items[0].hash+ assert torch.equal(a.input_ids, b.input_ids)+ c = manager.tokenize([_msg([_png(256, 256)])])[0]+ assert c.mm_items[0].hash != a.mm_items[0].hash+++def test_image_max_tokens_clamps(manager):+ from freetoken.mm.config import MultimodalConfig+ from freetoken.mm.processor import get_mm_processor+ from freetoken.tokenizer.tokenize import TokenizeManager++ path = manager.tokenizer.name_or_path+ budgeted = TokenizeManager(manager.tokenizer, get_mm_processor(path, MultimodalConfig(image_max_tokens=2048)))+ assert manager.tokenize([_msg([_png(3840, 2160)])])[0].mm_items[0].num_tokens > 2048+ assert budgeted.tokenize([_msg([_png(3840, 2160)])])[0].mm_items[0].num_tokens <= 2048+++def test_count_prompt_tokens_counts_the_expanded_image(manager):+ import asyncio+ import base64+ from types import SimpleNamespace++ from freetoken.server.generation import count_prompt_tokens++ png = _png(512, 512)+ messages = [{"role": "user", "content": [+ {"type": "image", "freetoken_ref": {"kind": "b64", "data": base64.b64encode(png).decode()}},+ {"type": "text", "text": "描述"},+ ]}]+ state = SimpleNamespace(+ frontend_tokenizer=lambda: manager,+ config=SimpleNamespace(served_modalities=frozenset({"image"}), mm=SimpleNamespace(text_model_only=False, disabled_encoders=frozenset())),+ )+ counted = asyncio.run(count_prompt_tokens(messages, None, {}, state))+ expanded = manager.tokenize([_msg([png])])[0].input_ids.numel()+ assert counted == expanded and counted > 256diff --git a/tests/tokenizer/test_tokenize.py b/tests/tokenizer/test_tokenize.pyindex 65fa5f2d..369576a1 100644--- a/tests/tokenizer/test_tokenize.py+++ b/tests/tokenizer/test_tokenize.py@@ -37,7 +37,7 @@ def test_tokenize_manager_passes_chat_template_kwargs():chat_template_kwargs={"enable_thinking": True},)- [input_ids] = manager.tokenize([msg])+ input_ids = manager.tokenize([msg])[0].input_idsassert tokenizer.chat_template_kwargs == {"tokenize": False,@@ -64,7 +64,7 @@ def test_tokenize_manager_passes_tools_to_chat_template():tools=tools,)- [input_ids] = manager.tokenize([msg])+ input_ids = manager.tokenize([msg])[0].input_idsassert tokenizer.chat_template_kwargs == {"tokenize": False,@@ -120,7 +120,7 @@ def encode_messages(messages, thinking_mode, reasoning_effort=None):tools=tools,)- [input_ids] = manager.tokenize([msg])+ input_ids = manager.tokenize([msg])[0].input_idsassert tokenizer.prompt == "dsv4 prompt"assert input_ids.tolist() == [4, 5, 6]@@ -163,7 +163,7 @@ def encode_messages(messages, thinking_mode, reasoning_effort=None):]msg = TokenizeMsg(uid=1, text=messages, sampling_params=SamplingParams())- [input_ids] = manager.tokenize([msg])+ input_ids = manager.tokenize([msg])[0].input_idsassert tokenizer.prompt == "dsv4 prompt"assert input_ids.tolist() == [4, 5, 6]