Skip to content

FreeToken brings datacenter-scale model serving to your desktop. Run massive models locally, fast and efficiently.

feat(mm): serve image input on the Qwen families (#454)

Xiaoze Fan08d728d5e385
diff --git a/docs/cli.md b/docs/cli.md
index e1eaa8aa..1611214b 100644
--- a/docs/cli.md
+++ b/docs/cli.md
@@ -99,6 +99,26 @@ See [models.md](models.md#moe-strategies) for what each strategy does.
| `--reasoning-parser` | auto | Splits chain-of-thought into `reasoning_content`; auto-inferred; `off` disables |
| `--enable-cache-report` | off | Report prefix-cache hits in each response's usage block |
+### Image input
+
+Experimental. Needs a vision-capable checkpoint (Qwen3.6, Qwen3.8-Flash-Next, Qwen3-VL); a request carrying images is
+rejected otherwise. Images are accepted on all three protocols (OpenAI `image_url`,
+Anthropic `image` blocks, Responses `input_image`) as an http(s) URL or base64. Images inside Anthropic
+`tool_result` blocks are rejected with a 400: chat templates render tool messages as plain text.
+`GET /v1/stats` reports what the server accepts as `model.input_modalities` (`["text"]` or `["text", "image"]`),
+so a client can gate its attachment controls without reading the checkpoint config.
+
+| Flag | Default | Meaning |
+| --- | --- | --- |
+| `--text-model-only` | off | Serve a multimodal checkpoint text-only: no encoder tower is built (its VRAM goes to the KV/expert pools) and every multimodal input is rejected. Same as `--mm-disable` with every encoder kind |
+| `--mm-disable` | none | Encoder towers to leave unbuilt (`vision`, `audio`); every input they would serve is rejected |
+| `--mm-encoder-weights` | host | Where the encoder tower's block weights live. `host` streams them from pinned host banks two blocks at a time behind the compute (about 60 MiB of VRAM instead of the whole tower; small images encode slower, about 17 ms instead of 7 ms for 448x448); `gpu` keeps them resident |
+| `--image-min-tokens`, `--image-max-tokens` | processor defaults | Per-image token budget: the image processor resizes every image to take between these many tokens, converted to the family's own units (Qwen VL: one token per 32x32 pixels of the resized image, defaults 64 to 16384). Only families with dynamic resolution honor them |
+| `--mm-processor-kwargs` | none | JSON object of extra keyword arguments for the checkpoint's image processor call, for family-specific knobs (Qwen VL: `{"size": {"longest_edge": 1048576}}`); applied after the token budget |
+| `--mm-embed-cache-device` | cpu | Where encoded image embeddings live between prefill chunks. `cpu` keeps them out of the VRAM budget; `cuda` skips the copy back |
+| `--allowed-media-domains` | any | Comma-separated hostname allowlist for image URLs; requests for other domains are rejected with a 400. Empty allows any domain |
+| `--allowed-local-media-path` | off | Directory `file://` image refs may be read from; unset rejects local files |
+
## ft shell
```bash
@@ -118,7 +138,7 @@ ft ctl [--base-url http://127.0.0.1:1919] [--timeout 10] [--json] <subcommand>
| Subcommand | Endpoint | Purpose |
|---|---|---|
| `health` | `GET /health` | Server status, model, load progress |
-| `stats` | `GET /v1/stats` | Throughput, latency, VRAM, pool occupancy |
+| `stats` | `GET /v1/stats` | Throughput, latency, VRAM, pool occupancy, accepted input modalities |
| `generate [prompt] [--max-tokens N] [--ignore-eos]` | `POST /generate` | Raw completion smoke test (no chat template) |
| `cache` | `GET /v1/cache/status` | Cache pool table |
| `cache --moe N \| --kv N \| --mamba N \| --swa N [--wait 300]` | `POST /v1/cache/rebuild` | Live pool resizing without a restart (`k`/`m` suffixes; `--kv`/`--swa` in tokens) |
diff --git a/docs/models.md b/docs/models.md
index e4115356..0a9e214a 100644
--- a/docs/models.md
+++ b/docs/models.md
@@ -13,11 +13,15 @@ for them; other checkpoints of the same architectures work too.
| Qwen3.6 / Qwen3.5 MoE | [Qwen/Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B) ([-FP8](https://huggingface.co/Qwen/Qwen3.6-35B-A3B-FP8)), [nvidia/Qwen3.6-35B-A3B-NVFP4](https://huggingface.co/nvidia/Qwen3.6-35B-A3B-NVFP4), [Qwen/Qwen3.5-35B-A3B](https://huggingface.co/Qwen/Qwen3.5-35B-A3B) ([-FP8](https://huggingface.co/Qwen/Qwen3.5-35B-A3B-FP8)) |
| Qwen3.8 / Qwen3.6 dense | [Qwen/Qwen3.8-27B](https://huggingface.co/Qwen/Qwen3.8-27B) ([-FP8](https://huggingface.co/Qwen/Qwen3.8-27B-FP8)), [RadixArk/Qwen3.8-27B-NVFP4](https://huggingface.co/RadixArk/Qwen3.8-27B-NVFP4), [Qwen/Qwen3.6-27B](https://huggingface.co/Qwen/Qwen3.6-27B) ([-FP8](https://huggingface.co/Qwen/Qwen3.6-27B-FP8)), [nvidia/Qwen3.6-27B-NVFP4](https://huggingface.co/nvidia/Qwen3.6-27B-NVFP4) |
| Qwen3-MoE | [Qwen/Qwen3-30B-A3B](https://huggingface.co/Qwen/Qwen3-30B-A3B) |
+| Qwen3-VL | [Qwen/Qwen3-VL-8B-Instruct](https://huggingface.co/Qwen/Qwen3-VL-8B-Instruct), [Qwen/Qwen3-VL-30B-A3B-Instruct](https://huggingface.co/Qwen/Qwen3-VL-30B-A3B-Instruct) |
| gpt-oss | [openai/gpt-oss-120b](https://huggingface.co/openai/gpt-oss-120b), [openai/gpt-oss-20b](https://huggingface.co/openai/gpt-oss-20b) |
| Gemma-4 | [google/gemma-4-26B-A4B-it](https://huggingface.co/google/gemma-4-26B-A4B-it), [nvidia/Gemma-4-26B-A4B-NVFP4](https://huggingface.co/nvidia/Gemma-4-26B-A4B-NVFP4), [google/gemma-4-12B-it](https://huggingface.co/google/gemma-4-12B-it), [nvidia/Gemma-4-31B-IT-NVFP4](https://huggingface.co/nvidia/Gemma-4-31B-IT-NVFP4) .. |
| MiniMax-M2.5 | [nvidia/MiniMax-M2.5-NVFP4](https://huggingface.co/nvidia/MiniMax-M2.5-NVFP4) |
| Muse-Glimmer | [meta-models/Muse-Glimmer-30B](https://huggingface.co/meta-models/Muse-Glimmer-30B), [RedHatAI/Muse-Glimmer-30B-NVFP4](https://huggingface.co/RedHatAI/Muse-Glimmer-30B-NVFP4) |
+Qwen3.6 (both variants, every listed weight format), Qwen3.8-Flash-Next and Qwen3-VL also accept image input by default;
+pass `--text-model-only` to skip the vision tower; see [CLI reference](cli.md#image-input).
+
## MoE strategies
`ft serve --moe-strategy {auto,fused,offload,cpu,hybrid}` (`--moe-backend` is the deprecated old spelling):
diff --git a/pyproject.toml b/pyproject.toml
index 8bd653f8..228108f0 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -47,6 +47,7 @@ dependencies = [
"numpy>=2.0,<2.5",
"openai>=2.0,<3",
"partial-json-parser>=0.2,<1",
+ "pillow>=10,<13",
"prompt_toolkit>=3.0,<4",
"pydantic>=2.9,<3",
"pyzmq>=27,<28",
@@ -55,8 +56,9 @@ dependencies = [
# PyPI's torch 2.11.0 wheel is itself the cu130 build, so plain pip resolves
# correctly from PyPI alone; uv additionally pins the index below.
"torch>=2.11,<2.12",
+ "torchvision>=0.26,<0.27",
"tqdm>=4.66,<5",
- "transformers>=5.5,<6",
+ "transformers>=5.15,<6",
"triton==3.6.0; platform_system == 'Linux'",
"uvicorn>=0.30,<1",
]
diff --git a/python/freetoken/attention/qsa_sparse.py b/python/freetoken/attention/qsa_sparse.py
index 4a28dc85..b07c9523 100644
--- a/python/freetoken/attention/qsa_sparse.py
+++ b/python/freetoken/attention/qsa_sparse.py
@@ -88,6 +88,10 @@ class QSASparseMetadata(BaseAttnMetadata):
cmp_rows: torch.Tensor | None = None # [T] int32, compressed slab destination
ring_rows: torch.Tensor | None = None # [T] int32, flat ring row or -1
positions: torch.Tensor | None = None # [T] int32, logical query positions
+ # mrope only, built once per forward: row r of the caches is token r's [cos | sin] (queries) or its group start's (keys)
+ rope_rows: torch.Tensor | None = None # [T] int32 arange
+ q_rope_cache: torch.Tensor | None = None # [T, rotary_dim] float32
+ k_rope_cache: torch.Tensor | None = None # [T, rotary_dim] float32
# fmt: on
def get_last_indices(self, bs: int) -> torch.Tensor:
@@ -129,6 +133,13 @@ class QSASparseAttnBackend(BaseAttnBackend):
self._idx_slot = {lid: i for i, lid in enumerate(group.layer_ids)}
self.rotary_config = group.rotary_config
self._index_cos_sin: torch.Tensor | None = None
+ self._section_table: torch.Tensor | None = None
+ if group.rotary_config.mrope_section is not None:
+ from freetoken.layers.rotary import build_section_table
+
+ self._section_table = build_section_table(
+ tuple(group.rotary_config.mrope_section), group.rotary_config.mrope_layout
+ ).to(self.device)
self._block_topk_kernel = _resolve_block_topk()
# decode staging (static buffers under CUDA graphs; eager decode snapshots per step)
@@ -175,6 +186,12 @@ class QSASparseAttnBackend(BaseAttnBackend):
self._index_cos_sin = rope._cos_sin_cache.to(self.device)
return self._index_cos_sin
+ def _index_rope_rows(self, positions: torch.Tensor) -> torch.Tensor:
+ """Per-token indexer cos/sin rows for [3, n] mrope positions."""
+ from freetoken.layers.rotary import mrope_cos_sin_rows
+
+ return mrope_cos_sin_rows(self._index_rope_cache(), positions, self._section_table)
+
# ----- metadata -----------------------------------------------------------------------
def prepare_metadata(self, batch: Batch) -> None:
reqs = batch.padded_reqs if hasattr(batch, "padded_reqs") else batch.reqs
@@ -298,6 +315,17 @@ class QSASparseAttnBackend(BaseAttnBackend):
md.positions = batch.positions
out_loc = batch.out_loc.to(torch.int64)
positions = batch.positions.to(torch.int64)
+ if self._section_table is not None:
+ rope_positions = batch.get_attn_positions()
+ if rope_positions.dim() == 1:
+ rope_positions = rope_positions.unsqueeze(0).expand(3, -1)
+ rope_positions = rope_positions.to(torch.int32)
+ self.kvcache.rope_positions.index_copy_(0, out_loc, rope_positions.t().contiguous())
+ # groups are ratio-aligned within a page, so the group start is the slot rounded down
+ first_pos = self.kvcache.rope_positions.index_select(0, out_loc - out_loc % self.ratio).t()
+ md.rope_rows = torch.arange(out_loc.numel(), dtype=torch.int32, device=self.device)
+ md.q_rope_cache = self._index_rope_rows(rope_positions)
+ md.k_rope_cache = self._index_rope_rows(first_pos)
rows = torch.arange(out_loc.numel(), device=self.device)
req = md.token_to_req.to(torch.int64)
slots = md.ring_slots.to(torch.int64).index_select(0, req)
@@ -338,10 +366,14 @@ class QSASparseAttnBackend(BaseAttnBackend):
pooled,
first,
)
+ if md.k_rope_cache is None:
+ rope_positions, rope_cache = first, self._index_rope_cache()
+ else:
+ rope_positions, rope_cache = md.rope_rows, md.k_rope_cache
qsa_index_norm_rope(
pooled,
- first,
- self._index_rope_cache(),
+ rope_positions,
+ rope_cache,
index.k_norm_weight,
index.eps,
self.kvcache.cmp_k_cache(slot),
@@ -364,10 +396,14 @@ class QSASparseAttnBackend(BaseAttnBackend):
q_index = self._scratch(
"q_index", rows, self.index_heads, self.index_head_dim, dtype=self.dtype
)
+ if md.q_rope_cache is None:
+ rope_positions, rope_cache = positions, self._index_rope_cache()
+ else:
+ rope_positions, rope_cache = md.rope_rows, md.q_rope_cache
qsa_index_norm_rope(
index.q.view(-1, self.index_head_dim),
- positions,
- self._index_rope_cache(),
+ rope_positions,
+ rope_cache,
index.q_norm_weight,
index.eps,
q_index.view(-1, self.index_head_dim),
diff --git a/python/freetoken/core.py b/python/freetoken/core.py
index 281ff41a..269b94ff 100644
--- a/python/freetoken/core.py
+++ b/python/freetoken/core.py
@@ -39,9 +39,10 @@ class Req:
uid: int
sampling_params: SamplingParams
cache_handle: BaseCacheHandle
- # Optional precomputed multimodal soft-token embeddings (GPU, [num_image_tokens,
- # hidden]) scattered at image-token positions during this request's prefill.
- mm_embeds: torch.Tensor | None = None
+ # per-item processor outputs and the tokenizer's precomputed mrope rows and delta
+ mm_items: list | None = None
+ mrope_positions_full: torch.Tensor | None = None # [3, prompt_len] int32, CPU
+ mrope_delta: int = 0
# --- hybrid-radix (GDN linear-state) per-request slots; None for non-hybrid models or
# until allocated from LinearStatePool. Set by the scheduler (P2). ---
@@ -115,6 +116,8 @@ class Batch:
# these fields should be set by scheduler
input_ids: torch.Tensor = field(init=False)
positions: torch.Tensor = field(init=False)
+ # [3, n] t/h/w rope positions on mrope models; positions keeps its sequence-index meaning for token_pool / page_table
+ mrope_positions: torch.Tensor | None = field(default=None, init=False)
out_loc: torch.Tensor | None = field(init=False)
# Per-(padded-)request table_idx as a GPU int64 tensor, used by GatedDeltaNet
# decode to gather/scatter recurrent+conv state without host-side loops (so the
@@ -132,8 +135,12 @@ class Batch:
active_table_idx: "torch.Tensor | None" = None
# this field should be set by attention backend
attn_metadata: BaseAttnMetadata = field(init=False)
- # concatenated multimodal soft-token embeddings for a prefill batch (or None)
+ # concatenated multimodal soft-token embeddings for a prefill batch (or None) and the batch rows they land on
mm_embeds: torch.Tensor | None = field(default=None, init=False)
+ mm_rows: torch.Tensor | None = field(default=None, init=False)
+ # this chunk's cache-miss items to encode and the gather plan [(uid, hash, row_lo, row_hi, n, pos), ...] in scatter order
+ mm_encoder_jobs: list | None = field(default=None, init=False)
+ mm_gather_plan: list | None = field(default=None, init=False)
# Prefill log stats snapshotted at schedule time (before forward's complete_one()
# advances cached_len), so the prefill log reports the tokens actually forwarded and
# the prefix-cache hit -- matching SGLang's #new-token / #cached-token. Set by the
@@ -154,6 +161,9 @@ class Batch:
def is_decode(self) -> bool:
return self.phase == "decode"
+ def get_attn_positions(self) -> torch.Tensor:
+ return self.mrope_positions if self.mrope_positions is not None else self.positions
+
@property
def size(self) -> int:
return len(self.reqs)
diff --git a/python/freetoken/engine/config.py b/python/freetoken/engine/config.py
index 7ab792f9..44c5a38e 100644
--- a/python/freetoken/engine/config.py
+++ b/python/freetoken/engine/config.py
@@ -1,5 +1,6 @@
from __future__ import annotations
+import copy
from dataclasses import dataclass, field, replace
from functools import cached_property
from typing import TYPE_CHECKING, List
@@ -7,7 +8,8 @@ from typing import TYPE_CHECKING, List
import torch
from freetoken.distributed import DistributedInfo
from freetoken.layers.quantization import set_quant_config
-from freetoken.models.register import _load_attr, checkpoint_quant_config, get_model_spec
+from freetoken.mm.config import ENCODER_SECTIONS, MultimodalConfig
+from freetoken.models.register import EncoderSpec, ModelSpec, _load_attr, checkpoint_quant_config, get_model_spec
from freetoken.utils import cached_load_hf_config, init_logger
if TYPE_CHECKING:
@@ -87,6 +89,8 @@ class EngineConfig:
# KV capacity in tokens; resolved into num_page_override by _adjust_config once page_size
# is final. Mutually exclusive with num_page_override.
num_token_override: int | None = None
+ # Runtime knobs of the multimodal path; the architecture side (vision_config, mrope) lives in ModelConfig.
+ mm: MultimodalConfig = field(default_factory=MultimodalConfig)
def __post_init__(self):
if self.moe_backend is None:
@@ -101,13 +105,38 @@ class EngineConfig:
def hf_config(self):
return cached_load_hf_config(self.model_path)
+ @cached_property
+ def model_spec(self) -> ModelSpec:
+ return get_model_spec(self.hf_config.architectures[0])
+
+ @cached_property
+ def active_encoders(self) -> tuple[EncoderSpec, ...]:
+ """The encoder towers this process builds: the family registers them, the checkpoint config carries their section, --mm-disable did not name them."""
+ return tuple(
+ e
+ for e in self.model_spec.encoders
+ if getattr(self.hf_config, e.config_key, None) is not None
+ and e.kind not in self.mm.disabled_encoders
+ )
+
+ @cached_property
+ def served_modalities(self) -> frozenset[str]:
+ """Modalities this process accepts."""
+ return frozenset(m for e in self.active_encoders for m in e.modalities)
+
@cached_property
def model_config(self) -> ModelConfig:
- spec = get_model_spec(self.hf_config.architectures[0])
- quant = checkpoint_quant_config(self.model_path, self.hf_config, spec)
+ # the parser sees no section for a tower this process does not build (for the vision tower that also means 1-D rope)
+ hf_config = copy.copy(self.hf_config)
+ built = {e.config_key for e in self.active_encoders}
+ for key in set(ENCODER_SECTIONS) | {e.config_key for e in self.model_spec.encoders}:
+ if key not in built:
+ setattr(hf_config, key, None)
+ spec = self.model_spec
+ quant = checkpoint_quant_config(self.model_path, hf_config, spec)
set_quant_config(quant)
- parse_config = _load_attr(spec.module, spec.parse_config)
- return replace(parse_config(self.hf_config), quant=quant)
+ model_config = _load_attr(spec.module, spec.parse_config)(hf_config)
+ return replace(model_config, quant=quant)
@property
def max_seq_len(self) -> int:
diff --git a/python/freetoken/engine/engine.py b/python/freetoken/engine/engine.py
index 006c1908..c92e1d88 100644
--- a/python/freetoken/engine/engine.py
+++ b/python/freetoken/engine/engine.py
@@ -14,6 +14,7 @@ from freetoken.gpu_select import gpu_identity
from freetoken.layers import set_rope_device
from freetoken.layers.quantization import LayerKind, QuantBackend, finalize_quant, set_quant_backend
from freetoken.moe.offload_cache import iter_offload_moe_layers
+from freetoken.mm.config import ENCODER_SECTIONS
from freetoken.models import create_model, load_weight
from freetoken.moe import is_offload_moe_strategy
from freetoken.moe.expert_banks import load_expert_banks
@@ -33,7 +34,6 @@ from freetoken.kvcache.linear_state_pool import (
logger = init_logger(__name__)
-
def _require_offload_cache_size(cache_size: int, num_experts: int) -> None:
"""The offload MoE cache needs at least one slot per expert per layer. A too-small size
(e.g. a bare offload run with moe_cache_size unset and auto disabled) must fail loudly."""
@@ -328,6 +328,16 @@ class Engine:
self.model = create_model(config.model_config)
self.model.load_state_dict(self._load_weight_state_dict(config))
finalize_quant(self.model)
+ if config.active_encoders:
+ from freetoken.models.blocks import SupportsMultimodal
+
+ if not isinstance(self.model, SupportsMultimodal):
+ raise TypeError(
+ f"{type(self.model).__name__} has encoders registered but lacks the SupportsMultimodal hooks; "
+ "run with --text-model-only"
+ )
+ # before the residency snapshot, so streamed blocks are not charged as resident weights
+ self.model.place_encoder_weights(config.mm.encoder_weights)
post_weights_free = self._sync_get_memory()[0]
self._weights_bytes = self._baseline_free - post_weights_free
# Pool-budget baseline for the desktop cache sliders: free VRAM after the weights are
@@ -348,6 +358,27 @@ class Engine:
self._init_offload_moe_cache(config)
if hasattr(self.model, "prepare_for_runtime"):
self.model.prepare_for_runtime()
+ self.encoder_cache = None
+ self.mm_processor = None
+ if config.active_encoders:
+ from freetoken.mm.encoder_cache import EncoderCache
+ from freetoken.mm.processor import get_mm_processor
+
+ self.mm_processor = get_mm_processor(config.model_path, config.mm)
+ self.encoder_cache = EncoderCache(storage=config.mm.embed_cache_device)
+ logger.info_rank0(
+ f"Multimodal enabled: {type(self.mm_processor).__name__}, encoders "
+ f"{[e.kind for e in config.active_encoders]} on {config.mm.encoder_weights}, serving {sorted(config.served_modalities)}"
+ )
+ self._warmup_encoders()
+ elif any(getattr(config.hf_config, key, None) is not None for key in ENCODER_SECTIONS):
+ logger.info_rank0(
+ "Multimodal disabled: --text-model-only"
+ if config.mm.text_model_only
+ else "Multimodal disabled: --mm-disable"
+ if config.mm.disabled_encoders
+ else "Multimodal disabled: no encoder registered for this architecture"
+ )
# ======================= KV cache initialization ========================
new_free = self._sync_get_memory()[1]
@@ -429,6 +460,7 @@ class Engine:
vocab_size=config.model_config.vocab_size,
dummy_req=self.dummy_req,
moe_offload_cache=self.moe_offload_cache,
+ mrope=config.model_config.model_is_mrope,
)
if config.attention_backend.split(",")[0] == "triton":
# Prefill runs on the first comma part; warm its autotune cache.
@@ -474,10 +506,43 @@ class Engine:
config.model_path,
self.device,
include_moe_experts=not is_offload_moe_strategy(config.moe_strategy),
+ include_vision=bool(config.active_encoders),
),
device=self.device,
)
+ @torch.inference_mode()
+ def _warmup_encoders(self) -> None:
+ for item in self.mm_processor.dummy_items(self.dtype, self.device):
+ if item.modality in self.config.served_modalities:
+ self.model.encode(item)
+ torch.cuda.synchronize(self.device)
+
+ @torch.inference_mode()
+ def _run_mm_encoder(self, batch: Batch) -> None:
+ """Encode the chunk's cache-miss items and gather its embedding rows into batch.mm_embeds, right before the LM forward."""
+ cache = self.encoder_cache
+ jobs = batch.mm_encoder_jobs or ()
+ # a job whose rows are not gathered this chunk would be encoded and freed unread
+ planned = {row[1] for row in batch.mm_gather_plan}
+ orphans = [item.hash for item in jobs if item.hash not in planned]
+ assert not orphans, f"encoder jobs without gather rows: {orphans}"
+ for item in jobs:
+ if not cache.has(item.hash):
+ if item.precomputed_embeddings is not None:
+ emb = item.precomputed_embeddings.to(self.device, non_blocking=True)
+ else:
+ emb = self.model.encode(item)
+ cache.put(item.hash, emb)
+ # cached now; free the ~MiB feature buffer
+ item.feature = None
+ item.precomputed_embeddings = None
+ parts = []
+ for uid, item_hash, lo, hi, _, _ in batch.mm_gather_plan:
+ parts.append(cache.get_slice(item_hash, lo, hi, self.device))
+ cache.consume(item_hash, uid, hi - lo)
+ if parts:
+ batch.mm_embeds = torch.cat([p.to(self.dtype) for p in parts], dim=0)
def _resolve_auto_moe_cache_size(self, config: EngineConfig, banks, method=None) -> tuple[int, int, bool]:
"""Resolve --moe-cache-auto into (moe_cache_size, num_pages, prefill_overlap).
@@ -906,10 +971,13 @@ class Engine:
vocab_size=config.model_config.vocab_size,
dummy_req=self.dummy_req,
moe_offload_cache=self.moe_offload_cache,
+ mrope=config.model_config.model_is_mrope,
)
def forward_batch(self, batch: Batch, args: BatchSamplingArgs) -> ForwardOutput:
assert torch.cuda.current_stream() == self.stream
+ if batch.mm_gather_plan:
+ self._run_mm_encoder(batch)
use_graph = self.graph_runner.can_use_cuda_graph(batch)
with self.ctx.forward_batch(batch), self.model.forward_host_ctx(batch, use_graph):
logits = self.graph_runner.replay(batch) if use_graph else self.model.forward()
@@ -970,6 +1038,10 @@ class Engine:
batch.padded_reqs = batch.reqs
batch.input_ids = torch.zeros(length, dtype=torch.int32, device=self.device)
batch.positions = torch.arange(length, dtype=torch.int32, device=self.device)
+ if self.config.model_config.model_is_mrope:
+ batch.mrope_positions = (
+ batch.positions.unsqueeze(0).expand(3, -1).contiguous()
+ )
batch.out_loc = dummy_row[:length]
self.attn_backend.prepare_metadata(batch)
with self.ctx.forward_batch(batch):
diff --git a/python/freetoken/engine/graph.py b/python/freetoken/engine/graph.py
index 4f202502..eb9684f6 100644
--- a/python/freetoken/engine/graph.py
+++ b/python/freetoken/engine/graph.py
@@ -24,17 +24,24 @@ class GraphCaptureBuffer:
input_ids: torch.Tensor
out_loc: torch.Tensor
positions: torch.Tensor
+ # [3, bs] t/h/w rope positions; allocated only for mrope models (else None).
+ mrope_positions: torch.Tensor | None
logits: torch.Tensor
table_idx: torch.Tensor # per-request slot id for GatedDeltaNet state gather/scatter
# Decode GDN query indptr = arange(bs+1); a constant per captured bs, filled once.
fla_cu_seqlens: torch.Tensor
@classmethod
- def init(cls, bs: int, vocab_size: int, device: torch.device) -> GraphCaptureBuffer:
+ def init(
+ cls, bs: int, vocab_size: int, device: torch.device, mrope: bool = False
+ ) -> GraphCaptureBuffer:
return GraphCaptureBuffer(
input_ids=torch.zeros(bs, dtype=torch.int32, device=device),
out_loc=torch.zeros(bs, dtype=torch.int32, device=device),
positions=torch.zeros(bs, dtype=torch.int32, device=device),
+ mrope_positions=(
+ torch.zeros(3, bs, dtype=torch.int32, device=device) if mrope else None
+ ),
logits=torch.empty(bs, vocab_size, dtype=torch.float32, device=device),
table_idx=torch.zeros(bs, dtype=torch.int32, device=device),
fla_cu_seqlens=torch.arange(bs + 1, dtype=torch.int32, device=device),
@@ -48,6 +55,8 @@ class GraphCaptureBuffer:
batch.input_ids = self.input_ids[_slice]
batch.out_loc = self.out_loc[_slice]
batch.positions = self.positions[_slice]
+ if self.mrope_positions is not None:
+ batch.mrope_positions = self.mrope_positions[:, _slice]
batch.linear_table_idx = self.table_idx[_slice]
# Decode GDN metadata reads the persistent cu_seqlens (constant arange) and the
# persistent table_idx slot map, so the captured kernels see stable addresses.
@@ -61,6 +70,8 @@ class GraphCaptureBuffer:
if batch.out_loc is not None:
self.out_loc[_slice] = batch.out_loc
self.positions[_slice] = batch.positions
+ if self.mrope_positions is not None:
+ self.mrope_positions[:, _slice] = batch.mrope_positions
if batch.linear_table_idx is not None:
self.table_idx[_slice] = batch.linear_table_idx
@@ -105,6 +116,7 @@ class GraphRunner:
vocab_size: int,
dummy_req: Req,
moe_offload_cache: OffloadMoeCache | None = None,
+ mrope: bool = False,
) -> None:
cuda_graph_bs = _determine_cuda_graph_bs(
cuda_graph_bs=cuda_graph_bs,
@@ -116,6 +128,7 @@ class GraphRunner:
self.graph_bs_list = sorted(cuda_graph_bs)
self.dummy_req = dummy_req
self.moe_offload_cache = moe_offload_cache
+ self.mrope = mrope
self.stream = stream
self.device = device
self._capture_graphs(max_seq_len, vocab_size, model)
@@ -145,7 +158,9 @@ class GraphRunner:
free_memory = get_free_memory(self.device)
logger.info_rank0(f"Free GPU memory before capturing CUDA graphs: {mem_GB(free_memory)}")
- self.buffer = GraphCaptureBuffer.init(self.max_graph_bs, vocab_size, self.device)
+ self.buffer = GraphCaptureBuffer.init(
+ self.max_graph_bs, vocab_size, self.device, mrope=self.mrope
+ )
self._reset_moe_offload_cache()
pbar = tqdm(
diff --git a/python/freetoken/kernel/aot_models.py b/python/freetoken/kernel/aot_models.py
index 8d066d43..a8ef85d4 100644
--- a/python/freetoken/kernel/aot_models.py
+++ b/python/freetoken/kernel/aot_models.py
@@ -145,6 +145,7 @@ SUPPORTED_MODELS: tuple[AotModel, ...] = (
moe_intermediate_size=768,
expert_formats=("bf16",),
aliases=("Qwen/Qwen3-30B-A3B-Thinking-2507",),
+ arch_aliases=("Qwen3VLMoeForConditionalGeneration",), # Qwen3-VL-30B-A3B: same text tower
),
AotModel(
name="Qwen/Qwen3.5-35B-A3B",
@@ -154,6 +155,7 @@ SUPPORTED_MODELS: tuple[AotModel, ...] = (
top_k=8,
moe_intermediate_size=512,
expert_formats=("bf16",),
+ arch_aliases=("Qwen3_5MoeForCausalLM",), # text-only release of the same tower
),
AotModel(
name="Qwen/Qwen3.5-35B-A3B-FP8",
@@ -337,6 +339,7 @@ SUPPORTED_MODELS: tuple[AotModel, ...] = (
hidden_size=5120,
kv_groups=((4, 256),),
aliases=("Qwen/Qwen3.6-27B-FP8", "nvidia/Qwen3.6-27B-NVFP4"),
+ arch_aliases=("Qwen3_5ForCausalLM",), # text-only release of the same tower
),
AotModel(
name="google/gemma-4-12B-it",
@@ -375,6 +378,7 @@ SUPPORTED_MODELS: tuple[AotModel, ...] = (
architecture="Qwen3ForCausalLM",
hidden_size=4096,
kv_groups=((8, 128),),
+ arch_aliases=("Qwen3VLForConditionalGeneration",), # Qwen3-VL-8B: same text tower
),
AotModel(
name="mistralai/Mistral-7B-Instruct-v0.3",
diff --git a/python/freetoken/kernel/triton/rope.py b/python/freetoken/kernel/triton/rope.py
index 0284c82d..4cd25c91 100644
--- a/python/freetoken/kernel/triton/rope.py
+++ b/python/freetoken/kernel/triton/rope.py
@@ -141,3 +141,130 @@ def apply_rope_with_cos_sin_cache_inplace(
__all__ = ["apply_rope_with_cos_sin_cache_inplace"]
+
+
[email protected](do_not_specialize=["nnz"])
+def _mrope_tiled(
+ Q, K, POS, CACHE, SEC,
+ stride_qbs, stride_qh, stride_qd,
+ stride_kbs, stride_kh, stride_kd,
+ nnz,
+ HEAD_Q, HEAD_K, rotary_dim, half,
+ BLOCK_SEQ: tl.constexpr,
+ BLOCK_HEAD: tl.constexpr,
+ BLOCK_DHALF: tl.constexpr,
+):
+ # _rope_tiled with 3-axis positions: POS is [3, nnz], SEC[half] picks the axis each frequency slot reads
+ seq_pid = tl.program_id(0)
+ head_pid = tl.program_id(1)
+
+ seq_range = seq_pid * BLOCK_SEQ + tl.arange(0, BLOCK_SEQ)
+ head_range = head_pid * BLOCK_HEAD + tl.arange(0, BLOCK_HEAD)
+ d = tl.arange(0, BLOCK_DHALF)
+
+ seq_mask = seq_range < nnz
+ dmask = d < half
+
+ sec = tl.load(SEC + d, mask=dmask, other=0).to(tl.int64) # (D,) in {0,1,2}
+ pos_ptr = POS + sec[None, :] * nnz + seq_range[:, None]
+ cs_mask = seq_mask[:, None] & dmask[None, :]
+ pos = tl.load(pos_ptr, mask=cs_mask, other=0).to(tl.int64) # (S,D)
+ cache_row = CACHE + pos * rotary_dim
+ cos = tl.load(cache_row + d[None, :], mask=cs_mask, other=0.0)
+ sin = tl.load(cache_row + half + d[None, :], mask=cs_mask, other=0.0)
+ cos = cos[:, None, :]
+ sin = sin[:, None, :]
+
+ d0 = d[None, None, :]
+ d1 = (half + d)[None, None, :]
+
+ qmask = (seq_mask[:, None, None]
+ & (head_range[None, :, None] < HEAD_Q)
+ & dmask[None, None, :])
+ base_q = seq_range[:, None, None] * stride_qbs + head_range[None, :, None] * stride_qh
+ q0 = tl.load(Q + base_q + d0 * stride_qd, mask=qmask, other=0.0).to(tl.float32)
+ q1 = tl.load(Q + base_q + d1 * stride_qd, mask=qmask, other=0.0).to(tl.float32)
+ tl.store(Q + base_q + d0 * stride_qd, (q0 * cos - q1 * sin).to(Q.dtype.element_ty), mask=qmask)
+ tl.store(Q + base_q + d1 * stride_qd, (q1 * cos + q0 * sin).to(Q.dtype.element_ty), mask=qmask)
+
+ kmask = (seq_mask[:, None, None]
+ & (head_range[None, :, None] < HEAD_K)
+ & dmask[None, None, :])
+ base_k = seq_range[:, None, None] * stride_kbs + head_range[None, :, None] * stride_kh
+ k0 = tl.load(K + base_k + d0 * stride_kd, mask=kmask, other=0.0).to(tl.float32)
+ k1 = tl.load(K + base_k + d1 * stride_kd, mask=kmask, other=0.0).to(tl.float32)
+ tl.store(K + base_k + d0 * stride_kd, (k0 * cos - k1 * sin).to(K.dtype.element_ty), mask=kmask)
+ tl.store(K + base_k + d1 * stride_kd, (k1 * cos + k0 * sin).to(K.dtype.element_ty), mask=kmask)
+
+
+def apply_mrope_with_cos_sin_cache_inplace(
+ positions: torch.Tensor,
+ query: torch.Tensor,
+ key: torch.Tensor,
+ head_size: int,
+ cos_sin_cache: torch.Tensor,
+ section_table: torch.Tensor,
+) -> None:
+ """3-axis rope: positions is [3, nnz], section_table [rotary_dim//2] maps each frequency slot to its axis row; NeoX layout only."""
+ if str(cos_sin_cache.dtype) != "torch.float32":
+ raise ValueError("cos_sin_cache should be float32")
+ assert positions.dim() == 2 and positions.shape[0] == 3
+ assert query.is_cuda and key.is_cuda and positions.is_cuda
+ assert cos_sin_cache.is_contiguous() and positions.is_contiguous()
+
+ nnz = query.shape[0]
+ if nnz == 0:
+ return
+ rotary_dim = cos_sin_cache.shape[1]
+ half = rotary_dim // 2
+
+ head_q = query.shape[1] // head_size
+ head_k = key.shape[1] // head_size
+ qv = query.view(nnz, head_q, head_size)
+ kv = key.view(nnz, head_k, head_size)
+ max_head = max(head_q, head_k)
+
+ grid = lambda META: (
+ triton.cdiv(nnz, META["BLOCK_SEQ"]),
+ triton.cdiv(max_head, META["BLOCK_HEAD"]),
+ )
+ _mrope_tiled[grid](
+ qv, kv, positions, cos_sin_cache, section_table,
+ qv.stride(0), qv.stride(1), qv.stride(2),
+ kv.stride(0), kv.stride(1), kv.stride(2),
+ nnz, head_q, head_k, rotary_dim, half,
+ BLOCK_DHALF=triton.next_power_of_2(half),
+ BLOCK_SEQ=16,
+ BLOCK_HEAD=1,
+ num_warps=4,
+ num_stages=1,
+ )
+
+
+def apply_mrope_torch_fallback(
+ positions: torch.Tensor,
+ query: torch.Tensor,
+ key: torch.Tensor,
+ head_size: int,
+ cos_sin_cache: torch.Tensor,
+ section_table: torch.Tensor,
+) -> None:
+ """Pure-torch mrope (Triton unavailable); same in-place contract as the kernel."""
+ nnz = query.shape[0]
+ if nnz == 0:
+ return
+ rotary_dim = cos_sin_cache.shape[1]
+ half = rotary_dim // 2
+ pos = positions[section_table.long(), :].transpose(0, 1) # [nnz, half]
+ cs = cos_sin_cache[pos] # [nnz, half, 2*half] rows gathered per slot
+ idx = torch.arange(half, device=query.device)
+ cos = cs[:, idx, idx].unsqueeze(1).float()
+ sin = cs[:, idx, half + idx].unsqueeze(1).float()
+ for t, heads in ((query, query.shape[1] // head_size), (key, key.shape[1] // head_size)):
+ v = t.view(nnz, heads, head_size)
+ lo, hi = v[..., :half].float(), v[..., half:rotary_dim].float()
+ v[..., :half] = (lo * cos - hi * sin).to(v.dtype)
+ v[..., half:rotary_dim] = (hi * cos + lo * sin).to(v.dtype)
+
+
+__all__ = ["apply_rope_with_cos_sin_cache_inplace", "apply_mrope_with_cos_sin_cache_inplace"]
diff --git a/python/freetoken/kvcache/__init__.py b/python/freetoken/kvcache/__init__.py
index 41b82f1d..1792da2b 100644
--- a/python/freetoken/kvcache/__init__.py
+++ b/python/freetoken/kvcache/__init__.py
@@ -204,6 +204,7 @@ def create_kvcache_pool(
index_ratio=spec.index_ratio,
num_req_slots=num_req_slots,
layer_ids=spec.layer_ids,
+ mrope=model_config.model_is_mrope,
)
if len(kv_specs) == 1 and kv_specs[0].mla:
diff --git a/python/freetoken/kvcache/qsa_pool.py b/python/freetoken/kvcache/qsa_pool.py
index fddcdbd3..7de70977 100644
--- a/python/freetoken/kvcache/qsa_pool.py
+++ b/python/freetoken/kvcache/qsa_pool.py
@@ -31,6 +31,8 @@ from .mha_pool import MHAKVCache
# The index tiers are always 2-byte (compute dtype); spec_kv_bytes_per_token budgets the same.
_INDEX_DTYPE_BYTES = 2
+# t/h/w int32 rope position kept per KV slot on mrope models
+_ROPE_POS_BYTES = 3 * 4
class QSAKVCache(MHAKVCache):
@@ -63,6 +65,7 @@ class QSAKVCache(MHAKVCache):
num_req_slots: int,
ring_capacity: int | None = None,
layer_ids: Sequence[int] | None = None,
+ mrope: bool = False,
) -> None:
if index_ratio < 1 or page_size % index_ratio != 0:
# slot // index_ratio only names one group when a group never straddles a page.
@@ -89,6 +92,7 @@ class QSAKVCache(MHAKVCache):
self._ring_capacity = ring_capacity
self._index_dtype = dtype
self._page_size = page_size
+ self._mrope = mrope
super().__init__(
num_kv_heads=num_kv_heads,
num_layers=num_layers,
@@ -128,6 +132,12 @@ class QSAKVCache(MHAKVCache):
dtype=self._index_dtype,
device=self._device,
)
+ # 3-axis rope position of every stored token: a compressed group ropes at its first token, which under mrope is not derivable from the logical position
+ self._rope_positions = (
+ torch.zeros(num_pages * self._page_size, 3, dtype=torch.int32, device=self._device)
+ if self._mrope
+ else None
+ )
def rebuild(self, num_pages: int) -> None:
# Free the index tiers BEFORE the K/V realloc (super().rebuild frees + syncs +
@@ -137,6 +147,7 @@ class QSAKVCache(MHAKVCache):
# cannot drop a live request's pending members.
self._cmp_k_buffer = None
self._pending_ring = None
+ self._rope_positions = None
super().rebuild(num_pages)
self._zero_kv_slabs()
try:
@@ -163,6 +174,8 @@ class QSAKVCache(MHAKVCache):
# One index-key row = all index layers at one position.
row = spec.index_head_dim * spec.num_index_layers * _INDEX_DTYPE_BYTES
fixed += num_req_slots * row * (cls.ring_capacity_for(spec.index_ratio) + 1)
+ if config.model_config.model_is_mrope:
+ per_token += _ROPE_POS_BYTES
return per_token * config.page_size, fixed, config.page_size, 0
def unit_bytes(self) -> tuple[int, int]:
@@ -176,7 +189,7 @@ class QSAKVCache(MHAKVCache):
* self._index_head_dim
* self._index_dtype.itemsize
)
- return kv + slab // tokens, swa
+ return kv + slab // tokens + (_ROPE_POS_BYTES if self._mrope else 0), swa
def cmp_k_cache(self, slot: int) -> torch.Tensor:
"""Compressed index keys of one sparse layer: ``[rows, index_head_dim]``."""
@@ -186,6 +199,12 @@ class QSAKVCache(MHAKVCache):
"""One sparse layer's pending ring: ``[num_req_slots, ring_capacity, index_head_dim]``."""
return self._pending_ring[:, slot]
+ @property
+ def rope_positions(self) -> torch.Tensor:
+ """``[num_tokens, 3]`` int32 t/h/w rope position per KV slot (written by the QSA backend)."""
+ assert self._rope_positions is not None, "rope positions are only kept on mrope models"
+ return self._rope_positions
+
@property
def cmp_scratch_base(self) -> int:
"""First scratch row of ``cmp_k_cache``; row ``cmp_scratch_base + table_idx`` sinks a
diff --git a/python/freetoken/layers/rotary.py b/python/freetoken/layers/rotary.py
index 3756a299..0d04645d 100644
--- a/python/freetoken/layers/rotary.py
+++ b/python/freetoken/layers/rotary.py
@@ -85,6 +85,114 @@ class RotaryEmbedding(StateLessOP):
return query, key
+def mrope_cos_sin_rows(
+ cos_sin_cache: torch.Tensor, positions: torch.Tensor, section_table: torch.Tensor
+) -> torch.Tensor:
+ """Per-token [cos | sin] rows for [3, n] positions: frequency slot i reads the cache row of axis section_table[i]."""
+ half = cos_sin_cache.shape[1] // 2
+ pos = positions[section_table.long(), :].transpose(0, 1).long() # [n, half]
+ idx = torch.arange(half, device=cos_sin_cache.device)
+ return torch.cat((cos_sin_cache[pos, idx], cos_sin_cache[pos, half + idx]), dim=1)
+
+
+def _mrope_torch(
+ positions: torch.Tensor,
+ query: torch.Tensor,
+ key: torch.Tensor,
+ head_size: int,
+ cos_sin_cache: torch.Tensor,
+ section_table: torch.Tensor,
+) -> None:
+ """Pure-torch mrope for triton-less installs; same in-place NeoX contract as the kernel."""
+ nnz = query.shape[0]
+ if nnz == 0:
+ return
+ rotary_dim = cos_sin_cache.shape[1]
+ half = rotary_dim // 2
+ rows = mrope_cos_sin_rows(cos_sin_cache, positions, section_table)
+ cos = rows[:, :half].unsqueeze(1).float()
+ sin = rows[:, half:].unsqueeze(1).float()
+ for t, heads in ((query, query.shape[1] // head_size), (key, key.shape[1] // head_size)):
+ v = t.view(nnz, heads, head_size)
+ lo, hi = v[..., :half].float(), v[..., half:rotary_dim].float()
+ v[..., :half] = (lo * cos - hi * sin).to(v.dtype)
+ v[..., half:rotary_dim] = (hi * cos + lo * sin).to(v.dtype)
+
+
+MROPE_LAYOUTS = ("contiguous", "interleaved", "interleaved_glm")
+
+
+def build_section_table(mrope_section: tuple[int, int, int], layout: str) -> torch.Tensor:
+ """Axis id (0=t, 1=h, 2=w) per frequency slot for a [t, h, w] section split.
+
+ contiguous: T|H|W blocks. interleaved: H at slots 1,4,.. and W at 2,5,.. until each share is used, T takes the rest.
+ interleaved_glm: round-robin over the three axes, skipping an axis once its share is used up.
+ """
+ st, sh, sw = mrope_section
+ half = st + sh + sw
+ sec = torch.zeros(half, dtype=torch.int32)
+ if layout == "contiguous":
+ sec[st : st + sh] = 1
+ sec[st + sh :] = 2
+ elif layout == "interleaved":
+ sec[1 : 3 * sh : 3] = 1
+ sec[2 : 3 * sw : 3] = 2
+ if int((sec == 1).sum()) != sh or int((sec == 2).sum()) != sw:
+ raise ValueError(f"mrope_section {mrope_section} is not representable in the interleaved layout")
+ elif layout == "interleaved_glm":
+ counts = [0, 0, 0]
+ for i in range(half):
+ ax = i % 3
+ while counts[ax] >= mrope_section[ax]:
+ ax = (ax + 1) % 3
+ sec[i] = ax
+ counts[ax] += 1
+ else:
+ raise ValueError(f"unknown mrope layout {layout!r}; expected one of {MROPE_LAYOUTS}")
+ return sec
+
+
+class MRotaryEmbedding(RotaryEmbedding):
+ """3-axis (t/h/w) rope over the parent's cos_sin_cache; section_table picks the axis row per frequency slot. Consumes positions [3, n]."""
+
+ def __init__(
+ self, *args, mrope_section: tuple, layout: str = "interleaved", **kwargs
+ ) -> None:
+ super().__init__(*args, **kwargs)
+ assert self.is_neox, "mrope is defined on the NeoX half-rotation layout"
+ half = self.rotary_dim // 2
+ assert sum(mrope_section) == half, (mrope_section, half)
+ self._section_table = build_section_table(tuple(mrope_section), layout)
+ try:
+ from freetoken.kernel.triton.rope import apply_mrope_with_cos_sin_cache_inplace
+
+ self._kernel = apply_mrope_with_cos_sin_cache_inplace
+ except ImportError:
+ self._kernel = None
+
+ def forward(
+ self,
+ positions: torch.Tensor,
+ query: torch.Tensor,
+ key: torch.Tensor,
+ ) -> Tuple[torch.Tensor, torch.Tensor]:
+ assert positions.dim() == 2, "mrope models feed [3, n] positions"
+ if self._section_table.device != query.device:
+ self._section_table = self._section_table.to(query.device)
+ positions = positions.contiguous()
+ if self._kernel is not None:
+ self._kernel(
+ positions=positions, query=query, key=key, head_size=self.head_size,
+ cos_sin_cache=self._cos_sin_cache, section_table=self._section_table,
+ )
+ else:
+ _mrope_torch(
+ positions, query, key, self.head_size,
+ self._cos_sin_cache, self._section_table,
+ )
+ return query, key
+
+
def _get_rope(
head_dim: int,
rotary_dim: int,
@@ -217,8 +325,21 @@ def get_rope(
base: float,
rope_scaling: Tuple[Tuple[str, Any], ...] | None = None,
is_neox: bool = True,
+ mrope_section: Tuple[int, ...] | None = None,
+ mrope_layout: str = "interleaved",
) -> RotaryEmbedding:
rope_map = dict(rope_scaling) if rope_scaling is not None else None
+
+ def build() -> RotaryEmbedding:
+ if mrope_section is not None:
+ assert rope_map is None or rope_map.get("rope_type", "default") == "default"
+ return MRotaryEmbedding(
+ head_dim, rotary_dim, max_position, base,
+ is_neox=is_neox, mrope_section=tuple(mrope_section),
+ layout=mrope_layout,
+ )
+ return _get_rope(head_dim, rotary_dim, max_position, base, rope_map, is_neox)
+
t = torch.tensor([])
if t.device == torch.device("meta"):
# we cannot use meta device for rope
@@ -227,8 +348,15 @@ def get_rope(
"We cannot use meta device for rope. Please call set_rope_device() first."
)
with torch.device(_ROPE_DEVICE):
- return _get_rope(head_dim, rotary_dim, max_position, base, rope_map, is_neox)
- return _get_rope(head_dim, rotary_dim, max_position, base, rope_map, is_neox)
-
-
-__all__ = ["get_rope", "RotaryEmbedding", "set_rope_device"]
+ return build()
+ return build()
+
+
+__all__ = [
+ "MROPE_LAYOUTS",
+ "MRotaryEmbedding",
+ "RotaryEmbedding",
+ "build_section_table",
+ "get_rope",
+ "set_rope_device",
+]
diff --git a/python/freetoken/llm/llm.py b/python/freetoken/llm/llm.py
index 1d4aaef0..20ae0874 100644
--- a/python/freetoken/llm/llm.py
+++ b/python/freetoken/llm/llm.py
@@ -37,23 +37,12 @@ class LLM(Scheduler):
**kwargs,
)
super().__init__(config)
- self.pending_requests: List[Tuple[List[int] | str, SamplingParams]] = []
+ self.pending_requests: List[Tuple[List[int] | str, SamplingParams, List[bytes] | None]] = []
self.status_map: Dict[int, RequestStatus] = {}
- self.mm_embeds_map: Dict[int, torch.Tensor] = {}
self.counter = 0
+ from freetoken.mm.processor import get_mm_processor
- @torch.inference_mode()
- def encode_images(
- self, pixel_values: torch.Tensor, image_position_ids: torch.Tensor
- ) -> torch.Tensor:
- """Run the vision tower + projector on processor outputs, returning the
- ``[num_image_tokens, hidden]`` soft-token embeddings (on device)."""
- model = self.engine.model
- if not hasattr(model, "encode_images"):
- raise RuntimeError(f"{type(model).__name__} does not support image inputs")
- return model.encode_images(
- pixel_values.to(self.device), image_position_ids.to(self.device)
- )
+ self._mm_processor = get_mm_processor(model_path, config.mm)
def _tokenize_one(self, prompt: List[int] | str) -> torch.Tensor:
if isinstance(prompt, str):
@@ -66,25 +55,31 @@ class LLM(Scheduler):
raise RequestAllFinished()
results: List[BaseBackendMsg] = []
added, sum_input_len = 0, 0
- for tokens_or_prompt, sampling_params in self.pending_requests:
+ for tokens_or_prompt, sampling_params, images in self.pending_requests:
if sum_input_len >= self.prefill_budget:
break
input_ids = self._tokenize_one(tokens_or_prompt)
- sum_input_len += len(input_ids)
- uid, added = self.counter + added, added + 1
- results.append(
- UserMsg(
- uid=uid,
+ msg = UserMsg(uid=0, input_ids=input_ids, sampling_params=sampling_params)
+ if images:
+ if self._mm_processor is None:
+ raise ValueError("image input is not supported for this model")
+ r = self._mm_processor.apply(input_ids, images)
+ input_ids = r.input_ids
+ msg = UserMsg(
+ uid=0,
input_ids=input_ids,
sampling_params=sampling_params,
- mm_embeds=self.mm_embeds_map.get(uid),
+ mm_items=r.mm_items,
+ mrope_positions=r.mrope_positions,
+ mrope_delta=r.mrope_delta,
)
- )
+ sum_input_len += len(input_ids)
+ uid, added = self.counter + added, added + 1
+ msg.uid = uid
+ results.append(msg)
self.status_map[uid] = RequestStatus(
uid=uid,
- input_ids=(
- input_ids.tolist() if isinstance(tokens_or_prompt, str) else tokens_or_prompt
- ),
+ input_ids=input_ids.tolist(),
output_ids=[],
)
self.counter += added
@@ -106,30 +101,18 @@ class LLM(Scheduler):
self,
prompts: List[str] | List[List[int]],
sampling_params: List[SamplingParams] | SamplingParams,
- mm_inputs: List[Dict[str, torch.Tensor] | None] | None = None,
+ images: List[List[bytes] | None] | None = None,
) -> List[Dict[str, str | List[int]]]:
- """Offline generation.
-
- ``mm_inputs`` (optional) is aligned with ``prompts``; each entry is either
- ``None`` (text-only) or a dict with ``pixel_values`` ``[N, P, 3*patch**2]`` and
- ``image_position_ids`` ``[N, P, 2]`` from the HF processor. For multimodal
- prompts pass token-id ``prompts`` containing ``image_token_id`` placeholders.
- """
+ """Offline generation; images is aligned with prompts: the raw image files of each prompt in placeholder order, or None."""
self.pending_requests = []
self.status_map = {}
- self.mm_embeds_map = {}
self.counter = 0
if isinstance(sampling_params, SamplingParams):
sampling_params = [sampling_params] * len(prompts)
- for prompt, sp in zip(prompts, sampling_params):
- self.pending_requests.append((prompt, sp))
- if mm_inputs is not None:
- for uid, mm in enumerate(mm_inputs):
- if mm is not None:
- self.mm_embeds_map[uid] = self.encode_images(
- mm["pixel_values"], mm["image_position_ids"]
- )
- torch.cuda.synchronize(self.device)
+ if images is None:
+ images = [None] * len(prompts)
+ for prompt, sp, imgs in zip(prompts, sampling_params, images, strict=True):
+ self.pending_requests.append((prompt, sp, imgs))
try:
self.run_forever()
except RequestAllFinished:
diff --git a/python/freetoken/message/__init__.py b/python/freetoken/message/__init__.py
index e9279f0b..c22cb70d 100644
--- a/python/freetoken/message/__init__.py
+++ b/python/freetoken/message/__init__.py
@@ -4,6 +4,7 @@ from .backend import (
BatchBackendMsg,
CacheRebuildBackendMsg,
ExitMsg,
+ MMItem,
UserMsg,
)
from .frontend import BaseFrontendMsg, BatchFrontendMsg, CacheRebuildReply, UserReply
@@ -26,6 +27,7 @@ __all__ = [
"BatchBackendMsg",
"CacheRebuildBackendMsg",
"ExitMsg",
+ "MMItem",
"UserMsg",
"BaseTokenizerMsg",
"BatchTokenizerMsg",
diff --git a/python/freetoken/message/backend.py b/python/freetoken/message/backend.py
index c42ecc5a..ccc1d792 100644
--- a/python/freetoken/message/backend.py
+++ b/python/freetoken/message/backend.py
@@ -1,7 +1,7 @@
from __future__ import annotations
-from dataclasses import dataclass
-from typing import Dict, List
+from dataclasses import dataclass, field
+from typing import Any, Dict, List
import torch
from freetoken.core import SamplingParams
@@ -29,14 +29,56 @@ class ExitMsg(BaseBackendMsg):
pass
+@dataclass
+class MMItem:
+ """One multimodal input of a request (an image, video or audio clip).
+
+ Family extras from the processor live in model_specific_data and read back as attributes (item.grid_thw).
+ offsets are half-open [start, end) spans in input_ids; an item may occupy several.
+ Exactly one of feature / precomputed_embeddings is set.
+ """
+
+ modality: str
+ hash: int
+ pad_value: int
+ offsets: List[List[int]]
+ feature: torch.Tensor | None = None # raw processor output (CPU), encoded by the engine
+ precomputed_embeddings: torch.Tensor | None = None # final encoder output, skips encoding
+ model_specific_data: Dict[str, Any] = field(default_factory=dict)
+
+ def __getattr__(self, name: str) -> Any:
+ data = self.__dict__.get("model_specific_data")
+ if data is not None and name in data:
+ return data[name]
+ raise AttributeError(f"'{type(self).__name__}' object has no attribute '{name}'")
+
+ @property
+ def num_tokens(self) -> int:
+ return sum(end - start for start, end in self.offsets)
+
+ def is_image(self) -> bool:
+ return self.modality == "image"
+
+ def validate(self) -> None:
+ if not self.offsets or any(end <= start for start, end in self.offsets):
+ raise ValueError(f"MMItem offsets must be non-empty half-open spans: {self.offsets}")
+ if (self.feature is None) == (self.precomputed_embeddings is None):
+ raise ValueError("MMItem needs exactly one of feature / precomputed_embeddings")
+
+ def encoder(self) -> Dict:
+ return serialize_type(self)
+
+
@dataclass
class UserMsg(BaseBackendMsg):
uid: int
input_ids: torch.Tensor # CPU 1D int32 tensor
sampling_params: SamplingParams
- # Optional precomputed multimodal soft-token embeddings (GPU tensor). Only used by
- # the in-process offline path; remains None for the (serialized) online path.
- mm_embeds: torch.Tensor | None = None
+ # per-image processor outputs, in prompt order
+ mm_items: List[MMItem] | None = None
+ # precomputed [3, len(input_ids)] mrope positions and decode delta; None for text-only requests and 1-D rope models
+ mrope_positions: torch.Tensor | None = None
+ mrope_delta: int = 0
@dataclass
diff --git a/python/freetoken/message/tokenizer.py b/python/freetoken/message/tokenizer.py
index 33b75c78..9442ee05 100644
--- a/python/freetoken/message/tokenizer.py
+++ b/python/freetoken/message/tokenizer.py
@@ -72,6 +72,7 @@ class TokenizeMsg(BaseTokenizerMsg):
sampling_params: SamplingParams
chat_template_kwargs: Dict[str, Any] | None = None
tools: List[Dict[str, Any]] | None = None
+ images: List[bytes] | None = None
@dataclass
diff --git a/python/freetoken/message/utils.py b/python/freetoken/message/utils.py
index ee92adf5..bf2c260e 100644
--- a/python/freetoken/message/utils.py
+++ b/python/freetoken/message/utils.py
@@ -12,7 +12,6 @@ _TYPE_KEY = "__type__"
# reading it as a serialized class -- without this, a request could crash the tokenizer worker.
_RAW_DICT_KEY = "__raw_dict__"
-
def _serialize_any(value: Any) -> Any:
if isinstance(value, dict):
encoded = {k: _serialize_any(v) for k, v in value.items()}
@@ -32,10 +31,16 @@ def serialize_type(self) -> Dict:
serialized = {}
if isinstance(self, torch.Tensor):
- assert self.dim() == 1, "we can only serialize 1D tensor for now"
+ assert not self.is_cuda, "wire tensors must live on CPU"
+ t = self.contiguous()
serialized["__type__"] = "Tensor"
- serialized["buffer"] = self.numpy().tobytes()
- serialized["dtype"] = str(self.dtype)
+ serialized["dtype"] = str(t.dtype)
+ # 1-D tensors omit the shape so the payload matches the legacy wire format.
+ if t.dim() != 1:
+ serialized["shape"] = list(t.shape)
+ if t.dtype == torch.bfloat16:
+ t = t.view(torch.uint16) # numpy has no bf16; ship the raw bytes
+ serialized["buffer"] = t.numpy().tobytes()
return serialized
# normal type
@@ -64,14 +69,17 @@ def _deserialize_any(cls_map: Dict[str, Type], data: Any) -> Any:
def deserialize_type(cls_map: Dict[str, Type], data: Dict) -> Any:
type_name = data["__type__"]
- # we can only serialize 1D tensor for now
if type_name == "Tensor":
buffer = data["buffer"]
dtype_str = data["dtype"].replace("torch.", "")
- np_dtype = getattr(np, dtype_str)
assert isinstance(buffer, bytes)
- np_tensor = np.frombuffer(buffer, dtype=np_dtype)
- return torch.from_numpy(np_tensor.copy())
+ is_bf16 = dtype_str == "bfloat16"
+ np_tensor = np.frombuffer(buffer, dtype=getattr(np, "uint16" if is_bf16 else dtype_str))
+ tensor = torch.from_numpy(np_tensor.copy())
+ if is_bf16:
+ tensor = tensor.view(torch.bfloat16)
+ shape = data.get("shape")
+ return tensor if shape is None else tensor.view(shape)
cls = cls_map.get(type_name)
if cls is None:
diff --git a/python/freetoken/mm/__init__.py b/python/freetoken/mm/__init__.py
new file mode 100644
index 00000000..4a7ebcd9
--- /dev/null
+++ b/python/freetoken/mm/__init__.py
@@ -0,0 +1,33 @@
+"""Multimodal input handling: pad-id scheme, per-family processors, media fetch, embedding cache."""
+
+from __future__ import annotations
+
+import torch
+
+# image spans become content pseudo ids above every real token id: the radix cache keys them by content and the model finds image rows by id >= MM_PAD_SHIFT_VALUE
+# they are clamped back into the vocab before the embedding lookup; those rows are overwritten anyway
+MM_PAD_SHIFT_VALUE = 1_000_000
+
+
+def restore_placeholder(input_ids: torch.Tensor, token_id: int) -> torch.Tensor:
+ """input_ids with every content pad id put back to the family's placeholder token."""
+ return input_ids.masked_fill(input_ids >= MM_PAD_SHIFT_VALUE, token_id)
+
+
+def mm_pad_value(item_hash: int) -> int:
+ return MM_PAD_SHIFT_VALUE + item_hash % (1 << 30)
+
+
+def check_mm_pad_shift(vocab_size: int) -> None:
+ if vocab_size > MM_PAD_SHIFT_VALUE:
+ raise ValueError(
+ f"vocab_size ({vocab_size}) exceeds MM_PAD_SHIFT_VALUE ({MM_PAD_SHIFT_VALUE}); "
+ "image pseudo-ids would alias real token ids"
+ )
+
+
+__all__ = [
+ "MM_PAD_SHIFT_VALUE",
+ "check_mm_pad_shift",
+ "mm_pad_value",
+]
diff --git a/python/freetoken/mm/config.py b/python/freetoken/mm/config.py
new file mode 100644
index 00000000..e76322a7
--- /dev/null
+++ b/python/freetoken/mm/config.py
@@ -0,0 +1,33 @@
+"""Runtime knobs of the multimodal path. The architecture side (vision_config, mrope) lives in ModelConfig."""
+
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+from typing import Any, Literal
+
+# encoder tower kinds a family can register; --mm-disable <kind> leaves that tower unbuilt and refuses its inputs
+ENCODER_KINDS = ("vision", "audio")
+# checkpoint config sections that describe encoder towers; the parser never sees the section of a tower this process does not build
+ENCODER_SECTIONS = ("vision_config", "audio_config")
+
+
+@dataclass(frozen=True)
+class MultimodalConfig:
+ # encoder kinds (ENCODER_KINDS) whose tower this process does not build; --text-model-only names them all
+ disabled_encoders: frozenset[str] = frozenset()
+ # Where encoded items wait between prefill chunks. "cpu": pinned host memory, "cuda": the device.
+ embed_cache_device: Literal["cpu", "cuda"] = "cpu"
+ # Encoder tower block weights. "host": pinned host banks streamed two blocks at a time behind the compute, "gpu": resident.
+ encoder_weights: Literal["gpu", "host"] = "host"
+ # per-image token budget; the family's MMProcessor converts it to its image processor's own limits, None keeps the checkpoint defaults
+ image_min_tokens: int | None = None
+ image_max_tokens: int | None = None
+ # extra keyword arguments the family's MMProcessor passes to the image processor call, after the token budget
+ processor_kwargs: dict[str, Any] = field(default_factory=dict)
+
+ @property
+ def text_model_only(self) -> bool:
+ return set(ENCODER_KINDS) <= self.disabled_encoders
+
+
+__all__ = ["ENCODER_KINDS", "ENCODER_SECTIONS", "MultimodalConfig"]
diff --git a/python/freetoken/mm/encoder_cache.py b/python/freetoken/mm/encoder_cache.py
new file mode 100644
index 00000000..dab51bf0
--- /dev/null
+++ b/python/freetoken/mm/encoder_cache.py
@@ -0,0 +1,87 @@
+"""Encoder embedding cache keyed by content hash; an entry lives while any request still has rows of it to gather."""
+
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+from typing import Dict
+
+import torch
+
+from freetoken.utils import init_logger
+
+logger = init_logger(__name__)
+
+
+@dataclass
+class _Entry:
+ tensor: torch.Tensor | None = None
+ # uid -> embedding rows the request still has to gather, summed over its occurrences of the image
+ remaining: Dict[int, int] = field(default_factory=dict)
+
+
+class EncoderCache:
+ def __init__(self, storage: str = "cpu") -> None:
+ assert storage in ("cpu", "cuda")
+ self._storage = storage
+ self._entries: Dict[int, _Entry] = {}
+
+ def register(self, item_hash: int, uid: int, rows: int) -> None:
+ """Claim rows of the image for uid before any of its chunks run; a repeated image adds up."""
+ if rows <= 0:
+ return
+ entry = self._entries.setdefault(item_hash, _Entry())
+ entry.remaining[uid] = entry.remaining.get(uid, 0) + rows
+
+ def has(self, item_hash: int) -> bool:
+ entry = self._entries.get(item_hash)
+ return entry is not None and entry.tensor is not None
+
+ def put(self, item_hash: int, embedding: torch.Tensor) -> None:
+ entry = self._entries.get(item_hash)
+ assert entry is not None, f"image {item_hash} encoded before any request registered it"
+ if entry.tensor is not None:
+ return
+ if self._storage == "cpu":
+ stored = torch.empty(
+ embedding.shape, dtype=embedding.dtype, device="cpu",
+ pin_memory=torch.cuda.is_available(),
+ )
+ stored.copy_(embedding, non_blocking=True)
+ else:
+ stored = embedding
+ entry.tensor = stored
+
+ def get_slice(self, item_hash: int, lo: int, hi: int, device: torch.device) -> torch.Tensor:
+ view = self._entries[item_hash].tensor[lo:hi]
+ if view.device == device:
+ return view
+ return view.to(device, non_blocking=True)
+
+ def consume(self, item_hash: int, uid: int, rows: int) -> None:
+ """Account rows uid gathered; the last row of the last holder frees the entry."""
+ entry = self._entries[item_hash]
+ left = entry.remaining[uid] - rows
+ assert left >= 0, f"request {uid} gathered more rows of image {item_hash} than it registered"
+ if left:
+ entry.remaining[uid] = left
+ return
+ del entry.remaining[uid]
+ if not entry.remaining:
+ del self._entries[item_hash]
+
+ def release(self, uid: int, hashes: list[int]) -> None:
+ """Drop uid's claims whatever it gathered (abort); entries nobody else holds are freed."""
+ for h in hashes:
+ entry = self._entries.get(h)
+ if entry is None:
+ continue
+ entry.remaining.pop(uid, None)
+ if not entry.remaining:
+ del self._entries[h]
+
+ def stats(self) -> tuple[int, int]:
+ stored = [e.tensor for e in self._entries.values() if e.tensor is not None]
+ return len(stored), sum(t.numel() * t.element_size() for t in stored)
+
+
+__all__ = ["EncoderCache"]
diff --git a/python/freetoken/mm/media.py b/python/freetoken/mm/media.py
new file mode 100644
index 00000000..5c3c4ef7
--- /dev/null
+++ b/python/freetoken/mm/media.py
@@ -0,0 +1,112 @@
+"""Client-supplied image handling: ref collection, the accept gate, and byte fetching.
+
+Raises plain ValueError on bad input; the server layer maps it to its wire error type.
+"""
+
+from __future__ import annotations
+
+import base64
+from typing import TYPE_CHECKING, Any
+
+if TYPE_CHECKING:
+ from freetoken.server.args import ServerArgs
+
+_MAX_IMAGE_BYTES = 32 << 20
+
+
+def collect_image_refs(messages: list[dict[str, Any]]) -> list[dict[str, Any]]:
+ """Pop the image refs out of rendered messages in prompt order; the bare {"type": "image"} parts stay for the chat template."""
+ refs: list[dict[str, Any]] = []
+ for m in messages:
+ content = m.get("content")
+ if not isinstance(content, list):
+ continue
+ for part in content:
+ if isinstance(part, dict) and part.get("type") == "image":
+ ref = part.pop("freetoken_ref", None)
+ if ref:
+ refs.append(ref)
+ return refs
+
+
+def image_reject_reason(config: ServerArgs) -> str | None:
+ """None when this server can accept image inputs, else the client-facing reason."""
+ if "image" in config.served_modalities:
+ return None
+ if config.mm.text_model_only:
+ return "image input is disabled on this server (--text-model-only)"
+ if "vision" in config.mm.disabled_encoders:
+ return "image input is disabled on this server (--mm-disable)"
+ return "the served model has no vision support"
+
+
+def _check_media_domain(url: str, config: ServerArgs) -> None:
+ """Reject a URL whose hostname is not in --allowed-media-domains; empty allowlist admits any domain."""
+ from urllib.parse import urlparse
+
+ raw = config.allowed_media_domains
+ allowed = {d.strip().lower().rstrip(".") for d in raw.split(",") if d.strip()}
+ if not allowed:
+ return
+ host = (urlparse(url).hostname or "").rstrip(".")
+ if host not in allowed:
+ raise ValueError(
+ f"the URL must be from one of the allowed domains: {sorted(allowed)}; "
+ f"input URL domain: {host or '<none>'}"
+ )
+
+
+def _load_local_media(url: str, config: ServerArgs) -> bytes:
+ """Read a file:// ref; the resolved path must be a strict subpath of --allowed-local-media-path."""
+ from pathlib import Path
+ from urllib.parse import urlparse
+ from urllib.request import url2pathname
+
+ root = config.allowed_local_media_path
+ if not root:
+ raise ValueError("cannot load local files without --allowed-local-media-path")
+ spec = urlparse(url)
+ filepath = Path(url2pathname((spec.netloc or "") + (spec.path or "")))
+ # resolve() follows symlinks, so a link inside the root escaping it is rejected too
+ resolved = filepath.resolve()
+ if Path(root).resolve() not in resolved.parents:
+ raise ValueError(
+ f"the file path {filepath} must be a subpath of "
+ f"--allowed-local-media-path {root}"
+ )
+ return resolved.read_bytes()
+
+
+async def fetch_image_bytes(refs: list[dict[str, Any]], config: ServerArgs) -> list[bytes]:
+ """Resolve collected image refs (URLs / base64) to raw bytes, in order."""
+ out: list[bytes] = []
+ for ref in refs:
+ data = ref.get("data") or ""
+ try:
+ if ref.get("kind") == "b64":
+ out.append(base64.b64decode(data))
+ elif data.startswith("data:"):
+ out.append(base64.b64decode(data.split(",", 1)[1]))
+ elif data.startswith(("http://", "https://")):
+ import httpx
+
+ _check_media_domain(data, config)
+ # redirect targets are not re-checked against the allowlist
+ async with httpx.AsyncClient(timeout=30.0, follow_redirects=True) as client:
+ resp = await client.get(data)
+ resp.raise_for_status()
+ if len(resp.content) > _MAX_IMAGE_BYTES:
+ raise ValueError(f"image exceeds {_MAX_IMAGE_BYTES} bytes")
+ out.append(resp.content)
+ elif data.startswith("file://"):
+ out.append(_load_local_media(data, config))
+ else:
+ raise ValueError(
+ "unsupported image source (expect http(s)/file url or base64)"
+ )
+ except Exception as exc: # noqa: BLE001 -- input-driven, client-classifiable
+ raise ValueError(f"could not load image: {exc}") from exc
+ return out
+
+
+__all__ = ["collect_image_refs", "fetch_image_bytes", "image_reject_reason"]
diff --git a/python/freetoken/mm/processor.py b/python/freetoken/mm/processor.py
new file mode 100644
index 00000000..48211a5b
--- /dev/null
+++ b/python/freetoken/mm/processor.py
@@ -0,0 +1,215 @@
+"""Per-family multimodal processors and the architecture registry; MMProcessor.apply is the image half of tokenization."""
+
+from __future__ import annotations
+
+from abc import ABC, abstractmethod
+import hashlib
+import importlib
+import io
+import threading
+from dataclasses import dataclass
+from typing import Any
+
+import torch
+
+from freetoken.message import MMItem
+from freetoken.mm.config import MultimodalConfig
+from freetoken.mm import check_mm_pad_shift
+
+
+@dataclass
+class PromptReplacement:
+ """The token sequence that replaces one placeholder; is_embed marks the positions that take image embeddings (None = all of them)."""
+
+ full: list[int]
+ is_embed: list[bool] | None = None
+
+ @classmethod
+ def select_token_id(cls, full: list[int], embed_token_id: int) -> PromptReplacement:
+ return cls(full, [t == embed_token_id for t in full])
+
+ def embed_spans(self) -> list[list[int]]:
+ """Half-open [lo, hi) runs of embedding positions, relative to the start of full."""
+ mask = self.is_embed if self.is_embed is not None else [True] * len(self.full)
+ spans: list[list[int]] = []
+ lo: int | None = None
+ for i, flag in enumerate([*mask, False]):
+ if flag and lo is None:
+ lo = i
+ elif not flag and lo is not None:
+ spans.append([lo, i])
+ lo = None
+ return spans
+
+
+@dataclass
+class MMResult:
+ input_ids: torch.Tensor
+ mm_items: list[MMItem]
+ mrope_positions: torch.Tensor | None
+ mrope_delta: int
+
+
+def _find_all(ids: list[int], target: list[int]) -> list[int]:
+ """Start index of every non-overlapping occurrence of target."""
+ n = len(target)
+ found: list[int] = []
+ i = 0
+ while i <= len(ids) - n:
+ if ids[i : i + n] == target:
+ found.append(i)
+ i += n
+ else:
+ i += 1
+ return found
+
+
+class MMProcessor(ABC):
+ """How a checkpoint's images become items, tokens and positions; apply is the family-agnostic driver, subclasses fill the hooks.
+
+ Built from the checkpoint config alone and import-light: tokenizer workers call apply without the model layer, the engine calls dummy_items.
+ Image preprocessing must stay on the CPU with one fixed implementation: radix keys and the embedding cache are content hashes of pixel_values, and resize results differ across backends/devices.
+ """
+
+ # the token sequence the chat template renders once per image
+ placeholder: list[int]
+
+ def __init__(self, model_path: str, mm: MultimodalConfig) -> None:
+ self.model_path = model_path
+ self.mm = mm
+ self._image_processor_instance: Any = None
+ self._image_processor_lock = threading.Lock()
+
+ def _image_processor(self) -> Any:
+ """The checkpoint's image processor, loaded on first use (tokenizer workers share the instance across threads)."""
+ with self._image_processor_lock:
+ if self._image_processor_instance is None:
+ from transformers import AutoImageProcessor
+
+ self._image_processor_instance = AutoImageProcessor.from_pretrained(self.model_path)
+ return self._image_processor_instance
+
+ def get_mm_processor_kwargs(self, mm: MultimodalConfig) -> dict[str, Any]:
+ """Keyword arguments for the checkpoint processor call: the family's reading of the mm knobs, then mm.processor_kwargs on top; families with a token budget override this."""
+ return dict(mm.processor_kwargs)
+
+ @abstractmethod
+ def process(self, images: list[Any]) -> list[MMItem]:
+ """One MMItem per image with feature and hash; offsets are assigned by apply."""
+
+ @abstractmethod
+ def prompt_replacement(self, item: MMItem) -> PromptReplacement:
+ """The token sequence that stands in for this item's placeholder in the prompt."""
+
+ def positions(self, length: int, items: list[MMItem]) -> tuple[torch.Tensor, int] | None:
+ """[3, length] t/h/w rope positions and the decode delta; None for 1-D rope families."""
+ return None
+
+ @abstractmethod
+ def dummy_items(self, dtype: torch.dtype, device: torch.device) -> list[MMItem]:
+ """The smallest encodable item of each modality the family serves, for warming the encoders."""
+
+ def apply(self, input_ids: torch.Tensor, images: list[bytes]) -> MMResult:
+ """Swap each image placeholder for its replacement sequence and precompute rope positions."""
+ from PIL import Image
+
+ ids = input_ids.tolist()
+ target = self.placeholder
+ slots = _find_all(ids, target)
+ if len(slots) != len(images):
+ raise ValueError(
+ f"prompt renders {len(slots)} image placeholders but the request "
+ f"carries {len(images)} images"
+ )
+
+ pils = [Image.open(io.BytesIO(raw)).convert("RGB") for raw in images]
+ items = self.process(pils)
+ out: list[int] = []
+ cursor = 0
+ for slot, item in zip(slots, items):
+ repl = self.prompt_replacement(item)
+ out.extend(ids[cursor:slot])
+ base = len(out)
+ full = list(repl.full)
+ spans = repl.embed_spans()
+ # embedding slots carry the content pad id so radix keys and the model's scatter mask see the image
+ for lo, hi in spans:
+ full[lo:hi] = [item.pad_value] * (hi - lo)
+ out.extend(full)
+ item.offsets = [[base + lo, base + hi] for lo, hi in spans]
+ item.validate()
+ cursor = slot + len(target)
+ out.extend(ids[cursor:])
+ new_ids = torch.tensor(out, dtype=input_ids.dtype)
+
+ positions, delta = self.positions(len(out), items) or (None, 0)
+ return MMResult(new_ids, items, positions, delta)
+
+
+def image_positions(
+ length: int, blocks: list[tuple[int, int, int, int]]
+) -> tuple[torch.Tensor, int]:
+ """t/h/w positions: text advances all three rows, an image freezes t and spreads h/w over its grid, then the running position jumps by max(h, w).
+
+ blocks are (offset, n_tokens, grid_h, grid_w) in prompt order; delta shifts decode positions (sequence index + delta)."""
+ pos = torch.empty((3, length), dtype=torch.int32)
+ st = 0
+ cursor = 0
+ for offset, n_tokens, h_m, w_m in blocks:
+ run = offset - cursor
+ if run:
+ text = torch.arange(st, st + run, dtype=torch.int32)
+ pos[:, cursor:offset] = text
+ st += run
+ cursor = offset
+ idx = torch.arange(n_tokens, dtype=torch.int32)
+ pos[0, cursor : cursor + n_tokens] = st
+ pos[1, cursor : cursor + n_tokens] = st + idx // w_m
+ pos[2, cursor : cursor + n_tokens] = st + idx % w_m
+ st += max(h_m, w_m)
+ cursor += n_tokens
+ run = length - cursor
+ if run:
+ pos[:, cursor:] = torch.arange(st, st + run, dtype=torch.int32)
+ st += run
+ delta = int(pos.max().item()) + 1 - length
+ return pos, delta
+
+
+def content_hash(feature: torch.Tensor, extra: bytes = b"") -> int:
+ """62-bit hash of the wire bytes plus family extras (e.g. the grid, so transposed images differ)."""
+ buf = feature.contiguous()
+ if buf.dtype == torch.bfloat16:
+ buf = buf.view(torch.uint16)
+ hasher = hashlib.sha256(memoryview(buf.numpy()))
+ hasher.update(extra)
+ return int.from_bytes(hasher.digest()[:8], "little") % (1 << 62)
+
+
+def get_mm_processor(model_path: str, mm: MultimodalConfig | None = None) -> MMProcessor | None:
+ """A processor for the checkpoint under the mm knobs, or None when no encoder is left to serve: the family registers none, the checkpoint ships none, or mm.disabled_encoders names them all; callers keep the instance, it holds the loaded image processor."""
+ from freetoken.models.register import get_model_spec
+ from freetoken.utils import cached_load_hf_config
+
+ try:
+ config = cached_load_hf_config(model_path)
+ spec = get_model_spec((config.architectures or [None])[0])
+ except Exception: # noqa: BLE001 -- missing/foreign config or unknown architecture: no multimodal input
+ return None
+ mm = mm or MultimodalConfig()
+ served = [e for e in spec.encoders if getattr(config, e.config_key, None) is not None and e.kind not in mm.disabled_encoders]
+ if spec.mm_processor is None or not served:
+ return None
+ check_mm_pad_shift(config.text_config.vocab_size)
+ module, _, cls = spec.mm_processor.partition(":")
+ return getattr(importlib.import_module(module), cls)(config, model_path, mm)
+
+
+__all__ = [
+ "MMProcessor",
+ "MMResult",
+ "PromptReplacement",
+ "content_hash",
+ "get_mm_processor",
+ "image_positions",
+]
diff --git a/python/freetoken/mm/processors/qwen_vl.py b/python/freetoken/mm/processors/qwen_vl.py
new file mode 100644
index 00000000..669f7163
--- /dev/null
+++ b/python/freetoken/mm/processors/qwen_vl.py
@@ -0,0 +1,95 @@
+"""Qwen VL family image processor (Qwen2-VL through Qwen3.8): patch-grid ViT with a spatial merger, 3-axis rope positions."""
+
+from __future__ import annotations
+
+import struct
+from typing import Any
+
+import torch
+
+from freetoken.message import MMItem
+from freetoken.mm import mm_pad_value
+from freetoken.mm.config import MultimodalConfig
+from freetoken.mm.processor import MMProcessor, PromptReplacement, content_hash, image_positions
+
+
+class QwenVLMMProcessor(MMProcessor):
+ def __init__(self, hf_config: Any, model_path: str, mm: MultimodalConfig) -> None:
+ super().__init__(model_path, mm)
+ vc = hf_config.vision_config
+ self.image_token_id = hf_config.image_token_id
+ self.placeholder = [self.image_token_id]
+ self.merge = vc.spatial_merge_size
+ self.pixels_per_token = (vc.patch_size * vc.spatial_merge_size) ** 2
+ self.patch_dim = vc.in_channels * vc.temporal_patch_size * vc.patch_size**2
+ self.is_mrope = "mrope_section" in hf_config.text_config.rope_parameters
+
+ def get_mm_processor_kwargs(self, mm: MultimodalConfig) -> dict[str, Any]:
+ kwargs: dict[str, Any] = {"return_tensors": "pt"}
+ if mm.image_min_tokens is not None or mm.image_max_tokens is not None:
+ # the budget travels as pixel areas in size.shortest_edge / longest_edge; bare min/max_pixels kwargs are ignored
+ size = dict(self._image_processor().size)
+ if mm.image_min_tokens is not None:
+ size["shortest_edge"] = mm.image_min_tokens * self.pixels_per_token
+ if mm.image_max_tokens is not None:
+ size["longest_edge"] = mm.image_max_tokens * self.pixels_per_token
+ kwargs["size"] = size
+ return {**kwargs, **mm.processor_kwargs}
+
+ def process(self, images: list[Any]) -> list[MMItem]:
+ processor = self._image_processor()
+ kwargs = self.get_mm_processor_kwargs(self.mm)
+ items: list[MMItem] = []
+ for pil in images:
+ out = processor(images=pil, **kwargs)
+ pixel_values = out["pixel_values"].to(torch.bfloat16)
+ grid = out["image_grid_thw"][0].tolist()
+ t, h, w = int(grid[0]), int(grid[1]), int(grid[2])
+ if t != 1:
+ raise ValueError("video/temporal input is not supported")
+ # hash the bf16 wire buffer: half the bytes, and the exact tensor the cache and radix key agree on
+ item_hash = content_hash(pixel_values, struct.pack("<3i", t, h, w))
+ items.append(
+ MMItem(
+ modality="image",
+ hash=item_hash,
+ pad_value=mm_pad_value(item_hash),
+ offsets=[],
+ feature=pixel_values,
+ model_specific_data={"grid_thw": [t, h, w]},
+ )
+ )
+ return items
+
+ def prompt_replacement(self, item: MMItem) -> PromptReplacement:
+ t, h, w = item.grid_thw
+ n_tokens = (t * h * w) // (self.merge * self.merge)
+ return PromptReplacement([self.image_token_id] * n_tokens)
+
+ def image_grid(self, item: MMItem) -> tuple[int, int]:
+ _, h, w = item.grid_thw
+ return h // self.merge, w // self.merge
+
+ def positions(self, length: int, items: list[MMItem]) -> tuple[torch.Tensor, int] | None:
+ if not self.is_mrope:
+ return None
+ blocks = []
+ for item in items:
+ ((start, end),) = item.offsets
+ rows, cols = self.image_grid(item)
+ blocks.append((start, end - start, rows, cols))
+ return image_positions(length, blocks)
+
+ def dummy_items(self, dtype: torch.dtype, device: torch.device) -> list[MMItem]:
+ merge = self.merge
+ return [MMItem(
+ modality="image",
+ hash=0,
+ pad_value=0,
+ offsets=[[0, 1]],
+ feature=torch.zeros(merge * merge, self.patch_dim, dtype=dtype, device=device),
+ model_specific_data={"grid_thw": [1, merge, merge]},
+ )]
+
+
+__all__ = ["QwenVLMMProcessor"]
diff --git a/python/freetoken/models/blocks.py b/python/freetoken/models/blocks.py
index 0fa5fe4d..4f039fd3 100644
--- a/python/freetoken/models/blocks.py
+++ b/python/freetoken/models/blocks.py
@@ -2,7 +2,9 @@ from __future__ import annotations
from abc import ABC, abstractmethod
from contextlib import contextmanager
-from typing import TYPE_CHECKING
+from typing import TYPE_CHECKING, Protocol, runtime_checkable
+
+import torch
from freetoken.layers import (
BaseOP,
@@ -33,6 +35,29 @@ class BaseLLMModel(ABC, BaseOP):
yield
+@runtime_checkable
+class SupportsMultimodal(Protocol):
+ """What the engine asks of a model that serves multimodal items; the text forward takes its token embeddings from embed_input_ids."""
+
+ def encode(self, item: MMItem) -> torch.Tensor:
+ """``[item.num_tokens, D]`` soft tokens for one item of any modality the family serves, on the model device."""
+ ...
+
+ def place_encoder_weights(self, mode: str) -> None:
+ """Every encoder tower's weights: ``gpu`` resident or ``host`` streamed from pinned banks."""
+ ...
+
+
+def embed_input_ids(embed_tokens, input_ids: torch.Tensor, batch: Batch) -> torch.Tensor:
+ """Token embeddings of the batch; on a chunk with multimodal rows, batch.mm_embeds' leading columns replace the rows batch.mm_rows."""
+ if batch.mm_embeds is None:
+ return embed_tokens.forward(input_ids)
+ # multimodal rows carry content pad ids above the vocab: clamp for the lookup, the copy overwrites those rows
+ x = embed_tokens.forward(input_ids.clamp(max=embed_tokens.num_embeddings - 1))
+ x.index_copy_(0, batch.mm_rows, batch.mm_embeds[:, : x.shape[1]].to(x.dtype))
+ return x
+
+
class GatedMLP(BaseOP):
def __init__(self, config: ModelConfig, *, quant_config: QuantConfig | None = None, prefix: str = ""):
self.gate_up_proj = LinearColParallelMerged(
diff --git a/python/freetoken/models/config.py b/python/freetoken/models/config.py
index 07a16df5..273662a9 100644
--- a/python/freetoken/models/config.py
+++ b/python/freetoken/models/config.py
@@ -1,21 +1,11 @@
from __future__ import annotations
-import os
from dataclasses import dataclass
from typing import Any, ClassVar, Dict, List, Literal, Tuple, TypeAlias
from freetoken.attention.base import AttnType
-# State-dict key prefixes for the (optional) vision stack. Used both to drop the vision
-# config (so the tower is never built) and to skip the matching tensors in the FTW reader.
-VISION_KEY_PREFIXES = ("vision_tower.", "embed_vision.")
-_VISION_TRUE = {"1", "true", "yes", "on"}
-
-
-def vision_load_enabled() -> bool:
- """Vision is opt-in (default OFF). The vision tower + multimodal embedder are ~1 GiB of
- resident, never-quantized (bf16) GPU weights that text-only serving never touches, so we
- skip building and loading them unless ``FREETOKEN_LOAD_VISION=1`` is set."""
- return os.getenv("FREETOKEN_LOAD_VISION", "0").strip().lower() in _VISION_TRUE
+# State-dict key prefixes of the vision stack; load_weight drops them when the engine serves text-only.
+VISION_KEY_PREFIXES = ("vision_tower.", "embed_vision.", "visual.")
def detect_expert_quant(hf_config: Any) -> str:
@@ -98,6 +88,18 @@ class RotaryConfig:
max_position: int
base: float
scaling: Dict[str, Any] | None
+ # 3-axis rope sections; None keeps the 1-D rope path, set only when the model serves vision
+ mrope_section: list | None = None
+ mrope_layout: str = "contiguous" # see freetoken.layers.rotary.build_section_table
+
+
+def mrope_layout_from_rope_params(rope_params: Any) -> str:
+ """rope_parameters flags -> layout name: mrope_interleaved_glm, else mrope_interleaved, else contiguous."""
+ if rope_params.get("mrope_interleaved_glm"):
+ return "interleaved_glm"
+ if rope_params.get("mrope_interleaved"):
+ return "interleaved"
+ return "contiguous"
@dataclass(frozen=True)
@@ -357,6 +359,14 @@ class ModelConfig:
def is_multimodal(self) -> bool:
return self.vision_config is not None
+ @property
+ def model_is_mrope(self) -> bool:
+ """True when any full-attention layer uses 3-axis (t/h/w) rope positions."""
+ return any(
+ getattr(getattr(g, "rotary_config", None), "mrope_section", None) is not None
+ for g in self.attention_groups
+ )
+
@property
def has_hybrid_attention(self) -> bool:
return len(self.attention_groups) > 1
diff --git a/python/freetoken/models/gemma4/config.py b/python/freetoken/models/gemma4/config.py
index 057e0a55..94718603 100644
--- a/python/freetoken/models/gemma4/config.py
+++ b/python/freetoken/models/gemma4/config.py
@@ -9,7 +9,6 @@ from freetoken.models.config import (
RotaryConfig,
SWAAttentionGroupConfig,
detect_expert_quant,
- vision_load_enabled,
)
@@ -44,15 +43,6 @@ def _parse_vision_config(top_cfg: Any, text_hidden_size: int) -> VisionConfig |
vc = getattr(top_cfg, "vision_config", None)
if vc is None:
return None
- # Vision is opt-in (default OFF). The vision tower + multimodal embedder are ~1 GiB of
- # resident, never-quantized (bf16) GPU weights that text-only serving never touches:
- # forward() ignores them and _merge_multimodal is a no-op without image embeds. Default
- # to text-only so that VRAM goes to KV / the expert cache; set FREETOKEN_LOAD_VISION=1
- # to load it. Returning None here is the single switch -- both the model build (model.py)
- # and weight loading (weight.py `include_vision = config.is_multimodal`) flow through
- # parse_config, so is_multimodal flips off consistently across both.
- if not vision_load_enabled():
- return None
rope_params = getattr(vc, "rope_parameters", None) or {}
act = getattr(vc, "hidden_activation", "gelu_pytorch_tanh")
act = "gelu_tanh" if "tanh" in act else act
diff --git a/python/freetoken/models/gemma4/model.py b/python/freetoken/models/gemma4/model.py
index 83e49c4d..d151540b 100644
--- a/python/freetoken/models/gemma4/model.py
+++ b/python/freetoken/models/gemma4/model.py
@@ -123,16 +123,8 @@ class Gemma4ForCausalLM(BaseLLMModel):
convert_gemma4_to_gguf(self, config)
@torch.inference_mode()
- def encode_images(
- self, pixel_values: torch.Tensor, image_position_ids: torch.Tensor
- ) -> torch.Tensor:
- """Run the vision tower + projector. Returns ``[num_valid_soft_tokens, text_hidden]``.
-
- ``pixel_values``: ``[num_images, num_patches, 3*patch**2]``;
- ``image_position_ids``: ``[num_images, num_patches, 2]`` with ``(-1, -1)`` padding.
- """
- features = self.vision_tower.forward(pixel_values, image_position_ids)
- return self.embed_vision.forward(features)
+ def encode(self, item) -> torch.Tensor:
+ raise NotImplementedError("gemma4 image input needs an MMProcessor (freetoken.mm.processors)")
def forward(self) -> torch.Tensor:
output = self.model.forward(get_global_ctx().batch.input_ids)
diff --git a/python/freetoken/models/gemma4/weight.py b/python/freetoken/models/gemma4/weight.py
index 740f4660..4413b67f 100644
--- a/python/freetoken/models/gemma4/weight.py
+++ b/python/freetoken/models/gemma4/weight.py
@@ -133,20 +133,19 @@ def iter_weights(
include_moe_experts: bool,
include_non_moe: bool,
) -> Iterator[tuple[str, torch.Tensor]]:
- def rename_key(raw_name: str, *, include_vision: bool) -> str | None:
+ def rename_key(raw_name: str) -> str | None:
prefix = "model.language_model."
if raw_name.startswith(prefix):
return _rename_language_key(raw_name)
if raw_name.startswith("language_model."):
return _rename_language_key(raw_name)
- if include_vision:
- if raw_name.startswith("model.vision_tower."):
- return ("vision_tower." + raw_name[len("model.vision_tower.") :]).replace(
- ".linear.",
- ".",
- )
- if raw_name.startswith("model.embed_vision."):
- return "embed_vision." + raw_name[len("model.embed_vision.") :]
+ if raw_name.startswith("model.vision_tower."):
+ return ("vision_tower." + raw_name[len("model.vision_tower.") :]).replace(
+ ".linear.",
+ ".",
+ )
+ if raw_name.startswith("model.embed_vision."):
+ return "embed_vision." + raw_name[len("model.embed_vision.") :]
return None
def merge_info(key: str) -> tuple[str, MergeRule] | None:
@@ -160,7 +159,6 @@ def iter_weights(
if tp_info.size > 1:
raise NotImplementedError("Gemma 4 weight loading currently supports TP=1 only")
- include_vision = config.is_multimodal
k_eq_v_layers = {
layer_id
for layer_id in range(config.num_layers)
@@ -178,7 +176,7 @@ def iter_weights(
):
with safetensors.safe_open(file, framework="pt", device=str(device)) as f:
for raw_name in f.keys():
- name = rename_key(raw_name, include_vision=include_vision)
+ name = rename_key(raw_name)
if name is None:
continue
diff --git a/python/freetoken/models/qwen3/attention.py b/python/freetoken/models/qwen3/attention.py
index 039dc50d..5ec37013 100644
--- a/python/freetoken/models/qwen3/attention.py
+++ b/python/freetoken/models/qwen3/attention.py
@@ -57,6 +57,12 @@ class Qwen3Attention(BaseOP):
if config.rotary_config.scaling
else None
),
+ mrope_section=(
+ tuple(config.rotary_config.mrope_section)
+ if config.rotary_config.mrope_section is not None
+ else None
+ ),
+ mrope_layout=config.rotary_config.mrope_layout,
)
self.o_proj = LinearOProj(
head_dim * config.num_qo_heads,
@@ -76,7 +82,7 @@ class Qwen3Attention(BaseOP):
self.q_norm.forward_inplace(q.view(-1, self.num_qo_heads, self.head_dim))
if self.k_norm is not None:
self.k_norm.forward_inplace(k.view(-1, self.num_kv_heads, self.head_dim))
- q, k = self.rotary.forward(ctx.batch.positions, q, k)
+ q, k = self.rotary.forward(ctx.batch.get_attn_positions(), q, k)
q = q.view(-1, self.num_qo_heads, self.head_dim)
o = ctx.attn_backend.forward(q, k, v, self.layer_id, ctx.batch)
return self.o_proj.forward(o.view(-1, self.qo_attn_dim))
diff --git a/python/freetoken/models/qwen3/model.py b/python/freetoken/models/qwen3/model.py
index dc35930d..6ac7d2fa 100644
--- a/python/freetoken/models/qwen3/model.py
+++ b/python/freetoken/models/qwen3/model.py
@@ -67,8 +67,10 @@ class Qwen3Model(BaseOP):
class Qwen3ForCausalLM(BaseLLMModel):
+ model_cls = Qwen3Model
+
def __init__(self, config: ModelConfig):
- self.model = Qwen3Model(config)
+ self.model = self.model_cls(config)
self.lm_head = ParallelLMHead(
num_embeddings=config.vocab_size,
embedding_dim=config.hidden_size,
diff --git a/python/freetoken/models/qwen3_5_moe/__init__.py b/python/freetoken/models/qwen3_5_moe/__init__.py
index 8620ce9c..9acc5aac 100644
--- a/python/freetoken/models/qwen3_5_moe/__init__.py
+++ b/python/freetoken/models/qwen3_5_moe/__init__.py
@@ -1,9 +1,17 @@
from .config import parse_config
-from .model import Qwen3_5MoEForCausalLM
+from .model import (
+ Qwen3_5ForCausalLM,
+ Qwen3_5ForConditionalGeneration,
+ Qwen3_5MoeForCausalLM,
+ Qwen3_5MoeForConditionalGeneration,
+)
from .weight import iter_expert_pieces, iter_weights, iter_weights_parallel, nvfp4_expert_spec
__all__ = [
- "Qwen3_5MoEForCausalLM",
+ "Qwen3_5ForCausalLM",
+ "Qwen3_5ForConditionalGeneration",
+ "Qwen3_5MoeForCausalLM",
+ "Qwen3_5MoeForConditionalGeneration",
"parse_config",
"iter_weights",
"iter_weights_parallel",
diff --git a/python/freetoken/models/qwen3_5_moe/attention.py b/python/freetoken/models/qwen3_5_moe/attention.py
index 12a91bb3..7a9e6a33 100644
--- a/python/freetoken/models/qwen3_5_moe/attention.py
+++ b/python/freetoken/models/qwen3_5_moe/attention.py
@@ -55,6 +55,12 @@ class Qwen3_5Attention(BaseOP):
if config.rotary_config.scaling
else None
),
+ mrope_section=(
+ tuple(config.rotary_config.mrope_section)
+ if config.rotary_config.mrope_section is not None
+ else None
+ ),
+ mrope_layout=config.rotary_config.mrope_layout,
)
self.o_proj = LinearReplicated(
self.qo_attn_dim, config.hidden_size, has_bias=False,
@@ -64,7 +70,7 @@ class Qwen3_5Attention(BaseOP):
def _project(self, x: torch.Tensor):
"""Returns (q, k, v, gate): q [N, num_q, head_dim] post qk-norm+rope,
k [N, num_kv*head_dim] post norm+rope, v [N, num_kv*head_dim], gate [N, num_q*head_dim]."""
- positions = get_global_ctx().batch.positions
+ positions = get_global_ctx().batch.get_attn_positions()
qkv = self.qkv_proj.forward(x)
qg, k, v = torch.split(qkv, self._qkv_split, dim=-1)
qg = qg.view(-1, self.num_q, self.head_dim * 2)
diff --git a/python/freetoken/models/qwen3_5_moe/config.py b/python/freetoken/models/qwen3_5_moe/config.py
index cc5a62b9..70cfbcf2 100644
--- a/python/freetoken/models/qwen3_5_moe/config.py
+++ b/python/freetoken/models/qwen3_5_moe/config.py
@@ -8,7 +8,9 @@ from freetoken.models.config import (
LinearGatedDeltaGroupConfig,
ModelConfig,
RotaryConfig,
+ mrope_layout_from_rope_params,
)
+from freetoken.models.qwen3_vl.config import parse_vision_config
def _expert_quant(hf_config: Any, text: Any) -> tuple[str, tuple[int, int] | None]:
@@ -74,12 +76,20 @@ def parse_config(hf_config: Any) -> ModelConfig:
full_ids = tuple(i for i, t in enumerate(layer_types) if t == "full_attention")
linear_ids = tuple(i for i, t in enumerate(layer_types) if t == "linear_attention")
+ # 3-axis rope only with vision; text-only serving keeps the 1-D partial rope and the decode-graph layout
+ vision_config = parse_vision_config(hf_config)
full_rotary = RotaryConfig(
head_dim=head_dim,
rotary_dim=rotary_dim,
max_position=text.max_position_embeddings,
base=rope_theta,
scaling=rope_scaling,
+ mrope_section=(
+ list(rope_params["mrope_section"])
+ if vision_config is not None and "mrope_section" in rope_params
+ else None
+ ),
+ mrope_layout=mrope_layout_from_rope_params(rope_params),
)
full_group = FullAttentionGroupConfig(
name="full",
@@ -127,7 +137,7 @@ def parse_config(hf_config: Any) -> ModelConfig:
use_qk_norm=True,
model_type=getattr(hf_config, "model_type", "qwen3_5_moe"),
architectures=getattr(hf_config, "architectures", ["Qwen3_5MoeForConditionalGeneration"]),
- vision_config=None, # text-only milestone
+ vision_config=vision_config,
image_token_id=getattr(hf_config, "image_token_id", None),
attention_groups=groups,
expert_quant=expert_quant,
diff --git a/python/freetoken/models/qwen3_5_moe/model.py b/python/freetoken/models/qwen3_5_moe/model.py
index d34793ad..8cdd41ae 100644
--- a/python/freetoken/models/qwen3_5_moe/model.py
+++ b/python/freetoken/models/qwen3_5_moe/model.py
@@ -12,6 +12,8 @@ from freetoken.layers import (
VocabParallelEmbedding,
)
from freetoken.models.blocks import BaseLLMModel
+from freetoken.models.blocks import embed_input_ids
+from freetoken.models.qwen3_vl.vision import Qwen3VLVisionModel, QwenVLVisionMixin
from freetoken.utils import nvtx_annotate
from .attention import Qwen3_5Attention
@@ -87,7 +89,7 @@ class Qwen3_5Model(BaseOP):
self.norm = GemmaRMSNorm(config.hidden_size, eps=config.rms_norm_eps)
def forward(self, input_ids: torch.Tensor) -> torch.Tensor:
- x = self.embed_tokens.forward(input_ids)
+ x = embed_input_ids(self.embed_tokens, input_ids, get_global_ctx().batch)
residual: torch.Tensor | None = None
for layer in self.layers.op_list:
x, residual = layer.forward(x, residual)
@@ -95,7 +97,7 @@ class Qwen3_5Model(BaseOP):
return x
-class Qwen3_5MoEForCausalLM(BaseLLMModel):
+class Qwen3_5ForCausalLM(BaseLLMModel):
def __init__(self, config: ModelConfig):
self.model = Qwen3_5Model(config)
self.lm_head = ParallelLMHead(
@@ -113,4 +115,25 @@ class Qwen3_5MoEForCausalLM(BaseLLMModel):
return self.lm_head.forward(output)
-__all__ = ["Qwen3_5MoEForCausalLM"]
+class Qwen3_5MoeForCausalLM(Qwen3_5ForCausalLM):
+ """The MoE releases share the dense code path: the decoder picks the routed or dense MLP from config.num_experts."""
+
+
+class Qwen3_5ForConditionalGeneration(QwenVLVisionMixin, Qwen3_5ForCausalLM):
+ def __init__(self, config: ModelConfig):
+ super().__init__(config)
+ if config.is_multimodal:
+ assert not config.vision_config.deepstack_visual_indexes, "Qwen3.5 consumes no DeepStack features"
+ self.visual = Qwen3VLVisionModel(config.vision_config, quant_config=config.quant, prefix="visual")
+
+
+class Qwen3_5MoeForConditionalGeneration(Qwen3_5ForConditionalGeneration):
+ """The MoE releases with the vision tower; see Qwen3_5MoeForCausalLM."""
+
+
+__all__ = [
+ "Qwen3_5ForCausalLM",
+ "Qwen3_5ForConditionalGeneration",
+ "Qwen3_5MoeForCausalLM",
+ "Qwen3_5MoeForConditionalGeneration",
+]
diff --git a/python/freetoken/models/qwen3_5_moe/weight.py b/python/freetoken/models/qwen3_5_moe/weight.py
index 5fc12e74..c74117b3 100644
--- a/python/freetoken/models/qwen3_5_moe/weight.py
+++ b/python/freetoken/models/qwen3_5_moe/weight.py
@@ -13,6 +13,7 @@ import torch
from freetoken.distributed import get_tp_info
from freetoken.kernel.triton.nvfp4_dequant import dequant_nvfp4
from freetoken.layers.quantization import QuantConfig, QuantKind, QuantScheme, get_quant_config
+from freetoken.models.config import VISION_KEY_PREFIXES
from freetoken.models.loader import ShardReader, iter_weight_files
from freetoken.models.nvfp4_banks import Nvfp4ExpertSourceSpec
from freetoken.models.register import ModelSpec, get_model_spec
@@ -20,6 +21,7 @@ from freetoken.utils import cached_load_hf_config
from tqdm import tqdm
from .config import parse_config
+from freetoken.models.qwen3_vl.weight import rename_vl_prefix
# bf16 checkpoints store the routed experts pre-stacked per layer
_STACKED_EXPERT_RE = re.compile(r"^model\.layers\.\d+\.mlp\.experts\.(gate_up_proj|down_proj)$")
@@ -53,16 +55,12 @@ _QUANT_DTYPES = (torch.float8_e4m3fn, torch.float8_e5m2, torch.uint8, torch.int8
def _rename(raw_name: str) -> str | None:
"""Checkpoint key -> FreeToken state-dict key, or None to skip."""
- if raw_name.startswith(("mtp.", "model.visual.", "visual.")):
+ if raw_name.startswith("mtp."):
return None
# static KV-cache scales of the quantizers; the KV cache runs in the engine's dtype
if raw_name.endswith((".k_scale", ".v_scale", ".q_scale", ".prob_scale")):
return None
- if raw_name.startswith("model.language_model."):
- return "model." + raw_name[len("model.language_model."):]
- if raw_name.startswith("language_model."):
- return "model." + raw_name[len("language_model."):]
- return raw_name
+ return rename_vl_prefix(raw_name)
def _is_gemma_norm(name: str) -> bool:
@@ -225,6 +223,7 @@ def iter_weights(
*,
include_moe_experts: bool,
include_non_moe: bool,
+ include_vision: bool = True,
) -> Iterator[tuple[str, torch.Tensor]]:
"""Yield the dense weights fused to the model's buffers, and the routed experts only where a resident path takes them from here: bf16 stacked experts as stored, block-fp8 experts restacked per layer.
@@ -237,18 +236,20 @@ def iter_weights(
stacked = include_moe_experts and config.is_moe and config.expert_quant == "none"
if include_non_moe or stacked:
reader = _DenseReader(get_quant_config(), get_model_spec(hf_config.architectures[0])) if include_non_moe else None
- yield from _iter_shards(model_path, device, reader, stacked=stacked)
+ yield from _iter_shards(model_path, device, reader, stacked=stacked, include_vision=include_vision)
if include_moe_experts and config.is_moe and config.expert_quant == "fp8_block":
yield from _resident_fp8_experts(model_path, config)
-def _iter_shards(model_path: str, device: torch.device, reader: _DenseReader | None, *, stacked: bool):
+def _iter_shards(model_path: str, device: torch.device, reader: _DenseReader | None, *, stacked: bool, include_vision: bool):
for file in tqdm(iter_weight_files(model_path), desc="Loading weights", disable=not get_tp_info().is_primary()):
with safetensors.safe_open(file, framework="pt", device=str(device)) as f:
for raw_name in f.keys():
name = _rename(raw_name)
if name is None or _EXPERT_RE.search(name):
continue
+ if not include_vision and name.startswith(VISION_KEY_PREFIXES):
+ continue
if _STACKED_EXPERT_RE.match(name):
if stacked:
yield name, f.get_tensor(raw_name)
diff --git a/python/freetoken/models/qwen3_moe/attention.py b/python/freetoken/models/qwen3_moe/attention.py
index 8a737b27..2c2e84b4 100644
--- a/python/freetoken/models/qwen3_moe/attention.py
+++ b/python/freetoken/models/qwen3_moe/attention.py
@@ -57,6 +57,12 @@ class Qwen3MoeAttention(BaseOP):
if config.rotary_config.scaling
else None
),
+ mrope_section=(
+ tuple(config.rotary_config.mrope_section)
+ if config.rotary_config.mrope_section is not None
+ else None
+ ),
+ mrope_layout=config.rotary_config.mrope_layout,
)
self.o_proj = LinearOProj(
head_dim * config.num_qo_heads,
@@ -76,7 +82,7 @@ class Qwen3MoeAttention(BaseOP):
self.q_norm.forward_inplace(q.view(-1, self.num_qo_heads, self.head_dim))
if self.k_norm is not None:
self.k_norm.forward_inplace(k.view(-1, self.num_kv_heads, self.head_dim))
- q, k = self.rotary.forward(ctx.batch.positions, q, k)
+ q, k = self.rotary.forward(ctx.batch.get_attn_positions(), q, k)
q = q.view(-1, self.num_qo_heads, self.head_dim)
o = ctx.attn_backend.forward(q, k, v, self.layer_id, ctx.batch)
return self.o_proj.forward(o.view(-1, self.qo_attn_dim))
diff --git a/python/freetoken/models/qwen3_moe/model.py b/python/freetoken/models/qwen3_moe/model.py
index 4918092e..5d89656b 100644
--- a/python/freetoken/models/qwen3_moe/model.py
+++ b/python/freetoken/models/qwen3_moe/model.py
@@ -68,8 +68,10 @@ class Qwen3Model(BaseOP):
class Qwen3MoeForCausalLM(BaseLLMModel):
+ model_cls = Qwen3Model
+
def __init__(self, config: ModelConfig):
- self.model = Qwen3Model(config)
+ self.model = self.model_cls(config)
self.lm_head = ParallelLMHead(
num_embeddings=config.vocab_size,
embedding_dim=config.hidden_size,
diff --git a/python/freetoken/models/qwen3_vl/__init__.py b/python/freetoken/models/qwen3_vl/__init__.py
new file mode 100644
index 00000000..d79c838f
--- /dev/null
+++ b/python/freetoken/models/qwen3_vl/__init__.py
@@ -0,0 +1,21 @@
+from .config import VisionConfig, parse_config, parse_vision_config
+from .model import (
+ Qwen3VLForConditionalGeneration,
+ Qwen3VLMoeForConditionalGeneration,
+ deepstack_add,
+)
+from .vision import Qwen3VLVisionModel
+from .weight import iter_weights, iter_weights_parallel, rename_vl_prefix
+
+__all__ = [
+ "Qwen3VLForConditionalGeneration",
+ "Qwen3VLMoeForConditionalGeneration",
+ "Qwen3VLVisionModel",
+ "VisionConfig",
+ "deepstack_add",
+ "iter_weights",
+ "iter_weights_parallel",
+ "parse_config",
+ "parse_vision_config",
+ "rename_vl_prefix",
+]
diff --git a/python/freetoken/models/qwen3_vl/config.py b/python/freetoken/models/qwen3_vl/config.py
new file mode 100644
index 00000000..0a5c26b3
--- /dev/null
+++ b/python/freetoken/models/qwen3_vl/config.py
@@ -0,0 +1,101 @@
+from __future__ import annotations
+
+import dataclasses
+from dataclasses import dataclass
+from typing import Any
+
+from freetoken.models.config import (
+ FullAttentionGroupConfig,
+ ModelConfig,
+ RotaryConfig,
+ mrope_layout_from_rope_params,
+)
+from freetoken.models.qwen3.config import parse_config as parse_qwen3
+from freetoken.models.qwen3_moe.config import parse_config as parse_qwen3_moe
+
+
+@dataclass
+class VisionConfig:
+ hidden_size: int
+ depth: int
+ num_heads: int
+ intermediate_size: int
+ patch_size: int
+ temporal_patch_size: int
+ spatial_merge_size: int
+ num_position_embeddings: int
+ out_hidden_size: int
+ in_channels: int
+ # ViT block indices whose merged features are added to the first decoder layers; () disables DeepStack
+ deepstack_visual_indexes: tuple[int, ...] = ()
+
+
+def parse_vision_config(hf_config: Any) -> VisionConfig | None:
+ """None when the config carries no vision section, which is how a text-only engine asks for no tower and 1-D rope."""
+ vc = getattr(hf_config, "vision_config", None)
+ if vc is None:
+ return None
+ return VisionConfig(
+ hidden_size=vc.hidden_size,
+ depth=vc.depth,
+ num_heads=vc.num_heads,
+ intermediate_size=vc.intermediate_size,
+ patch_size=vc.patch_size,
+ temporal_patch_size=vc.temporal_patch_size,
+ spatial_merge_size=vc.spatial_merge_size,
+ num_position_embeddings=vc.num_position_embeddings,
+ out_hidden_size=vc.out_hidden_size,
+ in_channels=vc.in_channels,
+ deepstack_visual_indexes=tuple(getattr(vc, "deepstack_visual_indexes", None) or ()),
+ )
+
+
+def parse_config(hf_config: Any) -> ModelConfig:
+ """Qwen3-VL: a Qwen3 (dense or MoE) text tower under text_config plus the shared Qwen VL vision tower."""
+ text = hf_config.text_config
+ base = parse_qwen3_moe(text) if getattr(text, "num_experts", 0) else parse_qwen3(text)
+
+ rope_params = text.rope_parameters
+ rope_type = rope_params.get("rope_type", "default")
+ # the default rope type needs no scaling dict, and the mrope_section list must stay out of get_rope's cache key
+ rope_scaling = (
+ None
+ if rope_type in (None, "default")
+ else {k: v for k, v in rope_params.items() if not isinstance(v, (list, dict))}
+ )
+ vision_config = parse_vision_config(hf_config)
+ rotary = RotaryConfig(
+ head_dim=base.head_dim,
+ rotary_dim=base.head_dim,
+ max_position=text.max_position_embeddings,
+ base=rope_params["rope_theta"],
+ scaling=rope_scaling,
+ mrope_section=(
+ list(rope_params["mrope_section"])
+ if vision_config is not None and "mrope_section" in rope_params
+ else None
+ ),
+ mrope_layout=mrope_layout_from_rope_params(rope_params),
+ )
+ # an explicit group carries the mrope rotary config, which model_is_mrope reads from the groups
+ full_group = FullAttentionGroupConfig(
+ name="full",
+ layer_ids=tuple(range(base.num_layers)),
+ num_kv_heads=base.num_kv_heads,
+ head_dim=base.head_dim,
+ rotary_config=rotary,
+ )
+ return dataclasses.replace(
+ base,
+ rotary_config=rotary,
+ attention_groups=(full_group,),
+ moe_enabled=bool(getattr(text, "num_experts", 0)),
+ vision_config=vision_config,
+ image_token_id=hf_config.image_token_id,
+ tie_word_embeddings=bool(getattr(hf_config, "tie_word_embeddings", False)),
+ model_type=hf_config.model_type,
+ architectures=list(hf_config.architectures),
+ )
+
+
+__all__ = ["VisionConfig", "parse_config", "parse_vision_config"]
diff --git a/python/freetoken/models/qwen3_vl/model.py b/python/freetoken/models/qwen3_vl/model.py
new file mode 100644
index 00000000..cd13945f
--- /dev/null
+++ b/python/freetoken/models/qwen3_vl/model.py
@@ -0,0 +1,80 @@
+from __future__ import annotations
+
+from typing import TYPE_CHECKING
+
+import torch
+from freetoken.core import get_global_ctx
+
+from freetoken.models.qwen3.model import Qwen3ForCausalLM, Qwen3Model
+from freetoken.models.blocks import embed_input_ids
+from freetoken.models.qwen3_moe.model import Qwen3MoeForCausalLM, Qwen3Model as Qwen3MoeModel
+
+from .vision import Qwen3VLVisionModel, QwenVLVisionMixin
+
+if TYPE_CHECKING:
+ from freetoken.models.config import ModelConfig
+
+
+def deepstack_add(
+ x: torch.Tensor, rows: torch.Tensor, mm_embeds: torch.Tensor, level: int, hidden_size: int
+) -> None:
+ """Add DeepStack level `level` (the column block after the main embedding) onto the image rows of x, in place."""
+ lo = hidden_size * (level + 1)
+ x.index_add_(0, rows, mm_embeds[:, lo : lo + hidden_size].to(x.dtype))
+
+
+class Qwen3VLTextModel(Qwen3Model):
+ def __init__(self, config: ModelConfig):
+ super().__init__(config)
+ self._deepstack_levels = len(config.vision_config.deepstack_visual_indexes) if config.vision_config else 0
+
+ def forward(self, input_ids: torch.Tensor) -> torch.Tensor:
+ batch = get_global_ctx().batch
+ x = embed_input_ids(self.embed_tokens, input_ids, batch)
+ residual: torch.Tensor | None = None
+ for level, layer in enumerate(self.layers.op_list):
+ x, residual = layer.forward(x, residual)
+ if batch.mm_embeds is not None and level < self._deepstack_levels:
+ deepstack_add(x, batch.mm_rows, batch.mm_embeds, level, x.shape[1])
+ return self.norm.forward(x, residual)[0]
+
+
+class Qwen3VLMoeTextModel(Qwen3MoeModel):
+ def __init__(self, config: ModelConfig):
+ super().__init__(config)
+ self._deepstack_levels = len(config.vision_config.deepstack_visual_indexes) if config.vision_config else 0
+
+ def forward(self, input_ids: torch.Tensor) -> torch.Tensor:
+ batch = get_global_ctx().batch
+ x = embed_input_ids(self.embed_tokens, input_ids, batch)
+ residual: torch.Tensor | None = None
+ for level, layer in enumerate(self.layers.op_list):
+ x, residual = layer.forward(x, residual)
+ if batch.mm_embeds is not None and level < self._deepstack_levels:
+ deepstack_add(x, batch.mm_rows, batch.mm_embeds, level, x.shape[1])
+ return self.norm.forward(x, residual)[0]
+
+
+class Qwen3VLForConditionalGeneration(QwenVLVisionMixin, Qwen3ForCausalLM):
+ model_cls = Qwen3VLTextModel
+
+ def __init__(self, config: ModelConfig):
+ super().__init__(config)
+ if config.is_multimodal:
+ self.visual = Qwen3VLVisionModel(config.vision_config, quant_config=config.quant, prefix="visual")
+
+
+class Qwen3VLMoeForConditionalGeneration(QwenVLVisionMixin, Qwen3MoeForCausalLM):
+ model_cls = Qwen3VLMoeTextModel
+
+ def __init__(self, config: ModelConfig):
+ super().__init__(config)
+ if config.is_multimodal:
+ self.visual = Qwen3VLVisionModel(config.vision_config, quant_config=config.quant, prefix="visual")
+
+
+__all__ = [
+ "Qwen3VLForConditionalGeneration",
+ "Qwen3VLMoeForConditionalGeneration",
+ "deepstack_add",
+]
diff --git a/python/freetoken/models/qwen3_vl/vision.py b/python/freetoken/models/qwen3_vl/vision.py
new file mode 100644
index 00000000..7c4e0df2
--- /dev/null
+++ b/python/freetoken/models/qwen3_vl/vision.py
@@ -0,0 +1,375 @@
+"""Qwen VL vision tower: full-attention ViT blocks and a 2x2 patch merger; DeepStack taps ride along as extra output columns."""
+
+from __future__ import annotations
+
+import os
+from typing import TYPE_CHECKING, List
+
+import torch
+import torch.nn.functional as F
+from freetoken.distributed import get_tp_info
+from freetoken.layers import (
+ BaseOP,
+ LinearColParallelMerged,
+ LinearOProj,
+ LinearQKVMerged,
+ LinearRowParallel,
+ OPList,
+)
+from freetoken.utils import div_even
+
+from freetoken.models.weight_stream import BlockWeightStreamer
+
+if TYPE_CHECKING:
+ from freetoken.layers.quantization import QuantConfig
+ from freetoken.message import MMItem
+ from .config import VisionConfig
+
+
+class VisionLayerNorm(BaseOP):
+ def __init__(self, size: int, eps: float = 1e-6):
+ self.weight = torch.empty(size)
+ self.bias = torch.empty(size)
+ self._size = size
+ self._eps = eps
+
+ def forward(self, x: torch.Tensor) -> torch.Tensor:
+ return F.layer_norm(x, (self._size,), self.weight, self.bias, self._eps)
+
+
+def _rotate_half(x: torch.Tensor) -> torch.Tensor:
+ half = x.shape[-1] // 2
+ return torch.cat((-x[..., half:], x[..., :half]), dim=-1)
+
+
+_warned_rope_fallback = False
+
+
+def _apply_vision_rope(
+ q: torch.Tensor, k: torch.Tensor, cache: torch.Tensor, positions: torch.Tensor, head_dim: int
+) -> None:
+ """In-place 2D rope through the NeoX text kernel: the [c,c]/[s,s] layout is the half-rotation layout and positions index per-token cache rows."""
+ global _warned_rope_fallback
+ S = q.shape[0]
+ try:
+ from freetoken.kernel.triton.rope import apply_rope_with_cos_sin_cache_inplace
+
+ apply_rope_with_cos_sin_cache_inplace(
+ positions, q.view(S, -1), k.view(S, -1), head_dim, cache, is_neox=True
+ )
+ return
+ except ImportError:
+ # only a missing triton falls back; a CUDA error must not re-run eager on a half-rotated q/k
+ if not _warned_rope_fallback:
+ _warned_rope_fallback = True
+ from freetoken.utils import init_logger
+
+ init_logger(__name__).warning("vision rope triton kernel unavailable; using eager")
+ half = cache.shape[1] // 2
+ cos = torch.cat((cache[:, :half], cache[:, :half]), dim=-1).unsqueeze(-2)
+ sin = torch.cat((cache[:, half:], cache[:, half:]), dim=-1).unsqueeze(-2)
+ q.copy_((q.float() * cos + _rotate_half(q.float()) * sin).to(q.dtype))
+ k.copy_((k.float() * cos + _rotate_half(k.float()) * sin).to(k.dtype))
+
+
+class VisionAttention(BaseOP):
+ def __init__(self, vc: VisionConfig, *, quant_config: QuantConfig | None = None, prefix: str = ""):
+ self.num_heads = div_even(vc.num_heads, get_tp_info().size)
+ self.head_dim = vc.hidden_size // vc.num_heads
+ self.qkv = LinearQKVMerged(
+ vc.hidden_size, self.head_dim, vc.num_heads, vc.num_heads, has_bias=True,
+ quant_config=quant_config, prefix=f"{prefix}.qkv",
+ )
+ self.proj = LinearOProj(
+ vc.hidden_size, vc.hidden_size, has_bias=True, quant_config=quant_config, prefix=f"{prefix}.proj"
+ )
+
+ def attend(
+ self, qkv: torch.Tensor, cache: torch.Tensor, positions: torch.Tensor, lengths: List[int]
+ ) -> torch.Tensor:
+ """Rope + bidirectional attention within each image on the fused qkv projection; returns [S, heads * head_dim]."""
+ S = qkv.shape[0]
+ # q/k/v stay views into the fused projection: the rope kernel takes strides, SDPA needs only head_dim contiguous
+ q, k, v = qkv.view(S, 3, self.num_heads, self.head_dim).unbind(1)
+ _apply_vision_rope(q, k, cache, positions, self.head_dim)
+ q = q.transpose(0, 1).unsqueeze(0)
+ k = k.transpose(0, 1).unsqueeze(0)
+ v = v.transpose(0, 1).unsqueeze(0)
+ # bidirectional attention within each image; images never attend across
+ outs = [
+ F.scaled_dot_product_attention(qs, ks, vs)
+ for qs, ks, vs in zip(
+ torch.split(q, lengths, dim=2),
+ torch.split(k, lengths, dim=2),
+ torch.split(v, lengths, dim=2),
+ )
+ ]
+ o = outs[0] if len(outs) == 1 else torch.cat(outs, dim=2)
+ return o[0].transpose(0, 1).reshape(S, -1)
+
+
+class VisionMLP(BaseOP):
+ def __init__(self, vc: VisionConfig, *, quant_config: QuantConfig | None = None, prefix: str = ""):
+ self.linear_fc1 = LinearColParallelMerged(
+ vc.hidden_size, [vc.intermediate_size], has_bias=True, quant_config=quant_config, prefix=f"{prefix}.linear_fc1"
+ )
+ self.linear_fc2 = LinearRowParallel(
+ vc.intermediate_size, vc.hidden_size, has_bias=True, quant_config=quant_config, prefix=f"{prefix}.linear_fc2"
+ )
+
+ def forward(self, x: torch.Tensor) -> torch.Tensor:
+ if x.shape[0] <= _MLP_ROWS:
+ return self.linear_fc2.forward(F.gelu(self.linear_fc1.forward(x), approximate="tanh"))
+ # per-token op: row chunks bound the two [rows, intermediate] temporaries while the GEMMs stay large
+ out = torch.empty_like(x)
+ for lo in range(0, x.shape[0], _MLP_ROWS):
+ rows = x[lo : lo + _MLP_ROWS]
+ out[lo : lo + _MLP_ROWS] = self.linear_fc2.forward(F.gelu(self.linear_fc1.forward(rows), approximate="tanh"))
+ return out
+
+
+# 4096 rows x 4304 x bf16 = 34 MiB per temporary while the GEMM still fills the GPU
+# FREETOKEN_VIT_MLP_ROWS=0 disables the chunking; cuBLAS may round a row chunk one bf16 ulp differently from the whole image
+_MLP_ROWS = int(os.environ.get("FREETOKEN_VIT_MLP_ROWS", 4096)) or 1 << 62
+# 16384 rows x 1152 x fp32 = 72 MiB per interpolation temporary
+_POS_ROWS = 16384
+
+
+class VisionBlock(BaseOP):
+ def __init__(self, vc: VisionConfig, *, quant_config: QuantConfig | None = None, prefix: str = ""):
+ self.norm1 = VisionLayerNorm(vc.hidden_size)
+ self.norm2 = VisionLayerNorm(vc.hidden_size)
+ self.attn = VisionAttention(vc, quant_config=quant_config, prefix=f"{prefix}.attn")
+ self.mlp = VisionMLP(vc, quant_config=quant_config, prefix=f"{prefix}.mlp")
+
+ def forward(
+ self, x: torch.Tensor, cache: torch.Tensor, positions: torch.Tensor, lengths: List[int]
+ ) -> torch.Tensor:
+ x = x + self._attention(x, cache, positions, lengths)
+ return x + self.mlp.forward(self.norm2.forward(x))
+
+ def _attention(
+ self, x: torch.Tensor, cache: torch.Tensor, positions: torch.Tensor, lengths: List[int]
+ ) -> torch.Tensor:
+ # each temporary dies inside the next call: norm out in the qkv GEMM, qkv in attention, its output in proj
+ o = self.attn.attend(self.attn.qkv.forward(self.norm1.forward(x)), cache, positions, lengths)
+ return self.attn.proj.forward(o)
+
+
+class _Conv3dParams(BaseOP):
+ def __init__(self, out_ch: int, shape: tuple):
+ self.weight = torch.empty(out_ch, *shape)
+ self.bias = torch.empty(out_ch)
+
+ forward = None
+
+
+class _EmbeddingParams(BaseOP):
+ def __init__(self, rows: int, dim: int):
+ self.weight = torch.empty(rows, dim)
+
+ forward = None
+
+
+class VisionConv3dPatchEmbed(BaseOP):
+ """Kept as a real Conv3d (kernel == stride): an equivalent unfolded matmul rounds differently."""
+
+ def __init__(self, vc: VisionConfig):
+ self._shape = (vc.in_channels, vc.temporal_patch_size, vc.patch_size, vc.patch_size)
+ self.proj = _Conv3dParams(vc.hidden_size, self._shape)
+
+ def forward(self, pixel_values: torch.Tensor) -> torch.Tensor:
+ x = pixel_values.view(-1, *self._shape).to(self.proj.weight.dtype)
+ out = F.conv3d(x, self.proj.weight, self.proj.bias, stride=self._shape[1:])
+ return out.view(x.shape[0], -1)
+
+
+class VisionPatchMerger(BaseOP):
+ def __init__(
+ self,
+ vc: VisionConfig,
+ use_postshuffle_norm: bool = False,
+ *,
+ quant_config: QuantConfig | None = None,
+ prefix: str = "",
+ ):
+ merged = vc.hidden_size * vc.spatial_merge_size**2
+ # the DeepStack mergers normalize the 2x2-merged vector, the final merger each patch
+ self.norm = VisionLayerNorm(merged if use_postshuffle_norm else vc.hidden_size)
+ self.linear_fc1 = LinearColParallelMerged(
+ merged, [merged], has_bias=True, quant_config=quant_config, prefix=f"{prefix}.linear_fc1"
+ )
+ self.linear_fc2 = LinearRowParallel(
+ merged, vc.out_hidden_size, has_bias=True, quant_config=quant_config, prefix=f"{prefix}.linear_fc2"
+ )
+ self._merged = merged
+ self._postshuffle_norm = use_postshuffle_norm
+
+ def forward(self, x: torch.Tensor) -> torch.Tensor:
+ if self._postshuffle_norm:
+ x = self.norm.forward(x.view(-1, self._merged))
+ else:
+ x = self.norm.forward(x).view(-1, self._merged)
+ return self.linear_fc2.forward(F.gelu(self.linear_fc1.forward(x)))
+
+
+class Qwen3VLVisionModel(BaseOP):
+ """Pixels -> per-image soft tokens [sum(h*w)/merge^2, out_hidden x (1 + DeepStack levels)].
+
+ Columns past out_hidden are the DeepStack side features in deepstack_visual_indexes order.
+ """
+
+ def __init__(self, vc: VisionConfig, *, quant_config: QuantConfig | None = None, prefix: str = "visual"):
+ self.patch_embed = VisionConv3dPatchEmbed(vc)
+ self.pos_embed = _EmbeddingParams(vc.num_position_embeddings, vc.hidden_size)
+ self.blocks = OPList(
+ [VisionBlock(vc, quant_config=quant_config, prefix=f"{prefix}.blocks.{i}") for i in range(vc.depth)]
+ )
+ self.merger = VisionPatchMerger(vc, quant_config=quant_config, prefix=f"{prefix}.merger")
+ self.deepstack_merger_list = OPList(
+ [
+ VisionPatchMerger(
+ vc, use_postshuffle_norm=True, quant_config=quant_config, prefix=f"{prefix}.deepstack_merger_list.{k}"
+ )
+ for k in range(len(vc.deepstack_visual_indexes))
+ ]
+ )
+ self._vc = vc
+ self._num_grid_per_side = int(vc.num_position_embeddings**0.5)
+ self._inv_dim = (vc.hidden_size // vc.num_heads) // 2
+ self._inv_freq: torch.Tensor | None = None
+ self._streamer: BlockWeightStreamer | None = None
+
+ def place_weights(self, mode: str) -> None:
+ """gpu: every tensor resident; host: block tensors in pinned banks streamed two blocks at a time (mergers and embeddings stay resident)."""
+ if mode == "host" and self._streamer is None:
+ self._streamer = BlockWeightStreamer(self.blocks.op_list, self.pos_embed.weight.device)
+ elif mode == "gpu" and self._streamer is not None:
+ self._streamer.unstream()
+ self._streamer = None
+ elif mode not in ("gpu", "host"):
+ raise ValueError(f"unknown vision weight placement {mode!r}")
+
+ def _blocks(self):
+ if self._streamer is None:
+ return enumerate(self.blocks.op_list)
+ return self._streamer.blocks(self.blocks.op_list)
+
+ def _add_pos_embed(self, x: torch.Tensor, grid: torch.Tensor) -> None:
+ """x += bilinear resample of the position table, in row chunks so the fp32 temporaries stay small."""
+ from transformers.vision_utils import get_vision_interpolation_indices_and_weights
+
+ interp_idx, interp_w = get_vision_interpolation_indices_and_weights(
+ grid,
+ num_grid_per_side=self._num_grid_per_side,
+ mode="bilinear",
+ align_corners=True,
+ spatial_merge_size=self._vc.spatial_merge_size,
+ )
+ table = self.pos_embed.weight
+ for lo in range(0, x.shape[0], _POS_ROWS):
+ idx, w = interp_idx[lo : lo + _POS_ROWS], interp_w[lo : lo + _POS_ROWS]
+ # one corner at a time, multiply then add: the same per-element arithmetic as the fused [S, 4, hidden] .sum(1)
+ acc = F.embedding(idx[:, 0], table).float() * w[:, 0, None]
+ for k in range(1, idx.shape[1]):
+ acc += F.embedding(idx[:, k], table).float() * w[:, k, None]
+ x[lo : lo + _POS_ROWS].add_(acc.to(x.dtype))
+
+ def _rope_table(self, grid: torch.Tensor, device: torch.device) -> tuple[torch.Tensor, torch.Tensor]:
+ from transformers.vision_utils import get_vision_position_ids
+
+ if self._inv_freq is None or self._inv_freq.device != device:
+ # built lazily: the model itself is constructed on the meta device
+ # kept in the model dtype (bf16) on purpose; fp32 here would change the rope numerics
+ inv_dim = self._inv_dim
+ self._inv_freq = (
+ 1.0 / (10000.0 ** (torch.arange(0, inv_dim, 2, dtype=torch.float32, device=device) / inv_dim))
+ ).to(self.pos_embed.weight.dtype)
+ pos_ids = get_vision_position_ids(grid, self._vc.spatial_merge_size)
+ # long * bf16 -> bf16 freqs on purpose; fp32 freqs would round differently
+ freqs = (pos_ids.unsqueeze(-1) * self._inv_freq).flatten(1) # [S, head_dim/2]
+ # per-token rope rows [cos | sin] consumed by the NeoX kernel via positions=arange
+ cache = torch.cat((freqs.cos(), freqs.sin()), dim=-1).to(torch.float32).contiguous()
+ positions = torch.arange(pos_ids.shape[0], dtype=torch.int32, device=device)
+ return cache, positions
+
+ def _embed(self, pixel_values: torch.Tensor, grid_thw: List[List[int]]) -> tuple[torch.Tensor, tuple]:
+ """Patch embeddings with the position table added, and the attention inputs every block takes."""
+ from transformers.vision_utils import get_vision_cu_seqlens
+
+ device = self.pos_embed.weight.device
+ grid = torch.tensor(grid_thw, dtype=torch.long, device=device)
+ x = self.patch_embed.forward(pixel_values.to(device))
+ self._add_pos_embed(x, grid)
+ cache, positions = self._rope_table(grid, device)
+ return x, (cache, positions, torch.diff(get_vision_cu_seqlens(grid)).tolist())
+
+ def _encode(
+ self, pixel_values: torch.Tensor, grid_thw: List[List[int]], groups: torch.Tensor | None = None
+ ) -> torch.Tensor:
+ """Pixels through the patch embedding and the block stack; with ``groups`` each DeepStack tap's merged feature is copied into groups[1 + k]."""
+ taps = self._vc.deepstack_visual_indexes
+ x, attn = self._embed(pixel_values, grid_thw)
+ for i, blk in self._blocks():
+ x = blk.forward(x, *attn)
+ if groups is not None and i in taps:
+ k = taps.index(i)
+ groups[1 + k].copy_(self.deepstack_merger_list.op_list[k].forward(x), non_blocking=True)
+ return x
+
+ @staticmethod
+ def _unpark(groups: torch.Tensor, device: torch.device) -> torch.Tensor:
+ """[rows, n * width] on the device from host slabs [n, rows, width]; group k fills columns k*width:(k+1)*width."""
+ n, rows, width = groups.shape
+ out = torch.empty(rows, n * width, dtype=groups.dtype, device=device)
+ for k in range(n):
+ out[:, k * width : (k + 1) * width].copy_(groups[k], non_blocking=True)
+ return out
+
+ @torch.inference_mode()
+ def forward(self, pixel_values: torch.Tensor, grid_thw: List[List[int]]) -> torch.Tensor:
+ """Soft tokens [rows, width x (1 + taps)], rows = sum(h*w) / merge^2: the merger's columns first, then one DeepStack tap per range.
+
+ Same numbers as ``forward_naive``.
+ """
+ vc = self._vc
+ taps = vc.deepstack_visual_indexes
+ if not taps:
+ return self.merger.forward(self._encode(pixel_values, grid_thw))
+ # the merger outputs leave the GPU as they are produced and come back after the last block, so only the block working set is resident while the blocks run
+ rows = pixel_values.shape[0] // vc.spatial_merge_size**2
+ groups = torch.empty(
+ (1 + len(taps), rows, vc.out_hidden_size), dtype=self.pos_embed.weight.dtype, device="cpu",
+ pin_memory=torch.cuda.is_available(),
+ )
+ x = self._encode(pixel_values, grid_thw, groups)
+ groups[0].copy_(self.merger.forward(x), non_blocking=True)
+ del x
+ return self._unpark(groups, self.pos_embed.weight.device)
+
+ @torch.inference_mode()
+ def forward_naive(self, pixel_values: torch.Tensor, grid_thw: List[List[int]]) -> torch.Tensor:
+ """Reference forward, written the plain way: the block stack, then the merger outputs side by side."""
+ taps = self._vc.deepstack_visual_indexes
+ x, attn = self._embed(pixel_values, grid_thw)
+ features = []
+ for i, blk in self._blocks():
+ x = blk.forward(x, *attn)
+ if i in taps:
+ features.append(self.deepstack_merger_list.op_list[taps.index(i)].forward(x))
+ return torch.cat([self.merger.forward(x)] + features, dim=1)
+
+
+class QwenVLVisionMixin:
+ """The engine hooks of a wrapper that owns a Qwen VL tower as ``self.visual``."""
+
+ visual: Qwen3VLVisionModel
+
+ def place_encoder_weights(self, mode: str) -> None:
+ self.visual.place_weights(mode)
+
+ def encode(self, item: MMItem) -> torch.Tensor:
+ return self.visual.forward(item.feature, [item.grid_thw])
+
+
+__all__ = ["Qwen3VLVisionModel", "QwenVLVisionMixin"]
diff --git a/python/freetoken/models/qwen3_vl/weight.py b/python/freetoken/models/qwen3_vl/weight.py
new file mode 100644
index 00000000..3a7df174
--- /dev/null
+++ b/python/freetoken/models/qwen3_vl/weight.py
@@ -0,0 +1,117 @@
+from __future__ import annotations
+
+from typing import Iterator
+
+import safetensors
+import torch
+from freetoken.distributed import get_tp_info
+from freetoken.models.config import VISION_KEY_PREFIXES
+from freetoken.models.loader import MergeRule, iter_merged_tensors, iter_weight_files, shard_tensor
+from freetoken.utils import cached_load_hf_config
+from tqdm import tqdm
+
+from .config import parse_config
+
+_MERGE_RULES = {
+ ".q_proj": MergeRule(".qkv_proj", "q", ("q", "k", "v")),
+ ".k_proj": MergeRule(".qkv_proj", "k", ("q", "k", "v")),
+ ".v_proj": MergeRule(".qkv_proj", "v", ("q", "k", "v")),
+ ".gate_proj": MergeRule(".gate_up_proj", "gate", ("gate", "up")),
+ ".up_proj": MergeRule(".gate_up_proj", "up", ("gate", "up")),
+}
+_FUSED_EXPERT_KEYS = ("gate_up_proj", "down_proj")
+
+
+def rename_vl_prefix(raw_name: str) -> str:
+ """Checkpoint wrapper prefixes -> state-dict keys: tower under visual., text under model."""
+ if raw_name.startswith(("model.visual.", "visual.")):
+ return "visual." + raw_name.removeprefix("model.").removeprefix("visual.")
+ if raw_name.startswith("model.language_model."):
+ return "model." + raw_name[len("model.language_model.") :]
+ if raw_name.startswith("language_model."):
+ return "model." + raw_name[len("language_model.") :]
+ return raw_name
+
+
+def _is_fused_experts(name: str) -> bool:
+ parts = name.split(".")
+ return len(parts) > 2 and parts[-2] == "experts" and parts[-1] in _FUSED_EXPERT_KEYS
+
+
+def _linear_layout(tensor: torch.Tensor) -> torch.Tensor:
+ # the checkpoint stacks experts as [E, in, out]; the banks and fused kernels take [E, out, in] and copy from the strided view
+ return tensor.transpose(1, 2)
+
+
+def iter_weights(
+ model_path: str,
+ device: torch.device,
+ *,
+ include_moe_experts: bool,
+ include_non_moe: bool,
+ include_vision: bool = True,
+) -> Iterator[tuple[str, torch.Tensor]]:
+ config = parse_config(cached_load_hf_config(model_path))
+ tp_info = get_tp_info()
+
+ def tensors() -> Iterator[tuple[str, torch.Tensor]]:
+ for file in tqdm(
+ iter_weight_files(model_path),
+ desc="Loading weights",
+ disable=not tp_info.is_primary(),
+ ):
+ with safetensors.safe_open(file, framework="pt", device=str(device)) as f:
+ for raw_name in f.keys():
+ name = rename_vl_prefix(raw_name)
+ if not include_vision and name.startswith(VISION_KEY_PREFIXES):
+ continue
+ is_expert = _is_fused_experts(name)
+ if is_expert and not include_moe_experts:
+ continue
+ if not is_expert and not include_non_moe:
+ continue
+ raw = f.get_tensor(raw_name)
+ if is_expert:
+ if tp_info.size > 1:
+ raise NotImplementedError("Qwen3-VL-MoE experts are not tensor-parallel sharded")
+ yield name, _linear_layout(raw)
+ continue
+ if name.startswith("visual.") and tp_info.size > 1:
+ raise NotImplementedError("Qwen VL vision tower weights are not tensor-parallel sharded")
+ tensor = shard_tensor(
+ name,
+ raw,
+ rank=tp_info.rank,
+ world_size=tp_info.size,
+ num_kv_heads=config.num_kv_heads,
+ )
+ del raw
+ yield name, tensor
+
+ yield from iter_merged_tensors(tensors(), _MERGE_RULES, model_name="qwen3_vl")
+
+
+def iter_weights_parallel(
+ model_path: str,
+ device: torch.device,
+ *,
+ include_moe_experts: bool,
+ include_non_moe: bool,
+ workers: int = 8,
+ chunk: int = 8 << 20,
+) -> Iterator[tuple[str, torch.Tensor]]:
+ """experts-only iter_weights over the common chunked multi-threaded O_DIRECT reader."""
+ assert include_moe_experts and not include_non_moe, (
+ "qwen3_vl parallel reader is experts-only (used by load_moe_expert_sources)"
+ )
+ from freetoken.models.weight import iter_expert_tensors_parallel
+
+ if get_tp_info().size > 1:
+ raise NotImplementedError("Qwen3-VL-MoE experts are not tensor-parallel sharded")
+ for raw_name, raw in iter_expert_tensors_parallel(
+ model_path, _is_fused_experts, workers=workers, chunk=chunk
+ ):
+ yield rename_vl_prefix(raw_name), _linear_layout(raw)
+
+
+__all__ = ["iter_weights", "iter_weights_parallel", "rename_vl_prefix"]
diff --git a/python/freetoken/models/qwen4_exp/__init__.py b/python/freetoken/models/qwen4_exp/__init__.py
index 6715e67e..b063da6a 100644
--- a/python/freetoken/models/qwen4_exp/__init__.py
+++ b/python/freetoken/models/qwen4_exp/__init__.py
@@ -1,4 +1,4 @@
-"""Qwen3.8-Flash-Next (model_type qwen4_exp), served text-only.
+"""Qwen3.8-Flash-Next (model_type qwen4_exp); image input rides on the shared Qwen VL tower.
48 decoder layers on hc_count=4 hyper-connection residual streams R [T, 4*hidden]:
embed -> repeat(1, 4) -> [PLE at zero-based layer 1] -> per layer attn_hc.mix -> (GDN | QSA) -> attn_hc.combine -> mlp_hc.mix -> MoE -> mlp_hc.combine -> top-level mixer.mix -> lm_head.
@@ -10,7 +10,7 @@ Contracts shared across modules (do not rename):
"""
from .config import parse_config
-from .model import Qwen4ExpForCausalLM
+from .model import Qwen4ExpForCausalLM, Qwen4ExpForConditionalGeneration
from .weight import (
ftw_side_files,
nvfp4_expert_spec,
@@ -26,6 +26,7 @@ __all__ = [
"ftw_side_files",
"nvfp4_expert_spec",
"Qwen4ExpForCausalLM",
+ "Qwen4ExpForConditionalGeneration",
"iter_weights",
"load_ple_table",
"parse_config",
diff --git a/python/freetoken/models/qwen4_exp/attention.py b/python/freetoken/models/qwen4_exp/attention.py
index 66211c7c..8f6b9388 100644
--- a/python/freetoken/models/qwen4_exp/attention.py
+++ b/python/freetoken/models/qwen4_exp/attention.py
@@ -141,6 +141,8 @@ class Qwen4ExpAttention(BaseOP):
max_position=rotary.max_position,
base=rotary.base,
rope_scaling=tuple(rotary.scaling.items()) if rotary.scaling else None,
+ mrope_section=tuple(rotary.mrope_section) if rotary.mrope_section is not None else None,
+ mrope_layout=rotary.mrope_layout,
)
self.indexer = Qwen4ExpIndexer(config, layer_id, prefix=f"{prefix}.indexer")
@@ -155,7 +157,7 @@ class Qwen4ExpAttention(BaseOP):
self.q_norm.forward_inplace(q)
self.k_norm.forward_inplace(k)
q, k = self.rotary.forward(
- batch.positions, q.view(-1, self.qo_attn_dim), k.view(-1, self.kv_attn_dim)
+ batch.get_attn_positions(), q.view(-1, self.qo_attn_dim), k.view(-1, self.kv_attn_dim)
)
index = self.indexer.forward(x)
o = get_global_ctx().attn_backend.qsa_forward(
diff --git a/python/freetoken/models/qwen4_exp/config.py b/python/freetoken/models/qwen4_exp/config.py
index 4bcd28de..e4137862 100644
--- a/python/freetoken/models/qwen4_exp/config.py
+++ b/python/freetoken/models/qwen4_exp/config.py
@@ -7,12 +7,14 @@ import torch
from freetoken.layers.quantization import QuantConfig
from freetoken.models.config import (
+ mrope_layout_from_rope_params,
FullAttentionGroupConfig,
LinearGatedDeltaGroupConfig,
ModelConfig,
RotaryConfig,
SlotStateSpec,
)
+from freetoken.models.qwen3_vl.config import parse_vision_config
@dataclass(frozen=True)
@@ -40,6 +42,7 @@ class Qwen4ExpArgs:
index_head_dim: int
index_budget: int
index_ratio: int
+ image_token_id: int | None = None
@property
def index_topk_blocks(self) -> int:
@@ -152,12 +155,19 @@ def parse_config(hf_config: Any) -> ModelConfig:
if layer_types[lid] != "linear_attention":
raise ValueError(f"PLE must sit on a linear_attention layer, got layer {lid}")
+ vision_config = parse_vision_config(hf_config)
full_rotary = RotaryConfig(
head_dim=head_dim,
rotary_dim=rotary_dim,
max_position=text.max_position_embeddings,
base=rope_theta,
scaling=rope_scaling,
+ mrope_section=(
+ list(rope_params["mrope_section"])
+ if vision_config is not None and "mrope_section" in rope_params
+ else None
+ ),
+ mrope_layout=mrope_layout_from_rope_params(rope_params),
)
full_group = FullAttentionGroupConfig(
name="full",
@@ -214,6 +224,7 @@ def parse_config(hf_config: Any) -> ModelConfig:
index_head_dim=int(text.indexer_head_dim),
index_budget=int(text.indexer_budget),
index_ratio=int(text.indexer_compress_ratio),
+ image_token_id=getattr(hf_config, "image_token_id", None),
)
return ModelConfig(
@@ -241,7 +252,7 @@ def parse_config(hf_config: Any) -> ModelConfig:
use_qk_norm=True,
model_type=getattr(hf_config, "model_type", "qwen4_exp"),
architectures=getattr(hf_config, "architectures", ["Qwen4ExpForConditionalGeneration"]),
- vision_config=None, # served text-only
+ vision_config=vision_config,
image_token_id=getattr(hf_config, "image_token_id", None),
attention_groups=groups,
expert_quant=expert_quant,
diff --git a/python/freetoken/models/qwen4_exp/model.py b/python/freetoken/models/qwen4_exp/model.py
index a4b63362..258badac 100644
--- a/python/freetoken/models/qwen4_exp/model.py
+++ b/python/freetoken/models/qwen4_exp/model.py
@@ -27,6 +27,8 @@ from .attention import Qwen4ExpAttention
from .hc import GatedResidual
from .moe import Qwen4ExpMoE
from .ple import PLELayer
+from freetoken.models.blocks import embed_input_ids
+from freetoken.models.qwen3_vl.vision import Qwen3VLVisionModel, QwenVLVisionMixin
if TYPE_CHECKING:
from freetoken.core import Batch
@@ -107,7 +109,8 @@ class Qwen4ExpModel(BaseOP):
return list(self._ple)
def forward(self, input_ids: torch.Tensor, batch: Batch) -> torch.Tensor:
- hidden = self.embed_tokens.forward(input_ids).repeat(1, self.hc_count)
+ hidden = embed_input_ids(self.embed_tokens, input_ids, batch)
+ hidden = hidden.repeat(1, self.hc_count)
meta = None
if self._ple:
from .ple import build_ple_metadata, commit_ngram_context
@@ -180,6 +183,7 @@ class Qwen4ExpForCausalLM(BaseLLMModel):
"per_head_vocab_sizes": emb.ngram_heads_vocab_sizes.tolist(),
"per_head_offsets": emb.ngram_heads_offsets.tolist(),
"eos_token_id": args.ngram_boundary_token_id,
+ "image_token_id": args.image_token_id,
}
disk_table = DiskRowTable(
resolve_row_source(folder),
@@ -209,4 +213,18 @@ class Qwen4ExpForCausalLM(BaseLLMModel):
return self.lm_head.forward(self.model.forward(batch.input_ids, batch))
-__all__ = ["Qwen4ExpDecoderLayer", "Qwen4ExpForCausalLM", "Qwen4ExpModel", "build_linear_mixer"]
+class Qwen4ExpForConditionalGeneration(QwenVLVisionMixin, Qwen4ExpForCausalLM):
+ def __init__(self, config: ModelConfig) -> None:
+ super().__init__(config)
+ if config.is_multimodal:
+ assert not config.vision_config.deepstack_visual_indexes, "Qwen3.8 consumes no DeepStack features"
+ self.visual = Qwen3VLVisionModel(config.vision_config, quant_config=config.quant, prefix="visual")
+
+
+__all__ = [
+ "Qwen4ExpDecoderLayer",
+ "Qwen4ExpForCausalLM",
+ "Qwen4ExpForConditionalGeneration",
+ "Qwen4ExpModel",
+ "build_linear_mixer",
+]
diff --git a/python/freetoken/models/qwen4_exp/ple.py b/python/freetoken/models/qwen4_exp/ple.py
index 13157cb8..c944c494 100644
--- a/python/freetoken/models/qwen4_exp/ple.py
+++ b/python/freetoken/models/qwen4_exp/ple.py
@@ -26,6 +26,7 @@ import torch
import torch.nn.functional as F
from freetoken.core import get_global_ctx
from freetoken.layers import BaseOP, LinearReplicated
+from freetoken.mm import restore_placeholder
from .config import PLE_CONV_STATE, PLE_NGRAM_STATE
from .hc import GroupedPlusOneRMSNorm
@@ -333,12 +334,16 @@ def build_ple_metadata(
)
fla = getattr(batch, "fla_metadata", None)
slots_dev = getattr(batch, "linear_table_idx", None)
+ # image rows hash as the placeholder token, never as the content pad ids
+ input_ids = batch.input_ids
+ if args.image_token_id is not None:
+ input_ids = restore_placeholder(input_ids, args.image_token_id)
if batch.is_decode and slots_dev is not None:
slots = slots_dev.long()
bs = slots.numel()
return PLEMetadata(
- input_ids=batch.input_ids,
+ input_ids=input_ids,
cu_seqlens=torch.arange(bs + 1, dtype=torch.int32, device=device),
seq_lens=(1,) * bs,
ngram_context=context_pool.index_select(0, slots).long(),
@@ -360,7 +365,7 @@ def build_ple_metadata(
context = context_pool.index_select(0, slots).long()
context = torch.where(fresh.unsqueeze(1), context.new_full((), eos), context)
return PLEMetadata(
- input_ids=batch.input_ids,
+ input_ids=input_ids,
cu_seqlens=cu,
seq_lens=tuple(lens),
ngram_context=context,
diff --git a/python/freetoken/models/qwen4_exp/ple_disk.py b/python/freetoken/models/qwen4_exp/ple_disk.py
index 799866a6..cea62693 100644
--- a/python/freetoken/models/qwen4_exp/ple_disk.py
+++ b/python/freetoken/models/qwen4_exp/ple_disk.py
@@ -13,6 +13,8 @@ from typing import Sequence
import safetensors
import torch
+from freetoken.mm import MM_PAD_SHIFT_VALUE, restore_placeholder
+
from freetoken.core import Batch
from freetoken.kernel.pinned import alloc_pinned_tensor
from freetoken.utils import init_logger
@@ -118,6 +120,7 @@ class DiskRowTable:
self.heads = int(hash_constants["num_ngram_heads"])
self.scale = source.scale
self.eos_token_id = int(hash_constants["eos_token_id"])
+ self.image_token_id = hash_constants.get("image_token_id")
sizes = [int(x) for x in hash_constants["per_head_vocab_sizes"]]
offsets = [int(x) for x in hash_constants["per_head_offsets"]]
need = max(o + s for o, s in zip(offsets, sizes))
@@ -191,6 +194,16 @@ class DiskRowTable:
offset += run.numel() - 2
self._store.flush(self._flag.data_ptr() if graph and self._wait_sync else 0)
+ def _ple_ids(self, input_ids: torch.Tensor) -> torch.Tensor:
+ if self.image_token_id is None:
+ return input_ids
+ return restore_placeholder(input_ids, self.image_token_id)
+
+ def _ple_context(self, ids: torch.Tensor, position: int) -> list[int]:
+ if self.image_token_id is None:
+ return _context(ids, position, self.eos_token_id)
+ return [self.image_token_id if t >= MM_PAD_SHIFT_VALUE else t for t in _context(ids, position, self.eos_token_id)]
+
def host_fill_batch(self, batch: Batch, use_graph: bool):
"""Stage this batch's rows; returns the post-dispatch fill callable under flag-sync, else None."""
eos = self.eos_token_id
@@ -205,7 +218,7 @@ class DiskRowTable:
try:
self._readback_event.synchronize()
tokens = self._token_readback[:bs].to(torch.int64).tolist()
- runs = [torch.tensor([*_context(r.input_ids, r.device_len - 1, eos), t], dtype=torch.int64)
+ runs = [torch.tensor([*self._ple_context(r.input_ids, r.device_len - 1), t], dtype=torch.int64)
for r, t in zip(reqs, tokens)]
self.fill(runs, graph=True)
except BaseException:
@@ -218,14 +231,14 @@ class DiskRowTable:
return _complete
# launch-gating: this D2H is the step's readback and orders the fill after sampling
tokens = batch.input_ids.to("cpu").to(torch.int64).tolist()
- runs = [torch.tensor([*_context(r.input_ids, r.device_len - 1, eos), t], dtype=torch.int64)
+ runs = [torch.tensor([*self._ple_context(r.input_ids, r.device_len - 1), t], dtype=torch.int64)
for r, t in zip(reqs, tokens)]
self.fill(runs, graph=use_graph)
return None
runs = [
torch.cat((
- torch.tensor(_context(req.input_ids, req.cached_len, eos), dtype=torch.int64),
- req.input_ids[req.cached_len : req.device_len].to(torch.int64),
+ torch.tensor(self._ple_context(req.input_ids, req.cached_len), dtype=torch.int64),
+ self._ple_ids(req.input_ids[req.cached_len : req.device_len]).to(torch.int64),
))
for req in batch.padded_reqs
]
diff --git a/python/freetoken/models/qwen4_exp/weight.py b/python/freetoken/models/qwen4_exp/weight.py
index 4a2a5b61..0ad04011 100644
--- a/python/freetoken/models/qwen4_exp/weight.py
+++ b/python/freetoken/models/qwen4_exp/weight.py
@@ -6,7 +6,7 @@ Three separate paths, because the checkpoint's three weight classes live in diff
* :func:`load_ple_table` -- the 47.7 GiB FP8 n-gram table, 128 checkpoint shards concatenated into one pinned :class:`HostBank`.
* :func:`nvfp4_expert_spec` -- how the routed NVFP4 experts are named, for the offload cache's expert reader.
-Dropped: ``mtp.*`` (speculative head, including its stacked ``mtp.layers.0.mlp.experts.*``) and ``model.visual.*`` (served text-only).
+Dropped: ``mtp.*`` (speculative head, including its stacked ``mtp.layers.0.mlp.experts.*``); ``model.visual.*`` is kept only when the model built the tower.
"""
from __future__ import annotations
@@ -21,6 +21,9 @@ from typing import Iterator
import safetensors
import torch
from freetoken.distributed import get_tp_info
+from freetoken.models.qwen3_vl.weight import rename_vl_prefix
+
+from freetoken.models.config import VISION_KEY_PREFIXES
from freetoken.models.loader import drop_page_cache, iter_weight_files
from freetoken.models.nvfp4_banks import (
Nvfp4ExpertSourceSpec,
@@ -83,7 +86,7 @@ _ELEM_DTYPES = {"e4m3": torch.float8_e4m3fn}
def _rename(raw_name: str) -> str | None:
"""Checkpoint key -> FreeToken state-dict key, or None to skip."""
- if raw_name.startswith(("mtp.", "model.visual.", "visual.")):
+ if raw_name.startswith("mtp."):
return None
if _PLE_TABLE_INFIX in raw_name:
return None # n-gram table + its scale: load_ple_table
@@ -91,11 +94,7 @@ def _rename(raw_name: str) -> str | None:
return None # routed experts: offload source banks
if raw_name.endswith(_SCALE_SUFFIXES):
return None
- if raw_name.startswith("model.language_model."):
- return "model." + raw_name[len("model.language_model.") :]
- if raw_name.startswith("language_model."):
- return "model." + raw_name[len("language_model.") :]
- return raw_name
+ return rename_vl_prefix(raw_name)
def _split_kind(name: str) -> tuple[str, str]:
@@ -195,6 +194,7 @@ def iter_weights(
*,
include_moe_experts: bool,
include_non_moe: bool,
+ include_vision: bool = True,
) -> Iterator[tuple[str, torch.Tensor]]:
"""Yield the dense (non-expert) weights, prefix-stripped and fused to the model's buffers.
@@ -221,6 +221,8 @@ def iter_weights(
name = _rename(raw_name)
if name is None:
continue
+ if not include_vision and name.startswith(VISION_KEY_PREFIXES):
+ continue
tensor = f.get_tensor(raw_name)
fused = fuser.fuse(name, tensor)
if fused is None:
diff --git a/python/freetoken/models/register.py b/python/freetoken/models/register.py
index 957c0fa9..d0666f9d 100644
--- a/python/freetoken/models/register.py
+++ b/python/freetoken/models/register.py
@@ -8,6 +8,13 @@ if TYPE_CHECKING:
from .config import ModelConfig
+@dataclass(frozen=True)
+class EncoderSpec:
+ kind: str # "vision" | "audio", the name --mm-disable takes
+ config_key: str # the checkpoint config section that builds the tower
+ modalities: tuple[str, ...] # inputs it serves, as implemented
+
+
@dataclass(frozen=True)
class ModelSpec:
module: str
@@ -22,10 +29,17 @@ class ModelSpec:
packed_modules_mapping: tuple[tuple[str, tuple[str, ...]], ...] = ()
# checkpoint-name globs the family serves in bf16 although the quantization_config covers them
unquantized_modules: tuple[str, ...] = ()
+ # "module:Class" turning the checkpoint's media into items; None: the family takes no multimodal input
+ mm_processor: str | None = None
+ encoders: tuple[EncoderSpec, ...] = ()
# Multimodal wrappers store the text tower under model.language_model.
_LANGUAGE_MODEL_ROOT = (("model", "model.language_model"),)
+_QWEN_VL_ROOTS = _LANGUAGE_MODEL_ROOT + (("visual", "model.visual"),)
+_QWEN_VL_PROCESSOR = "freetoken.mm.processors.qwen_vl:QwenVLMMProcessor"
+# TODO: video once the processor samples frames; the tower already takes grid_thw with t > 1
+_QWEN_VL_ENCODERS = (EncoderSpec("vision", "vision_config", ("image",)),)
# Fused projections split back into their HF leaves so the quantization_config lookup sees the stored names.
_DENSE_PACKED = (
("qkv_proj", ("q_proj", "k_proj", "v_proj")),
@@ -34,6 +48,8 @@ _DENSE_PACKED = (
# per-expert checkpoints: probe the first expert's projections for the experts container
_EXPERTS_PACKED = (("experts", ("experts.0.gate_proj", "experts.0.up_proj", "experts.0.down_proj")),)
_EXPERTS_W123_PACKED = (("experts", ("experts.0.w1", "experts.0.w2", "experts.0.w3")),)
+# pre-stacked expert checkpoints (Qwen3-VL-MoE): the container's own two tensors are the probe
+_STACKED_EXPERTS_PACKED = (("experts", ("experts.gate_up_proj", "experts.down_proj")),)
_QWEN3_5_PACKED = _DENSE_PACKED + (
("in_proj_qkvz", ("in_proj_qkv", "in_proj_z")),
("in_proj_ba", ("in_proj_b", "in_proj_a")),
@@ -81,6 +97,23 @@ _MODEL_REGISTRY: dict[str, ModelSpec] = {
"Qwen3MoeForCausalLM",
packed_modules_mapping=_DENSE_PACKED + _EXPERTS_PACKED,
),
+ # Qwen3-VL: the Qwen3 text tower under model.language_model. plus the shared Qwen VL vision tower with DeepStack; the MoE variant ships its experts pre-stacked.
+ "Qwen3VLForConditionalGeneration": ModelSpec(
+ "freetoken.models.qwen3_vl",
+ "Qwen3VLForConditionalGeneration",
+ checkpoint_roots=_QWEN_VL_ROOTS,
+ mm_processor=_QWEN_VL_PROCESSOR,
+ encoders=_QWEN_VL_ENCODERS,
+ packed_modules_mapping=_DENSE_PACKED,
+ ),
+ "Qwen3VLMoeForConditionalGeneration": ModelSpec(
+ "freetoken.models.qwen3_vl",
+ "Qwen3VLMoeForConditionalGeneration",
+ checkpoint_roots=_QWEN_VL_ROOTS,
+ mm_processor=_QWEN_VL_PROCESSOR,
+ encoders=_QWEN_VL_ENCODERS,
+ packed_modules_mapping=_DENSE_PACKED + _STACKED_EXPERTS_PACKED,
+ ),
"MiniMaxM2ForCausalLM": ModelSpec(
"freetoken.models.minimax_m2",
"MiniMaxM2ForCausalLM",
@@ -113,19 +146,36 @@ _MODEL_REGISTRY: dict[str, ModelSpec] = {
),
"Qwen3_5MoeForConditionalGeneration": ModelSpec(
"freetoken.models.qwen3_5_moe",
- "Qwen3_5MoEForCausalLM",
- checkpoint_roots=_LANGUAGE_MODEL_ROOT,
+ "Qwen3_5MoeForConditionalGeneration",
+ checkpoint_roots=_QWEN_VL_ROOTS,
+ mm_processor=_QWEN_VL_PROCESSOR,
+ encoders=_QWEN_VL_ENCODERS,
+ packed_modules_mapping=_QWEN3_5_PACKED,
+ unquantized_modules=_QWEN3_5_UNQUANTIZED,
+ ),
+ # text-only Qwen3.5 releases: text_config is the top-level config and there is no tower
+ "Qwen3_5MoeForCausalLM": ModelSpec(
+ "freetoken.models.qwen3_5_moe",
+ "Qwen3_5MoeForCausalLM",
+ packed_modules_mapping=_QWEN3_5_PACKED,
+ unquantized_modules=_QWEN3_5_UNQUANTIZED,
+ ),
+ "Qwen3_5ForCausalLM": ModelSpec(
+ "freetoken.models.qwen3_5_moe",
+ "Qwen3_5ForCausalLM",
packed_modules_mapping=_QWEN3_5_PACKED,
unquantized_modules=_QWEN3_5_UNQUANTIZED,
),
# Qwen3.8-Flash-Next (model_type qwen4_exp): multimodal wrapper config (text tower in
- # text_config, weights under model.language_model.); served text-only. 36 GDN + 12 QSA
+ # text_config, weights under model.language_model.). 36 GDN + 12 QSA
# compressed-sparse attention layers on 4 hyper-connection residual streams, a PLE
# n-gram embedding layer, 512 NVFP4 routed experts top-10 + a gated shared expert.
"Qwen4ExpForConditionalGeneration": ModelSpec(
"freetoken.models.qwen4_exp",
- "Qwen4ExpForCausalLM",
- checkpoint_roots=_LANGUAGE_MODEL_ROOT,
+ "Qwen4ExpForConditionalGeneration",
+ checkpoint_roots=_QWEN_VL_ROOTS,
+ mm_processor=_QWEN_VL_PROCESSOR,
+ encoders=_QWEN_VL_ENCODERS,
packed_modules_mapping=_QWEN4_EXP_PACKED,
),
# Dense Qwen3.x (no "Moe" in the arch name, num_experts==0, e.g. Qwen3.6-27B). Shares the
@@ -133,8 +183,10 @@ _MODEL_REGISTRY: dict[str, ModelSpec] = {
# loader handles the compressed-tensors NVFP4 layout.
"Qwen3_5ForConditionalGeneration": ModelSpec(
"freetoken.models.qwen3_5_moe",
- "Qwen3_5MoEForCausalLM",
- checkpoint_roots=_LANGUAGE_MODEL_ROOT,
+ "Qwen3_5ForConditionalGeneration",
+ checkpoint_roots=_QWEN_VL_ROOTS,
+ mm_processor=_QWEN_VL_PROCESSOR,
+ encoders=_QWEN_VL_ENCODERS,
packed_modules_mapping=_QWEN3_5_PACKED,
unquantized_modules=_QWEN3_5_UNQUANTIZED,
),
diff --git a/python/freetoken/models/weight.py b/python/freetoken/models/weight.py
index 0cff7eec..5165d149 100644
--- a/python/freetoken/models/weight.py
+++ b/python/freetoken/models/weight.py
@@ -211,6 +211,7 @@ def load_weight(
device: torch.device,
*,
include_moe_experts: bool = True,
+ include_vision: bool = True,
) -> Iterator[Tuple[str, torch.Tensor]]:
# FTW checkpoint: dense weights are stored post-iter_weights, so we replay them
# model-agnostically instead of re-running the per-model reader. Which tensors exist is
@@ -218,28 +219,27 @@ def load_weight(
# fails loudly in load_state_dict (strict missing/unexpected expert keys), so the reader
# just yields the stored weight tensors regardless of the include_moe_experts flag.
from freetoken.checkpoint.ftw import is_ftw_checkpoint, iter_ftw_weights
- from freetoken.models.config import VISION_KEY_PREFIXES, vision_load_enabled
+ from freetoken.models.config import VISION_KEY_PREFIXES
if is_ftw_checkpoint(model_path):
- # The FTW dense shard stores whatever existed at conversion, including the vision
- # stack. Vision is opt-in (default OFF, see vision_load_enabled): when it is off the
- # model never builds the tower, so replaying those tensors would trip load_state_dict's
- # strict unexpected-key check. Skip them here to match the model the engine built.
- skip_vision = not vision_load_enabled()
- for name, tensor in iter_ftw_weights(model_path):
- if skip_vision and name.startswith(VISION_KEY_PREFIXES):
- continue
- yield name, tensor
- return
-
- _config, spec = _spec_for_model_path(model_path)
- iter_weights = _load_attr(spec.module, spec.iter_weights)
- yield from iter_weights(
- model_path,
- device,
- include_moe_experts=include_moe_experts,
- include_non_moe=True,
- )
+ weights = iter_ftw_weights(model_path)
+ else:
+ _config, spec = _spec_for_model_path(model_path)
+ iter_weights = _load_attr(spec.module, spec.iter_weights)
+ # only a family that registers an encoder is asked about the tower; the others never load one
+ kwargs = {"include_vision": include_vision} if spec.encoders else {}
+ weights = iter_weights(
+ model_path,
+ device,
+ include_moe_experts=include_moe_experts,
+ include_non_moe=True,
+ **kwargs,
+ )
+ for name, tensor in weights:
+ # the FTW replay has no reader to skip the tower; a text-only engine never built it
+ if not include_vision and name.startswith(VISION_KEY_PREFIXES):
+ continue
+ yield name, tensor
def load_q4_0_moe_expert_sources(
diff --git a/python/freetoken/models/weight_stream.py b/python/freetoken/models/weight_stream.py
new file mode 100644
index 00000000..ac529eca
--- /dev/null
+++ b/python/freetoken/models/weight_stream.py
@@ -0,0 +1,122 @@
+"""Block weights streamed from pinned host banks through two device staging buffers: block i computes from buffer i % 2 while the copy stream fills the other with block i + 1."""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+from typing import Any, Iterator, List, Sequence
+
+import torch
+
+from freetoken.layers import BaseOP
+
+_ALIGN = 256
+
+
+@dataclass(frozen=True)
+class _Slot:
+ owner: Any
+ attr: str
+ offset: int
+ nbytes: int
+ dtype: torch.dtype
+ shape: torch.Size
+
+ def view(self, row: torch.Tensor) -> torch.Tensor:
+ return row[self.offset : self.offset + self.nbytes].view(self.dtype).view(self.shape)
+
+
+def _slots(block: BaseOP) -> tuple[List[_Slot], int]:
+ """Every tensor attribute of the block (recursively), laid out back to back with 256-byte alignment."""
+ slots: List[_Slot] = []
+ offset = 0
+
+ def walk(op: BaseOP) -> None:
+ nonlocal offset
+ for name, value in op.__dict__.items():
+ if name.startswith("_"):
+ continue
+ if isinstance(value, torch.Tensor):
+ nbytes = value.numel() * value.element_size()
+ slots.append(_Slot(op, name, offset, nbytes, value.dtype, value.shape))
+ offset += -(-nbytes // _ALIGN) * _ALIGN
+ elif isinstance(value, BaseOP):
+ walk(value)
+
+ walk(block)
+ return slots, offset
+
+
+class BlockWeightStreamer:
+ """Owns the host banks and the two staging buffers of a block stack; drives one forward with `blocks()`."""
+
+ def __init__(self, blocks: Sequence[BaseOP], device: torch.device):
+ self.device = device
+ self._layouts = [_slots(b) for b in blocks]
+ row_bytes = self._layouts[0][1]
+ assert all(nbytes == row_bytes for _, nbytes in self._layouts), "streamed blocks must share one layout"
+ self.bank = torch.empty((len(blocks), row_bytes), dtype=torch.uint8, pin_memory=True)
+ self.staging = torch.empty((2, row_bytes), dtype=torch.uint8, device=device)
+ for b, (slots, _) in enumerate(self._layouts):
+ for s in slots:
+ s.view(self.bank[b]).copy_(getattr(s.owner, s.attr))
+ torch.cuda.synchronize(device)
+ self._bind_all_to_host()
+ self.copy_stream = torch.cuda.Stream(device=device)
+ self.begin_event = torch.cuda.Event()
+ self.ready_events = [torch.cuda.Event() for _ in range(2)]
+ self.release_events = [torch.cuda.Event() for _ in range(2)]
+ self._holder: List[int | None] = [None, None]
+ self._has_release: List[bool] = [False, False]
+
+ @property
+ def device_bytes(self) -> int:
+ return self.staging.numel()
+
+ def _bind_all_to_host(self) -> None:
+ # between forwards the attributes point at the pinned bank, so state_dict() stays complete and correct
+ for b, (slots, _) in enumerate(self._layouts):
+ for s in slots:
+ setattr(s.owner, s.attr, s.view(self.bank[b]))
+
+ def unstream(self) -> None:
+ """Put every block's tensors back on the device as plain resident copies."""
+ for b, (slots, _) in enumerate(self._layouts):
+ for s in slots:
+ setattr(s.owner, s.attr, s.view(self.bank[b]).to(self.device))
+ torch.cuda.synchronize(self.device)
+
+ def _prefetch(self, i: int) -> None:
+ if i >= len(self._layouts):
+ return
+ buf = i % 2
+ if self._holder[buf] == i:
+ return
+ with torch.cuda.stream(self.copy_stream):
+ if self._has_release[buf]:
+ self.copy_stream.wait_event(self.release_events[buf])
+ self.staging[buf].copy_(self.bank[i], non_blocking=True)
+ self.ready_events[buf].record(self.copy_stream)
+ self._holder[buf] = i
+
+ def blocks(self, ops: Sequence[BaseOP]) -> Iterator[tuple[int, BaseOP]]:
+ """Yield (i, block) with block i's tensors bound to a filled staging buffer while block i + 1 streams in."""
+ compute = torch.cuda.current_stream(self.device)
+ # the copy stream must not overwrite staging that a still-running previous forward reads
+ self.begin_event.record(compute)
+ self.copy_stream.wait_event(self.begin_event)
+ for i, op in enumerate(ops):
+ buf = i % 2
+ self._prefetch(i)
+ self._prefetch(i + 1)
+ compute.wait_event(self.ready_events[buf])
+ slots, _ = self._layouts[i]
+ for s in slots:
+ setattr(s.owner, s.attr, s.view(self.staging[buf]))
+ yield i, op
+ self.release_events[buf].record(compute)
+ self._has_release[buf] = True
+ for s in slots:
+ setattr(s.owner, s.attr, s.view(self.bank[i]))
+
+
+__all__ = ["BlockWeightStreamer"]
diff --git a/python/freetoken/scheduler/cache.py b/python/freetoken/scheduler/cache.py
index 44adde42..8f77be93 100644
--- a/python/freetoken/scheduler/cache.py
+++ b/python/freetoken/scheduler/cache.py
@@ -93,10 +93,7 @@ class CacheManager:
def match_req(self, req: PendingReq) -> MatchResult:
input_len = req.input_len
assert input_len > 0, "Input length must be greater than 0."
- # Multimodal requests must not reuse a shared prefix: image-placeholder tokens
- # have identical ids across images but carry different content (and KV), so a
- # match would serve the wrong image's KV. Match against the empty prefix.
- ids = req.input_ids[:0] if req.mm_embeds is not None else req.input_ids[: input_len - 1]
+ ids = req.input_ids[: input_len - 1]
if self.is_swa:
from freetoken.kvcache.swa_radix_cache import SWACacheHandle
m = self.prefix_cache.match_prefix(ids)
@@ -299,17 +296,6 @@ class CacheManager:
# We should free it if the request has finished.
page_indices = self.page_table[req.table_idx, : req.cached_len]
old_handle = req.cache_handle
- # Multimodal requests are never inserted into the shared prefix cache (see
- # ``match_req``). Their KV pages stay owned by the active request and are freed
- # on completion; nothing is exposed for cross-request reuse.
- if req.mm_embeds is not None:
- self.unlock(old_handle)
- if finished:
- tail = self._padded_tail(req, old_handle.cached_len)
- if self.swa_paged:
- self._free_swa(tail)
- self._free(tail)
- return
insert_ids = req.input_ids[: req.cached_len]
cached_len, new_handle = self.prefix_cache.insert_prefix(insert_ids, page_indices)
# unlock until all operations on handle is done
@@ -350,13 +336,6 @@ class CacheManager:
old_handle = req.cache_handle
page_indices = self.page_table[req.table_idx, : req.cached_len]
- if req.mm_embeds is not None:
- self.unlock(old_handle)
- if finished:
- self._free(page_indices[old_handle.cached_len :])
- self._free_req_slots(req)
- return
-
if finished:
# A pending freeze (the tool-call anchor, or a prefill ×64 track the request
# finished too early to chunk-commit) is a strictly shorter prefix than the live
@@ -444,14 +423,6 @@ class CacheManager:
old_handle = req.cache_handle
page_indices = self.page_table[req.table_idx, : req.cached_len]
- if req.mm_embeds is not None:
- self.unlock(old_handle)
- if finished:
- tail = self._padded_tail(req, old_handle.cached_len)
- self._free_swa(tail)
- self._free(tail)
- return
-
insert_len = align_down(req.cached_len, self.page_size)
freed = page_indices[:0]
if insert_len > 0:
diff --git a/python/freetoken/scheduler/mm.py b/python/freetoken/scheduler/mm.py
new file mode 100644
index 00000000..5d9e4d86
--- /dev/null
+++ b/python/freetoken/scheduler/mm.py
@@ -0,0 +1,64 @@
+"""Scheduler-side multimodal planning: the embedding rows a chunk gathers and the items it encodes."""
+
+from __future__ import annotations
+
+from typing import TYPE_CHECKING, List, Tuple
+
+if TYPE_CHECKING:
+ from freetoken.message import MMItem
+ from freetoken.mm.encoder_cache import EncoderCache
+
+
+def mm_rows_after(item: MMItem, cached_len: int) -> int:
+ """Embedding rows of the item that prefill will gather once the first cached_len tokens are skipped."""
+ return sum(max(0, span_hi - max(span_lo, cached_len)) for span_lo, span_hi in item.offsets)
+
+
+def plan_mm_chunk(
+ uid: int,
+ items: List[MMItem],
+ window_lo: int,
+ window_hi: int,
+ encoder_cache: EncoderCache | None,
+) -> tuple[List[MMItem], List[Tuple[int, int, int, int, int, int]]]:
+ """Encoder jobs and the embedding-row gather plan for the items overlapping the chunk window [window_lo, window_hi).
+
+ Plan rows are (uid, hash, row_lo, row_hi, num_tokens, pos): embedding rows [row_lo, row_hi) land at chunk position pos. A repeated image is one job.
+ """
+ jobs: List[MMItem] = []
+ queued: set[int] = set()
+ plan: List[Tuple[int, int, int, int, int, int]] = []
+ for item in items:
+ num_tokens = item.num_tokens
+ needs_encode = item.hash not in queued and (encoder_cache is None or not encoder_cache.has(item.hash))
+ row_base = 0
+ for span_lo, span_hi in item.offsets:
+ lo, hi = max(span_lo, window_lo), min(span_hi, window_hi)
+ if lo < hi:
+ if needs_encode:
+ jobs.append(item)
+ queued.add(item.hash)
+ needs_encode = False
+ plan.append((uid, item.hash, row_base + lo - span_lo, row_base + hi - span_lo, num_tokens, lo - window_lo))
+ row_base += span_hi - span_lo
+ return jobs, plan
+
+
+def plan_mm_batch(reqs, encoder_cache: EncoderCache | None) -> tuple[List[MMItem], List[Tuple[int, int, int, int, int, int]], List[int]]:
+ """Jobs, plan and the batch rows of every gathered embedding row, over the reqs in batch order (each spans [cached_len, device_len))."""
+ jobs: List[MMItem] = []
+ plan: List[Tuple[int, int, int, int, int, int]] = []
+ rows: List[int] = []
+ offset = 0
+ for req in reqs:
+ if req.mm_items:
+ req_jobs, req_plan = plan_mm_chunk(req.uid, req.mm_items, req.cached_len, req.device_len, encoder_cache)
+ jobs.extend(req_jobs)
+ plan.extend(req_plan)
+ for _, _, row_lo, row_hi, _, pos in req_plan:
+ rows.extend(range(offset + pos, offset + pos + row_hi - row_lo))
+ offset += req.extend_len
+ return jobs, plan, rows
+
+
+__all__ = ["mm_rows_after", "plan_mm_batch", "plan_mm_chunk"]
diff --git a/python/freetoken/scheduler/prefill.py b/python/freetoken/scheduler/prefill.py
index 60becf0e..3d94b53a 100644
--- a/python/freetoken/scheduler/prefill.py
+++ b/python/freetoken/scheduler/prefill.py
@@ -7,11 +7,13 @@ import torch
from freetoken.core import Batch, Req
from freetoken.utils import align_down, div_ceil, init_logger
+from .mm import mm_rows_after
from .utils import PendingReq
if TYPE_CHECKING:
from freetoken.kvcache import BaseCacheHandle
from freetoken.message import UserMsg
+ from freetoken.mm.encoder_cache import EncoderCache
from .cache import CacheManager
from .decode import DecodeManager
@@ -43,6 +45,7 @@ class PrefillAdder:
reserved_size: int
cache_manager: CacheManager
table_manager: TableManager
+ encoder_cache: EncoderCache | None = None
# SWA-pool tokens charged to reqs admitted so far this pass. Mirrors reserved_size: swa is
# allocated only in allocate_paged (after the pass), so swa_available_size does not decrement
# across the admission loop -- without this, successive admits all see the full pool.
@@ -185,11 +188,6 @@ class PrefillAdder:
_slice = slice(cached_len, cached_len + chunk_size)
device_ids = self.table_manager.token_pool[table_idx, _slice]
device_ids.copy_(_maybe_pinned(pending_req.input_ids[_slice]), non_blocking=True)
- if is_chunked and pending_req.mm_embeds is not None:
- raise NotImplementedError(
- "Multimodal prompts must fit in a single prefill chunk; increase "
- "--max-extend-tokens or shrink the prompt."
- )
req = CLS(
input_ids=pending_req.input_ids[: cached_len + chunk_size],
table_idx=table_idx,
@@ -198,8 +196,10 @@ class PrefillAdder:
uid=pending_req.uid,
cache_handle=cache_handle,
sampling_params=pending_req.sampling_params,
- mm_embeds=pending_req.mm_embeds,
)
+ req.mm_items = pending_req.mm_items
+ req.mrope_positions_full = pending_req.mrope_positions_full
+ req.mrope_delta = pending_req.mrope_delta
# Hybrid GDN per-request state slots (None for non-hybrid). On a fresh admit these are
# freshly allocated; on a chunked continuation they are inherited from the prior chunk.
req.linear_slot_idx = linear_slot_idx
@@ -255,11 +255,19 @@ class PrefillManager:
cache_manager: CacheManager
table_manager: TableManager
decode_manager: DecodeManager
+ encoder_cache: EncoderCache | None = None
pending_list: List[PendingReq] = field(default_factory=list)
def add_one_req(self, req: UserMsg) -> None:
self.pending_list.append(
- PendingReq(req.uid, req.input_ids, req.sampling_params, mm_embeds=req.mm_embeds)
+ PendingReq(
+ req.uid,
+ req.input_ids,
+ req.sampling_params,
+ mm_items=req.mm_items,
+ mrope_positions_full=req.mrope_positions,
+ mrope_delta=req.mrope_delta,
+ )
)
def schedule_next_batch(self, prefill_budget: int) -> Batch | None:
@@ -272,6 +280,7 @@ class PrefillManager:
reserved_size=self.decode_manager.inflight_tokens,
cache_manager=self.cache_manager,
table_manager=self.table_manager,
+ encoder_cache=self.encoder_cache,
)
reqs: List[Req] = []
chunked_list: List[PendingReq] = []
@@ -296,6 +305,12 @@ class PrefillManager:
prompt_admissions.append(
(req.uid, pending_req.input_len, req.cache_handle.cached_len)
)
+ if pending_req.mm_items and self.encoder_cache is not None:
+ # claim the rows every chunk of this request will gather; the entry outlives the chunks
+ for item in pending_req.mm_items:
+ self.encoder_cache.register(
+ item.hash, req.uid, mm_rows_after(item, req.cache_handle.cached_len)
+ )
log_new_tokens += req.extend_len
if not is_continuation:
log_cached_tokens += req.cache_handle.cached_len
diff --git a/python/freetoken/scheduler/scheduler.py b/python/freetoken/scheduler/scheduler.py
index 48923e3b..28520e13 100644
--- a/python/freetoken/scheduler/scheduler.py
+++ b/python/freetoken/scheduler/scheduler.py
@@ -1,5 +1,7 @@
from __future__ import annotations
+import os
+
from typing import TYPE_CHECKING, List, NamedTuple, NoReturn, Set, Tuple, TypeAlias
import torch
@@ -30,6 +32,7 @@ from .cache import CacheManager
from .config import SchedulerConfig
from .decode import DecodeManager
from .io import SchedulerIOMixin
+from .mm import plan_mm_batch
from .prefill import ChunkedReq, PrefillManager
from .status import SchedulerStatusReporter
from .table import TableManager
@@ -89,7 +92,10 @@ class Scheduler(SchedulerIOMixin):
)
self.decode_manager = DecodeManager(config.page_size)
self.prefill_manager = PrefillManager(
- self.cache_manager, self.table_manager, self.decode_manager
+ self.cache_manager,
+ self.table_manager,
+ self.decode_manager,
+ encoder_cache=self.engine.encoder_cache,
)
# some alias for easy access
@@ -133,6 +139,7 @@ class Scheduler(SchedulerIOMixin):
min(config.max_extend_tokens, _chunk_cap) if _chunk_cap else config.max_extend_tokens
)
self.config = config
+ self._model_is_mrope = config.model_config.model_is_mrope
self.status_reporter = SchedulerStatusReporter(
log=logger.info_rank0,
decode_log_interval=config.decode_log_interval,
@@ -489,6 +496,17 @@ class Scheduler(SchedulerIOMixin):
"Dropping request %d because its abort arrived before admission", msg.uid
)
return
+ if msg.mm_items and self.engine.encoder_cache is None:
+ # no encoder runtime: fail loudly instead of decoding unexpanded placeholders
+ self.send_result(
+ [
+ ErrorReplyMsg(
+ uid=msg.uid,
+ error="image input is not supported by this server",
+ )
+ ]
+ )
+ return
input_len, max_seq_len = len(msg.input_ids), self.engine.max_seq_len
max_output_len = max_seq_len - input_len
if max_output_len <= 0:
@@ -534,6 +552,15 @@ class Scheduler(SchedulerIOMixin):
tombstones.pop(next(iter(tombstones)))
req_to_free = self.prefill_manager.abort_req(msg.uid)
req_to_free = req_to_free or self.decode_manager.abort_req(msg.uid)
+ if (
+ req_to_free is not None
+ and req_to_free.mm_items
+ and self.engine.encoder_cache is not None
+ ):
+ # drop the aborted request's claims; entries it held alone die here
+ self.engine.encoder_cache.release(
+ msg.uid, [item.hash for item in req_to_free.mm_items]
+ )
if req_to_free is not None:
# SGLang-style abort: never free resources under an in-flight forward. If the
# request is in the launched-but-not-drained batch (overlap), only mark it;
@@ -781,6 +808,8 @@ class Scheduler(SchedulerIOMixin):
if batch.is_prefill:
self._gather_multimodal(batch)
batch.positions = _make_positions(batch, self.device)
+ if self._model_is_mrope:
+ batch.mrope_positions = _make_mrope_positions(batch, self.device)
input_mapping = _make_input_tuple(batch, self.device)
write_mapping = _make_write_tuple(batch, self.device)
batch.out_loc = self.engine.page_table[input_mapping]
@@ -819,14 +848,12 @@ class Scheduler(SchedulerIOMixin):
)
def _gather_multimodal(self, batch: Batch) -> None:
- """Concatenate per-request vision soft tokens (in request order) for a prefill
- batch so the model can scatter them at image-token positions. ``req.mm_embeds``
- is kept (not cleared) so the cache manager can recognize multimodal requests and
- keep them out of the shared prefix cache (image placeholders share a token id but
- carry per-image content)."""
- parts = [req.mm_embeds for req in batch.reqs if req.mm_embeds is not None]
- if parts:
- batch.mm_embeds = torch.cat(parts, dim=0)
+ """Plan the chunk's encoder jobs, gather rows and scatter rows over the batch; the engine runs them before the LM forward."""
+ jobs, plan, rows = plan_mm_batch(batch.padded_reqs, self.engine.encoder_cache)
+ if plan:
+ batch.mm_encoder_jobs = jobs
+ batch.mm_gather_plan = plan
+ batch.mm_rows = torch.tensor(rows, dtype=torch.int64, pin_memory=True).to(self.device, non_blocking=True)
def _schedule_next_batch(self) -> ForwardInput | None:
# TODO: support other policies: e.g. DECODE first
@@ -874,6 +901,28 @@ class Scheduler(SchedulerIOMixin):
return forward_output
+def _make_mrope_positions(batch: Batch, device: torch.device) -> torch.Tensor:
+ """[3, N] rope rows: an image request's prompt tokens use their precomputed columns, everything else is sequence index + per-request delta."""
+ needed = sum(r.extend_len for r in batch.padded_reqs)
+ host = torch.empty((3, needed), dtype=torch.int32, pin_memory=True)
+ offset = 0
+ for req in batch.padded_reqs:
+ length = req.extend_len
+ out = host[:, offset : offset + length]
+ full = req.mrope_positions_full
+ if full is not None and req.device_len <= full.shape[1]:
+ out.copy_(full[:, req.cached_len : req.device_len])
+ else:
+ row = torch.arange(
+ req.cached_len + req.mrope_delta,
+ req.device_len + req.mrope_delta,
+ dtype=torch.int32,
+ )
+ out.copy_(row.unsqueeze(0).expand(3, -1))
+ offset += length
+ return host.to(device, non_blocking=True)
+
+
def _make_positions(batch: Batch, device: torch.device) -> torch.Tensor:
needed_size = sum(r.extend_len for r in batch.padded_reqs)
indices_host = torch.empty(needed_size, dtype=torch.int32, pin_memory=True)
diff --git a/python/freetoken/scheduler/utils.py b/python/freetoken/scheduler/utils.py
index 0d6dd512..83b79f2a 100644
--- a/python/freetoken/scheduler/utils.py
+++ b/python/freetoken/scheduler/utils.py
@@ -17,7 +17,9 @@ class PendingReq:
input_ids: torch.Tensor
sampling_params: SamplingParams
chunked_req: ChunkedReq | None = None
- mm_embeds: torch.Tensor | None = None
+ mm_items: list | None = None
+ mrope_positions_full: torch.Tensor | None = None
+ mrope_delta: int = 0
@property
def input_len(self) -> int:
diff --git a/python/freetoken/server/anthropic_api.py b/python/freetoken/server/anthropic_api.py
index 18911673..e42e0805 100644
--- a/python/freetoken/server/anthropic_api.py
+++ b/python/freetoken/server/anthropic_api.py
@@ -218,7 +218,21 @@ def convert_anthropic_prompt(
# -> reasoning_content; redacted_thinking stays skipped (opaque payload).
thinking_parts.append(block.thinking)
elif block.type == "image":
- # Text-only server: drop image blocks rather than failing the request.
+ src = block.source or {}
+ stype = src.get("type")
+ data = src.get("data") if stype == "base64" else src.get("url")
+ if not data:
+ # an unsupported image source must fail the request, not degrade to a text-only answer
+ raise ValueError(f"unsupported image source type: {stype!r}")
+ content_parts.append(
+ {
+ "type": "image",
+ "freetoken_ref": {
+ "kind": "b64" if stype == "base64" else "url",
+ "data": data,
+ },
+ }
+ )
continue
elif block.type == "tool_use":
tool_calls.append(
@@ -347,6 +361,9 @@ def _tool_result_text(content) -> str:
parts: list[str] = []
for item in content:
if isinstance(item, dict):
+ if item.get("type") == "image":
+ # chat templates render tool messages as plain text, so an image here has nowhere to go
+ raise ValueError("images inside tool results are not supported")
parts.append(item.get("text") or "")
else:
parts.append(str(item))
diff --git a/python/freetoken/server/api_server.py b/python/freetoken/server/api_server.py
index 3e2acc85..5592a2b0 100644
--- a/python/freetoken/server/api_server.py
+++ b/python/freetoken/server/api_server.py
@@ -206,10 +206,13 @@ class FrontendManager:
``_frontend_tokenizer`` field). Called from a worker thread — never on the event loop."""
with self._frontend_tokenizer_lock:
if self._frontend_tokenizer is None:
+ from freetoken.mm.processor import get_mm_processor
from freetoken.tokenizer.tokenize import TokenizeManager
from freetoken.utils import load_tokenizer
- self._frontend_tokenizer = TokenizeManager(load_tokenizer(self.config.model_path))
+ self._frontend_tokenizer = TokenizeManager(
+ load_tokenizer(self.config.model_path), get_mm_processor(self.config.model_path, self.config.mm)
+ )
return self._frontend_tokenizer
def warm_frontend_tokenizer(self) -> None:
diff --git a/python/freetoken/server/args.py b/python/freetoken/server/args.py
index 32e8d126..ab2fb9b7 100644
--- a/python/freetoken/server/args.py
+++ b/python/freetoken/server/args.py
@@ -1,11 +1,13 @@
from __future__ import annotations
import argparse
+import json
import os
from dataclasses import dataclass
from typing import List, Tuple
import torch
+from freetoken.mm.config import ENCODER_KINDS, MultimodalConfig
from freetoken.distributed import DistributedInfo
from freetoken.scheduler import SchedulerConfig
from freetoken.utils import init_logger
@@ -57,6 +59,10 @@ class ServerArgs(SchedulerConfig):
# prompt_tokens_details.cached_tokens, Anthropic cache_read_input_tokens, Responses
# input_tokens_details.cached_tokens). Mirrors sglang's --enable-cache-report.
enable_cache_report: bool = False
+ # Comma-separated hostname allowlist for client-supplied image URLs; empty admits any domain.
+ allowed_media_domains: str = ""
+ # Directory file:// image refs may be read from; empty rejects local files.
+ allowed_local_media_path: str = ""
# Comma-separated CORS allow-list for browser/webview clients (e.g. the desktop
# app). Empty string disables CORS headers entirely; "*" allows any origin.
cors_origins: str = "tauri://localhost,http://tauri.localhost,http://localhost:1420"
@@ -98,6 +104,16 @@ class ServerArgs(SchedulerConfig):
return f"tcp://127.0.0.1:{self.server_port + 1}"
+def _json_object(text: str) -> dict:
+ try:
+ value = json.loads(text)
+ except json.JSONDecodeError as exc:
+ raise argparse.ArgumentTypeError(f"not valid JSON: {exc}") from None
+ if not isinstance(value, dict):
+ raise argparse.ArgumentTypeError("expected a JSON object")
+ return value
+
+
def parse_args(
args: List[str],
run_shell: bool = False,
@@ -424,6 +440,77 @@ def parse_args(
"as a GDN-aware radix (cross-request GDN-state prefix reuse); pass 'naive' to opt out.",
)
+ parser.add_argument(
+ "--text-model-only",
+ action="store_true",
+ default=False,
+ help="Serve a multimodal checkpoint text-only: no encoder tower is built (its VRAM goes to "
+ "the KV/expert pools) and every multimodal input is rejected. Same as --mm-disable with "
+ "every encoder kind.",
+ )
+ parser.add_argument(
+ "--mm-disable",
+ nargs="+",
+ choices=list(ENCODER_KINDS),
+ default=[],
+ metavar="{vision,audio}",
+ help="Encoder towers to leave unbuilt; every input they would serve is rejected.",
+ )
+
+ parser.add_argument(
+ "--image-min-tokens",
+ type=_positive_int,
+ default=MultimodalConfig.image_min_tokens,
+ help="Fewest tokens an image may take: the image processor scales smaller images up to it, "
+ "in the family's own units. Default: the processor's own limit.",
+ )
+ parser.add_argument(
+ "--image-max-tokens",
+ type=_positive_int,
+ default=MultimodalConfig.image_max_tokens,
+ help="Most tokens an image may take: the image processor scales larger images down to it, "
+ "in the family's own units (Qwen VL: one token per 32x32 pixels). Default: the processor's own limit.",
+ )
+ parser.add_argument(
+ "--mm-processor-kwargs",
+ type=_json_object,
+ default=None,
+ metavar="JSON",
+ help="JSON object of extra keyword arguments for the checkpoint's image processor call, "
+ "for family-specific knobs; applied after the token budget.",
+ )
+
+ parser.add_argument(
+ "--mm-embed-cache-device",
+ choices=["cpu", "cuda"],
+ default=MultimodalConfig.embed_cache_device,
+ help="Storage for encoded image embeddings between prefill chunks.",
+ )
+
+ parser.add_argument(
+ "--mm-encoder-weights",
+ choices=["gpu", "host"],
+ default=MultimodalConfig.encoder_weights,
+ help="Encoder tower block weights: pinned host banks streamed two blocks at a time behind the "
+ "compute (default, about 60 MiB of VRAM instead of the whole tower), or resident on the GPU.",
+ )
+
+ parser.add_argument(
+ "--allowed-media-domains",
+ type=str,
+ default=ServerArgs.allowed_media_domains,
+ help="Comma-separated hostname allowlist for client-supplied image URLs. "
+ "Empty (default) allows any domain.",
+ )
+
+ parser.add_argument(
+ "--allowed-local-media-path",
+ type=str,
+ default=ServerArgs.allowed_local_media_path,
+ help="Directory that file:// image refs may be read from. "
+ "Unset (default) rejects local files.",
+ )
+
parser.add_argument(
"--enable-cache-report",
action="store_true",
@@ -724,6 +811,13 @@ def parse_args(
if kwargs["model_path"].startswith("~"):
kwargs["model_path"] = os.path.expanduser(kwargs["model_path"])
+ # a bad media root is a deployment mistake; fail at startup, not per request
+ if kwargs["allowed_local_media_path"]:
+ media_root = os.path.realpath(os.path.expanduser(kwargs["allowed_local_media_path"]))
+ if not os.path.isdir(media_root):
+ parser.error(f"--allowed-local-media-path {media_root} is not a directory")
+ kwargs["allowed_local_media_path"] = media_root
+
if kwargs["served_model_name"] is None:
kwargs["served_model_name"] = (
os.path.basename(os.path.normpath(kwargs["model_path"])) or kwargs["model_path"]
@@ -785,6 +879,19 @@ def parse_args(
kwargs["tp_info"] = DistributedInfo(0, kwargs["tensor_parallel_size"])
del kwargs["tensor_parallel_size"]
+ disabled = set(ENCODER_KINDS) if kwargs.pop("text_model_only") else set()
+ disabled.update(kwargs.pop("mm_disable"))
+ image_min_tokens, image_max_tokens = kwargs.pop("image_min_tokens"), kwargs.pop("image_max_tokens")
+ if image_min_tokens is not None and image_max_tokens is not None and image_min_tokens > image_max_tokens:
+ parser.error(f"--image-min-tokens {image_min_tokens} exceeds --image-max-tokens {image_max_tokens}")
+ kwargs["mm"] = MultimodalConfig(
+ disabled_encoders=frozenset(disabled),
+ embed_cache_device=kwargs.pop("mm_embed_cache_device"),
+ encoder_weights=kwargs.pop("mm_encoder_weights"),
+ image_min_tokens=image_min_tokens,
+ image_max_tokens=image_max_tokens,
+ processor_kwargs=kwargs.pop("mm_processor_kwargs") or {},
+ )
result = ServerArgs(**kwargs)
logger.info(f"Parsed arguments:\n{result}")
return result, run_shell
diff --git a/python/freetoken/server/generation.py b/python/freetoken/server/generation.py
index 28f85d87..91056e8d 100644
--- a/python/freetoken/server/generation.py
+++ b/python/freetoken/server/generation.py
@@ -23,6 +23,7 @@ from typing import Any
from . import request_ring
from freetoken.core import SamplingParams
from freetoken.message import TokenizeMsg
+from freetoken.mm.media import collect_image_refs, fetch_image_bytes, image_reject_reason
from freetoken.tokenizer.tokenize import resolve_thinking_mode
try:
@@ -42,12 +43,14 @@ from .reasoning_parser import (
)
-class GenerationError(Exception):
+class GenerationError(ValueError):
"""A request failed before producing output (surfaced via ``UserReply.error`` — e.g. a
chat template the tokenizer cannot render, or a prompt that exceeds the KV budget). Each
adapter turns this into its own wire-level error instead of hanging on a reply that never
arrives. ``code`` carries the stable class (``UserReply.error_code``) when the failure has
- one, so an adapter can emit it as OpenAI's error ``code`` and clients need not parse prose."""
+ one, so an adapter can emit it as OpenAI's error ``code`` and clients need not parse prose.
+
+ Subclasses ValueError so every adapter's invalid-request handler maps it to a 400."""
def __init__(self, message: str, code: str | None = None):
super().__init__(message)
@@ -231,15 +234,41 @@ def _render_message(message: dict[str, Any]) -> dict[str, Any]:
return m
-def _flatten_text_parts(parts: list[Any]) -> str:
- texts: list[str] = []
+def _flatten_text_parts(parts: list[Any]) -> str | list[dict[str, Any]]:
+ """Plain string for text-only parts; a template-ready part list with {"type": "image"} entries (source kept under freetoken_ref) when images are present."""
+ out: list[dict[str, Any]] = []
+ has_image = False
for part in parts:
ptype = part.get("type") if isinstance(part, dict) else None
if ptype == "text":
- texts.append((part.get("text") if isinstance(part, dict) else None) or "")
+ out.append({"type": "text", "text": part.get("text") or ""})
+ elif ptype in ("image_url", "input_image"):
+ url = part.get("image_url")
+ if isinstance(url, dict):
+ url = url.get("url")
+ url = url or part.get("url")
+ if not url:
+ raise ValueError("image content part carries no url")
+ out.append({"type": "image", "freetoken_ref": {"kind": "url", "data": url}})
+ has_image = True
+ elif ptype == "image" and isinstance(part.get("freetoken_ref"), dict):
+ out.append(part)
+ has_image = True
else:
- raise ValueError(f"Unsupported content part type for text-only server: {ptype}")
- return "".join(texts)
+ raise ValueError(f"Unsupported content part type: {ptype}")
+ if not has_image:
+ return "".join(p["text"] for p in out)
+ return out
+
+
+async def _resolve_images(refs: list[dict[str, Any]], state: Any) -> list[bytes]:
+ """Gate and fetch a request's images; every failure is a GenerationError."""
+ if reason := image_reject_reason(state.config):
+ raise GenerationError(reason)
+ try:
+ return await fetch_image_bytes(refs, state.config)
+ except ValueError as exc:
+ raise GenerationError(str(exc)) from exc
def split_tool_lists(
@@ -262,6 +291,8 @@ def split_tool_lists(
async def submit_generation(spec: GenSpec, state: Any) -> int:
"""Enqueue one generation from a GenSpec; return its uid. Every protocol adapter
calls this — it takes the neutral spec, not a wire request type."""
+ refs = collect_image_refs(spec.messages)
+ images = await _resolve_images(refs, state) if refs else None
uid = state.new_user()
await state.send_one(
TokenizeMsg(
@@ -270,6 +301,7 @@ async def submit_generation(spec: GenSpec, state: Any) -> int:
sampling_params=spec.sampling_params,
chat_template_kwargs=spec.chat_template_kwargs,
tools=spec.template_tools,
+ images=images,
)
)
return uid
@@ -293,19 +325,22 @@ async def count_prompt_tokens(
class the generation path maps to a 400. A tokenizer *initialization* failure (missing
template, load error) propagates as its original exception, a server fault. Load + tokenize
run in a worker thread so the event loop is never blocked."""
+ refs = collect_image_refs(messages)
+ images = await _resolve_images(refs, state) if refs else None
msg = TokenizeMsg(
uid=0,
text=messages,
sampling_params=SamplingParams(),
chat_template_kwargs=chat_template_kwargs,
tools=tools,
+ images=images,
)
manager = await asyncio.to_thread(state.frontend_tokenizer) # init failure -> server fault
try:
- input_ids = (await asyncio.to_thread(manager.tokenize, [msg]))[0]
+ (user_msg,) = await asyncio.to_thread(manager.tokenize, [msg])
except _TemplateError as exc:
raise GenerationError(str(exc)) from exc
- return int(input_ids.numel())
+ return int(user_msg.input_ids.numel())
async def prerender_error(spec: GenSpec, state: Any) -> GenerationError | None:
diff --git a/python/freetoken/server/launch.py b/python/freetoken/server/launch.py
index acef5517..bb0c256f 100644
--- a/python/freetoken/server/launch.py
+++ b/python/freetoken/server/launch.py
@@ -183,6 +183,7 @@ def launch_server(
"backend_addr": server_args.zmq_backend_addr,
"frontend_addr": server_args.zmq_frontend_addr,
"local_bs": 1,
+ "mm": server_args.mm,
"create": server_args.tokenizer_create_addr,
"tokenizer_id": num_tokenizers,
"ack_queue": ack_queue,
@@ -202,6 +203,7 @@ def launch_server(
"backend_addr": server_args.zmq_backend_addr,
"frontend_addr": server_args.zmq_frontend_addr,
"local_bs": 1,
+ "mm": server_args.mm,
"create": server_args.tokenizer_create_addr,
"tokenizer_id": i,
"ack_queue": ack_queue,
diff --git a/python/freetoken/server/openai_api.py b/python/freetoken/server/openai_api.py
index a76898ab..dc2f73a9 100644
--- a/python/freetoken/server/openai_api.py
+++ b/python/freetoken/server/openai_api.py
@@ -196,7 +196,10 @@ async def handle_chat_completion(
if err is not None:
return create_error_response(str(err), code=err.code)
- uid = await submit_generation(spec, state)
+ try:
+ uid = await submit_generation(spec, state)
+ except GenerationError as exc:
+ return create_error_response(str(exc), code=exc.code)
if req.stream:
chunks = stream_chat_completion_chunks(uid, req, state, spec)
diff --git a/python/freetoken/server/responses_api.py b/python/freetoken/server/responses_api.py
index 60a1442f..fc1cde9e 100644
--- a/python/freetoken/server/responses_api.py
+++ b/python/freetoken/server/responses_api.py
@@ -157,6 +157,8 @@ async def handle_responses(
reasoning_parser=getattr(state.config, "reasoning_parser", None),
)
uid = await submit_generation(spec, state)
+ except GenerationError as exc:
+ return _error_response(400, str(exc), exc.code)
except ValueError as exc:
return _error_response(400, str(exc))
@@ -260,7 +262,7 @@ def _convert_input_item(item: dict[str, Any]) -> list[dict[str, Any]]:
role = item.get("role", "user")
if role == "developer":
role = "system"
- return [{"role": role, "content": _input_text(item.get("content"))}]
+ return [{"role": role, "content": _input_content(item.get("content"))}]
if itype == "function_call":
return [
{
@@ -328,6 +330,29 @@ def _merge_assistant_run(messages: list[dict[str, Any]]) -> list[dict[str, Any]]
return merged
+def _input_content(content: Any) -> str | list[dict[str, Any]]:
+ """Like _input_text, but keeps input_image parts as template-ready image parts."""
+ if not isinstance(content, list):
+ return _input_text(content)
+ parts: list[dict[str, Any]] = []
+ has_image = False
+ for part in content:
+ if isinstance(part, dict) and part.get("type") == "input_image":
+ url = part.get("image_url") or part.get("url")
+ if isinstance(url, dict):
+ url = url.get("url")
+ if not url:
+ # an input_image without a url (e.g. a file_id) is not servable; fail rather than answer text-only
+ raise ValueError("input_image without image_url is not supported")
+ parts.append({"type": "image", "freetoken_ref": {"kind": "url", "data": url}})
+ has_image = True
+ continue
+ parts.append({"type": "text", "text": _input_text([part])})
+ if not has_image:
+ return "".join(p["text"] for p in parts)
+ return parts
+
+
def _input_text(content: Any) -> str:
if content is None:
return ""
diff --git a/python/freetoken/server/stats.py b/python/freetoken/server/stats.py
index 76c6c308..1857d3d7 100644
--- a/python/freetoken/server/stats.py
+++ b/python/freetoken/server/stats.py
@@ -102,7 +102,7 @@ class StatsTracker:
def derive_model_card(config: Any) -> dict:
- """attn enum + moe bool + ctx from the model config."""
+ """attn enum + moe bool + ctx from the model config; input_modalities is what the API accepts right now."""
mc = config.model_config
if getattr(mc, "has_linear_attention", False):
attn = "hybrid_linear"
@@ -115,6 +115,7 @@ def derive_model_card(config: Any) -> dict:
"ctx": config.max_seq_len,
"attn": attn,
"moe": bool(getattr(mc, "is_moe", False)),
+ "input_modalities": ["text", *sorted(config.served_modalities)],
}
diff --git a/python/freetoken/tokenizer/server.py b/python/freetoken/tokenizer/server.py
index 530e862d..d3365bfe 100644
--- a/python/freetoken/tokenizer/server.py
+++ b/python/freetoken/tokenizer/server.py
@@ -24,6 +24,7 @@ from freetoken.message import (
UserMsg,
UserReply,
)
+from freetoken.mm.config import MultimodalConfig
from freetoken.utils import (
ZmqPullQueue,
ZmqPushQueue,
@@ -84,18 +85,17 @@ def _tokenize_requests(
tokenize_manager: Any,
messages: List[TokenizeMsg],
logger: Any,
-) -> tuple[List[TokenizeMsg], List[torch.Tensor], List[UserReply]]:
+) -> tuple[List[UserMsg], List[UserReply]]:
"""Tokenize independently, returning backend work plus terminal frontend errors.
Successful tokenization deliberately emits no prompt-token reply: accounting starts
only when the scheduler later confirms first-prefill admission.
"""
- ok_msgs: List[TokenizeMsg] = []
- ok_tensors: List[torch.Tensor] = []
+ backend: List[UserMsg] = []
errors: List[UserReply] = []
for msg in messages:
try:
- tokens = tokenize_manager.tokenize([msg])[0]
+ user_msg = tokenize_manager.tokenize([msg])[0]
except Exception as exc: # noqa: BLE001 — isolate, never crash the worker
logger.warning(f"tokenization failed for request {msg.uid}: {exc!r}")
errors.append(
@@ -109,7 +109,7 @@ def _tokenize_requests(
continue
# A zero-token prompt would trip the scheduler's input_len > 0 invariant and
# crash the worker; reject it here as a terminal error instead.
- if tokens.numel() == 0:
+ if user_msg.input_ids.numel() == 0:
errors.append(
UserReply(
uid=msg.uid,
@@ -119,9 +119,8 @@ def _tokenize_requests(
)
)
continue
- ok_msgs.append(msg)
- ok_tensors.append(tokens)
- return ok_msgs, ok_tensors, errors
+ backend.append(user_msg)
+ return backend, errors
@torch.inference_mode()
@@ -136,6 +135,7 @@ def tokenize_worker(
tokenizer_id: int = -1,
model_source: str = "huggingface",
ack_queue: mp.Queue[str] | None = None,
+ mm: MultimodalConfig | None = None,
) -> None:
send_backend = ZmqPushQueue(backend_addr, create=False, encoder=BaseBackendMsg.encoder)
send_frontend = ZmqPushQueue(frontend_addr, create=False, encoder=BaseFrontendMsg.encoder)
@@ -144,10 +144,12 @@ def tokenize_worker(
tokenizer = load_tokenizer(tokenizer_path)
logger = init_logger(__name__, f"tokenizer_{tokenizer_id}")
+ from freetoken.mm.processor import get_mm_processor
+
from .detokenize import DetokenizeManager
from .tokenize import TokenizeManager
- tokenize_manager = TokenizeManager(tokenizer)
+ tokenize_manager = TokenizeManager(tokenizer, get_mm_processor(tokenizer_path, mm))
detokenize_manager = DetokenizeManager(
tokenizer, load_eos_token_ids(tokenizer_path, tokenizer)
)
@@ -245,18 +247,12 @@ def tokenize_worker(
# Tokenize per-message so a single un-renderable request (e.g. a chat template
# that rejects the message layout) becomes a terminal error reply for THAT uid
# instead of an uncaught exception that kills the worker and bricks the server.
- ok_msgs, ok_tensors, errors = _tokenize_requests(
- tokenize_manager, tokenize_msg, logger
- )
+ backend, errors = _tokenize_requests(tokenize_manager, tokenize_msg, logger)
if errors:
send_frontend.put(
errors[0] if len(errors) == 1 else BatchFrontendMsg(data=errors)
)
- if ok_msgs:
- backend = [
- UserMsg(uid=msg.uid, input_ids=t, sampling_params=msg.sampling_params)
- for msg, t in zip(ok_msgs, ok_tensors, strict=True)
- ]
+ if backend:
send_backend.put(backend[0] if len(backend) == 1 else BatchBackendMsg(data=backend))
if len(abort_msg) > 0:
batch_output = BatchBackendMsg(
diff --git a/python/freetoken/tokenizer/tokenize.py b/python/freetoken/tokenizer/tokenize.py
index 0636b342..758ffd4d 100644
--- a/python/freetoken/tokenizer/tokenize.py
+++ b/python/freetoken/tokenizer/tokenize.py
@@ -5,13 +5,16 @@ import json
import os
import threading
from types import ModuleType
-from typing import Any, List
+from typing import TYPE_CHECKING, Any, List
import torch
-from freetoken.message import TokenizeMsg
+from freetoken.message import TokenizeMsg, UserMsg
from freetoken.utils import init_logger
from transformers import PreTrainedTokenizerBase
+if TYPE_CHECKING:
+ from freetoken.mm.processor import MMProcessor
+
from .effort import (
EffortProfile,
ThinkingProfile,
@@ -47,16 +50,17 @@ _EFFORT_PROBE_MESSAGES = [{"role": "user", "content": "ping"}]
class TokenizeManager:
- def __init__(self, tokenizer: PreTrainedTokenizerBase) -> None:
+ def __init__(self, tokenizer: PreTrainedTokenizerBase, mm_processor: MMProcessor | None = None) -> None:
self.tokenizer = tokenizer
+ self.mm_processor = mm_processor # None: the model takes no images
self._dsv4_encoder = _load_dsv4_encoder_if_needed(tokenizer)
self._effort_profile: EffortProfile | None = None
self._thinking_profile: ThinkingProfile | None = None
self._effort_lock = threading.Lock()
self._logged_effort_maps: set[tuple[Any, str | None]] = set()
- def tokenize(self, msgs: List[TokenizeMsg]) -> List[torch.Tensor]:
- results: List[torch.Tensor] = []
+ def tokenize(self, msgs: List[TokenizeMsg]) -> List[UserMsg]:
+ results: List[UserMsg] = []
# TODO: batch tokenization
for msg in msgs:
prompt = self.render_prompt(msg)
@@ -71,7 +75,25 @@ class TokenizeManager:
prompt, return_tensors="pt", add_special_tokens=not templated
)
)
- results.append(input_ids.view(-1).to(torch.int32))
+ input_ids = input_ids.view(-1).to(torch.int32)
+ if msg.images:
+ if self.mm_processor is None:
+ raise ValueError("image input is not supported for this model")
+ mm = self.mm_processor.apply(input_ids, msg.images)
+ results.append(
+ UserMsg(
+ uid=msg.uid,
+ input_ids=mm.input_ids,
+ sampling_params=msg.sampling_params,
+ mm_items=mm.mm_items,
+ mrope_positions=mm.mrope_positions,
+ mrope_delta=mm.mrope_delta,
+ )
+ )
+ else:
+ results.append(
+ UserMsg(uid=msg.uid, input_ids=input_ids, sampling_params=msg.sampling_params)
+ )
return results
def render_prompt(self, msg: TokenizeMsg) -> str:
diff --git a/tests/README.md b/tests/README.md
index 83cd79f2..33e5756c 100644
--- a/tests/README.md
+++ b/tests/README.md
@@ -48,6 +48,8 @@ checkpoint is set:
| `FREETOKEN_AIME_MAX_TOKENS`| `e2e/test_aime.py` — per-sample token budget (default `16384`) |
| `FREETOKEN_AIME_SAMPLES` | `e2e/test_aime.py` — pass@N sample count when sampling (default `3`) |
| `FREETOKEN_AIME_MIN_FREE_GIB` | `e2e/test_aime.py` — required free GPU memory (default `70`) |
+| `FREETOKEN_QWEN36_MODEL` | `tokenizer/test_mm_tokenize.py` — local Qwen3.6 checkpoint (image processor + tokenizer) |
+| `FREETOKEN_QWEN3VL_MODEL` | `tokenizer/test_mm_tokenize.py` — local Qwen3-VL checkpoint (same expansion, DeepStack family) |
| `FREETOKEN_TEST_MOE_CACHE_SIZE` | `e2e/test_aime.py` — >0 switches to the offload MoE backend with this cache size |
| `FREETOKEN_TEST_MEM_RATIO` | `e2e/test_aime.py` — offload-mode memory_ratio (default `0.9`) |
| `FREETOKEN_REBUILD_TEST_MODEL` | `e2e/test_cache_rebuild.py` — a SMALL local model dir; boots a real server (falls back to `FREETOKEN_TEST_MODEL`) |
diff --git a/tests/engine/test_mm_encoder.py b/tests/engine/test_mm_encoder.py
new file mode 100644
index 00000000..1c12de58
--- /dev/null
+++ b/tests/engine/test_mm_encoder.py
@@ -0,0 +1,96 @@
+"""Engine._run_mm_encoder against a fake model: chunked gathers, a shared image, precomputed embeddings, orphan jobs."""
+
+from __future__ import annotations
+
+from types import SimpleNamespace
+
+import pytest
+import torch
+
+from freetoken.engine.engine import Engine
+from freetoken.message.backend import MMItem
+from freetoken.mm.encoder_cache import EncoderCache
+
+H = 8
+
+
+class _FakeVision:
+ def __init__(self):
+ self.calls = 0
+
+ def encode(self, item: MMItem) -> torch.Tensor:
+ self.calls += 1
+ return torch.full((item.num_tokens, H), float(item.hash))
+
+
+def _engine(cache: EncoderCache, model=None) -> SimpleNamespace:
+ return SimpleNamespace(
+ encoder_cache=cache, device=torch.device("cpu"), dtype=torch.float32,
+ model=model or _FakeVision(),
+ )
+
+
+def _item(h: int, n_tokens: int, precomputed: torch.Tensor | None = None) -> MMItem:
+ feature = None if precomputed is not None else torch.zeros(1)
+ return MMItem(
+ modality="image", hash=h, pad_value=0, offsets=[[0, n_tokens]],
+ feature=feature, precomputed_embeddings=precomputed,
+ )
+
+
+def _batch(jobs, plan) -> SimpleNamespace:
+ return SimpleNamespace(mm_encoder_jobs=jobs, mm_gather_plan=plan, mm_embeds=None, mm_rows=None)
+
+
+def test_entry_lives_until_its_consumer_gathers_the_last_row():
+ cache = EncoderCache(storage="cpu")
+ eng = _engine(cache)
+ item = _item(h=3, n_tokens=4)
+ cache.register(3, 7, 4) # admission claims the whole image
+ # chunk 1 consumes rows [0, 2) of a 4-token image
+ Engine._run_mm_encoder(eng, _batch([item], [(7, 3, 0, 2, 4, 0)]))
+ assert cache._entries[3].remaining == {7: 2}
+ assert item.feature is None
+ # chunk 2 finishes the image; the entry dies with its last claim
+ Engine._run_mm_encoder(eng, _batch([], [(7, 3, 2, 4, 4, 0)]))
+ assert not cache.has(3)
+ assert eng.model.calls == 1
+
+
+def test_shared_image_is_encoded_once_and_sliced_per_request():
+ cache = EncoderCache(storage="cpu")
+ eng = _engine(cache)
+ a, b = _item(h=5, n_tokens=4), _item(h=5, n_tokens=4)
+ cache.register(5, 1, 4)
+ cache.register(5, 2, 4)
+ batch = _batch([a, b], [(1, 5, 0, 4, 4, 0), (2, 5, 0, 2, 4, 4)])
+ Engine._run_mm_encoder(eng, batch)
+ assert eng.model.calls == 1
+ assert batch.mm_embeds.shape == (6, H)
+ assert torch.equal(batch.mm_embeds, torch.full((6, H), 5.0))
+ # request 1 consumed its image; request 2 still has rows to gather in its next chunk
+ assert cache._entries[5].remaining == {2: 2}
+
+
+def test_precomputed_embeddings_bypass_the_encoder():
+ cache = EncoderCache(storage="cpu")
+
+ class _NoVision:
+ def encode(self, item):
+ raise AssertionError("encoder must not run for precomputed embeddings")
+
+ eng = _engine(cache, model=_NoVision())
+ emb = torch.arange(2 * H, dtype=torch.float32).view(2, H)
+ item = _item(h=4, n_tokens=2, precomputed=emb)
+ cache.register(4, 1, 2)
+ batch = _batch([item], [(1, 4, 0, 2, 2, 0)])
+ Engine._run_mm_encoder(eng, batch)
+ assert torch.equal(batch.mm_embeds, emb)
+ assert item.precomputed_embeddings is None
+ assert not cache.has(4)
+
+
+def test_job_without_gather_row_fails_loudly():
+ eng = _engine(EncoderCache(storage="cpu"))
+ with pytest.raises(AssertionError, match="encoder jobs without gather rows"):
+ Engine._run_mm_encoder(eng, _batch([_item(h=1, n_tokens=2)], []))
diff --git a/tests/engine/test_mm_repeated_image.py b/tests/engine/test_mm_repeated_image.py
new file mode 100644
index 00000000..a87b0424
--- /dev/null
+++ b/tests/engine/test_mm_repeated_image.py
@@ -0,0 +1,48 @@
+"""A request that carries the same image twice, chunked between the two occurrences (a real checkpoint)."""
+
+from __future__ import annotations
+
+import os
+import subprocess
+import sys
+
+import pytest
+
+CHECKPOINT = os.environ.get("FREETOKEN_QWEN3VL_MODEL", "")
+
+pytestmark = [
+ pytest.mark.slow,
+ pytest.mark.skipif(not os.path.exists(os.path.join(CHECKPOINT, "config.json")), reason="FREETOKEN_QWEN3VL_MODEL not set"),
+]
+
+# the engine wants a fresh CUDA context, so each run gets its own process
+_SCRIPT = """
+import io, sys, torch
+from PIL import Image
+from freetoken.core import SamplingParams
+from freetoken.llm import LLM
+from freetoken.mm.config import MultimodalConfig
+
+checkpoint, storage = sys.argv[1], sys.argv[2]
+llm = LLM(model_path=checkpoint, dtype=torch.bfloat16, attention_backend="auto", max_running_req=1, cuda_graph_max_bs=1,
+ max_extend_tokens=128, max_seq_len_override=4096, mm=MultimodalConfig(embed_cache_device=storage))
+messages = [{"role": "user", "content": [{"type": "image"}, {"type": "image"},
+ {"type": "text", "text": "What color are these two images? Answer in one word."}]}]
+prompt = llm.tokenizer.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
+ids = llm.tokenizer.encode(prompt, add_special_tokens=False)
+buf = io.BytesIO(); Image.new("RGB", (256, 256), (255, 0, 0)).save(buf, format="PNG"); red = buf.getvalue()
+text = llm.generate([ids], SamplingParams(max_tokens=8, temperature=0.0), images=[[red, red]])[0]["text"]
+print("RESULT", repr(text), llm.engine.encoder_cache.stats())
+"""
+
+
[email protected]("storage", ["cpu", "cuda"])
+def test_repeated_image_survives_the_chunk_boundary(storage):
+ run = subprocess.run(
+ [sys.executable, "-c", _SCRIPT, CHECKPOINT, storage],
+ capture_output=True, text=True, timeout=900, env={**os.environ, "PYTHONPATH": os.environ.get("PYTHONPATH", "")},
+ )
+ assert run.returncode == 0, run.stderr[-2000:]
+ result = [line for line in run.stdout.splitlines() if line.startswith("RESULT")][-1]
+ assert "red" in result.lower(), result
+ assert result.endswith("(0, 0)"), result # every claim gathered, nothing left behind
diff --git a/tests/kernels/test_mrope.py b/tests/kernels/test_mrope.py
new file mode 100644
index 00000000..f8402f87
--- /dev/null
+++ b/tests/kernels/test_mrope.py
@@ -0,0 +1,114 @@
+"""MRotaryEmbedding correctness: text degeneracy, section layouts, kernel-vs-fallback."""
+
+from __future__ import annotations
+
+import pytest
+import torch
+
+cuda = pytest.mark.skipif(not torch.cuda.is_available(), reason="needs CUDA")
+
+HEAD = 256
+ROT = 64
+SECTION = (11, 11, 10)
+
+# independent expectations for SECTION under each layout (half = 32 slots)
+REFERENCE_TABLES = {
+ "contiguous": [0] * 11 + [1] * 11 + [2] * 10,
+ "interleaved": [1 if i % 3 == 1 and i < 33 else 2 if i % 3 == 2 and i < 30 else 0 for i in range(32)],
+ "interleaved_glm": [0, 1, 2] * 10 + [0, 1],
+}
+
+
+def test_section_tables():
+ from freetoken.layers.rotary import build_section_table
+
+ for layout, expected in REFERENCE_TABLES.items():
+ assert build_section_table(SECTION, layout).tolist() == expected, layout
+ # GLM-V's [8,12,12]: T runs out first and H/W fill the tail
+ assert build_section_table((8, 12, 12), "interleaved_glm").tolist() == [0, 1, 2] * 8 + [1, 1, 2, 1, 1, 2, 2, 2]
+ with pytest.raises(ValueError, match="not representable"):
+ build_section_table((8, 12, 12), "interleaved")
+ with pytest.raises(ValueError, match="unknown mrope layout"):
+ build_section_table(SECTION, "diagonal")
+
+
+def _make(mrope: bool, layout: str = "interleaved"):
+ from freetoken.layers.rotary import get_rope
+
+ # the engine builds rope layers inside a cuda device context
+ with torch.device("cuda"):
+ return get_rope(
+ head_dim=HEAD, rotary_dim=ROT, max_position=4096, base=1e7,
+ mrope_section=SECTION if mrope else None, mrope_layout=layout,
+ )
+
+
+def _qk(n, heads=4, kv=2, seed=0):
+ g = torch.Generator(device="cuda").manual_seed(seed)
+ q = torch.randn(n, heads * HEAD, generator=g, dtype=torch.bfloat16, device="cuda")
+ k = torch.randn(n, kv * HEAD, generator=g, dtype=torch.bfloat16, device="cuda")
+ return q, k
+
+
+@cuda
+def test_text_positions_degenerate_to_1d_rope():
+ n = 64
+ pos1 = torch.arange(n, dtype=torch.int32, device="cuda")
+ pos3 = pos1.unsqueeze(0).expand(3, -1).contiguous()
+ q1, k1 = _qk(n)
+ q3, k3 = q1.clone(), k1.clone()
+ _make(False).forward(pos1, q1, k1)
+ _make(True).forward(pos3, q3, k3)
+ # 1-D path may use flashinfer while mrope uses the triton kernel: allclose, not bitwise
+ assert torch.allclose(q1.float(), q3.float(), atol=2e-2, rtol=2e-2)
+ assert torch.allclose(k1.float(), k3.float(), atol=2e-2, rtol=2e-2)
+
+
+def _torch_reference(pos3, q, k, table):
+ """Per-axis freqs, per-slot axis selection, NeoX rotation, fp32 math."""
+ half = ROT // 2
+ inv = 1.0 / (1e7 ** (torch.arange(0, ROT, 2, dtype=torch.float32, device="cuda") / ROT))
+ sec = torch.tensor(table, dtype=torch.long, device="cuda")
+ pos = pos3[sec, :].transpose(0, 1).float() # [n, half]
+ freqs = pos * inv.unsqueeze(0) # [n, half]
+ cos, sin = freqs.cos().unsqueeze(1), freqs.sin().unsqueeze(1)
+ for t in (q, k):
+ v = t.view(t.shape[0], -1, HEAD)
+ lo, hi = v[..., :half].float(), v[..., half:ROT].float()
+ v[..., :half] = (lo * cos - hi * sin).to(v.dtype)
+ v[..., half:ROT] = (hi * cos + lo * sin).to(v.dtype)
+
+
+@cuda
[email protected]("layout", sorted(REFERENCE_TABLES))
+def test_matches_reference_semantics(layout):
+ n = 97
+ g = torch.Generator(device="cuda").manual_seed(7)
+ pos3 = torch.randint(0, 4000, (3, n), generator=g, dtype=torch.int32, device="cuda")
+ q, k = _qk(n, seed=7)
+ q_ref, k_ref = q.clone(), k.clone()
+ _make(True, layout).forward(pos3, q, k)
+ _torch_reference(pos3, q_ref, k_ref, REFERENCE_TABLES[layout])
+ assert torch.allclose(q.float(), q_ref.float(), atol=2e-2, rtol=2e-2)
+ assert torch.allclose(k.float(), k_ref.float(), atol=2e-2, rtol=2e-2)
+
+
+@cuda
+def test_kernel_matches_torch_fallback():
+ from freetoken.kernel.triton.rope import (
+ apply_mrope_torch_fallback,
+ apply_mrope_with_cos_sin_cache_inplace,
+ )
+
+ rope = _make(True)
+ n = 33
+ g = torch.Generator(device="cuda").manual_seed(3)
+ pos3 = torch.randint(0, 4000, (3, n), generator=g, dtype=torch.int32, device="cuda")
+ q, k = _qk(n, seed=3)
+ q2, k2 = q.clone(), k.clone()
+ cache = rope._cos_sin_cache
+ sec = rope._section_table.cuda()
+ apply_mrope_with_cos_sin_cache_inplace(pos3, q, k, HEAD, cache, sec)
+ apply_mrope_torch_fallback(pos3, q2, k2, HEAD, cache, sec)
+ assert torch.allclose(q.float(), q2.float(), atol=1e-2, rtol=1e-2)
+ assert torch.allclose(k.float(), k2.float(), atol=1e-2, rtol=1e-2)
diff --git a/tests/kvcache/test_qsa_pool.py b/tests/kvcache/test_qsa_pool.py
index 9aec2bb6..dea722f6 100644
--- a/tests/kvcache/test_qsa_pool.py
+++ b/tests/kvcache/test_qsa_pool.py
@@ -71,7 +71,7 @@ def _spec(*, index_ratio=4, attn_type=AttnType.QSA, num_kv_heads=2, head_dim=64,
def _config(spec, *, page_size=64, max_running_req=3):
- mc = SimpleNamespace(num_layers=8, has_swa_attention=False, has_linear_attention=True)
+ mc = SimpleNamespace(num_layers=8, has_swa_attention=False, has_linear_attention=True, model_is_mrope=False)
mc.kv_cache_group_specs = lambda: (spec,)
return SimpleNamespace(
model_config=mc,
@@ -201,6 +201,7 @@ def test_resolve_pool_class_and_factory():
spec = _spec()
mc = SimpleNamespace(
+ model_is_mrope=False,
num_layers=8, has_swa_attention=False, has_linear_attention=True,
num_kv_heads=2, head_dim=64, dsv4_args=None,
)
diff --git a/tests/mm/test_encoder_cache.py b/tests/mm/test_encoder_cache.py
new file mode 100644
index 00000000..ee028d4c
--- /dev/null
+++ b/tests/mm/test_encoder_cache.py
@@ -0,0 +1,46 @@
+"""EncoderCache ownership: rows claimed at admission, freed when the last claim is gathered."""
+
+from __future__ import annotations
+
+import torch
+
+from freetoken.mm.encoder_cache import EncoderCache
+
+CPU = torch.device("cpu")
+
+
+def test_entry_lives_until_every_claim_is_gathered():
+ cache = EncoderCache(storage="cpu")
+ emb = torch.arange(12, dtype=torch.float32).reshape(6, 2)
+ cache.register(7, 1, 6)
+ cache.register(7, 2, 6)
+ assert not cache.has(7) # claimed, not encoded yet
+ cache.put(7, emb)
+ assert cache.has(7)
+ assert torch.equal(cache.get_slice(7, 2, 4, CPU), emb[2:4])
+ cache.consume(7, 1, 4)
+ cache.consume(7, 1, 2)
+ assert cache.has(7) # request 2 still has rows to gather
+ cache.consume(7, 2, 6)
+ assert not cache.has(7) and cache.stats() == (0, 0)
+
+
+def test_repeated_image_in_one_request_adds_up():
+ cache = EncoderCache(storage="cpu")
+ cache.register(7, 1, 4)
+ cache.register(7, 1, 4)
+ cache.put(7, torch.ones(4, 2))
+ cache.consume(7, 1, 4) # the first occurrence is done
+ assert cache.has(7)
+ cache.consume(7, 1, 4)
+ assert not cache.has(7)
+
+
+def test_release_drops_a_claim_even_before_the_tensor_arrives():
+ cache = EncoderCache(storage="cpu")
+ cache.register(5, 1, 3)
+ cache.register(5, 2, 3)
+ cache.release(1, [5, 99]) # unknown hash: no-op
+ assert 5 in cache._entries
+ cache.release(2, [5])
+ assert cache._entries == {} and cache.stats() == (0, 0)
diff --git a/tests/mm/test_processor.py b/tests/mm/test_processor.py
new file mode 100644
index 00000000..87559d29
--- /dev/null
+++ b/tests/mm/test_processor.py
@@ -0,0 +1,186 @@
+"""MMProcessor: placeholder expansion on synthetic families, the Qwen VL processor and its positions, the registry (no checkpoint)."""
+
+from __future__ import annotations
+
+import io
+from types import SimpleNamespace
+
+import pytest
+import torch
+
+from freetoken.message import MMItem
+from freetoken.mm import MM_PAD_SHIFT_VALUE
+from freetoken.mm.config import MultimodalConfig
+from freetoken.mm.processor import MMProcessor, PromptReplacement, image_positions
+from freetoken.mm.processors.qwen_vl import QwenVLMMProcessor
+
+PLACEHOLDER = 7
+BOI, SOFT, EOI = 90, 91, 92
+ARCH = "Qwen3_5ForConditionalGeneration"
+
+
+def _png():
+ from PIL import Image
+
+ buf = io.BytesIO()
+ Image.new("RGB", (8, 8)).save(buf, format="PNG")
+ return buf.getvalue()
+
+
+class _Family(MMProcessor):
+ """Fixed-size images wrapped in begin/end tokens; only the soft tokens take embeddings."""
+
+ placeholder = [PLACEHOLDER]
+
+ def __init__(self, n_soft):
+ super().__init__("/fake", MultimodalConfig())
+ self.n_soft = n_soft
+
+ def process(self, images):
+ return [
+ MMItem(modality="image", hash=100 + i, pad_value=MM_PAD_SHIFT_VALUE + 100 + i, offsets=[], feature=torch.zeros(1))
+ for i, _ in enumerate(images)
+ ]
+
+ def prompt_replacement(self, item):
+ return PromptReplacement.select_token_id([BOI] + [SOFT] * self.n_soft + [EOI], SOFT)
+
+ def dummy_items(self, dtype, device):
+ raise NotImplementedError
+
+
+class _Tiled(_Family):
+ """Rows of soft tokens separated by a break token, like a tiled image."""
+
+ def prompt_replacement(self, item):
+ full = ([SOFT] * 2 + [BOI]) * 3
+ return PromptReplacement.select_token_id(full, SOFT)
+
+
+def test_replacement_keeps_wrapper_tokens_and_pads_only_embedding_slots():
+ ids = torch.tensor([1, 2, PLACEHOLDER, 3, PLACEHOLDER, 4], dtype=torch.int32)
+ r = _Family(n_soft=3).apply(ids, [_png(), _png()])
+ a, b = r.mm_items
+ assert r.input_ids.tolist() == [1, 2, BOI, a.pad_value, a.pad_value, a.pad_value, EOI, 3, BOI, b.pad_value, b.pad_value, b.pad_value, EOI, 4]
+ assert a.offsets == [[3, 6]] and b.offsets == [[9, 12]]
+ assert a.num_tokens == 3 and r.mrope_positions is None and r.mrope_delta == 0
+
+
+def test_replacement_with_several_embedding_runs_yields_several_spans():
+ ids = torch.tensor([PLACEHOLDER, 5], dtype=torch.int32)
+ r = _Tiled(n_soft=0).apply(ids, [_png()])
+ (item,) = r.mm_items
+ p = item.pad_value
+ assert r.input_ids.tolist() == [p, p, BOI, p, p, BOI, p, p, BOI, 5]
+ assert item.offsets == [[0, 2], [3, 5], [6, 8]] and item.num_tokens == 6
+
+
+def test_placeholder_count_is_checked_before_any_image_is_decoded():
+ ids = torch.tensor([PLACEHOLDER, PLACEHOLDER], dtype=torch.int32)
+ with pytest.raises(ValueError, match="2 image placeholders but the request carries 1"):
+ _Family(n_soft=1).apply(ids, [b"not an image"])
+
+
+def test_image_positions_freeze_t_spread_hw_and_advance_by_the_longer_side():
+ # [text x2][img 2x3 = 6][text x1][img 3x1 = 3][text x2] -> 14 tokens
+ pos, delta = image_positions(14, [(2, 6, 2, 3), (9, 3, 3, 1)])
+ assert pos[:, :2].tolist() == [[0, 1]] * 3
+ assert pos[0, 2:8].tolist() == [2] * 6
+ assert pos[1, 2:8].tolist() == [2, 2, 2, 3, 3, 3]
+ assert pos[2, 2:8].tolist() == [2, 3, 4, 2, 3, 4]
+ assert pos[:, 8].tolist() == [5, 5, 5] # 2 + max(2, 3)
+ assert pos[0, 9:12].tolist() == [6] * 3
+ assert pos[1, 9:12].tolist() == [6, 7, 8]
+ assert pos[2, 9:12].tolist() == [6, 6, 6]
+ assert pos[:, 12:].tolist() == [[9, 10]] * 3 # 6 + max(3, 1)
+ assert delta == 10 + 1 - 14
+
+
+def _hf_config(vision=True, mrope=True, arch=ARCH):
+ rope = {"rope_theta": 1e7}
+ if mrope:
+ rope.update(mrope_section=[24, 20, 20], mrope_interleaved=True)
+ return SimpleNamespace(
+ architectures=[arch],
+ image_token_id=151655,
+ vision_config=(
+ SimpleNamespace(spatial_merge_size=2, patch_size=16, temporal_patch_size=2, in_channels=3)
+ if vision
+ else None
+ ),
+ text_config=SimpleNamespace(rope_parameters=rope, vocab_size=248320),
+ )
+
+
+def _grid_item(h, w, offsets):
+ return MMItem(
+ modality="image", hash=1, pad_value=1, offsets=offsets,
+ feature=torch.zeros(1), model_specific_data={"grid_thw": [1, h, w]},
+ )
+
+
+def test_qwen_processor_reads_config():
+ proc = QwenVLMMProcessor(_hf_config(), "/nonexistent", MultimodalConfig())
+ assert proc.image_token_id == 151655 and proc.is_mrope and proc.merge == 2
+ assert proc.image_grid(_grid_item(8, 6, [[0, 12]])) == (4, 3)
+ repl = proc.prompt_replacement(_grid_item(8, 6, []))
+ assert proc.placeholder == [151655] and repl.full == [151655] * 12 and repl.embed_spans() == [[0, 12]]
+ (dummy,) = proc.dummy_items(torch.bfloat16, torch.device("cpu"))
+ dummy.validate()
+ assert dummy.feature.shape == (4, 3 * 2 * 16 * 16) and dummy.num_tokens == 1
+ assert dummy.grid_thw == [1, 2, 2]
+
+
+def test_qwen_token_budget_and_kwargs_reach_the_image_processor():
+ from freetoken.mm.config import MultimodalConfig
+
+ calls = []
+
+ class _FakeImageProcessor:
+ size = {"shortest_edge": 65536, "longest_edge": 16777216}
+
+ def __call__(self, images, **kwargs):
+ calls.append(kwargs)
+ return {"pixel_values": torch.zeros(4, 3 * 2 * 16 * 16), "image_grid_thw": torch.tensor([[1, 2, 2]])}
+
+ mm = MultimodalConfig(image_min_tokens=100, image_max_tokens=1000, processor_kwargs={"do_convert_rgb": False})
+ proc = QwenVLMMProcessor(_hf_config(), "/nonexistent", mm)
+ proc._image_processor = lambda: _FakeImageProcessor()
+ (item,) = proc.process([object()])
+ assert item.grid_thw == [1, 2, 2]
+ # one token is a 32x32 patch of the resized image; the budget travels as pixel areas in size, the extra kwarg rides along
+ assert calls == [{"return_tensors": "pt", "size": {"shortest_edge": 100 * 1024, "longest_edge": 1000 * 1024}, "do_convert_rgb": False}]
+ proc = QwenVLMMProcessor(_hf_config(), "/nonexistent", MultimodalConfig())
+ proc._image_processor = lambda: _FakeImageProcessor()
+ proc.process([object()])
+ assert calls[-1] == {"return_tensors": "pt"} # no budget, no size override
+
+
+def test_qwen_processor_positions_follow_the_grid():
+ proc = QwenVLMMProcessor(_hf_config(), "/nonexistent", MultimodalConfig())
+ # 3 text tokens, one 8x6-patch image (4x3 = 12 llm tokens), 2 text tokens
+ pos, delta = proc.positions(17, [_grid_item(8, 6, [[3, 15]])])
+ assert pos.shape == (3, 17)
+ assert pos[:, :3].tolist() == [[0, 1, 2]] * 3
+ assert pos[0, 3:15].tolist() == [3] * 12
+ assert pos[1, 3:15].tolist() == [3 + i // 3 for i in range(12)]
+ assert pos[2, 3:15].tolist() == [3 + i % 3 for i in range(12)]
+ assert pos[:, 15:].tolist() == [[7, 8]] * 3 # resumes at 3 + max(4, 3)
+ assert delta == 8 + 1 - 17
+
+
+def test_registry_resolves_by_architecture(monkeypatch):
+ import freetoken.utils
+ from freetoken.mm.processor import get_mm_processor
+
+ configs = {
+ "/qwen": _hf_config(),
+ "/text-only": _hf_config(vision=False),
+ "/unknown-vlm": _hf_config(arch="SomeOtherForConditionalGeneration"),
+ }
+ monkeypatch.setattr(freetoken.utils, "cached_load_hf_config", lambda p: configs[p])
+ assert isinstance(get_mm_processor("/qwen"), QwenVLMMProcessor)
+ assert get_mm_processor("/qwen", MultimodalConfig(disabled_encoders=frozenset({"vision"}))) is None # --mm-disable vision
+ assert get_mm_processor("/text-only") is None
+ assert get_mm_processor("/unknown-vlm") is None
+ assert get_mm_processor("/missing") is None # a config that fails to load means no vision
diff --git a/tests/models/qwen4_exp/common.py b/tests/models/qwen4_exp/common.py
index aa6f2399..0a52fb0e 100644
--- a/tests/models/qwen4_exp/common.py
+++ b/tests/models/qwen4_exp/common.py
@@ -231,6 +231,8 @@ class Fixture:
is_prefill=phase == "prefill",
is_decode=phase == "decode",
positions=positions,
+ get_attn_positions=lambda: positions,
+ mm_embeds=None,
out_loc=out_loc,
attn_metadata=None,
active_table_idx=torch.tensor(
@@ -353,12 +355,14 @@ def meta_state_dict(model_path: str) -> dict[str, torch.Tensor]:
from freetoken.engine.config import EngineConfig
from freetoken.engine.engine import _decode_target
from freetoken.layers import rotary
+ from freetoken.mm.config import ENCODER_KINDS, MultimodalConfig
from freetoken.models import create_model
from freetoken.utils.torch_utils import torch_dtype
if try_get_tp_info() is None:
set_tp_info(rank=0, size=1)
- config = EngineConfig(model_path=model_path, tp_info=try_get_tp_info(), dtype=torch.bfloat16, moe_strategy="offload")
+ config = EngineConfig(model_path=model_path, tp_info=try_get_tp_info(), dtype=torch.bfloat16, moe_strategy="offload",
+ mm=MultimodalConfig(disabled_encoders=frozenset(ENCODER_KINDS)))
object.__setattr__(config.model_config, "moe_strategy", "offload")
object.__setattr__(config.model_config, "decode_target", _decode_target(config))
saved = rotary._ROPE_DEVICE
diff --git a/tests/models/qwen4_exp/test_config.py b/tests/models/qwen4_exp/test_config.py
index b81e02f4..be532eec 100644
--- a/tests/models/qwen4_exp/test_config.py
+++ b/tests/models/qwen4_exp/test_config.py
@@ -5,7 +5,10 @@ from types import SimpleNamespace
import pytest
from freetoken.attention import AttnType
-from freetoken.models.config import FullAttentionGroupConfig, LinearGatedDeltaGroupConfig
+from freetoken.models.config import (
+ FullAttentionGroupConfig,
+ LinearGatedDeltaGroupConfig,
+)
from freetoken.models.qwen4_exp.config import parse_config
from .common import LOVEDHEART_NVFP4_FP8, NVIDIA_NVFP4, QWEN_FP8, RADIXARK_NVFP4
@@ -157,6 +160,28 @@ def test_eos_token_id_list_uses_the_first_entry():
assert parse_config(hf).qwen4_args.ngram_boundary_token_id == base
+def _vision_config():
+ return SimpleNamespace(
+ hidden_size=1152, depth=27, num_heads=16, intermediate_size=4304, patch_size=16, temporal_patch_size=2,
+ spatial_merge_size=2, num_position_embeddings=2304, out_hidden_size=2560, in_channels=3,
+ deepstack_visual_indexes=[],
+ )
+
+
+def test_vision_turns_on_mrope_and_the_tower():
+ hf = _hf_config()
+ hf.vision_config = _vision_config()
+ config = parse_config(hf)
+ assert config.is_multimodal and config.model_is_mrope
+ assert config.rotary_config.mrope_section == [11, 11, 10]
+ assert config.rotary_config.mrope_layout == "interleaved" and config.rotary_config.rotary_dim == 64
+ assert config.vision_config.out_hidden_size == 2560 and config.vision_config.deepstack_visual_indexes == ()
+ assert config.qwen4_args.image_token_id == 248056
+
+
+def test_text_only_keeps_the_1d_rope():
+ config = parse_config(_hf_config())
+ assert not config.is_multimodal and not config.model_is_mrope and config.rotary_config.mrope_section is None
# the merged-projection prefixes the model asks the QuantConfig about (attention.py / gdn.py)
DENSE_PREFIXES = (
"model.layers.3.self_attn.qkv_proj", "model.layers.3.self_attn.o_proj",
diff --git a/tests/models/qwen4_exp/test_qsa_backend.py b/tests/models/qwen4_exp/test_qsa_backend.py
index 1d3b944c..23661079 100644
--- a/tests/models/qwen4_exp/test_qsa_backend.py
+++ b/tests/models/qwen4_exp/test_qsa_backend.py
@@ -174,6 +174,7 @@ def test_decode_graph_replay_matches_eager():
capture_batch = SimpleNamespace(
padded_reqs=[dummy] * bs, reqs=[dummy] * bs, phase="decode", size=bs, padded_size=bs,
is_prefill=False, is_decode=True, positions=static["positions"],
+ get_attn_positions=lambda: static["positions"],
out_loc=static["out_loc"], attn_metadata=None, active_table_idx=None,
)
fixture.backend.prepare_for_capture(capture_batch)
diff --git a/tests/models/qwen4_exp/test_skeleton.py b/tests/models/qwen4_exp/test_skeleton.py
index fa75d6ee..ac598e65 100644
--- a/tests/models/qwen4_exp/test_skeleton.py
+++ b/tests/models/qwen4_exp/test_skeleton.py
@@ -346,7 +346,9 @@ def test_qsa_layer_matches_hf_dense():
x = (torch.randn(seq_len, config.hidden_size, device=device, dtype=dtype) * 0.5)
positions = torch.arange(seq_len, device=device, dtype=torch.int64)
req = SimpleNamespace(extend_len=seq_len, cached_len=0, table_idx=1)
- batch = SimpleNamespace(padded_reqs=[req], reqs=[req], positions=positions)
+ batch = SimpleNamespace(
+ padded_reqs=[req], reqs=[req], positions=positions, get_attn_positions=lambda: positions
+ )
backend = TorchDenseQSAReference(config, num_slots=4, max_len=64, device=device, dtype=dtype)
_fresh_ctx(attn_backend=backend)
@@ -473,10 +475,11 @@ def test_decoder_stack_prefill_and_decode(monkeypatch):
last = torch.tensor(
[sum(len(p) for p in prompts[: i + 1]) - 1 for i in range(len(prompts))], device=device
)
+ positions = torch.cat([torch.arange(len(p)) for p in prompts]).to(device)
batch = SimpleNamespace(
padded_reqs=reqs, reqs=reqs, size=len(reqs), is_prefill=True, is_decode=False,
input_ids=torch.tensor(flat, dtype=torch.int64, device=device),
- positions=torch.cat([torch.arange(len(p)) for p in prompts]).to(device),
+ positions=positions, get_attn_positions=lambda: positions, mm_embeds=None,
attn_metadata=SimpleNamespace(get_last_indices=lambda bs: last[:bs]),
)
with ctx.forward_batch(batch):
@@ -488,10 +491,11 @@ def test_decoder_stack_prefill_and_decode(monkeypatch):
r.cached_len = len(p)
r.extend_len = 1
r.input_ids = torch.cat([r.input_ids, torch.tensor([14], dtype=torch.int64)])
+ decode_positions = torch.tensor([len(p) for p in prompts], dtype=torch.int64, device=device)
decode = SimpleNamespace(
padded_reqs=reqs, reqs=reqs, size=len(reqs), is_prefill=False, is_decode=True,
input_ids=torch.tensor([14] * len(reqs), dtype=torch.int64, device=device),
- positions=torch.tensor([len(p) for p in prompts], dtype=torch.int64, device=device),
+ positions=decode_positions, get_attn_positions=lambda: decode_positions, mm_embeds=None,
attn_metadata=None,
)
with ctx.forward_batch(decode):
diff --git a/tests/models/qwen4_exp/test_weight.py b/tests/models/qwen4_exp/test_weight.py
index 1ddbf425..1a9a100e 100644
--- a/tests/models/qwen4_exp/test_weight.py
+++ b/tests/models/qwen4_exp/test_weight.py
@@ -193,12 +193,12 @@ def _write_checkpoint(folder, raw: dict[str, torch.Tensor], quantization_config)
return str(folder), {**raw, **table}
-def _load(folder: str) -> dict[str, torch.Tensor]:
+def _load(folder: str, *, vision: bool = True) -> dict[str, torch.Tensor]:
install_quant_config(folder)
return {
name: tensor.clone()
for name, tensor in iter_weights(
- folder, torch.device("cpu"), include_moe_experts=True, include_non_moe=True
+ folder, torch.device("cpu"), include_moe_experts=True, include_non_moe=True, include_vision=vision
)
}
@@ -227,38 +227,13 @@ def loaded_fp8(checkpoint_fp8) -> dict[str, torch.Tensor]:
return _load(checkpoint_fp8[0])
-def _expected_names() -> set[str]:
- names = {"model.embed_tokens.weight", "lm_head.weight"}
- names |= {f"model.hyper_connection_mixer.{leaf}" for leaf in
- ("hc_norm.weight", "input_mix_weight_down.weight", "input_mix_weight_up.weight")}
- for layer in (0, 1):
- for hc in ("attn_hyper_connection", "mlp_hyper_connection"):
- names |= {f"model.layers.{layer}.{hc}.{leaf}" for leaf in (
- "hc_norm.weight", "input_mix_weight_down_block_inject.weight",
- "input_mix_weight_up.weight")}
- names |= {f"model.layers.{layer}.mlp.{leaf}" for leaf in (
- "gate.weight", "shared_expert.gate_up_proj.weight",
- "shared_expert.down_proj.weight", "shared_expert_gate.weight")}
- names |= {f"model.layers.0.linear_attn.{leaf}" for leaf in (
- "in_proj.weight", "conv1d.weight", "A_log", "dt_bias", "norm.weight", "out_proj.weight")}
- names |= {f"model.layers.0.ple.{leaf}" for leaf in (
- "key_proj.weight", "value_proj.weight", "norm_key.weight", "norm_query.weight",
- "norm_conv.weight", "conv1d.weight", "ple_embedding.layer_multipliers",
- "ple_embedding.ngram_heads_offsets", "ple_embedding.ngram_heads_vocab_sizes")}
- names |= {f"model.layers.1.self_attn.{leaf}" for leaf in (
- "qkv_proj.weight", "o_proj.weight", "q_norm.weight", "k_norm.weight",
- "indexer.index_qk_proj.weight", "indexer.q_layernorm.weight",
- "indexer.k_layernorm.weight")}
- return names
-
-
-def test_key_map_is_exactly_the_model_state_dict(loaded):
- assert set(loaded) == _expected_names()
-
-
-def test_mtp_visual_experts_and_table_never_loaded(loaded):
+def test_tower_keys_come_out_under_the_prefix_load_weight_filters(loaded):
+ assert {n for n in loaded if "visual" in n} == {"visual.blocks.0.attn.qkv.weight", "visual.merger.norm.weight"}
+
+
+def test_mtp_experts_and_table_never_loaded(loaded):
for name in loaded:
- assert not name.startswith(("mtp.", "model.visual."))
+ assert not name.startswith("mtp.")
assert ".mlp.experts." not in name
assert "ngram_embedding" not in name
assert not name.endswith((".weight_scale", ".weight_scale_2", ".input_scale", ".weight_scale_inv"))
@@ -486,7 +461,7 @@ FP8_MODULES = (
def test_emitted_keys_are_the_model_state_dict(fixture, request):
"""The reader fills exactly the buffers the engine builds from the same config, block-fp8 ones with the stored dtypes."""
folder, _raw = request.getfixturevalue(fixture)
- loaded, state = _load(folder), meta_state_dict(folder)
+ loaded, state = _load(folder, vision=False), meta_state_dict(folder)
assert set(loaded) == set(state)
if fixture != "checkpoint_fp8":
assert loaded["model.layers.0.linear_attn.in_proj.weight"].dtype is torch.bfloat16
diff --git a/tests/models/test_qwen3_5_moe_weight.py b/tests/models/test_qwen3_5_moe_weight.py
index 1efec609..f5dc2c3c 100644
--- a/tests/models/test_qwen3_5_moe_weight.py
+++ b/tests/models/test_qwen3_5_moe_weight.py
@@ -278,9 +278,9 @@ def _install(folder: str) -> None:
set_quant_config(checkpoint_quant_config(folder, hf, get_model_spec(hf.architectures[0])))
-def _load(folder: str, *, experts: bool = False) -> dict[str, torch.Tensor]:
+def _load(folder: str, *, experts: bool = False, vision: bool = True) -> dict[str, torch.Tensor]:
_install(folder)
- return {n: t.clone() for n, t in iter_weights(folder, torch.device("cpu"), include_moe_experts=experts, include_non_moe=True)}
+ return {n: t.clone() for n, t in iter_weights(folder, torch.device("cpu"), include_moe_experts=experts, include_non_moe=True, include_vision=vision)}
def _meta_state_dict(folder: str) -> dict[str, torch.Tensor]:
@@ -288,11 +288,13 @@ def _meta_state_dict(folder: str) -> dict[str, torch.Tensor]:
from freetoken.engine.config import EngineConfig
from freetoken.engine.engine import _decode_target
from freetoken.layers import rotary
+ from freetoken.mm.config import ENCODER_KINDS, MultimodalConfig
from freetoken.models import create_model
from freetoken.utils.torch_utils import torch_dtype
strategy = "offload" if cached_load_hf_config(folder).architectures[0].startswith("Qwen3_5Moe") else "auto"
- config = EngineConfig(model_path=folder, tp_info=try_get_tp_info(), dtype=torch.bfloat16, moe_strategy=strategy)
+ config = EngineConfig(model_path=folder, tp_info=try_get_tp_info(), dtype=torch.bfloat16, moe_strategy=strategy,
+ mm=MultimodalConfig(disabled_encoders=frozenset(ENCODER_KINDS)))
object.__setattr__(config.model_config, "moe_strategy", strategy)
object.__setattr__(config.model_config, "decode_target", _decode_target(config))
saved = rotary._ROPE_DEVICE
@@ -319,14 +321,14 @@ def checkpoint(request, tmp_path_factory):
def test_emitted_keys_are_the_model_state_dict(checkpoint):
"""Every layout fills exactly the buffers the engine builds from the same config, with the buffers' shapes and (for the weights) dtypes."""
_name, folder, _raw = checkpoint
- loaded, state = _load(folder), _meta_state_dict(folder)
+ loaded, state = _load(folder, vision=False), _meta_state_dict(folder)
assert set(loaded) == set(state)
for key, tensor in loaded.items():
assert tensor.shape == state[key].shape, key
if key.endswith(".weight"):
assert tensor.dtype is state[key].dtype, key
assert not any(k.endswith((".input_global_scale", ".weight_scale_2", ".weight_global_scale", ".k_scale")) for k in loaded)
- assert not any(".mlp.experts." in k or k.startswith(("mtp.", "model.visual.")) for k in loaded)
+ assert not any(".mlp.experts." in k or k.startswith("mtp.") for k in loaded)
def test_expert_quant_tag_follows_the_config(checkpoint):
diff --git a/tests/models/test_qwen3_vl.py b/tests/models/test_qwen3_vl.py
new file mode 100644
index 00000000..c45a9f15
--- /dev/null
+++ b/tests/models/test_qwen3_vl.py
@@ -0,0 +1,153 @@
+"""Qwen VL on the CPU side: parse_config, the engine's encoder decision, the registry invariants, the text-side image scatter (no checkpoint)."""
+
+from __future__ import annotations
+
+from types import SimpleNamespace
+
+import pytest
+import torch
+
+from freetoken.distributed import DistributedInfo
+from freetoken.engine.config import EngineConfig
+from freetoken.mm import MM_PAD_SHIFT_VALUE
+from freetoken.mm.config import ENCODER_KINDS, MultimodalConfig
+from freetoken.models.blocks import embed_input_ids
+from freetoken.models.qwen3_vl import deepstack_add, parse_config
+
+ROPE = {"rope_theta": 5_000_000, "rope_type": "default", "mrope_section": [24, 20, 20], "mrope_interleaved": True}
+
+
+def _hf_config(num_experts=0, arch="Qwen3VLForConditionalGeneration"):
+ text = SimpleNamespace(
+ hidden_size=64, num_hidden_layers=4, num_attention_heads=4, num_key_value_heads=2, head_dim=16,
+ intermediate_size=128, rms_norm_eps=1e-6, hidden_act="silu", vocab_size=1000,
+ max_position_embeddings=4096, rope_parameters=ROPE, rope_scaling=ROPE, model_type="qwen3_vl_text",
+ num_experts=num_experts, num_experts_per_tok=2, moe_intermediate_size=32, norm_topk_prob=True,
+ )
+ vision = SimpleNamespace(
+ hidden_size=32, depth=3, num_heads=4, intermediate_size=64, patch_size=16, temporal_patch_size=2,
+ spatial_merge_size=2, num_position_embeddings=16, out_hidden_size=64, in_channels=3,
+ deepstack_visual_indexes=[0, 1],
+ )
+ return SimpleNamespace(
+ text_config=text, vision_config=vision, image_token_id=151655, tie_word_embeddings=False,
+ model_type="qwen3_vl", architectures=[arch],
+ )
+
+
+def _engine_config(monkeypatch, hf, processor, **overrides):
+ import freetoken.engine.config as engine_config
+ import freetoken.mm.processor as mm_processor
+
+ monkeypatch.setattr(engine_config, "cached_load_hf_config", lambda path: hf)
+ monkeypatch.setattr(engine_config, "checkpoint_quant_config", lambda *args: None)
+ monkeypatch.setattr(mm_processor, "get_mm_processor", lambda path, mm=None: processor)
+ return EngineConfig(
+ model_path="/fake", tp_info=DistributedInfo(rank=0, size=1), dtype=torch.bfloat16, **overrides
+ )
+
+
+def test_vision_carries_mrope_and_deepstack():
+ c = parse_config(_hf_config())
+ assert c.is_multimodal and c.model_is_mrope
+ assert c.rotary_config.mrope_section == [24, 20, 20] and c.rotary_config.mrope_layout == "interleaved"
+ assert c.rotary_config.base == 5_000_000 and c.rotary_config.scaling is None
+ assert c.vision_config.deepstack_visual_indexes == (0, 1) and c.image_token_id == 151655
+ assert c.num_layers == 4 and c.num_experts == 0 and c.architectures == ["Qwen3VLForConditionalGeneration"]
+ assert [g.attn_type.name for g in c.kv_cache_group_specs()] == ["FULL"]
+
+
+def test_no_vision_section_is_a_plain_qwen3():
+ hf = _hf_config()
+ hf.vision_config = None
+ c = parse_config(hf)
+ assert not c.is_multimodal and not c.model_is_mrope and c.rotary_config.mrope_section is None
+
+
+def test_engine_builds_the_tower_for_a_registered_family(monkeypatch):
+ c = _engine_config(monkeypatch, _hf_config(), object())
+ assert [e.kind for e in c.active_encoders] == ["vision"] and c.served_modalities == {"image"}
+ assert c.model_config.is_multimodal and c.model_config.model_is_mrope
+
+
+def test_text_model_only_hands_the_parser_a_config_without_vision(monkeypatch):
+ hf = _hf_config()
+ c = _engine_config(monkeypatch, hf, object(), mm=MultimodalConfig(disabled_encoders=frozenset(ENCODER_KINDS)))
+ assert not c.served_modalities
+ assert not c.model_config.is_multimodal and not c.model_config.model_is_mrope
+ assert hf.vision_config is not None # the cached checkpoint config is left alone
+
+
[email protected]("kinds, served", [({"vision"}, set()), ({"audio"}, {"image"})])
+def test_mm_disable_drops_only_the_named_tower(monkeypatch, kinds, served):
+ c = _engine_config(monkeypatch, _hf_config(), object(), mm=MultimodalConfig(disabled_encoders=frozenset(kinds)))
+ assert c.served_modalities == served and not c.mm.text_model_only
+ assert c.model_config.is_multimodal == bool(served) and c.model_config.model_is_mrope == bool(served)
+
+
+def test_registered_encoders_come_with_a_processor_and_the_model_hooks():
+ from freetoken.models.blocks import SupportsMultimodal
+ from freetoken.models.register import _MODEL_REGISTRY, _load_attr
+
+ for arch, spec in _MODEL_REGISTRY.items():
+ assert (spec.mm_processor is None) == (not spec.encoders), arch
+ for e in spec.encoders:
+ assert e.kind in ENCODER_KINDS and e.modalities, arch
+ if spec.encoders:
+ assert issubclass(_load_attr(spec.module, spec.model_cls), SupportsMultimodal), arch
+
+
+def test_a_family_without_registered_encoders_is_served_text_only(monkeypatch):
+ import freetoken.engine.config as engine_config
+ from dataclasses import replace
+ from freetoken.models.register import get_model_spec
+
+ monkeypatch.setattr(engine_config, "get_model_spec", lambda arch: replace(get_model_spec(arch), mm_processor=None, encoders=()))
+ c = _engine_config(monkeypatch, _hf_config(), None)
+ assert not c.active_encoders and not c.served_modalities and not c.model_config.is_multimodal
+
+
+def test_tower_roots_make_the_fp8_ignore_list_match():
+ from freetoken.layers.quantization import NameMap, QuantConfig
+ from freetoken.models.register import get_model_spec
+
+ spec = get_model_spec("Qwen3VLForConditionalGeneration")
+ # quantized Qwen VL exports list every tower module under its stored name and keep it bf16
+ hf = SimpleNamespace(
+ architectures=["Qwen3VLForConditionalGeneration"],
+ quantization_config={
+ "quant_method": "fp8",
+ "weight_block_size": [128, 128],
+ "modules_to_not_convert": [
+ "lm_head",
+ "model.visual.blocks.0.attn.qkv",
+ "model.visual.blocks.0.attn.proj",
+ "model.visual.blocks.0.mlp.linear_fc1",
+ "model.visual.blocks.0.mlp.linear_fc2",
+ "model.visual.merger.linear_fc1",
+ "model.visual.merger.linear_fc2",
+ ],
+ },
+ )
+ with_roots = QuantConfig.from_hf(hf, name_map=NameMap(roots=spec.checkpoint_roots, packed=spec.packed_modules_mapping))
+ assert with_roots.scheme_for("model.layers.0.self_attn.qkv_proj") is not None
+ assert with_roots.scheme_for("visual.blocks.0.attn.qkv") is None
+ assert with_roots.scheme_for("visual.merger.linear_fc2") is None
+ # without the visual -> model.visual root the tower would be built fp8 against bf16 tensors
+ without = QuantConfig.from_hf(hf, name_map=NameMap(roots=(), packed=spec.packed_modules_mapping))
+ assert without.scheme_for("visual.blocks.0.attn.qkv") is not None
+
+
+def test_image_rows_take_the_leading_columns_and_deepstack_adds_the_next_block():
+ H = 4
+ table = torch.arange(10 * H, dtype=torch.float32).view(10, H)
+ embed = SimpleNamespace(forward=lambda ids: table[ids], num_embeddings=10)
+ assert torch.equal(embed_input_ids(embed, torch.tensor([1, 2]), SimpleNamespace(mm_embeds=None)), table[[1, 2]])
+ ids = torch.tensor([1, MM_PAD_SHIFT_VALUE + 7, 2, MM_PAD_SHIFT_VALUE + 7])
+ mm = torch.arange(2 * 3 * H, dtype=torch.float32).view(2, 3 * H)
+ rows = torch.tensor([1, 3])
+ x = embed_input_ids(embed, ids, SimpleNamespace(mm_embeds=mm, mm_rows=rows))
+ assert torch.equal(x[0], table[1]) and torch.equal(x[2], table[2])
+ assert torch.equal(x[1], mm[0, :H]) and torch.equal(x[3], mm[1, :H])
+ deepstack_add(x, rows, mm, level=1, hidden_size=H)
+ assert torch.equal(x[1], mm[0, :H] + mm[0, 2 * H : 3 * H]) and torch.equal(x[0], table[1])
diff --git a/tests/models/test_qwen3_vl_vision.py b/tests/models/test_qwen3_vl_vision.py
new file mode 100644
index 00000000..f8a18d3d
--- /dev/null
+++ b/tests/models/test_qwen3_vl_vision.py
@@ -0,0 +1,72 @@
+"""Qwen3VLVisionModel DeepStack side outputs on a tiny random tower."""
+
+from __future__ import annotations
+
+import pytest
+import torch
+
+from freetoken.distributed import set_tp_info, try_get_tp_info
+from freetoken.models.qwen3_vl import Qwen3VLVisionModel, VisionConfig
+
+pytestmark = pytest.mark.skipif(not torch.cuda.is_available(), reason="needs CUDA")
+
+
[email protected](autouse=True)
+def _single_rank():
+ # the tower is built from the same TP-aware layers as the text tower
+ if try_get_tp_info() is None:
+ set_tp_info(rank=0, size=1)
+
+
+def _tower(deepstack):
+ vc = VisionConfig(
+ hidden_size=64, depth=3, num_heads=4, intermediate_size=128, patch_size=16, temporal_patch_size=2,
+ spatial_merge_size=2, num_position_embeddings=16, out_hidden_size=32, in_channels=3,
+ deepstack_visual_indexes=deepstack,
+ )
+ torch.set_default_dtype(torch.bfloat16)
+ try:
+ with torch.device("cuda"):
+ tower = Qwen3VLVisionModel(vc)
+ finally:
+ torch.set_default_dtype(torch.float32)
+ for p in tower.state_dict().values():
+ p.normal_(0, 0.02)
+ return tower, vc
+
+
+def test_deepstack_levels_ride_as_extra_columns():
+ tower, vc = _tower((0, 1))
+ feature = torch.zeros(16, 3 * 2 * 16 * 16, dtype=torch.bfloat16, device="cuda")
+ out = tower.forward(feature, [[1, 4, 4]])
+ assert out.shape == (4, vc.out_hidden_size * 3)
+ keys = tower.state_dict()
+ # the DeepStack mergers normalize the merged vector, the final merger each patch
+ assert keys["deepstack_merger_list.0.norm.weight"].shape == (64 * 4,)
+ assert keys["merger.norm.weight"].shape == (64,)
+ assert "deepstack_merger_list.1.linear_fc2.weight" in keys and "deepstack_merger_list.2.norm.weight" not in keys
+
+
[email protected]("deepstack", [(), (0, 1)])
+def test_forward_matches_the_naive_forward(deepstack):
+ tower, vc = _tower(deepstack)
+ feature = torch.randn(64, 3 * 2 * 16 * 16, dtype=torch.bfloat16, device="cuda")
+ out = tower.forward(feature, [[1, 8, 8]])
+ assert out.shape == (16, vc.out_hidden_size * (1 + len(deepstack)))
+ assert torch.equal(out, tower.forward_naive(feature, [[1, 8, 8]]))
+
+
+def test_host_streamed_weights_compute_the_same_output():
+ tower, vc = _tower((1,))
+ feature = torch.randn(64, 3 * 2 * 16 * 16, dtype=torch.bfloat16, device="cuda")
+ resident = tower.forward(feature, [[1, 8, 8]])
+ keys = tower.state_dict()
+ tower.place_weights("host")
+ assert tower.state_dict().keys() == keys.keys()
+ assert tower.state_dict()["blocks.0.attn.qkv.weight"].device.type == "cpu"
+ assert tower._streamer.device_bytes < sum(t.numel() * 2 for k, t in keys.items() if k.startswith("blocks."))
+ for _ in range(2): # a second forward reuses the staging buffers behind the release events
+ assert torch.equal(tower.forward(feature, [[1, 8, 8]]), resident)
+ tower.place_weights("gpu")
+ assert tower.state_dict()["blocks.0.attn.qkv.weight"].device.type == "cuda"
+ assert torch.equal(tower.forward(feature, [[1, 8, 8]]), resident)
diff --git a/tests/scheduler/test_abort_inflight_prefill.py b/tests/scheduler/test_abort_inflight_prefill.py
index a50d0d32..85c21785 100644
--- a/tests/scheduler/test_abort_inflight_prefill.py
+++ b/tests/scheduler/test_abort_inflight_prefill.py
@@ -79,7 +79,7 @@ def _launch_req(pool, cm, tm, prompt, *, cls=Req, track_seqlen=None):
"""A launched (forward in flight) hybrid req: handle locked, pages allocated,
GDN slots held, cached_len advanced -- the state _process_last_data will drain."""
mr = cm.match_req(SimpleNamespace(input_ids=prompt, input_len=len(prompt),
- mm_embeds=None))
+ ))
req = cls(input_ids=prompt, table_idx=tm.allocate(), cached_len=0, output_len=4,
uid=UID, sampling_params=SamplingParams(max_tokens=4),
cache_handle=mr.cuda_handle)
diff --git a/tests/scheduler/test_commit_repoints_page_table.py b/tests/scheduler/test_commit_repoints_page_table.py
index 9e770b06..50db29dc 100644
--- a/tests/scheduler/test_commit_repoints_page_table.py
+++ b/tests/scheduler/test_commit_repoints_page_table.py
@@ -18,7 +18,7 @@ PROMPT = [1, 2, 3, 4, 5, 6, 7, 8]
def _pend(ids):
t = torch.tensor(ids, dtype=torch.int32)
- return SimpleNamespace(input_ids=t, input_len=len(ids), mm_embeds=None)
+ return SimpleNamespace(input_ids=t, input_len=len(ids))
def _admit(cm, page_table, table_idx, ids, handle):
diff --git a/tests/scheduler/test_cost_accounting_core.py b/tests/scheduler/test_cost_accounting_core.py
index 8942c4ca..045010c6 100644
--- a/tests/scheduler/test_cost_accounting_core.py
+++ b/tests/scheduler/test_cost_accounting_core.py
@@ -51,27 +51,28 @@ def _tokenize_msg(uid: int) -> TokenizeMsg:
def test_successful_tokenization_does_not_account_prompt_before_admission():
class Tokenizer:
def tokenize(self, messages):
- return [torch.tensor([10, 11, 12], dtype=torch.int32)]
+ (msg,) = messages
+ return [UserMsg(uid=msg.uid, input_ids=torch.tensor([10, 11, 12], dtype=torch.int32), sampling_params=msg.sampling_params)]
- ok, tensors, errors = _tokenize_requests(Tokenizer(), [_tokenize_msg(1)], _Logger())
- assert [msg.uid for msg in ok] == [1]
- assert tensors[0].tolist() == [10, 11, 12]
+ backend, errors = _tokenize_requests(Tokenizer(), [_tokenize_msg(1)], _Logger())
+ assert [msg.uid for msg in backend] == [1]
+ assert backend[0].input_ids.tolist() == [10, 11, 12]
assert errors == [] # in particular, no early prompt_tokens_delta UserReply
def test_tokenization_failure_and_empty_prompt_are_terminal_without_usage():
class Tokenizer:
def tokenize(self, messages):
- uid = messages[0].uid
- if uid == 2:
+ (msg,) = messages
+ if msg.uid == 2:
raise ValueError("bad template")
- return [torch.empty(0, dtype=torch.int32)]
+ return [UserMsg(uid=msg.uid, input_ids=torch.empty(0, dtype=torch.int32), sampling_params=msg.sampling_params)]
logger = _Logger()
- ok, tensors, errors = _tokenize_requests(
+ backend, errors = _tokenize_requests(
Tokenizer(), [_tokenize_msg(2), _tokenize_msg(3)], logger
)
- assert ok == [] and tensors == []
+ assert backend == []
assert [reply.uid for reply in errors] == [2, 3]
assert all(reply.finished and reply.prompt_tokens_delta == 0 for reply in errors)
assert "could not encode request" in errors[0].error
diff --git a/tests/scheduler/test_hybrid_cache_manager.py b/tests/scheduler/test_hybrid_cache_manager.py
index 080999ad..098f2766 100644
--- a/tests/scheduler/test_hybrid_cache_manager.py
+++ b/tests/scheduler/test_hybrid_cache_manager.py
@@ -25,7 +25,7 @@ def _pool(num_slots=16):
def _pend(ids):
# int32 to match production Req.input_ids dtype (fast_compare_key needs consistent dtype)
t = torch.tensor(ids, dtype=torch.int32)
- return SimpleNamespace(input_ids=t, input_len=len(ids), mm_embeds=None)
+ return SimpleNamespace(input_ids=t, input_len=len(ids))
def test_hybrid_cache_manager_donate_then_hit():
diff --git a/tests/scheduler/test_mm.py b/tests/scheduler/test_mm.py
new file mode 100644
index 00000000..1910adf2
--- /dev/null
+++ b/tests/scheduler/test_mm.py
@@ -0,0 +1,92 @@
+"""Chunk-window planning over image spans: rows, jobs, cache hits, multi-span items, a repeated image across a cut."""
+
+from __future__ import annotations
+
+import torch
+
+from freetoken.message import MMItem
+from freetoken.mm.encoder_cache import EncoderCache
+from freetoken.scheduler.mm import mm_rows_after, plan_mm_batch, plan_mm_chunk
+
+CPU = torch.device("cpu")
+
+
+def _item(h, offsets):
+ return MMItem(modality="image", hash=h, pad_value=h, offsets=offsets, feature=torch.zeros(1))
+
+
+class _Cache:
+ def __init__(self, *hashes):
+ self._h = set(hashes)
+
+ def has(self, h):
+ return h in self._h
+
+
+def test_single_span_split_across_two_chunks():
+ item = _item(7, [[10, 20]])
+ jobs, plan = plan_mm_chunk(1, [item], 0, 16, None)
+ assert jobs == [item] and plan == [(1, 7, 0, 6, 10, 10)]
+ jobs, plan = plan_mm_chunk(1, [item], 16, 32, None)
+ assert jobs == [item] and plan == [(1, 7, 6, 10, 10, 0)] # row_hi == num_tokens: last rows
+
+
+def test_cached_prefix_and_cache_hit_produce_no_job():
+ item = _item(7, [[10, 20]])
+ assert plan_mm_chunk(1, [item], 20, 40, None) == ([], []) # span entirely in the cached prefix
+ jobs, plan = plan_mm_chunk(1, [item], 0, 40, _Cache(7))
+ assert jobs == [] and plan == [(1, 7, 0, 10, 10, 10)] # gather only
+
+
+def test_multi_span_item_maps_to_consecutive_rows():
+ item = _item(9, [[2, 5], [8, 12]]) # 3 + 4 rows
+ jobs, plan = plan_mm_chunk(3, [item], 0, 10, None)
+ assert jobs == [item] # one job per item even when two spans overlap the window
+ assert plan == [(3, 9, 0, 3, 7, 2), (3, 9, 3, 5, 7, 8)]
+ jobs, plan = plan_mm_chunk(3, [item], 10, 20, None)
+ assert jobs == [item] and plan == [(3, 9, 5, 7, 7, 0)]
+
+
+def test_rows_after_the_cached_prefix():
+ item = _item(1, [[2, 5], [8, 12]])
+ assert mm_rows_after(item, 0) == 7
+ assert mm_rows_after(item, 4) == 1 + 4 # the hit ends inside the first span
+ assert mm_rows_after(item, 12) == 0
+
+
+def test_repeated_image_survives_a_chunk_cut_between_its_occurrences():
+ cache = EncoderCache(storage="cpu")
+ items = [_item(7, [[0, 4]]), _item(7, [[5, 9]])]
+ for item in items:
+ cache.register(item.hash, 1, mm_rows_after(item, 0))
+ encoded = []
+
+ def run_chunk(lo, hi):
+ jobs, plan = plan_mm_chunk(1, items, lo, hi, cache)
+ for item in jobs:
+ if not cache.has(item.hash):
+ assert item.feature is not None
+ encoded.append(item.hash)
+ cache.put(item.hash, torch.ones(4, 8))
+ item.feature = None
+ for uid, h, row_lo, row_hi, _, _ in plan:
+ cache.get_slice(h, row_lo, row_hi, CPU)
+ cache.consume(h, uid, row_hi - row_lo)
+ return jobs
+
+ assert len(run_chunk(0, 6)) == 1 # both occurrences overlap the window: one job
+ assert cache.has(7) # the second occurrence still has rows to gather
+ assert run_chunk(6, 10) == [] # nothing to encode, both features already gone
+ assert encoded == [7] and not cache.has(7)
+
+
+def test_batch_rows_follow_the_reqs_in_batch_order():
+ from types import SimpleNamespace
+
+ # req 1: 6 tokens, image on [2, 5); req 2: chunk [4, 10) of a prompt whose image spans [3, 8)
+ a = SimpleNamespace(uid=1, mm_items=[_item(7, [[2, 5]])], cached_len=0, device_len=6, extend_len=6)
+ b = SimpleNamespace(uid=2, mm_items=[_item(8, [[3, 8]])], cached_len=4, device_len=10, extend_len=6)
+ jobs, plan, rows = plan_mm_batch([a, b], None)
+ assert [j.hash for j in jobs] == [7, 8]
+ assert plan == [(1, 7, 0, 3, 3, 2), (2, 8, 1, 5, 5, 0)]
+ assert rows == [2, 3, 4, 6, 7, 8, 9] # req 2 starts at batch row 6; its image rows 1..5 land on its first four tokens
diff --git a/tests/scheduler/test_swa_pagesize.py b/tests/scheduler/test_swa_pagesize.py
index 74c2982d..3f55c5e8 100644
--- a/tests/scheduler/test_swa_pagesize.py
+++ b/tests/scheduler/test_swa_pagesize.py
@@ -135,7 +135,7 @@ def test_hybrid_chunk_donate_skips_unaligned_boundary(ps):
req = _req(3 * ps + 3, 1)
h = cm.match_req(SimpleNamespace(input_ids=req.input_ids, input_len=req.input_len,
- mm_embeds=None)).cuda_handle
+ )).cuda_handle
req.cache_handle = h
req.cached_len = h.cached_len
cm.lock(h)
diff --git a/tests/server/test_anthropic_api.py b/tests/server/test_anthropic_api.py
index 8ea1d1d1..59ad45da 100644
--- a/tests/server/test_anthropic_api.py
+++ b/tests/server/test_anthropic_api.py
@@ -17,11 +17,14 @@ import os
import sys
from types import SimpleNamespace
+import pytest
+
_ROOT = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
_PY = os.path.join(_ROOT, "python")
if _PY not in sys.path:
sys.path.insert(0, _PY)
+from freetoken.message import UserMsg # noqa: E402
from freetoken.message.frontend import UserReply # noqa: E402
from freetoken.server import anthropic_api as A # noqa: E402
from freetoken.server.anthropic_models import AnthropicMessagesRequest # noqa: E402
@@ -421,10 +424,12 @@ class FakeState:
self._outputs = outputs
self.maintenance_state = "serving"
self.config = SimpleNamespace(
+ mm=SimpleNamespace(text_model_only=False, disabled_encoders=frozenset()),
reasoning_parser=None,
tool_call_parser="llama3",
served_model_name="test-model",
model_path="/test",
+ served_modalities=frozenset(),
)
self._uid = 0
self._count_manager = None
@@ -605,23 +610,20 @@ def test_convert_native_thinking_toggle():
# --------------------------------------------------------------------------- #
# /v1/messages/count_tokens (route + neutral count_prompt_tokens primitive)
# --------------------------------------------------------------------------- #
-class _FakeIds:
- def __init__(self, n):
- self._n = n
-
- def numel(self):
- return self._n
-
-
class _FakeTokenizeManager:
- """Counts tokens as len(str(prompt)) so tests are deterministic without a model."""
+ """Counts tokens as len(str(prompt)) so tests are deterministic without a model; returns the wire message like the real manager."""
def __init__(self):
self.msgs = []
def tokenize(self, msgs):
+ import torch
+
self.msgs.extend(msgs)
- return [_FakeIds(len(str(m.text))) for m in msgs]
+ return [
+ UserMsg(uid=m.uid, input_ids=torch.zeros(len(str(m.text)), dtype=torch.int32), sampling_params=m.sampling_params)
+ for m in msgs
+ ]
def _count_client(manager=None, maintenance_state="serving"):
@@ -811,10 +813,10 @@ def test_frontend_tokenizer_concurrent_first_build_dedupes():
orig_load, orig_tm = _utils.load_tokenizer, _tok.TokenizeManager
_utils.load_tokenizer = _slow_load
- _tok.TokenizeManager = lambda tok: tok
+ _tok.TokenizeManager = lambda tok, mm=None: tok
try:
fm = FrontendManager(
- config=SimpleNamespace(model_path="dedupe-test"),
+ config=SimpleNamespace(model_path="dedupe-test", mm=None),
send_tokenizer=None,
recv_tokenizer=None,
)
@@ -865,3 +867,49 @@ def test_tool_result_carries_the_wire_tool_use_id():
results = [m for m in spec.messages if m["role"] == "tool"]
assert [(m["tool_call_id"], m["content"]) for m in results] == [
("toolu_b", "60F"), ("toolu_a", "72F")]
+
+
+def test_count_tokens_reads_the_real_manager_return_contract():
+ import torch
+ from freetoken.tokenizer.tokenize import TokenizeManager
+
+ class _Tokenizer:
+ def apply_chat_template(self, messages, **kwargs):
+ return "rendered"
+
+ def encode(self, prompt, return_tensors=None, add_special_tokens=True):
+ return torch.tensor([[7, 8, 9, 10, 11]], dtype=torch.long)
+
+ client, _ = _count_client(TokenizeManager(_Tokenizer()))
+ r = client.post(
+ "/v1/messages/count_tokens",
+ json={"model": "claude-x", "messages": [{"role": "user", "content": "hi"}]},
+ )
+ assert r.status_code == 200, r.text
+ assert r.json() == {"input_tokens": 5}
+
+
+def test_tool_result_image_is_rejected_not_dropped():
+ body = {
+ "model": "claude-x",
+ "max_tokens": 16,
+ "messages": [
+ {"role": "user", "content": "take a screenshot"},
+ {"role": "assistant", "content": [{"type": "tool_use", "id": "toolu_1", "name": "shot", "input": {}}]},
+ {
+ "role": "user",
+ "content": [
+ {
+ "type": "tool_result",
+ "tool_use_id": "toolu_1",
+ "content": [{"type": "image", "source": {"type": "base64", "media_type": "image/png", "data": "aGk="}}],
+ }
+ ],
+ },
+ ],
+ }
+ with pytest.raises(ValueError, match="tool results"):
+ A.convert_anthropic_to_genspec(AnthropicMessagesRequest.model_validate(body), {})
+ r = _client(FakeState([])).post("/v1/messages", json=body)
+ assert r.status_code == 400, r.text
+ assert "tool results" in r.json()["error"]["message"]
diff --git a/tests/server/test_effort_dialect.py b/tests/server/test_effort_dialect.py
index 3b04ec54..6a68a4ec 100644
--- a/tests/server/test_effort_dialect.py
+++ b/tests/server/test_effort_dialect.py
@@ -31,6 +31,7 @@ def run(coro):
class FakeState:
def __init__(self, reasoning_parser: str | None = None) -> None:
self.config = SimpleNamespace(
+ mm=SimpleNamespace(text_model_only=False, disabled_encoders=frozenset()),
model_path="/models/unit-model",
served_model_name="unit-model",
tool_call_parser="llama3",
diff --git a/tests/server/test_generation_accounting.py b/tests/server/test_generation_accounting.py
index 204c1f8c..8276163b 100644
--- a/tests/server/test_generation_accounting.py
+++ b/tests/server/test_generation_accounting.py
@@ -46,6 +46,7 @@ class FakeState:
def __init__(self, replies: list[UserReply]) -> None:
self.config = SimpleNamespace(
+ mm=SimpleNamespace(text_model_only=False, disabled_encoders=frozenset()),
model_path="/m",
served_model_name="unit-model",
tool_call_parser="llama3",
diff --git a/tests/server/test_message_wire.py b/tests/server/test_message_wire.py
index 3bd7cc6b..e2286671 100644
--- a/tests/server/test_message_wire.py
+++ b/tests/server/test_message_wire.py
@@ -7,9 +7,13 @@ wire with its fields intact; these pin the ones carrying state a later consumer
from __future__ import annotations
+import pytest
+import torch
+
from freetoken.message import (
BaseBackendMsg,
DetokenizeMsg,
+ MMItem,
BaseFrontendMsg,
BaseTokenizerMsg,
CacheRebuildBackendMsg,
@@ -18,6 +22,7 @@ from freetoken.message import (
CacheRebuildResultMsg,
PromptAdmittedMsg,
TokenizeMsg,
+ UserMsg,
UserReply,
)
from freetoken.core import SamplingParams
@@ -122,3 +127,78 @@ def test_client_dicts_with_the_wire_tag_key_survive_intact():
assert isinstance(out, TokenizeMsg)
assert out.chat_template_kwargs == payload
assert out.tools[0]["function"]["parameters"] == payload
+
+
+def test_tensor_wire_nd_and_dtype_roundtrip():
+ import numpy as np
+ import torch
+ from freetoken.message.utils import deserialize_type, serialize_type
+
+ cases = [
+ torch.arange(6, dtype=torch.int32),
+ torch.arange(12, dtype=torch.int64).reshape(3, 4),
+ torch.randn(2, 3, 5, dtype=torch.float32),
+ torch.randn(4, 1536, dtype=torch.bfloat16),
+ torch.randn(3, 4)[:, :2], # non-contiguous input
+ torch.tensor(7, dtype=torch.int32), # 0-d
+ ]
+ for t in cases:
+ out = deserialize_type({}, serialize_type(t))
+ assert out.dtype == t.dtype
+ assert out.shape == t.shape
+ assert torch.equal(out, t)
+
+ # legacy payloads carry no shape and decode as 1-D
+ legacy = {
+ "__type__": "Tensor",
+ "buffer": np.arange(4, dtype=np.int32).tobytes(),
+ "dtype": "torch.int32",
+ }
+ out = deserialize_type({}, legacy)
+ assert out.shape == (4,) and out.dtype == torch.int32
+
+ # 1-D payloads still omit the shape field (old decoders can read them)
+ assert "shape" not in serialize_type(torch.arange(3, dtype=torch.int32))
+
+
+def test_user_msg_with_mm_items_survives_the_wire():
+ item = MMItem(
+ modality="image",
+ hash=0x1234ABCD,
+ pad_value=1_000_000 + 0x1234ABCD,
+ offsets=[[7, 11]],
+ feature=torch.randn(16, 1536, dtype=torch.bfloat16),
+ model_specific_data={"grid_thw": [1, 4, 4]},
+ )
+ msg = UserMsg(
+ uid=9,
+ input_ids=torch.arange(16, dtype=torch.int32),
+ sampling_params=SamplingParams(),
+ mm_items=[item],
+ mrope_positions=torch.zeros(3, 16, dtype=torch.int32),
+ mrope_delta=-3,
+ )
+ out = BaseBackendMsg.decoder(msg.encoder())
+ assert isinstance(out, UserMsg)
+ got = out.mm_items[0]
+ assert (got.hash, got.pad_value, got.offsets) == (0x1234ABCD, 1_000_000 + 0x1234ABCD, [[7, 11]])
+ assert got.num_tokens == 4 and got.grid_thw == [1, 4, 4] # model_specific_data reads as attributes
+ assert got.precomputed_embeddings is None
+ assert torch.equal(got.feature, item.feature)
+ assert out.mrope_positions.shape == (3, 16) and out.mrope_delta == -3
+
+
+def test_mm_item_shape_rules():
+ t = torch.zeros(2, 4)
+ item = MMItem(modality="image", hash=1, pad_value=1, offsets=[[3, 5], [9, 12]], feature=t)
+ item.validate()
+ assert item.num_tokens == 5
+ assert getattr(item, "grid_thw", None) is None # missing model-specific key -> AttributeError path
+ with pytest.raises(ValueError, match="half-open"):
+ MMItem(modality="image", hash=1, pad_value=1, offsets=[[5, 5]], feature=t).validate()
+ with pytest.raises(ValueError, match="exactly one"):
+ MMItem(modality="image", hash=1, pad_value=1, offsets=[[0, 2]]).validate()
+ with pytest.raises(ValueError, match="exactly one"):
+ MMItem(
+ modality="image", hash=1, pad_value=1, offsets=[[0, 2]], feature=t, precomputed_embeddings=t
+ ).validate()
diff --git a/tests/server/test_mm_input.py b/tests/server/test_mm_input.py
new file mode 100644
index 00000000..19d9f865
--- /dev/null
+++ b/tests/server/test_mm_input.py
@@ -0,0 +1,142 @@
+"""Image input plumbing: content-part rendering, ref collection, fetch, and the gate.
+
+No GPU and no model checkpoint: everything here is pure frontend logic."""
+
+from __future__ import annotations
+
+import asyncio
+import base64
+from types import SimpleNamespace
+
+import pytest
+
+from freetoken.mm.media import collect_image_refs, fetch_image_bytes, image_reject_reason
+from freetoken.server.generation import GenerationError, render_messages
+from freetoken.server.stats import derive_model_card
+
+PNG = base64.b64encode(b"fakepng").decode()
+
+
+def _config(**overrides):
+ fields = dict(
+ model_path="/nonexistent", allowed_media_domains="", allowed_local_media_path="",
+ served_model_name="unit-model", max_seq_len=8192, model_config=SimpleNamespace(),
+ )
+ text_model_only = overrides.pop("text_model_only", False)
+ serves_images = overrides.pop("vision_enabled", False)
+ mm = SimpleNamespace(
+ text_model_only=text_model_only,
+ disabled_encoders=frozenset({"vision", "audio"}) if text_model_only else frozenset(),
+ )
+ return SimpleNamespace(mm=mm, served_modalities=frozenset({"image"}) if serves_images else frozenset(), **{**fields, **overrides})
+
+
+def test_image_url_part_becomes_template_image_part():
+ msgs = render_messages(
+ [{"role": "user", "content": [
+ {"type": "text", "text": "hi"},
+ {"type": "image_url", "image_url": {"url": "https://x/y.png"}},
+ ]}]
+ )
+ content = msgs[0]["content"]
+ assert isinstance(content, list)
+ assert content[1]["type"] == "image"
+ refs = collect_image_refs(msgs)
+ assert refs == [{"kind": "url", "data": "https://x/y.png"}]
+ # refs are popped; the template-facing part stays
+ assert msgs[0]["content"][1] == {"type": "image"}
+ assert collect_image_refs(msgs) == []
+
+
+def test_collect_refs_preserves_prompt_order():
+ msgs = render_messages(
+ [
+ {"role": "user", "content": [{"type": "image_url", "image_url": {"url": "u1"}}]},
+ {"role": "user", "content": [
+ {"type": "image_url", "image_url": {"url": "u2"}},
+ {"type": "image_url", "image_url": {"url": "u3"}},
+ ]},
+ ]
+ )
+ assert [r["data"] for r in collect_image_refs(msgs)] == ["u1", "u2", "u3"]
+
+
+def test_fetch_decodes_data_uri_and_raw_b64():
+ refs = [
+ {"kind": "url", "data": f"data:image/png;base64,{PNG}"},
+ {"kind": "b64", "data": PNG},
+ ]
+ assert asyncio.run(fetch_image_bytes(refs, _config())) == [b"fakepng", b"fakepng"]
+
+
+def test_fetch_failures_surface_as_generation_error(monkeypatch):
+ from freetoken.server import generation as gen
+
+ # bypass the capability gate; the fetch failure itself must become a GenerationError
+ monkeypatch.setattr(gen, "image_reject_reason", lambda config: None)
+ state = SimpleNamespace(config=_config())
+ with pytest.raises(GenerationError):
+ asyncio.run(gen._resolve_images([{"kind": "url", "data": "ftp://nope"}], state))
+
+
+def test_image_gate_reasons():
+ assert "text-model-only" in image_reject_reason(_config(text_model_only=True))
+ assert "vision" in image_reject_reason(_config(model_path="/nonexistent"))
+
+
+def test_stats_model_card_lists_the_accepted_input_modalities():
+ assert derive_model_card(_config())["input_modalities"] == ["text"]
+ assert derive_model_card(_config(vision_enabled=True))["input_modalities"] == ["text", "image"]
+
+
+def test_media_domain_allowlist():
+ from freetoken.mm.media import _check_media_domain
+
+ config = _config(allowed_media_domains="cdn.example.com, Other.COM.")
+ _check_media_domain("https://cdn.example.com/a.png", config) # allowed: no raise
+ _check_media_domain("https://OTHER.com./b.png", config) # case/root-dot normalized
+ with pytest.raises(ValueError, match="allowed domains"):
+ _check_media_domain("https://evil.com/a.png", config)
+ # empty allowlist admits any domain
+ _check_media_domain("https://evil.com/a.png", _config())
+
+ with pytest.raises(ValueError, match="allowed domains"):
+ asyncio.run(
+ fetch_image_bytes([{"kind": "url", "data": "https://evil.com/a.png"}], config)
+ )
+
+
+def test_local_media_requires_allowlisted_root(tmp_path):
+ img = tmp_path / "img.png"
+ img.write_bytes(b"fakepng")
+ url = f"file://{img}"
+
+ # gate off (default): rejected
+ with pytest.raises(ValueError, match="allowed-local-media-path"):
+ asyncio.run(fetch_image_bytes([{"kind": "url", "data": url}], _config()))
+
+ # gate on, file under the root: served
+ config = _config(allowed_local_media_path=str(tmp_path))
+ assert asyncio.run(fetch_image_bytes([{"kind": "url", "data": url}], config)) == [b"fakepng"]
+
+ # a path outside the root is rejected even with the gate on
+ with pytest.raises(ValueError, match="subpath"):
+ asyncio.run(fetch_image_bytes([{"kind": "url", "data": "file:///etc/hostname"}], config))
+
+
+def test_image_token_budget_flags_land_in_the_multimodal_config():
+ from unittest.mock import patch
+
+ from freetoken.server.args import parse_args
+
+ hf = SimpleNamespace(to_dict=lambda: {"architectures": ["Qwen3VLForConditionalGeneration"], "torch_dtype": "bfloat16"})
+ with patch("freetoken.utils.cached_load_hf_config", lambda _path: hf):
+ args, _ = parse_args([
+ "--model", "/models/anon", "--image-min-tokens", "64", "--image-max-tokens", "1024",
+ "--mm-processor-kwargs", '{"size": {"longest_edge": 4096}}',
+ ])
+ assert (args.mm.image_min_tokens, args.mm.image_max_tokens) == (64, 1024)
+ assert args.mm.processor_kwargs == {"size": {"longest_edge": 4096}}
+ assert parse_args(["--model", "/models/anon"])[0].mm.processor_kwargs == {}
+ with pytest.raises(SystemExit): # argparse reports the bad pair and exits
+ parse_args(["--model", "/models/anon", "--image-min-tokens", "2048", "--image-max-tokens", "1024"])
diff --git a/tests/server/test_openai_api.py b/tests/server/test_openai_api.py
index 58d15d7c..e33018e1 100644
--- a/tests/server/test_openai_api.py
+++ b/tests/server/test_openai_api.py
@@ -30,6 +30,7 @@ class FakeState:
reasoning_parser: str | None = None,
) -> None:
self.config = SimpleNamespace(
+ mm=SimpleNamespace(text_model_only=False, disabled_encoders=frozenset()),
model_path="/models/unit-model",
served_model_name="unit-model",
tool_call_parser=tool_call_parser,
diff --git a/tests/server/test_responses_api.py b/tests/server/test_responses_api.py
index d283de88..0da62172 100644
--- a/tests/server/test_responses_api.py
+++ b/tests/server/test_responses_api.py
@@ -317,6 +317,7 @@ class FakeState:
self._cached_tokens = cached_tokens # stamped on the first ack (admission reply)
self.maintenance_state = "serving"
self.config = SimpleNamespace(
+ mm=SimpleNamespace(text_model_only=False, disabled_encoders=frozenset()),
reasoning_parser=None, tool_call_parser="llama3",
served_model_name="test-model", model_path="/test",
)
diff --git a/tests/tokenizer/test_mm_tokenize.py b/tests/tokenizer/test_mm_tokenize.py
new file mode 100644
index 00000000..5dbf2822
--- /dev/null
+++ b/tests/tokenizer/test_mm_tokenize.py
@@ -0,0 +1,105 @@
+"""Image tokenization against real Qwen VL checkpoints (skipped when absent)."""
+
+from __future__ import annotations
+
+import io
+import os
+
+import pytest
+import torch
+
+from freetoken.mm import MM_PAD_SHIFT_VALUE, mm_pad_value
+
+MODELS = [
+ p
+ for p in (os.environ.get("FREETOKEN_QWEN36_MODEL", ""), os.environ.get("FREETOKEN_QWEN3VL_MODEL", ""))
+ if os.path.exists(os.path.join(p, "config.json"))
+]
+
+pytestmark = pytest.mark.skipif(
+ not MODELS, reason="neither FREETOKEN_QWEN36_MODEL nor FREETOKEN_QWEN3VL_MODEL points at a checkpoint"
+)
+
+
+def _png(w, h):
+ from PIL import Image
+
+ buf = io.BytesIO()
+ Image.new("RGB", (w, h), (120, 30, 200)).save(buf, format="PNG")
+ return buf.getvalue()
+
+
[email protected](scope="module", params=MODELS, ids=os.path.basename)
+def manager(request):
+ from freetoken.mm.processor import get_mm_processor
+ from freetoken.tokenizer.tokenize import TokenizeManager
+ from freetoken.utils.hf import load_tokenizer
+
+ return TokenizeManager(load_tokenizer(request.param), get_mm_processor(request.param))
+
+
+def _msg(images, n_parts=1):
+ from freetoken.core import SamplingParams
+ from freetoken.message import TokenizeMsg
+
+ content = [{"type": "text", "text": "描述"}] + [{"type": "image"}] * n_parts
+ return TokenizeMsg(
+ uid=1, text=[{"role": "user", "content": content}],
+ sampling_params=SamplingParams(), images=images,
+ )
+
+
+def test_expansion_mrope_and_radix_ids(manager):
+ r = manager.tokenize([_msg([_png(512, 512)])])[0]
+ item = r.mm_items[0]
+ L = r.input_ids.numel()
+ assert item.num_tokens == 256 and item.grid_thw == [1, 32, 32]
+ assert r.mrope_positions.shape == (3, L)
+ assert int(r.mrope_positions.max()) + 1 - L == r.mrope_delta
+ # the image span carries one content pad id above the vocab; text is untouched
+ (start, end), = item.offsets
+ span = r.input_ids[start:end]
+ assert span.unique().numel() == 1
+ assert span[0].item() == item.pad_value == mm_pad_value(item.hash) >= MM_PAD_SHIFT_VALUE
+ assert bool((r.input_ids[:start] < MM_PAD_SHIFT_VALUE).all())
+
+
+def test_same_image_same_hash_and_pad(manager):
+ a = manager.tokenize([_msg([_png(512, 512)])])[0]
+ b = manager.tokenize([_msg([_png(512, 512)])])[0]
+ assert a.mm_items[0].hash == b.mm_items[0].hash
+ assert torch.equal(a.input_ids, b.input_ids)
+ c = manager.tokenize([_msg([_png(256, 256)])])[0]
+ assert c.mm_items[0].hash != a.mm_items[0].hash
+
+
+def test_image_max_tokens_clamps(manager):
+ from freetoken.mm.config import MultimodalConfig
+ from freetoken.mm.processor import get_mm_processor
+ from freetoken.tokenizer.tokenize import TokenizeManager
+
+ path = manager.tokenizer.name_or_path
+ budgeted = TokenizeManager(manager.tokenizer, get_mm_processor(path, MultimodalConfig(image_max_tokens=2048)))
+ assert manager.tokenize([_msg([_png(3840, 2160)])])[0].mm_items[0].num_tokens > 2048
+ assert budgeted.tokenize([_msg([_png(3840, 2160)])])[0].mm_items[0].num_tokens <= 2048
+
+
+def test_count_prompt_tokens_counts_the_expanded_image(manager):
+ import asyncio
+ import base64
+ from types import SimpleNamespace
+
+ from freetoken.server.generation import count_prompt_tokens
+
+ png = _png(512, 512)
+ messages = [{"role": "user", "content": [
+ {"type": "image", "freetoken_ref": {"kind": "b64", "data": base64.b64encode(png).decode()}},
+ {"type": "text", "text": "描述"},
+ ]}]
+ state = SimpleNamespace(
+ frontend_tokenizer=lambda: manager,
+ config=SimpleNamespace(served_modalities=frozenset({"image"}), mm=SimpleNamespace(text_model_only=False, disabled_encoders=frozenset())),
+ )
+ counted = asyncio.run(count_prompt_tokens(messages, None, {}, state))
+ expanded = manager.tokenize([_msg([png])])[0].input_ids.numel()
+ assert counted == expanded and counted > 256
diff --git a/tests/tokenizer/test_tokenize.py b/tests/tokenizer/test_tokenize.py
index 65fa5f2d..369576a1 100644
--- a/tests/tokenizer/test_tokenize.py
+++ b/tests/tokenizer/test_tokenize.py
@@ -37,7 +37,7 @@ def test_tokenize_manager_passes_chat_template_kwargs():
chat_template_kwargs={"enable_thinking": True},
)
- [input_ids] = manager.tokenize([msg])
+ input_ids = manager.tokenize([msg])[0].input_ids
assert tokenizer.chat_template_kwargs == {
"tokenize": False,
@@ -64,7 +64,7 @@ def test_tokenize_manager_passes_tools_to_chat_template():
tools=tools,
)
- [input_ids] = manager.tokenize([msg])
+ input_ids = manager.tokenize([msg])[0].input_ids
assert tokenizer.chat_template_kwargs == {
"tokenize": False,
@@ -120,7 +120,7 @@ def encode_messages(messages, thinking_mode, reasoning_effort=None):
tools=tools,
)
- [input_ids] = manager.tokenize([msg])
+ input_ids = manager.tokenize([msg])[0].input_ids
assert tokenizer.prompt == "dsv4 prompt"
assert input_ids.tolist() == [4, 5, 6]
@@ -163,7 +163,7 @@ def encode_messages(messages, thinking_mode, reasoning_effort=None):
]
msg = TokenizeMsg(uid=1, text=messages, sampling_params=SamplingParams())
- [input_ids] = manager.tokenize([msg])
+ input_ids = manager.tokenize([msg])[0].input_ids
assert tokenizer.prompt == "dsv4 prompt"
assert input_ids.tolist() == [4, 5, 6]