FreeToken brings datacenter-scale model serving to your desktop. Run massive models locally, fast and efficiently.
feat(gguf): take image input on Qwen and Gemma-4 GGUF checkpoints from an mmproj (#803)
Xiaoze Fan609191d81f21
diff --git a/docs/cli.md b/docs/cli.mdindex 5a759374..138683b0 100644--- a/docs/cli.md+++ b/docs/cli.md@@ -122,6 +122,7 @@ so a client can gate its attachment controls without reading the checkpoint conf| --- | --- | --- || `--text-model-only` | off | Serve a multimodal checkpoint text-only: no encoder tower is built (its VRAM goes to the KV/expert pools) and every multimodal input is rejected. Same as `--mm-disable` with every encoder kind || `--mm-disable` | none | Encoder towers to leave unbuilt (`vision`, `audio`); every input they would serve is rejected |+| `--mmproj` | found next to the model | GGUF models only: the llama.cpp vision tower file (`mmproj-*.gguf`), see [Image input](models.md#image-input). Without it, FreeToken takes a `*mmproj*.gguf` that fits the model (output width, tower type, image tokens) from the model file's own folder, BF16 before F32, F16 and quantized files. With none, or with files of several models it cannot tell apart, the model serves text only; a repo that keeps each quant in a subfolder and the mmproj at the top needs `--mmproj`. The log names the file it loads || `--mm-encoder-weights` | host | Where the encoder tower's block weights live. `host` streams them from pinned host banks two blocks at a time behind the compute, so the GPU holds two blocks instead of the whole tower; small images pay the copy time, large ones hide it behind the compute. `gpu` keeps them resident. An encoder without a block stack stays resident either way || `--image-min-tokens`, `--image-max-tokens` | processor defaults | Per-image token budget: the image processor resizes every image to take between these many tokens, converted to the family's own units by its processor. A family with fixed budgets honors the maximum only and refuses one below its smallest budget at start-up || `--mm-processor-kwargs` | none | JSON object of extra keyword arguments for the checkpoint's image processor call, for knobs the token budget does not cover; applied after the budget, so an explicit key wins |diff --git a/docs/models.md b/docs/models.mdindex a9b66a24..914e813b 100644--- a/docs/models.md+++ b/docs/models.md@@ -37,6 +37,11 @@ These families accept image input by default; pass `--text-model-only` to skip t| MiniMax-M3 (`minimax_m3`: CLIP-style ViT tower, streamed under `--mm-encoder-weights host`) | one token per 28x28 pixels of the resized image, dynamic resolution | pixel areas in `size.shortest_edge` / `longest_edge`; checkpoint defaults 4 to 576 tokens | `{"size": {"longest_edge": 1048576}}` || DeepSeek-V4-Flash-Vision (`deepseek_v4`: native-resolution ViT tower, streamed under `--mm-encoder-weights host`) | one token per 3x3 grid of 14x14 patches, against a per-image cap; the block carries sentinel tokens and lead pads that align it to a 4-token compression stride, so its length depends on where it lands in the prompt | the maximum is a cap on the block (`vision_max_n_token`, checkpoint default 384) and lowers it, the minimum is read as the checkpoint's upscale floor and has no effect | not read |+A GGUF model takes image input from a separate llama.cpp mmproj file, found in the model file's own folder or passed with `--mmproj` (see the+[CLI reference](cli.md#image-input)). A GGUF file that carries the vision tensors itself (the Ollama layout) serves text only. The tower runs in bf16+whatever the mmproj stores; a quantized mmproj is dequantized at load and keeps its quantization error. Image tokens and budgets are those of the+family's row above.+## MoE strategies`ft serve --moe-strategy {auto,fused,offload,cpu,hybrid}` (`--moe-backend` is the deprecated old spelling):diff --git a/python/freetoken/checkpoint/convert.py b/python/freetoken/checkpoint/convert.pyindex 5551759f..3c496d8a 100644--- a/python/freetoken/checkpoint/convert.py+++ b/python/freetoken/checkpoint/convert.py@@ -220,8 +220,12 @@ def convert_checkpoint(# 1) dense weights (host tensors; load straight to CPU to avoid GPU pressure)_progress("dense", 0, 0) # phase start; per-tensor cumulative bytes follow (total unknown)dense_bytes = 0+ from freetoken.models.gguf.reader import gguf_config_source++ # an FTW keeps only a GGUF's text metadata, so it cannot build an mmproj tower+ include_vision = gguf_config_source(model_path) is Nonefor name, tensor in count_bar(load_weight(model_path, torch.device("cpu"),- include_moe_experts=include_moe_experts),+ include_moe_experts=include_moe_experts, include_vision=include_vision),"Converting dense weights"):writer.add_tensor(name, tensor, kind="weight")n_weight += 1diff --git a/python/freetoken/kernel/aot_models.py b/python/freetoken/kernel/aot_models.pyindex e45b98d5..3773d86d 100644--- a/python/freetoken/kernel/aot_models.py+++ b/python/freetoken/kernel/aot_models.py@@ -360,7 +360,7 @@ SUPPORTED_MODELS: tuple[AotModel, ...] = (architecture="Gemma4UnifiedForConditionalGeneration",hidden_size=3840,kv_groups=((1, 512), (8, 256)),- arch_aliases=("Gemma4UnifiedForCausalLM",),+ arch_aliases=("Gemma4UnifiedForCausalLM", "Gemma4UnifiedGGUFForCausalLM"),),AotModel(name="nvidia/Gemma-4-31B-IT-NVFP4",diff --git a/python/freetoken/mm/processor.py b/python/freetoken/mm/processor.pyindex 84463bc2..7872b7db 100644--- a/python/freetoken/mm/processor.py+++ b/python/freetoken/mm/processor.py@@ -84,9 +84,9 @@ class MMProcessor(ABC):"""The checkpoint's image processor, loaded on first use (tokenizer workers share the instance across threads)."""with self._image_processor_lock:if self._image_processor_instance is None:- from transformers import AutoImageProcessor+ from freetoken.utils import load_image_processor- self._image_processor_instance = AutoImageProcessor.from_pretrained(self.model_path)+ self._image_processor_instance = load_image_processor(self.model_path)return self._image_processor_instancedef get_mm_processor_kwargs(self, mm: MultimodalConfig) -> dict[str, Any]:diff --git a/python/freetoken/models/gemma4/__init__.py b/python/freetoken/models/gemma4/__init__.pyindex 4fcf15a3..4432c38e 100644--- a/python/freetoken/models/gemma4/__init__.py+++ b/python/freetoken/models/gemma4/__init__.py@@ -1,6 +1,6 @@from .attention import Gemma4Attentionfrom .config import parse_config-from .gguf import iter_gguf_weights, parse_gguf_config, to_gguf+from .gguf import gguf_mm_config, iter_gguf_weights, parse_gguf_config, to_gguffrom .model import Gemma4ForCausalLM, Gemma4ForConditionalGeneration, Gemma4UnifiedForConditionalGenerationfrom .moe import Gemma4Routerfrom .vision import Gemma4MultimodalEmbedder, Gemma4UnifiedVisionEmbedder, Gemma4VisionModel@@ -25,6 +25,7 @@ __all__ = ["iter_vision_weights","iter_weights","iter_weights_parallel",+ "gguf_mm_config","parse_gguf_config","iter_gguf_weights","to_gguf",diff --git a/python/freetoken/models/gemma4/gguf.py b/python/freetoken/models/gemma4/gguf.pyindex 3d4de920..047d089d 100644--- a/python/freetoken/models/gemma4/gguf.py+++ b/python/freetoken/models/gemma4/gguf.py@@ -13,8 +13,9 @@ routed experts load as GGUF expert banks (``freetoken.models.gguf.experts``).from __future__ import annotationsimport re-from typing import TYPE_CHECKING, Iterator+from typing import TYPE_CHECKING, Any, Iterator+import numpy as npimport torchfrom freetoken.models.config import (@@ -23,9 +24,12 @@ from freetoken.models.config import (RotaryConfig,SWAAttentionGroupConfig,)-from freetoken.models.gguf.reader import gguf_tensor_index+from freetoken.models.gguf.mmproj import Mmproj, find_mmproj, log_mmproj, mmproj_tensor+from freetoken.models.gguf.reader import gguf_tensor_index, load_gguf_metadatafrom freetoken.models.gguf.tensors import check_scales, float_tensor, fused, matrix, weight_scale+from .config import _parse_vision_config+if TYPE_CHECKING:from freetoken.models.gguf.config import GgufConfigShim@@ -47,6 +51,162 @@ def _full_rotary_dim(shim: "GgufConfigShim", full_head_dim: int) -> int:return full_head_dim // 4+# --------------------------------------------------------------------------------------+# Vision: the tower from a llama.cpp mmproj file. gemma4v is the ViT of the 26B-A4B and 31B releases, gemma4uv the linear embedder of the 12B "unified" release.+# --------------------------------------------------------------------------------------++_VIT, _UNIFIED = "gemma4v", "gemma4uv"+_TOKENS = {"image_token_id": "<|image|>", "boi_token_id": "<|image>", "eoi_token_id": "<image|>"}+# llama.cpp hardcodes these and the mmproj does not store them; the values match the HF configs+_POOLING_KERNEL = 3+_ROPE_THETA = 100.0+_MAX_SOFT_TOKENS = 280++_VIT_TOP = {+ "v.position_embd.weight": "vision_tower.patch_embedder.position_embedding_table",+ "v.std_bias": "vision_tower.std_bias",+ "v.std_scale": "vision_tower.std_scale",+ "mm.input_projection.weight": "embed_vision.embedding_projection.weight",+}+_VIT_BLOCK = {+ "attn_q": "self_attn.q_proj",+ "attn_k": "self_attn.k_proj",+ "attn_v": "self_attn.v_proj",+ "attn_out": "self_attn.o_proj",+ "attn_q_norm": "self_attn.q_norm",+ "attn_k_norm": "self_attn.k_norm",+ "ln1": "input_layernorm",+ "attn_post_norm": "post_attention_layernorm",+ "ln2": "pre_feedforward_layernorm",+ "ffn_post_norm": "post_feedforward_layernorm",+ "ffn_gate": "mlp.gate_proj",+ "ffn_up": "mlp.up_proj",+ "ffn_down": "mlp.down_proj",+}+_VIT_BLOCK_RE = re.compile(r"^v\.blk\.(\d+)\.(\w+)\.weight$")+# tensors over the patch features, stored channel-major+_UNIFIED_PATCH = {+ "v.patch_embd.weight": "vision_embedder.patch_dense.weight",+ "v.patch_norm.1.weight": "vision_embedder.patch_ln1.weight",+ "v.patch_norm.1.bias": "vision_embedder.patch_ln1.bias",+}+_UNIFIED_TOP = {+ "v.patch_embd.bias": "vision_embedder.patch_dense.bias",+ "v.patch_norm.2.weight": "vision_embedder.patch_ln2.weight",+ "v.patch_norm.2.bias": "vision_embedder.patch_ln2.bias",+ "v.patch_norm.3.weight": "vision_embedder.pos_norm.weight",+ "v.patch_norm.3.bias": "vision_embedder.pos_norm.bias",+ "mm.input_projection.weight": "embed_vision.embedding_projection.weight",+}+++def _f32_value(v: float) -> float:+ """The shortest decimal of a float32 metadata value: 1e-06, as the HF config writes it, not 9.99999997e-07."""+ return float(str(np.float32(v)))+++def gguf_mm_config(text_md: dict[str, Any], mmproj: Mmproj, vocab_size: int) -> tuple[dict[str, Any], dict[str, Any]]:+ """The HF config.json keys and the preprocessor_config.json of the tower in ``mmproj``, for the text GGUF whose metadata is ``text_md``."""+ if mmproj.projector not in (_VIT, _UNIFIED):+ raise ValueError(f"mmproj {mmproj.path} is a {mmproj.projector!r} tower; Gemma 4 needs {_VIT!r} or {_UNIFIED!r}")+ md, index = load_gguf_metadata(mmproj.path), gguf_tensor_index(mmproj.path)++ def kv(key: str):+ return md[f"clip.vision.{key}"]++ text_hidden = int(text_md["gemma4.embedding_length"])+ if int(kv("projection_dim")) != text_hidden:+ raise ValueError(f"mmproj {mmproj.path} projects to {int(kv('projection_dim'))} but the text model is {text_hidden} wide")+ tokens = text_md["tokenizer.ggml.tokens"]+ missing = [t for t in _TOKENS.values() if t not in tokens]+ if missing:+ raise ValueError(f"the text GGUF tokenizer has no {missing} tokens")+ patch_size = int(kv("patch_size"))+ eps = _f32_value(kv("attention.layer_norm_epsilon"))+ if mmproj.projector == _VIT:+ head_dim = int(index["v.blk.0.attn_q_norm.weight"].ne[0])+ vision_config = {+ "hidden_size": int(kv("embedding_length")),+ "num_hidden_layers": int(kv("block_count")),+ "num_attention_heads": int(kv("attention.head_count")),+ "num_key_value_heads": int(index["v.blk.0.attn_k.weight"].ne[1]) // head_dim,+ "head_dim": head_dim,+ "intermediate_size": int(kv("feed_forward_length")),+ "patch_size": patch_size,+ # ggml order [hidden, table, axis]+ "position_embedding_size": int(index["v.position_embd.weight"].ne[1]),+ "pooling_kernel_size": _POOLING_KERNEL,+ "rms_norm_eps": eps,+ "rope_parameters": {"rope_theta": _ROPE_THETA},+ "hidden_activation": "gelu_pytorch_tanh",+ "standardize": "v.std_bias" in index,+ }+ processor_type = "Gemma4ImageProcessor"+ else:+ # ggml order [patch_dim, hidden]; patch_dim = 3 * model_patch_size**2+ patch_dim = int(index["v.patch_embd.weight"].ne[0])+ vision_config = {+ "mm_embed_dim": int(index["v.patch_embd.weight"].ne[1]),+ "model_patch_size": round((patch_dim // 3) ** 0.5),+ "mm_posemb_size": int(index["v.position_embd.weight"].ne[1]),+ "patch_size": patch_size,+ "pooling_kernel_size": _POOLING_KERNEL,+ "rms_norm_eps": eps,+ }+ processor_type = "Gemma4UnifiedImageProcessor"+ hf_fields = {"vision_config": vision_config, **{key: tokens.index(t) for key, t in _TOKENS.items()}}+ preprocessor = {+ "image_processor_type": processor_type,+ "patch_size": patch_size,+ "pooling_kernel_size": _POOLING_KERNEL,+ "max_soft_tokens": _MAX_SOFT_TOKENS,+ "image_mean": [float(v) for v in kv("image_mean")],+ "image_std": [float(v) for v in kv("image_std")],+ "resample": 3,+ "do_convert_rgb": True,+ "do_resize": True,+ "do_rescale": True,+ "rescale_factor": 1 / 255,+ "do_normalize": False,+ }+ return hf_fields, preprocessor+++def _hwc(t: torch.Tensor, patch: int) -> torch.Tensor:+ """Patch features from llama.cpp's channel-major order back to the HF processor's (y, x, channel) order, along the last dim."""+ lead = t.shape[:-1]+ return t.reshape(*lead, 3, patch, patch).movedim(-3, -1).reshape(*lead, -1)+++def iter_mmproj_weights(mmproj: Mmproj) -> Iterator[tuple[str, torch.Tensor]]:+ """The tower state dict, named as the HF checkpoints load; audio tensors of a unified mmproj are skipped."""+ index = gguf_tensor_index(mmproj.path)+ for name, info in index.items():+ if name.startswith(("a.", "mm.a.")):+ continue+ t = mmproj_tensor(info)+ if mmproj.projector == _VIT:+ if name == "v.patch_embd.weight":+ # [hidden, 3, p, p] -> [hidden, p * p * 3]+ yield "vision_tower.patch_embedder.input_proj.weight", _hwc(t.flatten(1), t.shape[-1])+ elif name in _VIT_TOP:+ yield _VIT_TOP[name], t+ elif (m := _VIT_BLOCK_RE.match(name)) and m[2] in _VIT_BLOCK:+ yield f"vision_tower.encoder.layers.{m[1]}.{_VIT_BLOCK[m[2]]}.weight", t+ else:+ raise ValueError(f"unexpected tensor {name!r} in a {_VIT} mmproj")+ continue+ if name in _UNIFIED_PATCH:+ yield _UNIFIED_PATCH[name], _hwc(t, round((t.shape[-1] // 3) ** 0.5))+ elif name == "v.position_embd.weight":+ # [axis, table, hidden] -> [table, axis, hidden]+ yield "vision_embedder.pos_embedding", t.transpose(0, 1)+ elif name in _UNIFIED_TOP:+ yield _UNIFIED_TOP[name], t+ else:+ raise ValueError(f"unexpected tensor {name!r} in a {_UNIFIED} mmproj")++def parse_gguf_config(shim: "GgufConfigShim") -> ModelConfig:check_scales(shim.model_path, known=("ffn_gate_inp.scale", "ffn_down_exps.scale"))m = shim.metadata@@ -68,6 +228,7 @@ def parse_gguf_config(shim: "GgufConfigShim") -> ModelConfig:swa_pattern = [bool(x) for x in g("attention.sliding_window_pattern")]assert len(swa_pattern) == num_layers, "sliding_window_pattern length != block_count"+ vision_config = _parse_vision_config(shim, hidden)swa_layer_ids = tuple(i for i, is_swa in enumerate(swa_pattern) if is_swa)full_layer_ids = tuple(i for i, is_swa in enumerate(swa_pattern) if not is_swa)@@ -120,6 +281,8 @@ def parse_gguf_config(shim: "GgufConfigShim") -> ModelConfig:attn_sm_scale=1.0,final_logit_softcapping=float(g("final_logit_softcapping")),embedding_scale=float(hidden) ** 0.5,+ vision_config=vision_config,+ image_token_id=getattr(shim, "image_token_id", None),attention_groups=(FullAttentionGroupConfig(name="full",@@ -137,6 +300,8 @@ def parse_gguf_config(shim: "GgufConfigShim") -> ModelConfig:head_dim=swa_head_dim,rotary_config=swa_rotary,sliding_window=int(g("attention.sliding_window")),+ # every Gemma 4 vision release sets use_bidirectional_attention="vision"+ bidirectional_mm_blocks=vision_config is not None,),),)@@ -221,6 +386,7 @@ def iter_gguf_weights(*,include_moe_experts: bool,include_non_moe: bool,+ include_vision: bool = True,) -> Iterator[tuple[str, torch.Tensor]]:"""The dense state dict from the GGUF; the routed experts load as GGUF expert banks (``freetoken.models.gguf.experts``)."""if not include_non_moe:@@ -229,6 +395,8 @@ def iter_gguf_weights(_require_tp1("weight loading")config = parse_gguf_config(cached_load_hf_config(model_path))+ vision = include_vision and config.is_multimodal+ log_mmproj(model_path, vision)index = gguf_tensor_index(model_path)hidden = config.hidden_size@@ -264,8 +432,13 @@ def iter_gguf_weights(yield from fused(f"{base}.feed_forward.shared_mlp.gate_up_proj", [gate, up], [matrix(gate), matrix(up)], hidden)yield from fused(f"{base}.feed_forward.shared_mlp.down_proj", [down], [matrix(down)], int(down.ne[0]))+ if vision:+ yield from iter_mmproj_weights(find_mmproj(model_path))+__all__ = [+ "gguf_mm_config",+ "iter_mmproj_weights","parse_gguf_config","iter_gguf_weights","to_gguf",diff --git a/python/freetoken/models/gguf/config.py b/python/freetoken/models/gguf/config.pyindex 214fb51b..eb859abe 100644--- a/python/freetoken/models/gguf/config.py+++ b/python/freetoken/models/gguf/config.py@@ -9,9 +9,12 @@ the raw GGUF metadata dict, and a few derived facts that need the tensor tablefrom __future__ import annotations-from dataclasses import dataclass+import functools+import importlib+from dataclasses import dataclass, fieldfrom typing import Any+from .mmproj import Mmproj, find_mmproj, warn_inline_visionfrom .reader import gguf_architecture, load_gguf_metadata, gguf_tensor_names# GGUF ``general.architecture`` -> FreeToken registry key (a GGUF-specific spec that@@ -34,9 +37,13 @@ GGUF_ARCH_TO_REGISTRY: dict[str, str] = {"qwen35moe": "Qwen3_5MoeGGUFForCausalLM","qwen4exp": "Qwen4ExpGGUFForCausalLM",}+# (GGUF architecture, mmproj projector) -> registry key, for text GGUFs whose release is told apart only by its mmproj+GGUF_MMPROJ_TO_REGISTRY: dict[tuple[str, str], str] = {+ ("gemma4", "gemma4uv"): "Gemma4UnifiedGGUFForCausalLM",+}-@dataclass(frozen=True)+@dataclassclass GgufConfigShim:architectures: list[str]model_path: str@@ -44,6 +51,16 @@ class GgufConfigShim:metadata: dict[str, Any]vocab_size: inttie_word_embeddings: bool+ # HF config.json keys the family derives for its encoders (vision_config, image_token_id, ...), read as attributes like a HF config+ hf_fields: dict[str, Any] = field(default_factory=dict)+ # the encoder's preprocessor_config.json; None without an encoder+ preprocessor_config: dict[str, Any] | None = None++ def __getattr__(self, name: str) -> Any:+ from freetoken.utils.hf import RawConfigShim++ # __dict__ lookup: copy.copy probes attributes before the fields exist+ return getattr(RawConfigShim(self.__dict__.get("hf_fields") or {}), name)def to_dict(self) -> dict[str, Any]:"""Minimal HF-config-like dict for trunk code that introspects the config@@ -55,6 +72,7 @@ class GgufConfigShim:"torch_dtype": "bfloat16","vocab_size": self.vocab_size,"tie_word_embeddings": self.tie_word_embeddings,+ **self.hf_fields,}@@ -73,6 +91,19 @@ def _vocab_size(model_path: str) -> int:raise ValueError(f"GGUF {model_path}: no token_embd.weight to size the vocab")+def _mm_fields(module: str, model_path: str, mmproj: Mmproj, vocab_size: int) -> tuple[dict[str, Any], dict[str, Any] | None]:+ return importlib.import_module(module).gguf_mm_config(load_gguf_metadata(model_path), mmproj, vocab_size)+++def mm_fields(model_path: str, mmproj: Mmproj) -> tuple[dict[str, Any], dict[str, Any] | None]:+ """The HF config keys and the preprocessor config that the family of the text GGUF ``model_path`` derives from ``mmproj``."""+ from freetoken.models.register import get_model_spec++ spec = get_model_spec(GGUF_ARCH_TO_REGISTRY[gguf_architecture(model_path)])+ return _mm_fields(spec.module, model_path, mmproj, _vocab_size(model_path))++def build_gguf_shim(model_path: str) -> GgufConfigShim:arch = gguf_architecture(model_path)registry_key = GGUF_ARCH_TO_REGISTRY.get(arch)@@ -82,6 +113,7 @@ def build_gguf_shim(model_path: str) -> GgufConfigShim:f"(known: {sorted(GGUF_ARCH_TO_REGISTRY)})")names = gguf_tensor_names(model_path)+ warn_inline_vision(model_path, names)metadata = load_gguf_metadata(model_path)if names:# No separate output projection -> embeddings are tied.@@ -98,14 +130,24 @@ def build_gguf_shim(model_path: str) -> GgufConfigShim:"reconvert the checkpoint with the current freetoken.checkpoint.convert")tie_word_embeddings = not present+ from freetoken.models.register import get_model_spec++ vocab_size = _vocab_size(model_path)+ spec = get_model_spec(registry_key)+ mmproj = find_mmproj(model_path) if spec.encoders else None+ if mmproj is not None:+ registry_key = GGUF_MMPROJ_TO_REGISTRY.get((arch, mmproj.projector), registry_key)+ hf_fields, preprocessor_config = ({}, None) if mmproj is None else mm_fields(model_path, mmproj)return GgufConfigShim(architectures=[registry_key],model_path=model_path,model_type=arch,metadata=metadata,- vocab_size=_vocab_size(model_path),+ vocab_size=vocab_size,tie_word_embeddings=tie_word_embeddings,+ hf_fields=hf_fields,+ preprocessor_config=preprocessor_config,)-__all__ = ["GgufConfigShim", "GGUF_ARCH_TO_REGISTRY", "build_gguf_shim"]+__all__ = ["GgufConfigShim", "GGUF_ARCH_TO_REGISTRY", "GGUF_MMPROJ_TO_REGISTRY", "build_gguf_shim"]diff --git a/python/freetoken/models/gguf/mmproj.py b/python/freetoken/models/gguf/mmproj.pynew file mode 100644index 00000000..a8582816--- /dev/null+++ b/python/freetoken/models/gguf/mmproj.py@@ -0,0 +1,179 @@+"""llama.cpp vision towers (``mmproj-*.gguf``): find the file next to a text GGUF, read it, and dequantize its tensors.++llama.cpp keeps the tower in its own file (``general.architecture = clip``). FreeToken loads it from ``--mmproj`` (passed to every process as ``FREETOKEN_MMPROJ``) or finds a ``*mmproj*.gguf`` that fits the text model in the model's own folder, as llama-server does for a ``--models-dir`` entry. Every tensor loads as a float tensor: the towers run in bf16 whatever the file stores.+"""++from __future__ import annotations++import functools+import glob+import os+import re+from dataclasses import dataclass++import numpy as np+import torch++from freetoken.layers.quantization.ggml_types import is_quantized+from freetoken.utils.logger import init_logger++from .reader import FTW_METADATA_GGUF, GgufTensorInfo, gguf_tensor_index, is_gguf_path, load_gguf_metadata+from .tensors import float_tensor++logger = init_logger(__name__)++MMPROJ_ENV = "FREETOKEN_MMPROJ"++# BF16 and F32 convert to bf16 exactly, F16 loses only its subnormals, the rest is lossy in the file+_TYPE_RANK = {"BF16": 0, "F32": 1, "F16": 2}+++@dataclass(frozen=True)+class Mmproj:+ path: str+ source: str # "--mmproj" or "auto-discovered"+ projector: str+ weight_type: str # ggml type of the block weights (largest matrix without blocks), e.g. "BF16", "Q8_0"+ skipped: tuple[str, ...] = ()+++def _header(path: str):+ from freetoken.third_party import gguf++ # uncached: candidates that are not picked must not stay mapped in this process+ return gguf.GGUFReader(path)+++def _field(reader, name: str):+ field = reader.fields.get(name)+ return None if field is None else field.contents()+++def _model_key(name: str | None) -> str | None:+ """A model name compared across converters: ``unsloth/Qwen3.6-35B-A3B`` and ``Qwen3.6-35B-A3B`` are one model."""+ if name is None:+ return None+ return re.sub(r"[^0-9a-z]", "", str(name).rsplit("/", 1)[-1].lower())+++def _describe(path: str) -> tuple[str, str, int, str | None] | None:+ """(projector type, block weight type, output width, model name) of a vision mmproj, or None for any other GGUF."""+ from freetoken.third_party.gguf import GGMLQuantizationType++ reader = _header(path)+ if _field(reader, "general.architecture") != "clip" or not _field(reader, "clip.has_vision_encoder"):+ return None+ projector = _field(reader, "clip.vision.projector_type") or _field(reader, "clip.projector_type")+ if projector is None:+ return None+ types = {t.name: t.tensor_type for t in reader.tensors}+ weight = types.get("v.blk.0.attn_qkv.weight", types.get("v.blk.0.attn_q.weight"))+ if weight is None and reader.tensors:+ # no blocks: the largest matrix; converters keep patch embeddings F32 in every file type, so F32 ranks last+ weight = max(reader.tensors, key=lambda t: (t.tensor_type != GGMLQuantizationType.F32, len(t.shape) >= 2, int(t.n_elements))).tensor_type+ width = int(_field(reader, "clip.vision.projection_dim") or 0)+ name = _model_key(_field(reader, "general.basename") or _field(reader, "general.name"))+ return str(projector), "unknown" if weight is None else GGMLQuantizationType(weight).name, width, name+++def _find(model_path: str, override: str) -> Mmproj | None:+ if override:+ described = _describe(override) if is_gguf_path(override) else None+ if described is None:+ raise ValueError(f"--mmproj {override!r} is not a llama.cpp vision mmproj GGUF")+ return Mmproj(override, "--mmproj", *described[:2])+ md = load_gguf_metadata(model_path)+ width = md.get(f"{md.get('general.architecture')}.embedding_length")+ folder = os.path.dirname(os.path.abspath(model_path))+ found = []+ for path in sorted(glob.glob(os.path.join(glob.escape(folder), "*mmproj*.gguf"))):+ try:+ described = _describe(path)+ except Exception as e: # noqa: BLE001 -- a broken download next to the model must not stop it from serving text+ logger.warning("vision: skipping unreadable %s: %s", path, e)+ continue+ if described is not None and (width is None or described[2] == int(width)):+ found.append((_TYPE_RANK.get(described[1], len(_TYPE_RANK)), path, described))+ # same-width towers of several models in one folder: take the one named like the text model, else do not guess+ names = {f[2][3] for f in found}+ if len(names) > 1:+ text_name = _model_key(md.get("general.basename") or md.get("general.name"))+ found = [f for f in found if text_name is not None and f[2][3] == text_name]+ if not found:+ logger.warning("vision: %s holds mmproj files of several models (%s); pick one with --mmproj", folder, ", ".join(sorted(map(str, names))))+ return None+ from .config import mm_fields++ found.sort(key=lambda f: f[:2])+ for i, (_, path, described) in enumerate(found):+ mmproj = Mmproj(path, "auto-discovered", *described[:2], skipped=tuple(f[1] for f in found[i + 1 :]))+ try:+ # the family's own checks (tower type, image tokens); a file that fails them leaves the model text only+ mm_fields(model_path, mmproj)+ except (KeyError, ValueError) as e:+ logger.warning("vision: skipping %s: %s", path, e)+ continue+ return mmproj+ return None+++def find_mmproj(model_path: str) -> Mmproj | None:+ """The vision mmproj that goes with the text GGUF ``model_path``, or None when it has none."""+ # an FTW converted from a GGUF keeps only the text metadata, and no tower+ if not is_gguf_path(model_path) or os.path.basename(model_path) == FTW_METADATA_GGUF:+ return None+ return _find(model_path, os.environ.get(MMPROJ_ENV, ""))+++def _warn_inline_vision(model_path: str) -> None:+ logger.warning("vision: %s carries vision tensors itself (Ollama layout), which are not read; use a llama.cpp mmproj file", model_path)+++def warn_inline_vision(model_path: str, tensor_names: set[str]) -> None:+ """Old Ollama GGUFs carry the tower inside the text file under another layout; FreeToken reads only llama.cpp mmproj files."""+ if any(n.startswith(("v.", "mm.")) for n in tensor_names):+ _warn_inline_vision(model_path)+++def mmproj_tensor(info: GgufTensorInfo) -> torch.Tensor:+ """A tensor in torch order: float types as stored, quantized types dequantized to fp32."""+ if not is_quantized(info.ggml_type):+ return float_tensor(info)+ from freetoken.third_party.gguf import GGMLQuantizationType, quants++ flat = quants.dequantize(np.array(info.data()), GGMLQuantizationType(info.ggml_type))+ return torch.from_numpy(np.ascontiguousarray(flat)).reshape(*reversed(info.ne))+++def log_mmproj(model_path: str, enabled: bool) -> None:+ """One line per process on the vision tower the text GGUF ``model_path`` loads, or why it loads none."""+ mmproj = find_mmproj(model_path)+ if mmproj is None:+ logger.info("vision: no mmproj found for %s, vision disabled", model_path)+ return+ if not enabled:+ logger.info("vision: mmproj %s ignored (--mm-disable)", mmproj.path)+ return+ tensors = gguf_tensor_index(mmproj.path)+ gib = sum(i.nbytes for i in tensors.values()) / (1 << 30)+ logger.info(+ "vision: mmproj=%s (%s) projector=%s type=%s -> bf16, %d tensors, %.2f GiB",+ mmproj.path, mmproj.source, mmproj.projector, mmproj.weight_type, len(tensors), gib,+ )+ if mmproj.skipped:+ logger.info("vision: other mmproj files not used: %s (pick one with --mmproj)", ", ".join(mmproj.skipped))+ if mmproj.weight_type not in _TYPE_RANK:+ logger.warning("vision: mmproj %s is %s; it runs in bf16 but keeps the quantization error of the file", mmproj.path, mmproj.weight_type)+++__all__ = [+ "MMPROJ_ENV",+ "Mmproj",+ "find_mmproj",+ "log_mmproj",+ "mmproj_tensor",+ "warn_inline_vision",+]diff --git a/python/freetoken/models/qwen3_5_moe/__init__.py b/python/freetoken/models/qwen3_5_moe/__init__.pyindex abe8fd1c..24bb6dd3 100644--- a/python/freetoken/models/qwen3_5_moe/__init__.py+++ b/python/freetoken/models/qwen3_5_moe/__init__.py@@ -1,5 +1,5 @@from .config import parse_config-from .gguf import iter_gguf_weights, parse_gguf_config, to_gguf+from .gguf import gguf_mm_config, iter_gguf_weights, parse_gguf_config, to_gguffrom .model import (Qwen3_5ForCausalLM,Qwen3_5ForConditionalGeneration,@@ -19,6 +19,7 @@ __all__ = ["iter_weights_parallel","iter_expert_pieces","nvfp4_expert_spec",+ "gguf_mm_config","iter_gguf_weights","parse_gguf_config","to_gguf",diff --git a/python/freetoken/models/qwen3_5_moe/gguf.py b/python/freetoken/models/qwen3_5_moe/gguf.pyindex caf12680..2b80e57f 100644--- a/python/freetoken/models/qwen3_5_moe/gguf.py+++ b/python/freetoken/models/qwen3_5_moe/gguf.py@@ -13,8 +13,10 @@ from typing import TYPE_CHECKING, Iteratorimport torchfrom freetoken.layers.quantization.ggml_types import is_quantized+from freetoken.models.gguf.mmproj import find_mmproj, log_mmprojfrom freetoken.models.gguf.reader import GgufTensorInfo, gguf_tensor_indexfrom freetoken.models.gguf.tensors import check_scales, float_tensor, fused, matrix+from freetoken.models.qwen3_vl.gguf import gguf_mm_config, iter_mmproj_weightsfrom .config import parse_config@@ -123,7 +125,8 @@ def parse_gguf_config(shim: GgufConfigShim) -> ModelConfig:text_config=text,model_type="qwen3_5_moe" if moe else "qwen3_5",architectures=list(shim.architectures),- vision_config=None,+ vision_config=getattr(shim, "vision_config", None),+ image_token_id=getattr(shim, "image_token_id", None),)config = parse_config(hf_like)return replace(config, expert_quant="gguf") if moe else config@@ -155,6 +158,8 @@ def iter_gguf_weights(from freetoken.utils import cached_load_hf_configconfig = parse_gguf_config(cached_load_hf_config(model_path))+ vision = include_vision and config.is_multimodal+ log_mmproj(model_path, vision)index = gguf_tensor_index(model_path)g = config.linear_attention_group()hidden = config.hidden_size@@ -217,5 +222,8 @@ def iter_gguf_weights(yield from fused(f"{mlp}.gate_up_proj", [fg, fu], [matrix(fg), matrix(fu)], hidden)yield from fused(f"{mlp}.down_proj", [fd], [matrix(fd)], config.intermediate_size)+ if vision:+ yield from iter_mmproj_weights(find_mmproj(model_path))-__all__ = ["iter_gguf_weights", "parse_gguf_config", "to_gguf"]++__all__ = ["gguf_mm_config", "iter_gguf_weights", "parse_gguf_config", "to_gguf"]diff --git a/python/freetoken/models/qwen3_vl/gguf.py b/python/freetoken/models/qwen3_vl/gguf.pynew file mode 100644index 00000000..42c40c2e--- /dev/null+++ b/python/freetoken/models/qwen3_vl/gguf.py@@ -0,0 +1,114 @@+"""The Qwen VL vision tower from a llama.cpp ``qwen3vl_merger`` mmproj: the HF config keys, the image processor config, and the tower weights.++llama.cpp splits the Conv3d patch embedding into one tensor per temporal slice (``v.patch_embd.weight`` and ``.weight.1``); everything else is stored in HF layout.+"""++from __future__ import annotations++import re+from typing import Any, Iterator++import torch++from freetoken.models.gguf.mmproj import Mmproj, mmproj_tensor+from freetoken.models.gguf.reader import gguf_tensor_index, load_gguf_metadata++_PROJECTOR = "qwen3vl_merger"+_IMAGE_TOKEN = "<|image_pad|>"+# preprocessor_config.json of the HF releases; the mmproj does not store the pixel limits+_IMAGE_SIZE = {"shortest_edge": 65536, "longest_edge": 16777216}++_TOP = {+ "v.patch_embd.bias": "patch_embed.proj.bias",+ "v.position_embd.weight": "pos_embed.weight",+ "v.post_ln.weight": "merger.norm.weight",+ "v.post_ln.bias": "merger.norm.bias",+ "mm.0.weight": "merger.linear_fc1.weight",+ "mm.0.bias": "merger.linear_fc1.bias",+ "mm.2.weight": "merger.linear_fc2.weight",+ "mm.2.bias": "merger.linear_fc2.bias",+}+_BLOCK = {+ "attn_qkv": "attn.qkv",+ "attn_out": "attn.proj",+ "ffn_up": "mlp.linear_fc1",+ "ffn_down": "mlp.linear_fc2",+ "ln1": "norm1",+ "ln2": "norm2",+}+_BLOCK_RE = re.compile(r"^v\.blk\.(\d+)\.(\w+)\.(weight|bias)$")+_PATCH = ("v.patch_embd.weight", "v.patch_embd.weight.1")+++def gguf_mm_config(text_md: dict[str, Any], mmproj: Mmproj, vocab_size: int) -> tuple[dict[str, Any], dict[str, Any]]:+ """The HF config.json keys and the preprocessor_config.json of the tower in ``mmproj``, for the text GGUF whose metadata is ``text_md``."""+ if mmproj.projector != _PROJECTOR:+ raise ValueError(f"mmproj {mmproj.path} is a {mmproj.projector!r} tower; Qwen VL needs {_PROJECTOR!r}")+ md, index = load_gguf_metadata(mmproj.path), gguf_tensor_index(mmproj.path)+ if any(md.get("clip.vision.is_deepstack_layers") or []):+ raise ValueError(f"mmproj {mmproj.path} is a Qwen3-VL tower with DeepStack layers; Qwen3.5 and later read none")++ def kv(key: str):+ return md[f"clip.vision.{key}"]++ arch = text_md["general.architecture"]+ text_hidden = int(text_md[f"{arch}.embedding_length"])+ if int(kv("projection_dim")) != text_hidden:+ raise ValueError(f"mmproj {mmproj.path} projects to {int(kv('projection_dim'))} but the text model is {text_hidden} wide")+ tokens = text_md["tokenizer.ggml.tokens"]+ if _IMAGE_TOKEN not in tokens:+ raise ValueError(f"the text GGUF tokenizer has no {_IMAGE_TOKEN} token")+ temporal = 2 if _PATCH[1] in index else 1+ vision_config = {+ "hidden_size": int(kv("embedding_length")),+ "depth": int(kv("block_count")),+ "num_heads": int(kv("attention.head_count")),+ "intermediate_size": int(kv("feed_forward_length")),+ "patch_size": int(kv("patch_size")),+ "temporal_patch_size": temporal,+ "spatial_merge_size": int(kv("spatial_merge_size")),+ "num_position_embeddings": int(index["v.position_embd.weight"].ne[1]),+ "out_hidden_size": text_hidden,+ # ggml order [kw, kh, in, out]+ "in_channels": int(index[_PATCH[0]].ne[2]),+ "deepstack_visual_indexes": [],+ }+ hf_fields = {+ "vision_config": vision_config,+ "image_token_id": tokens.index(_IMAGE_TOKEN),+ "text_config": {+ "vocab_size": vocab_size,+ "rope_parameters": {"mrope_section": [int(s) for s in text_md[f"{arch}.rope.dimension_sections"]][:3]},+ },+ }+ preprocessor = {+ "image_processor_type": "Qwen2VLImageProcessor",+ "size": dict(_IMAGE_SIZE),+ "patch_size": vision_config["patch_size"],+ "temporal_patch_size": temporal,+ "merge_size": vision_config["spatial_merge_size"],+ "image_mean": [float(v) for v in kv("image_mean")],+ "image_std": [float(v) for v in kv("image_std")],+ }+ return hf_fields, preprocessor+++def _name(gguf_name: str) -> str:+ if gguf_name in _TOP:+ return _TOP[gguf_name]+ if (m := _BLOCK_RE.match(gguf_name)) and m[2] in _BLOCK:+ return f"blocks.{m[1]}.{_BLOCK[m[2]]}.{m[3]}"+ raise ValueError(f"unexpected tensor {gguf_name!r} in a {_PROJECTOR} mmproj")+++def iter_mmproj_weights(mmproj: Mmproj) -> Iterator[tuple[str, torch.Tensor]]:+ """The tower state dict under ``visual.``, named as the HF checkpoints load."""+ index = gguf_tensor_index(mmproj.path)+ slices = [mmproj_tensor(index[n]) for n in _PATCH if n in index]+ yield "visual.patch_embed.proj.weight", torch.stack(slices, dim=2)+ for gguf_name, info in index.items():+ if gguf_name not in _PATCH:+ yield f"visual.{_name(gguf_name)}", mmproj_tensor(info)+++__all__ = ["gguf_mm_config", "iter_mmproj_weights"]diff --git a/python/freetoken/models/qwen4_exp/__init__.py b/python/freetoken/models/qwen4_exp/__init__.pyindex 15f866ca..0a97233b 100644--- a/python/freetoken/models/qwen4_exp/__init__.py+++ b/python/freetoken/models/qwen4_exp/__init__.py@@ -10,7 +10,7 @@ Contracts shared across modules (do not rename):"""from .config import parse_config-from .gguf import iter_gguf_weights, parse_gguf_config, to_gguf+from .gguf import gguf_mm_config, iter_gguf_weights, parse_gguf_config, to_gguffrom .model import Qwen4ExpForCausalLM, Qwen4ExpForConditionalGenerationfrom .weight import (ftw_side_files,@@ -34,6 +34,7 @@ __all__ = ["load_ple_table","parse_config","iter_expert_pieces",+ "gguf_mm_config","iter_gguf_weights","parse_gguf_config","to_gguf",diff --git a/python/freetoken/models/qwen4_exp/gguf.py b/python/freetoken/models/qwen4_exp/gguf.pyindex f29f6533..1ff181f5 100644--- a/python/freetoken/models/qwen4_exp/gguf.py+++ b/python/freetoken/models/qwen4_exp/gguf.py@@ -13,8 +13,10 @@ from typing import TYPE_CHECKING, Iteratorimport torchfrom freetoken.layers.quantization.ggml_types import is_quantized+from freetoken.models.gguf.mmproj import find_mmproj, log_mmprojfrom freetoken.models.gguf.reader import GgufTensorInfo, gguf_tensor_indexfrom freetoken.models.gguf.tensors import check_scales, float_tensor, fused, matrix+from freetoken.models.qwen3_vl.gguf import gguf_mm_config, iter_mmproj_weightsfrom .config import parse_config@@ -142,13 +144,16 @@ def parse_gguf_config(shim: GgufConfigShim) -> ModelConfig:heads_per_ngram=int(kv("ple.heads_per_ngram")),eos_token_id=int(kv("ple.eos_token_id")),)- image_token_id = md.get(f"{_ARCH}.ple.image_token_id")+ # with an mmproj, the id the image processor places; ple.image_token_id names the same token for text-only files+ image_token_id = getattr(shim, "image_token_id", None)+ if image_token_id is None:+ image_token_id = md.get(f"{_ARCH}.ple.image_token_id")hf_like = SimpleNamespace(text_config=text,model_type="qwen4_exp",architectures=list(shim.architectures),image_token_id=None if image_token_id is None else int(image_token_id),- vision_config=None,+ vision_config=getattr(shim, "vision_config", None),)config = parse_config(hf_like)return replace(config, expert_quant="gguf", qwen4_args=replace(config.qwen4_args, gdn_out_heads_tiled=True))@@ -180,6 +185,8 @@ def iter_gguf_weights(from freetoken.utils import cached_load_hf_configconfig = parse_gguf_config(cached_load_hf_config(model_path))+ vision = include_vision and config.is_multimodal+ log_mmproj(model_path, vision)index = gguf_tensor_index(model_path)md = cached_load_hf_config(model_path).metadatag = config.linear_attention_group()@@ -263,9 +270,12 @@ def iter_gguf_weights(yield f"{ple}.ple_embedding.ngram_heads_vocab_sizes", torch.tensor([int(x) for x in md[f"{_ARCH}.ple.head_vocab_sizes"]], dtype=torch.int64)yield f"{ple}.ple_embedding.ngram_heads_offsets", torch.tensor([int(x) for x in md[f"{_ARCH}.ple.head_offsets"]], dtype=torch.int64)+ if vision:+ yield from iter_mmproj_weights(find_mmproj(model_path))+def ple_table_info(model_path: str) -> GgufTensorInfo:return gguf_tensor_index(model_path)["per_layer_token_embd.weight"]-__all__ = ["iter_gguf_weights", "parse_gguf_config", "ple_table_info", "to_gguf"]+__all__ = ["gguf_mm_config", "iter_gguf_weights", "parse_gguf_config", "ple_table_info", "to_gguf"]diff --git a/python/freetoken/models/register.py b/python/freetoken/models/register.pyindex f4238668..bbe34f06 100644--- a/python/freetoken/models/register.py+++ b/python/freetoken/models/register.py@@ -266,24 +266,28 @@ _MODEL_REGISTRY: dict[str, ModelSpec] = {packed_modules_mapping=_QWEN3_5_PACKED,unquantized_modules=_QWEN3_5_UNQUANTIZED,),- # llama.cpp qwen35 / qwen35moe GGUF (text only): the same model classes with every tensor in its packed ggml type+ # llama.cpp qwen35 / qwen35moe GGUF: the same model classes with every tensor in its packed ggml type; the vision tower comes from an mmproj file and runs in bf16"Qwen3_5GGUFForCausalLM": ModelSpec("freetoken.models.qwen3_5_moe",- "Qwen3_5ForCausalLM",+ "Qwen3_5ForConditionalGeneration",parse_config="parse_gguf_config",iter_weights="iter_gguf_weights",checkpoint_roots=_LANGUAGE_MODEL_ROOT,packed_modules_mapping=_QWEN3_5_PACKED,unquantized_modules=_QWEN3_5_UNQUANTIZED,+ mm_processor=_QWEN_VL_PROCESSOR,+ encoders=_QWEN_VL_ENCODERS,),"Qwen3_5MoeGGUFForCausalLM": ModelSpec("freetoken.models.qwen3_5_moe",- "Qwen3_5MoeForCausalLM",+ "Qwen3_5MoeForConditionalGeneration",parse_config="parse_gguf_config",iter_weights="iter_gguf_weights",checkpoint_roots=_LANGUAGE_MODEL_ROOT,packed_modules_mapping=_QWEN3_5_PACKED,unquantized_modules=_QWEN3_5_UNQUANTIZED,+ mm_processor=_QWEN_VL_PROCESSOR,+ encoders=_QWEN_VL_ENCODERS,),# Qwen3.8-Flash-Next (model_type qwen4_exp): multimodal wrapper config (text tower in# text_config, weights under model.language_model.). 36 GDN + 12 QSA@@ -297,14 +301,16 @@ _MODEL_REGISTRY: dict[str, ModelSpec] = {encoders=_QWEN_VL_ENCODERS,packed_modules_mapping=_QWEN4_EXP_PACKED,),- # llama.cpp qwen4exp GGUF (text only): the same model classes with every tensor in its packed ggml type+ # llama.cpp qwen4exp GGUF: the same model classes with every tensor in its packed ggml type; the vision tower comes from an mmproj file and runs in bf16"Qwen4ExpGGUFForCausalLM": ModelSpec("freetoken.models.qwen4_exp",- "Qwen4ExpForCausalLM",+ "Qwen4ExpForConditionalGeneration",parse_config="parse_gguf_config",iter_weights="iter_gguf_weights",checkpoint_roots=_LANGUAGE_MODEL_ROOT,packed_modules_mapping=_QWEN4_EXP_GGUF_PACKED,+ mm_processor=_QWEN_VL_PROCESSOR,+ encoders=_QWEN_VL_ENCODERS,),# Dense Qwen3.x (no "Moe" in the arch name, num_experts==0, e.g. Qwen3.6-27B). Shares the# qwen3_5_moe package: the decoder routes its MLP through the dense Qwen3_5DenseMLP and the@@ -380,14 +386,27 @@ _MODEL_REGISTRY: dict[str, ModelSpec] = {checkpoint_segments=_GEMMA4_SEGMENTS,packed_modules_mapping=_GEMMA4_PACKED,),- # llama.cpp gemma4 GGUF (text only): the same model classes with every tensor in its packed ggml type+ # llama.cpp gemma4 GGUF: the same model classes with every tensor in its packed ggml type; the vision tower comes from an mmproj file and runs in bf16"Gemma4GGUFForCausalLM": ModelSpec("freetoken.models.gemma4",- "Gemma4ForCausalLM",+ "Gemma4ForConditionalGeneration",+ parse_config="parse_gguf_config",+ iter_weights="iter_gguf_weights",+ checkpoint_segments=_GEMMA4_SEGMENTS,+ packed_modules_mapping=_GEMMA4_PACKED,+ mm_processor=_GEMMA4_PROCESSOR,+ encoders=_GEMMA4_ENCODERS,+ ),+ # the same text GGUF with a gemma4uv mmproj (12B): the text GGUF does not tell the 12B apart, so build_gguf_shim picks this row from the projector+ "Gemma4UnifiedGGUFForCausalLM": ModelSpec(+ "freetoken.models.gemma4",+ "Gemma4UnifiedForConditionalGeneration",parse_config="parse_gguf_config",iter_weights="iter_gguf_weights",checkpoint_segments=_GEMMA4_SEGMENTS,packed_modules_mapping=_GEMMA4_PACKED,+ mm_processor=_GEMMA4_UNIFIED_PROCESSOR,+ encoders=_GEMMA4_ENCODERS,),"GptOssForCausalLM": ModelSpec("freetoken.models.gpt_oss",diff --git a/python/freetoken/server/args.py b/python/freetoken/server/args.pyindex cca78e63..efe3b8c0 100644--- a/python/freetoken/server/args.py+++ b/python/freetoken/server/args.py@@ -503,6 +503,15 @@ def parse_args(metavar="{vision,audio}",help="Encoder towers to leave unbuilt; every input they would serve is rejected.",)+ parser.add_argument(+ "--mmproj",+ type=str,+ default=None,+ metavar="FILE",+ help="llama.cpp vision tower (mmproj GGUF) for a GGUF model. Default: a *mmproj*.gguf that fits the model "+ "in the model file's folder (BF16 > F32 > F16 > quantized when several are there); none found, or files of "+ "several models, serves text only.",+ )parser.add_argument("--image-min-tokens",@@ -949,6 +958,17 @@ def parse_args(if kwargs["model_path"].startswith("~"):kwargs["model_path"] = os.path.expanduser(kwargs["model_path"])+ # every process reads the GGUF config, tokenizer workers included, so the choice travels in the environment+ from freetoken.models.gguf.mmproj import MMPROJ_ENV++ os.environ.pop(MMPROJ_ENV, None)+ if (mmproj := kwargs.pop("mmproj")) is not None:+ from freetoken.models.gguf.reader import is_gguf_path++ if not is_gguf_path(kwargs["model_path"]):+ parser.error("--mmproj needs a .gguf model file")+ os.environ[MMPROJ_ENV] = os.path.abspath(os.path.expanduser(mmproj))+# a bad media root is a deployment mistake; fail at startup, not per requestif kwargs["allowed_local_media_path"]:media_root = os.path.realpath(os.path.expanduser(kwargs["allowed_local_media_path"]))diff --git a/python/freetoken/utils/__init__.py b/python/freetoken/utils/__init__.pyindex af68bf58..b435628c 100644--- a/python/freetoken/utils/__init__.py+++ b/python/freetoken/utils/__init__.py@@ -12,6 +12,7 @@ from .hf import (download_hf_weight,load_eos_token_ids,load_generation_sampling,+ load_image_processor,load_tokenizer,load_toolcall_anchor_id,)diff --git a/python/freetoken/utils/hf.py b/python/freetoken/utils/hf.pyindex 5a7b7e69..2a06ebbf 100644--- a/python/freetoken/utils/hf.py+++ b/python/freetoken/utils/hf.py@@ -47,6 +47,22 @@ def load_tokenizer(model_path: str) -> PreTrainedTokenizerBase:return tokenizer+def load_image_processor(model_path: str) -> Any:+ """The checkpoint's image processor; a GGUF builds it from the preprocessor_config its family derives."""+ from freetoken.models.gguf.reader import gguf_config_source++ if gguf_config_source(model_path) is not None:+ from transformers.models.auto.image_processing_auto import get_image_processor_class_from_name++ config = cached_load_hf_config(model_path).preprocessor_config+ if config is None:+ raise ValueError(f"{model_path}: GGUF has no vision mmproj to build an image processor from")+ return get_image_processor_class_from_name(config["image_processor_type"]).from_dict(config)+ from transformers import AutoImageProcessor++ return AutoImageProcessor.from_pretrained(model_path)++def load_eos_token_ids(model_path: str, tokenizer: PreTrainedTokenizerBase) -> FrozenSet[int]:diff --git a/scripts/gguf_check/README.md b/scripts/gguf_check/README.mdindex a1f369f7..8ec9a814 100644--- a/scripts/gguf_check/README.md+++ b/scripts/gguf_check/README.md@@ -7,7 +7,7 @@ Run from the repo root with `PYTHONPATH=python` and transformers 5.18.|---|---|---|| `fetch_headers.py` | network | nothing, writes header-only stubs of HF GGUF files || `probe_kernels.py` | GPU | which ggml ops this build runs per ggml type |-| `check.py` | CPU | config, dry load, expert banks, kernel coverage, tokenizer, chat template per file |+| `check.py` | CPU | config, dry load, expert banks, kernel coverage, tokenizer, chat template, and the vision tower of an mmproj next to the file, per file || `tok_compare.py` | CPU | GGUF tokenizer vs the model's HF `tokenizer.json` |## Coverage: most published files loaddiff --git a/scripts/gguf_check/check.py b/scripts/gguf_check/check.pyindex 390a52b4..69900889 100644--- a/scripts/gguf_check/check.py+++ b/scripts/gguf_check/check.py@@ -9,6 +9,7 @@ Per file, through the code the server runs:- dry load: the model builds on the meta device, and the family's weight reader yields exactly its state dict (names, shapes, packed vs float); the expert banks build for every MoE layer with the shapes the model needs- kernels: every quantized ggml type a GGUF linear, embedding or expert bank uses has the ops it needs in the ``probe_kernels.py`` table- tokenizer: it builds, the chat template renders, and the file names an EOS token+- vision: with an ``mmproj`` next to the file (or ``FREETOKEN_MMPROJ``), the tower config, its weights in the dry load, the image processor, and an image message rendering the processor's placeholderThe weight reader runs on meta tensors, so no weight bytes are read or copied. A reader that needs real values (a numpy dequant) gets a second run on the real bytes, once enough RAM is free.The files can be real or ``fetch_headers.py`` stubs. Passing does not prove the weights are right or that the model fits in memory.@@ -148,12 +149,14 @@ def _prod(dims) -> int:def _clear_caches() -> None:"""Drop the caches that hold a file's metadata, config or rope tables."""+ import freetoken.models.gguf.config as gguf_config+ import freetoken.models.gguf.mmproj as mmprojimport freetoken.models.gguf.reader as readerfrom freetoken.layers import rotaryfrom freetoken.utils import hffor fn in (reader._reader, reader.load_gguf_metadata, reader.gguf_split_paths, reader.gguf_tensor_index,- rotary.get_rope, rotary._cos_sin_table, hf._load_hf_config):+ rotary.get_rope, rotary._cos_sin_table, hf._load_hf_config, mmproj._find, mmproj.log_mmproj, gguf_config._mm_fields):fn.cache_clear()@@ -181,18 +184,17 @@ def _init_worker(kernels: dict | None, in_flight, min_free_gb: float) -> None:def _build_model(path: str):- """The engine's config and the model it builds for ``path`` (experts offloaded), on the meta device."""+ """The engine's config and the model it builds for ``path`` (experts offloaded, encoders the file serves), on the meta device."""import torchfrom freetoken.distributed import try_get_tp_infofrom freetoken.engine.config import EngineConfigfrom freetoken.engine.engine import _decode_target- from freetoken.mm.config import ENCODER_KINDS, MultimodalConfig+ from freetoken.mm.config import MultimodalConfigfrom freetoken.models import create_modelfrom freetoken.utils.torch_utils import torch_dtype- config = EngineConfig(model_path=path, tp_info=try_get_tp_info(), dtype=torch.bfloat16, moe_strategy="offload",- mm=MultimodalConfig(disabled_encoders=frozenset(ENCODER_KINDS)))+ config = EngineConfig(model_path=path, tp_info=try_get_tp_info(), dtype=torch.bfloat16, moe_strategy="offload", mm=MultimodalConfig())object.__setattr__(config.model_config, "moe_strategy", "offload")object.__setattr__(config.model_config, "decode_target", _decode_target(config))with torch.device("meta"), torch_dtype(torch.bfloat16):@@ -331,10 +333,12 @@ def _dry_load(path: str, config, model, *, meta: bool) -> list[str]:spec = config.model_specexpected = model.state_dict()problems, seen = [], set()+ # the same keyword the engine passes, only to families that register an encoder+ kwargs = {"include_vision": bool(config.active_encoders)} if spec.encoders else {}_swap_readers(meta)try:with _meta_wins()() if meta else contextlib.nullcontext():- for key, t in _load_attr(spec.module, spec.iter_weights)(path, torch.device("cpu"), include_moe_experts=False, include_non_moe=True):+ for key, t in _load_attr(spec.module, spec.iter_weights)(path, torch.device("cpu"), include_moe_experts=False, include_non_moe=True, **kwargs):ref = expected.get(key)if ref is None:problems.append(f"unexpected tensor {key}")@@ -398,6 +402,28 @@ def _check_tokenizer(path: str) -> dict:return {"problems": problems, "eos": eos, "template_tail": tail}+def _check_vision(path: str) -> tuple[dict, list[str]]:+ """The mmproj in use, and problems of the image processor and of an image message in the chat template."""+ from freetoken.mm.processor import get_mm_processor+ from freetoken.models.gguf.mmproj import find_mmproj+ from freetoken.models.gguf.tokenizer import load_gguf_tokenizer++ mm = find_mmproj(path)+ info = {"mmproj": mm.path, "source": mm.source, "projector": mm.projector, "type": mm.weight_type}+ problems = []+ processor = get_mm_processor(path)+ processor._image_processor()+ tok = load_gguf_tokenizer(path)+ if tok.chat_template is not None:+ message = [{"role": "user", "content": [{"type": "image"}, {"type": "text", "text": "hi"}]}]+ text = tok.apply_chat_template(message, tokenize=False, add_generation_prompt=True)+ ids = tok.encode(text, add_special_tokens=False)+ n = len(processor.placeholder)+ if not any(list(ids[i : i + n]) == processor.placeholder for i in range(len(ids) - n + 1)):+ problems.append(f"chat template: an image message renders no image placeholder {processor.placeholder}")+ return info, problems++def check(path: str) -> dict:from freetoken.models.gguf.config import GGUF_ARCH_TO_REGISTRYfrom freetoken.models.gguf.reader import gguf_architecture, load_gguf_metadata@@ -436,6 +462,9 @@ def check(path: str) -> dict:_TOKENIZERS[key] = {"problems": [f"tokenizer: {type(e).__name__}: {str(e)[:200]}"]}tok = _TOKENIZERS[key]problems.extend(tok["problems"])+ if config.active_encoders:+ out["vision"], vision_problems = _check_vision(path)+ problems.extend(vision_problems)out["tokenizer"] = {"hash": key, "pre": meta.get("tokenizer.ggml.pre"), "eos": tok.get("eos"), "template_tail": tok.get("template_tail")}except Exception as e:problems.append(f"{type(e).__name__}: {str(e)[:300]}")