Skip to content

FreeToken brings datacenter-scale model serving to your desktop. Run massive models locally, fast and efficiently.

feat(gemma4): serve image input on the Gemma-4 releases (#467)

Xiaoze Fan84d236c4c7cb
diff --git a/docs/cli.md b/docs/cli.md
index bf480d42..0c339439 100644
--- a/docs/cli.md
+++ b/docs/cli.md
@@ -101,8 +101,8 @@ See [models.md](models.md#moe-strategies) for what each strategy does.
### Image input
-Experimental. Needs a vision-capable checkpoint (Qwen3.6, Qwen3.8-Flash-Next, Qwen3-VL); a request carrying images is
-rejected otherwise. Images are accepted on all three protocols (OpenAI `image_url`,
+Experimental. Needs a checkpoint whose family registers a vision encoder ([models.md](models.md#image-input) lists them and how each one
+maps the flags below); a request carrying images is rejected otherwise. Images are accepted on all three protocols (OpenAI `image_url`,
Anthropic `image` blocks, Responses `input_image`) as an http(s) URL or base64. Images inside a tool
result (an Anthropic `tool_result` block from Claude Code's Read, a Responses `function_call_output`
from Codex's view_image) are moved to the user turn that follows the tool message, as vLLM does,
@@ -114,9 +114,9 @@ so a client can gate its attachment controls without reading the checkpoint conf
| --- | --- | --- |
| `--text-model-only` | off | Serve a multimodal checkpoint text-only: no encoder tower is built (its VRAM goes to the KV/expert pools) and every multimodal input is rejected. Same as `--mm-disable` with every encoder kind |
| `--mm-disable` | none | Encoder towers to leave unbuilt (`vision`, `audio`); every input they would serve is rejected |
-| `--mm-encoder-weights` | host | Where the encoder tower's block weights live. `host` streams them from pinned host banks two blocks at a time behind the compute (about 60 MiB of VRAM instead of the whole tower; small images encode slower, about 17 ms instead of 7 ms for 448x448); `gpu` keeps them resident |
-| `--image-min-tokens`, `--image-max-tokens` | processor defaults | Per-image token budget: the image processor resizes every image to take between these many tokens, converted to the family's own units (Qwen VL: one token per 32x32 pixels of the resized image, defaults 64 to 16384). Only families with dynamic resolution honor them |
-| `--mm-processor-kwargs` | none | JSON object of extra keyword arguments for the checkpoint's image processor call, for family-specific knobs (Qwen VL: `{"size": {"longest_edge": 1048576}}`); applied after the token budget |
+| `--mm-encoder-weights` | host | Where the encoder tower's block weights live. `host` streams them from pinned host banks two blocks at a time behind the compute, so the GPU holds two blocks instead of the whole tower; small images pay the copy time, large ones hide it behind the compute. `gpu` keeps them resident. An encoder without a block stack stays resident either way |
+| `--image-min-tokens`, `--image-max-tokens` | processor defaults | Per-image token budget: the image processor resizes every image to take between these many tokens, converted to the family's own units by its processor. A family with fixed budgets honors the maximum only and refuses one below its smallest budget at start-up |
+| `--mm-processor-kwargs` | none | JSON object of extra keyword arguments for the checkpoint's image processor call, for knobs the token budget does not cover; applied after the budget, so an explicit key wins |
| `--mm-embed-cache-device` | cpu | Where encoded image embeddings live between prefill chunks. `cpu` keeps them out of the VRAM budget; `cuda` skips the copy back |
| `--allowed-media-domains` | any | Comma-separated hostname allowlist for image URLs; requests for other domains are rejected with a 400. Empty allows any domain |
| `--allowed-local-media-path` | off | Directory `file://` image refs may be read from; unset rejects local files |
diff --git a/docs/models.md b/docs/models.md
index 0a9e214a..7f7f7b6d 100644
--- a/docs/models.md
+++ b/docs/models.md
@@ -19,8 +19,16 @@ for them; other checkpoints of the same architectures work too.
| MiniMax-M2.5 | [nvidia/MiniMax-M2.5-NVFP4](https://huggingface.co/nvidia/MiniMax-M2.5-NVFP4) |
| Muse-Glimmer | [meta-models/Muse-Glimmer-30B](https://huggingface.co/meta-models/Muse-Glimmer-30B), [RedHatAI/Muse-Glimmer-30B-NVFP4](https://huggingface.co/RedHatAI/Muse-Glimmer-30B-NVFP4) |
-Qwen3.6 (both variants, every listed weight format), Qwen3.8-Flash-Next and Qwen3-VL also accept image input by default;
-pass `--text-model-only` to skip the vision tower; see [CLI reference](cli.md#image-input).
+### Image input
+
+These families accept image input by default; pass `--text-model-only` to skip the vision encoder. The flags are described in the
+[CLI reference](cli.md#image-input); each family reads them in its own units.
+
+| Family | Image tokens | `--image-min-tokens` / `--image-max-tokens` | `--mm-processor-kwargs` example |
+| --- | --- | --- | --- |
+| Qwen3.6 (both variants, every listed weight format), Qwen3.8-Flash-Next, Qwen3-VL | one token per 32x32 pixels of the resized image, dynamic resolution | pixel areas in `size.shortest_edge` / `longest_edge`; checkpoint defaults 64 to 16384 tokens | `{"size": {"longest_edge": 1048576}}` |
+| Gemma-4 26B-A4B, 31B (`gemma4`: ViT tower, streamed under `--mm-encoder-weights host`) | one of the soft-token budgets 70 / 140 / 280 / 560 / 1120, every image scaled to its budget as far as the aspect ratio allows | the maximum picks the largest budget within it, below 70 is refused at start-up; the minimum has no effect | `{"max_soft_tokens": 1120}` |
+| Gemma-4 12B (`gemma4_unified`: linear patch embedder, resident under either placement) | same budgets, one 48x48 super-patch per soft token | same as the tower releases | same |
## MoE strategies
diff --git a/python/freetoken/attention/base.py b/python/freetoken/attention/base.py
index ca36fb88..9115b809 100644
--- a/python/freetoken/attention/base.py
+++ b/python/freetoken/attention/base.py
@@ -41,6 +41,8 @@ class AttentionSpec:
sliding_window: int | None = None
sm_scale: float | None = None
sinks: torch.Tensor | None = None
+ # rows of a multimodal span (batch.mm_block_ends) also attend to the span's later keys
+ bidirectional_mm_blocks: bool = False
@dataclass
diff --git a/python/freetoken/attention/triton.py b/python/freetoken/attention/triton.py
index 9eed1e1d..8bf9792d 100644
--- a/python/freetoken/attention/triton.py
+++ b/python/freetoken/attention/triton.py
@@ -157,6 +157,7 @@ class TritonAttentionBackend(BaseAttnBackend):
v_cache = v_raw.view(-1, kv_heads, head_dim)
spec = attn_spec or AttentionSpec()
+ block_ends = batch.mm_block_ends if spec.bidirectional_mm_blocks else None
indices = metadata.indices
if spec.sliding_window is not None and metadata.swa_indices is not None:
indices = metadata.swa_indices
@@ -201,7 +202,10 @@ class TritonAttentionBackend(BaseAttnBackend):
sinks=spec.sinks,
k_extend=k.view(q.shape[0], kv_heads, head_dim),
v_extend=v.view(q.shape[0], kv_heads, head_dim),
+ block_ends=block_ends,
)
+ if block_ends is not None:
+ raise NotImplementedError("bidirectional multimodal blocks need the extend kernel path")
return paged_attention(
q=q,
k_cache=k_cache,
diff --git a/python/freetoken/core.py b/python/freetoken/core.py
index 269b94ff..e07e442c 100644
--- a/python/freetoken/core.py
+++ b/python/freetoken/core.py
@@ -138,6 +138,8 @@ class Batch:
# concatenated multimodal soft-token embeddings for a prefill batch (or None) and the batch rows they land on
mm_embeds: torch.Tensor | None = field(default=None, init=False)
mm_rows: torch.Tensor | None = field(default=None, init=False)
+ # per batch token, the end (exclusive, in its request) of the image span holding it, 0 for text: the block a bidirectional layer attends within
+ mm_block_ends: torch.Tensor | None = field(default=None, init=False)
# this chunk's cache-miss items to encode and the gather plan [(uid, hash, row_lo, row_hi, n, pos), ...] in scatter order
mm_encoder_jobs: list | None = field(default=None, init=False)
mm_gather_plan: list | None = field(default=None, init=False)
diff --git a/python/freetoken/kernel/triton/attention.py b/python/freetoken/kernel/triton/attention.py
index c2358d84..3f53c5fc 100644
--- a/python/freetoken/kernel/triton/attention.py
+++ b/python/freetoken/kernel/triton/attention.py
@@ -476,6 +476,7 @@ def _extend_attention_kernel(
kv_indptr_ptr,
kv_indices_ptr,
prefix_lens_ptr,
+ block_ends_ptr,
sm_scale,
sinks_ptr,
stride_qt,
@@ -494,6 +495,7 @@ def _extend_attention_kernel(
BLOCK_N: tl.constexpr,
SLIDING_WINDOW: tl.constexpr,
HAS_SINKS: tl.constexpr,
+ HAS_BLOCKS: tl.constexpr,
):
seq_id = tl.program_id(0)
q_head = tl.program_id(1)
@@ -517,6 +519,12 @@ def _extend_attention_kernel(
q_abs_pos = prefix_len + offs_m
block_q_end = tl.minimum(q_len, (block_m_id + 1) * BLOCK_M)
kv_loop_end = tl.minimum(kv_len, prefix_len + block_q_end)
+ if HAS_BLOCKS:
+ # rows inside a multimodal span also attend forward to the span's later keys: the tile loop must reach them
+ block_end = tl.load(block_ends_ptr + q_start + offs_m, mask=mask_m, other=0)
+ kv_loop_end = tl.minimum(kv_len, tl.maximum(kv_loop_end, tl.max(block_end, axis=0)))
+ else:
+ block_end = tl.zeros((BLOCK_M,), dtype=tl.int32)
q = tl.load(
q_ptr + (q_start + offs_m[:, None]) * stride_qt + q_head * stride_qh + offs_d[None, :],
@@ -538,6 +546,8 @@ def _extend_attention_kernel(
mask_n = kv_offsets < kv_len
key_pos = kv_offsets
causal_mask = key_pos[None, :] <= q_abs_pos[:, None]
+ if HAS_BLOCKS:
+ causal_mask = causal_mask | (key_pos[None, :] < block_end[:, None])
if SLIDING_WINDOW > 0:
causal_mask = causal_mask & ((key_pos[None, :] + SLIDING_WINDOW) > q_abs_pos[:, None])
final_mask = mask_m[:, None] & mask_n[None, :] & causal_mask
@@ -597,6 +607,7 @@ def _extend_attention_split_kernel(
kv_indptr_ptr,
kv_indices_ptr,
prefix_lens_ptr,
+ block_ends_ptr,
sm_scale,
sinks_ptr,
stride_qt,
@@ -619,6 +630,7 @@ def _extend_attention_split_kernel(
BLOCK_N: tl.constexpr,
SLIDING_WINDOW: tl.constexpr,
HAS_SINKS: tl.constexpr,
+ HAS_BLOCKS: tl.constexpr,
):
seq_id = tl.program_id(0)
q_head = tl.program_id(1)
@@ -702,11 +714,19 @@ def _extend_attention_split_kernel(
m_i = m_new
current_end = tl.minimum(q_len, (block_m_id + 1) * BLOCK_M)
+ if HAS_BLOCKS:
+ # rows inside a multimodal span also attend forward to the span's later keys: the tile loop must reach them
+ block_end = tl.load(block_ends_ptr + q_start + offs_m, mask=mask_m, other=0) - prefix_len
+ current_end = tl.minimum(q_len, tl.maximum(current_end, tl.max(block_end, axis=0)))
+ else:
+ block_end = tl.zeros((BLOCK_M,), dtype=tl.int32)
for start_n in tl.range(0, current_end, BLOCK_N):
local_kv_offsets = start_n + offs_n
mask_n = local_kv_offsets < current_end
local_q_pos = offs_m
causal_mask = local_kv_offsets[None, :] <= local_q_pos[:, None]
+ if HAS_BLOCKS:
+ causal_mask = causal_mask | (local_kv_offsets[None, :] < block_end[:, None])
if SLIDING_WINDOW > 0:
causal_mask = causal_mask & (
(local_kv_offsets[None, :] + SLIDING_WINDOW) > local_q_pos[:, None]
@@ -773,8 +793,9 @@ def extend_paged_attention(
out: torch.Tensor | None = None,
k_extend: torch.Tensor | None = None,
v_extend: torch.Tensor | None = None,
+ block_ends: torch.Tensor | None = None,
) -> torch.Tensor:
- """Block-tiled causal prefill/extend attention over paged KV cache."""
+ """Block-tiled causal prefill/extend attention over paged KV cache; block_ends holds per query token the end of the multimodal span it sits in (0 for none), whose later keys the row also attends."""
assert q.is_cuda and k_cache.is_cuda and v_cache.is_cuda
assert q.dim() == 3 and k_cache.dim() == 3 and v_cache.dim() == 3
@@ -791,8 +812,11 @@ def extend_paged_attention(
assert sinks.numel() >= num_q_heads
sinks = sinks.contiguous()
+ if block_ends is not None:
+ assert block_ends.is_cuda and block_ends.dtype == torch.int32 and block_ends.numel() == num_q_tokens
o = out if out is not None else torch.empty_like(q)
sinks_arg = sinks if sinks is not None else q
+ block_ends_arg = block_ends if block_ends is not None else qo_indptr
block_d = triton.next_power_of_2(head_dim)
block_dv = triton.next_power_of_2(head_dim)
# Tile size is shared-memory bound: keep the fast (large) tiles on GPUs whose opt-in
@@ -820,6 +844,7 @@ def extend_paged_attention(
kv_indptr,
kv_indices,
prefix_lens,
+ block_ends_arg,
sm_scale,
sinks_arg,
q.stride(0),
@@ -842,6 +867,7 @@ def extend_paged_attention(
BLOCK_N=block_n,
SLIDING_WINDOW=sliding_window or 0,
HAS_SINKS=sinks is not None,
+ HAS_BLOCKS=block_ends is not None,
num_warps=8,
num_stages=1,
)
@@ -856,6 +882,7 @@ def extend_paged_attention(
kv_indptr,
kv_indices,
prefix_lens,
+ block_ends_arg,
sm_scale,
sinks_arg,
q.stride(0),
@@ -874,6 +901,7 @@ def extend_paged_attention(
BLOCK_N=block_n,
SLIDING_WINDOW=sliding_window or 0,
HAS_SINKS=sinks is not None,
+ HAS_BLOCKS=block_ends is not None,
num_warps=8,
num_stages=1,
)
diff --git a/python/freetoken/layers/__init__.py b/python/freetoken/layers/__init__.py
index 579dc6e4..d868751a 100644
--- a/python/freetoken/layers/__init__.py
+++ b/python/freetoken/layers/__init__.py
@@ -22,6 +22,7 @@ from .norm import (
GemmaPlusOneRMSNorm,
GemmaPlusOneRMSNormFused,
GemmaRMSNorm,
+ LayerNorm,
RMSNorm,
RMSNormFused,
)
@@ -44,6 +45,7 @@ __all__ = [
"LinearRowParallel",
"LinearOProj",
"LinearQKVMerged",
+ "LayerNorm",
"RMSNorm",
"RMSNormFused",
"GatedRMSNorm",
diff --git a/python/freetoken/layers/norm.py b/python/freetoken/layers/norm.py
index 5eca789b..ae3e7253 100644
--- a/python/freetoken/layers/norm.py
+++ b/python/freetoken/layers/norm.py
@@ -1,6 +1,7 @@
from typing import Tuple
import torch
+import torch.nn.functional as F
from .base import BaseOP
@@ -190,3 +191,15 @@ class GatedRMSNorm(BaseOP):
x=x, weight=self.weight, bias=None, z=z, eps=self.eps,
is_rms_norm=True, norm_before_gate=True, activation=self.activation,
)
+
+
+class LayerNorm(BaseOP):
+ """LayerNorm with bias on torch's fused kernel; the decoders use the RMSNorm family."""
+
+ def __init__(self, size: int, eps: float) -> None:
+ self.eps = eps
+ self.weight = torch.empty(size)
+ self.bias = torch.empty(size)
+
+ def forward(self, x: torch.Tensor) -> torch.Tensor:
+ return F.layer_norm(x, (x.shape[-1],), self.weight, self.bias, self.eps)
diff --git a/python/freetoken/mm/processors/gemma4.py b/python/freetoken/mm/processors/gemma4.py
new file mode 100644
index 00000000..88ae75da
--- /dev/null
+++ b/python/freetoken/mm/processors/gemma4.py
@@ -0,0 +1,104 @@
+"""Gemma 4 image processor: aspect-ratio-preserving 16x16 patches, pooled 3x3 into soft tokens, 1-D rope."""
+
+from __future__ import annotations
+
+import struct
+from typing import Any
+
+import torch
+
+from freetoken.message import MMItem
+from freetoken.mm import mm_pad_value
+from freetoken.mm.config import MultimodalConfig
+from freetoken.mm.processor import MMProcessor, PromptReplacement, content_hash
+
+# the only soft-token budgets the image processor accepts
+_SOFT_TOKEN_BUDGETS = (70, 140, 280, 560, 1120)
+
+
+def _soft_token_budget(max_tokens: int) -> int:
+ """Largest accepted budget within max_tokens."""
+ return max(b for b in _SOFT_TOKEN_BUDGETS if b <= max_tokens)
+
+
+class Gemma4MMProcessor(MMProcessor):
+ def __init__(self, hf_config: Any, model_path: str, mm: MultimodalConfig) -> None:
+ super().__init__(model_path, mm)
+ if mm.image_max_tokens is not None and mm.image_max_tokens < _SOFT_TOKEN_BUDGETS[0]:
+ raise ValueError(f"--image-max-tokens {mm.image_max_tokens} is below the smallest soft-token budget this image processor accepts ({_SOFT_TOKEN_BUDGETS[0]})")
+ vc = hf_config.vision_config
+ self.image_token_id = hf_config.image_token_id
+ self.boi_token_id = hf_config.boi_token_id
+ self.eoi_token_id = hf_config.eoi_token_id
+ self.placeholder = [self.image_token_id]
+ self.pooling_kernel_size = vc.pooling_kernel_size
+ self.patch_dim = 3 * vc.patch_size**2
+
+ def get_mm_processor_kwargs(self, mm: MultimodalConfig) -> dict[str, Any]:
+ kwargs: dict[str, Any] = {"return_tensors": "pt"}
+ if mm.image_max_tokens is not None:
+ # the processor scales every image to its soft-token budget as far as the aspect ratio allows, so a lower bound has no effect
+ kwargs["max_soft_tokens"] = _soft_token_budget(mm.image_max_tokens)
+ return {**kwargs, **mm.processor_kwargs}
+
+ def process(self, images: list[Any]) -> list[MMItem]:
+ processor = self._image_processor()
+ kwargs = self.get_mm_processor_kwargs(self.mm)
+ items: list[MMItem] = []
+ for pil in images:
+ out = processor(images=pil, **kwargs)
+ positions = out["image_position_ids"][0]
+ valid = (positions != -1).all(dim=-1)
+ positions = positions[valid].to(torch.int32)
+ feature = out["pixel_values"][0][valid]
+ n_soft = int(out["num_soft_tokens_per_image"][0])
+ item_hash = content_hash(feature, positions.numpy().tobytes() + struct.pack("<i", n_soft))
+ items.append(
+ MMItem(
+ modality="image",
+ hash=item_hash,
+ pad_value=mm_pad_value(item_hash),
+ offsets=[],
+ feature=feature,
+ model_specific_data={"position_ids": positions, "num_soft_tokens": n_soft},
+ )
+ )
+ return items
+
+ def prompt_replacement(self, item: MMItem) -> PromptReplacement:
+ full = [self.boi_token_id] + [self.image_token_id] * item.num_soft_tokens + [self.eoi_token_id]
+ return PromptReplacement.select_token_id(full, self.image_token_id)
+
+ def dummy_items(self, dtype: torch.dtype, device: torch.device) -> list[MMItem]:
+ k = self.pooling_kernel_size
+ ys, xs = torch.meshgrid(torch.arange(k), torch.arange(k), indexing="ij")
+ positions = torch.stack([xs.reshape(-1), ys.reshape(-1)], dim=-1).to(torch.int32)
+ return [MMItem(
+ modality="image",
+ hash=0,
+ pad_value=0,
+ offsets=[[0, 1]],
+ feature=torch.zeros(k * k, self.patch_dim, dtype=dtype, device=device),
+ model_specific_data={"position_ids": positions.to(device), "num_soft_tokens": 1},
+ )]
+
+
+class Gemma4UnifiedMMProcessor(Gemma4MMProcessor):
+ """The gemma4_unified release: one 48x48 super-patch per soft token and no pooling; the processor call and its budgets are the same."""
+
+ def __init__(self, hf_config: Any, model_path: str, mm: MultimodalConfig) -> None:
+ super().__init__(hf_config, model_path, mm)
+ self.patch_dim = 3 * hf_config.vision_config.model_patch_size**2
+
+ def dummy_items(self, dtype: torch.dtype, device: torch.device) -> list[MMItem]:
+ return [MMItem(
+ modality="image",
+ hash=0,
+ pad_value=0,
+ offsets=[[0, 1]],
+ feature=torch.zeros(1, self.patch_dim, dtype=dtype, device=device),
+ model_specific_data={"position_ids": torch.zeros(1, 2, dtype=torch.int32, device=device), "num_soft_tokens": 1},
+ )]
+
+
+__all__ = ["Gemma4MMProcessor", "Gemma4UnifiedMMProcessor"]
diff --git a/python/freetoken/models/config.py b/python/freetoken/models/config.py
index 273662a9..d7b504a7 100644
--- a/python/freetoken/models/config.py
+++ b/python/freetoken/models/config.py
@@ -5,7 +5,7 @@ from typing import Any, ClassVar, Dict, List, Literal, Tuple, TypeAlias
from freetoken.attention.base import AttnType
# State-dict key prefixes of the vision stack; load_weight drops them when the engine serves text-only.
-VISION_KEY_PREFIXES = ("vision_tower.", "embed_vision.", "visual.")
+VISION_KEY_PREFIXES = ("vision_tower.", "embed_vision.", "vision_embedder.", "visual.")
def detect_expert_quant(hf_config: Any) -> str:
@@ -172,6 +172,8 @@ class SWAAttentionGroupConfig(BaseAttentionGroupConfig):
head_dim: int
rotary_config: RotaryConfig
sliding_window: int
+ # image token spans attend to each other in both directions on these layers
+ bidirectional_mm_blocks: bool = False
@dataclass(frozen=True)
diff --git a/python/freetoken/models/gemma4/__init__.py b/python/freetoken/models/gemma4/__init__.py
index a6b285f2..925bbdb1 100644
--- a/python/freetoken/models/gemma4/__init__.py
+++ b/python/freetoken/models/gemma4/__init__.py
@@ -6,9 +6,9 @@ from .gguf import (
load_q4_0_expert_sources,
parse_gguf_config,
)
-from .model import Gemma4ForCausalLM
+from .model import Gemma4ForCausalLM, Gemma4ForConditionalGeneration, Gemma4UnifiedForConditionalGeneration
from .moe import Gemma4Router
-from .vision import Gemma4MultimodalEmbedder, Gemma4VisionModel
+from .vision import Gemma4MultimodalEmbedder, Gemma4UnifiedVisionEmbedder, Gemma4VisionModel
from .weight import (
nvfp4_expert_spec,
iter_weights,
@@ -19,7 +19,10 @@ __all__ = [
"nvfp4_expert_spec",
"Gemma4Attention",
"Gemma4ForCausalLM",
+ "Gemma4ForConditionalGeneration",
"Gemma4MultimodalEmbedder",
+ "Gemma4UnifiedForConditionalGeneration",
+ "Gemma4UnifiedVisionEmbedder",
"Gemma4Router",
"Gemma4VisionModel",
"parse_config",
diff --git a/python/freetoken/models/gemma4/attention.py b/python/freetoken/models/gemma4/attention.py
index 4f85255f..3f34e1cd 100644
--- a/python/freetoken/models/gemma4/attention.py
+++ b/python/freetoken/models/gemma4/attention.py
@@ -50,6 +50,7 @@ class Gemma4Attention(BaseOP):
self.attn_spec = AttentionSpec(
sliding_window=group.sliding_window if self.is_swa else None,
sm_scale=config.attn_sm_scale,
+ bidirectional_mm_blocks=self.is_swa and group.bidirectional_mm_blocks,
)
self.rotary = get_rope(
head_dim=self.head_dim,
diff --git a/python/freetoken/models/gemma4/config.py b/python/freetoken/models/gemma4/config.py
index 94718603..0f99fd77 100644
--- a/python/freetoken/models/gemma4/config.py
+++ b/python/freetoken/models/gemma4/config.py
@@ -28,7 +28,18 @@ class VisionConfig:
hidden_act: str
standardize: bool
use_clipped_linears: bool
- soft_tokens_per_image: int
+ text_hidden_size: int
+
+
+@dataclass(frozen=True)
+class UnifiedVisionConfig:
+ """The gemma4_unified release's encoder-free vision path: one super-patch becomes one soft token."""
+
+ hidden_size: int
+ patch_dim: int
+ posemb_size: int
+ layer_norm_eps: float
+ rms_norm_eps: float
text_hidden_size: int
@@ -39,10 +50,22 @@ def _text_config(hf_config: Any) -> tuple[Any, list[str] | None, Any]:
return hf_config, top_architectures, hf_config
-def _parse_vision_config(top_cfg: Any, text_hidden_size: int) -> VisionConfig | None:
+def _parse_vision_config(top_cfg: Any, text_hidden_size: int) -> VisionConfig | UnifiedVisionConfig | None:
vc = getattr(top_cfg, "vision_config", None)
if vc is None:
return None
+ if getattr(vc, "num_attention_heads", None) is None:
+ # the gemma4_unified release's section describes a linear embedder without attention; any other tower-less section is not served
+ if getattr(vc, "model_patch_size", None) is None:
+ return None
+ return UnifiedVisionConfig(
+ hidden_size=vc.mm_embed_dim,
+ patch_dim=3 * vc.model_patch_size**2,
+ posemb_size=vc.mm_posemb_size,
+ layer_norm_eps=1e-5, # the reference builds torch LayerNorms with the default eps
+ rms_norm_eps=vc.rms_norm_eps,
+ text_hidden_size=text_hidden_size,
+ )
rope_params = getattr(vc, "rope_parameters", None) or {}
act = getattr(vc, "hidden_activation", "gelu_pytorch_tanh")
act = "gelu_tanh" if "tanh" in act else act
@@ -62,10 +85,6 @@ def _parse_vision_config(top_cfg: Any, text_hidden_size: int) -> VisionConfig |
hidden_act=act,
standardize=bool(getattr(vc, "standardize", True)),
use_clipped_linears=bool(getattr(vc, "use_clipped_linears", False)),
- soft_tokens_per_image=int(
- getattr(top_cfg, "vision_soft_tokens_per_image", None)
- or getattr(vc, "default_output_length", 280)
- ),
text_hidden_size=text_hidden_size,
)
@@ -186,9 +205,10 @@ def parse_config(hf_config: Any) -> ModelConfig:
head_dim=sliding_head_dim,
rotary_config=swa_rotary_config,
sliding_window=cfg.sliding_window,
+ bidirectional_mm_blocks=getattr(cfg, "use_bidirectional_attention", None) == "vision",
),
),
)
-__all__ = ["VisionConfig", "parse_config"]
+__all__ = ["UnifiedVisionConfig", "VisionConfig", "parse_config"]
diff --git a/python/freetoken/models/gemma4/model.py b/python/freetoken/models/gemma4/model.py
index d151540b..5c91b905 100644
--- a/python/freetoken/models/gemma4/model.py
+++ b/python/freetoken/models/gemma4/model.py
@@ -13,13 +13,14 @@ from freetoken.layers import (
)
from freetoken.utils import nvtx_annotate
-from freetoken.models.blocks import BaseLLMModel
+from freetoken.models.blocks import BaseLLMModel, embed_input_ids
from .attention import Gemma4Attention
from .moe import Gemma4DenseMLP, Gemma4MLP
-from .vision import Gemma4MultimodalEmbedder, Gemma4VisionModel
+from .vision import Gemma4MultimodalEmbedder, Gemma4UnifiedVisionEmbedder, Gemma4VisionModel
if TYPE_CHECKING:
+ from freetoken.message import MMItem
from freetoken.models.config import ModelConfig
@@ -68,31 +69,9 @@ class Gemma4Model(BaseOP):
]
)
self.norm = GemmaRMSNorm(config.hidden_size, eps=config.rms_norm_eps)
- self._image_token_id = config.image_token_id
-
- def _merge_multimodal(self, input_ids: torch.Tensor, x: torch.Tensor) -> torch.Tensor:
- """Scatter precomputed image soft-token embeddings at image-token positions.
-
- ``mm_embeds`` (set by the scheduler from each request's vision features) is a
- ``[num_image_tokens, hidden]`` tensor whose rows replace the placeholder
- embeddings produced for ``image_token_id``. Only runs during prefill batches
- that carry images; decode batches never do.
- """
- batch = get_global_ctx().batch
- mm_embeds = getattr(batch, "mm_embeds", None)
- if mm_embeds is None or self._image_token_id is None:
- return x
- mask = input_ids == self._image_token_id
- n_slots = int(mask.sum().item())
- assert n_slots == mm_embeds.shape[0], (
- f"image-token slots ({n_slots}) != vision features ({mm_embeds.shape[0]}); "
- "image tokens must not be split across prefill chunks"
- )
- return x.masked_scatter(mask.unsqueeze(-1), mm_embeds.to(x.dtype))
def forward(self, input_ids: torch.Tensor) -> torch.Tensor:
- x = self.embed_tokens.forward(input_ids)
- x = self._merge_multimodal(input_ids, x)
+ x = embed_input_ids(self.embed_tokens, input_ids, get_global_ctx().batch)
for layer in self.layers.op_list:
x = layer.forward(x)
return self.norm.forward(x)
@@ -110,9 +89,6 @@ class Gemma4ForCausalLM(BaseLLMModel):
prefix="lm_head",
)
self._final_logit_softcapping = config.final_logit_softcapping
- if config.is_multimodal:
- self.vision_tower = Gemma4VisionModel(config.vision_config)
- self.embed_vision = Gemma4MultimodalEmbedder(config.vision_config)
super().__init__()
# GGUF checkpoints carry native block-quantized weights: swap the dense
@@ -122,10 +98,6 @@ class Gemma4ForCausalLM(BaseLLMModel):
if is_gguf_model(config):
convert_gemma4_to_gguf(self, config)
- @torch.inference_mode()
- def encode(self, item) -> torch.Tensor:
- raise NotImplementedError("gemma4 image input needs an MMProcessor (freetoken.mm.processors)")
-
def forward(self) -> torch.Tensor:
output = self.model.forward(get_global_ctx().batch.input_ids)
logits = self.lm_head.forward(output)
@@ -135,4 +107,42 @@ class Gemma4ForCausalLM(BaseLLMModel):
return logits
-__all__ = ["Gemma4ForCausalLM"]
+class Gemma4ForConditionalGeneration(Gemma4ForCausalLM):
+ def __init__(self, config: ModelConfig):
+ super().__init__(config)
+ if config.is_multimodal:
+ self.vision_tower = Gemma4VisionModel(config.vision_config)
+ self.embed_vision = Gemma4MultimodalEmbedder(config.vision_config)
+
+ def place_encoder_weights(self, mode: str) -> None:
+ self.vision_tower.place_weights(mode)
+
+ def encode(self, item: MMItem) -> torch.Tensor:
+ device = self.embed_vision.embedding_projection.weight.device
+ feature = item.feature.to(device, non_blocking=True)[None]
+ positions = item.position_ids.to(device, non_blocking=True).long()[None]
+ return self.embed_vision.forward(self.vision_tower.forward(feature, positions))
+
+
+class Gemma4UnifiedForConditionalGeneration(Gemma4ForCausalLM):
+ """The gemma4_unified release: no vision tower, a linear embedder turns each 48x48 super-patch into one soft token."""
+
+ def __init__(self, config: ModelConfig):
+ super().__init__(config)
+ if config.is_multimodal:
+ self.vision_embedder = Gemma4UnifiedVisionEmbedder(config.vision_config)
+ self.embed_vision = Gemma4MultimodalEmbedder(config.vision_config)
+
+ def place_encoder_weights(self, mode: str) -> None:
+ """Nothing to stream: the embedder is a few dense tensors, resident under either placement."""
+ if mode not in ("gpu", "host"):
+ raise ValueError(f"unknown vision weight placement {mode!r}")
+
+ def encode(self, item: MMItem) -> torch.Tensor:
+ device = self.embed_vision.embedding_projection.weight.device
+ feature = item.feature.to(device, non_blocking=True)[None]
+ positions = item.position_ids.to(device, non_blocking=True).long()[None]
+ return self.embed_vision.forward(self.vision_embedder.forward(feature, positions))[0]
+
+
+__all__ = ["Gemma4ForCausalLM", "Gemma4ForConditionalGeneration", "Gemma4UnifiedForConditionalGeneration"]
diff --git a/python/freetoken/models/gemma4/vision.py b/python/freetoken/models/gemma4/vision.py
index 7c260c48..37446e74 100644
--- a/python/freetoken/models/gemma4/vision.py
+++ b/python/freetoken/models/gemma4/vision.py
@@ -4,10 +4,11 @@ from typing import TYPE_CHECKING, Tuple
import torch
import torch.nn.functional as F
-from freetoken.layers import BaseOP, GemmaRMSNorm, LinearReplicated, OPList
+from freetoken.layers import BaseOP, GemmaRMSNorm, LayerNorm, LinearReplicated, OPList
+from freetoken.models.weight_stream import BlockWeightStreamer
if TYPE_CHECKING:
- from freetoken.models.gemma4.config import VisionConfig
+ from freetoken.models.gemma4.config import UnifiedVisionConfig, VisionConfig
def _rotate_half(x: torch.Tensor) -> torch.Tensor:
@@ -108,7 +109,7 @@ class Gemma4VisionAttention(BaseOP):
k = _apply_multidim_rope(k, cos, sin).transpose(1, 2)
v = v.transpose(1, 2)
- o = F.scaled_dot_product_attention(q, k, v, attn_mask=attn_mask, scale=1.0)
+ o = F.scaled_dot_product_attention(q, k, v, attn_mask=attn_mask, scale=1.0, enable_gqa=self.num_kv_heads != self.num_heads)
o = o.transpose(1, 2).reshape(B, P, self.num_heads * self.head_dim)
return self.o_proj.forward(o)
@@ -189,15 +190,33 @@ class Gemma4VisionModel(BaseOP):
"""Pixels -> soft tokens. Output is ``[num_valid_soft_tokens, hidden]`` (padding stripped)."""
def __init__(self, vc: VisionConfig):
+ if vc.use_clipped_linears:
+ raise NotImplementedError("the gemma4 vision tower does not apply clipped linears")
self.patch_embedder = Gemma4VisionPatchEmbedder(vc)
self.encoder = Gemma4VisionEncoder(vc)
self._standardize = vc.standardize
self._pooling_kernel_size = vc.pooling_kernel_size
self._root_hidden = vc.hidden_size**0.5
+ self._streamer: BlockWeightStreamer | None = None
if vc.standardize:
self.std_bias = torch.empty(vc.hidden_size)
self.std_scale = torch.empty(vc.hidden_size)
+ def place_weights(self, mode: str) -> None:
+ """gpu: every tensor resident; host: encoder layer tensors in pinned banks streamed two layers at a time (patch embedder and pooling stay resident)."""
+ if mode == "host" and self._streamer is None:
+ self._streamer = BlockWeightStreamer(self.encoder.layers.op_list, self.patch_embedder.input_proj.weight.device)
+ elif mode == "gpu" and self._streamer is not None:
+ self._streamer.unstream()
+ self._streamer = None
+ elif mode not in ("gpu", "host"):
+ raise ValueError(f"unknown vision weight placement {mode!r}")
+
+ def _layers(self):
+ if self._streamer is None:
+ return enumerate(self.encoder.layers.op_list)
+ return self._streamer.blocks(self.encoder.layers.op_list)
+
def forward(self, pixel_values: torch.Tensor, position_ids: torch.Tensor) -> torch.Tensor:
k = self._pooling_kernel_size
output_length = pixel_values.shape[-2] // (k * k)
@@ -206,7 +225,7 @@ class Gemma4VisionModel(BaseOP):
h = self.patch_embedder.forward(pixel_values, position_ids, padding)
cos, sin = self.encoder._rotary.cos_sin(position_ids, h.dtype)
attn_mask = (~padding)[:, None, None, :] # [B, 1, 1, P] True = attend
- for layer in self.encoder.layers.op_list:
+ for _, layer in self._layers():
h = layer.forward(h, cos, sin, attn_mask)
h = h.masked_fill(padding.unsqueeze(-1), 0.0)
@@ -233,4 +252,22 @@ class Gemma4MultimodalEmbedder(BaseOP):
return self.embedding_projection.forward(self.embedding_pre_projection_norm.forward(x))
-__all__ = ["Gemma4VisionModel", "Gemma4MultimodalEmbedder"]
+class Gemma4UnifiedVisionEmbedder(BaseOP):
+ """The gemma4_unified release's stand-in for the tower: LayerNorm, dense projection, LayerNorm, factorized 2-D position embedding, LayerNorm; one 48x48 super-patch per soft token."""
+
+ def __init__(self, vc: UnifiedVisionConfig):
+ self.patch_ln1 = LayerNorm(vc.patch_dim, vc.layer_norm_eps)
+ self.patch_dense = LinearReplicated(vc.patch_dim, vc.hidden_size, has_bias=True)
+ self.patch_ln2 = LayerNorm(vc.hidden_size, vc.layer_norm_eps)
+ self.pos_embedding = torch.empty(vc.posemb_size, 2, vc.hidden_size)
+ self.pos_norm = LayerNorm(vc.hidden_size, vc.layer_norm_eps)
+
+ def forward(self, pixel_values: torch.Tensor, position_ids: torch.Tensor) -> torch.Tensor:
+ h = self.patch_ln1.forward(pixel_values.to(self.patch_dense.weight.dtype))
+ h = self.patch_ln2.forward(self.patch_dense.forward(h))
+ clamped = position_ids.clamp(min=0)
+ pos = self.pos_embedding[clamped[..., 0], 0] + self.pos_embedding[clamped[..., 1], 1]
+ return self.pos_norm.forward(h + pos)
+
+
+__all__ = ["Gemma4MultimodalEmbedder", "Gemma4UnifiedVisionEmbedder", "Gemma4VisionModel"]
diff --git a/python/freetoken/models/gemma4/weight.py b/python/freetoken/models/gemma4/weight.py
index 4413b67f..19b23ed7 100644
--- a/python/freetoken/models/gemma4/weight.py
+++ b/python/freetoken/models/gemma4/weight.py
@@ -132,6 +132,7 @@ def iter_weights(
*,
include_moe_experts: bool,
include_non_moe: bool,
+ include_vision: bool = True,
) -> Iterator[tuple[str, torch.Tensor]]:
def rename_key(raw_name: str) -> str | None:
prefix = "model.language_model."
@@ -146,6 +147,8 @@ def iter_weights(
)
if raw_name.startswith("model.embed_vision."):
return "embed_vision." + raw_name[len("model.embed_vision.") :]
+ if raw_name.startswith("model.vision_embedder."):
+ return "vision_embedder." + raw_name[len("model.vision_embedder.") :]
return None
def merge_info(key: str) -> tuple[str, MergeRule] | None:
@@ -189,7 +192,9 @@ def iter_weights(
if raw_name.endswith(_NVFP4_DENSE_SCALE_SUFFIXES):
continue
- is_vision = name.startswith(("vision_tower.", "embed_vision."))
+ is_vision = name.startswith(("vision_tower.", "embed_vision.", "vision_embedder."))
+ if is_vision and not include_vision:
+ continue
is_expert = (
not is_vision and _PACKED_EXPERT_PATTERN.match(name) is not None
)
diff --git a/python/freetoken/models/register.py b/python/freetoken/models/register.py
index 2396bb64..62eade18 100644
--- a/python/freetoken/models/register.py
+++ b/python/freetoken/models/register.py
@@ -72,6 +72,9 @@ _GEMMA4_SEGMENTS = (
("feed_forward.router", "router"),
)
_GEMMA4_PACKED = _DENSE_PACKED + _EXPERTS_PACKED
+_GEMMA4_PROCESSOR = "freetoken.mm.processors.gemma4:Gemma4MMProcessor"
+_GEMMA4_UNIFIED_PROCESSOR = "freetoken.mm.processors.gemma4:Gemma4UnifiedMMProcessor"
+_GEMMA4_ENCODERS = (EncoderSpec("vision", "vision_config", ("image",)),)
_MINIMAX_M3_PACKED = _DENSE_PACKED + (
("index_qk_proj", ("index_q_proj", "index_k_proj")),
) + _EXPERTS_W123_PACKED
@@ -215,10 +218,12 @@ _MODEL_REGISTRY: dict[str, ModelSpec] = {
),
"Gemma4ForConditionalGeneration": ModelSpec(
"freetoken.models.gemma4",
- "Gemma4ForCausalLM",
+ "Gemma4ForConditionalGeneration",
checkpoint_roots=_LANGUAGE_MODEL_ROOT,
checkpoint_segments=_GEMMA4_SEGMENTS,
packed_modules_mapping=_GEMMA4_PACKED,
+ mm_processor=_GEMMA4_PROCESSOR,
+ encoders=_GEMMA4_ENCODERS,
),
"Gemma4ForCausalLM": ModelSpec(
"freetoken.models.gemma4",
@@ -226,14 +231,15 @@ _MODEL_REGISTRY: dict[str, ModelSpec] = {
checkpoint_segments=_GEMMA4_SEGMENTS,
packed_modules_mapping=_GEMMA4_PACKED,
),
- # Dense text tower of the gemma-4-12B "Unified"/omni model (model_type gemma4_unified_text).
- # Same decoder as gemma4; the dense feed-forward is selected via config.is_moe.
+ # the gemma-4-12B "Unified" release (model_type gemma4_unified): the gemma4 decoder with a dense feed-forward and a linear vision embedder in place of the ViT tower; audio is not wired
"Gemma4UnifiedForConditionalGeneration": ModelSpec(
"freetoken.models.gemma4",
- "Gemma4ForCausalLM",
+ "Gemma4UnifiedForConditionalGeneration",
checkpoint_roots=_LANGUAGE_MODEL_ROOT,
checkpoint_segments=_GEMMA4_SEGMENTS,
packed_modules_mapping=_GEMMA4_PACKED,
+ mm_processor=_GEMMA4_UNIFIED_PROCESSOR,
+ encoders=_GEMMA4_ENCODERS,
),
"Gemma4UnifiedForCausalLM": ModelSpec(
"freetoken.models.gemma4",
diff --git a/python/freetoken/scheduler/mm.py b/python/freetoken/scheduler/mm.py
index 5d9e4d86..8c656c47 100644
--- a/python/freetoken/scheduler/mm.py
+++ b/python/freetoken/scheduler/mm.py
@@ -4,6 +4,8 @@ from __future__ import annotations
from typing import TYPE_CHECKING, List, Tuple
+from freetoken.utils import align_down
+
if TYPE_CHECKING:
from freetoken.message import MMItem
from freetoken.mm.encoder_cache import EncoderCache
@@ -14,6 +16,24 @@ def mm_rows_after(item: MMItem, cached_len: int) -> int:
return sum(max(0, span_hi - max(span_lo, cached_len)) for span_lo, span_hi in item.offsets)
+def mm_chunk_end(items: List[MMItem], chunk_lo: int, end: int, align: int) -> int:
+ """The chunk end pulled back to an align multiple before any image span it would cut; end itself when that span already begins at or before chunk_lo (the image is longer than the budget)."""
+ cut = end
+ for span_lo, span_hi in sorted((lo, hi) for item in items for lo, hi in item.offsets)[::-1]:
+ if span_lo < cut < span_hi:
+ if span_lo <= chunk_lo:
+ return end
+ cut = align_down(span_lo, align)
+ if cut <= chunk_lo:
+ return end
+ return cut
+
+
+def cut_image_spans(reqs) -> List[Tuple[int, int]]:
+ """The image spans the reqs' current chunks end inside of; mm_chunk_end leaves such a cut only for an image longer than the chunk."""
+ return [(lo, hi) for req in reqs if req.mm_items for item in req.mm_items for lo, hi in item.offsets if lo < req.device_len < hi]
+
+
def plan_mm_chunk(
uid: int,
items: List[MMItem],
@@ -44,11 +64,12 @@ def plan_mm_chunk(
return jobs, plan
-def plan_mm_batch(reqs, encoder_cache: EncoderCache | None) -> tuple[List[MMItem], List[Tuple[int, int, int, int, int, int]], List[int]]:
- """Jobs, plan and the batch rows of every gathered embedding row, over the reqs in batch order (each spans [cached_len, device_len))."""
+def plan_mm_batch(reqs, encoder_cache: EncoderCache | None) -> tuple[List[MMItem], List[Tuple[int, int, int, int, int, int]], List[int], List[int]]:
+ """Jobs, plan, the batch rows of every gathered embedding row and, per batch token, the end of the image span holding it (0 for text), over the reqs in batch order (each spans [cached_len, device_len))."""
jobs: List[MMItem] = []
plan: List[Tuple[int, int, int, int, int, int]] = []
rows: List[int] = []
+ block_ends: List[int] = []
offset = 0
for req in reqs:
if req.mm_items:
@@ -57,8 +78,14 @@ def plan_mm_batch(reqs, encoder_cache: EncoderCache | None) -> tuple[List[MMItem
plan.extend(req_plan)
for _, _, row_lo, row_hi, _, pos in req_plan:
rows.extend(range(offset + pos, offset + pos + row_hi - row_lo))
+ if not block_ends:
+ block_ends = [0] * sum(r.extend_len for r in reqs)
+ for item in req.mm_items:
+ for span_lo, span_hi in item.offsets:
+ for i in range(max(span_lo, req.cached_len), min(span_hi, req.device_len)):
+ block_ends[offset + i - req.cached_len] = span_hi
offset += req.extend_len
- return jobs, plan, rows
+ return jobs, plan, rows, block_ends
-__all__ = ["mm_rows_after", "plan_mm_batch", "plan_mm_chunk"]
+__all__ = ["cut_image_spans", "mm_chunk_end", "mm_rows_after", "plan_mm_batch", "plan_mm_chunk"]
diff --git a/python/freetoken/scheduler/prefill.py b/python/freetoken/scheduler/prefill.py
index 3d94b53a..bd8129aa 100644
--- a/python/freetoken/scheduler/prefill.py
+++ b/python/freetoken/scheduler/prefill.py
@@ -1,5 +1,6 @@
from __future__ import annotations
+import math
from dataclasses import dataclass, field
from typing import TYPE_CHECKING, List, Tuple
@@ -7,7 +8,7 @@ import torch
from freetoken.core import Batch, Req
from freetoken.utils import align_down, div_ceil, init_logger
-from .mm import mm_rows_after
+from .mm import mm_chunk_end, mm_rows_after
from .utils import PendingReq
if TYPE_CHECKING:
@@ -46,11 +47,19 @@ class PrefillAdder:
cache_manager: CacheManager
table_manager: TableManager
encoder_cache: EncoderCache | None = None
+ # end a chunk before an image it would cut; only models whose image spans attend in both directions need it
+ keep_images_whole: bool = False
+ # the whole budget of this pass; token_budget shrinks as requests are admitted
+ pass_budget: int = 0
# SWA-pool tokens charged to reqs admitted so far this pass. Mirrors reserved_size: swa is
# allocated only in allocate_paged (after the pass), so swa_available_size does not decrement
# across the admission loop -- without this, successive admits all see the full pool.
reserved_swa: int = 0
+ def __post_init__(self) -> None:
+ if not self.pass_budget:
+ self.pass_budget = self.token_budget
+
def _kv_reservation_size(self, total_len: int, cached_len: int) -> int:
"""Return the token-equivalent cost of the additional KV pages for a request."""
page_size = self.cache_manager.page_size
@@ -165,9 +174,6 @@ class PrefillAdder:
if aligned <= 0:
return None
chunk_size = aligned
- self.reserved_swa += (
- div_ceil(cached_len + chunk_size, ps) - div_ceil(cached_len, ps)
- ) * ps
align = self.cache_manager.prefill_chunk_align
if align > 1 and 0 < chunk_size < remain_len:
# An unaligned chunk end is correct, it just loses this prompt's snapshot boundaries --
@@ -175,6 +181,18 @@ class PrefillAdder:
# the request until it gets a bigger turn.
aligned = align_down(cached_len + chunk_size, align) - cached_len
chunk_size = aligned if aligned > 0 else chunk_size
+ if self.keep_images_whole and pending_req.mm_items and chunk_size < remain_len:
+ # a cut image would attend within only the part already in the cache: end the chunk before it, decided last because the caps above only move the end earlier and would undo it
+ unit = math.lcm(self.cache_manager.page_size if self.cache_manager.swa_paged else 1, align if align > 1 else 1)
+ end = mm_chunk_end(pending_req.mm_items, cached_len, cached_len + chunk_size, unit)
+ cut = next((hi for item in pending_req.mm_items for lo, hi in item.offsets if lo < end < hi), None)
+ if cut is not None and self.token_budget < self.pass_budget and cut - cached_len <= self.pass_budget:
+ # other requests took part of this pass; a pass of its own holds the image whole
+ return None
+ chunk_size = end - cached_len
+ if self.cache_manager.swa_paged:
+ ps = self.cache_manager.page_size
+ self.reserved_swa += (div_ceil(cached_len + chunk_size, ps) - div_ceil(cached_len, ps)) * ps
is_chunked = chunk_size < remain_len
CLS = ChunkedReq if is_chunked else Req
self.token_budget -= chunk_size
@@ -256,6 +274,7 @@ class PrefillManager:
table_manager: TableManager
decode_manager: DecodeManager
encoder_cache: EncoderCache | None = None
+ keep_images_whole: bool = False
pending_list: List[PendingReq] = field(default_factory=list)
def add_one_req(self, req: UserMsg) -> None:
@@ -281,6 +300,7 @@ class PrefillManager:
cache_manager=self.cache_manager,
table_manager=self.table_manager,
encoder_cache=self.encoder_cache,
+ keep_images_whole=self.keep_images_whole,
)
reqs: List[Req] = []
chunked_list: List[PendingReq] = []
diff --git a/python/freetoken/scheduler/scheduler.py b/python/freetoken/scheduler/scheduler.py
index 28520e13..ac1bf322 100644
--- a/python/freetoken/scheduler/scheduler.py
+++ b/python/freetoken/scheduler/scheduler.py
@@ -32,7 +32,7 @@ from .cache import CacheManager
from .config import SchedulerConfig
from .decode import DecodeManager
from .io import SchedulerIOMixin
-from .mm import plan_mm_batch
+from .mm import cut_image_spans, plan_mm_batch
from .prefill import ChunkedReq, PrefillManager
from .status import SchedulerStatusReporter
from .table import TableManager
@@ -91,11 +91,13 @@ class Scheduler(SchedulerIOMixin):
) or getattr(self.engine.kv_cache, "sliding_window_size", None),
)
self.decode_manager = DecodeManager(config.page_size)
+ self._bidirectional_mm = any(getattr(g, "bidirectional_mm_blocks", False) for g in config.model_config.attention_groups)
self.prefill_manager = PrefillManager(
self.cache_manager,
self.table_manager,
self.decode_manager,
encoder_cache=self.engine.encoder_cache,
+ keep_images_whole=self._bidirectional_mm,
)
# some alias for easy access
@@ -140,6 +142,7 @@ class Scheduler(SchedulerIOMixin):
)
self.config = config
self._model_is_mrope = config.model_config.model_is_mrope
+ self._warned_cut_image = False
self.status_reporter = SchedulerStatusReporter(
log=logger.info_rank0,
decode_log_interval=config.decode_log_interval,
@@ -849,11 +852,19 @@ class Scheduler(SchedulerIOMixin):
def _gather_multimodal(self, batch: Batch) -> None:
"""Plan the chunk's encoder jobs, gather rows and scatter rows over the batch; the engine runs them before the LM forward."""
- jobs, plan, rows = plan_mm_batch(batch.padded_reqs, self.engine.encoder_cache)
+ jobs, plan, rows, block_ends = plan_mm_batch(batch.padded_reqs, self.engine.encoder_cache)
if plan:
batch.mm_encoder_jobs = jobs
batch.mm_gather_plan = plan
batch.mm_rows = torch.tensor(rows, dtype=torch.int64, pin_memory=True).to(self.device, non_blocking=True)
+ batch.mm_block_ends = torch.tensor(block_ends, dtype=torch.int32, pin_memory=True).to(self.device, non_blocking=True)
+ if self._bidirectional_mm and not self._warned_cut_image and (cut := cut_image_spans(batch.padded_reqs)):
+ # only a bidirectional image span loses context when cut, and only an image longer than the chunk still gets cut
+ lo, hi = cut[0]
+ self._warned_cut_image = True
+ logger.warning_rank0(
+ f"an image of {hi - lo} tokens does not fit one prefill chunk (--max-extend-tokens {self.prefill_budget}, or the sliding-window pool's share of it): its earlier rows attend within the first part only"
+ )
def _schedule_next_batch(self) -> ForwardInput | None:
# TODO: support other policies: e.g. DECODE first
diff --git a/tests/README.md b/tests/README.md
index 33e5756c..cb594663 100644
--- a/tests/README.md
+++ b/tests/README.md
@@ -50,6 +50,8 @@ checkpoint is set:
| `FREETOKEN_AIME_MIN_FREE_GIB` | `e2e/test_aime.py` — required free GPU memory (default `70`) |
| `FREETOKEN_QWEN36_MODEL` | `tokenizer/test_mm_tokenize.py` — local Qwen3.6 checkpoint (image processor + tokenizer) |
| `FREETOKEN_QWEN3VL_MODEL` | `tokenizer/test_mm_tokenize.py` — local Qwen3-VL checkpoint (same expansion, DeepStack family) |
+| `FREETOKEN_GEMMA4_MODEL` | `models/test_gemma4_vision.py`, `tokenizer/test_mm_tokenize.py` — local Gemma-4 `gemma4` checkpoint (tower parity, boi/eoi expansion; the tokenizer test also takes the 12B `gemma4_unified` checkpoint) |
+| `FREETOKEN_GEMMA4_UNIFIED_MODEL` | `models/test_gemma4_vision.py` — local gemma-4-12B `gemma4_unified` checkpoint (embedder parity) |
| `FREETOKEN_TEST_MOE_CACHE_SIZE` | `e2e/test_aime.py` — >0 switches to the offload MoE backend with this cache size |
| `FREETOKEN_TEST_MEM_RATIO` | `e2e/test_aime.py` — offload-mode memory_ratio (default `0.9`) |
| `FREETOKEN_REBUILD_TEST_MODEL` | `e2e/test_cache_rebuild.py` — a SMALL local model dir; boots a real server (falls back to `FREETOKEN_TEST_MODEL`) |
diff --git a/tests/kernels/test_triton_attention.py b/tests/kernels/test_triton_attention.py
index 6f4afca9..a29f621d 100644
--- a/tests/kernels/test_triton_attention.py
+++ b/tests/kernels/test_triton_attention.py
@@ -17,6 +17,7 @@ def _reference_paged_attention(
sm_scale: float,
sliding_window: int | None,
sinks: torch.Tensor | None = None,
+ block_ends: torch.Tensor | None = None,
) -> torch.Tensor:
outs = []
group = q.shape[1] // k_cache.shape[1]
@@ -30,6 +31,8 @@ def _reference_paged_attention(
scores = torch.einsum("hd,hkd->hk", q[tok].float(), k.float()) * sm_scale
key_pos = torch.arange(end - start, device=q.device)
mask = key_pos <= q_positions[tok]
+ if block_ends is not None:
+ mask = mask | (key_pos < block_ends[tok])
if sliding_window is not None:
mask = mask & (key_pos + sliding_window > q_positions[tok])
scores = scores.masked_fill(~mask.unsqueeze(0), float("-inf"))
@@ -511,6 +514,55 @@ def test_extend_triton_attention_matches_reference(
torch.testing.assert_close(actual.float(), expected.float(), atol=2e-2, rtol=2e-2)
[email protected](not torch.cuda.is_available(), reason="Triton attention needs CUDA")
[email protected]("use_split_inputs", [False, True])
[email protected]("sliding_window", [None, 48])
+def test_extend_triton_attention_with_bidirectional_blocks_matches_reference(use_split_inputs: bool, sliding_window: int | None):
+ """Rows of an image span see the span's later keys across q tiles, text rows stay causal, the window still bounds the past."""
+ from freetoken.kernel.triton.attention import extend_paged_attention
+
+ torch.manual_seed(3)
+ device = torch.device("cuda")
+ head_dim, num_q_heads, num_kv_heads = 256, 16, 8
+ cached_lens, extend_lens = [2, 0], [300, 40]
+ spans = [(70, 260), (30, 60)] # req 0: straddles the q tiles; req 1: cut off by the chunk end
+ seq_lens = [c + e for c, e in zip(cached_lens, extend_lens)]
+ total_q, total_kv = sum(extend_lens), sum(seq_lens)
+ q = torch.randn(total_q, num_q_heads, head_dim, device=device, dtype=torch.bfloat16)
+ k_cache = torch.randn(total_kv, num_kv_heads, head_dim, device=device, dtype=torch.bfloat16)
+ v_cache = torch.randn(total_kv, num_kv_heads, head_dim, device=device, dtype=torch.bfloat16)
+ qo_indptr = torch.tensor([0] + extend_lens, dtype=torch.int32, device=device).cumsum_(0)
+ kv_indptr = torch.tensor([0] + seq_lens, dtype=torch.int32, device=device).cumsum_(0)
+ indices = torch.arange(total_kv, dtype=torch.int32, device=device)
+ prefix_lens = torch.tensor(cached_lens, dtype=torch.int32, device=device)
+ q_to_req = torch.empty(total_q, dtype=torch.int32, device=device)
+ q_positions = torch.empty(total_q, dtype=torch.int64, device=device)
+ block_ends = torch.zeros(total_q, dtype=torch.int32, device=device)
+ offset = kv_offset = 0
+ for req_idx, (cached_len, extend_len, (lo, hi)) in enumerate(zip(cached_lens, extend_lens, spans)):
+ q_to_req[offset : offset + extend_len].fill_(req_idx)
+ q_positions[offset : offset + extend_len] = torch.arange(cached_len, cached_len + extend_len, device=device)
+ block_ends[offset + max(lo, cached_len) - cached_len : offset + min(hi, cached_len + extend_len) - cached_len] = hi
+ offset += extend_len
+ kv_offset += cached_len + extend_len
+ k_extend = torch.cat([k_cache[kv_indptr[i] + cached_lens[i] : kv_indptr[i + 1]] for i in range(2)])
+ v_extend = torch.cat([v_cache[kv_indptr[i] + cached_lens[i] : kv_indptr[i + 1]] for i in range(2)])
+ sm_scale = head_dim**-0.5
+
+ actual = extend_paged_attention(
+ q, k_cache, v_cache, qo_indptr, kv_indptr, indices, prefix_lens, max(extend_lens), sm_scale,
+ sliding_window=sliding_window, block_ends=block_ends,
+ k_extend=k_extend if use_split_inputs else None, v_extend=v_extend if use_split_inputs else None,
+ )
+ expected = _reference_paged_attention(
+ q, k_cache, v_cache, kv_indptr, indices, q_to_req, q_positions, sm_scale, sliding_window, block_ends=block_ends
+ )
+ causal_only = _reference_paged_attention(q, k_cache, v_cache, kv_indptr, indices, q_to_req, q_positions, sm_scale, sliding_window)
+
+ torch.testing.assert_close(actual.float(), expected.float(), atol=2e-2, rtol=2e-2)
+ assert not torch.allclose(expected.float(), causal_only.float(), atol=2e-2, rtol=2e-2)
+
+
@pytest.mark.skipif(not torch.cuda.is_available(), reason="Triton attention needs CUDA")
@pytest.mark.parametrize("use_split_inputs", [False, True])
def test_extend_triton_attention_with_sinks_matches_reference(use_split_inputs: bool):
@@ -684,6 +736,57 @@ def test_triton_backend_stores_kv_and_matches_reference(monkeypatch):
torch.testing.assert_close(actual, expected, atol=2e-5, rtol=2e-5)
[email protected](not torch.cuda.is_available(), reason="Triton attention needs CUDA")
+def test_triton_backend_applies_the_batch_block_ends_only_when_the_spec_asks(monkeypatch):
+ from freetoken.attention import AttentionSpec
+ from freetoken.attention.triton import TritonAttentionBackend
+
+ class FakeKVCache:
+ def __init__(self, device: torch.device, head_dim: int):
+ self.device = device
+ self.dtype = torch.bfloat16
+ self.k = torch.randn(3, 1, head_dim, device=device, dtype=torch.bfloat16)
+ self.v = torch.randn(3, 1, head_dim, device=device, dtype=torch.bfloat16)
+
+ def store_kv(self, k, v, out_loc, layer_id):
+ self.k[out_loc.to(torch.long)] = k.view(k.shape[0], 1, -1)
+ self.v[out_loc.to(torch.long)] = v.view(v.shape[0], 1, -1)
+
+ def k_cache(self, layer_id):
+ return self.k
+
+ def v_cache(self, layer_id):
+ return self.v
+
+ device = torch.device("cuda")
+ head_dim = 256
+ kv_cache = FakeKVCache(device, head_dim)
+ ctx = SimpleNamespace(kv_cache=kv_cache, page_table=torch.tensor([[0, 1, 2]], dtype=torch.int32, device=device))
+ monkeypatch.setattr("freetoken.attention.triton.get_global_ctx", lambda: ctx)
+ backend = TritonAttentionBackend(SimpleNamespace())
+ # one request: cached token 0, chunk tokens 1 and 2 forming one image span [1, 3)
+ batch = SimpleNamespace(
+ padded_reqs=[SimpleNamespace(extend_len=2, device_len=3, cached_len=1, table_idx=0)],
+ positions=torch.tensor([1, 2], dtype=torch.int64, device=device),
+ out_loc=torch.tensor([1, 2], dtype=torch.int32, device=device),
+ mm_block_ends=torch.tensor([3, 3], dtype=torch.int32, device=device),
+ )
+ # bf16 takes the extend kernel path; fp32 would fall back to the paged kernel, which has no block mask
+ q = torch.randn(2, 2, head_dim, device=device, dtype=torch.bfloat16)
+ k = torch.randn(2, head_dim, device=device, dtype=torch.bfloat16)
+ v = torch.randn(2, head_dim, device=device, dtype=torch.bfloat16)
+ backend.prepare_metadata(batch)
+ md = batch.attn_metadata
+ ref = lambda ends: _reference_paged_attention(
+ q, kv_cache.k, kv_cache.v, md.indptr, md.indices, md.q_to_req, md.q_positions, head_dim**-0.5, None, block_ends=ends
+ )
+ causal = backend.forward(q, k, v, layer_id=0, batch=batch, attn_spec=AttentionSpec(sm_scale=head_dim**-0.5))
+ torch.testing.assert_close(causal.float(), ref(None).float(), atol=2e-2, rtol=2e-2)
+ blocks = backend.forward(q, k, v, layer_id=0, batch=batch, attn_spec=AttentionSpec(sm_scale=head_dim**-0.5, bidirectional_mm_blocks=True))
+ torch.testing.assert_close(blocks.float(), ref(batch.mm_block_ends).float(), atol=2e-2, rtol=2e-2)
+ assert not torch.allclose(blocks[0].float(), causal[0].float(), atol=2e-2, rtol=2e-2) # token 1 now sees token 2
+
+
@pytest.mark.skipif(not torch.cuda.is_available(), reason="Triton attention needs CUDA")
def test_triton_backend_replay_metadata_uses_capture_buffers(monkeypatch):
from freetoken.attention.triton import TritonAttentionBackend, TritonMetadata
diff --git a/tests/mm/test_processor.py b/tests/mm/test_processor.py
index 87559d29..6f5fe703 100644
--- a/tests/mm/test_processor.py
+++ b/tests/mm/test_processor.py
@@ -12,6 +12,7 @@ from freetoken.message import MMItem
from freetoken.mm import MM_PAD_SHIFT_VALUE
from freetoken.mm.config import MultimodalConfig
from freetoken.mm.processor import MMProcessor, PromptReplacement, image_positions
+from freetoken.mm.processors.gemma4 import Gemma4MMProcessor, Gemma4UnifiedMMProcessor
from freetoken.mm.processors.qwen_vl import QwenVLMMProcessor
PLACEHOLDER = 7
@@ -169,18 +170,127 @@ def test_qwen_processor_positions_follow_the_grid():
assert delta == 8 + 1 - 17
+def _gemma_config():
+ return SimpleNamespace(
+ architectures=["Gemma4ForConditionalGeneration"],
+ image_token_id=258880,
+ boi_token_id=255999,
+ eoi_token_id=258882,
+ vision_soft_tokens_per_image=280,
+ vision_config=SimpleNamespace(patch_size=16, pooling_kernel_size=3),
+ text_config=SimpleNamespace(vocab_size=262144),
+ )
+
+
+class _FakeGemmaImageProcessor:
+ """Two soft tokens (18 valid patches) padded out to a 2520-row batch, like the real processor."""
+
+ def __init__(self):
+ self.calls = []
+
+ def __call__(self, images, return_tensors, max_soft_tokens=None):
+ self.calls.append(max_soft_tokens)
+ rows = 2520
+ pixels = torch.zeros(1, rows, 768)
+ pixels[0, :18] = 0.75
+ positions = torch.full((1, rows, 2), -1, dtype=torch.int64)
+ positions[0, :18, 0] = torch.arange(18) % 6
+ positions[0, :18, 1] = torch.arange(18) // 6
+ return {"pixel_values": pixels, "image_position_ids": positions, "num_soft_tokens_per_image": torch.tensor([2])}
+
+
+def _gemma_processor(mm=None):
+ proc = Gemma4MMProcessor(_gemma_config(), "/nonexistent", mm or MultimodalConfig())
+ fake = _FakeGemmaImageProcessor()
+ proc._image_processor = lambda: fake
+ return proc, fake
+
+
+def test_gemma_items_strip_padding_and_wrap_the_soft_tokens_in_boi_eoi():
+ proc, _ = _gemma_processor()
+ (item,) = proc.process([object()])
+ assert item.feature.shape == (18, 768) and item.feature.dtype == torch.float32
+ assert item.feature[0, 0].item() == 0.75 # the [0, 1] pixels of the reference processor; the tower maps them to [-1, 1]
+ assert item.position_ids.shape == (18, 2) and item.position_ids[7].tolist() == [1, 1]
+ assert item.num_soft_tokens == 2 and item.pad_value >= MM_PAD_SHIFT_VALUE
+ repl = proc.prompt_replacement(item)
+ assert proc.placeholder == [258880]
+ assert repl.full == [255999, 258880, 258880, 258882] and repl.embed_spans() == [[1, 3]]
+ assert proc.positions(10, [item]) is None
+
+
+def test_gemma_image_max_tokens_picks_an_accepted_soft_token_budget():
+ for budget, tier in ((910, 560), (70, 70), (None, None)): # the largest accepted budget within the limit
+ proc, fake = _gemma_processor(MultimodalConfig(image_max_tokens=budget))
+ proc.process([object()])
+ assert fake.calls == [tier]
+ with pytest.raises(ValueError, match="below the smallest"): # a maximum no budget honors is refused at start-up instead of rounded up
+ _gemma_processor(MultimodalConfig(image_max_tokens=60))
+ proc, _ = _gemma_processor(MultimodalConfig(image_min_tokens=500, processor_kwargs={"max_soft_tokens": 1120}))
+ # every image is scaled to its budget, so the minimum has no effect; an explicit kwarg wins over the budget
+ assert proc.get_mm_processor_kwargs(proc.mm) == {"return_tensors": "pt", "max_soft_tokens": 1120}
+
+
+def test_gemma_dummy_item_is_one_pooled_soft_token():
+ proc, _ = _gemma_processor()
+ (dummy,) = proc.dummy_items(torch.bfloat16, torch.device("cpu"))
+ dummy.validate()
+ assert dummy.feature.shape == (9, 768) and dummy.num_tokens == 1 and dummy.num_soft_tokens == 1
+ assert dummy.position_ids.tolist() == [[x, y] for y in range(3) for x in range(3)]
+
+
+def _gemma_unified_config():
+ return SimpleNamespace(
+ architectures=["Gemma4UnifiedForConditionalGeneration"],
+ image_token_id=258880,
+ boi_token_id=255999,
+ eoi_token_id=258882,
+ vision_config=SimpleNamespace(patch_size=16, pooling_kernel_size=3, model_patch_size=48, num_soft_tokens=280),
+ text_config=SimpleNamespace(vocab_size=262144),
+ )
+
+
+class _FakeGemmaUnifiedImageProcessor:
+ """Six super-patches padded out to the budget's rows, like the real processor: one row per soft token."""
+
+ def __call__(self, images, return_tensors, max_soft_tokens=None):
+ rows = max_soft_tokens or 280
+ pixels = torch.zeros(1, rows, 6912)
+ pixels[0, :6] = 0.25
+ positions = torch.full((1, rows, 2), -1, dtype=torch.int64)
+ positions[0, :6, 0] = torch.arange(6) % 3
+ positions[0, :6, 1] = torch.arange(6) // 3
+ return {"pixel_values": pixels, "image_position_ids": positions, "num_soft_tokens_per_image": torch.tensor([6])}
+
+
+def test_gemma_unified_items_are_one_row_per_soft_token():
+ proc = Gemma4UnifiedMMProcessor(_gemma_unified_config(), "/nonexistent", MultimodalConfig(image_max_tokens=100))
+ proc._image_processor = lambda: _FakeGemmaUnifiedImageProcessor()
+ (item,) = proc.process([object()])
+ assert item.feature.shape == (6, 6912) and item.num_soft_tokens == 6 and item.position_ids[4].tolist() == [1, 1]
+ assert proc.prompt_replacement(item).full == [255999] + [258880] * 6 + [258882]
+ assert proc.get_mm_processor_kwargs(proc.mm) == {"return_tensors": "pt", "max_soft_tokens": 70} # the same budgets as the tower release
+ (dummy,) = proc.dummy_items(torch.bfloat16, torch.device("cpu"))
+ dummy.validate()
+ assert dummy.feature.shape == (1, 6912) and dummy.num_tokens == 1 and dummy.position_ids.tolist() == [[0, 0]]
+
+
def test_registry_resolves_by_architecture(monkeypatch):
import freetoken.utils
from freetoken.mm.processor import get_mm_processor
configs = {
"/qwen": _hf_config(),
+ "/gemma": _gemma_config(),
+ "/gemma-unified": _gemma_unified_config(),
"/text-only": _hf_config(vision=False),
"/unknown-vlm": _hf_config(arch="SomeOtherForConditionalGeneration"),
}
monkeypatch.setattr(freetoken.utils, "cached_load_hf_config", lambda p: configs[p])
assert isinstance(get_mm_processor("/qwen"), QwenVLMMProcessor)
assert get_mm_processor("/qwen", MultimodalConfig(disabled_encoders=frozenset({"vision"}))) is None # --mm-disable vision
+ assert isinstance(get_mm_processor("/gemma"), Gemma4MMProcessor)
+ assert isinstance(get_mm_processor("/gemma-unified"), Gemma4UnifiedMMProcessor)
assert get_mm_processor("/text-only") is None
assert get_mm_processor("/unknown-vlm") is None
assert get_mm_processor("/missing") is None # a config that fails to load means no vision
diff --git a/tests/models/test_gemma4_vision.py b/tests/models/test_gemma4_vision.py
new file mode 100644
index 00000000..085afb54
--- /dev/null
+++ b/tests/models/test_gemma4_vision.py
@@ -0,0 +1,227 @@
+"""Gemma 4 vision: shapes and host streaming on a tiny random tower, the unified release's embedder, parity with the reference on real checkpoints."""
+
+from __future__ import annotations
+
+import glob
+import os
+
+import pytest
+import torch
+
+from freetoken.models.gemma4.config import UnifiedVisionConfig, VisionConfig
+from freetoken.models.gemma4.vision import Gemma4MultimodalEmbedder, Gemma4UnifiedVisionEmbedder, Gemma4VisionModel
+
+pytestmark = pytest.mark.skipif(not torch.cuda.is_available(), reason="needs CUDA")
+
+CHECKPOINT = os.environ.get("FREETOKEN_GEMMA4_MODEL", "")
+UNIFIED_CHECKPOINT = os.environ.get("FREETOKEN_GEMMA4_UNIFIED_MODEL", "")
+needs_checkpoint = pytest.mark.skipif(not os.path.exists(os.path.join(CHECKPOINT, "config.json")), reason="FREETOKEN_GEMMA4_MODEL not set")
+needs_unified_checkpoint = pytest.mark.skipif(not os.path.exists(os.path.join(UNIFIED_CHECKPOINT, "config.json")), reason="FREETOKEN_GEMMA4_UNIFIED_MODEL not set")
+
+
+def _tiny_vc():
+ return VisionConfig(
+ hidden_size=32, num_layers=2, num_heads=2, num_kv_heads=2, head_dim=16, intermediate_size=64, patch_size=16,
+ position_embedding_size=64, pooling_kernel_size=3, rms_norm_eps=1e-6, rope_theta=100.0, hidden_act="gelu_tanh",
+ standardize=True, use_clipped_linears=False, text_hidden_size=48,
+ )
+
+
+def _build(vc):
+ torch.set_default_dtype(torch.bfloat16)
+ try:
+ with torch.device("cuda"):
+ tower, embedder = Gemma4VisionModel(vc), Gemma4MultimodalEmbedder(vc)
+ finally:
+ torch.set_default_dtype(torch.float32)
+ for p in list(tower.state_dict().values()) + [embedder.embedding_projection.weight]:
+ p.normal_(0, 0.02)
+ return tower, embedder
+
+
+def _grid_positions(width, height):
+ ys, xs = torch.meshgrid(torch.arange(height), torch.arange(width), indexing="ij")
+ return torch.stack([xs.reshape(-1), ys.reshape(-1)], dim=-1)
+
+
+def test_tiny_tower_pools_nine_patches_per_soft_token():
+ tower, embedder = _build(_tiny_vc())
+ feature = torch.randn(1, 18, 768, dtype=torch.bfloat16, device="cuda")
+ positions = _grid_positions(6, 3)[None].cuda()
+ out = embedder.forward(tower.forward(feature, positions))
+ assert out.shape == (2, 48) and out.dtype == torch.bfloat16
+
+
+def test_host_streamed_weights_compute_the_same_output():
+ tower, _ = _build(_tiny_vc())
+ feature = torch.randn(1, 18, 768, dtype=torch.bfloat16, device="cuda")
+ positions = _grid_positions(6, 3)[None].cuda()
+ resident = tower.forward(feature, positions)
+ keys = tower.state_dict().keys()
+ tower.place_weights("host")
+ assert tower.state_dict().keys() == keys
+ assert tower.state_dict()["encoder.layers.0.self_attn.q_proj.weight"].device.type == "cpu"
+ assert tower.state_dict()["patch_embedder.input_proj.weight"].device.type == "cuda"
+ for _ in range(2): # a second forward reuses the staging buffers behind the release events
+ assert torch.equal(tower.forward(feature, positions), resident)
+ tower.place_weights("gpu")
+ assert tower.state_dict()["encoder.layers.0.self_attn.q_proj.weight"].device.type == "cuda"
+ assert torch.equal(tower.forward(feature, positions), resident)
+
+
+@needs_checkpoint
+def test_vision_mode_marks_only_the_sliding_layers_bidirectional():
+ from transformers import AutoConfig
+
+ from freetoken.models.config import FullAttentionGroupConfig, SWAAttentionGroupConfig
+ from freetoken.models.gemma4 import parse_config
+
+ groups = {type(g): g for g in parse_config(AutoConfig.from_pretrained(CHECKPOINT)).attention_groups}
+ assert groups[SWAAttentionGroupConfig].bidirectional_mm_blocks
+ assert not hasattr(groups[FullAttentionGroupConfig], "bidirectional_mm_blocks")
+
+
+@needs_checkpoint
+def test_tower_matches_the_reference_on_a_checkpoint():
+ """The bf16 tower stays within the reference's own bf16 noise around its fp32 result."""
+ import json
+
+ from PIL import Image, ImageDraw
+ from safetensors import safe_open
+ from transformers import AutoConfig, AutoImageProcessor
+ from transformers.models.gemma4.modeling_gemma4 import Gemma4MultimodalEmbedder as HFEmbedder
+ from transformers.models.gemma4.modeling_gemma4 import Gemma4VisionModel as HFVision
+ from transformers.models.gemma4.modeling_gemma4 import Gemma4VisionRotaryEmbedding
+
+ from freetoken.mm.config import MultimodalConfig
+ from freetoken.mm.processors.gemma4 import Gemma4MMProcessor
+ from freetoken.models.gemma4 import parse_config
+
+ torch.set_float32_matmul_precision("highest")
+ hf_config = AutoConfig.from_pretrained(CHECKPOINT)
+ hf_config.vision_config._attn_implementation = "sdpa"
+ index = json.load(open(os.path.join(CHECKPOINT, "model.safetensors.index.json")))["weight_map"]
+ tower_weights, embed_weights = {}, {}
+ for name, file in index.items():
+ if name.startswith(("model.vision_tower.", "model.embed_vision.")):
+ with safe_open(os.path.join(CHECKPOINT, file), "pt") as f:
+ (tower_weights if "vision_tower" in name else embed_weights)[name.split(".", 2)[2]] = f.get_tensor(name)
+
+ def reference(dtype):
+ vision = HFVision(hf_config.vision_config).to("cuda", dtype).eval()
+ vision.load_state_dict({k: v.to(dtype) for k, v in tower_weights.items()}, strict=True)
+ # .to(dtype) also casts the rotary table, which from_pretrained keeps in fp32
+ vision.encoder.rotary_emb.inv_freq = Gemma4VisionRotaryEmbedding.compute_default_rope_parameters(hf_config.vision_config)[0].cuda()
+ embed = HFEmbedder(hf_config.vision_config, hf_config.text_config).to("cuda", dtype).eval()
+ embed.load_state_dict({k: v.to(dtype) for k, v in embed_weights.items()}, strict=True)
+ return lambda pixels, positions: embed(vision(pixel_values=pixels, pixel_position_ids=positions).last_hidden_state)
+
+ config = parse_config(hf_config)
+ tower, embedder = _build(config.vision_config)
+ tower.load_state_dict({k.replace(".linear.", "."): v.cuda() for k, v in tower_weights.items()})
+ embedder.load_state_dict({k: v.cuda() for k, v in embed_weights.items()})
+
+ img = Image.new("RGB", (640, 400), (30, 144, 255))
+ draw = ImageDraw.Draw(img)
+ draw.ellipse((100, 60, 340, 300), fill=(255, 215, 0))
+ draw.rectangle((40, 340, 600, 380), fill=(220, 20, 60))
+ out = AutoImageProcessor.from_pretrained(CHECKPOINT)(images=img, return_tensors="pt")
+ pixels, positions = out["pixel_values"].cuda(), out["image_position_ids"].cuda()
+ with torch.inference_mode():
+ ref32 = reference(torch.float32)(pixels, positions)
+ ref16 = reference(torch.bfloat16)(pixels, positions).float()
+ (item,) = Gemma4MMProcessor(hf_config, CHECKPOINT, MultimodalConfig()).process([img])
+ ours = embedder.forward(tower.forward(item.feature.cuda()[None], item.position_ids.cuda().long()[None])).float()
+ assert ours.shape == ref32.shape == (item.num_soft_tokens, config.hidden_size)
+ cos = torch.nn.functional.cosine_similarity
+ noise, error = (ref32 - ref16).norm(), (ref32 - ours).norm()
+ print(f"gemma4 tower: ours vs fp32 mean cos {cos(ref32, ours, dim=-1).mean():.5f}, error/noise {error / noise:.3f}")
+ assert cos(ref32, ours, dim=-1).mean() > 0.99
+ assert error <= 1.2 * noise
+
+
+# the 12B gemma4_unified release: a linear embedder in place of the tower
+
+
+def _tiny_unified_vc():
+ return UnifiedVisionConfig(hidden_size=32, patch_dim=48, posemb_size=16, layer_norm_eps=1e-5, rms_norm_eps=1e-6, text_hidden_size=40)
+
+
+def _build_unified(vc):
+ torch.set_default_dtype(torch.bfloat16)
+ try:
+ with torch.device("cuda"):
+ embedder, proj = Gemma4UnifiedVisionEmbedder(vc), Gemma4MultimodalEmbedder(vc)
+ finally:
+ torch.set_default_dtype(torch.float32)
+ for p in list(embedder.state_dict().values()) + [proj.embedding_projection.weight]:
+ p.normal_(0, 0.02)
+ return embedder, proj
+
+
+def test_tiny_unified_embedder_maps_one_super_patch_to_one_soft_token():
+ embedder, proj = _build_unified(_tiny_unified_vc())
+ feature = torch.rand(1, 6, 48, device="cuda")
+ positions = torch.tensor([[[x, y] for y in range(2) for x in range(3)]], device="cuda")
+ out = proj.forward(embedder.forward(feature, positions))
+ assert out.shape == (1, 6, 40) and out.dtype == torch.bfloat16
+
+
+@needs_unified_checkpoint
+def test_unified_embedder_matches_the_reference_on_a_checkpoint():
+ """The bf16 embedder stays within the reference's own bf16 noise around its fp32 result."""
+ from PIL import Image, ImageDraw
+ from safetensors import safe_open
+ from transformers import AutoConfig, AutoImageProcessor
+ from transformers.models.gemma4_unified.modeling_gemma4_unified import Gemma4UnifiedVisionEmbedder as HFEmbedder
+
+ from freetoken.mm.config import MultimodalConfig
+ from freetoken.mm.processors.gemma4 import Gemma4UnifiedMMProcessor
+ from freetoken.models.gemma4 import parse_config
+
+ torch.set_float32_matmul_precision("highest")
+ hf_config = AutoConfig.from_pretrained(UNIFIED_CHECKPOINT)
+ weights = {}
+ for file in glob.glob(os.path.join(UNIFIED_CHECKPOINT, "*.safetensors")):
+ with safe_open(file, "pt") as f:
+ for name in f.keys():
+ if name.startswith(("model.vision_embedder.", "model.embed_vision.")):
+ weights[name[len("model."):]] = f.get_tensor(name)
+ embedder_weights = {k[len("vision_embedder."):]: v for k, v in weights.items() if k.startswith("vision_embedder.")}
+ projection = weights["embed_vision.embedding_projection.weight"]
+
+ def reference(dtype):
+ ref = HFEmbedder(hf_config.vision_config, hf_config.text_config).to("cuda", dtype).eval()
+ ref.load_state_dict({**{k: v.to(dtype) for k, v in embedder_weights.items()}, "multimodal_embedder.embedding_projection.weight": projection.to(dtype)}, strict=True)
+ def run(pixels, positions):
+ out = ref(pixel_values=pixels, image_position_ids=positions)
+ return out.pooler_output if hasattr(out, "pooler_output") else out # a plain tensor on transformers 5.15
+
+ return run
+
+ config = parse_config(hf_config)
+ embedder, proj = _build_unified(config.vision_config)
+ embedder.load_state_dict({k: v.cuda() for k, v in embedder_weights.items()})
+ proj.load_state_dict({"embedding_projection.weight": projection.cuda()})
+
+ img = Image.new("RGB", (640, 400), (30, 144, 255))
+ draw = ImageDraw.Draw(img)
+ draw.ellipse((100, 60, 340, 300), fill=(255, 215, 0))
+ draw.rectangle((40, 340, 600, 380), fill=(220, 20, 60))
+ out = AutoImageProcessor.from_pretrained(UNIFIED_CHECKPOINT)(images=img, return_tensors="pt")
+ pixels, positions = out["pixel_values"].cuda(), out["image_position_ids"].cuda()
+ valid = (positions[0] != -1).all(dim=-1)
+ with torch.inference_mode():
+ ref32 = reference(torch.float32)(pixels, positions)[0][valid]
+ ref16 = reference(torch.bfloat16)(pixels, positions)[0][valid].float()
+ (item,) = Gemma4UnifiedMMProcessor(hf_config, UNIFIED_CHECKPOINT, MultimodalConfig()).process([img])
+ ours = proj.forward(embedder.forward(item.feature.cuda()[None], item.position_ids.cuda().long()[None]))[0].float()
+ assert ours.shape == ref32.shape == (item.num_soft_tokens, hf_config.text_config.hidden_size)
+ cos = torch.nn.functional.cosine_similarity
+ noise, error = (ref32 - ref16).norm(), (ref32 - ours).norm()
+ print(f"gemma4 unified embedder: ours vs fp32 mean cos {cos(ref32, ours, dim=-1).mean():.5f}, error/noise {error / noise:.3f}")
+ assert cos(ref32, ours, dim=-1).mean() > 0.99
+ assert error <= 1.2 * noise
diff --git a/tests/scheduler/test_mm.py b/tests/scheduler/test_mm.py
index 1910adf2..13447634 100644
--- a/tests/scheduler/test_mm.py
+++ b/tests/scheduler/test_mm.py
@@ -6,7 +6,7 @@ import torch
from freetoken.message import MMItem
from freetoken.mm.encoder_cache import EncoderCache
-from freetoken.scheduler.mm import mm_rows_after, plan_mm_batch, plan_mm_chunk
+from freetoken.scheduler.mm import cut_image_spans, mm_chunk_end, mm_rows_after, plan_mm_batch, plan_mm_chunk
CPU = torch.device("cpu")
@@ -86,7 +86,29 @@ def test_batch_rows_follow_the_reqs_in_batch_order():
# req 1: 6 tokens, image on [2, 5); req 2: chunk [4, 10) of a prompt whose image spans [3, 8)
a = SimpleNamespace(uid=1, mm_items=[_item(7, [[2, 5]])], cached_len=0, device_len=6, extend_len=6)
b = SimpleNamespace(uid=2, mm_items=[_item(8, [[3, 8]])], cached_len=4, device_len=10, extend_len=6)
- jobs, plan, rows = plan_mm_batch([a, b], None)
+ jobs, plan, rows, block_ends = plan_mm_batch([a, b], None)
assert [j.hash for j in jobs] == [7, 8]
assert plan == [(1, 7, 0, 3, 3, 2), (2, 8, 1, 5, 5, 0)]
assert rows == [2, 3, 4, 6, 7, 8, 9] # req 2 starts at batch row 6; its image rows 1..5 land on its first four tokens
+ assert block_ends == [0, 0, 5, 5, 5, 0, 8, 8, 8, 8, 0, 0] # every image row carries its span's end in request positions
+
+
+def test_chunk_end_never_lands_inside_an_image_span():
+ items = [_item(1, [[10, 20]]), _item(2, [[22, 40]])]
+ assert mm_chunk_end(items, 0, 8, 1) == 8 # before the images: untouched
+ assert mm_chunk_end(items, 0, 15, 1) == 10 # inside the first image: back to its start
+ assert mm_chunk_end(items, 0, 30, 1) == 22 # inside the second
+ assert mm_chunk_end(items, 0, 30, 8) == 8 # the aligned start 16 lands in the first image, so back again
+ assert mm_chunk_end(items, 0, 20, 1) == 20 and mm_chunk_end(items, 0, 40, 1) == 40 # ends on a boundary are fine
+ assert mm_chunk_end(items, 22, 30, 1) == 30 # the image begins at the chunk start: longer than the budget, split it
+ assert mm_chunk_end(items, 12, 15, 1) == 15 # an image already split by an earlier chunk keeps going
+
+
+def test_cut_image_spans_reports_the_image_a_chunk_ends_inside():
+ from types import SimpleNamespace
+
+ image = MMItem(modality="image", hash=1, pad_value=1, offsets=[[10, 20], [22, 40]], feature=torch.zeros(1))
+ req = lambda device_len: SimpleNamespace(mm_items=[image], device_len=device_len)
+ text = SimpleNamespace(mm_items=None, device_len=5)
+ assert cut_image_spans([text, req(30)]) == [(22, 40)]
+ assert cut_image_spans([req(8)]) == cut_image_spans([req(20)]) == cut_image_spans([req(40)]) == [] # outside every span or on a boundary
diff --git a/tests/scheduler/test_scheduler_chunked_prefill.py b/tests/scheduler/test_scheduler_chunked_prefill.py
index f3e510ea..a4c7e065 100644
--- a/tests/scheduler/test_scheduler_chunked_prefill.py
+++ b/tests/scheduler/test_scheduler_chunked_prefill.py
@@ -150,3 +150,90 @@ def test_batched_prefill_carries_each_new_prompt_admission():
batch = pm.schedule_next_batch(16)
assert batch is not None
assert batch.prompt_admissions == [(1, 3, 0), (2, 5, 0)]
+
+
+def test_chunk_ends_before_an_image_it_would_cut_on_bidirectional_models():
+ from freetoken.core import SamplingParams
+ from freetoken.message import MMItem
+ from freetoken.scheduler.utils import PendingReq
+
+ def chunk_lens(keep_images_whole):
+ cm, tm, dm, pm = _build_managers(num_pages=64)
+ pm.keep_images_whole = keep_images_whole
+ image = MMItem(modality="image", hash=1, pad_value=1, offsets=[[10, 20]], feature=torch.zeros(1))
+ pm.pending_list = [PendingReq(uid=UID, input_ids=torch.arange(30, dtype=torch.int32),
+ sampling_params=SamplingParams(max_tokens=4), mm_items=[image])]
+ lens = []
+ while pm.runnable:
+ batch = pm.schedule_next_batch(15)
+ cm.allocate_paged(batch.reqs)
+ for r in batch.reqs:
+ lens.append(r.extend_len)
+ r.complete_one()
+ return lens
+
+ assert chunk_lens(True) == [10, 15, 5] # a 15-token chunk would cut the image at 10..20: stop at 10, the image goes whole into the next chunk
+ assert chunk_lens(False) == [15, 15] # causal models lose nothing to a cut image and keep the plain chunking
+
+
+def test_a_shared_pass_defers_an_image_it_cannot_hold_whole():
+ """Another request took most of the pass: the image waits for a pass of its own instead of being cut at the leftover."""
+ from freetoken.core import SamplingParams
+ from freetoken.message import MMItem
+ from freetoken.scheduler.utils import PendingReq
+
+ cm, tm, dm, pm = _build_managers(num_pages=128)
+ pm.keep_images_whole = True
+ text = PendingReq(uid=UID, input_ids=torch.arange(35, dtype=torch.int32), sampling_params=SamplingParams(max_tokens=4))
+ image = MMItem(modality="image", hash=1, pad_value=1, offsets=[[0, 20]], feature=torch.zeros(1))
+ with_image = PendingReq(uid=UID + 1, input_ids=torch.arange(100, 130, dtype=torch.int32),
+ sampling_params=SamplingParams(max_tokens=4), mm_items=[image])
+ pm.pending_list = [text, with_image]
+ first = pm.schedule_next_batch(40)
+ assert [(r.uid, r.extend_len) for r in first.reqs] == [(UID, 35)] # the 5 tokens left cannot hold the 20-token image
+ cm.allocate_paged(first.reqs)
+ for r in first.reqs:
+ r.complete_one()
+ second = pm.schedule_next_batch(40)
+ assert [(r.uid, r.extend_len) for r in second.reqs] == [(UID + 1, 30)]
+
+
+def test_the_sliding_window_pool_cap_cannot_recut_an_image():
+ """The image boundary is decided after the pool cap: a pool of 5000 tokens ends the chunk before the image at 4900, not inside it."""
+ from freetoken.core import SamplingParams
+ from freetoken.distributed import set_tp_info, try_get_tp_info
+ from freetoken.kvcache.hybrid_swa_pool import HybridSWAKVCache
+ from freetoken.message import MMItem
+ from freetoken.models.config import KVCacheGroupSpec
+ from freetoken.scheduler.cache import CacheManager
+ from freetoken.scheduler.decode import DecodeManager
+ from freetoken.scheduler.mm import cut_image_spans
+ from freetoken.scheduler.prefill import PrefillManager
+ from freetoken.scheduler.table import TableManager
+ from freetoken.scheduler.utils import PendingReq
+
+ _setup_context()
+ if try_get_tp_info() is None:
+ set_tp_info(rank=0, size=1)
+ groups = (
+ KVCacheGroupSpec(name="full", layer_ids=(1,), num_kv_heads=1, head_dim=8, sliding_window=None),
+ KVCacheGroupSpec(name="swa", layer_ids=(0,), num_kv_heads=1, head_dim=8, sliding_window=1024),
+ )
+ pool = HybridSWAKVCache(groups=groups, num_layers=2, num_full_pages=8192, page_size=1, dtype=torch.bfloat16,
+ device=torch.device("cpu"), num_swa_tokens=5000)
+ pt = torch.zeros((MAX_RUNNING + 1, 8192), dtype=torch.int32)
+ cm = CacheManager(num_pages=8192, page_size=1, page_table=pt, type="swa_radix", swa_pool=pool, sliding_window_size=1024)
+ pm = PrefillManager(cm, TableManager(max_running_reqs=MAX_RUNNING, page_table=pt), DecodeManager(1), keep_images_whole=True)
+ image = MMItem(modality="image", hash=1, pad_value=1, offsets=[[4900, 5156]], feature=torch.zeros(1))
+ pm.pending_list = [PendingReq(uid=UID, input_ids=torch.arange(6000, dtype=torch.int32),
+ sampling_params=SamplingParams(max_tokens=1), mm_items=[image])]
+ ends = []
+ while pm.runnable:
+ batch = pm.schedule_next_batch(8192)
+ assert cut_image_spans(batch.reqs) == []
+ ends.append(batch.reqs[0].device_len)
+ cm.free_swa_out_of_window_extend(batch.reqs)
+ cm.allocate_paged(batch.reqs)
+ for r in batch.reqs:
+ r.complete_one()
+ assert ends[0] == 4900 and ends[-1] == 6000
diff --git a/tests/tokenizer/test_mm_tokenize.py b/tests/tokenizer/test_mm_tokenize.py
index 5dbf2822..ea3bcfa9 100644
--- a/tests/tokenizer/test_mm_tokenize.py
+++ b/tests/tokenizer/test_mm_tokenize.py
@@ -16,9 +16,15 @@ MODELS = [
if os.path.exists(os.path.join(p, "config.json"))
]
-pytestmark = pytest.mark.skipif(
- not MODELS, reason="neither FREETOKEN_QWEN36_MODEL nor FREETOKEN_QWEN3VL_MODEL points at a checkpoint"
-)
+GEMMA = os.environ.get("FREETOKEN_GEMMA4_MODEL", "")
+
+pytestmark = [
+ pytest.mark.needs_weights,
+ pytest.mark.skipif(
+ not MODELS and not os.path.exists(os.path.join(GEMMA, "config.json")),
+ reason="no FREETOKEN_QWEN36_MODEL / FREETOKEN_QWEN3VL_MODEL / FREETOKEN_GEMMA4_MODEL checkpoint",
+ ),
+]
def _png(w, h):
@@ -103,3 +109,34 @@ def test_count_prompt_tokens_counts_the_expanded_image(manager):
counted = asyncio.run(count_prompt_tokens(messages, None, {}, state))
expanded = manager.tokenize([_msg([png])])[0].input_ids.numel()
assert counted == expanded and counted > 256
+
+
[email protected](not os.path.exists(os.path.join(GEMMA, "config.json")), reason="FREETOKEN_GEMMA4_MODEL not set")
+def test_gemma_wraps_the_pad_span_in_boi_eoi():
+ from freetoken.mm.processor import get_mm_processor
+ from freetoken.tokenizer.tokenize import TokenizeManager
+ from freetoken.utils import cached_load_hf_config
+ from freetoken.utils.hf import load_tokenizer
+
+ hf = cached_load_hf_config(GEMMA)
+ r = TokenizeManager(load_tokenizer(GEMMA), get_mm_processor(GEMMA)).tokenize([_msg([_png(640, 400)])])[0]
+ item = r.mm_items[0]
+ ((start, end),) = item.offsets
+ ids = r.input_ids
+ assert ids[start - 1].item() == hf.boi_token_id and ids[end].item() == hf.eoi_token_id
+ assert end - start == item.num_soft_tokens and r.mrope_positions is None
+ assert bool((ids[start:end] == item.pad_value).all())
+
+
[email protected](not os.path.exists(os.path.join(GEMMA, "config.json")), reason="FREETOKEN_GEMMA4_MODEL not set")
+def test_gemma_image_max_tokens_picks_a_smaller_soft_token_budget():
+ from freetoken.mm.config import MultimodalConfig
+ from freetoken.mm.processor import get_mm_processor
+ from freetoken.tokenizer.tokenize import TokenizeManager
+ from freetoken.utils.hf import load_tokenizer
+
+ tokenizer = load_tokenizer(GEMMA)
+ default = TokenizeManager(tokenizer, get_mm_processor(GEMMA)).tokenize([_msg([_png(640, 400)])])[0]
+ budgeted = TokenizeManager(tokenizer, get_mm_processor(GEMMA, MultimodalConfig(image_max_tokens=100))).tokenize([_msg([_png(640, 400)])])[0]
+ # the processor scales the image to the budget as far as its aspect ratio allows: 10x6 pooled cells for 640x400
+ assert budgeted.mm_items[0].num_soft_tokens == 60 < default.mm_items[0].num_soft_tokens