Skip to content

FreeToken brings datacenter-scale model serving to your desktop. Run massive models locally, fast and efficiently.

feat(moe): serve NVFP4 and MXFP4 experts fused through the expert banks (#601)

Xiaoze Fan189c8ad0c335
diff --git a/docs/cli.md b/docs/cli.md
index fb5e613f..deeda184 100644
--- a/docs/cli.md
+++ b/docs/cli.md
@@ -80,7 +80,7 @@ See [models.md](models.md#moe-strategies) for what each strategy does.
| Flag | Default | Meaning |
|---|---|---|
-| `--moe-strategy` | auto | `fused`/`offload`/`cpu`/`hybrid`; auto → offload, or hybrid with a `ft bench bw` profile. `--moe-backend` is the deprecated old spelling |
+| `--moe-strategy` | auto | `fused`/`offload`/`cpu`/`hybrid`; auto → offload, or hybrid with a `ft bench bw` profile; fused on unified-memory GPUs (GB10), see [models.md](models.md#unified-memory-gpus-gb10--dgx-spark). `--moe-backend` is the deprecated old spelling |
| `--quant-backend` | auto | Kernel per quantized layer type, `layer[.kind]=name` entries: `linear=marlin,moe=b12x` or `moe.nvfp4=triton`. A layer-level entry applies to every kind whose table lists the name |
| `--nvfp4-backend` | — | Deprecated: stands in for `--quant-backend moe.nvfp4=<marlin\|b12x\|triton>` (`flashinfer` means b12x); cannot be combined with `--quant-backend` |
| `--moe-cache-size` / `--moe-cache-rate` / `--moe-cache-auto` | auto | GPU expert-cache size as slots / fraction of all experts / sized from free VRAM (mutually exclusive; auto is enabled by default for offload-family strategies) |
@@ -179,9 +179,10 @@ ft checkpoint --model <hf_dir> --out <ftw_dir> [--dtype bfloat16] [--moe-backend
```
Converts an HF safetensors checkpoint to FTW, FreeToken's self-contained
-fast-load format; point `ft serve --model` at the output dir. `--moe-backend
-offload` (default) packs experts into offload banks; `--moe-backend triton`
-keeps them dense for resident serving. See the FTW caveats in
+fast-load format; point `ft serve --model` at the output dir. MoE experts are
+always packed into expert banks, which serve the fused and the offload
+strategies alike (GGUF and DeepSeek-V4 experts serve offload only);
+`--moe-backend` is kept for compatibility. See the FTW caveats in
[models.md](models.md#notes); FTW files from older builds can be repaired with
[scripts/ftw_hotfix.py](ftw-hotfix.md) instead of reconverting.
diff --git a/docs/models.md b/docs/models.md
index 548a363c..efd5268b 100644
--- a/docs/models.md
+++ b/docs/models.md
@@ -38,7 +38,8 @@ These families accept image input by default; pass `--text-model-only` to skip t
`ft serve --moe-strategy {auto,fused,offload,cpu,hybrid}` (`--moe-backend` is the deprecated old spelling):
-- **fused** — experts resident on GPU (needs the VRAM); never auto-selected.
+- **fused** — experts resident on GPU (needs the VRAM). Serves every expert
+ format except GGUF and DeepSeek-V4's.
- **offload** — experts live in host RAM, an LRU cache of expert slots on GPU;
misses stream over PCIe.
- **cpu** — misses are computed on the CPU instead of fetched.
@@ -46,12 +47,24 @@ These families accept image input by default; pass `--text-model-only` to skip t
CPU, overlapped. Run `ft bench bw` once per machine to calibrate the split.
- **auto** — dense models always resolve to `fused`; MoE models resolve to
`offload`, upgraded to `hybrid` when a cached `ft bench bw` profile
- recommends it.
+ recommends it. Unified-memory GPUs differ, see below.
+
+### Unified-memory GPUs (GB10 / DGX Spark)
+
+The GPU and the CPU share one DRAM, so offload only copies experts between two
+names for the same memory. On these GPUs `auto` resolves every MoE model to
+`fused`: the default is the resident path only. GGUF and DeepSeek-V4 experts
+have no resident path and stay on `offload`. Offload-only flags
+(`--moe-cpu-layers`, `--moe-cache-size` / `-rate` / `-auto`,
+`--moe-prefill-hit-d2d`, `--disable-moe-prefill-overlap`) are ignored with a
+warning; pass `--moe-strategy offload` to use them.
## Notes
- `ft checkpoint` conversion is optional — it pre-converts a checkpoint into
FreeToken's fast-load format, and `ft serve --model` auto-detects the result.
+- An FTW converted by an older build with `--moe-backend fused` or `triton` keeps
+ its experts as dense weights and no longer loads; reconvert it with `ft checkpoint`.
- FTW files converted by builds before the quantization refactor may fail to load;
see [ftw-hotfix.md](ftw-hotfix.md) for the affected checkpoints and the repair tool.
- An FTW converted before its family served images holds no vision encoder: `ft serve`
diff --git a/python/freetoken/checkpoint/__main__.py b/python/freetoken/checkpoint/__main__.py
index 2b7e35f1..0d7c2d7e 100644
--- a/python/freetoken/checkpoint/__main__.py
+++ b/python/freetoken/checkpoint/__main__.py
@@ -37,7 +37,7 @@ def main(argv: list[str] | None = None, prog: str = "freetoken.checkpoint") -> i
p.add_argument("--out", required=True, help="output FTW checkpoint dir")
p.add_argument("--dtype", choices=sorted(_DTYPES), default="bfloat16")
p.add_argument("--moe-backend", default="offload",
- help="offload (experts -> banks) or e.g. triton (experts stay dense)")
+ help="kept for compatibility: MoE experts always go to banks, which every strategy reads")
p.add_argument("--quant-backend", type=_parse_quant_backend, default=None,
help="kernel per quantized layer type, as for ft serve; the expert banks are packed for the "
"chosen MoE kernel and the server picks that kernel back up from the FTW")
diff --git a/python/freetoken/checkpoint/convert.py b/python/freetoken/checkpoint/convert.py
index a875d77f..152f3871 100644
--- a/python/freetoken/checkpoint/convert.py
+++ b/python/freetoken/checkpoint/convert.py
@@ -200,7 +200,8 @@ def convert_checkpoint(
cfg = EngineConfig(model_path=model_path, tp_info=DistributedInfo(tp.rank, tp.size),
dtype=dtype, moe_strategy=moe_backend, quant_backend=quant_backend)
mc = cfg.model_config
- offload = moe_backend == "offload" and getattr(mc, "is_moe", False)
+ # every MoE strategy reads the routed experts from the banks, fused included
+ offload = getattr(mc, "is_moe", False)
include_moe_experts = not offload
method = None
if offload:
@@ -208,6 +209,7 @@ def convert_checkpoint(
from freetoken.layers import set_rope_device
set_rope_device(dev)
+ object.__setattr__(cfg, "moe_strategy", "offload")
method = offload_expert_method(cfg)
from freetoken.utils.progress import byte_bar, count_bar
diff --git a/python/freetoken/checkpoint/ftw.py b/python/freetoken/checkpoint/ftw.py
index e2bd0de2..e31e9d82 100644
--- a/python/freetoken/checkpoint/ftw.py
+++ b/python/freetoken/checkpoint/ftw.py
@@ -432,6 +432,43 @@ def iter_ftw_weights(path: str, *, kinds=("weight",), keep: Callable[[str], bool
raise err[0]
+def _split_bank_entries(reader: FTWReader, path: str, num_layers: int):
+ """``(all, alphas, flat regions, {base: {layer: entry}})`` of the ``experts_bank`` entries; None when there are none."""
+ bank_entries = reader.entries("experts_bank")
+ if not bank_entries:
+ return None
+
+ alpha_entries = [e for e in bank_entries if e["name"] in _ALPHA_NAMES]
+ row_entries = [e for e in bank_entries if e["name"] not in _ALPHA_NAMES]
+
+ meta_layers = reader.meta("expert_bank_num_layers")
+ if meta_layers is not None and meta_layers != num_layers:
+ raise RuntimeError(
+ f"{path!r} was converted with {meta_layers} expert-bank layers but the "
+ f"model config says num_moe_layers={num_layers}; the checkpoint does not "
+ "match its config"
+ )
+
+ # Split row entries into the two layouts by name.
+ flat_entries: list[dict] = []
+ per_layer_groups: dict[str, dict[int, dict]] = {}
+ for e in row_entries:
+ m = _LAYER_ENTRY_RE.match(e["name"])
+ if m is None:
+ flat_entries.append(e)
+ continue
+ per_layer_groups.setdefault(m.group("base"), {})[int(m.group("layer"))] = e
+
+ mixed = {e["name"] for e in flat_entries} & per_layer_groups.keys()
+ assert not mixed, f"FTW bank(s) mix flat and per-layer row layouts: {sorted(mixed)}"
+ for base, by_layer in per_layer_groups.items():
+ assert sorted(by_layer) == list(range(num_layers)), (
+ f"FTW bank {base!r} has per-layer entries for layers {sorted(by_layer)}, "
+ f"expected exactly range({num_layers})"
+ )
+ return bank_entries, alpha_entries, flat_entries, per_layer_groups
+
+
def load_ftw_banks(
path: str, *, num_layers: int, workers: int = 8, chunk: int = _DEFAULT_CHUNK,
layer_residency: list[str] | None = None,
@@ -482,40 +519,20 @@ def load_ftw_banks(
return "mmap"
reader = FTWReader(path)
- bank_entries = reader.entries("experts_bank")
- if not bank_entries:
+ try:
+ split = _split_bank_entries(reader, path, num_layers)
+ except BaseException:
reader.close()
- return None
-
- alpha_entries = [e for e in bank_entries if e["name"] in _ALPHA_NAMES]
- row_entries = [e for e in bank_entries if e["name"] not in _ALPHA_NAMES]
-
- meta_layers = reader.meta("expert_bank_num_layers")
- if meta_layers is not None and meta_layers != num_layers:
+ raise
+ if split is None:
reader.close()
- raise RuntimeError(
- f"{path!r} was converted with {meta_layers} expert-bank layers but the "
- f"model config says num_moe_layers={num_layers}; the checkpoint does not "
- "match its config"
- )
+ return None
+ bank_entries, alpha_entries, flat_entries, per_layer_groups = split
# Alphas: unchanged, one flat HostBank per entry.
alpha_specs = {e["name"]: (tuple(e["shape"]), _dtype_of(e["dtype"])) for e in alpha_entries}
alpha_hb = alloc_banks(alpha_specs)
- # Split row entries into the two layouts by name.
- flat_entries: list[dict] = []
- per_layer_groups: dict[str, dict[int, dict]] = {}
- for e in row_entries:
- m = _LAYER_ENTRY_RE.match(e["name"])
- if m is None:
- flat_entries.append(e)
- continue
- per_layer_groups.setdefault(m.group("base"), {})[int(m.group("layer"))] = e
-
- mixed = {e["name"] for e in flat_entries} & per_layer_groups.keys()
- assert not mixed, f"FTW bank(s) mix flat and per-layer row layouts: {sorted(mixed)}"
-
# Row banks: one padded-window HostBank per (name, layer_id) for the flat layout, plus
# how to carve the real [num_experts, *row_shape] tensor out of its head; ``None`` marks
# a per-layer entry (direct view, no carving needed).
@@ -546,10 +563,6 @@ def load_ftw_banks(
row_jobs.append((name, bank, win_off, win_end - win_off, layer_bytes, layer_id))
for base, by_layer in per_layer_groups.items():
- assert sorted(by_layer) == list(range(num_layers)), (
- f"FTW bank {base!r} has per-layer entries for layers {sorted(by_layer)}, "
- f"expected exactly range({num_layers})"
- )
row_hb[base] = []
row_view_args[base] = []
for layer_id in range(num_layers):
@@ -659,8 +672,71 @@ def load_ftw_banks(
)
+def load_ftw_banks_to_device(path: str, *, num_layers: int, device: torch.device, workers: int = 8, chunk: int = _DEFAULT_CHUNK):
+ """:func:`load_ftw_banks` for resident experts: every bank lands on ``device``, one (bank, layer) at a time through a transient host buffer."""
+ from freetoken.moe.expert_banks import ExpertBanks
+ from freetoken.moe.legacy_format import canonical_role, kind_kernel_for
+ from freetoken.utils.progress import byte_bar
+
+ reader = FTWReader(path)
+ try:
+ split = _split_bank_entries(reader, path, num_layers)
+ if split is None:
+ return None
+ bank_entries, alpha_entries, flat_entries, per_layer_groups = split
+
+ # (name, layer_id or None for an alpha, read offset, read length, head pad, shape, dtype)
+ jobs = [(e["name"], None, e["global_off"], e["nbytes"], 0, tuple(e["shape"]), _dtype_of(e["dtype"])) for e in alpha_entries]
+ for e in flat_entries:
+ total, *row_shape = e["shape"]
+ assert total % num_layers == 0, (e["name"], total, num_layers)
+ layer_bytes = total // num_layers * math.prod(row_shape) * _elsize(_dtype_of(e["dtype"]))
+ assert layer_bytes * num_layers == e["nbytes"], (e["name"], layer_bytes, num_layers, e["nbytes"])
+ for layer_id in range(num_layers):
+ off = e["global_off"] + layer_id * layer_bytes
+ win_off = (off // ALIGN) * ALIGN
+ jobs.append((e["name"], layer_id, win_off, _align_up(off + layer_bytes) - win_off, off - win_off, (total // num_layers, *row_shape), _dtype_of(e["dtype"])))
+ for base, by_layer in per_layer_groups.items():
+ for layer_id in range(num_layers):
+ e = by_layer[layer_id]
+ jobs.append((base, layer_id, e["global_off"], e["nbytes"], 0, tuple(e["shape"]), _dtype_of(e["dtype"])))
+
+ def _read(job) -> torch.Tensor:
+ _name, _layer, off, nbytes, head_pad, shape, dtype = job
+ buf = _transient_buffer(nbytes)
+ reader.read_into(memoryview(buf), {"global_off": off, "nbytes": nbytes}, workers=workers, chunk=chunk)
+ count = math.prod(shape) * _elsize(dtype)
+ return torch.frombuffer(buf, dtype=torch.uint8, count=head_pad + count)[head_pad:].view(dtype).view(shape)
+
+ sources: dict[str, list] = {}
+ alphas: dict[str, torch.Tensor] = {}
+ bar = byte_bar(sum(e["nbytes"] for e in bank_entries), "Loading expert banks (FTW)")
+ try:
+ # read the next (bank, layer) while the current one copies to the device
+ with ThreadPoolExecutor(1) as ex:
+ ahead = ex.submit(_read, jobs[0])
+ for k, (name, layer_id, *_rest) in enumerate(jobs):
+ host = ahead.result()
+ if k + 1 < len(jobs):
+ ahead = ex.submit(_read, jobs[k + 1])
+ if layer_id is None:
+ alphas[name] = host.to(device)
+ else:
+ sources.setdefault(canonical_role(name), [None] * num_layers)[layer_id] = host.to(device)
+ bar.update(host.numel() * host.element_size())
+ del host
+ finally:
+ bar.close()
+ quant_format = reader.meta("quant_format")
+ finally:
+ reader.close()
+
+ kind, kernel = kind_kernel_for(quant_format) if quant_format is not None else (None, None)
+ return ExpertBanks(quant_format, sources, **alphas, kind=kind, kernel=kernel)
+
+
__all__ = [
"INDEX_NAME", "FORMAT_TAG", "FORMAT_VERSION", "ALIGN", "DEFAULT_SHARD_LIMIT",
"is_ftw_checkpoint", "ftw_tensor_names", "FTWWriter", "FTWReader",
- "iter_ftw_weights", "load_ftw_banks", "layer_bank_entry_name",
+ "iter_ftw_weights", "load_ftw_banks", "load_ftw_banks_to_device", "layer_bank_entry_name",
]
diff --git a/python/freetoken/engine/engine.py b/python/freetoken/engine/engine.py
index e855bcd3..e5ccf710 100644
--- a/python/freetoken/engine/engine.py
+++ b/python/freetoken/engine/engine.py
@@ -13,14 +13,15 @@ from freetoken.attention import AttnType, attention_backend_info, create_attenti
from freetoken.core import Batch, Context, Req, set_global_ctx
from freetoken.distributed import destroy_distributed, enable_pynccl_distributed, set_tp_info
from freetoken.gpu_select import gpu_identity
-from freetoken.layers import set_rope_device
+from freetoken.checkpoint.ftw import ftw_tensor_names, is_ftw_checkpoint
+from freetoken.layers import OffloadMoELayer, iter_moe_layers, set_rope_device
from freetoken.layers.quantization import LayerKind, QuantBackend, finalize_quant, set_quant_backend
from freetoken.moe.offload_cache import iter_offload_moe_layers
from freetoken.mm.config import ENCODER_SECTIONS
from freetoken.models import create_model, load_weight
from freetoken.models.weight import ftw_lacks_vision
from freetoken.moe import is_offload_moe_strategy
-from freetoken.moe.expert_banks import load_expert_banks
+from freetoken.moe.expert_banks import attach_resident_banks, load_expert_banks
from freetoken.moe.host_banks import PinFailed
from freetoken.moe.offload_cache import OffloadMoeCache, attach_offload_moe_cache
from freetoken.utils import align_ceil, init_logger, is_sm90_family, is_sm100_family, mem_GB, torch_dtype
@@ -531,9 +532,14 @@ class Engine:
"family served images. Reconvert it with `ft checkpoint`, add the encoder in place with "
"scripts/ftw_hotfix.py (docs/ftw-hotfix.md), or start with --text-model-only"
)
+ resident = [layer for layer in iter_moe_layers(self.model) if not isinstance(layer, OffloadMoELayer)]
+ if resident and not config.use_dummy_weight and is_ftw_checkpoint(config.model_path) and not ftw_tensor_names(config.model_path, "experts_bank"):
+ raise ValueError(f"{config.model_path} keeps its routed experts as dense weights (an older `ft checkpoint --moe-backend fused`); reconvert it with `ft checkpoint`")
with _weight_load_context():
self.model.load_state_dict(self._load_weight_state_dict(config))
finalize_quant(self.model)
+ if resident:
+ self._load_resident_experts(config, resident)
def _load_weight_state_dict(self, config: EngineConfig) -> Dict[str, torch.Tensor]:
model_state = self.model.state_dict()
@@ -541,18 +547,32 @@ class Engine:
return _make_dummy_weight_state_dict(model_state, device=self.device)
# _materialize casts each loaded tensor to its model-param dtype (model_state), so
# models declaring per-tensor dtypes (e.g. DSV4's mixed fp8/fp32/bf16) are preserved;
- # offload models exclude experts (served from the offload cache, not dense weights).
+ # MoE models exclude routed experts (they load as expert banks, not dense weights).
return _materialize_loaded_weight_state_dict(
model_state,
load_weight(
config.model_path,
self.device,
- include_moe_experts=not is_offload_moe_strategy(config.moe_strategy),
+ include_moe_experts=not getattr(config.model_config, "is_moe", False),
include_vision=bool(config.active_encoders),
),
device=self.device,
)
+ def _load_resident_experts(self, config: EngineConfig, layers) -> None:
+ with _weight_load_context():
+ banks = load_expert_banks(
+ config.model_path,
+ config.model_config,
+ method=layers[0].quant_method,
+ device=self.device,
+ dtype=self.dtype,
+ dummy=config.use_dummy_weight,
+ parallel={"serial": False, "parallel": True}.get(config.expert_load, None),
+ resident=True,
+ )
+ attach_resident_banks(layers, banks)
+
@torch.inference_mode()
def _warmup_encoders(self) -> None:
for item in self.mm_processor.dummy_items(self.dtype, self.device):
@@ -1135,15 +1155,14 @@ def _is_unified_memory_gpu(index: "int | None" = None) -> bool:
return False
+_RESIDENT_EXPERT_QUANTS = ("none", "fp8_block", "nvfp4")
+
+
def _fused_resident_ok(model_config) -> bool:
- """Whether the resident ('fused') MoE path can hold this model's experts.
-
- FIXME: auto resolves to fused only for bf16 and fp8_block experts; drop this gate once the other quant formats support fused.
- """
- expert_quant = getattr(model_config, "expert_quant", "none")
- if expert_quant not in ("none", "fp8_block"):
+ """Whether the resident ('fused') MoE path can hold this model's experts; GGUF q4_0 and DeepSeek-V4 experts stay offload."""
+ if getattr(model_config, "expert_quant", "none") not in _RESIDENT_EXPERT_QUANTS:
return False
- return getattr(model_config, "moe_weight_format", None) in (None, "bf16")
+ return getattr(model_config, "moe_weight_format", None) in (None, "bf16", "mxfp4")
def _ensure_expandable_segments() -> None:
@@ -1594,8 +1613,8 @@ def _adjust_config(config: EngineConfig):
if is_moe and config.moe_strategy == "auto":
# A MoE model always defaults to the offload family: experts stream from pinned host
# banks into an auto-sized GPU slot cache, which is the only default that serves a model
- # bigger than the GPU. The resident 'fused' path (bf16 / block-fp8 experts, the two
- # formats MoELayer can allocate) is still reachable, but only when asked for explicitly
+ # bigger than the GPU. The resident 'fused' path (every expert format but GGUF q4_0 and
+ # DeepSeek-V4's) is still reachable, but only when asked for explicitly
# -- auto never picks it, because nothing here knows whether the experts would fit in
# HBM and a wrong guess is a weight-load OOM rather than a slower-but-working run.
default_backend = "offload"
@@ -1614,6 +1633,18 @@ def _adjust_config(config: EngineConfig):
"Unified-memory GPU detected; auto-selecting 'fused' MoE strategy "
"(resident experts) instead of offload"
)
+ # the cache sizing flags get their own warning in the 'fused' block below
+ inert = [flag for flag, on in (
+ (f"--moe-cpu-layers={config.moe_cpu_layers}", config.moe_cpu_layers),
+ ("--moe-prefill-hit-d2d", config.moe_prefill_hit_d2d),
+ ("--disable-moe-prefill-overlap", not config.moe_prefill_overlap),
+ ) if on]
+ if inert:
+ logger.warning_rank0(
+ f"Unified-memory GPU serves MoE experts resident; ignoring {', '.join(inert)} "
+ "(use --moe-strategy offload to keep the offload flags)"
+ )
+ override("moe_cpu_layers", None)
# Hardware-adaptive config: a cached `ft bench bw` profile can upgrade
# the offload default to hybrid when this machine's CPU MoE bandwidth clears its PCIe
# gather bandwidth by the bench threshold (default 2x). hybrid is VRAM-equivalent to
@@ -1733,7 +1764,7 @@ def _adjust_config(config: EngineConfig):
if (
is_moe
- and expert_quant not in ("none", "fp8_block")
+ and expert_quant not in _RESIDENT_EXPERT_QUANTS
and not is_offload_moe_strategy(config.moe_strategy)
):
raise ValueError(
diff --git a/python/freetoken/layers/__init__.py b/python/freetoken/layers/__init__.py
index d868751a..a0748462 100644
--- a/python/freetoken/layers/__init__.py
+++ b/python/freetoken/layers/__init__.py
@@ -16,7 +16,7 @@ from .linear import (
LinearReplicated,
LinearRowParallel,
)
-from .moe import MoELayer, OffloadMoELayer, make_moe_layer
+from .moe import MoELayer, OffloadMoELayer, iter_moe_layers, make_moe_layer
from .norm import (
GatedRMSNorm,
GemmaPlusOneRMSNorm,
@@ -57,5 +57,6 @@ __all__ = [
"LinearReplicated",
"MoELayer",
"OffloadMoELayer",
+ "iter_moe_layers",
"make_moe_layer",
]
diff --git a/python/freetoken/layers/moe.py b/python/freetoken/layers/moe.py
index c2094549..996beaa7 100644
--- a/python/freetoken/layers/moe.py
+++ b/python/freetoken/layers/moe.py
@@ -1,5 +1,5 @@
import os
-from typing import TYPE_CHECKING, Tuple
+from typing import TYPE_CHECKING, Iterator, Tuple
import torch
from freetoken.core import get_global_ctx
@@ -435,6 +435,19 @@ class OffloadMoELayer(MoELayer):
raise AssertionError(f"offload experts without a quant method only serve q4_0 banks, got {fmt!r}")
+def iter_moe_layers(model) -> Iterator[MoELayer]:
+ if isinstance(model, MoELayer):
+ yield model
+ if not isinstance(model, BaseOP):
+ return
+ for value in model.__dict__.values():
+ if isinstance(value, BaseOP):
+ yield from iter_moe_layers(value)
+ elif isinstance(value, (list, tuple)):
+ for item in value:
+ yield from iter_moe_layers(item)
+
+
def make_moe_layer(
config: "ModelConfig",
*,
@@ -483,10 +496,10 @@ def make_moe_layer(
has_bias=has_bias,
quant_config=quant_config,
prefix=prefix,
+ layer_id=layer_id,
)
if offload:
assert layer_id is not None, "offload MoE backends need the layer_id"
- kwargs["layer_id"] = layer_id
kwargs["strategy"] = config.moe_strategy
kwargs["decode_target"] = config.decode_target
return layer_cls(**kwargs)
diff --git a/python/freetoken/layers/quantization/moe/base.py b/python/freetoken/layers/quantization/moe/base.py
index 89f13ed0..e4a428d3 100644
--- a/python/freetoken/layers/quantization/moe/base.py
+++ b/python/freetoken/layers/quantization/moe/base.py
@@ -32,6 +32,7 @@ class MoEConfig:
apply_router_weight_on_input: bool = False
strategy: str = "resident"
decode_target: str = "gpu"
+ dtype: torch.dtype = torch.bfloat16
@classmethod
def from_layer(cls, layer: Any, scheme: QuantScheme | None) -> "MoEConfig":
@@ -52,6 +53,7 @@ class MoEConfig:
apply_router_weight_on_input=bool(layer.apply_router_weight_on_input),
strategy=layer.strategy,
decode_target=layer.decode_target,
+ dtype=torch.get_default_dtype(),
)
@property
@@ -147,9 +149,7 @@ class MoEKernel(ABC):
"""Most GPU cache slots the kernel can address for ``cfg``; None for no limit."""
return self.max_slots
- def _common_reject(self, cfg: MoEConfig, *, resident_ok: bool, tp_ok: bool, cpu_ok: bool, plain_silu_only: bool) -> str | None:
- if not resident_ok and cfg.strategy == "resident":
- return "not served resident; use --moe-strategy offload or cpu"
+ def _common_reject(self, cfg: MoEConfig, *, tp_ok: bool, cpu_ok: bool, plain_silu_only: bool) -> str | None:
if not tp_ok and cfg.tp_size > 1:
return "TP > 1 is not supported for this expert format"
if not cpu_ok and cfg.decode_target != "gpu":
@@ -190,12 +190,15 @@ class MoEMethod(QuantMethod):
def cpu_format(self) -> str | None:
return self.kernel.cpu_format
- @abstractmethod
- def create_weights(self, layer: Any) -> None: ...
+ def create_weights(self, layer: Any) -> None:
+ # resident experts stay out of the state dict; attach_resident_banks fills these after the dense weights load
+ layer._expert_banks = None
+ layer._expert_alphas = None
def finalize(self, layer: Any) -> None:
pass
- @abstractmethod
def resident_view(self, layer: Any) -> ExpertView:
"""Layout roles -> the resident layer's tensors, for apply()."""
+ assert layer._expert_banks is not None, "resident expert banks were never loaded"
+ return ExpertView(layer._expert_banks, n=self.cfg.num_experts, alphas=layer._expert_alphas)
diff --git a/python/freetoken/layers/quantization/moe/fp8_block.py b/python/freetoken/layers/quantization/moe/fp8_block.py
index 6c0167b7..20c3dde9 100644
--- a/python/freetoken/layers/quantization/moe/fp8_block.py
+++ b/python/freetoken/layers/quantization/moe/fp8_block.py
@@ -21,7 +21,7 @@ class TritonFp8BlockMoEKernel(MoEKernel):
name = "triton"
def unusable_reason(self, cfg: MoEConfig) -> str | None:
- reason = self._common_reject(cfg, resident_ok=True, tp_ok=False, cpu_ok=False, plain_silu_only=False)
+ reason = self._common_reject(cfg, tp_ok=False, cpu_ok=False, plain_silu_only=False)
if reason:
return reason
reason = gated_epilogue_reason(cfg)
@@ -64,17 +64,3 @@ class TritonFp8BlockMoEKernel(MoEKernel):
@register_method(QuantKind.FP8_BLOCK, LayerKind.MOE)
class Fp8BlockMoEMethod(MoEMethod):
candidates = (TritonFp8BlockMoEKernel,)
-
- def create_weights(self, layer) -> None:
- g = self.cfg
- e, i, h, b = g.num_experts, g.intermediate, g.hidden, BLOCK
- layer.gate_up_proj = torch.empty(e, 2 * i, h, dtype=FP8)
- layer.gate_up_scale_inv = torch.empty(e, 2 * i // b, h // b, dtype=torch.bfloat16)
- layer.down_proj = torch.empty(e, h, i, dtype=FP8)
- layer.down_scale_inv = torch.empty(e, h // b, i // b, dtype=torch.bfloat16)
-
- def resident_view(self, layer) -> ExpertView:
- return ExpertView({
- "gate_up": layer.gate_up_proj, "gate_up_scale": layer.gate_up_scale_inv,
- "down": layer.down_proj, "down_scale": layer.down_scale_inv,
- })
diff --git a/python/freetoken/layers/quantization/moe/mxfp4.py b/python/freetoken/layers/quantization/moe/mxfp4.py
index aa020ee3..13cd9746 100644
--- a/python/freetoken/layers/quantization/moe/mxfp4.py
+++ b/python/freetoken/layers/quantization/moe/mxfp4.py
@@ -22,7 +22,7 @@ class TritonMxfp4MoEKernel(MoEKernel):
return "standard MXFP4 kernel reads the concatenated gate|up row order"
if (cfg.alpha, cfg.beta) != (1.0, 0.0):
return "standard MXFP4 kernel has no alpha / beta in its swiglu"
- return self._common_reject(cfg, resident_ok=False, tp_ok=False, cpu_ok=True, plain_silu_only=False)
+ return self._common_reject(cfg, tp_ok=False, cpu_ok=True, plain_silu_only=False)
def layout(self, cfg: MoEConfig) -> dict[str, BankSpec]:
i, h = cfg.intermediate, cfg.hidden
@@ -106,37 +106,3 @@ class TritonGptossMxfp4MoEKernel(MoEKernel):
@register_method(QuantKind.MXFP4, LayerKind.MOE)
class Mxfp4MoEMethod(MoEMethod):
candidates = (TritonMxfp4MoEKernel, TritonGptossMxfp4MoEKernel)
-
- def create_weights(self, layer) -> None:
- if not self.cfg.has_bias:
- raise NotImplementedError("standard MXFP4 experts are served from the offload cache, not resident")
- g = self.cfg
- e, i, h = g.num_experts, g.local_intermediate, g.hidden
- if h % GROUP:
- raise ValueError(f"MXFP4 hidden size must be divisible by {GROUP}")
- layer.gate_up_proj_blocks = torch.empty(e, 2 * i, h // GROUP, 16, dtype=torch.uint8)
- layer.gate_up_proj_scales = torch.empty(e, 2 * i, h // GROUP, dtype=torch.uint8)
- layer.gate_up_proj_bias = torch.empty(e, 2 * i, dtype=torch.bfloat16)
- layer.down_proj_blocks = torch.empty(e, h, i // GROUP, 16, dtype=torch.uint8)
- layer.down_proj_scales = torch.empty(e, h, i // GROUP, dtype=torch.uint8)
- layer.down_proj_bias = torch.empty(e, h, dtype=torch.bfloat16)
-
- def finalize(self, layer) -> None:
- if getattr(layer, "gate_up_proj_blocks", None) is None:
- return
- from freetoken.moe.fused_mxfp4 import _transpose_mxfp4_for_decode
-
- # one transposed copy serves prefill and decode; the HF blocks are freed so 120B fits
- layer._gu_blocks_t, layer._gu_scales_t = _transpose_mxfp4_for_decode(layer.gate_up_proj_blocks, layer.gate_up_proj_scales)
- layer._dn_blocks_t, layer._dn_scales_t = _transpose_mxfp4_for_decode(layer.down_proj_blocks, layer.down_proj_scales)
- layer.gate_up_proj_blocks = None
- layer.gate_up_proj_scales = None
- layer.down_proj_blocks = None
- layer.down_proj_scales = None
- torch.cuda.empty_cache()
-
- def resident_view(self, layer) -> ExpertView:
- return ExpertView({
- "gate_up": layer._gu_blocks_t, "gate_up_scale": layer._gu_scales_t, "gate_up_bias": layer.gate_up_proj_bias,
- "down": layer._dn_blocks_t, "down_scale": layer._dn_scales_t, "down_bias": layer.down_proj_bias,
- })
diff --git a/python/freetoken/layers/quantization/moe/nvfp4.py b/python/freetoken/layers/quantization/moe/nvfp4.py
index bb341cdf..cf66e207 100644
--- a/python/freetoken/layers/quantization/moe/nvfp4.py
+++ b/python/freetoken/layers/quantization/moe/nvfp4.py
@@ -1,4 +1,4 @@
-"""NVFP4 experts: served from the offload cache through the Triton inline-dequant kernels, vLLM's Marlin or flashinfer's b12x.
+"""NVFP4 experts: served resident or from the offload cache through the Triton inline-dequant kernels, vLLM's Marlin or flashinfer's b12x.
FreeToken stores MoE experts as ModelOpt NVFP4 (packed e2m1 codes + fp8-e4m3 per-16
block scales + per-tensor global scale) in pinned host banks and gathers the routed
@@ -41,7 +41,7 @@ class TritonNvfp4MoEKernel(MoEKernel):
cpu_format = "nvfp4"
def unusable_reason(self, cfg: MoEConfig) -> str | None:
- reason = self._common_reject(cfg, resident_ok=False, tp_ok=False, cpu_ok=True, plain_silu_only=False)
+ reason = self._common_reject(cfg, tp_ok=False, cpu_ok=True, plain_silu_only=False)
if reason:
return reason
reason = gated_epilogue_reason(cfg)
@@ -230,7 +230,7 @@ class MarlinNvfp4MoEKernel(MoEKernel):
def unusable_reason(self, cfg: MoEConfig) -> str | None:
if not backend.is_vllm_installed():
return "vLLM is not installed"
- reason = self._common_reject(cfg, resident_ok=False, tp_ok=False, cpu_ok=False, plain_silu_only=True)
+ reason = self._common_reject(cfg, tp_ok=False, cpu_ok=False, plain_silu_only=True)
if reason:
return reason
if not _marlin_symbols_ok():
@@ -496,7 +496,7 @@ class B12xNvfp4MoEKernel(MoEKernel):
return f"b12x requires sm_120+, got sm_{cc[0]}{cc[1]}"
if not backend.is_flashinfer_installed():
return "flashinfer is not installed"
- reason = self._common_reject(cfg, resident_ok=False, tp_ok=False, cpu_ok=False, plain_silu_only=True)
+ reason = self._common_reject(cfg, tp_ok=False, cpu_ok=False, plain_silu_only=True)
if reason:
return reason
return _b12x_unusable_reason(cc)
@@ -567,9 +567,3 @@ class B12xNvfp4MoEKernel(MoEKernel):
@register_method(QuantKind.NVFP4, LayerKind.MOE)
class Nvfp4MoEMethod(MoEMethod):
candidates = (TritonNvfp4MoEKernel, MarlinNvfp4MoEKernel, B12xNvfp4MoEKernel)
-
- def create_weights(self, layer) -> None:
- raise NotImplementedError("NVFP4 experts are served from the offload cache, not resident")
-
- def resident_view(self, layer) -> ExpertView:
- raise NotImplementedError("NVFP4 experts are not resident")
diff --git a/python/freetoken/layers/quantization/moe/unquantized.py b/python/freetoken/layers/quantization/moe/unquantized.py
index ef435686..ded7d813 100644
--- a/python/freetoken/layers/quantization/moe/unquantized.py
+++ b/python/freetoken/layers/quantization/moe/unquantized.py
@@ -19,7 +19,9 @@ class FusedMoEKernel(MoEKernel):
def layout(self, cfg: MoEConfig) -> dict[str, BankSpec]:
i = cfg.local_intermediate
- return {"gate_up": BankSpec((2 * i, cfg.hidden), torch.bfloat16), "down": BankSpec((cfg.hidden, i), torch.bfloat16)}
+ # resident experts compute in the model dtype; offload banks stay bf16, the FTW and CPU executor format
+ dt = cfg.dtype if cfg.strategy == "resident" else torch.bfloat16
+ return {"gate_up": BankSpec((2 * i, cfg.hidden), dt), "down": BankSpec((cfg.hidden, i), dt)}
def pack(self, pieces, cfg: MoEConfig, out):
out["gate_up"].copy_(fused_piece(pieces, "gate_up"))
@@ -37,12 +39,3 @@ class FusedMoEKernel(MoEKernel):
@register_method(QuantKind.NONE, LayerKind.MOE)
class UnquantizedMoEMethod(MoEMethod):
candidates = (FusedMoEKernel,)
-
- def create_weights(self, layer) -> None:
- g = self.cfg
- i = g.local_intermediate
- layer.gate_up_proj = torch.empty(g.num_experts, 2 * i, g.hidden)
- layer.down_proj = torch.empty(g.num_experts, g.hidden, i)
-
- def resident_view(self, layer) -> ExpertView:
- return ExpertView({"gate_up": layer.gate_up_proj, "down": layer.down_proj})
diff --git a/python/freetoken/models/qwen3_5_moe/weight.py b/python/freetoken/models/qwen3_5_moe/weight.py
index 1bac8227..de5b9e89 100644
--- a/python/freetoken/models/qwen3_5_moe/weight.py
+++ b/python/freetoken/models/qwen3_5_moe/weight.py
@@ -1,6 +1,6 @@
"""Qwen3.5 / 3.6 / 3.8 checkpoint reader.
-The dense pass reads every Linear module under the scheme the checkpoint's QuantConfig gives it, the same answer the model built its buffers from, so bf16, block-fp8, ModelOpt and llm-compressor exports in any mix all land as the model's state dict. Routed experts are read by the offload cache (``nvfp4_expert_spec`` / ``iter_expert_pieces``); only bf16 stacked experts and resident block-fp8 experts come from here.
+The dense pass reads every Linear module under the scheme the checkpoint's QuantConfig gives it, the same answer the model built its buffers from, so bf16, block-fp8, ModelOpt and llm-compressor exports in any mix all land as the model's state dict. Routed experts are read by the expert-bank loader (``nvfp4_expert_spec`` / ``iter_expert_pieces``); only the bf16 stacked experts it reads come from here.
"""
from __future__ import annotations
@@ -241,9 +241,9 @@ def iter_weights(
include_non_moe: bool,
include_vision: bool = True,
) -> Iterator[tuple[str, torch.Tensor]]:
- """Yield the dense weights fused to the model's buffers, and the routed experts only where a resident path takes them from here: bf16 stacked experts as stored, block-fp8 experts restacked per layer.
+ """Yield the dense weights fused to the model's buffers, and with ``include_moe_experts`` the bf16 stacked experts as stored.
- Per-expert NVFP4 experts always come from the offload cache's expert reader.
+ Block-fp8 and NVFP4 experts always come from the expert-bank reader.
"""
if get_tp_info().size > 1:
raise NotImplementedError("qwen3_5_moe weight loading supports TP=1 only")
@@ -253,8 +253,6 @@ def iter_weights(
if include_non_moe or stacked:
reader = _DenseReader(get_quant_config(), get_model_spec(hf_config.architectures[0])) if include_non_moe else None
yield from _iter_shards(model_path, device, reader, stacked=stacked, include_vision=include_vision)
- if include_moe_experts and config.is_moe and config.expert_quant == "fp8_block":
- yield from _resident_fp8_experts(model_path, config)
def _iter_shards(model_path: str, device: torch.device, reader: _DenseReader | None, *, stacked: bool, include_vision: bool):
@@ -327,7 +325,7 @@ def iter_weights_parallel(
# ======================================================================================
-# Block-FP8 routed experts (Qwen3.5-35B-A3B-FP8): offload expert pieces and resident stacks.
+# Block-FP8 routed experts (Qwen3.5-35B-A3B-FP8): expert pieces.
# ======================================================================================
# Routed-expert checkpoint key (per-expert, un-fused). ``mtp.layers...`` is excluded by the
@@ -338,35 +336,6 @@ _FP8_EXPERT_KEY_RE = (
)
-def _resident_fp8_experts(model_path, config):
- from freetoken.kernel.triton.fp8_block_linear import FP8
-
- B = 128
- L, E, H, I, dense = _moe_dims(config)
- shapes = {
- "gate_up_proj": ((E, 2 * I, H), FP8),
- "gate_up_scale_inv": ((E, 2 * I // B, H // B), torch.bfloat16),
- "down_proj": ((E, H, I), FP8),
- "down_scale_inv": ((E, H // B, I // B), torch.bfloat16),
- }
- layers: dict[int, dict[str, torch.Tensor]] = {}
- placed = [0] * L
- for li, e0, e1, piece in iter_expert_pieces(model_path, config, QuantKind.FP8_BLOCK, parallel=None):
- stack = layers.setdefault(li, {n: torch.empty(shape, dtype=dt) for n, (shape, dt) in shapes.items()})
- stack["gate_up_proj"][e0:e1, :I] = piece["gate"]
- stack["gate_up_proj"][e0:e1, I:] = piece["up"]
- stack["gate_up_scale_inv"][e0:e1, : I // B] = piece["gate_scale"]
- stack["gate_up_scale_inv"][e0:e1, I // B :] = piece["up_scale"]
- stack["down_proj"][e0:e1] = piece["down"]
- stack["down_scale_inv"][e0:e1] = piece["down_scale"]
- placed[li] += e1 - e0
- if placed[li] == E:
- pre = f"model.layers.{dense + li}.mlp.experts"
- for name, tensor in layers.pop(li).items():
- yield f"{pre}.{name}", tensor
- assert not layers, f"incomplete resident fp8 experts for layers {sorted(layers)}"
-
-
def _moe_dims(model_config):
L = model_config.num_moe_layers
return (
diff --git a/python/freetoken/moe/expert_banks.py b/python/freetoken/moe/expert_banks.py
index e3a768b6..7500174a 100644
--- a/python/freetoken/moe/expert_banks.py
+++ b/python/freetoken/moe/expert_banks.py
@@ -75,6 +75,7 @@ def build_expert_banks(
device: torch.device,
layer_sink=None,
dummy: bool = False,
+ resident: bool = False,
) -> ExpertBanks:
"""Fill host banks in the kernel's layout from a stream of expert pieces.
@@ -83,6 +84,7 @@ def build_expert_banks(
complete once its ``num_experts`` rows have arrived: with ``layer_sink=None`` its banks
are pinned in the background, otherwise the sink receives them (converter). ``dummy``
skips the pieces and fills the banks with finite random contents.
+ ``resident`` packs into banks on ``device`` instead of host banks (the resident 'fused' experts).
"""
from freetoken.moe.host_banks import LayerCompletionTracker, PinPipeline, pin_banks
from freetoken.moe.legacy_format import legacy_format_for
@@ -91,8 +93,13 @@ def build_expert_banks(
layout = method.layout()
E = method.cfg.num_experts
specs = {role: ((E, *spec.shape), spec.dtype) for role, spec in layout.items() if not spec.resident}
- hb = alloc_layer_banks(specs, num_layers)
- banks = {role: [b.tensor for b in hb[role]] for role in specs}
+ if resident:
+ assert layer_sink is None
+ hb = None
+ banks = {role: [torch.empty(shape, dtype=dtype, device=device) for _ in range(num_layers)] for role, (shape, dtype) in specs.items()}
+ else:
+ hb = alloc_layer_banks(specs, num_layers)
+ banks = {role: [b.tensor for b in hb[role]] for role in specs}
alphas = {
role: torch.empty(num_layers * E, dtype=spec.dtype, device=device)
for role, spec in layout.items() if spec.resident
@@ -104,7 +111,7 @@ def build_expert_banks(
_dummy_fill(role, tensor)
for alpha in alphas.values():
alpha.fill_(1.0)
- if torch.cuda.is_available():
+ if hb is not None and torch.cuda.is_available():
pin_banks(hb)
return ExpertBanks(
legacy_format_for(method.kind, kernel.name), banks,
@@ -136,7 +143,7 @@ def build_expert_banks(
if layer_sink is not None:
_fill(layer_sink)
- elif torch.cuda.is_available():
+ elif torch.cuda.is_available() and not resident:
with PinPipeline() as pins:
_fill(pins)
else:
@@ -192,16 +199,16 @@ def _legacy_expert_banks(model_path, model_config, device, dtype, dummy, paralle
)
-def _method_expert_banks(model_path, model_config, method, device, dummy, parallel, workers, chunk, layer_sink=None) -> ExpertBanks:
+def _method_expert_banks(model_path, model_config, method, device, dummy, parallel, workers, chunk, layer_sink=None, resident=False) -> ExpertBanks:
from freetoken.moe.expert_pieces import iter_expert_pieces
num_layers = model_config.num_moe_layers
if dummy:
- return build_expert_banks(method, num_layers, None, device=device, dummy=True)
+ return build_expert_banks(method, num_layers, None, device=device, dummy=True, resident=resident)
pieces = iter_expert_pieces(
model_path, model_config, method.kind, parallel=parallel, workers=workers, chunk=chunk
)
- return build_expert_banks(method, num_layers, pieces, device=device, layer_sink=layer_sink)
+ return build_expert_banks(method, num_layers, pieces, device=device, layer_sink=layer_sink, resident=resident)
def _host_ram_fits_parallel(model_path: str) -> bool:
@@ -286,6 +293,7 @@ def load_expert_banks(
decode_target: str = "gpu",
layer_sink=None,
layer_residency: list[str] | None = None,
+ resident: bool = False,
) -> ExpertBanks:
"""Load (or fabricate, with ``dummy=True``) the expert banks. Two paths, both returning
the same normalized ``ExpertBanks`` and both pinning after fill:
@@ -310,14 +318,21 @@ def load_expert_banks(
``layer_residency``: per-layer ``HostResidency`` labels applied at settle time -- explicitly on the FTW fast path, ambiently (``requested_residency``) in the slow-path providers.
Applied labels are echoed on ``ExpertBanks.layer_residency``; a loader that settles some other way leaves it ``None`` (CPU-layer decode still works on pinned banks, it just saves no pin quota).
+
+ ``resident`` (with ``method``): the banks land on ``device`` for the resident 'fused' experts; no host bank outlives the load.
"""
- from freetoken.checkpoint.ftw import is_ftw_checkpoint, load_ftw_banks
+ from freetoken.checkpoint.ftw import is_ftw_checkpoint, load_ftw_banks, load_ftw_banks_to_device
+ if resident:
+ assert method is not None and layer_sink is None and layer_residency is None
if model_path and is_ftw_checkpoint(model_path) and not dummy:
- banks = load_ftw_banks(
- model_path, num_layers=model_config.num_moe_layers, workers=workers, chunk=chunk,
- layer_residency=layer_residency,
- )
+ if resident:
+ banks = load_ftw_banks_to_device(model_path, num_layers=model_config.num_moe_layers, device=device, workers=workers, chunk=chunk)
+ else:
+ banks = load_ftw_banks(
+ model_path, num_layers=model_config.num_moe_layers, workers=workers, chunk=chunk,
+ layer_residency=layer_residency,
+ )
if banks is not None:
logger.info_rank0(f"expert banks: FTW fast path (FTW checkpoint {model_path})")
return banks
@@ -354,7 +369,7 @@ def load_expert_banks(
def _build(par: bool) -> ExpertBanks:
if method is not None:
- return _method_expert_banks(model_path, model_config, method, device, dummy, par, workers, chunk, layer_sink)
+ return _method_expert_banks(model_path, model_config, method, device, dummy, par, workers, chunk, layer_sink, resident)
return _legacy_expert_banks(model_path, model_config, device, dtype, dummy, par, workers, chunk, decode_target, layer_sink)
with requested_residency(layer_residency) as residency_plan:
@@ -393,3 +408,33 @@ def _echo_residency(banks: ExpertBanks, requested, plan) -> ExpertBanks:
"but saves no pinned quota"
)
return banks
+
+
+def attach_resident_banks(layers, banks: ExpertBanks) -> None:
+ """Give each resident MoE layer its rows of the device ``banks``; bank layer ``i`` goes to the layer whose ``layer_id`` is ``i``."""
+ ids = sorted(layer.layer_id for layer in layers if layer.layer_id is not None)
+ num_layers = len(next(iter(banks.sources.values())))
+ if ids != list(range(num_layers)) or len(ids) != len(layers):
+ raise ValueError(f"resident MoE layer ids {ids} do not cover the {num_layers} expert bank layers")
+ for layer in layers:
+ i, e = layer.layer_id, layer.num_experts
+ layout = layer.quant_method.layout()
+ rows = {role: (e, *spec.shape) for role, spec in layout.items() if not spec.resident}
+ if set(banks.sources) != set(rows):
+ raise ValueError(f"expert banks hold {sorted(banks.sources)}, the {layer.quant_method.kernel.name} kernel reads {sorted(rows)}")
+ for role, shape in rows.items():
+ bank = banks.sources[role][i]
+ if tuple(bank.shape) != shape or bank.dtype != layout[role].dtype:
+ raise ValueError(
+ f"bank {role!r} of layer {i} is {tuple(bank.shape)} {bank.dtype} but the {layer.quant_method.kernel.name} kernel "
+ f"wants {shape} {layout[role].dtype}; the banks were packed for another kernel or model"
+ )
+ alphas = []
+ # apply() reads view.alphas as (gate_up, down)
+ for role in (role for role in ("gate_up_alpha", "down_alpha") if role in layout):
+ a = getattr(banks, role)
+ if a is None or a.numel() != num_layers * e or a.dtype != layout[role].dtype:
+ raise ValueError(f"the {layer.quant_method.kernel.name} kernel needs {num_layers * e} {layout[role].dtype} {role} values, the expert banks carry {None if a is None else (a.numel(), a.dtype)}")
+ alphas.append(a)
+ layer._expert_banks = {role: per_layer[i] for role, per_layer in banks.sources.items()}
+ layer._expert_alphas = tuple(a[i * e : (i + 1) * e] for a in alphas) or None
diff --git a/python/freetoken/moe/offload_cache.py b/python/freetoken/moe/offload_cache.py
index 32e0f3cc..c6f975df 100644
--- a/python/freetoken/moe/offload_cache.py
+++ b/python/freetoken/moe/offload_cache.py
@@ -1054,20 +1054,9 @@ class OffloadMoeCache:
def iter_offload_moe_layers(model) -> Iterator:
- from freetoken.layers import BaseOP, OffloadMoELayer
+ from freetoken.layers import OffloadMoELayer, iter_moe_layers
- if isinstance(model, OffloadMoELayer):
- yield model
-
- if not isinstance(model, BaseOP):
- return
-
- for value in model.__dict__.values():
- if isinstance(value, BaseOP):
- yield from iter_offload_moe_layers(value)
- elif isinstance(value, (list, tuple)):
- for item in value:
- yield from iter_offload_moe_layers(item)
+ return (layer for layer in iter_moe_layers(model) if isinstance(layer, OffloadMoELayer))
def attach_offload_moe_cache(model, cache: OffloadMoeCache) -> list:
diff --git a/python/freetoken/server/args.py b/python/freetoken/server/args.py
index 23a62f84..4dd57b1a 100644
--- a/python/freetoken/server/args.py
+++ b/python/freetoken/server/args.py
@@ -600,8 +600,8 @@ def parse_args(
choices=["auto", *MOE_STRATEGIES],
help=(
"How the routed experts are served. 'auto' resolves a MoE model to the offload family "
- "(offload, or hybrid when a `ft bench bw` profile recommends it); resident "
- "'fused' experts must be requested explicitly."
+ "(offload, or hybrid when a `ft bench bw` profile recommends it), and to resident "
+ "'fused' experts on unified-memory GPUs (GB10 / DGX Spark)."
),
)
diff --git a/tests/engine/test_cache_budget.py b/tests/engine/test_cache_budget.py
index 073112bb..a77409f0 100644
--- a/tests/engine/test_cache_budget.py
+++ b/tests/engine/test_cache_budget.py
@@ -367,7 +367,7 @@ def _offload_engine_config(**overrides):
num_layers=10,
num_moe_layers=10,
num_experts=8,
- expert_quant="nvfp4", # quantized experts -> must resolve to an offload backend
+ expert_quant="nvfp4",
moe_strategy="auto",
),
)
@@ -389,7 +389,7 @@ def test_guard_raises_actionable_error_when_too_small():
assert "128" in msg and "moe-cache" in msg
-def test_adjust_config_defaults_moe_cache_auto_for_auto_resolved_offload_backend():
+def test_adjust_config_defaults_moe_cache_auto_for_auto_resolved_offload_backend(monkeypatch):
"""Bare `ft serve <FTW MoE checkpoint>`: no --moe-backend, no --moe-cache-* flags at all.
args.py's parse-time default only fires when the backend is *already*
@@ -400,9 +400,12 @@ def test_adjust_config_defaults_moe_cache_auto_for_auto_resolved_offload_backend
moe_cache_auto=True, so _init_offload_moe_cache's _require_offload_cache_size guard is never
reached with moe_cache_size still 0.
"""
+ import freetoken.engine.engine as engine_module
from freetoken.engine.engine import _adjust_config
from freetoken.moe import is_offload_moe_strategy
+ # a unified-memory host (GB10) resolves auto to fused instead
+ monkeypatch.setattr(engine_module, "_is_unified_memory_gpu", lambda index=None: False)
config = _offload_engine_config()
_adjust_config(config)
diff --git a/tests/models/qwen4_exp/test_skeleton.py b/tests/models/qwen4_exp/test_skeleton.py
index ac598e65..8fb9112d 100644
--- a/tests/models/qwen4_exp/test_skeleton.py
+++ b/tests/models/qwen4_exp/test_skeleton.py
@@ -16,7 +16,7 @@ import pytest
import torch
import torch.nn.functional as F
-from freetoken.layers import BaseOP, LinearReplicated
+from freetoken.layers import BaseOP, LinearReplicated, iter_moe_layers
from freetoken.models.config import ModelConfig
from freetoken.models.qwen4_exp.config import parse_config
from freetoken.models.qwen4_exp.hc import GatedResidual
@@ -30,12 +30,17 @@ def _config(num_layers: int = 4) -> ModelConfig:
def _fill(op, gen: torch.Generator, scale: float = 0.05) -> None:
- """Random floats / zeroed ints for every state-dict tensor of an op tree."""
+ """Random floats / zeroed ints for every state-dict tensor of an op tree, and random expert banks for its resident MoE layers."""
for tensor in op.state_dict().values():
if tensor.is_floating_point():
tensor.normal_(0.0, scale, generator=gen)
else:
tensor.zero_()
+ for layer in iter_moe_layers(op):
+ layer._expert_banks = {
+ role: torch.randn((layer.num_experts, *spec.shape), generator=gen, device=gen.device, dtype=spec.dtype) * scale
+ for role, spec in layer.quant_method.layout().items()
+ }
def _group_norm(x, weight, eps, groups):
diff --git a/tests/models/test_quant_config.py b/tests/models/test_quant_config.py
index 6ac1b84b..084a2e4a 100644
--- a/tests/models/test_quant_config.py
+++ b/tests/models/test_quant_config.py
@@ -121,14 +121,6 @@ class Case:
check: Callable[[Any, dict], None] | None = None # family extras beyond the method table
-def _installed(module: str) -> bool:
- try:
- __import__(module)
- return True
- except ImportError:
- return False
-
-
def _qwen35_fp8(model, idx):
assert model.model.layers.op_list[idx["lin"]].linear_attn._split_in_proj
@@ -162,13 +154,18 @@ def _dsv4(model, idx):
assert "model.head.weight" in keys and not any(".ffn.experts." in k for k in keys)
+def _banked_experts(model, idx):
+ experts = op_at(model, f"model.layers.{idx['lin']}.mlp.experts")
+ assert not experts.state_dict()
+
+
def _gpt_oss(cls):
def check(model, idx):
experts = op_at(model, "model.layers.0.mlp.experts")
assert type(experts) is cls
assert experts.interleaved and experts.alpha == pytest.approx(1.702) and experts.limit == 7.0
if cls is GptOssMoELayer:
- assert experts.gate_up_proj_blocks.dtype is torch.uint8 and experts.gate_up_proj_blocks.dim() == 4
+ assert not experts.state_dict()
return check
@@ -198,9 +195,8 @@ CASES = [
"model.layers.{lin}.mlp.shared_expert.down_proj": NVFP4, "model.layers.{lin}.mlp.experts": Nvfp4MoEMethod,
"lm_head": NVFP4,
}, check=_qwen35_nvfp4),
- # resident NVFP4 experts have no kernel without vLLM's Marlin
- Case("Qwen3.6-35B-A3B-NVFP4", ModelOptConfig, {}, strategy="fused", raises=KernelSelectionError,
- skip_if=lambda: not _installed("flashinfer") or _installed("vllm")),
+ # resident NVFP4 experts load through the expert-bank loader, not the state dict
+ Case("Qwen3.6-35B-A3B-NVFP4", ModelOptConfig, {"model.layers.{lin}.mlp.experts": (Nvfp4MoEMethod, "triton")}, strategy="fused", check=_banked_experts),
Case("Qwen3.8-27B-NVFP4", ModelOptConfig, {
"model.layers.{lin}.linear_attn.in_proj_qkvz": FP8T, "model.layers.{full}.self_attn.o_proj": FP8T,
"model.layers.{lin}.mlp.gate_up_proj": NVFP4, "model.layers.{lin}.mlp.down_proj": NVFP4, "lm_head": NVFP4,