FreeToken brings datacenter-scale model serving to your desktop. Run massive models locally, fast and efficiently.
feat(moe): serve NVFP4 and MXFP4 experts fused through the expert banks (#601)
Xiaoze Fan189c8ad0c335
diff --git a/docs/cli.md b/docs/cli.mdindex fb5e613f..deeda184 100644--- a/docs/cli.md+++ b/docs/cli.md@@ -80,7 +80,7 @@ See [models.md](models.md#moe-strategies) for what each strategy does.| Flag | Default | Meaning ||---|---|---|-| `--moe-strategy` | auto | `fused`/`offload`/`cpu`/`hybrid`; auto → offload, or hybrid with a `ft bench bw` profile. `--moe-backend` is the deprecated old spelling |+| `--moe-strategy` | auto | `fused`/`offload`/`cpu`/`hybrid`; auto → offload, or hybrid with a `ft bench bw` profile; fused on unified-memory GPUs (GB10), see [models.md](models.md#unified-memory-gpus-gb10--dgx-spark). `--moe-backend` is the deprecated old spelling || `--quant-backend` | auto | Kernel per quantized layer type, `layer[.kind]=name` entries: `linear=marlin,moe=b12x` or `moe.nvfp4=triton`. A layer-level entry applies to every kind whose table lists the name || `--nvfp4-backend` | — | Deprecated: stands in for `--quant-backend moe.nvfp4=<marlin\|b12x\|triton>` (`flashinfer` means b12x); cannot be combined with `--quant-backend` || `--moe-cache-size` / `--moe-cache-rate` / `--moe-cache-auto` | auto | GPU expert-cache size as slots / fraction of all experts / sized from free VRAM (mutually exclusive; auto is enabled by default for offload-family strategies) |@@ -179,9 +179,10 @@ ft checkpoint --model <hf_dir> --out <ftw_dir> [--dtype bfloat16] [--moe-backend```Converts an HF safetensors checkpoint to FTW, FreeToken's self-contained-fast-load format; point `ft serve --model` at the output dir. `--moe-backend-offload` (default) packs experts into offload banks; `--moe-backend triton`-keeps them dense for resident serving. See the FTW caveats in+fast-load format; point `ft serve --model` at the output dir. MoE experts are+always packed into expert banks, which serve the fused and the offload+strategies alike (GGUF and DeepSeek-V4 experts serve offload only);+`--moe-backend` is kept for compatibility. See the FTW caveats in[models.md](models.md#notes); FTW files from older builds can be repaired with[scripts/ftw_hotfix.py](ftw-hotfix.md) instead of reconverting.diff --git a/docs/models.md b/docs/models.mdindex 548a363c..efd5268b 100644--- a/docs/models.md+++ b/docs/models.md@@ -38,7 +38,8 @@ These families accept image input by default; pass `--text-model-only` to skip t`ft serve --moe-strategy {auto,fused,offload,cpu,hybrid}` (`--moe-backend` is the deprecated old spelling):-- **fused** — experts resident on GPU (needs the VRAM); never auto-selected.+- **fused** — experts resident on GPU (needs the VRAM). Serves every expert+ format except GGUF and DeepSeek-V4's.- **offload** — experts live in host RAM, an LRU cache of expert slots on GPU;misses stream over PCIe.- **cpu** — misses are computed on the CPU instead of fetched.@@ -46,12 +47,24 @@ These families accept image input by default; pass `--text-model-only` to skip tCPU, overlapped. Run `ft bench bw` once per machine to calibrate the split.- **auto** — dense models always resolve to `fused`; MoE models resolve to`offload`, upgraded to `hybrid` when a cached `ft bench bw` profile- recommends it.+ recommends it. Unified-memory GPUs differ, see below.++### Unified-memory GPUs (GB10 / DGX Spark)++The GPU and the CPU share one DRAM, so offload only copies experts between two+names for the same memory. On these GPUs `auto` resolves every MoE model to+`fused`: the default is the resident path only. GGUF and DeepSeek-V4 experts+have no resident path and stay on `offload`. Offload-only flags+(`--moe-cpu-layers`, `--moe-cache-size` / `-rate` / `-auto`,+`--moe-prefill-hit-d2d`, `--disable-moe-prefill-overlap`) are ignored with a+warning; pass `--moe-strategy offload` to use them.## Notes- `ft checkpoint` conversion is optional — it pre-converts a checkpoint intoFreeToken's fast-load format, and `ft serve --model` auto-detects the result.+- An FTW converted by an older build with `--moe-backend fused` or `triton` keeps+ its experts as dense weights and no longer loads; reconvert it with `ft checkpoint`.- FTW files converted by builds before the quantization refactor may fail to load;see [ftw-hotfix.md](ftw-hotfix.md) for the affected checkpoints and the repair tool.- An FTW converted before its family served images holds no vision encoder: `ft serve`diff --git a/python/freetoken/checkpoint/__main__.py b/python/freetoken/checkpoint/__main__.pyindex 2b7e35f1..0d7c2d7e 100644--- a/python/freetoken/checkpoint/__main__.py+++ b/python/freetoken/checkpoint/__main__.py@@ -37,7 +37,7 @@ def main(argv: list[str] | None = None, prog: str = "freetoken.checkpoint") -> ip.add_argument("--out", required=True, help="output FTW checkpoint dir")p.add_argument("--dtype", choices=sorted(_DTYPES), default="bfloat16")p.add_argument("--moe-backend", default="offload",- help="offload (experts -> banks) or e.g. triton (experts stay dense)")+ help="kept for compatibility: MoE experts always go to banks, which every strategy reads")p.add_argument("--quant-backend", type=_parse_quant_backend, default=None,help="kernel per quantized layer type, as for ft serve; the expert banks are packed for the ""chosen MoE kernel and the server picks that kernel back up from the FTW")diff --git a/python/freetoken/checkpoint/convert.py b/python/freetoken/checkpoint/convert.pyindex a875d77f..152f3871 100644--- a/python/freetoken/checkpoint/convert.py+++ b/python/freetoken/checkpoint/convert.py@@ -200,7 +200,8 @@ def convert_checkpoint(cfg = EngineConfig(model_path=model_path, tp_info=DistributedInfo(tp.rank, tp.size),dtype=dtype, moe_strategy=moe_backend, quant_backend=quant_backend)mc = cfg.model_config- offload = moe_backend == "offload" and getattr(mc, "is_moe", False)+ # every MoE strategy reads the routed experts from the banks, fused included+ offload = getattr(mc, "is_moe", False)include_moe_experts = not offloadmethod = Noneif offload:@@ -208,6 +209,7 @@ def convert_checkpoint(from freetoken.layers import set_rope_deviceset_rope_device(dev)+ object.__setattr__(cfg, "moe_strategy", "offload")method = offload_expert_method(cfg)from freetoken.utils.progress import byte_bar, count_bardiff --git a/python/freetoken/checkpoint/ftw.py b/python/freetoken/checkpoint/ftw.pyindex e2bd0de2..e31e9d82 100644--- a/python/freetoken/checkpoint/ftw.py+++ b/python/freetoken/checkpoint/ftw.py@@ -432,6 +432,43 @@ def iter_ftw_weights(path: str, *, kinds=("weight",), keep: Callable[[str], boolraise err[0]+def _split_bank_entries(reader: FTWReader, path: str, num_layers: int):+ """``(all, alphas, flat regions, {base: {layer: entry}})`` of the ``experts_bank`` entries; None when there are none."""+ bank_entries = reader.entries("experts_bank")+ if not bank_entries:+ return None++ alpha_entries = [e for e in bank_entries if e["name"] in _ALPHA_NAMES]+ row_entries = [e for e in bank_entries if e["name"] not in _ALPHA_NAMES]++ meta_layers = reader.meta("expert_bank_num_layers")+ if meta_layers is not None and meta_layers != num_layers:+ raise RuntimeError(+ f"{path!r} was converted with {meta_layers} expert-bank layers but the "+ f"model config says num_moe_layers={num_layers}; the checkpoint does not "+ "match its config"+ )++ # Split row entries into the two layouts by name.+ flat_entries: list[dict] = []+ per_layer_groups: dict[str, dict[int, dict]] = {}+ for e in row_entries:+ m = _LAYER_ENTRY_RE.match(e["name"])+ if m is None:+ flat_entries.append(e)+ continue+ per_layer_groups.setdefault(m.group("base"), {})[int(m.group("layer"))] = e++ mixed = {e["name"] for e in flat_entries} & per_layer_groups.keys()+ assert not mixed, f"FTW bank(s) mix flat and per-layer row layouts: {sorted(mixed)}"+ for base, by_layer in per_layer_groups.items():+ assert sorted(by_layer) == list(range(num_layers)), (+ f"FTW bank {base!r} has per-layer entries for layers {sorted(by_layer)}, "+ f"expected exactly range({num_layers})"+ )+ return bank_entries, alpha_entries, flat_entries, per_layer_groups++def load_ftw_banks(path: str, *, num_layers: int, workers: int = 8, chunk: int = _DEFAULT_CHUNK,layer_residency: list[str] | None = None,@@ -482,40 +519,20 @@ def load_ftw_banks(return "mmap"reader = FTWReader(path)- bank_entries = reader.entries("experts_bank")- if not bank_entries:+ try:+ split = _split_bank_entries(reader, path, num_layers)+ except BaseException:reader.close()- return None-- alpha_entries = [e for e in bank_entries if e["name"] in _ALPHA_NAMES]- row_entries = [e for e in bank_entries if e["name"] not in _ALPHA_NAMES]-- meta_layers = reader.meta("expert_bank_num_layers")- if meta_layers is not None and meta_layers != num_layers:+ raise+ if split is None:reader.close()- raise RuntimeError(- f"{path!r} was converted with {meta_layers} expert-bank layers but the "- f"model config says num_moe_layers={num_layers}; the checkpoint does not "- "match its config"- )+ return None+ bank_entries, alpha_entries, flat_entries, per_layer_groups = split# Alphas: unchanged, one flat HostBank per entry.alpha_specs = {e["name"]: (tuple(e["shape"]), _dtype_of(e["dtype"])) for e in alpha_entries}alpha_hb = alloc_banks(alpha_specs)- # Split row entries into the two layouts by name.- flat_entries: list[dict] = []- per_layer_groups: dict[str, dict[int, dict]] = {}- for e in row_entries:- m = _LAYER_ENTRY_RE.match(e["name"])- if m is None:- flat_entries.append(e)- continue- per_layer_groups.setdefault(m.group("base"), {})[int(m.group("layer"))] = e-- mixed = {e["name"] for e in flat_entries} & per_layer_groups.keys()- assert not mixed, f"FTW bank(s) mix flat and per-layer row layouts: {sorted(mixed)}"-# Row banks: one padded-window HostBank per (name, layer_id) for the flat layout, plus# how to carve the real [num_experts, *row_shape] tensor out of its head; ``None`` marks# a per-layer entry (direct view, no carving needed).@@ -546,10 +563,6 @@ def load_ftw_banks(row_jobs.append((name, bank, win_off, win_end - win_off, layer_bytes, layer_id))for base, by_layer in per_layer_groups.items():- assert sorted(by_layer) == list(range(num_layers)), (- f"FTW bank {base!r} has per-layer entries for layers {sorted(by_layer)}, "- f"expected exactly range({num_layers})"- )row_hb[base] = []row_view_args[base] = []for layer_id in range(num_layers):@@ -659,8 +672,71 @@ def load_ftw_banks()+def load_ftw_banks_to_device(path: str, *, num_layers: int, device: torch.device, workers: int = 8, chunk: int = _DEFAULT_CHUNK):+ """:func:`load_ftw_banks` for resident experts: every bank lands on ``device``, one (bank, layer) at a time through a transient host buffer."""+ from freetoken.moe.expert_banks import ExpertBanks+ from freetoken.moe.legacy_format import canonical_role, kind_kernel_for+ from freetoken.utils.progress import byte_bar++ reader = FTWReader(path)+ try:+ split = _split_bank_entries(reader, path, num_layers)+ if split is None:+ return None+ bank_entries, alpha_entries, flat_entries, per_layer_groups = split++ # (name, layer_id or None for an alpha, read offset, read length, head pad, shape, dtype)+ jobs = [(e["name"], None, e["global_off"], e["nbytes"], 0, tuple(e["shape"]), _dtype_of(e["dtype"])) for e in alpha_entries]+ for e in flat_entries:+ total, *row_shape = e["shape"]+ assert total % num_layers == 0, (e["name"], total, num_layers)+ layer_bytes = total // num_layers * math.prod(row_shape) * _elsize(_dtype_of(e["dtype"]))+ assert layer_bytes * num_layers == e["nbytes"], (e["name"], layer_bytes, num_layers, e["nbytes"])+ for layer_id in range(num_layers):+ off = e["global_off"] + layer_id * layer_bytes+ win_off = (off // ALIGN) * ALIGN+ jobs.append((e["name"], layer_id, win_off, _align_up(off + layer_bytes) - win_off, off - win_off, (total // num_layers, *row_shape), _dtype_of(e["dtype"])))+ for base, by_layer in per_layer_groups.items():+ for layer_id in range(num_layers):+ e = by_layer[layer_id]+ jobs.append((base, layer_id, e["global_off"], e["nbytes"], 0, tuple(e["shape"]), _dtype_of(e["dtype"])))++ def _read(job) -> torch.Tensor:+ _name, _layer, off, nbytes, head_pad, shape, dtype = job+ buf = _transient_buffer(nbytes)+ reader.read_into(memoryview(buf), {"global_off": off, "nbytes": nbytes}, workers=workers, chunk=chunk)+ count = math.prod(shape) * _elsize(dtype)+ return torch.frombuffer(buf, dtype=torch.uint8, count=head_pad + count)[head_pad:].view(dtype).view(shape)++ sources: dict[str, list] = {}+ alphas: dict[str, torch.Tensor] = {}+ bar = byte_bar(sum(e["nbytes"] for e in bank_entries), "Loading expert banks (FTW)")+ try:+ # read the next (bank, layer) while the current one copies to the device+ with ThreadPoolExecutor(1) as ex:+ ahead = ex.submit(_read, jobs[0])+ for k, (name, layer_id, *_rest) in enumerate(jobs):+ host = ahead.result()+ if k + 1 < len(jobs):+ ahead = ex.submit(_read, jobs[k + 1])+ if layer_id is None:+ alphas[name] = host.to(device)+ else:+ sources.setdefault(canonical_role(name), [None] * num_layers)[layer_id] = host.to(device)+ bar.update(host.numel() * host.element_size())+ del host+ finally:+ bar.close()+ quant_format = reader.meta("quant_format")+ finally:+ reader.close()++ kind, kernel = kind_kernel_for(quant_format) if quant_format is not None else (None, None)+ return ExpertBanks(quant_format, sources, **alphas, kind=kind, kernel=kernel)++__all__ = ["INDEX_NAME", "FORMAT_TAG", "FORMAT_VERSION", "ALIGN", "DEFAULT_SHARD_LIMIT","is_ftw_checkpoint", "ftw_tensor_names", "FTWWriter", "FTWReader",- "iter_ftw_weights", "load_ftw_banks", "layer_bank_entry_name",+ "iter_ftw_weights", "load_ftw_banks", "load_ftw_banks_to_device", "layer_bank_entry_name",]diff --git a/python/freetoken/engine/engine.py b/python/freetoken/engine/engine.pyindex e855bcd3..e5ccf710 100644--- a/python/freetoken/engine/engine.py+++ b/python/freetoken/engine/engine.py@@ -13,14 +13,15 @@ from freetoken.attention import AttnType, attention_backend_info, create_attentifrom freetoken.core import Batch, Context, Req, set_global_ctxfrom freetoken.distributed import destroy_distributed, enable_pynccl_distributed, set_tp_infofrom freetoken.gpu_select import gpu_identity-from freetoken.layers import set_rope_device+from freetoken.checkpoint.ftw import ftw_tensor_names, is_ftw_checkpoint+from freetoken.layers import OffloadMoELayer, iter_moe_layers, set_rope_devicefrom freetoken.layers.quantization import LayerKind, QuantBackend, finalize_quant, set_quant_backendfrom freetoken.moe.offload_cache import iter_offload_moe_layersfrom freetoken.mm.config import ENCODER_SECTIONSfrom freetoken.models import create_model, load_weightfrom freetoken.models.weight import ftw_lacks_visionfrom freetoken.moe import is_offload_moe_strategy-from freetoken.moe.expert_banks import load_expert_banks+from freetoken.moe.expert_banks import attach_resident_banks, load_expert_banksfrom freetoken.moe.host_banks import PinFailedfrom freetoken.moe.offload_cache import OffloadMoeCache, attach_offload_moe_cachefrom freetoken.utils import align_ceil, init_logger, is_sm90_family, is_sm100_family, mem_GB, torch_dtype@@ -531,9 +532,14 @@ class Engine:"family served images. Reconvert it with `ft checkpoint`, add the encoder in place with ""scripts/ftw_hotfix.py (docs/ftw-hotfix.md), or start with --text-model-only")+ resident = [layer for layer in iter_moe_layers(self.model) if not isinstance(layer, OffloadMoELayer)]+ if resident and not config.use_dummy_weight and is_ftw_checkpoint(config.model_path) and not ftw_tensor_names(config.model_path, "experts_bank"):+ raise ValueError(f"{config.model_path} keeps its routed experts as dense weights (an older `ft checkpoint --moe-backend fused`); reconvert it with `ft checkpoint`")with _weight_load_context():self.model.load_state_dict(self._load_weight_state_dict(config))finalize_quant(self.model)+ if resident:+ self._load_resident_experts(config, resident)def _load_weight_state_dict(self, config: EngineConfig) -> Dict[str, torch.Tensor]:model_state = self.model.state_dict()@@ -541,18 +547,32 @@ class Engine:return _make_dummy_weight_state_dict(model_state, device=self.device)# _materialize casts each loaded tensor to its model-param dtype (model_state), so# models declaring per-tensor dtypes (e.g. DSV4's mixed fp8/fp32/bf16) are preserved;- # offload models exclude experts (served from the offload cache, not dense weights).+ # MoE models exclude routed experts (they load as expert banks, not dense weights).return _materialize_loaded_weight_state_dict(model_state,load_weight(config.model_path,self.device,- include_moe_experts=not is_offload_moe_strategy(config.moe_strategy),+ include_moe_experts=not getattr(config.model_config, "is_moe", False),include_vision=bool(config.active_encoders),),device=self.device,)+ def _load_resident_experts(self, config: EngineConfig, layers) -> None:+ with _weight_load_context():+ banks = load_expert_banks(+ config.model_path,+ config.model_config,+ method=layers[0].quant_method,+ device=self.device,+ dtype=self.dtype,+ dummy=config.use_dummy_weight,+ parallel={"serial": False, "parallel": True}.get(config.expert_load, None),+ resident=True,+ )+ attach_resident_banks(layers, banks)+@torch.inference_mode()def _warmup_encoders(self) -> None:for item in self.mm_processor.dummy_items(self.dtype, self.device):@@ -1135,15 +1155,14 @@ def _is_unified_memory_gpu(index: "int | None" = None) -> bool:return False+_RESIDENT_EXPERT_QUANTS = ("none", "fp8_block", "nvfp4")++def _fused_resident_ok(model_config) -> bool:- """Whether the resident ('fused') MoE path can hold this model's experts.-- FIXME: auto resolves to fused only for bf16 and fp8_block experts; drop this gate once the other quant formats support fused.- """- expert_quant = getattr(model_config, "expert_quant", "none")- if expert_quant not in ("none", "fp8_block"):+ """Whether the resident ('fused') MoE path can hold this model's experts; GGUF q4_0 and DeepSeek-V4 experts stay offload."""+ if getattr(model_config, "expert_quant", "none") not in _RESIDENT_EXPERT_QUANTS:return False- return getattr(model_config, "moe_weight_format", None) in (None, "bf16")+ return getattr(model_config, "moe_weight_format", None) in (None, "bf16", "mxfp4")def _ensure_expandable_segments() -> None:@@ -1594,8 +1613,8 @@ def _adjust_config(config: EngineConfig):if is_moe and config.moe_strategy == "auto":# A MoE model always defaults to the offload family: experts stream from pinned host# banks into an auto-sized GPU slot cache, which is the only default that serves a model- # bigger than the GPU. The resident 'fused' path (bf16 / block-fp8 experts, the two- # formats MoELayer can allocate) is still reachable, but only when asked for explicitly+ # bigger than the GPU. The resident 'fused' path (every expert format but GGUF q4_0 and+ # DeepSeek-V4's) is still reachable, but only when asked for explicitly# -- auto never picks it, because nothing here knows whether the experts would fit in# HBM and a wrong guess is a weight-load OOM rather than a slower-but-working run.default_backend = "offload"@@ -1614,6 +1633,18 @@ def _adjust_config(config: EngineConfig):"Unified-memory GPU detected; auto-selecting 'fused' MoE strategy ""(resident experts) instead of offload")+ # the cache sizing flags get their own warning in the 'fused' block below+ inert = [flag for flag, on in (+ (f"--moe-cpu-layers={config.moe_cpu_layers}", config.moe_cpu_layers),+ ("--moe-prefill-hit-d2d", config.moe_prefill_hit_d2d),+ ("--disable-moe-prefill-overlap", not config.moe_prefill_overlap),+ ) if on]+ if inert:+ logger.warning_rank0(+ f"Unified-memory GPU serves MoE experts resident; ignoring {', '.join(inert)} "+ "(use --moe-strategy offload to keep the offload flags)"+ )+ override("moe_cpu_layers", None)# Hardware-adaptive config: a cached `ft bench bw` profile can upgrade# the offload default to hybrid when this machine's CPU MoE bandwidth clears its PCIe# gather bandwidth by the bench threshold (default 2x). hybrid is VRAM-equivalent to@@ -1733,7 +1764,7 @@ def _adjust_config(config: EngineConfig):if (is_moe- and expert_quant not in ("none", "fp8_block")+ and expert_quant not in _RESIDENT_EXPERT_QUANTSand not is_offload_moe_strategy(config.moe_strategy)):raise ValueError(diff --git a/python/freetoken/layers/__init__.py b/python/freetoken/layers/__init__.pyindex d868751a..a0748462 100644--- a/python/freetoken/layers/__init__.py+++ b/python/freetoken/layers/__init__.py@@ -16,7 +16,7 @@ from .linear import (LinearReplicated,LinearRowParallel,)-from .moe import MoELayer, OffloadMoELayer, make_moe_layer+from .moe import MoELayer, OffloadMoELayer, iter_moe_layers, make_moe_layerfrom .norm import (GatedRMSNorm,GemmaPlusOneRMSNorm,@@ -57,5 +57,6 @@ __all__ = ["LinearReplicated","MoELayer","OffloadMoELayer",+ "iter_moe_layers","make_moe_layer",]diff --git a/python/freetoken/layers/moe.py b/python/freetoken/layers/moe.pyindex c2094549..996beaa7 100644--- a/python/freetoken/layers/moe.py+++ b/python/freetoken/layers/moe.py@@ -1,5 +1,5 @@import os-from typing import TYPE_CHECKING, Tuple+from typing import TYPE_CHECKING, Iterator, Tupleimport torchfrom freetoken.core import get_global_ctx@@ -435,6 +435,19 @@ class OffloadMoELayer(MoELayer):raise AssertionError(f"offload experts without a quant method only serve q4_0 banks, got {fmt!r}")+def iter_moe_layers(model) -> Iterator[MoELayer]:+ if isinstance(model, MoELayer):+ yield model+ if not isinstance(model, BaseOP):+ return+ for value in model.__dict__.values():+ if isinstance(value, BaseOP):+ yield from iter_moe_layers(value)+ elif isinstance(value, (list, tuple)):+ for item in value:+ yield from iter_moe_layers(item)++def make_moe_layer(config: "ModelConfig",*,@@ -483,10 +496,10 @@ def make_moe_layer(has_bias=has_bias,quant_config=quant_config,prefix=prefix,+ layer_id=layer_id,)if offload:assert layer_id is not None, "offload MoE backends need the layer_id"- kwargs["layer_id"] = layer_idkwargs["strategy"] = config.moe_strategykwargs["decode_target"] = config.decode_targetreturn layer_cls(**kwargs)diff --git a/python/freetoken/layers/quantization/moe/base.py b/python/freetoken/layers/quantization/moe/base.pyindex 89f13ed0..e4a428d3 100644--- a/python/freetoken/layers/quantization/moe/base.py+++ b/python/freetoken/layers/quantization/moe/base.py@@ -32,6 +32,7 @@ class MoEConfig:apply_router_weight_on_input: bool = Falsestrategy: str = "resident"decode_target: str = "gpu"+ dtype: torch.dtype = torch.bfloat16@classmethoddef from_layer(cls, layer: Any, scheme: QuantScheme | None) -> "MoEConfig":@@ -52,6 +53,7 @@ class MoEConfig:apply_router_weight_on_input=bool(layer.apply_router_weight_on_input),strategy=layer.strategy,decode_target=layer.decode_target,+ dtype=torch.get_default_dtype(),)@property@@ -147,9 +149,7 @@ class MoEKernel(ABC):"""Most GPU cache slots the kernel can address for ``cfg``; None for no limit."""return self.max_slots- def _common_reject(self, cfg: MoEConfig, *, resident_ok: bool, tp_ok: bool, cpu_ok: bool, plain_silu_only: bool) -> str | None:- if not resident_ok and cfg.strategy == "resident":- return "not served resident; use --moe-strategy offload or cpu"+ def _common_reject(self, cfg: MoEConfig, *, tp_ok: bool, cpu_ok: bool, plain_silu_only: bool) -> str | None:if not tp_ok and cfg.tp_size > 1:return "TP > 1 is not supported for this expert format"if not cpu_ok and cfg.decode_target != "gpu":@@ -190,12 +190,15 @@ class MoEMethod(QuantMethod):def cpu_format(self) -> str | None:return self.kernel.cpu_format- @abstractmethod- def create_weights(self, layer: Any) -> None: ...+ def create_weights(self, layer: Any) -> None:+ # resident experts stay out of the state dict; attach_resident_banks fills these after the dense weights load+ layer._expert_banks = None+ layer._expert_alphas = Nonedef finalize(self, layer: Any) -> None:pass- @abstractmethoddef resident_view(self, layer: Any) -> ExpertView:"""Layout roles -> the resident layer's tensors, for apply()."""+ assert layer._expert_banks is not None, "resident expert banks were never loaded"+ return ExpertView(layer._expert_banks, n=self.cfg.num_experts, alphas=layer._expert_alphas)diff --git a/python/freetoken/layers/quantization/moe/fp8_block.py b/python/freetoken/layers/quantization/moe/fp8_block.pyindex 6c0167b7..20c3dde9 100644--- a/python/freetoken/layers/quantization/moe/fp8_block.py+++ b/python/freetoken/layers/quantization/moe/fp8_block.py@@ -21,7 +21,7 @@ class TritonFp8BlockMoEKernel(MoEKernel):name = "triton"def unusable_reason(self, cfg: MoEConfig) -> str | None:- reason = self._common_reject(cfg, resident_ok=True, tp_ok=False, cpu_ok=False, plain_silu_only=False)+ reason = self._common_reject(cfg, tp_ok=False, cpu_ok=False, plain_silu_only=False)if reason:return reasonreason = gated_epilogue_reason(cfg)@@ -64,17 +64,3 @@ class TritonFp8BlockMoEKernel(MoEKernel):@register_method(QuantKind.FP8_BLOCK, LayerKind.MOE)class Fp8BlockMoEMethod(MoEMethod):candidates = (TritonFp8BlockMoEKernel,)-- def create_weights(self, layer) -> None:- g = self.cfg- e, i, h, b = g.num_experts, g.intermediate, g.hidden, BLOCK- layer.gate_up_proj = torch.empty(e, 2 * i, h, dtype=FP8)- layer.gate_up_scale_inv = torch.empty(e, 2 * i // b, h // b, dtype=torch.bfloat16)- layer.down_proj = torch.empty(e, h, i, dtype=FP8)- layer.down_scale_inv = torch.empty(e, h // b, i // b, dtype=torch.bfloat16)-- def resident_view(self, layer) -> ExpertView:- return ExpertView({- "gate_up": layer.gate_up_proj, "gate_up_scale": layer.gate_up_scale_inv,- "down": layer.down_proj, "down_scale": layer.down_scale_inv,- })diff --git a/python/freetoken/layers/quantization/moe/mxfp4.py b/python/freetoken/layers/quantization/moe/mxfp4.pyindex aa020ee3..13cd9746 100644--- a/python/freetoken/layers/quantization/moe/mxfp4.py+++ b/python/freetoken/layers/quantization/moe/mxfp4.py@@ -22,7 +22,7 @@ class TritonMxfp4MoEKernel(MoEKernel):return "standard MXFP4 kernel reads the concatenated gate|up row order"if (cfg.alpha, cfg.beta) != (1.0, 0.0):return "standard MXFP4 kernel has no alpha / beta in its swiglu"- return self._common_reject(cfg, resident_ok=False, tp_ok=False, cpu_ok=True, plain_silu_only=False)+ return self._common_reject(cfg, tp_ok=False, cpu_ok=True, plain_silu_only=False)def layout(self, cfg: MoEConfig) -> dict[str, BankSpec]:i, h = cfg.intermediate, cfg.hidden@@ -106,37 +106,3 @@ class TritonGptossMxfp4MoEKernel(MoEKernel):@register_method(QuantKind.MXFP4, LayerKind.MOE)class Mxfp4MoEMethod(MoEMethod):candidates = (TritonMxfp4MoEKernel, TritonGptossMxfp4MoEKernel)-- def create_weights(self, layer) -> None:- if not self.cfg.has_bias:- raise NotImplementedError("standard MXFP4 experts are served from the offload cache, not resident")- g = self.cfg- e, i, h = g.num_experts, g.local_intermediate, g.hidden- if h % GROUP:- raise ValueError(f"MXFP4 hidden size must be divisible by {GROUP}")- layer.gate_up_proj_blocks = torch.empty(e, 2 * i, h // GROUP, 16, dtype=torch.uint8)- layer.gate_up_proj_scales = torch.empty(e, 2 * i, h // GROUP, dtype=torch.uint8)- layer.gate_up_proj_bias = torch.empty(e, 2 * i, dtype=torch.bfloat16)- layer.down_proj_blocks = torch.empty(e, h, i // GROUP, 16, dtype=torch.uint8)- layer.down_proj_scales = torch.empty(e, h, i // GROUP, dtype=torch.uint8)- layer.down_proj_bias = torch.empty(e, h, dtype=torch.bfloat16)-- def finalize(self, layer) -> None:- if getattr(layer, "gate_up_proj_blocks", None) is None:- return- from freetoken.moe.fused_mxfp4 import _transpose_mxfp4_for_decode-- # one transposed copy serves prefill and decode; the HF blocks are freed so 120B fits- layer._gu_blocks_t, layer._gu_scales_t = _transpose_mxfp4_for_decode(layer.gate_up_proj_blocks, layer.gate_up_proj_scales)- layer._dn_blocks_t, layer._dn_scales_t = _transpose_mxfp4_for_decode(layer.down_proj_blocks, layer.down_proj_scales)- layer.gate_up_proj_blocks = None- layer.gate_up_proj_scales = None- layer.down_proj_blocks = None- layer.down_proj_scales = None- torch.cuda.empty_cache()-- def resident_view(self, layer) -> ExpertView:- return ExpertView({- "gate_up": layer._gu_blocks_t, "gate_up_scale": layer._gu_scales_t, "gate_up_bias": layer.gate_up_proj_bias,- "down": layer._dn_blocks_t, "down_scale": layer._dn_scales_t, "down_bias": layer.down_proj_bias,- })diff --git a/python/freetoken/layers/quantization/moe/nvfp4.py b/python/freetoken/layers/quantization/moe/nvfp4.pyindex bb341cdf..cf66e207 100644--- a/python/freetoken/layers/quantization/moe/nvfp4.py+++ b/python/freetoken/layers/quantization/moe/nvfp4.py@@ -1,4 +1,4 @@-"""NVFP4 experts: served from the offload cache through the Triton inline-dequant kernels, vLLM's Marlin or flashinfer's b12x.+"""NVFP4 experts: served resident or from the offload cache through the Triton inline-dequant kernels, vLLM's Marlin or flashinfer's b12x.FreeToken stores MoE experts as ModelOpt NVFP4 (packed e2m1 codes + fp8-e4m3 per-16block scales + per-tensor global scale) in pinned host banks and gathers the routed@@ -41,7 +41,7 @@ class TritonNvfp4MoEKernel(MoEKernel):cpu_format = "nvfp4"def unusable_reason(self, cfg: MoEConfig) -> str | None:- reason = self._common_reject(cfg, resident_ok=False, tp_ok=False, cpu_ok=True, plain_silu_only=False)+ reason = self._common_reject(cfg, tp_ok=False, cpu_ok=True, plain_silu_only=False)if reason:return reasonreason = gated_epilogue_reason(cfg)@@ -230,7 +230,7 @@ class MarlinNvfp4MoEKernel(MoEKernel):def unusable_reason(self, cfg: MoEConfig) -> str | None:if not backend.is_vllm_installed():return "vLLM is not installed"- reason = self._common_reject(cfg, resident_ok=False, tp_ok=False, cpu_ok=False, plain_silu_only=True)+ reason = self._common_reject(cfg, tp_ok=False, cpu_ok=False, plain_silu_only=True)if reason:return reasonif not _marlin_symbols_ok():@@ -496,7 +496,7 @@ class B12xNvfp4MoEKernel(MoEKernel):return f"b12x requires sm_120+, got sm_{cc[0]}{cc[1]}"if not backend.is_flashinfer_installed():return "flashinfer is not installed"- reason = self._common_reject(cfg, resident_ok=False, tp_ok=False, cpu_ok=False, plain_silu_only=True)+ reason = self._common_reject(cfg, tp_ok=False, cpu_ok=False, plain_silu_only=True)if reason:return reasonreturn _b12x_unusable_reason(cc)@@ -567,9 +567,3 @@ class B12xNvfp4MoEKernel(MoEKernel):@register_method(QuantKind.NVFP4, LayerKind.MOE)class Nvfp4MoEMethod(MoEMethod):candidates = (TritonNvfp4MoEKernel, MarlinNvfp4MoEKernel, B12xNvfp4MoEKernel)-- def create_weights(self, layer) -> None:- raise NotImplementedError("NVFP4 experts are served from the offload cache, not resident")-- def resident_view(self, layer) -> ExpertView:- raise NotImplementedError("NVFP4 experts are not resident")diff --git a/python/freetoken/layers/quantization/moe/unquantized.py b/python/freetoken/layers/quantization/moe/unquantized.pyindex ef435686..ded7d813 100644--- a/python/freetoken/layers/quantization/moe/unquantized.py+++ b/python/freetoken/layers/quantization/moe/unquantized.py@@ -19,7 +19,9 @@ class FusedMoEKernel(MoEKernel):def layout(self, cfg: MoEConfig) -> dict[str, BankSpec]:i = cfg.local_intermediate- return {"gate_up": BankSpec((2 * i, cfg.hidden), torch.bfloat16), "down": BankSpec((cfg.hidden, i), torch.bfloat16)}+ # resident experts compute in the model dtype; offload banks stay bf16, the FTW and CPU executor format+ dt = cfg.dtype if cfg.strategy == "resident" else torch.bfloat16+ return {"gate_up": BankSpec((2 * i, cfg.hidden), dt), "down": BankSpec((cfg.hidden, i), dt)}def pack(self, pieces, cfg: MoEConfig, out):out["gate_up"].copy_(fused_piece(pieces, "gate_up"))@@ -37,12 +39,3 @@ class FusedMoEKernel(MoEKernel):@register_method(QuantKind.NONE, LayerKind.MOE)class UnquantizedMoEMethod(MoEMethod):candidates = (FusedMoEKernel,)-- def create_weights(self, layer) -> None:- g = self.cfg- i = g.local_intermediate- layer.gate_up_proj = torch.empty(g.num_experts, 2 * i, g.hidden)- layer.down_proj = torch.empty(g.num_experts, g.hidden, i)-- def resident_view(self, layer) -> ExpertView:- return ExpertView({"gate_up": layer.gate_up_proj, "down": layer.down_proj})diff --git a/python/freetoken/models/qwen3_5_moe/weight.py b/python/freetoken/models/qwen3_5_moe/weight.pyindex 1bac8227..de5b9e89 100644--- a/python/freetoken/models/qwen3_5_moe/weight.py+++ b/python/freetoken/models/qwen3_5_moe/weight.py@@ -1,6 +1,6 @@"""Qwen3.5 / 3.6 / 3.8 checkpoint reader.-The dense pass reads every Linear module under the scheme the checkpoint's QuantConfig gives it, the same answer the model built its buffers from, so bf16, block-fp8, ModelOpt and llm-compressor exports in any mix all land as the model's state dict. Routed experts are read by the offload cache (``nvfp4_expert_spec`` / ``iter_expert_pieces``); only bf16 stacked experts and resident block-fp8 experts come from here.+The dense pass reads every Linear module under the scheme the checkpoint's QuantConfig gives it, the same answer the model built its buffers from, so bf16, block-fp8, ModelOpt and llm-compressor exports in any mix all land as the model's state dict. Routed experts are read by the expert-bank loader (``nvfp4_expert_spec`` / ``iter_expert_pieces``); only the bf16 stacked experts it reads come from here."""from __future__ import annotations@@ -241,9 +241,9 @@ def iter_weights(include_non_moe: bool,include_vision: bool = True,) -> Iterator[tuple[str, torch.Tensor]]:- """Yield the dense weights fused to the model's buffers, and the routed experts only where a resident path takes them from here: bf16 stacked experts as stored, block-fp8 experts restacked per layer.+ """Yield the dense weights fused to the model's buffers, and with ``include_moe_experts`` the bf16 stacked experts as stored.- Per-expert NVFP4 experts always come from the offload cache's expert reader.+ Block-fp8 and NVFP4 experts always come from the expert-bank reader."""if get_tp_info().size > 1:raise NotImplementedError("qwen3_5_moe weight loading supports TP=1 only")@@ -253,8 +253,6 @@ def iter_weights(if include_non_moe or stacked:reader = _DenseReader(get_quant_config(), get_model_spec(hf_config.architectures[0])) if include_non_moe else Noneyield from _iter_shards(model_path, device, reader, stacked=stacked, include_vision=include_vision)- if include_moe_experts and config.is_moe and config.expert_quant == "fp8_block":- yield from _resident_fp8_experts(model_path, config)def _iter_shards(model_path: str, device: torch.device, reader: _DenseReader | None, *, stacked: bool, include_vision: bool):@@ -327,7 +325,7 @@ def iter_weights_parallel(# ======================================================================================-# Block-FP8 routed experts (Qwen3.5-35B-A3B-FP8): offload expert pieces and resident stacks.+# Block-FP8 routed experts (Qwen3.5-35B-A3B-FP8): expert pieces.# ======================================================================================# Routed-expert checkpoint key (per-expert, un-fused). ``mtp.layers...`` is excluded by the@@ -338,35 +336,6 @@ _FP8_EXPERT_KEY_RE = ()-def _resident_fp8_experts(model_path, config):- from freetoken.kernel.triton.fp8_block_linear import FP8-- B = 128- L, E, H, I, dense = _moe_dims(config)- shapes = {- "gate_up_proj": ((E, 2 * I, H), FP8),- "gate_up_scale_inv": ((E, 2 * I // B, H // B), torch.bfloat16),- "down_proj": ((E, H, I), FP8),- "down_scale_inv": ((E, H // B, I // B), torch.bfloat16),- }- layers: dict[int, dict[str, torch.Tensor]] = {}- placed = [0] * L- for li, e0, e1, piece in iter_expert_pieces(model_path, config, QuantKind.FP8_BLOCK, parallel=None):- stack = layers.setdefault(li, {n: torch.empty(shape, dtype=dt) for n, (shape, dt) in shapes.items()})- stack["gate_up_proj"][e0:e1, :I] = piece["gate"]- stack["gate_up_proj"][e0:e1, I:] = piece["up"]- stack["gate_up_scale_inv"][e0:e1, : I // B] = piece["gate_scale"]- stack["gate_up_scale_inv"][e0:e1, I // B :] = piece["up_scale"]- stack["down_proj"][e0:e1] = piece["down"]- stack["down_scale_inv"][e0:e1] = piece["down_scale"]- placed[li] += e1 - e0- if placed[li] == E:- pre = f"model.layers.{dense + li}.mlp.experts"- for name, tensor in layers.pop(li).items():- yield f"{pre}.{name}", tensor- assert not layers, f"incomplete resident fp8 experts for layers {sorted(layers)}"--def _moe_dims(model_config):L = model_config.num_moe_layersreturn (diff --git a/python/freetoken/moe/expert_banks.py b/python/freetoken/moe/expert_banks.pyindex e3a768b6..7500174a 100644--- a/python/freetoken/moe/expert_banks.py+++ b/python/freetoken/moe/expert_banks.py@@ -75,6 +75,7 @@ def build_expert_banks(device: torch.device,layer_sink=None,dummy: bool = False,+ resident: bool = False,) -> ExpertBanks:"""Fill host banks in the kernel's layout from a stream of expert pieces.@@ -83,6 +84,7 @@ def build_expert_banks(complete once its ``num_experts`` rows have arrived: with ``layer_sink=None`` its banksare pinned in the background, otherwise the sink receives them (converter). ``dummy``skips the pieces and fills the banks with finite random contents.+ ``resident`` packs into banks on ``device`` instead of host banks (the resident 'fused' experts)."""from freetoken.moe.host_banks import LayerCompletionTracker, PinPipeline, pin_banksfrom freetoken.moe.legacy_format import legacy_format_for@@ -91,8 +93,13 @@ def build_expert_banks(layout = method.layout()E = method.cfg.num_expertsspecs = {role: ((E, *spec.shape), spec.dtype) for role, spec in layout.items() if not spec.resident}- hb = alloc_layer_banks(specs, num_layers)- banks = {role: [b.tensor for b in hb[role]] for role in specs}+ if resident:+ assert layer_sink is None+ hb = None+ banks = {role: [torch.empty(shape, dtype=dtype, device=device) for _ in range(num_layers)] for role, (shape, dtype) in specs.items()}+ else:+ hb = alloc_layer_banks(specs, num_layers)+ banks = {role: [b.tensor for b in hb[role]] for role in specs}alphas = {role: torch.empty(num_layers * E, dtype=spec.dtype, device=device)for role, spec in layout.items() if spec.resident@@ -104,7 +111,7 @@ def build_expert_banks(_dummy_fill(role, tensor)for alpha in alphas.values():alpha.fill_(1.0)- if torch.cuda.is_available():+ if hb is not None and torch.cuda.is_available():pin_banks(hb)return ExpertBanks(legacy_format_for(method.kind, kernel.name), banks,@@ -136,7 +143,7 @@ def build_expert_banks(if layer_sink is not None:_fill(layer_sink)- elif torch.cuda.is_available():+ elif torch.cuda.is_available() and not resident:with PinPipeline() as pins:_fill(pins)else:@@ -192,16 +199,16 @@ def _legacy_expert_banks(model_path, model_config, device, dtype, dummy, paralle)-def _method_expert_banks(model_path, model_config, method, device, dummy, parallel, workers, chunk, layer_sink=None) -> ExpertBanks:+def _method_expert_banks(model_path, model_config, method, device, dummy, parallel, workers, chunk, layer_sink=None, resident=False) -> ExpertBanks:from freetoken.moe.expert_pieces import iter_expert_piecesnum_layers = model_config.num_moe_layersif dummy:- return build_expert_banks(method, num_layers, None, device=device, dummy=True)+ return build_expert_banks(method, num_layers, None, device=device, dummy=True, resident=resident)pieces = iter_expert_pieces(model_path, model_config, method.kind, parallel=parallel, workers=workers, chunk=chunk)- return build_expert_banks(method, num_layers, pieces, device=device, layer_sink=layer_sink)+ return build_expert_banks(method, num_layers, pieces, device=device, layer_sink=layer_sink, resident=resident)def _host_ram_fits_parallel(model_path: str) -> bool:@@ -286,6 +293,7 @@ def load_expert_banks(decode_target: str = "gpu",layer_sink=None,layer_residency: list[str] | None = None,+ resident: bool = False,) -> ExpertBanks:"""Load (or fabricate, with ``dummy=True``) the expert banks. Two paths, both returningthe same normalized ``ExpertBanks`` and both pinning after fill:@@ -310,14 +318,21 @@ def load_expert_banks(``layer_residency``: per-layer ``HostResidency`` labels applied at settle time -- explicitly on the FTW fast path, ambiently (``requested_residency``) in the slow-path providers.Applied labels are echoed on ``ExpertBanks.layer_residency``; a loader that settles some other way leaves it ``None`` (CPU-layer decode still works on pinned banks, it just saves no pin quota).++ ``resident`` (with ``method``): the banks land on ``device`` for the resident 'fused' experts; no host bank outlives the load."""- from freetoken.checkpoint.ftw import is_ftw_checkpoint, load_ftw_banks+ from freetoken.checkpoint.ftw import is_ftw_checkpoint, load_ftw_banks, load_ftw_banks_to_device+ if resident:+ assert method is not None and layer_sink is None and layer_residency is Noneif model_path and is_ftw_checkpoint(model_path) and not dummy:- banks = load_ftw_banks(- model_path, num_layers=model_config.num_moe_layers, workers=workers, chunk=chunk,- layer_residency=layer_residency,- )+ if resident:+ banks = load_ftw_banks_to_device(model_path, num_layers=model_config.num_moe_layers, device=device, workers=workers, chunk=chunk)+ else:+ banks = load_ftw_banks(+ model_path, num_layers=model_config.num_moe_layers, workers=workers, chunk=chunk,+ layer_residency=layer_residency,+ )if banks is not None:logger.info_rank0(f"expert banks: FTW fast path (FTW checkpoint {model_path})")return banks@@ -354,7 +369,7 @@ def load_expert_banks(def _build(par: bool) -> ExpertBanks:if method is not None:- return _method_expert_banks(model_path, model_config, method, device, dummy, par, workers, chunk, layer_sink)+ return _method_expert_banks(model_path, model_config, method, device, dummy, par, workers, chunk, layer_sink, resident)return _legacy_expert_banks(model_path, model_config, device, dtype, dummy, par, workers, chunk, decode_target, layer_sink)with requested_residency(layer_residency) as residency_plan:@@ -393,3 +408,33 @@ def _echo_residency(banks: ExpertBanks, requested, plan) -> ExpertBanks:"but saves no pinned quota")return banks+++def attach_resident_banks(layers, banks: ExpertBanks) -> None:+ """Give each resident MoE layer its rows of the device ``banks``; bank layer ``i`` goes to the layer whose ``layer_id`` is ``i``."""+ ids = sorted(layer.layer_id for layer in layers if layer.layer_id is not None)+ num_layers = len(next(iter(banks.sources.values())))+ if ids != list(range(num_layers)) or len(ids) != len(layers):+ raise ValueError(f"resident MoE layer ids {ids} do not cover the {num_layers} expert bank layers")+ for layer in layers:+ i, e = layer.layer_id, layer.num_experts+ layout = layer.quant_method.layout()+ rows = {role: (e, *spec.shape) for role, spec in layout.items() if not spec.resident}+ if set(banks.sources) != set(rows):+ raise ValueError(f"expert banks hold {sorted(banks.sources)}, the {layer.quant_method.kernel.name} kernel reads {sorted(rows)}")+ for role, shape in rows.items():+ bank = banks.sources[role][i]+ if tuple(bank.shape) != shape or bank.dtype != layout[role].dtype:+ raise ValueError(+ f"bank {role!r} of layer {i} is {tuple(bank.shape)} {bank.dtype} but the {layer.quant_method.kernel.name} kernel "+ f"wants {shape} {layout[role].dtype}; the banks were packed for another kernel or model"+ )+ alphas = []+ # apply() reads view.alphas as (gate_up, down)+ for role in (role for role in ("gate_up_alpha", "down_alpha") if role in layout):+ a = getattr(banks, role)+ if a is None or a.numel() != num_layers * e or a.dtype != layout[role].dtype:+ raise ValueError(f"the {layer.quant_method.kernel.name} kernel needs {num_layers * e} {layout[role].dtype} {role} values, the expert banks carry {None if a is None else (a.numel(), a.dtype)}")+ alphas.append(a)+ layer._expert_banks = {role: per_layer[i] for role, per_layer in banks.sources.items()}+ layer._expert_alphas = tuple(a[i * e : (i + 1) * e] for a in alphas) or Nonediff --git a/python/freetoken/moe/offload_cache.py b/python/freetoken/moe/offload_cache.pyindex 32e0f3cc..c6f975df 100644--- a/python/freetoken/moe/offload_cache.py+++ b/python/freetoken/moe/offload_cache.py@@ -1054,20 +1054,9 @@ class OffloadMoeCache:def iter_offload_moe_layers(model) -> Iterator:- from freetoken.layers import BaseOP, OffloadMoELayer+ from freetoken.layers import OffloadMoELayer, iter_moe_layers- if isinstance(model, OffloadMoELayer):- yield model-- if not isinstance(model, BaseOP):- return-- for value in model.__dict__.values():- if isinstance(value, BaseOP):- yield from iter_offload_moe_layers(value)- elif isinstance(value, (list, tuple)):- for item in value:- yield from iter_offload_moe_layers(item)+ return (layer for layer in iter_moe_layers(model) if isinstance(layer, OffloadMoELayer))def attach_offload_moe_cache(model, cache: OffloadMoeCache) -> list:diff --git a/python/freetoken/server/args.py b/python/freetoken/server/args.pyindex 23a62f84..4dd57b1a 100644--- a/python/freetoken/server/args.py+++ b/python/freetoken/server/args.py@@ -600,8 +600,8 @@ def parse_args(choices=["auto", *MOE_STRATEGIES],help=("How the routed experts are served. 'auto' resolves a MoE model to the offload family "- "(offload, or hybrid when a `ft bench bw` profile recommends it); resident "- "'fused' experts must be requested explicitly."+ "(offload, or hybrid when a `ft bench bw` profile recommends it), and to resident "+ "'fused' experts on unified-memory GPUs (GB10 / DGX Spark)."),)diff --git a/tests/engine/test_cache_budget.py b/tests/engine/test_cache_budget.pyindex 073112bb..a77409f0 100644--- a/tests/engine/test_cache_budget.py+++ b/tests/engine/test_cache_budget.py@@ -367,7 +367,7 @@ def _offload_engine_config(**overrides):num_layers=10,num_moe_layers=10,num_experts=8,- expert_quant="nvfp4", # quantized experts -> must resolve to an offload backend+ expert_quant="nvfp4",moe_strategy="auto",),)@@ -389,7 +389,7 @@ def test_guard_raises_actionable_error_when_too_small():assert "128" in msg and "moe-cache" in msg-def test_adjust_config_defaults_moe_cache_auto_for_auto_resolved_offload_backend():+def test_adjust_config_defaults_moe_cache_auto_for_auto_resolved_offload_backend(monkeypatch):"""Bare `ft serve <FTW MoE checkpoint>`: no --moe-backend, no --moe-cache-* flags at all.args.py's parse-time default only fires when the backend is *already*@@ -400,9 +400,12 @@ def test_adjust_config_defaults_moe_cache_auto_for_auto_resolved_offload_backendmoe_cache_auto=True, so _init_offload_moe_cache's _require_offload_cache_size guard is neverreached with moe_cache_size still 0."""+ import freetoken.engine.engine as engine_modulefrom freetoken.engine.engine import _adjust_configfrom freetoken.moe import is_offload_moe_strategy+ # a unified-memory host (GB10) resolves auto to fused instead+ monkeypatch.setattr(engine_module, "_is_unified_memory_gpu", lambda index=None: False)config = _offload_engine_config()_adjust_config(config)diff --git a/tests/models/qwen4_exp/test_skeleton.py b/tests/models/qwen4_exp/test_skeleton.pyindex ac598e65..8fb9112d 100644--- a/tests/models/qwen4_exp/test_skeleton.py+++ b/tests/models/qwen4_exp/test_skeleton.py@@ -16,7 +16,7 @@ import pytestimport torchimport torch.nn.functional as F-from freetoken.layers import BaseOP, LinearReplicated+from freetoken.layers import BaseOP, LinearReplicated, iter_moe_layersfrom freetoken.models.config import ModelConfigfrom freetoken.models.qwen4_exp.config import parse_configfrom freetoken.models.qwen4_exp.hc import GatedResidual@@ -30,12 +30,17 @@ def _config(num_layers: int = 4) -> ModelConfig:def _fill(op, gen: torch.Generator, scale: float = 0.05) -> None:- """Random floats / zeroed ints for every state-dict tensor of an op tree."""+ """Random floats / zeroed ints for every state-dict tensor of an op tree, and random expert banks for its resident MoE layers."""for tensor in op.state_dict().values():if tensor.is_floating_point():tensor.normal_(0.0, scale, generator=gen)else:tensor.zero_()+ for layer in iter_moe_layers(op):+ layer._expert_banks = {+ role: torch.randn((layer.num_experts, *spec.shape), generator=gen, device=gen.device, dtype=spec.dtype) * scale+ for role, spec in layer.quant_method.layout().items()+ }def _group_norm(x, weight, eps, groups):diff --git a/tests/models/test_quant_config.py b/tests/models/test_quant_config.pyindex 6ac1b84b..084a2e4a 100644--- a/tests/models/test_quant_config.py+++ b/tests/models/test_quant_config.py@@ -121,14 +121,6 @@ class Case:check: Callable[[Any, dict], None] | None = None # family extras beyond the method table-def _installed(module: str) -> bool:- try:- __import__(module)- return True- except ImportError:- return False--def _qwen35_fp8(model, idx):assert model.model.layers.op_list[idx["lin"]].linear_attn._split_in_proj@@ -162,13 +154,18 @@ def _dsv4(model, idx):assert "model.head.weight" in keys and not any(".ffn.experts." in k for k in keys)+def _banked_experts(model, idx):+ experts = op_at(model, f"model.layers.{idx['lin']}.mlp.experts")+ assert not experts.state_dict()++def _gpt_oss(cls):def check(model, idx):experts = op_at(model, "model.layers.0.mlp.experts")assert type(experts) is clsassert experts.interleaved and experts.alpha == pytest.approx(1.702) and experts.limit == 7.0if cls is GptOssMoELayer:- assert experts.gate_up_proj_blocks.dtype is torch.uint8 and experts.gate_up_proj_blocks.dim() == 4+ assert not experts.state_dict()return check@@ -198,9 +195,8 @@ CASES = ["model.layers.{lin}.mlp.shared_expert.down_proj": NVFP4, "model.layers.{lin}.mlp.experts": Nvfp4MoEMethod,"lm_head": NVFP4,}, check=_qwen35_nvfp4),- # resident NVFP4 experts have no kernel without vLLM's Marlin- Case("Qwen3.6-35B-A3B-NVFP4", ModelOptConfig, {}, strategy="fused", raises=KernelSelectionError,- skip_if=lambda: not _installed("flashinfer") or _installed("vllm")),+ # resident NVFP4 experts load through the expert-bank loader, not the state dict+ Case("Qwen3.6-35B-A3B-NVFP4", ModelOptConfig, {"model.layers.{lin}.mlp.experts": (Nvfp4MoEMethod, "triton")}, strategy="fused", check=_banked_experts),Case("Qwen3.8-27B-NVFP4", ModelOptConfig, {"model.layers.{lin}.linear_attn.in_proj_qkvz": FP8T, "model.layers.{full}.self_attn.o_proj": FP8T,"model.layers.{lin}.mlp.gate_up_proj": NVFP4, "model.layers.{lin}.mlp.down_proj": NVFP4, "lm_head": NVFP4,