FreeToken brings datacenter-scale model serving to your desktop. Run massive models locally, fast and efficiently.
feat(moe): bench hybrid fetch fractions per moe layer kind (#734)
Xiaoze Fan8f49e0af98ba
diff --git a/docs/cli.md b/docs/cli.mdindex 69848f2e..e109bde0 100644--- a/docs/cli.md+++ b/docs/cli.md@@ -11,7 +11,7 @@ ft <command> [args]| `ft ctl` | Query and manage a running server over HTTP || `ft launch` | Configure and launch a coding agent against a server || `ft checkpoint` | Convert an HF checkpoint to the FTW fast-load format |-| `ft bench bw` | Benchmark CPU vs PCIe bandwidth to calibrate the MoE backend |+| `ft bench bw` | Benchmark CPU vs PCIe bandwidth to calibrate hybrid MoE decode |`ft --version` prints the installed version (torch-free; nightly wheels carry a`+g<sha>` build stamp, tagged releases a bare version). Every command supports@@ -81,7 +81,7 @@ See [models.md](models.md#moe-strategies) for what each strategy does.| Flag | Default | Meaning ||---|---|---|-| `--moe-strategy` | auto | `fused`/`offload`/`cpu`/`hybrid`; auto → offload, or hybrid with a `ft bench bw` profile; fused on unified-memory GPUs (GB10), see [models.md](models.md#unified-memory-gpus-gb10--dgx-spark). `--moe-backend` is the deprecated old spelling |+| `--moe-strategy` | auto | `fused`/`offload`/`cpu`/`hybrid`; auto → offload; fused on unified-memory GPUs (GB10), see [models.md](models.md#unified-memory-gpus-gb10--dgx-spark). `--moe-backend` is the deprecated old spelling || `--quant-backend` | auto | Kernel per quantized layer type, `layer[.kind]=name` entries: `linear=marlin,moe=b12x` or `moe.nvfp4=triton`. A layer-level entry applies to every kind whose table lists the name || `--nvfp4-backend` | — | Deprecated: stands in for `--quant-backend moe.nvfp4=<marlin\|b12x\|triton>` (`flashinfer` means b12x); cannot be combined with `--quant-backend` || `--moe-cache-size` / `--moe-cache-rate` / `--moe-cache-mem` / `--moe-cache-auto` | auto | GPU expert-cache size as slots / fraction of all experts / memory such as `12GiB` / sized from free VRAM (mutually exclusive; auto is enabled by default for offload-family strategies) |@@ -197,20 +197,24 @@ strategies alike (DeepSeek-V4 experts serve offload only);## ft bench bw```bash-ft bench bw # once per GPU-ft bench bw --dtype nvfp4,bf16 # only the formats you serve-ft bench bw --gpu 1 # a specific GPU (UUID or nvidia-smi index, as for ft serve)+ft bench bw --model /path/to/model.gguf # every MoE layer kind of a checkpoint (GGUF or HF)+ft bench bw # canonical geometry per expert format+ft bench bw --gpu 1 # a specific GPU (UUID or nvidia-smi index, as for ft serve)```-Measures host-RAM vs PCIe bandwidth with the real cpu/offload MoE kernels and writes a-profile that `ft serve --moe-strategy auto` and `--moe-hybrid-max-fetch -1` then read.--- One profile per GPU, at `~/.cache/freetoken/benchbw/<gpu-uuid>.json`.-- Keyed on expert format + GPU, so a profile from other hardware is ignored rather than- misapplied. An older single `benchbw.json` still counts if its GPU name matches.-- What to measure: `--dtype`, `--model`, `--formats`, `--isa`.-- `--threshold` (default 2.0) sets the call: recommend hybrid when CPU bandwidth beats PCIe- by that factor.+Measures host-RAM vs PCIe bandwidth with the real cpu/offload MoE kernels and writes one+row per MoE layer kind, which `ft serve --moe-strategy hybrid --moe-hybrid-max-fetch -1` reads+to split each layer's misses.++- One profile per GPU, at `~/.cache/freetoken/benchbw/<gpu-uuid>.json`; a profile from other+ hardware or another profile version is ignored.+- A row's key is the layer's CPU format and options, expert shape, top_k and CPU worker+ threads; a layer without an exact row keeps a fetch cap of 1. Bench with the same+ `--cpu-threads` as the server's `--moe-cpu-threads`.+- Each run replaces the rows it benched and keeps the others.+- What to measure: `--model` (checkpoint paths or presets), `--dtype`, `--formats`, `--isa`.+- `--threshold` (default 2.0) sets each row's `recommended`: hybrid when CPU bandwidth beats+ PCIe by that factor.On NVLink-C2C hosts, see [models.md](models.md#nvlink-c2c-hosts-gh200--gb200)for `FREETOKEN_H2D_BLOCKS_PER_BANK` if `PCIe-gather` sits well below the linear H2D ceiling.diff --git a/docs/models.md b/docs/models.mdindex d3f26a08..4418d92b 100644--- a/docs/models.md+++ b/docs/models.md@@ -47,10 +47,10 @@ These families accept image input by default; pass `--text-model-only` to skip tmisses stream over PCIe.- **cpu** — misses are computed on the CPU instead of fetched.- **hybrid** — per step, fetches some misses over PCIe and computes the rest on- CPU, overlapped. Run `ft bench bw` once per machine to calibrate the split.+ CPU, overlapped. Run `ft bench bw --model <checkpoint>` once per machine and+ model to calibrate the split.- **auto** — dense models always resolve to `fused`; MoE models resolve to- `offload`, upgraded to `hybrid` when a cached `ft bench bw` profile- recommends it. Unified-memory GPUs differ, see below.+ `offload`. Unified-memory GPUs differ, see below.### Unified-memory GPUs (GB10 / DGX Spark)@@ -107,7 +107,7 @@ unset on PCIe GPUs: wider grids add no bandwidth there.Bounded output differs from exact by construction and does not depend on the prefill chunk size.Under expert offload the prefill time is bounded by streaming each layer's experts, so decoderreplay saves decoder-layer compute, not prefill time.- Run `ft bench bw --model dsv4.1-flash` to measure local PCIe and CPU bandwidth before+ Run `ft bench bw --model <checkpoint>` to measure local PCIe and CPU bandwidth beforechoosing a MoE strategy; `--moe-strategy hybrid` splits expert misses using thosemeasurements. With `--moe-cache-auto`, `--kv-reserve-tokens` sets the token capacity reservedbefore the expert cache is allocated, subject to the attention pool's structural minimum; size itdiff --git a/python/freetoken/engine/engine.py b/python/freetoken/engine/engine.pyindex ee655801..b93de55c 100644--- a/python/freetoken/engine/engine.py+++ b/python/freetoken/engine/engine.py@@ -1,5 +1,6 @@from __future__ import annotations+import collectionsimport contextlibimport errnoimport gc@@ -853,8 +854,6 @@ class Engine:cache.cpu_layer_ids = cpu_layer_idscache.set_bank_sources(banks.sources, layer_residency=banks.layer_residency, layer_tags=banks.layer_tags, row_tail_bytes=banks.row_tail_bytes)cache.set_alphas(banks.gate_up_alpha, banks.down_alpha)- if decode_target == "hybrid":- self._resolve_hybrid_fetch(config, cache)# Must be set before CUDA graph capture so the (device-side) accumulation ops are# captured and re-run on every decode replay.cache.collect_stats = config.moe_collect_stats@@ -862,6 +861,8 @@ class Engine:assert len(layers) == config.model_config.num_moe_layersif cache.decode_target in ("cpu", "hybrid"):self._init_cpu_moe_executor(config, cache, layers)+ if decode_target == "hybrid":+ self._resolve_hybrid_fetch(config, cache)self.ctx.moe_offload_cache = cacheself.moe_offload_cache = cachereturn cache@@ -882,30 +883,50 @@ class Engine:Perfect fetch/compute overlap wants fetched : cpu-computed misses = pcie_bw :(cpu_bw - pcie_bw), i.e. fetching a pcie_bw / cpu_bw fraction of each decodestep's misses -- both sides then finish together instead of one idling. The- achieved bandwidths come from the cached `ft bench bw` profile (the same one the- auto backend pick reads); without a usable profile the old fixed cap of 1 applies.+ achieved bandwidths come from the cached `ft bench bw` profile, from the row of each+ MoE layer's kind (format, options, expert shape, top_k, CPU worker threads); a layer+ without a row keeps the fixed cap of 1."""if config.moe_hybrid_max_fetch >= 0:return # explicit fixed cap- from freetoken.moe.bench_profile import load_hybrid_fetch_fraction+ from freetoken.moe.bench_profile import fetch_fraction, layer_kind_key, load_layer_kinds+ from freetoken.moe.cpu_executor import _format_label+ cache.hybrid_max_fetch = 1+ methods = offload_layer_methods(self.model)+ if methods is None:+ logger.warning_rank0("--moe-hybrid-max-fetch auto: the MoE layers bind no expert method to look up in the `ft bench bw` profile; using a fixed fetch cap of 1")+ returngpu_name, gpu_uuid = _profile_gpu(self.device.index)- fraction = load_hybrid_fetch_fraction(- cache.quant_format, gpu_name=gpu_name, gpu_uuid=gpu_uuid- )- if fraction is None:- cache.hybrid_max_fetch = 1+ rows = load_layer_kinds(gpu_name=gpu_name, gpu_uuid=gpu_uuid) or {}+ executor = self.cpu_moe_executor+ keys, labels = [], []+ for m in methods:+ fmt, opts = m.cpu_spec()+ c = m.cfg+ keys.append(layer_kind_key(fmt, opts, c.hidden, c.local_intermediate, c.top_k, executor.num_threads))+ labels.append(f"{_format_label(fmt, tuple(sorted(opts.items())))} {c.hidden}x{c.local_intermediate}")+ fractions = [fetch_fraction(rows.get(key)) for key in keys]+ cache.hybrid_fetch_fraction = [f or 0.0 for f in fractions]+ groups = collections.Counter(zip(labels, fractions))+ fetched = [f"{f:.1%} on {n} {label} layers" for (label, f), n in groups.items() if f is not None]+ if fetched:+ logger.info_rank0(+ "--moe-hybrid-max-fetch auto: each decode step fetches this share of a layer's expert misses over PCIe "+ f"(benched PCIe/CPU bandwidth ratio) and computes the rest on the CPU: {'; '.join(fetched)}"+ )+ if missing := sorted({label for label, f in groups if f is None}):+ capped = sum(n for (_, f), n in groups.items() if f is None)logger.warning_rank0(- "--moe-hybrid-max-fetch auto: no usable `ft bench bw` profile for "- f"{cache.quant_format!r} experts; using a fixed fetch cap of 1"+ f"--moe-hybrid-max-fetch auto: no `ft bench bw` row for {capped} MoE layers ({', '.join(missing)}) at "+ f"{executor.num_threads} CPU threads; using a fixed fetch cap of 1 there. `ft bench bw --model <checkpoint>` benches each layer kind"+ )+ benched_isas = {rows[key].get("cpu_moe_isa") for key, f in zip(keys, fractions) if f is not None} - {None}+ if other := sorted(benched_isas - set(executor.isa.split(","))):+ logger.warning_rank0(+ f"--moe-hybrid-max-fetch auto: some `ft bench bw` rows were benched on CPU kernels {other}, but this executor runs "+ f"{executor.isa}; using them anyway")- return- cache.hybrid_max_fetch = cache.num_experts # inert: the fraction is the cap- cache.hybrid_fetch_fraction = fraction- logger.info_rank0(- f"--moe-hybrid-max-fetch auto: fetching {fraction:.1%} of each decode step's "- "expert misses over PCIe (benched PCIe/CPU bandwidth ratio), the rest on the CPU"- )def _init_cpu_moe_executor(self, config: EngineConfig, cache, layers) -> None:"""Build the persistent CPU MoE executor (decode-time expert compute).@@ -1643,24 +1664,34 @@ def _check_layer_formats(banks, layer_methods) -> None:)+def _meta_model(config: EngineConfig, decode_target: str):+ from freetoken.models import create_model+ from freetoken.utils.torch_utils import torch_dtype++ set_quant_backend(_adjust_ftw_quant_backend(config.model_path, QuantBackend.parse(config.quant_backend)))+ object.__setattr__(config.model_config, "moe_strategy", config.moe_strategy)+ object.__setattr__(config.model_config, "decode_target", decode_target)+ with torch.device("meta"), torch_dtype(config.dtype):+ return create_model(config.model_config)++def offload_expert_methods(config: EngineConfig) -> tuple:"""``(method, layer_methods)`` of ``config``'s model, from a meta-device build: the most common expert method and,for experts of several formats, one per MoE layer (else None).For tools that load expert banks without an engine (the FTW converter); they pack for GPU decode."""- from freetoken.models import create_modelfrom freetoken.moe.expert_banks import is_mixed- from freetoken.utils.torch_utils import torch_dtype- set_quant_backend(_adjust_ftw_quant_backend(config.model_path, QuantBackend.parse(config.quant_backend)))- object.__setattr__(config.model_config, "moe_strategy", config.moe_strategy)- object.__setattr__(config.model_config, "decode_target", "gpu")- with torch.device("meta"), torch_dtype(config.dtype):- model = create_model(config.model_config)+ model = _meta_model(config, "gpu")layer_methods = offload_layer_methods(model)return shared_offload_method(model), (layer_methods if is_mixed(layer_methods) else None)+def meta_offload_layer_methods(config: EngineConfig, decode_target: str) -> list | None:+ """The expert method of each offload MoE layer of ``config``'s model, from a meta-device build for ``decode_target``; see ``offload_layer_methods``."""+ return offload_layer_methods(_meta_model(config, decode_target))++def offload_expert_method(config: EngineConfig):"""The (most common) offload expert method of ``config``'s model; see ``offload_expert_methods``."""return offload_expert_methods(config)[0]@@ -1789,7 +1820,7 @@ def _adjust_config(config: EngineConfig):# swigluoai (csrc ActKind; "gpt_oss_swiglu" rides inside the mxfp4 kernel and# swigluoai the generic GEMV epilogue). A model with any other expert# activation cannot decode on the CPU: reject an explicit cpu/hybrid pick at- # config time, and keep auto from upgrading offload -> hybrid off the profile.+ # config time.# hidden_act (the dense activation) stands proxy for the expert activation --# true for every in-tree model. mxfp4 experts pass regardless: their act runs# inside the mxfp4 kernel, not the generic epilogue.@@ -1825,8 +1856,7 @@ def _adjust_config(config: EngineConfig):# GB10 this added a measured ~130 s stall to every request, #369). Resident# experts are the safe default here, not the risky one: the model and its banks# page from the same pool, so the "wrong guess = weight-load OOM" rationale above- # does not apply. The benchbw hybrid upgrade is skipped too: CPU execution adds- # no bandwidth when both sides share one memory. Only formats the resident path+ # does not apply. Only formats the resident path# can actually hold take this branch; the rest stay on offload as before.unified_memory = _is_unified_memory_gpu()if unified_memory and _fused_resident_ok(model_config):@@ -1847,50 +1877,6 @@ def _adjust_config(config: EngineConfig):"(use --moe-strategy offload to keep the offload flags)")override("moe_cpu_layers", None)- # Hardware-adaptive config: a cached `ft bench bw` profile can upgrade- # the offload default to hybrid when this machine's CPU MoE bandwidth clears its PCIe- # gather bandwidth by the bench threshold (default 2x). hybrid is VRAM-equivalent to- # offload -- same auto-sized GPU slot cache (_resolve_auto_moe_cache_size), plus a- # host-RAM CPU executor -- so this never raises the OOM risk; with no profile (or one- # from different hardware) it stays offload. offload remains the always-safe fallback.- # Key the lookup on the real expert format: mxfp4 lives in moe_weight_format when- # expert_quant is "none", and "none" with no weight format means plain bf16 experts.- moe_wfmt = getattr(model_config, "moe_weight_format", None)- bench_fmt = expert_quant if expert_quant != "none" else (moe_wfmt or "bf16")- from freetoken.moe.bench_profile import load_backend_recommendation-- gpu_name, gpu_uuid = _profile_gpu()- if (- default_backend == "offload"- and not unified_memory- and load_backend_recommendation(- bench_fmt, gpu_name=gpu_name, gpu_uuid=gpu_uuid- )- == "hybrid"- ):- from freetoken.moe.cpu_executor import compiled_extension_supports-- _act = getattr(model_config, "hidden_act", "silu")- if not _cpu_moe_act_ok:- logger.info_rank0(- f"benchbw profile recommends hybrid, but the CPU MoE executor does not "- f"support this model's expert activation "- f"{getattr(model_config, 'hidden_act', None)!r}; staying on offload"- )- elif moe_wfmt != "mxfp4" and not compiled_extension_supports(_act):- # Stale prebuilt _cpu_moe.so: an explicit cpu/hybrid pick still- # hard-fails in the executor, but a default must not turn into a- # post-load crash -- degrade to offload.- logger.info_rank0(- f"benchbw profile recommends hybrid, but the compiled _cpu_moe "- f"extension predates activation {_act!r} (rebuild with "- f"`python setup.py build_ext --inplace`); staying on offload"- )- else:- default_backend = "hybrid"- logger.info_rank0(- f"benchbw profile recommends hybrid for {bench_fmt!r} experts on this GPU"- )override("moe_strategy", default_backend)logger.info_rank0(f"Auto-selected MoE strategy: {config.moe_strategy}")diff --git a/python/freetoken/moe/bench_profile.py b/python/freetoken/moe/bench_profile.pyindex 5fa38b6f..c939a947 100644--- a/python/freetoken/moe/bench_profile.py+++ b/python/freetoken/moe/bench_profile.py@@ -1,12 +1,8 @@"""Torch-free reader for the ``ft bench bw`` hardware profile (``benchbw/<gpu-uuid>.json``).-The engine consults this at MoE-backend *auto* resolution (``engine.py``) to make the-offload-vs-hybrid choice hardware-adaptive without importing the (torch-heavy) benchmark-itself. ``benchbw.py`` writes the profile; this module only reads it.--The join key is the expert *format*: the CPU-MoE-vs-PCIe-gather bandwidth ratio the choice-rides on is dominated by ``(format, hardware)``, not by the exact model, so a profile benched-on one workload transfers to any model with the same expert format on the same GPU.+The profile holds one row per MoE layer kind (``layer_kind_key``): the CPU MoE and PCIe gather+bandwidths benched for that kind on this GPU and host. The engine reads it to set each hybrid+layer's fetch split, without importing the (torch-heavy) benchmark. ``benchbw.py`` writes it."""from __future__ import annotations@@ -18,20 +14,8 @@ from freetoken.utils import init_loggerlogger = init_logger(__name__)-# Bump when a format gains or loses a CPU MoE path: verdicts and fetch fractions from an older profile are stale.-PROFILE_VERSION = 5--# Engine ``expert_quant`` (models/config.py) -> benchbw format key (offload_cache._BANK_SCHEMAS-# / benchbw._offload_bank_specs). Only the offload-family formats with a CPU MoE weight path can-# ever resolve to hybrid; anything not listed falls through unmapped and finds no profile entry-# (-> None -> offload), which is the safe default.-_QUANT_TO_BENCH_FORMAT = {- "nvfp4": "nvfp4",- "ds_fp4": "ds_fp4",- "mxfp4": "mxfp4_triton",- "bf16": "bf16",- "fp8_block": "fp8_block",-}+# Bump when the row key or the measurements change: rows of another version are not comparable.+PROFILE_VERSION = 6def _cache_dir() -> str:@@ -104,10 +88,10 @@ def _usable_profile(return Noneif not isinstance(prof, dict):return None- if not isinstance(prof.get("version"), int) or prof["version"] < PROFILE_VERSION:+ if prof.get("version") != PROFILE_VERSION:logger.warning(- f"benchbw profile {src} predates this engine (version {prof.get('version')}, "- f"need {PROFILE_VERSION}); ignoring it, rerun `ft bench bw`"+ f"benchbw profile {src} has version {prof.get('version')}, this engine reads "+ f"{PROFILE_VERSION}; ignoring it, rerun `ft bench bw`")return Noneprof_gpu = (prof.get("gpu") or {}).get("name")@@ -120,81 +104,34 @@ def _usable_profile(return prof-def load_backend_recommendation(- quant_format: str,- gpu_name: str | None = None,- path: str | None = None,- gpu_uuid: str | None = None,-) -> str | None:- """Bench-recommended offload-family backend for ``quant_format`` on this GPU, or ``None``.+def layer_kind_key(fmt: str, options: dict[str, int], hidden: int, inter: int, top_k: int, cpu_threads: int) -> str:+ """Row key of one MoE layer kind, e.g. ``gguf[down_type=8,gate_up_type=8]:2816x704:top8:t12``.- Returns ``"hybrid"`` only when *every* benched workload sharing this expert format- recommended hybrid (CPU MoE BW > threshold x PCIe gather BW); a mixed verdict (a- near-threshold format) resolves conservatively to ``"offload"``. ``None`` means "no usable- profile" (see ``_usable_profile``) or no entry for this format. The caller keeps its own- default (offload) on ``None``.- """- fmt = _QUANT_TO_BENCH_FORMAT.get(quant_format, quant_format)- prof = _usable_profile(gpu_name, path, gpu_uuid)- if prof is None:- return None+ ``cpu_threads`` is the CPU executor's worker count: the per-step pool overhead makes the CPU bandwidth depend on it and on ``top_k``."""+ opts = ",".join(f"{k}={v}" for k, v in sorted(options.items()))+ fmt = f"{fmt}[{opts}]" if opts else fmt+ return f"{fmt}:{hidden}x{inter}:top{top_k}:t{cpu_threads}"- # Preferred: the per-dtype tuning verdicts (`ft bench bw --dtype`), a direct format->backend- # map -- the axis the backend pick is meant to key on.- dtypes = prof.get("dtypes")- if isinstance(dtypes, dict) and dtypes.get(fmt) in ("hybrid", "offload"):- return dtypes[fmt]- # Fallback: a per-model profile (`ft bench bw --model`). Aggregate the workloads sharing this- # format -- unanimous hybrid -> hybrid; any offload (a near-threshold split) -> offload.- workloads = prof.get("workloads")- if not isinstance(workloads, dict):- return None- picks = [- entry["recommended"]- for wl in workloads.values()- if isinstance(wl, dict)- for entry in [(wl.get("kernels") or {}).get(fmt)]- if isinstance(entry, dict) and entry.get("recommended")- ]- if not picks:+def fetch_fraction(row: dict | None) -> float | None:+ """The hybrid fetch split of a row: of a decode step's expert misses, the fraction to fetch over PCIe so the fetch and the CPU compute finish together.++ From the overlapped pair (both sides measured while running concurrently): pcie_ov / (pcie_ov + cpu_ov); without it, pcie / cpu. Clamped to 1."""+ if not isinstance(row, dict):return None- return "hybrid" if all(p == "hybrid" for p in picks) else "offload"---def load_hybrid_fetch_fraction(- quant_format: str,- gpu_name: str | None = None,- path: str | None = None,- gpu_uuid: str | None = None,-) -> float | None:- """Benched hybrid fetch fraction for ``quant_format``, or ``None``.-- The hybrid backend's bandwidth-matched fetch split: of a decode step's expert misses,- fetch this fraction over PCIe and compute the rest on the CPU, so both finish together.- Preferred source is the *overlapped* pair (CPU MoE and PCIe gather measured while- running concurrently -- the real contention regime): fetched/misses = pcie_ov /- (pcie_ov + cpu_ov). Older profiles without it fall back to the standalone bandwidths- under a full-DRAM-contention assumption (cpu keeps cpu - pcie under DMA), which- reduces to pcie/cpu. Per-dtype entry first, then any per-model entry with this format.- ``None`` = no usable profile; clamped to [0, 1].- """- fmt = _QUANT_TO_BENCH_FORMAT.get(quant_format, quant_format)+ cpu_ov, pcie_ov = row.get("cpu_moe_overlap_gbs"), row.get("pcie_gather_overlap_gbs")+ if cpu_ov and pcie_ov:+ return min(1.0, pcie_ov / (pcie_ov + cpu_ov))+ cpu, pcie = row.get("cpu_moe_gbs"), row.get("pcie_gather_gbs")+ if cpu and pcie:+ return min(1.0, pcie / cpu)+ return None+++def load_layer_kinds(gpu_name: str | None = None, path: str | None = None, gpu_uuid: str | None = None) -> dict | None:+ """The rows of the usable profile for this GPU (key -> row), or None without one."""prof = _usable_profile(gpu_name, path, gpu_uuid)if prof is None:return None- entries = [(prof.get("dtype_kernels") or {}).get(fmt)] + [- (wl.get("kernels") or {}).get(fmt)- for wl in (prof.get("workloads") or {}).values()- if isinstance(wl, dict)- ]- for entry in entries:- if not isinstance(entry, dict):- continue- cpu_ov, pcie_ov = entry.get("cpu_moe_overlap_gbs"), entry.get("pcie_gather_overlap_gbs")- if cpu_ov and pcie_ov:- return min(1.0, pcie_ov / (pcie_ov + cpu_ov))- cpu, pcie = entry.get("cpu_moe_gbs"), entry.get("pcie_gather_gbs")- if cpu and pcie:- return min(1.0, pcie / cpu)- return None+ rows = prof.get("layer_kinds")+ return rows if isinstance(rows, dict) else {}diff --git a/python/freetoken/moe/benchbw.py b/python/freetoken/moe/benchbw.pyindex f7ea5aa3..057b86fd 100644--- a/python/freetoken/moe/benchbw.py+++ b/python/freetoken/moe/benchbw.py@@ -11,23 +11,24 @@ Two layers of measurement:PCIe on a decode miss). These are what the hybrid-vs-offload choice rides on.When both work, the same pair is also measured *overlapped* (running concurrently,contending for host DRAM) -- those two achieved bandwidths set the hybrid backend's- per-step fetch split (``load_hybrid_fetch_fraction``).+ per-step fetch split (``bench_profile.fetch_fraction``).The rule, per (workload, format): recommend ``hybrid`` when the CPU MoE kernelbandwidth exceeds ``threshold`` x the PCIe gather bandwidth (default 2x), otherwise``offload``. Formats with no CPU MoE weight path (e.g. block-fp8) can't be computed onthe CPU, so they always resolve to ``offload``.-The hybrid-vs-offload choice is dtype-dominated, so the default is a **per-dtype tuning bench**-(``--dtype``): one bench per expert format against a canonical geometry -- the minimal set the-runtime backend pick matches on. ``--model`` additionally benches specific model geometries for-per-model detail. Results are written to a JSON file per GPU (default ``$XDG_CACHE_HOME/-freetoken/benchbw/<gpu-uuid>.json``) so the choice is reproducible and a multi-GPU box keeps-one profile per card.+Each bench writes one row per MoE layer kind: format, options, expert shape, top_k and CPU+worker threads (``bench_profile.layer_kind_key``). ``--model <checkpoint>`` benches every kind+a real checkpoint has, which is what the hybrid fetch split matches per layer; ``--dtype`` and+the ``--model`` presets bench canonical geometries. The rows live in a JSON file per GPU+(default ``$XDG_CACHE_HOME/freetoken/benchbw/<gpu-uuid>.json``); a run replaces the rows it+benched and keeps the others.ft bench bw # per-dtype tuning bench (default: all formats)ft bench bw --dtype nvfp4,bf16 # only these formatsft bench bw --model qwen3.6-moe # per-model detail instead+ ft bench bw --model /path/to/model.gguf # every MoE layer kind of a checkpointft bench bw --dtype all --model all # bothft bench bw --gpu GPU-2f3a... # bench a specific GPU (UUID or nvidia-smi index)"""@@ -38,9 +39,11 @@ import argparseimport contextlibimport jsonimport logging+import mathimport osimport socketimport statistics+import subprocessimport threadingimport timefrom dataclasses import dataclass@@ -64,7 +67,7 @@ logger = init_logger(__name__)# Formats the CPU MoE C++ kernel can compute AND this bench can build banks for; anything# else is offload-only here.-_CPU_MOE_FORMATS = frozenset({"bf16", "nvfp4", "mxfp4_triton", "ds_fp4", "fp8_block"})+_CPU_MOE_FORMATS = frozenset({"bf16", "nvfp4", "mxfp4_triton", "ds_fp4", "fp8_block", "gguf", "gguf_scaled"})# Formats this bench can build synthetic (correctly-sized) banks for._BUILDABLE_FORMATS = frozenset({"bf16", "nvfp4", "fp8_block", "mxfp4_triton", "ds_fp4"})# Friendlier CLI/display aliases for the internal quant_format strings.@@ -110,6 +113,13 @@ class Workload:swiglu_alpha: float = 1.702swiglu_limit: float | None = Noneact_block: int | None = None # ds_fp4 W4A8 activation quant block+ options: tuple[tuple[str, int], ...] = () # CPU executor options of the format, e.g. the GGUF types++ def format_options(self, fmt: str) -> dict[str, int]:+ opts = dict(self.options)+ if self.act_block is not None and fmt == "ds_fp4":+ opts.setdefault("act_block", self.act_block)+ return opts# Preset workloads. Dims from the model configs / benchmarks/bench_offload_cache_copy.py.@@ -132,13 +142,11 @@ WORKLOADS: dict[str, Workload] = {"minimax-m2.5": Workload("minimax-m2.5", 3072, 1536, 256, 8, ("nvfp4",)),}-# Per-dtype canonical geometries for the TUNING bench (`--dtype`). The hybrid-vs-offload choice is-# dtype-dominated -- the CPU-MoE-vs-PCIe-gather bandwidth ratio rides on the expert *format* and-# the hardware, not the exact model -- so measuring one representative geometry per format is-# enough for the runtime backend pick (the engine also matches by dtype). This is the minimal set-# the Desktop "detect" flow runs; `--model` still benches specific model geometries for per-model-# detail. Geometries mirror the model families that ship each format (mxfp4 keeps gpt-oss's large-# experts -- the reason that format sits near the 2x threshold).+# Per-dtype canonical geometries for the TUNING bench (`--dtype`): one representative geometry per+# expert format, the set the Desktop "detect" flow runs. A model's layers match these rows only when+# their kind is the same; `--model <checkpoint>` benches a model's own kinds. Geometries mirror the+# model families that ship each format (mxfp4 keeps gpt-oss's large experts -- the reason that+# format sits near the 2x threshold).DTYPE_WORKLOADS: dict[str, Workload] = {"bf16": Workload("dtype:bf16", 2048, 768, 128, 8, ("bf16",)),"nvfp4": Workload("dtype:nvfp4", 3072, 1536, 128, 8, ("nvfp4",)),@@ -285,9 +293,25 @@ def measure_pcie_bw(device: torch.device, nbytes: int = 256 << 20, iters: int =# ============================== bank geometry ==============================-def _offload_bank_specs(fmt: str, H: int, I: int) -> dict[str, tuple[int, torch.dtype]]:+def _gguf_bank_shapes(fmt: str, opts: dict, H: int, I: int) -> dict[str, tuple[tuple[int, ...], torch.dtype]]:+ """Per-expert (row shape, dtype) of the GGUF banks: ggml rows as bytes, float rows in their dtype."""+ from freetoken.layers.quantization.ggml_types import float_dtype, row_bytes++ def rows(ggml_type: int, n: int, k: int):+ dtype = float_dtype(ggml_type)+ return ((n, k), dtype) if dtype is not None else ((n, row_bytes(ggml_type, k)), torch.uint8)++ shapes = {"gate_up": rows(opts["gate_up_type"], 2 * I, H), "down": rows(opts["down_type"], H, I)}+ if fmt == "gguf_scaled":+ shapes["global"] = ((4,), torch.float32)+ return shapes+++def _offload_bank_specs(fmt: str, H: int, I: int, opts: dict | None = None) -> dict[str, tuple[int, torch.dtype]]:"""Per-bank (elems_per_expert, dtype) for the flat offload host banks (the gather)."""u8, bf16, f16, f8 = torch.uint8, torch.bfloat16, torch.float16, torch.float8_e4m3fn+ if fmt in ("gguf", "gguf_scaled"):+ return {name: (math.prod(shape), dtype) for name, (shape, dtype) in _gguf_bank_shapes(fmt, opts, H, I).items()}if fmt == "bf16":return {"gate_up": (2 * I * H, bf16), "down": (H * I, bf16)}if fmt == "fp8_block":@@ -315,8 +339,8 @@ def _offload_bank_specs(fmt: str, H: int, I: int) -> dict[str, tuple[int, torch.raise NotImplementedError(fmt)-def _expert_bytes(fmt: str, H: int, I: int) -> int:- return sum(elems * dt.itemsize for elems, dt in _offload_bank_specs(fmt, H, I).values())+def _expert_bytes(fmt: str, H: int, I: int, opts: dict | None = None) -> int:+ return sum(elems * dt.itemsize for elems, dt in _offload_bank_specs(fmt, H, I, opts).values())def _synth_experts(E: int, expert_bytes: int) -> int:@@ -327,7 +351,7 @@ def _synth_experts(E: int, expert_bytes: int) -> int:return min(E, max(1, _SYNTH_BANK_BUDGET // max(1, expert_bytes)))-def _cpu_moe_bank_sources(fmt: str, H: int, I: int, E: int) -> dict:+def _cpu_moe_bank_sources(fmt: str, H: int, I: int, E: int, opts: dict | None = None) -> dict:"""3D pinned banks in the exact layout ``CpuMoeExecutor`` expects for ``fmt``.Scale banks that decode through an e8m0 exponent (mxfp4/ds_fp4) are set to a unit@@ -338,6 +362,22 @@ def _cpu_moe_bank_sources(fmt: str, H: int, I: int, E: int) -> dict:def pin(*shape, dtype):return alloc_pinned_tensor(*shape, dtype=dtype)+ if fmt in ("gguf", "gguf_scaled"):+ from freetoken.third_party.gguf import GGML_QUANT_SIZES, GGMLQuantizationType++ b = {}+ for name, (shape, dtype) in _gguf_bank_shapes(fmt, opts, H, I).items():+ b[name] = pin(E, *shape, dtype=dtype)+ if name == "global":+ b[name].fill_(1.0)+ elif dtype == torch.uint8:+ b[name].zero_() # zero ggml blocks: every fp16 block scale is 0, never denormal+ if opts["gate_up_type" if name == "gate_up" else "down_type"] == GGMLQuantizationType.MXFP4:+ # each block starts with an e8m0 scale byte; 0 decodes to 2^-127, a denormal that slows the CPU kernel+ b[name].view(E, -1, GGML_QUANT_SIZES[GGMLQuantizationType.MXFP4][1])[..., 0] = 127+ else:+ b[name].fill_(0.02)+ return bif fmt == "bf16":gate_up, down = pin(E, 2 * I, H, dtype=torch.bfloat16), pin(E, H, I, dtype=torch.bfloat16)gate_up.fill_(0.02) # uninitialized bf16 can be denormal -> x86 FP slowdown@@ -408,9 +448,9 @@ def _build_gather_rig(fmt: str, wl: Workload, device: torch.device):bench and the overlapped CPU+PCIe bench."""from freetoken.moe.offload_cache import OffloadMoeCache- H, I = wl.hidden, wl.inter- E = _synth_experts(wl.experts, _expert_bytes(fmt, H, I))- specs = _offload_bank_specs(fmt, H, I)+ H, I, opts = wl.hidden, wl.inter, wl.format_options(fmt)+ E = _synth_experts(wl.experts, _expert_bytes(fmt, H, I, opts))+ specs = _offload_bank_specs(fmt, H, I, opts)cache = OffloadMoeCache(num_layers=1, num_experts=E, cache_size=E, device=device, quant_format=fmt)total_bytes = 0sources = {}@@ -437,7 +477,7 @@ def measure_pcie_gather_bw(fmt: str, wl: Workload, device: torch.device, iters:(all synth experts, scattered sources). Reusing the cache gives the kernel-correctint64 ``num_indices``. Timed with CUDA events."""- eb = _expert_bytes(fmt, wl.hidden, wl.inter)+ eb = _expert_bytes(fmt, wl.hidden, wl.inter, wl.format_options(fmt))cache, total_bytes = _build_gather_rig(fmt, wl, device)E = cache.num_experts@@ -476,7 +516,7 @@ def _build_cpu_moe_executor(fmt: str, wl: Workload, banks: dict, num_threads: incache, top_k=wl.top_k, activation=wl.activation,apply_router_weight_on_input=False, num_threads=num_threads, max_tokens=1,device=torch.device("cuda"), swiglu_alpha=wl.swiglu_alpha,- swiglu_limit=wl.swiglu_limit, layer_formats=[(fmt, {"act_block": wl.act_block} if wl.act_block is not None else {})] * cache.num_layers,+ swiglu_limit=wl.swiglu_limit, layer_formats=[(fmt, wl.format_options(fmt))] * cache.num_layers,)@@ -528,25 +568,25 @@ def measure_cpu_moe_bw(fmt: str, wl: Workload, iters: int = 64, num_threads: intthe SAME banks (built once) to populate the diagnostic ``isa_sweep`` breakdown; tiersabove the CPU's support clamp down and dedupe by the resulting isa name."""- H, I = wl.hidden, wl.inter- eb = _expert_bytes(fmt, H, I)+ H, I, opts = wl.hidden, wl.inter, wl.format_options(fmt)+ eb = _expert_bytes(fmt, H, I, opts)E = _synth_experts(wl.experts, eb)- banks = _cpu_moe_bank_sources(fmt, H, I, E)+ banks = _cpu_moe_bank_sources(fmt, H, I, E, opts)- def run(isa: str | None) -> tuple[str, float]:+ def run(isa: str | None) -> tuple[str, float, list[int]]:with _forced_isa(isa):ex = _build_cpu_moe_executor(fmt, wl, banks, num_threads, E)bw = round(_time_cpu_moe(ex, wl, eb, E, iters), 2)- label = ex.isa+ label, cores = ex.isa, list(ex.core_ids)del ex- return label, bw+ return label, bw, cores- auto_isa, auto_bw = run(None) # recommendation basis: the tier production would pick+ auto_isa, auto_bw, cores = run(None) # recommendation basis: the tier production would picksweep = {auto_isa: auto_bw}for isa in isas or []:- label, bw = run(isa)+ label, bw, _ = run(isa)sweep[label] = max(sweep.get(label, 0.0), bw) # dedupe clamped tiers, keep best- return {"bw_gbs": auto_bw, "isa": auto_isa, "isa_sweep": sweep,+ return {"bw_gbs": auto_bw, "isa": auto_isa, "isa_sweep": sweep, "cores": cores,"expert_bytes": eb, "synth_experts": E}@@ -567,10 +607,10 @@ def measure_overlap_bw(fmt: str, wl: Workload, device: torch.device,so the DMA is really in flight, not just enqueued). Each side reports bytes / its ownelapsed; the two windows differ by at most one CPU step + one gather."""- H, I = wl.hidden, wl.inter- eb = _expert_bytes(fmt, H, I)+ H, I, opts = wl.hidden, wl.inter, wl.format_options(fmt)+ eb = _expert_bytes(fmt, H, I, opts)E = _synth_experts(wl.experts, eb)- ex = _build_cpu_moe_executor(fmt, wl, _cpu_moe_bank_sources(fmt, H, I, E), num_threads, E)+ ex = _build_cpu_moe_executor(fmt, wl, _cpu_moe_bank_sources(fmt, H, I, E, opts), num_threads, E)set_ids, run_step = _cpu_moe_step_fns(ex, wl, E)cache, gather_bytes = _build_gather_rig(fmt, wl, device)@@ -631,17 +671,15 @@ def _bench_format(fmt: str, wl: Workload, device: torch.device, threshold: floatone format can't kill the run; an unexpected exception is left to propagate."""entry: dict = {- "expert_bytes": None, "synth_experts": None, "cpu_moe_gbs": None, "cpu_moe_isa": None,- "isa_sweep": None, "pcie_gather_gbs": None,+ "cpu_moe_gbs": None, "cpu_moe_isa": None,+ "cpu_cores": None, "isa_sweep": None, "pcie_gather_gbs": None,"cpu_moe_overlap_gbs": None, "pcie_gather_overlap_gbs": None,- "ratio": None, "recommended": None,+ "recommended": None,"note": None,}try:g = measure_pcie_gather_bw(fmt, wl, device, pcie_iters)entry["pcie_gather_gbs"] = round(g["bw_gbs"], 2)- entry["expert_bytes"] = g["expert_bytes"]- entry["synth_experts"] = g["synth_experts"]except (ImportError, RuntimeError) as e:logger.warning(f"benchbw: PCIe gather bench failed for {wl.name}/{fmt}: {e}")_note(entry, f"pcie gather unavailable ({e})")@@ -655,6 +693,7 @@ def _bench_format(fmt: str, wl: Workload, device: torch.device, threshold: floatc = measure_cpu_moe_bw(fmt, wl, cpu_iters, cpu_threads, isas=isas)entry["cpu_moe_gbs"] = c["bw_gbs"] # auto/best tier (what production runs)entry["cpu_moe_isa"] = c["isa"]+ entry["cpu_cores"] = c["cores"]if isas: # a sweep was requested -> keep the per-tier breakdownentry["isa_sweep"] = c["isa_sweep"]# The tier label comes from the bf16 dot chooser, so it can over-report tiers@@ -665,18 +704,18 @@ def _bench_format(fmt: str, wl: Workload, device: torch.device, threshold: floatelif fmt in ("mxfp4_triton", "ds_fp4", "fp8_block"):_note(entry, "isa tier labels are nominal: avx512bf16 == avx512f here ""(3 real kernels: scalar/avx2/avx512, no bf16/VNNI variant)")- entry["expert_bytes"] = entry["expert_bytes"] or c["expert_bytes"]- entry["synth_experts"] = entry["synth_experts"] or c["synth_experts"]+ elif fmt in ("gguf", "gguf_scaled"):+ _note(entry, "isa tier labels are nominal: GGUF rows run on the ggml-cpu variant "+ "picked at load, regardless of the forced tier")except (ImportError, RuntimeError) as e:logger.warning(f"benchbw: CPU MoE bench failed for {wl.name}/{fmt}: {e}")_note(entry, f"cpu moe unavailable ({e})")cpu_g, pcie_g = entry["cpu_moe_gbs"], entry["pcie_gather_gbs"]if cpu_g is not None and pcie_g: # pcie_g truthy also rules out a div-by-zero- entry["ratio"] = round(cpu_g / pcie_g, 3)entry["recommended"] = recommend(cpu_g, pcie_g, threshold)# Both sides work standalone -> also measure them contended (concurrently). This- # pair sets the hybrid backend's fetch split (load_hybrid_fetch_fraction); the+ # pair sets the hybrid backend's fetch split (bench_profile.fetch_fraction); the# hybrid-vs-offload verdict above stays on the standalone numbers.try:o = measure_overlap_bw(fmt, wl, device, cpu_threads)@@ -694,6 +733,89 @@ def _bench_format(fmt: str, wl: Workload, device: torch.device, threshold: floatreturn entry+def _kind(fmt: str, wl: Workload) -> tuple:+ """A layer kind without the CPU thread count, which the row key adds once the executor has sized its pool."""+ return (fmt, tuple(sorted(wl.format_options(fmt).items())), wl.hidden, wl.inter, wl.top_k)+++def checkpoint_layer_kinds(model_path: str, device: torch.device) -> dict[tuple, Workload]:+ """Each distinct MoE layer kind of a checkpoint as hybrid decode would serve it: kind -> workload."""+ from freetoken.distributed import DistributedInfo, set_tp_info, try_get_tp_info+ from freetoken.engine.config import EngineConfig+ from freetoken.engine.engine import meta_offload_layer_methods+ from freetoken.layers import set_rope_device++ if try_get_tp_info() is None:+ set_tp_info(rank=0, size=1)+ tp = try_get_tp_info()+ cfg = EngineConfig(model_path=model_path, tp_info=DistributedInfo(tp.rank, tp.size), dtype=torch.bfloat16, moe_strategy="hybrid")+ if not getattr(cfg.model_config, "is_moe", False):+ raise RuntimeError(f"{model_path} has no MoE layers")+ set_rope_device(device)+ methods = meta_offload_layer_methods(cfg, "hybrid")+ if methods is None:+ raise RuntimeError(f"{model_path}: its MoE layers bind no expert method")+ kinds: dict[tuple, Workload] = {}+ for m in methods:+ fmt, opts = m.cpu_spec()+ c = m.cfg+ wl = Workload(+ f"{os.path.basename(model_path.rstrip('/'))}:{fmt}", c.hidden, c.local_intermediate, c.num_experts, c.top_k, (fmt,),+ activation=c.activation, swiglu_alpha=float(c.alpha), swiglu_limit=c.limit, options=tuple(sorted(opts.items())),+ )+ kinds.setdefault(_kind(fmt, wl), wl)+ return kinds+++def _freetoken_build() -> dict:+ """FreeToken version and, in a git checkout, its commit: rows benched on older kernels may be stale."""+ from freetoken.version import __version__++ try:+ commit = subprocess.run(+ ["git", "-C", os.path.dirname(os.path.abspath(__file__)), "rev-parse", "--short=12", "HEAD"],+ capture_output=True, text=True, timeout=5, check=True,+ ).stdout.strip() or None+ except (OSError, subprocess.SubprocessError):+ commit = None+ return {"version": __version__, "commit": commit}+++def _cpu_info() -> dict:+ model = numa = None+ try:+ with open("/proc/cpuinfo") as f:+ model = next((line.split(":", 1)[1].strip() for line in f if line.startswith("model name")), None)+ except OSError:+ pass+ try:+ numa = sum(1 for n in os.listdir("/sys/devices/system/node") if n.startswith("node") and n[4:].isdigit()) or None+ except OSError:+ pass+ return {"model": model, "physical_cores": len(physical_core_cpus()), "numa_nodes": numa}+++def _merge_previous(result: dict, path: str) -> None:+ """Keep the rows of the profile already at ``path`` (same GPU and version) that this run did not re-bench; a re-benched row keeps its other sources."""+ try:+ with open(path) as f:+ prev = json.load(f)+ except (OSError, ValueError):+ return+ if not isinstance(prev, dict) or prev.get("version") != result["version"] or (prev.get("gpu") or {}).get("uuid") != result["gpu"]["uuid"]:+ return+ prev_rows = prev.get("layer_kinds")+ if not isinstance(prev_rows, dict):+ return+ rows = dict(prev_rows)+ for key, row in result["layer_kinds"].items():+ old = prev_rows.get(key)+ if isinstance(old, dict) and isinstance(old.get("sources"), list):+ row["sources"] = sorted(set(old["sources"]) | set(row["sources"]))+ rows[key] = row+ result["layer_kinds"] = rows++def _atomic_write_json(path: str, obj: dict) -> None:os.makedirs(os.path.dirname(os.path.abspath(path)) or ".", exist_ok=True)tmp = f"{path}.tmp.{os.getpid()}"@@ -717,7 +839,11 @@ def run_benchbw(pcie_iters: int = 30,kernel_cpu_iters: int = 64,kernel_pcie_iters: int = 20,+ checkpoints: tuple[str, ...] = (),) -> dict:+ from freetoken.moe.bench_profile import layer_kind_key+ from freetoken.moe.cpu_executor import _format_label+if not torch.cuda.is_available():raise RuntimeError("benchbw needs a CUDA device to measure PCIe bandwidth (both offload and "@@ -735,12 +861,29 @@ def run_benchbw(torch.cuda.set_device(device)gpu = gpu_identity(device_index)+ # kind -> (workload, sources); checkpoints are read before any timing so a bad one fails fast+ units: dict[tuple, tuple[Workload, list[str]]] = {}++ def add(fmt: str, wl: Workload, source: str) -> None:+ sources = units.setdefault(_kind(fmt, wl), (wl, []))[1]+ if source not in sources:+ sources.append(source)++ for fmt in dtypes or ():+ add(fmt, DTYPE_WORKLOADS[fmt], f"dtype:{_FORMAT_DISPLAY.get(fmt, fmt)}")+ for mname in models:+ for fmt in formats or WORKLOADS[mname].formats:+ add(fmt, WORKLOADS[mname], f"preset:{mname}")+ for path in checkpoints:+ logger.info(f"benchbw: reading the MoE layer kinds of {path} ...")+ for kind, wl in checkpoint_layer_kinds(path, device).items():+ add(kind[0], wl, path)+# Machine-parseable progress on stdout (opt-in), so the daemon/Desktop can stream feedback# while the bench runs -- mirrors ft checkpoint's FTCONVERT lines. `done`/`total` count the- # measured units (ceilings + one per benched format); `label` is what's measuring now.+ # measured units (ceilings + one per layer kind); `label` is what's measuring now.prog_on = os.environ.get("FREETOKEN_BENCH_PROGRESS") == "1"- n_model_units = sum(len(formats or WORKLOADS[m].formats) for m in models)- total = 1 + len(dtypes or ()) + n_model_units+ total = 1 + len(units)done = 0def _prog(label: str) -> None:@@ -758,36 +901,24 @@ def run_benchbw(exec_log = logging.getLogger("freetoken.moe.cpu_executor")prev_level = exec_log.levelexec_log.setLevel(logging.WARNING)- dtypes_out: dict = {}- workloads_out: dict = {}+ now = datetime.now(timezone.utc).astimezone().isoformat(timespec="seconds")+ build = _freetoken_build()+ rows: dict = {}try:- # dtype tuning: one bench per expert format (canonical geometry). This is what the- # runtime backend pick keys off -- matched by dtype in bench_profile.py.- for fmt in dtypes or ():- wl = DTYPE_WORKLOADS[fmt]- _prog(f"dtype:{_FORMAT_DISPLAY.get(fmt, fmt)}")- logger.info(f"benchbw: dtype {_FORMAT_DISPLAY.get(fmt, fmt)} real kernels ...")- dtypes_out[fmt] = _bench_format(- fmt, wl, device, threshold, cpu_threads, kernel_cpu_iters, kernel_pcie_iters, isas- )+ for (fmt, opts, hidden, inter, top_k), (wl, sources) in units.items():+ label = f"{_format_label(fmt, opts)} {hidden}x{inter} top{top_k}"+ _prog(label)+ logger.info(f"benchbw: {label} real kernels ...")+ entry = _bench_format(fmt, wl, device, threshold, cpu_threads, kernel_cpu_iters, kernel_pcie_iters, isas)done += 1- # optional per-model detail (--model): specific geometries, unchanged.- for mname in models:- wl = WORKLOADS[mname]- fmts = formats or wl.formats- kernels = {}- for fmt in fmts:- _prog(f"{mname}:{_FORMAT_DISPLAY.get(fmt, fmt)}")- logger.info(f"benchbw: {mname}/{fmt} real kernels ...")- kernels[fmt] = _bench_format(- fmt, wl, device, threshold, cpu_threads, kernel_cpu_iters, kernel_pcie_iters, isas- )- done += 1- workloads_out[mname] = {- "model": {"name": wl.name, "hidden": wl.hidden, "inter": wl.inter,- "experts": wl.experts, "top_k": wl.top_k},- "kernels": kernels,- "recommended_moe_backend": {f: kernels[f]["recommended"] for f in fmts},+ if not entry["cpu_cores"]:+ logger.warning(f"benchbw: no CPU MoE result for {label}, so no row: {entry['note']}")+ continue+ threads = len(entry["cpu_cores"])+ rows[layer_kind_key(fmt, dict(opts), hidden, inter, top_k, threads)] = {+ "format": fmt, "options": dict(opts), "hidden": hidden, "inter": inter, "top_k": top_k,+ "cpu_threads": threads, "activation": wl.activation, "sources": sources,+ "threshold": threshold, "measured_at": now, "freetoken": build, **entry,}finally:exec_log.setLevel(prev_level)@@ -795,27 +926,27 @@ def run_benchbw(result = {"version": PROFILE_VERSION,- "timestamp": datetime.now(timezone.utc).astimezone().isoformat(timespec="seconds"),+ "timestamp": now,"epoch": int(time.time()),"host": socket.gethostname(),# index is the CUDA ordinal the bench ran on; uuid keys the profile file"gpu": {"index": device_index, "name": gpu["name"], "uuid": gpu["uuid"]},- "cpu": {"physical_cores": len(physical_core_cpus()), "threads_used": cpu["threads"]},- "threshold": threshold,+ "cpu": _cpu_info(),"ceilings": {"cpu_stream_read_gbs": round(cpu["bw_gbs"], 2),+ "cpu_stream_threads": cpu["threads"],"pcie_linear_h2d_gbs": round(pcie["h2d_gbs"], 2),"pcie_linear_d2h_gbs": round(pcie["d2h_gbs"], 2),},- # Per-dtype verdicts (the tuning axis) + optional per-model detail.- "dtypes": {f: e["recommended"] for f, e in dtypes_out.items()},- "dtype_kernels": dtypes_out,- "workloads": workloads_out,+ "layer_kinds": rows,}out_path = os.path.expanduser(out_path or default_out_path(gpu["uuid"]))+ benched = list(rows)+ _merge_previous(result, out_path)_atomic_write_json(out_path, result)result["out_path"] = out_path+ result["benched"] = benchedif prog_on:print(f"FTBENCH_OUT {out_path}", flush=True)return result@@ -833,8 +964,9 @@ def _print_kernels(kernels: dict, iw: int) -> None:f"{'CPU/PCIe':>9} backend")for fmt, e in kernels.items():disp = _FORMAT_DISPLAY.get(fmt, fmt)- eb = f"{e['expert_bytes'] / 2**20:.2f} MB" if e["expert_bytes"] else "n/a"- ratio = f"{e['ratio']:.2f}x" if e["ratio"] is not None else "—"+ eb = f"{_expert_bytes(fmt, e['hidden'], e['inter'], e['options']) / 2**20:.2f} MB"+ cpu_g, pcie_g = e["cpu_moe_gbs"], e["pcie_gather_gbs"]+ ratio = f"{cpu_g / pcie_g:.2f}x" if cpu_g is not None and pcie_g else "—"print(f" {disp:<8} {eb:>9} {_gbs(e['cpu_moe_gbs'])} {_gbs(e['pcie_gather_gbs'])} "f"{ratio:>9} {e['recommended']}")c_ov, p_ov = e.get("cpu_moe_overlap_gbs"), e.get("pcie_gather_overlap_gbs")@@ -851,27 +983,24 @@ def _print_kernels(kernels: dict, iw: int) -> None:def _print_report(r: dict) -> None:- c = r["ceilings"]+ from freetoken.moe.cpu_executor import _format_label++ c, cpu, rows = r["ceilings"], r["cpu"], r["layer_kinds"]print()print(f" host {r['host']} gpu cuda:{r['gpu']['index']} ({r['gpu']['name']}) "- f"cpu {r['cpu']['physical_cores']}c/{r['cpu']['threads_used']}t")+ f"cpu {cpu['model']} ({cpu['physical_cores']}c, {cpu['numa_nodes']} NUMA)")print(f" ceilings: CPU STREAM read {c['cpu_stream_read_gbs']:.1f} | "- f"PCIe linear H2D {c['pcie_linear_h2d_gbs']:.1f} D2H {c['pcie_linear_d2h_gbs']:.1f} GB/s"- f" (threshold {r['threshold']}x)")- # one global ISA-name width so every sweep's GB/s lands in the same column; the- # redundant "(<fmt>)" tag is dropped -- the format is already in the row header.- all_kernels = list((r.get("dtype_kernels") or {}).values()) + [- e for wl in r["workloads"].values() for e in wl["kernels"].values()- ]- iw = max((len(k.split("(")[0]) for e in all_kernels for k in (e.get("isa_sweep") or {})), default=0)- if r.get("dtype_kernels"):- print("\n per-dtype (tuning — what the runtime backend pick matches on)")- _print_kernels(r["dtype_kernels"], iw)- for name, w in r["workloads"].items():- m = w["model"]- print(f"\n {name} H={m['hidden']} I={m['inter']} E={m['experts']} top_k={m['top_k']}")- _print_kernels(w["kernels"], iw)- print(f"\n saved: {r['out_path']}\n")+ f"PCIe linear H2D {c['pcie_linear_h2d_gbs']:.1f} D2H {c['pcie_linear_d2h_gbs']:.1f} GB/s")+ benched = [rows[key] for key in r["benched"]]+ # one global ISA-name width so every sweep's GB/s lands in the same column+ iw = max((len(k.split("(")[0]) for e in benched for k in (e.get("isa_sweep") or {})), default=0)+ for e in benched:+ label = _format_label(e["format"], tuple(sorted(e["options"].items())))+ sources = ", ".join(os.path.basename(s.rstrip("/")) for s in e["sources"])+ print(f"\n {label} H={e['hidden']} I={e['inter']} top_k={e['top_k']} threads={e['cpu_threads']} "+ f" ({sources}; threshold {e['threshold']}x)")+ _print_kernels({e["format"]: e}, iw)+ print(f"\n saved: {r['out_path']} ({len(benched)} rows benched, {len(rows)} in the profile)\n")# ================================ CLI plumbing ================================@@ -899,13 +1028,14 @@ def _positive_float(s: str) -> float:def _model_list(s: str) -> tuple[str, ...]:+ """Preset names and checkpoint paths; a path is any item that exists on disk."""if s.strip() == "all":return tuple(WORKLOADS)items = [x.strip() for x in s.split(",") if x.strip()]- bad = [x for x in items if x not in WORKLOADS]+ bad = [x for x in items if x not in WORKLOADS and not os.path.exists(x)]if bad or not items:raise argparse.ArgumentTypeError(- f"'all' or a comma-separated subset of {sorted(WORKLOADS)}, got {s!r}"+ f"'all', or a comma-separated list of checkpoint paths and presets from {sorted(WORKLOADS)}, got {s!r}")return tuple(dict.fromkeys(items))@@ -951,11 +1081,12 @@ def main(argv: list[str] | None = None, prog: str = "ft bench bw") -> int:formatter_class=argparse.RawDescriptionHelpFormatter)p.add_argument("--dtype", type=_dtype_list, default=None,help=f"tuning bench: 'all' (default when no --model), or comma-separated from "- f"{sorted(DTYPE_WORKLOADS)} -- one bench per expert format (what the runtime "- f"backend pick matches on)")+ f"{sorted(DTYPE_WORKLOADS)} -- one bench per expert format at a canonical geometry")p.add_argument("--model", type=_model_list, default=None,help=f"per-model detail instead of/in addition to --dtype: 'all', or comma-"- f"separated from {sorted(WORKLOADS)}")+ f"separated presets from {sorted(WORKLOADS)} and checkpoint paths (a "+ f"checkpoint benches each of its MoE layer kinds, which the hybrid fetch "+ f"split matches per layer)")p.add_argument("--formats", type=_format_list, default=None,help="override a --model's native expert formats (comma-separated)")p.add_argument("--isa", type=_isa_list, default=None, dest="isas",@@ -986,11 +1117,11 @@ def main(argv: list[str] | None = None, prog: str = "ft bench bw") -> int:except (ValueError, RuntimeError) as e:p.error(str(e))- models = ns.model or ()+ models = tuple(m for m in ns.model or () if m in WORKLOADS)+ checkpoints = tuple(m for m in ns.model or () if m not in WORKLOADS)dtypes = ns.dtype- # Default (no selection): the full per-dtype tuning bench -- the minimal set the runtime- # backend pick needs. --model alone skips it; --dtype + --model runs both.- if not models and not dtypes:+ # Default (no selection): the full per-dtype tuning bench. --model alone skips it; --dtype + --model runs both.+ if not ns.model and not dtypes:dtypes = tuple(DTYPE_WORKLOADS)try:@@ -999,6 +1130,7 @@ def main(argv: list[str] | None = None, prog: str = "ft bench bw") -> int:dtypes=dtypes, formats=ns.formats, isas=ns.isas, cpu_threads=ns.cpu_threads,cpu_iters=ns.cpu_iters, pcie_bytes=ns.pcie_mib << 20, pcie_iters=ns.pcie_iters,kernel_cpu_iters=ns.kernel_cpu_iters, kernel_pcie_iters=ns.kernel_pcie_iters,+ checkpoints=checkpoints,)except (RuntimeError, OSError) as e:print(f"error: {e}")diff --git a/python/freetoken/moe/offload_cache.py b/python/freetoken/moe/offload_cache.pyindex 5291e621..6b428c41 100644--- a/python/freetoken/moe/offload_cache.py+++ b/python/freetoken/moe/offload_cache.py@@ -152,12 +152,12 @@ class OffloadMoeCache:# of that step's misses are computed on the CPU. 0 -> never fetch (CPU does every# miss, the GPU cache stays cold); large -> behaves like pure offload.hybrid_max_fetch: int = 1- # hybrid only: when > 0, replaces the fixed cap with a per-step fraction -- fetch+ # hybrid only, one per MoE layer: when > 0, replaces the fixed cap with a per-step fraction -- fetch# ~fraction * misses experts over PCIe (rounded to whichever integer balances the# overlap best), the CPU computes the rest. The engine sets it to the benched# pcie_bw / cpu_bw ratio so the PCIe fetch and the CPU overflow GEMV take equal# time (perfect overlap): fetched : cpu = pcie : cpu - pcie.- hybrid_fetch_fraction: float = 0.0+ hybrid_fetch_fraction: list[float] | None = None# bank layout from the expert kernel (a BankSpec per role); when given it replaces the _BANK_SCHEMAS lookup and the slot cap comes from max_slotslayout: dict | None = Nonemax_slots: int | None = None@@ -1095,9 +1095,8 @@ class OffloadMoeCache:self.decode_freq[layer_id].scatter_add_(0, ids, torch.ones_like(ids))self._pending_src_layer = layer_idself._pending_whole_layer = False- return ensure_experts_hybrid(- self, layer_id, expert_ids, weights, self.hybrid_max_fetch, self.hybrid_fetch_fraction, clamp- )+ fraction = self.hybrid_fetch_fraction[layer_id] if self.hybrid_fetch_fraction is not None else 0.0+ return ensure_experts_hybrid(self, layer_id, expert_ids, weights, self.hybrid_max_fetch, fraction, clamp)def materialize_layer(self, layer_id: int) -> None:from freetoken.moe.offload_kernels import materialize_layerdiff --git a/python/freetoken/server/args.py b/python/freetoken/server/args.pyindex be244a11..56d52ee7 100644--- a/python/freetoken/server/args.py+++ b/python/freetoken/server/args.py@@ -644,9 +644,8 @@ def parse_args(default=ServerArgs.moe_strategy,choices=["auto", *MOE_STRATEGIES],help=(- "How the routed experts are served. 'auto' resolves a MoE model to the offload family "- "(offload, or hybrid when a `ft bench bw` profile recommends it), and to resident "- "'fused' experts on unified-memory GPUs (GB10 / DGX Spark)."+ "How the routed experts are served. 'auto' resolves a MoE model to 'offload', and to "+ "resident 'fused' experts on unified-memory GPUs (GB10 / DGX Spark)."),)@@ -813,7 +812,8 @@ def parse_args("For --moe-strategy hybrid: max experts fetched over PCIe per (layer, decode ""step); the rest of that step's misses are computed on the CPU, overlapped. ""-1 (default) = auto: fetch the benched pcie/cpu bandwidth fraction of each "- "step's misses (perfect overlap; needs an `ft bench bw` profile, else 1). "+ "step's misses (perfect overlap; needs the `ft bench bw` row of the layer's kind, "+ "from `ft bench bw --model <checkpoint>`, else 1). ""0 = never fetch (all misses on CPU); large = behaves like plain offload."),)diff --git a/tests/engine/test_cache_budget.py b/tests/engine/test_cache_budget.pyindex 7b37af9e..00b31482 100644--- a/tests/engine/test_cache_budget.py+++ b/tests/engine/test_cache_budget.py@@ -409,9 +409,7 @@ def test_adjust_config_defaults_moe_cache_auto_for_auto_resolved_offload_backendconfig = _offload_engine_config()_adjust_config(config)- # Which member of the family gets picked is not this test's claim, and is not ours to- # decide: a bare "auto" consults ~/.cache/freetoken/benchbw.json, so a box that has run- # `ft bench bw` resolves nvfp4 experts to hybrid instead. Assert the family, not the member.+ # Which member of the family gets picked is not this test's claim. Assert the family, not the member.assert is_offload_moe_strategy(config.moe_strategy)assert config.moe_cache_auto is Trueassert config.moe_cache_size == 0 # still unresolved -- the scheduler sizes it from VRAMdiff --git a/tests/moe/test_hybrid_fetch.py b/tests/moe/test_hybrid_fetch.pyindex fad4fa78..5f36ac7b 100644--- a/tests/moe/test_hybrid_fetch.py+++ b/tests/moe/test_hybrid_fetch.py@@ -1,7 +1,7 @@"""Hybrid decode's bandwidth-matched fetch split.-Covers the profile reader of --moe-hybrid-max-fetch auto, which turns `ft bench bw`-kernel bandwidths into a fetch fraction.+Covers the profile reader of --moe-hybrid-max-fetch auto, which turns the `ft bench bw`+row of each MoE layer kind into a fetch fraction."""import json@@ -9,37 +9,36 @@ import osimport pytest-from freetoken.moe.bench_profile import PROFILE_VERSION, default_profile_path, load_backend_recommendation, load_hybrid_fetch_fraction+from freetoken.moe.bench_profile import PROFILE_VERSION, default_profile_path, fetch_fraction, layer_kind_key, load_layer_kinds-def test_load_hybrid_fetch_fraction(tmp_path):+def test_layer_kind_rows_give_fetch_fractions(tmp_path):+ bf16 = layer_kind_key("bf16", {}, 2048, 768, 8, 12)+ gguf = layer_kind_key("gguf", {"gate_up_type": 12, "down_type": 14}, 2048, 768, 8, 12)+ assert gguf == "gguf[down_type=14,gate_up_type=12]:2048x768:top8:t12"prof = {"version": PROFILE_VERSION,"gpu": {"name": "FAKE GPU"},- "dtype_kernels": {- "bf16": {"cpu_moe_gbs": 100.0, "pcie_gather_gbs": 40.0},+ "layer_kinds": {+ bf16: {"cpu_moe_gbs": 100.0, "pcie_gather_gbs": 40.0},# overlapped (contended) pair wins over the standalone numbers when present- "nvfp4_x": {"cpu_moe_gbs": 100.0, "pcie_gather_gbs": 40.0,- "cpu_moe_overlap_gbs": 90.0, "pcie_gather_overlap_gbs": 30.0},- },- "workloads": {- "m": {"kernels": {"ds_fp4": {"cpu_moe_gbs": 80.0, "pcie_gather_gbs": 50.0}}}+ gguf: {"cpu_moe_gbs": 100.0, "pcie_gather_gbs": 40.0, "cpu_moe_overlap_gbs": 90.0, "pcie_gather_overlap_gbs": 30.0},},}path = tmp_path / "benchbw.json"path.write_text(json.dumps(prof))+ rows = load_layer_kinds(path=str(path))# standalone fallback: full-contention assumption -> pcie / cpu- assert load_hybrid_fetch_fraction("bf16", path=str(path)) == pytest.approx(0.4)+ assert fetch_fraction(rows[bf16]) == pytest.approx(0.4)# overlapped pair preferred: pcie_ov / (pcie_ov + cpu_ov)- assert load_hybrid_fetch_fraction("nvfp4_x", path=str(path)) == pytest.approx(0.25)- # per-model fallback when there is no per-dtype entry for the format- assert load_hybrid_fetch_fraction("ds_fp4", path=str(path)) == pytest.approx(0.625)- assert load_hybrid_fetch_fraction("nvfp4", path=str(path)) is None+ assert fetch_fraction(rows[gguf]) == pytest.approx(0.25)+ # another thread count is another kind+ assert layer_kind_key("bf16", {}, 2048, 768, 8, 6) not in rows# a profile from different hardware is ignored- assert load_hybrid_fetch_fraction("bf16", gpu_name="OTHER", path=str(path)) is None- # so is one written before the current set of CPU formats+ assert load_layer_kinds(gpu_name="OTHER", path=str(path)) is None+ # so is one of another versionpath.write_text(json.dumps({**prof, "version": PROFILE_VERSION - 1}))- assert load_hybrid_fetch_fraction("bf16", path=str(path)) is None+ assert load_layer_kinds(path=str(path)) is Nonedef test_profile_lookup_prefers_the_gpu_uuid_file(tmp_path, monkeypatch):@@ -47,15 +46,15 @@ def test_profile_lookup_prefers_the_gpu_uuid_file(tmp_path, monkeypatch):monkeypatch.delenv("FREETOKEN_BENCHBW_PATH", raising=False)uuid = "GPU-2f3a9b1c-0000-1111-2222-333344445555"- def write(path, name, verdict):+ def write(path, name, row):os.makedirs(os.path.dirname(path), exist_ok=True)with open(path, "w") as f:- json.dump({"version": PROFILE_VERSION, "gpu": {"name": name}, "dtypes": {"bf16": verdict}}, f)+ json.dump({"version": PROFILE_VERSION, "gpu": {"name": name}, "layer_kinds": {"k": row}}, f)# legacy single file only: used when the name matches, ignored otherwise- write(default_profile_path(), "FAKE GPU", "hybrid")- assert load_backend_recommendation("bf16", gpu_name="FAKE GPU", gpu_uuid=uuid) == "hybrid"- assert load_backend_recommendation("bf16", gpu_name="OTHER", gpu_uuid=uuid) is None+ write(default_profile_path(), "FAKE GPU", "legacy")+ assert load_layer_kinds(gpu_name="FAKE GPU", gpu_uuid=uuid) == {"k": "legacy"}+ assert load_layer_kinds(gpu_name="OTHER", gpu_uuid=uuid) is None# this card's own file wins over the legacy one- write(default_profile_path(uuid), "FAKE GPU", "offload")- assert load_backend_recommendation("bf16", gpu_name="FAKE GPU", gpu_uuid=uuid) == "offload"+ write(default_profile_path(uuid), "FAKE GPU", "card")+ assert load_layer_kinds(gpu_name="FAKE GPU", gpu_uuid=uuid) == {"k": "card"}diff --git a/tests/moe/test_offload.py b/tests/moe/test_offload.pyindex a5a7adb9..5d7d0ab8 100644--- a/tests/moe/test_offload.py+++ b/tests/moe/test_offload.py@@ -451,7 +451,7 @@ def test_adjust_config_converts_moe_cache_rate_to_cache_size(monkeypatch):from freetoken.moe import is_offload_moe_strategyassert config.moe_cache_size == 24- # Family, not member: a box with a benchbw profile resolves bf16 experts to hybrid.+ # Family, not member: which offload-family backend auto picks is not this test's claim.assert is_offload_moe_strategy(config.moe_strategy)