FreeToken brings datacenter-scale model serving to your desktop. Run massive models locally, fast and efficiently.
feat(moe): fetch locked-layer experts through pinned staging (#742)
Xiaoze Fanc5b5b7243a74
diff --git a/docs/cli.md b/docs/cli.mdindex e109bde0..34a9ffaa 100644--- a/docs/cli.md+++ b/docs/cli.md@@ -89,8 +89,11 @@ See [models.md](models.md#moe-strategies) for what each strategy does.| `--moe-cache-split` | equal | How the pools share the slots: `equal` (in proportion to their layer counts), or `profile:<path>`: every pool keeps `top_k` slots and the rest go to the most used experts in the profile's per-layer expert counts. With `offload`, a pool smaller than the expert count lowers `--max-running-requests` to its slots / top_k || `--kv-reserve-tokens` | 8192 | KV token floor reserved before `--moe-cache-auto` fills experts || `--moe-cpu-threads` | physical cores | CPU worker threads for the cpu/hybrid executor |-| `--moe-cpu-layers` | all on GPU | With `offload`: which MoE layers decode on CPU (`3,7,11`, a count, a fraction, or `auto`). `auto` is for Windows/WSL only, where CUDA pinned memory is capped; every value needs an expert format the CPU executor serves (bf16, fp8_block, nvfp4, mxfp4, GGUF on x86 CPUs) |+| `--moe-cpu-layers` | all on GPU | With `offload`/`hybrid`: which MoE layers decode on CPU (`3,7,11`, a count, a fraction, or `auto`). `auto` is for hosts that cap CUDA pinned memory (Windows/WSL): it pins layers from the middle out until CUDA refuses one and OS-locks the rest (`FREETOKEN_PIN_BUDGET_GB` and FTW checkpoints lock from a size estimate instead), see [models.md](models.md#hosts-that-cap-pinned-memory-wsl); every value needs an expert format the CPU executor serves (bf16, fp8_block, nvfp4, mxfp4, GGUF on x86 CPUs) || `--moe-hybrid-max-fetch` | auto | With `hybrid`: max experts fetched over PCIe per layer per step; rest computed on CPU |+| `--moe-locked-fetch` | auto | With `hybrid` and locked layers: share of a locked layer's misses fetched per step into its GPU slots through pinned staging. Auto takes p / (2p + c) from the `ft bench bw` row of the layer's kind (at most 1 per step without a row); 0 = locked layers decode on CPU only |+| `--moe-locked-max-fetch` | 8 | Most experts a locked layer fetches per step; sizes the staging (that many rows of the widest expert, per bank) |+| `--moe-collect-stats` | off | Log decode MoE stats with each decode log line: hit rate, and per layer-step the misses, experts fetched and experts computed on CPU, for pinned, locked and CPU-only layers || `--moe-prefill-hit-d2d` | off | Prefill: copy cache-hit experts device-side, stream only misses (CUDA >= 13) || `--disable-moe-prefill-overlap` | overlap on | Disable the two-buffer prefill copy overlap |diff --git a/docs/models.md b/docs/models.mdindex f6d99af9..a9b66a24 100644--- a/docs/models.md+++ b/docs/models.md@@ -62,6 +62,28 @@ have no resident path and stay on `offload`. Offload-only flags`--moe-prefill-hit-d2d`, `--disable-moe-prefill-overlap`) are ignored with awarning; pass `--moe-strategy offload` to use them.+### Hosts that cap pinned memory (WSL)++WSL caps CUDA pinned memory near half of RAM, so the expert banks of a large+model may not all pin. With `--moe-cpu-layers auto`, FreeToken loads the banks+first, then pins them layer by layer from the middle out until CUDA refuses one;+the remaining head and tail layers are OS-locked instead (resident, but with no+device address). After a refusal under WDDM (WSL and native Windows) it+unpins about 3 GiB more (`FREETOKEN_PIN_PROBE_MARGIN_GB`).+The probe runs only where pinning is known to be capped (WDDM, or+`FREETOKEN_SIMULATE_PIN_CAP_GB` for testing). FTW checkpoints, dummy weights+and models the CPU executor cannot serve take the size estimate instead, as+does `FREETOKEN_PIN_BUDGET_GB`, which fixes the budget: the locked layers are+then picked from a size estimate before the load, and a refused pin stops the+boot.++Under `hybrid`, a locked layer keeps GPU slots of its own pools. Each decode+step it fetches a share of its misses (`--moe-locked-fetch`, by default from the `ft bench bw` profile): CPU threads+copy the experts into a small pinned staging buffer+(`--moe-locked-max-fetch` rows per bank) and the GPU copies them into the+slots. The CPU computes the remaining misses, as on any hybrid layer. Under+`offload`, locked layers decode on the CPU.+### NVLink-C2C hosts (GH200 / GB200)The host link is ~450 GB/s per direction instead of PCIe's ~32-64 GB/s. Thediff --git a/python/freetoken/engine/config.py b/python/freetoken/engine/config.pyindex a6c47dd0..9593667e 100644--- a/python/freetoken/engine/config.py+++ b/python/freetoken/engine/config.py@@ -74,6 +74,13 @@ class EngineConfig:# misses so the PCIe fetch and the CPU compute finish together (perfect overlap);# falls back to a fixed cap of 1 without a usable `ft bench bw` profile.moe_hybrid_max_fetch: int = -1+ # Hybrid with locked layers (banks the pin quota could not take, OS-locked instead): a locked layer keeps GPU+ # slots of its own pools and fetches this share of each decode step's misses through pinned staging; the rest+ # are computed on the CPU. -1 (default) = auto: per layer from its `ft bench bw` row (bench_profile.locked_fetch_fraction),+ # a fixed cap of 1 without one. 0 = locked layers decode on the CPU only. moe_locked_max_fetch caps the fetches per+ # (layer, step) and sizes the staging (that many rows of the widest expert, per bank).+ moe_locked_fetch: float = -1.0+ moe_locked_max_fetch: int = 8cuda_graph_bs: List[int] | None = Nonecuda_graph_max_bs: int | None = Nonepage_size: int = 1@@ -104,6 +111,13 @@ class EngineConfig:mm: MultimodalConfig = field(default_factory=MultimodalConfig)def __post_init__(self):+ if not (self.moe_locked_fetch == -1 or 0.0 <= self.moe_locked_fetch <= 1.0):+ raise ValueError(f"moe_locked_fetch must be -1 (auto) or in [0, 1], got {self.moe_locked_fetch}")+ if self.moe_locked_fetch > 0 and round(self.moe_locked_fetch * (1 << 16)) == 0:+ # the ensure kernel takes the share in Q16: this one rounds to 0 and the layers would never fetch+ raise ValueError(f"moe_locked_fetch {self.moe_locked_fetch} rounds to 0 in Q16; pass 0 to keep locked layers CPU-only")+ if self.moe_locked_max_fetch < 1:+ raise ValueError(f"moe_locked_max_fetch must be at least 1, got {self.moe_locked_max_fetch}")if self.moe_backend is None:returnif self.moe_strategy != "auto":diff --git a/python/freetoken/engine/engine.py b/python/freetoken/engine/engine.pyindex b93de55c..7982095a 100644--- a/python/freetoken/engine/engine.py+++ b/python/freetoken/engine/engine.py@@ -22,8 +22,9 @@ from freetoken.mm.config import ENCODER_SECTIONSfrom freetoken.models import create_model, load_weightfrom freetoken.models.weight import ftw_lacks_visionfrom freetoken.moe import is_offload_moe_strategy-from freetoken.moe.expert_banks import attach_resident_banks, load_expert_banks+from freetoken.moe.expert_banks import attach_resident_banks, load_expert_banks, pin_probe_enabledfrom freetoken.moe.host_banks import PinFailed+from freetoken.moe.locked_fetch import attach_locked_fetch, check_locked_fetch_config, locked_pool_tags, settle_banks, staging_reservefrom freetoken.moe.offload_cache import OffloadMoeCache, attach_offload_moe_cache, bank_layoutsfrom freetoken.moe.pool_policy import group_layers, make_pools, pool_of_layer, resolve_profile, split_slotsfrom freetoken.utils import align_ceil, init_logger, is_sm90_family, is_sm100_family, mem_GB, torch_dtype@@ -685,56 +686,35 @@ class Engine:"this model's experts come in several formats, which are served from per-format GPU slot pools; ""they do not combine with --moe-strategy cpu or --moe-cpu-layers (use offload or hybrid)")- cpu_layer_ids = _resolve_cpu_layers(config, num_moe_layers, reserved=self._host_tables_bytes, method=method)- _check_pin_budget(config, reserved=self._host_tables_bytes, method=method, layer_methods=layer_methods)- # the kernels were picked for model_config.decode_target; --moe-cpu-layers auto may still find that every bank fits the pin budget- decode_target = config.model_config.decode_target- if decode_target == "cpu" and not cpu_layer_ids:- decode_target = "gpu"- # split residency: where pinning is quota-capped (_pin_budget_bytes), pin only the GPU layers' banks and mlock the CPU layers'- # uncapped hosts keep every bank pinned (CPU decode reads them the same; overlap prefill stays on)- # not applied to plain --moe-strategy cpu; all-locked under a cap = --moe-strategy offload --moe-cpu-layers 1.0- split_residency = (- bool(cpu_layer_ids)- and config.moe_strategy in ("offload", "hybrid")- and _pin_budget_bytes(self._host_tables_bytes) is not None- )- if config.moe_strategy == "cpu" and not split_residency:- # cpu mode pins every bank for the prefill double buffer; over the pin cap that dies in cudaHostRegister, so lock everything instead- from freetoken.moe.expert_banks import bank_bytes_estimate, ftw_bank_bytes-- budget = _pin_budget_bytes(self._host_tables_bytes)- bank_bytes = None- if budget is not None:- bank_bytes = ftw_bank_bytes(config.model_path) or bank_bytes_estimate(config.model_config, method)- if bank_bytes and bank_bytes > budget:- split_residency = True- logger.info_rank0(- f"--moe-strategy cpu: banks {bank_bytes / 2**30:.2f} GiB exceed the "- f"pin budget; OS-locking all layers instead of pinning"- )- if split_residency and config.moe_prefill_overlap:- # locked (unregistered) layers cannot feed the async pinned H2D double buffer; their prefill is a synchronous pageable copy via materialize- logger.info_rank0(- "--moe-cpu-layers split residency: disabling MoE prefill overlap "- "(locked layers prefill via synchronous pageable copies)"- )- object.__setattr__(config, "moe_prefill_overlap", False)# Fast path: an FTW checkpoint loads its repacked banks directly.# Slow path: load_expert_banks auto-picks parallel vs serial baseline by# expert-tensor granularity. Both pin-after-fill.# --expert-load: serial/parallel force the read; auto (None) lets load_expert_banks# pick (parallel for scattered experts, with a low-RAM fallback to serial).expert_parallel = {"serial": False, "parallel": True}.get(config.expert_load, None)- requested_residency = None- if split_residency:+ check_locked_fetch_config(config)+ # --moe-cpu-layers auto without a manual pin budget: load unpinned, then pin layers until the driver refuses (settle_banks)+ # a model the CPU executor cannot serve takes the estimate path, which rejects it before the load when layers must lock+ probe = pin_probe_enabled(config, mixed_formats=layer_methods is not None) and _cpu_moe_executor_viable(config.model_config)+ if probe:from freetoken.moe.host_banks import HostResidency- requested_residency = [- HostResidency.LOCKED.value if i in cpu_layer_ids- else HostResidency.PINNED.value- for i in range(config.model_config.num_moe_layers)- ]+ requested_residency = [HostResidency.DEFERRED.value] * num_moe_layers+ cpu_layer_ids, decode_target, split_residency = frozenset(), None, False+ else:+ reserved = self._host_tables_bytes + staging_reserve(config, lambda: _bank_bytes(config, method, layer_methods))+ cpu_layer_ids = _resolve_cpu_layers(config, num_moe_layers, reserved=reserved, method=method)+ _check_pin_budget(config, reserved=reserved, method=method, layer_methods=layer_methods)+ decode_target, split_residency = self._residency_split(config, cpu_layer_ids, method, capped=_pin_budget_bytes(reserved) is not None)+ requested_residency = None+ if split_residency:+ from freetoken.moe.host_banks import HostResidency++ requested_residency = [+ HostResidency.LOCKED.value if i in cpu_layer_ids+ else HostResidency.PINNED.value+ for i in range(config.model_config.num_moe_layers)+ ]# read before the expert load, so a bad profile fails in secondsexpert_counts = resolve_profile(config.moe_cache_split, config.model_path,@@ -754,14 +734,25 @@ class Engine:)except PinFailed as exc:raise RuntimeError(f"{exc}; {_pin_hint(self._host_tables_bytes)}") from exc+ banks, probed, staging = settle_banks(config, banks, probe=probe, has_locked=split_residency)+ if probe:+ cpu_layer_ids = probed+ decode_target, split_residency = self._residency_split(config, cpu_layer_ids, method, capped=True)from freetoken.engine.cache_budget import expert_bytes_per_slot, layer_slot_bytesself._moe_layer_slot_bytes = layer_slot_bytes(banks.sources)- self._moe_pool_groups = group_layers(config.moe_cache_scope, bank_layouts(banks.sources, banks.layer_tags), self._moe_layer_slot_bytes)- if len(self._moe_pool_groups) > 1 and cpu_layer_ids:+ # hybrid: locked layers keep GPU slots of their own pools and fetch part of their misses through the pinned staging (moe/locked_fetch.py)+ locked_fetch = staging is not None+ layer_tags = locked_pool_tags(banks.layer_tags, cpu_layer_ids, num_moe_layers) if locked_fetch else banks.layer_tags+ self._moe_pool_groups = group_layers(config.moe_cache_scope, bank_layouts(banks.sources, layer_tags), self._moe_layer_slot_bytes)+ if locked_fetch:+ # a locked refill costs a staged copy: place the locked pools after the pinned ones, which hold the prefill staging when they are big enough+ self._moe_pool_groups.sort(key=lambda layers: layers[0] in cpu_layer_ids)+ if len(self._moe_pool_groups) > 1 and cpu_layer_ids and not locked_fetch:raise ValueError("several MoE slot pools (--moe-cache-scope per-layer, or experts in several quant formats) need every MoE "- "layer to decode from the GPU slot cache; they do not combine with --moe-strategy cpu or --moe-cpu-layers"+ "layer to decode from the GPU slot cache; they combine with --moe-cpu-layers only for locked layers under "+ "--moe-strategy hybrid unless --moe-locked-fetch is 0")self._moe_expert_counts = expert_counts if len(self._moe_pool_groups) > 1 else Noneif len(self._moe_pool_groups) == 1 and config.moe_cache_split != "equal":@@ -852,7 +843,9 @@ class Engine:)# before set_bank_sources: the residency validation and the copy plan's skip of non-pinned layers key on the CPU-layer setcache.cpu_layer_ids = cpu_layer_ids- cache.set_bank_sources(banks.sources, layer_residency=banks.layer_residency, layer_tags=banks.layer_tags, row_tail_bytes=banks.row_tail_bytes)+ if locked_fetch:+ cache.locked_hybrid = cpu_layer_ids+ cache.set_bank_sources(banks.sources, layer_residency=banks.layer_residency, layer_tags=layer_tags, row_tail_bytes=banks.row_tail_bytes)cache.set_alphas(banks.gate_up_alpha, banks.down_alpha)# Must be set before CUDA graph capture so the (device-side) accumulation ops are# captured and re-run on every decode replay.@@ -863,10 +856,49 @@ class Engine:self._init_cpu_moe_executor(config, cache, layers)if decode_target == "hybrid":self._resolve_hybrid_fetch(config, cache)+ if locked_fetch:+ attach_locked_fetch(cache, config, staging, self._profile_layer_rows)self.ctx.moe_offload_cache = cacheself.moe_offload_cache = cachereturn cache+ def _residency_split(self, config: EngineConfig, cpu_layer_ids, method, *, capped: bool):+ """(decode_target, split_residency) for a CPU-layer set; turns prefill overlap off when some layers stay unpinned."""+ # the kernels were picked for model_config.decode_target; --moe-cpu-layers auto may still find that every bank fits the pin budget+ decode_target = config.model_config.decode_target+ if decode_target == "cpu" and not cpu_layer_ids:+ decode_target = "gpu"+ # split residency: where pinning is quota-capped (_pin_budget_bytes), pin only the GPU layers' banks and mlock the CPU layers'+ # uncapped hosts keep every bank pinned (CPU decode reads them the same; overlap prefill stays on)+ # not applied to plain --moe-strategy cpu; all-locked under a cap = --moe-strategy offload --moe-cpu-layers 1.0+ split_residency = (+ bool(cpu_layer_ids)+ and config.moe_strategy in ("offload", "hybrid")+ and capped+ )+ if config.moe_strategy == "cpu" and not split_residency:+ # cpu mode pins every bank for the prefill double buffer; over the pin cap that dies in cudaHostRegister, so lock everything instead+ from freetoken.moe.expert_banks import bank_bytes_estimate, ftw_bank_bytes++ budget = _pin_budget_bytes(self._host_tables_bytes)+ bank_bytes = None+ if budget is not None:+ bank_bytes = ftw_bank_bytes(config.model_path) or bank_bytes_estimate(config.model_config, method)+ if bank_bytes and bank_bytes > budget:+ split_residency = True+ logger.info_rank0(+ f"--moe-strategy cpu: banks {bank_bytes / 2**30:.2f} GiB exceed the "+ f"pin budget; OS-locking all layers instead of pinning"+ )+ if split_residency and config.moe_prefill_overlap:+ # locked (unregistered) layers cannot feed the async pinned H2D double buffer; their prefill is a synchronous pageable copy via materialize+ logger.info_rank0(+ "--moe-cpu-layers split residency: disabling MoE prefill overlap "+ "(locked layers prefill via synchronous pageable copies)"+ )+ object.__setattr__(config, "moe_prefill_overlap", False)+ return decode_target, split_residency+def _moe_pool_bytes(self, pool_sizes: list[int]) -> int:return sum(s * self._moe_layer_slot_bytes[g[0]] for g, s in zip(self._moe_pool_groups, pool_sizes))@@ -889,26 +921,19 @@ class Engine:"""if config.moe_hybrid_max_fetch >= 0:return # explicit fixed cap- from freetoken.moe.bench_profile import fetch_fraction, layer_kind_key, load_layer_kinds- from freetoken.moe.cpu_executor import _format_label+ from freetoken.moe.bench_profile import fetch_fractioncache.hybrid_max_fetch = 1- methods = offload_layer_methods(self.model)- if methods is None:+ layer_rows = self._profile_layer_rows()+ if layer_rows is None:logger.warning_rank0("--moe-hybrid-max-fetch auto: the MoE layers bind no expert method to look up in the `ft bench bw` profile; using a fixed fetch cap of 1")return- gpu_name, gpu_uuid = _profile_gpu(self.device.index)- rows = load_layer_kinds(gpu_name=gpu_name, gpu_uuid=gpu_uuid) or {}executor = self.cpu_moe_executor- keys, labels = [], []- for m in methods:- fmt, opts = m.cpu_spec()- c = m.cfg- keys.append(layer_kind_key(fmt, opts, c.hidden, c.local_intermediate, c.top_k, executor.num_threads))- labels.append(f"{_format_label(fmt, tuple(sorted(opts.items())))} {c.hidden}x{c.local_intermediate}")+ keys, labels, rows = layer_rowsfractions = [fetch_fraction(rows.get(key)) for key in keys]cache.hybrid_fetch_fraction = [f or 0.0 for f in fractions]- groups = collections.Counter(zip(labels, fractions))+ # locked layers fetch the fetch_plan share instead; their own log line lists it+ groups = collections.Counter((labels[l], fractions[l]) for l in range(len(keys)) if l not in cache.locked_hybrid)fetched = [f"{f:.1%} on {n} {label} layers" for (label, f), n in groups.items() if f is not None]if fetched:logger.info_rank0(@@ -928,6 +953,24 @@ class Engine:f"{executor.isa}; using them anyway")+ def _profile_layer_rows(self):+ """(row key, label) of each MoE layer's kind in the `ft bench bw` profile, and the profile's rows; None without expert methods."""+ from freetoken.moe.bench_profile import layer_kind_key, load_layer_kinds+ from freetoken.moe.cpu_executor import _format_label++ methods = offload_layer_methods(self.model)+ if methods is None:+ return None+ gpu_name, gpu_uuid = _profile_gpu(self.device.index)+ rows = load_layer_kinds(gpu_name=gpu_name, gpu_uuid=gpu_uuid) or {}+ keys, labels = [], []+ for m in methods:+ fmt, opts = m.cpu_spec()+ c = m.cfg+ keys.append(layer_kind_key(fmt, opts, c.hidden, c.local_intermediate, c.top_k, self.cpu_moe_executor.num_threads))+ labels.append(f"{_format_label(fmt, tuple(sorted(opts.items())))} {c.hidden}x{c.local_intermediate}")+ return keys, labels, rows+def _init_cpu_moe_executor(self, config: EngineConfig, cache, layers) -> None:"""Build the persistent CPU MoE executor (decode-time expert compute).@@ -1587,6 +1630,8 @@ _DENSE_MOE_SETTINGS = {"moe_cpu_layers": None,"moe_cpu_threads": 0,"moe_hybrid_max_fetch": -1,+ "moe_locked_fetch": -1.0,+ "moe_locked_max_fetch": 8,"moe_prefill_overlap": True,"moe_prefill_hit_d2d": False,"expert_load": "auto",diff --git a/python/freetoken/kernel/csrc/cpu_moe/cpu_moe_ext.cpp b/python/freetoken/kernel/csrc/cpu_moe/cpu_moe_ext.cppindex 8162138c..b0de7085 100644--- a/python/freetoken/kernel/csrc/cpu_moe/cpu_moe_ext.cpp+++ b/python/freetoken/kernel/csrc/cpu_moe/cpu_moe_ext.cpp@@ -28,17 +28,10 @@#include <freetoken/hip_compat.h>#include <torch/extension.h>-#if defined(__linux__)-#include <pthread.h>-#include <sched.h>-#define CPU_MOE_HAS_AFFINITY 1-#else-#define CPU_MOE_HAS_AFFINITY 0-#endif-#include "format.h"#include "gpu_sync.h"#include "gpu_sync_io.h"+#include "threads.h"namespace freetoken::cpu_moe {@@ -50,28 +43,10 @@ std::unique_ptr<MoeFormat> make_fp8_block_format(const MoeParams& p, const Formastd::unique_ptr<MoeFormat> make_gguf_format(const MoeParams& p, const FormatArgs& a);std::unique_ptr<MoeFormat> make_gguf_scaled_format(const MoeParams& p, const FormatArgs& a);void set_ggml_cpu(uintptr_t addr);+void bind_locked_fetch(pybind11::module_& m);namespace {-inline void pin_this_thread(int cpu) {-#if CPU_MOE_HAS_AFFINITY- cpu_set_t set;- CPU_ZERO(&set);- CPU_SET(cpu, &set);- pthread_setaffinity_np(pthread_self(), sizeof(set), &set);-#else- (void)cpu;-#endif-}--inline void cpu_relax() {-#if CPU_MOE_X86- _mm_pause();-#elif defined(__aarch64__)- asm volatile("yield" ::: "memory");-#endif-}-constexpr int kMaxTopK = 64;// One MoE layer on the first num_tokens rows of the pinned IO.@@ -566,6 +541,7 @@ PYBIND11_MODULE(TORCH_EXTENSION_NAME, m) {[](CpuMoeExecutor& e, bool v) { e.spin_wait.store(v, std::memory_order_relaxed); },py::arg("enabled")).def("get_spin_wait", [](const CpuMoeExecutor& e) { return e.spin_wait.load(); });+ freetoken::cpu_moe::bind_locked_fetch(m);m.def("memops_probe", &GpuSync::probe, py::arg("stream"));m.def("kernel_io_probe", &freetoken::cpu_moe::KernelIo::probe, py::arg("stream"));m.def("host_func_capture_safe", &GpuSync::host_func_capture_safe);diff --git a/python/freetoken/kernel/csrc/cpu_moe/gpu_sync.h b/python/freetoken/kernel/csrc/cpu_moe/gpu_sync.hindex 829a8f34..0e447751 100644--- a/python/freetoken/kernel/csrc/cpu_moe/gpu_sync.h+++ b/python/freetoken/kernel/csrc/cpu_moe/gpu_sync.h@@ -44,8 +44,9 @@ inline void flag_store_release(volatile int64_t* p, int64_t v) {class GpuSync {public:- // Allocates the handshake memory; throws if this runtime cannot.- GpuSync();+ // Allocates the handshake memory; throws if this runtime cannot, or lacks stream memops while require_memops.+ // Without memops only the words, cpu_poll and cpu_done serve: kernels write the words and wait on them.+ explicit GpuSync(bool require_memops = true);~GpuSync();GpuSync(const GpuSync&) = delete;GpuSync& operator=(const GpuSync&) = delete;diff --git a/python/freetoken/kernel/csrc/cpu_moe/gpu_sync_cuda.cpp b/python/freetoken/kernel/csrc/cpu_moe/gpu_sync_cuda.cppindex a2a089bf..728776d3 100644--- a/python/freetoken/kernel/csrc/cpu_moe/gpu_sync_cuda.cpp+++ b/python/freetoken/kernel/csrc/cpu_moe/gpu_sync_cuda.cpp@@ -94,8 +94,8 @@ struct GpuSync::Impl {volatile int64_t* done = nullptr;};-GpuSync::GpuSync() : impl_(std::make_unique<Impl>()) {- if (!cumemop_resolve()) throw std::runtime_error("CPU MoE flag sync: CUDA stream memops are unavailable");+GpuSync::GpuSync(bool require_memops) : impl_(std::make_unique<Impl>()) {+ if (require_memops && !cumemop_resolve()) throw std::runtime_error("CPU MoE flag sync: CUDA stream memops are unavailable");const cudaError_t rc = cudaHostAlloc(&impl_->block, 128, cudaHostAllocPortable | cudaHostAllocMapped);if (rc != cudaSuccess)throw std::runtime_error(std::string("CPU MoE flag allocation failed: ") + cudaGetErrorString(rc));diff --git a/python/freetoken/kernel/csrc/cpu_moe/gpu_sync_hip.cpp b/python/freetoken/kernel/csrc/cpu_moe/gpu_sync_hip.cppindex 31ff9be5..395f2684 100644--- a/python/freetoken/kernel/csrc/cpu_moe/gpu_sync_hip.cpp+++ b/python/freetoken/kernel/csrc/cpu_moe/gpu_sync_hip.cpp@@ -162,7 +162,8 @@ struct GpuSync::Impl {hipStreamBatchMemOpParams wait_op{};};-GpuSync::GpuSync() : impl_(std::make_unique<Impl>()) {+// The memop-free mode serves only the locked fetch, which ROCm keeps off: every GpuSync here needs memops.+GpuSync::GpuSync(bool) : impl_(std::make_unique<Impl>()) {hipError_t rc = alloc_signal(&impl_->ready);if (rc == hipSuccess) rc = alloc_signal(&impl_->done);if (rc != hipSuccess) {diff --git a/python/freetoken/kernel/csrc/cpu_moe/locked_fetch.cpp b/python/freetoken/kernel/csrc/cpu_moe/locked_fetch.cppnew file mode 100644index 00000000..e2e7a895--- /dev/null+++ b/python/freetoken/kernel/csrc/cpu_moe/locked_fetch.cpp@@ -0,0 +1,205 @@+// Locked-layer fetch for hybrid decode (python/freetoken/moe/locked_fetch.py).+//+// A locked layer's host banks have no device address, so the GPU cannot gather its rows. Per decode step a publish kernel+// writes the fetch list (count + layer-local expert rows) into mapped meta and rings a doorbell (GpuSync's words); the+// coordinator and worker threads memcpy each listed row of every bank into pinned staging, then complete the doorbell,+// and fast_index_copy_multi copies staging row i into the i-th evicted slot after a stream memop wait, or with its own+// spin wait where memops do not work. Everything is kernels and memops, so it captures into the decode graph. Row i of a+// bank sits at i * the layer's own row bytes, so layers of different widths share staging.++#include <algorithm>+#include <atomic>+#include <chrono>+#include <cstdint>+#include <cstring>+#include <memory>+#include <stdexcept>+#include <thread>+#include <vector>++#include <torch/extension.h>++#include <freetoken/hip_compat.h>++#include "gpu_sync.h"+#include "locked_fetch_io.h"+#include "threads.h"++namespace freetoken::cpu_moe {+namespace {++using fetch_clock = std::chrono::steady_clock;+// same idle policy as the MoE coordinator: spin while decode traffic flows, sleep once it stops+constexpr auto kHotWindow = std::chrono::milliseconds(50);+constexpr int64_t kChunkBytes = 256 << 10;++struct CopyJob {+ char* dst;+ const char* src;+ int64_t bytes;+};++class LockedFetch {+ public:+ // src[l][b] / row_bytes[l][b]: host base and row bytes of layer l's bank b (0 for banks or layers it never fetches).+ // memops: wait on the doorbell with a stream memop, else the gather spins on done_word().+ LockedFetch(std::vector<std::vector<uintptr_t>> src, std::vector<std::vector<int64_t>> row_bytes, std::vector<uintptr_t> staging,+ int max_rows, std::vector<int> cpus, bool memops)+ : src_(std::move(src)), row_bytes_(std::move(row_bytes)), staging_(std::move(staging)), max_rows_(max_rows), memops_(memops) {+ sync_ = std::make_unique<GpuSync>(memops);+ void* meta = nullptr;+ void* meta_dev = nullptr;+ const size_t meta_bytes = sizeof(int64_t) * (1 + (max_rows + 1) / 2);+ if (cudaHostAlloc(&meta, meta_bytes, cudaHostAllocPortable | cudaHostAllocMapped) != cudaSuccess ||+ cudaHostGetDevicePointer(&meta_dev, meta, 0) != cudaSuccess) {+ (void)cudaGetLastError();+ if (meta) (void)cudaFreeHost(meta);+ throw std::runtime_error("locked fetch: mapped meta allocation failed");+ }+ meta_ = meta;+ count_ = static_cast<volatile int64_t*>(meta);+ rows_ = reinterpret_cast<const int32_t*>(static_cast<int64_t*>(meta) + 1);+ count_dev_ = static_cast<int64_t*>(meta_dev);+ rows_dev_ = reinterpret_cast<int32_t*>(static_cast<int64_t*>(meta_dev) + 1);+ const int coord_cpu = cpus.empty() ? -1 : cpus[0];+ for (size_t i = 1; i < cpus.size(); ++i) workers_.emplace_back([this, cpu = cpus[i]] { worker_loop(cpu); });+ coord_ = std::thread([this, coord_cpu] { coordinator_loop(coord_cpu); });+ }++ ~LockedFetch() {+ stop_.store(true);+ coord_.join();+ for (auto& t : workers_) t.join();+ (void)cudaFreeHost(meta_);+ }++ LockedFetch(const LockedFetch&) = delete;+ LockedFetch& operator=(const LockedFetch&) = delete;++ // Hands one layer step's fetch list (the hybrid ensure's num_indices and src_indices) to the copy threads, then waits for+ // them on `stream` when memops work; otherwise the gather that follows waits on done_word() itself.+ void publish(uintptr_t stream, int layer, uintptr_t num_indices, uintptr_t src_indices) {+ locked_publish(stream, reinterpret_cast<const int64_t*>(num_indices), reinterpret_cast<const int32_t*>(src_indices), max_rows_,+ count_dev_, rows_dev_, reinterpret_cast<volatile int64_t*>(sync_->ready_word()),+ reinterpret_cast<volatile int64_t*>(sync_->done_word()), static_cast<int64_t>(layer) + 1);+ if (memops_) sync_->gpu_wait(stream);+ }+ uintptr_t done_word() const { return sync_->done_word(); }++ private:+ void run_jobs() {+ for (;;) {+ const int64_t i = next_.fetch_add(1, std::memory_order_relaxed);+ if (i >= static_cast<int64_t>(jobs_.size())) return;+ std::memcpy(jobs_[i].dst, jobs_[i].src, static_cast<size_t>(jobs_[i].bytes));+ }+ }++ void worker_loop(int cpu) {+ if (cpu >= 0) pin_this_thread(cpu);+ uint64_t seen = 0;+ auto last_active = fetch_clock::now();+ int64_t sleep_us = 20;+ while (!stop_.load(std::memory_order_relaxed)) {+ const uint64_t gen = gen_.load(std::memory_order_acquire);+ if (gen == seen) {+ if (fetch_clock::now() - last_active < kHotWindow) {+ cpu_relax();+ } else {+ std::this_thread::sleep_for(std::chrono::microseconds(sleep_us));+ sleep_us = std::min<int64_t>(sleep_us * 2, 2000);+ }+ continue;+ }+ seen = gen;+ run_jobs();+ finished_.fetch_add(1, std::memory_order_acq_rel);+ last_active = fetch_clock::now();+ sleep_us = 20;+ }+ }++ // Splits this step's rows into chunk jobs and runs them with the workers.+ void copy_layer(int layer) {+ int64_t n = *count_;+ n = std::max<int64_t>(0, std::min<int64_t>(n, max_rows_));+ jobs_.clear();+ const auto& src = src_[layer];+ const auto& row_bytes = row_bytes_[layer];+ for (int64_t i = 0; i < n; ++i) {+ const int64_t row = rows_[i];+ for (size_t b = 0; b < staging_.size(); ++b) {+ const int64_t rb = row_bytes[b];+ if (rb == 0) continue;+ char* dst = reinterpret_cast<char*>(staging_[b]) + i * rb;+ const char* s = reinterpret_cast<const char*>(src[b]) + row * rb;+ for (int64_t off = 0; off < rb; off += kChunkBytes) jobs_.push_back({dst + off, s + off, std::min(kChunkBytes, rb - off)});+ }+ }+ if (jobs_.empty()) return;+ next_.store(0, std::memory_order_relaxed);+ finished_.store(0, std::memory_order_relaxed);+ gen_.fetch_add(1, std::memory_order_acq_rel);+ run_jobs();+ // on stop a worker may leave before it joins this generation; the coordinator's run_jobs already took every job+ while (finished_.load(std::memory_order_acquire) < static_cast<int64_t>(workers_.size()) && !stop_.load(std::memory_order_relaxed))+ cpu_relax();+ }++ void coordinator_loop(int cpu) {+ if (cpu >= 0) pin_this_thread(cpu);+ auto last_active = fetch_clock::now();+ int64_t sleep_us = 20;+ while (!stop_.load(std::memory_order_relaxed)) {+ const uint64_t task = sync_->cpu_poll();+ if (task == 0) {+ if (fetch_clock::now() - last_active < kHotWindow) {+ cpu_relax();+ } else {+ std::this_thread::sleep_for(std::chrono::microseconds(sleep_us));+ sleep_us = std::min<int64_t>(sleep_us * 2, 2000);+ }+ continue;+ }+ copy_layer(static_cast<int>(task - 1));+ // release: the staging stores are visible before the GPU sees done+ sync_->cpu_done();+ last_active = fetch_clock::now();+ sleep_us = 20;+ }+ sync_->cpu_done(); // a replay caught mid-teardown does not stall+ }++ std::vector<std::vector<uintptr_t>> src_;+ std::vector<std::vector<int64_t>> row_bytes_;+ std::vector<uintptr_t> staging_;+ int64_t max_rows_;+ bool memops_;+ std::unique_ptr<GpuSync> sync_;+ void* meta_ = nullptr;+ const volatile int64_t* count_ = nullptr;+ const int32_t* rows_ = nullptr;+ int64_t* count_dev_ = nullptr;+ int32_t* rows_dev_ = nullptr;+ std::vector<CopyJob> jobs_;+ std::atomic<int64_t> next_{0};+ std::atomic<int64_t> finished_{0};+ std::atomic<uint64_t> gen_{0};+ std::atomic<bool> stop_{false};+ std::vector<std::thread> workers_;+ std::thread coord_;+};++} // namespace++void bind_locked_fetch(pybind11::module_& m) {+ namespace py = pybind11;+ py::class_<LockedFetch>(m, "LockedFetch")+ .def(py::init<std::vector<std::vector<uintptr_t>>, std::vector<std::vector<int64_t>>, std::vector<uintptr_t>, int, std::vector<int>,+ bool>(),+ py::arg("src"), py::arg("row_bytes"), py::arg("staging"), py::arg("max_rows"), py::arg("cpus"), py::arg("memops"))+ .def("publish", &LockedFetch::publish, py::arg("stream"), py::arg("layer"), py::arg("num_indices"), py::arg("src_indices"))+ .def("done_word", &LockedFetch::done_word);+}++} // namespace freetoken::cpu_moediff --git a/python/freetoken/kernel/csrc/cpu_moe/locked_fetch_io.cu b/python/freetoken/kernel/csrc/cpu_moe/locked_fetch_io.cunew file mode 100644index 00000000..a264a549--- /dev/null+++ b/python/freetoken/kernel/csrc/cpu_moe/locked_fetch_io.cu@@ -0,0 +1,52 @@+#include <stdexcept>+#include <string>++#include <freetoken/hip_compat.h>++#include "locked_fetch_io.h"++namespace freetoken::cpu_moe {+namespace {++__device__ inline int64_t clamp_rows(const int64_t* num_indices, int max_rows) {+ const int64_t n = *num_indices;+ return n < 0 ? 0 : (n > max_rows ? max_rows : n);+}++// One block. The rows land before the doorbell, which the CPU reads with acquire.+__global__ void publish_kernel(const int64_t* num_indices, const int32_t* src_indices, int max_rows, int64_t* meta_count,+ int32_t* meta_rows, volatile int64_t* ready, volatile int64_t* done, int64_t task) {+ const int64_t n = clamp_rows(num_indices, max_rows);+ if (n == 0) {+ if (threadIdx.x == 0) {+ *done = 1;+ __threadfence_system();+ }+ return;+ }+ for (int64_t i = threadIdx.x; i < n; i += blockDim.x) meta_rows[i] = src_indices[i];+ if (threadIdx.x == 0) *meta_count = n;+ __threadfence_system();+ __syncthreads();+ if (threadIdx.x == 0) {+ *done = 0;+ __threadfence_system();+ *ready = task;+ __threadfence_system();+ }+}++void check(cudaError_t rc, const char* what) {+ if (rc != cudaSuccess) throw std::runtime_error(std::string("locked fetch: ") + what + ": " + cudaGetErrorString(rc));+}++} // namespace++void locked_publish(uintptr_t stream, const int64_t* num_indices, const int32_t* src_indices, int max_rows, int64_t* meta_count,+ int32_t* meta_rows, volatile int64_t* ready, volatile int64_t* done, int64_t task) {+ publish_kernel<<<1, 256, 0, reinterpret_cast<cudaStream_t>(stream)>>>(num_indices, src_indices, max_rows, meta_count, meta_rows, ready,+ done, task);+ check(cudaGetLastError(), "publish launch");+}++} // namespace freetoken::cpu_moediff --git a/python/freetoken/kernel/csrc/cpu_moe/locked_fetch_io.h b/python/freetoken/kernel/csrc/cpu_moe/locked_fetch_io.hnew file mode 100644index 00000000..57e6e2ef--- /dev/null+++ b/python/freetoken/kernel/csrc/cpu_moe/locked_fetch_io.h@@ -0,0 +1,14 @@+// GPU side of the locked-layer fetch (locked_fetch.cpp): one publish kernel in place of the meta copies and the doorbell writes.+// One source for both runtimes like gpu_sync_io.cu.+#pragma once++#include <cstdint>++namespace freetoken::cpu_moe {++// Writes min(*num_indices, max_rows) and that many src rows into the mapped meta, then done = 0 and ready = task.+// A step with nothing to fetch writes done = 1 and rings nothing, so the wait and the gather pass at once.+void locked_publish(uintptr_t stream, const int64_t* num_indices, const int32_t* src_indices, int max_rows, int64_t* meta_count,+ int32_t* meta_rows, volatile int64_t* ready, volatile int64_t* done, int64_t task);++} // namespace freetoken::cpu_moediff --git a/python/freetoken/kernel/csrc/cpu_moe/threads.h b/python/freetoken/kernel/csrc/cpu_moe/threads.hnew file mode 100644index 00000000..f7d9306b--- /dev/null+++ b/python/freetoken/kernel/csrc/cpu_moe/threads.h@@ -0,0 +1,35 @@+// Thread placement and spin helpers shared by the CPU MoE pool and the locked-layer fetch threads.+#pragma once++#include "common.h"++#if defined(__linux__)+#include <pthread.h>+#include <sched.h>+#define CPU_MOE_HAS_AFFINITY 1+#else+#define CPU_MOE_HAS_AFFINITY 0+#endif++namespace freetoken::cpu_moe {++inline void pin_this_thread(int cpu) {+#if CPU_MOE_HAS_AFFINITY+ cpu_set_t set;+ CPU_ZERO(&set);+ CPU_SET(cpu, &set);+ pthread_setaffinity_np(pthread_self(), sizeof(set), &set);+#else+ (void)cpu;+#endif+}++inline void cpu_relax() {+#if CPU_MOE_X86+ _mm_pause();+#elif defined(__aarch64__)+ asm volatile("yield" ::: "memory");+#endif+}++} // namespace freetoken::cpu_moediff --git a/python/freetoken/kernel/csrc/include/freetoken/hip_compat.h b/python/freetoken/kernel/csrc/include/freetoken/hip_compat.hindex 8abcd1a0..d1a22fe4 100644--- a/python/freetoken/kernel/csrc/include/freetoken/hip_compat.h+++ b/python/freetoken/kernel/csrc/include/freetoken/hip_compat.h@@ -72,6 +72,10 @@#define cudaHostRegister hipHostRegister#endif+#ifndef cudaHostUnregister+#define cudaHostUnregister hipHostUnregister+#endif+#ifndef cudaHostRegisterPortable#define cudaHostRegisterPortable hipHostRegisterPortable#endifdiff --git a/python/freetoken/kernel/csrc/jit/fast_index_copy.cuh b/python/freetoken/kernel/csrc/jit/fast_index_copy.cuhindex 43f3fc1b..279b15ae 100644--- a/python/freetoken/kernel/csrc/jit/fast_index_copy.cuh+++ b/python/freetoken/kernel/csrc/jit/fast_index_copy.cuh@@ -507,13 +507,15 @@ struct FastIndexCopyKernel {// block b copies bank `b = blockIdx.x / kBlocksPerBank`, grid-striding over that// bank's (num_indices * feat/16) 16-byte units. Pointers + feature sizes are passed// as small device arrays (built once by the cache), so it stays CUDA-graph capturable.+// With wait_flag set, each block first waits for *wait_flag >= 1, a flag in mapped host memory the CPU sets once the source rows are written.struct MultiIndexCopyParams {const int64_t* __restrict__ dst_ptrs; // [B] device, each base addr of a bank slot cacheconst int64_t* __restrict__ src_ptrs; // [B] device, each GPU-visible base addr of a bank host sourceconst int64_t* __restrict__ feat_bytes; // [B] device, per-row bytes (multiple of 16)const void* __restrict__ dst_indices; // [L]- const void* __restrict__ src_indices; // [L]+ const void* __restrict__ src_indices; // [L], or null: row i copies source row iconst int64_t* __restrict__ valid_length; // [1] or null+ const volatile int64_t* wait_flag; // GPU-visible address, or null: no waitint64_t length; // max Lint num_banks;};@@ -522,6 +524,18 @@ template <typename IdType, std::size_t kNumThreads, std::size_t kBlocksPerBank>__global__ __launch_bounds__(kNumThreads) void fast_index_copy_multi(const __grid_constant__ MultiIndexCopyParams p) {+ if (p.wait_flag != nullptr) {+ if (threadIdx.x == 0) {+ while (*p.wait_flag < 1) {+#if !FREETOKEN_USE_ROCM && __CUDA_ARCH__ >= 700+ __nanosleep(100);+#endif+ }+ // with the volatile read above, an acquire: the block reads the rows written before the flag+ __threadfence_system();+ }+ __syncthreads();+ }const int b = static_cast<int>(blockIdx.x / kBlocksPerBank);if (b >= p.num_banks) {return;@@ -540,7 +554,7 @@ __global__ __launch_bounds__(kNumThreads) void fast_index_copy_multi(const int64_t row = u / units;const int64_t col = (u - row * units) << 4; // byte offset within the rowconst int64_t pd = static_cast<int64_t>(di[row]);- const int64_t ps = static_cast<int64_t>(si[row]);+ const int64_t ps = si ? static_cast<int64_t>(si[row]) : row;const uint4 v = *reinterpret_cast<const uint4*>(src + ps * feat + col);*reinterpret_cast<uint4*>(dst + pd * feat + col) = v;}@@ -553,8 +567,9 @@ struct MultiIndexCopyKernel {tvm::ffi::TensorView src_ptrs,tvm::ffi::TensorView feat_bytes,tvm::ffi::TensorView dst_indices,- tvm::ffi::TensorView src_indices,- tvm::ffi::Optional<tvm::ffi::TensorView> num_indices+ tvm::ffi::Optional<tvm::ffi::TensorView> src_indices,+ tvm::ffi::Optional<tvm::ffi::TensorView> num_indices,+ int64_t wait_flag) {using namespace host;auto device = SymbolicDevice{};@@ -567,7 +582,13 @@ struct MultiIndexCopyKernel {TensorMatcher({B}).with_dtype<int64_t>(ptr_dtype).with_device<kDLCUDA, kDLROCM>(device).verify(dst_ptrs).verify(src_ptrs).verify(feat_bytes);TensorMatcher({L}).with_dtype<int32_t, int64_t>(indices_dtype).with_device<kDLCUDA, kDLROCM>(device)- .verify(dst_indices).verify(src_indices);+ .verify(dst_indices);+ const void* src_rows = nullptr;+ if (src_indices.has_value()) {+ TensorMatcher({L}).with_dtype<int32_t, int64_t>(indices_dtype).with_device<kDLCUDA, kDLROCM>(device)+ .verify(src_indices.value());+ src_rows = src_indices.value().data_ptr();+ }const int64_t* valid_length = nullptr;if (num_indices.has_value()) {@@ -582,8 +603,9 @@ struct MultiIndexCopyKernel {static_cast<const int64_t*>(src_ptrs.data_ptr()),static_cast<const int64_t*>(feat_bytes.data_ptr()),dst_indices.data_ptr(),- src_indices.data_ptr(),+ src_rows,valid_length,+ reinterpret_cast<const volatile int64_t*>(wait_flag),static_cast<int64_t>(L.unwrap()),num_banks,};diff --git a/python/freetoken/kernel/csrc/pinned_tensor.cpp b/python/freetoken/kernel/csrc/pinned_tensor.cppindex 9355f57a..1cdee5a6 100644--- a/python/freetoken/kernel/csrc/pinned_tensor.cpp+++ b/python/freetoken/kernel/csrc/pinned_tensor.cpp@@ -60,6 +60,10 @@ torch::Tensor alloc_pinned_tensor(std::vector<int64_t> sizes,void *data_ptr = nullptr;const cudaError_t alloc_err = cudaHostAlloc(&data_ptr, alloc_nbytes, cudaHostAllocPortable | cudaHostAllocMapped);+ if (alloc_err != cudaSuccess) {+ // callers may catch the refusal and go on: leave no error for the next launch+ (void)cudaGetLastError();+ }TORCH_CHECK(alloc_err == cudaSuccess,"cudaHostAlloc failed: ", cudaGetErrorString(alloc_err));@@ -98,10 +102,20 @@ void host_register(int64_t addr, int64_t nbytes) {const cudaError_t err =cudaHostRegister(reinterpret_cast<void *>(addr), static_cast<size_t>(nbytes),cudaHostRegisterPortable | cudaHostRegisterMapped);+ if (err != cudaSuccess) {+ // a refused pin is expected (the pin probe), so clear it before a later launch check reports it+ (void)cudaGetLastError();+ }TORCH_CHECK(err == cudaSuccess,"cudaHostRegister failed: ", cudaGetErrorString(err));}+void host_unregister(int64_t addr) {+ const cudaError_t err = cudaHostUnregister(reinterpret_cast<void *>(addr));+ TORCH_CHECK(err == cudaSuccess,+ "cudaHostUnregister failed: ", cudaGetErrorString(err));+}+int64_t driver_cuda_version() {int version = 0; // stays 0 when no driver is installedconst cudaError_t err = cudaDriverGetVersion(&version);@@ -123,6 +137,8 @@ PYBIND11_MODULE(TORCH_EXTENSION_NAME, m) {"Device-visible alias of a pinned+mapped host address");m.def("host_register", &host_register,"cudaHostRegister an existing host range as portable+mapped");+ m.def("host_unregister", &host_unregister,+ "cudaHostUnregister a range host_register pinned");m.def("driver_cuda_version", &driver_cuda_version,"Max CUDA version the installed NVIDIA driver supports (0 if none)");}diff --git a/python/freetoken/kernel/fast_index_copy.py b/python/freetoken/kernel/fast_index_copy.pyindex b2030620..38be995f 100644--- a/python/freetoken/kernel/fast_index_copy.py+++ b/python/freetoken/kernel/fast_index_copy.py@@ -172,11 +172,12 @@ def fast_index_copy_multi_jit(src_ptrs: torch.Tensor,feat_bytes: torch.Tensor,dst_indices: torch.Tensor,- src_indices: torch.Tensor,+ src_indices: torch.Tensor | None,num_indices: torch.Tensor | None = None,*,num_threads: int = 1024,blocks_per_bank: int = 8,+ wait_flag: int = 0,) -> None:"""Fused multi-bank index copy: copy the same rows for every bank in ONE launch.@@ -193,13 +194,16 @@ def fast_index_copy_multi_jit(once by the caller (the per-bank slot-cache base addr, host-source base addr, andper-row byte size). Every bank's per-row byte size must be a multiple of 16, and thebase addresses 16-byte aligned (true for contiguous torch allocations of these banks).++ ``src_indices`` None copies source row i to ``dst_indices[i]``. ``wait_flag``, a GPU-visible+ int64 address, makes every block first wait until it holds >= 1 (rows a CPU thread writes)."""if _skip_fast_index_copy_enabled():returnmodule = _jit_fast_index_copy_multi_module(num_threads=num_threads, blocks_per_bank=blocks_per_bank)- module.launch(dst_ptrs, src_ptrs, feat_bytes, dst_indices, src_indices, num_indices)+ module.launch(dst_ptrs, src_ptrs, feat_bytes, dst_indices, src_indices, num_indices, wait_flag)def update_copy_flag_jit(sync_flag: torch.Tensor, delta: int) -> None:diff --git a/python/freetoken/kernel/pinned.py b/python/freetoken/kernel/pinned.pyindex a21e78ac..897e5d6f 100644--- a/python/freetoken/kernel/pinned.py+++ b/python/freetoken/kernel/pinned.py@@ -50,6 +50,11 @@ def host_register(addr: int, nbytes: int) -> None:_load_pinned_extension().host_register(addr, nbytes)+def host_unregister(addr: int) -> None:+ """cudaHostUnregister a range :func:`host_register` pinned."""+ _load_pinned_extension().host_unregister(addr)++@lru_cache(maxsize=1)def _host_ptr_identity() -> bool:# cached per process: FreeToken pins one CUDA device per process (set at engine launch)diff --git a/python/freetoken/moe/bench_profile.py b/python/freetoken/moe/bench_profile.pyindex c939a947..319655cd 100644--- a/python/freetoken/moe/bench_profile.py+++ b/python/freetoken/moe/bench_profile.py@@ -128,6 +128,19 @@ def fetch_fraction(row: dict | None) -> float | None:return None+def locked_fetch_fraction(row: dict | None) -> float | None:+ """The fetch split of a locked layer of this row's kind: p / (2p + c) from the same pair as ``fetch_fraction``.++ A locked fetch is a CPU memcpy into staging, then the PCIe gather; taking the memcpy rate as the CPU MoE rate c, the two steps in series run at p*c / (p + c), which against the CPU compute gives p / (2p + c)."""+ if not isinstance(row, dict):+ return None+ for cpu_key, pcie_key in (("cpu_moe_overlap_gbs", "pcie_gather_overlap_gbs"), ("cpu_moe_gbs", "pcie_gather_gbs")):+ cpu, pcie = row.get(cpu_key), row.get(pcie_key)+ if cpu and pcie:+ return pcie / (2 * pcie + cpu)+ return None++def load_layer_kinds(gpu_name: str | None = None, path: str | None = None, gpu_uuid: str | None = None) -> dict | None:"""The rows of the usable profile for this GPU (key -> row), or None without one."""prof = _usable_profile(gpu_name, path, gpu_uuid)diff --git a/python/freetoken/moe/expert_banks.py b/python/freetoken/moe/expert_banks.pyindex b79d2860..dcb79797 100644--- a/python/freetoken/moe/expert_banks.py+++ b/python/freetoken/moe/expert_banks.py@@ -54,6 +54,8 @@ class ExpertBanks:row_tail_bytes: dict[str, list[int]] | None = None# per-layer quant_format of experts in several formats (``quant_format``/``kind``/``kernel`` then name the most common one); None = every layer is ``quant_format``layer_formats: list[str] | None = None+ # the residency plan holding DEFERRED layers' banks, for host_banks.probe_pin; None when nothing was deferred+ deferred_plan: object | None = Nonedef _dummy_fill(role: str, tensor: torch.Tensor) -> None:@@ -439,6 +441,45 @@ def load_expert_banks(return _echo_residency(banks, layer_residency, residency_plan)+def pin_probe_enabled(config, mixed_formats: bool) -> bool:+ """Whether ``--moe-cpu-layers auto`` finds the pin limit by pinning until refused.++ Only where pinning is known to be capped; FREETOKEN_PIN_BUDGET_GB fixes the limit instead (then a failed pin is an error)."""+ from freetoken.checkpoint.ftw import is_ftw_checkpoint+ from freetoken.moe.host_banks import pin_capped++ return (+ (config.moe_cpu_layers or "").strip() == "auto"+ and pin_capped()+ and not os.environ.get("FREETOKEN_PIN_BUDGET_GB")+ and config.moe_strategy in ("offload", "hybrid")+ and not mixed_formats+ and not config.use_dummy_weight+ and not (config.model_path and is_ftw_checkpoint(config.model_path)) # FTW settles its banks itself+ )+++def probe_banks(banks: ExpertBanks, num_layers: int) -> tuple[ExpertBanks, frozenset[int]]:+ """Settle DEFERRED banks with host_banks.probe_pin; returns the banks and the OS-locked layers."""+ import dataclasses+ import time++ from freetoken.moe.host_banks import HostResidency, probe_pin++ plan = banks.deferred_plan+ if plan is None:+ return banks, frozenset() # the loader settled the banks itself (pinned)+ t0 = time.perf_counter()+ labels = probe_pin(plan, num_layers)+ locked = frozenset(i for i, r in enumerate(labels) if r != HostResidency.PINNED.value)+ pinned_bytes = sum(b.nbytes for l, bs in plan.deferred.items() if l not in locked for b in bs)+ logger.info_rank0(+ f"--moe-cpu-layers auto (pin probe): pinned {num_layers - len(locked)} MoE layers ({pinned_bytes / 2**30:.2f} GiB) "+ f"in {time.perf_counter() - t0:.1f}s, locked {len(locked)} {sorted(locked)}"+ )+ return dataclasses.replace(banks, layer_residency=labels, deferred_plan=None), locked++def _echo_residency(banks: ExpertBanks, requested, plan) -> ExpertBanks:"""Stamp an honored residency request onto the ExpertBanks; keep None (and warn) when no settle point consulted the plan."""if requested is None or banks.layer_residency is not None:@@ -446,6 +487,8 @@ def _echo_residency(banks: ExpertBanks, requested, plan) -> ExpertBanks:if plan is not None and plan.applied:import dataclasses+ if plan.deferred:+ return dataclasses.replace(banks, layer_residency=list(requested), deferred_plan=plan)labels = [plan.actual.get(i, r) for i, r in enumerate(requested)]downgraded = [i for i, r in enumerate(requested) if labels[i] != r]if downgraded:diff --git a/python/freetoken/moe/host_banks.py b/python/freetoken/moe/host_banks.pyindex 95562a88..3cdc8d77 100644--- a/python/freetoken/moe/host_banks.py+++ b/python/freetoken/moe/host_banks.py@@ -23,6 +23,7 @@ import mathimport mmapimport osimport queue+import sysimport threadingfrom concurrent.futures import ThreadPoolExecutorfrom enum import Enum@@ -50,6 +51,34 @@ class HostResidency(str, Enum):PINNED = "pinned"LOCKED = "locked"PAGEABLE = "pageable"+ # filled but not settled: probe_pin settles the layer after the load (--moe-cpu-layers auto finds the pin limit)+ DEFERRED = "deferred"+++# FREETOKEN_SIMULATE_PIN_CAP_GB fails pins past that many GiB, like a driver pin quota (WSL/WDDM), to exercise the pin probe on hosts without one+_sim_pinned = 0+++def _sim_pin_cap() -> int | None:+ value = os.environ.get("FREETOKEN_SIMULATE_PIN_CAP_GB")+ return int(float(value) * 2**30) if value else None+++def charge_pin(nbytes: int) -> None:+ """Count ``nbytes`` against the simulated pin cap; raise PinFailed past it (no-op without the cap)."""+ global _sim_pinned+ cap = _sim_pin_cap()+ if cap is None:+ return+ if _sim_pinned + nbytes > cap:+ raise PinFailed(f"simulated pin cap {cap / 2**30:.1f} GiB reached")+ _sim_pinned += nbytes+++def refund_pin(nbytes: int) -> None:+ global _sim_pinned+ if _sim_pin_cap() is not None:+ _sim_pinned -= nbytes_DEFAULT_CHUNK = 8 << 20@@ -137,12 +166,24 @@ class HostBank:returnfrom freetoken.kernel.pinned import host_register+ charge_pin(len(self._buf))try:host_register(self.addr, len(self._buf))except RuntimeError as exc:+ refund_pin(len(self._buf))raise PinFailed(f"cudaHostRegister failed for {len(self._buf) / 2**30:.1f} GiB") from excself._pinned = True+ def unpin(self) -> None:+ """Undo ``pin()`` on an mmap bank; its pages stay resident (born-pinned banks cannot unpin)."""+ if not self._pinned or not isinstance(self._buf, mmap.mmap):+ return+ from freetoken.kernel.pinned import host_unregister++ host_unregister(self.addr)+ refund_pin(len(self._buf))+ self._pinned = False+def release(self) -> None:"""Drop the resident pages; the address space stays valid, the contents become undefined.@@ -152,7 +193,7 @@ class HostBank:self._buf.madvise(mmap.MADV_DONTNEED)def lock(self) -> None:- """mlock the (now-filled) buffer: resident without CUDA pin quota, but no device address -- only the CPU executor can serve a locked layer.+ """mlock the (now-filled) buffer: resident without CUDA pin quota, but with no device address, so the GPU cannot gather its rows.Lock after fill, or the lazy mmap faults+zero-fills every page. A failed lock (RLIMIT_MEMLOCK) warns once and leaves the bank PAGEABLE, which every consumer treats the same."""if self._locked or self._pinned: # cudaHostRegister already page-locks@@ -209,13 +250,15 @@ class _ResidencyPlan:Installed by ``load_expert_banks`` around the provider dispatch so every loader honors --moe-cpu-layers without a new parameter in each signature. ``applied`` flips once a settle point consults the plan."""- __slots__ = ("labels", "applied", "has_unpinned", "actual")+ __slots__ = ("labels", "applied", "has_unpinned", "actual", "deferred")def __init__(self, labels: list[str]):self.labels = list(labels)self.applied = Falseself.has_unpinned = any(r != HostResidency.PINNED.value for r in labels)self.actual: dict[int, str] = {}+ # DEFERRED layers' filled banks, for probe_pin+ self.deferred: dict[int, list[HostBank]] = {}def residency_for(self, layer_id: int) -> str:self.applied = True@@ -264,6 +307,9 @@ def pin_banks(banks: dict[str, HostBank | list[HostBank] | dict[int, HostBank]])HostResidency.PINNED.value if plan is Noneelse plan.residency_for(layer_id))+ if residency == HostResidency.DEFERRED.value:+ plan.deferred.setdefault(layer_id, []).append(layer_bank)+ continue_settle(layer_bank, residency)if plan is not None and residency == HostResidency.LOCKED.value:plan.record(layer_id, layer_bank.residency.value)@@ -271,6 +317,61 @@ def pin_banks(banks: dict[str, HostBank | list[HostBank] | dict[int, HostBank]])bank.pin()+def _wddm() -> bool:+ return sys.platform == "win32" or (hasattr(os, "uname") and "microsoft" in os.uname().release.lower())+++def pin_capped() -> bool:+ """Whether CUDA pinning is known to be capped: WDDM (WSL and native Windows), or FREETOKEN_SIMULATE_PIN_CAP_GB."""+ return _wddm() or _sim_pin_cap() is not None+++def _probe_margin() -> int:+ """FREETOKEN_PIN_PROBE_MARGIN_GB, else 3 GiB under WDDM (WSL and native Windows)."""+ return int(float(os.environ.get("FREETOKEN_PIN_PROBE_MARGIN_GB", "3" if _wddm() else "0")) * 2**30)+++def probe_pin(plan: _ResidencyPlan, num_layers: int, margin_bytes: int | None = None) -> list[str]:+ """Settle the DEFERRED layers: pin middle layers first until a pin fails, OS-lock the rest; returns each layer's residency.++ After a refusal, the last pinned layers give back at least ``margin_bytes`` (default ``_probe_margin()``)."""+ if margin_bytes is None:+ margin_bytes = _probe_margin()+ # per-layer decode miss rates are U-shaped, so the head and tail layers are the ones left unpinned+ order = sorted(plan.deferred, key=lambda l: (abs(l - (num_layers - 1) / 2), l))+ pinned: list[int] = []+ refused = None+ for layer in order:+ banks = plan.deferred[layer]+ try:+ for bank in banks:+ bank.pin()+ except PinFailed as exc:+ for bank in banks:+ bank.unpin()+ refused = exc+ break+ pinned.append(layer)+ if refused is not None:+ freed = 0+ while pinned and freed < margin_bytes:+ layer = pinned.pop()+ for bank in plan.deferred[layer]:+ freed += len(bank._buf) if bank._pinned else 0+ bank.unpin()+ labels = [HostResidency.PINNED.value] * num_layers+ kept = set(pinned)+ for layer, banks in plan.deferred.items():+ if layer in kept:+ continue+ for bank in banks:+ bank.lock()+ labels[layer] = (+ HostResidency.LOCKED.value if all(b.residency is HostResidency.LOCKED for b in banks) else HostResidency.PAGEABLE.value+ )+ return labels++class PinPipeline:"""Settle (pin or lock) filled banks while other banks are still being read.@@ -297,6 +398,9 @@ class PinPipeline:if self._exc is not None:continue # drain without settling after a failurebank, residency, plan, layer_id = item+ if residency == HostResidency.DEFERRED.value:+ plan.deferred.setdefault(layer_id, []).append(bank)+ continuetry:_settle(bank, residency)if plan is not None and residency == HostResidency.LOCKED.value:diff --git a/python/freetoken/moe/locked_fetch.py b/python/freetoken/moe/locked_fetch.pynew file mode 100644index 00000000..b2e4331e--- /dev/null+++ b/python/freetoken/moe/locked_fetch.py@@ -0,0 +1,235 @@+"""Locked-layer fetch for hybrid decode.++A locked layer's host banks are OS-locked but not registered with CUDA (the pin quota ran out), so the GPU cannot+gather their rows. Each decode step a publish kernel hands the fetch list chosen by the hybrid ensure to CPU threads+(``_cpu_moe.LockedFetch``), which copy each fetched row of every bank into pinned staging, and fast_index_copy_multi+copies staging row i into ``evict_slots[i]``. The misses it does not fetch go to the CPU executor as on any hybrid layer.+"""++from __future__ import annotations++import collections+import os++import torch++from freetoken.utils import init_logger++logger = init_logger(__name__)+++def locked_fetch_wanted(config) -> bool:+ """Whether locked layers fetch through pinned staging: hybrid, --moe-locked-fetch not 0, CUDA (the copy kernels are untested on ROCm)."""+ return config.moe_strategy == "hybrid" and config.moe_locked_fetch != 0 and torch.version.hip is None+++def check_locked_fetch_config(config) -> None:+ """Reject --moe-locked-max-fetch past the experts of a layer before the banks are read (only with --moe-cpu-layers)."""+ num_experts = config.model_config.num_experts+ if config.moe_cpu_layers and locked_fetch_wanted(config) and not 1 <= config.moe_locked_max_fetch <= num_experts:+ raise ValueError(f"--moe-locked-max-fetch {config.moe_locked_max_fetch} must be in [1, {num_experts}] (the experts of a layer)")+++def staging_reserve(config, bank_bytes) -> int:+ """Pin budget the staging takes when --moe-cpu-layers is settled before the load: ``max_fetch`` rows of twice the mean expert.++ ``bank_bytes()`` estimates the banks; it runs only when the staging is wanted."""+ if not (config.moe_cpu_layers and locked_fetch_wanted(config)) or not (total := bank_bytes()):+ return 0+ mc = config.model_config+ return 2 * (total // (mc.num_moe_layers * mc.num_experts)) * config.moe_locked_max_fetch+++def settle_banks(config, banks, *, probe: bool, has_locked: bool):+ """Pin the staging when locked layers may fetch, then with ``probe`` settle the DEFERRED banks (expert_banks.probe_banks).++ Pinning the staging first keeps the banks from taking its share of the pin budget. Returns the banks, the locked layers+ the probe found (None without ``probe``) and the staging (None when unused); a staging that cannot be pinned raises."""+ from freetoken.moe.expert_banks import probe_banks++ staging = None+ if locked_fetch_wanted(config) and (probe or has_locked):+ staging = pin_staging(banks.sources, config.moe_locked_max_fetch)+ locked = None+ if probe:+ banks, locked = probe_banks(banks, config.model_config.num_moe_layers)+ if staging is not None and not locked:+ refund_staging(staging) # every bank got pinned+ staging = None+ return banks, locked, staging+++def locked_pool_tags(layer_tags: list | None, locked, num_layers: int) -> list:+ """Bank layer tags with the locked layers marked, so they group into slot pools of their own."""+ return [(t, "locked") if i in locked else t for i, t in enumerate(layer_tags or [None] * num_layers)]+++def attach_locked_fetch(cache, config, staging: dict[str, torch.Tensor], layer_rows) -> None:+ """Set ``cache.locked_fetch`` for ``cache.locked_hybrid``; needs the cache's CPU executor. ``layer_rows`` as in fetch_plan."""+ locked = sorted(cache.locked_hybrid)+ plan = fetch_plan(config.moe_locked_fetch, locked, layer_rows)+ cache.locked_fetch = LockedLayerFetch(cache, locked, plan, config.moe_locked_max_fetch, cache.cpu_executor, staging=staging)+++def staging_bytes(sources: dict[str, list[torch.Tensor]], max_fetch: int) -> dict[str, int]:+ """Per canonical bank role: ``max_fetch`` rows of the bank's widest row over every layer."""+ from freetoken.moe.legacy_format import canonical_role++ out = {}+ for name, per_layer in sources.items():+ rows = [t[0].numel() * t.element_size() for t in per_layer if t.numel()]+ out[canonical_role(name)] = max([16] + [r * max_fetch for r in rows])+ return out+++def alloc_staging(sizes: dict[str, int]) -> dict[str, torch.Tensor]:+ """Pinned staging per bank role."""+ from freetoken.kernel.pinned import alloc_pinned_tensor+ from freetoken.moe.host_banks import charge_pin, refund_pin++ charge_pin(sum(sizes.values()))+ try:+ return {role: alloc_pinned_tensor(n, dtype=torch.uint8) for role, n in sizes.items()}+ except BaseException:+ refund_pin(sum(sizes.values()))+ raise+++def pin_staging(sources: dict[str, list[torch.Tensor]], max_fetch: int) -> dict[str, torch.Tensor]:+ """The staging for ``sources``; raises when it cannot be pinned."""+ from freetoken.moe.host_banks import PinFailed++ sizes = staging_bytes(sources, max_fetch)+ try:+ return alloc_staging(sizes)+ except (PinFailed, RuntimeError) as exc:+ raise RuntimeError(+ f"could not pin the {sum(sizes.values()) / 2**20:.1f} MiB locked-layer staging ({exc}); free pinned host memory, "+ "lower --moe-locked-max-fetch, or pass --moe-locked-fetch 0 to keep locked layers CPU-only"+ ) from exc+++def refund_staging(staging: dict[str, torch.Tensor]) -> None:+ """Give the staging's bytes back to the simulated pin cap; the memory itself goes with its last reference."""+ from freetoken.moe.host_banks import refund_pin++ refund_pin(sum(t.numel() for t in staging.values()))+++def fetch_plan(share: float, layers: list[int], layer_rows) -> dict[int, tuple[int, float]]:+ """(max_fetch, fraction) per locked layer for the hybrid ensure.++ ``share`` > 0 fetches that share of every step's misses; otherwise each layer takes p / (2p + c) from its `ft bench bw`+ row, or a fixed cap of 1 without one. ``layer_rows()`` returns (row key, label) per MoE layer and the profile's rows, or None.+ """+ if share > 0:+ logger.info_rank0(f"--moe-locked-fetch {share:g}: locked layers fetch this share of each step's misses")+ return {layer: (0, share) for layer in layers}+ from freetoken.moe.bench_profile import locked_fetch_fraction++ rows = layer_rows()+ keys, labels, rows = rows if rows is not None else ([None] * (max(layers) + 1), [""] * (max(layers) + 1), {})+ fractions = {layer: locked_fetch_fraction(rows.get(keys[layer])) for layer in layers}+ groups = collections.Counter((labels[layer], f) for layer, f in fractions.items())+ if fetched := [f"{f:.1%} on {n} {label} layers" for (label, f), n in groups.items() if f is not None]:+ logger.info_rank0(f"--moe-locked-fetch auto: locked layers fetch this share of each step's misses (benched p / (2p + c)): {'; '.join(fetched)}")+ if capped := [layer for layer, f in fractions.items() if f is None]:+ logger.warning_rank0(+ f"--moe-locked-fetch auto: no `ft bench bw` row for locked layers {capped}; they fetch at most 1 expert per step. "+ "`ft bench bw --model <checkpoint>` benches each layer kind"+ )+ return {layer: (0, f) if f is not None else (1, 0.0) for layer, f in fractions.items()}+++def _smt_siblings(cpus: list[int]) -> list[int]:+ out = []+ for cpu in cpus:+ try:+ with open(f"/sys/devices/system/cpu/cpu{cpu}/topology/thread_siblings_list") as f:+ text = f.read().strip()+ except OSError:+ continue+ for part in text.split(","):+ lo, _, hi = part.partition("-")+ out.extend(i for i in range(int(lo), int(hi or lo) + 1) if i != cpu)+ return out+++def _copy_threads() -> int:+ """FREETOKEN_LOCKED_FETCH_THREADS, else max(2, min(4, logical CPUs / 4)): a few threads already saturate memcpy."""+ if env := os.environ.get("FREETOKEN_LOCKED_FETCH_THREADS"):+ return max(1, int(env))+ logical = len(os.sched_getaffinity(0)) if hasattr(os, "sched_getaffinity") else (os.cpu_count() or 1)+ return max(2, min(4, logical // 4))+++def _copy_cpus(executor, count: int) -> list[int]:+ """CPUs for the copy threads: SMT siblings of the executor's workers, else physical cores the executor leaves free."""+ from freetoken.moe.cpu_executor import physical_core_cpus++ busy = set(executor.core_ids) | {getattr(executor, "_coord_core", -1)}+ cpus = [c for c in dict.fromkeys(_smt_siblings(executor.core_ids)) if c not in busy]+ cpus = cpus or [c for c in physical_core_cpus() if c not in busy]+ return cpus[:count]+++class LockedLayerFetch:+ """Staging and copy threads for the locked layers of one ``OffloadMoeCache``."""++ def __init__(self, cache, layers: list[int], plan: dict[int, tuple[int, float]], max_fetch: int, executor, staging: dict[str, torch.Tensor] | None = None) -> None:+ from freetoken.kernel import _cpu_moe+ from freetoken.kernel.pinned import device_ptr+ from freetoken.moe.legacy_format import canonical_role++ if not cache._copy_fused_ok:+ raise ValueError("locked layers fetch through the fused multi-bank copy, which this cache could not plan (bank alignment or FREETOKEN_FUSED_COPY=0)")+ # (max_fetch, fraction) of each locked layer for the hybrid ensure; max_fetch caps any of them+ self.plan = plan+ self.max_fetch = max_fetch+ roles = [canonical_role(name) for name in cache.bank_schema]+ nbanks = len(cache.banks)+ src = [[0] * nbanks for _ in range(cache.num_layers)]+ row_bytes = [[0] * nbanks for _ in range(cache.num_layers)]+ need = [16] * nbanks+ for b, (per_layer, _) in enumerate(cache.banks):+ for layer in layers:+ rb = int(cache._row_bytes[cache.layer_layout[layer]][b])+ if per_layer[layer].numel() == 0 or rb == 0:+ continue+ src[layer][b] = per_layer[layer].data_ptr()+ row_bytes[layer][b] = rb+ need[b] = max(need[b], rb * max_fetch)+ if staging is None:+ staging = alloc_staging(dict(zip(roles, need)))+ for b, role in enumerate(roles):+ assert staging[role].numel() >= need[b], (role, staging[role].numel(), need[b])+ self.staging = [staging[role] for role in roles]+ self.staging_ptrs = torch.tensor([device_ptr(t) for t in self.staging], dtype=torch.int64, device=cache.device)+ cpus = _copy_cpus(executor, _copy_threads())+ # the executor's doorbell decision (FREETOKEN_CPU_MOE_FLAG_SYNC and the memop probe); without memops the gather spins on the GPU+ memops = bool(getattr(executor, "_flag_sync", False))+ self._ext = _cpu_moe.LockedFetch(src, row_bytes, [t.data_ptr() for t in self.staging], max_fetch, cpus, memops)+ self._wait_flag = 0 if memops else self._ext.done_word()+ logger.info_rank0(+ f"locked layers {layers}: GPU slots + fetch up to {max_fetch} misses per step through "+ f"{sum(t.numel() for t in self.staging) / 2**20:.1f} MiB of pinned staging, {max(1, len(cpus))} copy thread(s) on cpus {cpus or 'any'}"+ + ("" if memops else ", the GPU spin-waits on them (no stream memops)")+ )++ def copy(self, cache, layer_id: int) -> None:+ """Stage and gather the rows the last hybrid ensure scheduled for ``layer_id``; graph capturable."""+ from freetoken.kernel.fast_index_copy import fast_index_copy_multi_jit++ # the gather reads num_indices rows of the staging, which has max_fetch+ assert cache._pending_fetch_cap == self.max_fetch, (cache._pending_fetch_cap, self.max_fetch)+ self._ext.publish(torch.cuda.current_stream().cuda_stream, layer_id, cache.num_indices.data_ptr(), cache.src_indices.data_ptr())+ fast_index_copy_multi_jit(+ cache._copy_dst_ptrs_pool[cache.pool_of_layer[layer_id]],+ self.staging_ptrs,+ cache._copy_feat_bytes[cache.layer_layout[layer_id]],+ cache.evict_slots,+ None,+ cache.num_indices,+ blocks_per_bank=cache._copy_blocks_per_bank,+ wait_flag=self._wait_flag,+ )diff --git a/python/freetoken/moe/offload_cache.py b/python/freetoken/moe/offload_cache.pyindex 0d702d8a..f555be96 100644--- a/python/freetoken/moe/offload_cache.py+++ b/python/freetoken/moe/offload_cache.py@@ -194,6 +194,9 @@ class OffloadMoeCache:# offload/PCIe path. Set by the engine after construction (empty = all-GPU,# all layers = the plain --moe-strategy cpu case).self.cpu_layer_ids: frozenset = frozenset()+ # hybrid: locked (unpinned) layers that keep slots of their own pools and fetch part of their misses through locked_fetch's pinned staging+ self.locked_hybrid: frozenset = frozenset()+ self.locked_fetch = None # moe.locked_fetch.LockedLayerFetch, set by the engine with locked_hybrid# Bank row layouts, set by set_bank_sources: layouts[g][k] is (dtype, row shape) of bank k (layouts of two quant tags may be equal), layer_layout[l] is layer l's g.# With one (dtype, row shape) per bank a bank cache is typed [cache_size, *row]; with several, or with pads, it is a byte segment each pool views with its own layout.self.layouts: list[tuple[tuple[torch.dtype, tuple[int, ...]], ...]] = []@@ -256,7 +259,7 @@ class OffloadMoeCache:self.bank_schema = _BANK_SCHEMAS[self.quant_format]self.bank_sources: dict[str, list[torch.Tensor]] = {}self.bank_caches: dict[str, torch.Tensor] = {}- # per-layer host residency: the GPU movement paths require "pinned"; LOCKED/PAGEABLE layers decode on the CPU executor and prefill via copy_missing's pageable branch+ # per-layer host residency: the GPU movement paths require "pinned"; LOCKED/PAGEABLE layers decode on the CPU executor (locked_hybrid ones also from their slots) and prefill via copy_missing's pageable branch# _unpinned_layers is the derived id set the hot paths test againstself.layer_residency: list[str] = []self._unpinned_layers: frozenset = frozenset()@@ -314,6 +317,7 @@ class OffloadMoeCache:# _pending_whole_layer records WHICH staged it: the pageable branch is only sound after materialize_layerself._pending_src_layer: int | None = Noneself._pending_whole_layer = False+ self._pending_fetch_cap: int | None = None # the last hybrid ensure's cap on its fetch count# The two prefill staging buffers are the first 2 * _stage_bytes[k] bytes of every bank, on top of the first pools.# _stage_views[g][b] are buffer b's per-bank [num_experts, ...] views for layout g; _inval[b] are the global slots it overwrites.self._stage_bytes: list[int] = []@@ -468,7 +472,7 @@ class OffloadMoeCache:-- the cache machinery is layout-agnostic and just moves rows.``layer_residency`` labels each layer with a ``HostResidency`` value (default: all pinned).- Non-pinned (LOCKED/PAGEABLE) layers have no device address: they must already be routed to the CPU executor (``cpu_layer_ids``, set BEFORE this call), the copy plan skips their rows, and their only movement is ``copy_missing``'s whole-layer pageable prefill branch -- which is why prefill overlap is incompatible with them.+ Non-pinned (LOCKED/PAGEABLE) layers have no device address: they must be in ``cpu_layer_ids`` (set BEFORE this call) and the copy plan skips their rows. Prefill moves them with ``copy_missing``'s whole-layer pageable branch, so prefill overlap is incompatible with them; ``locked_hybrid`` layers also fetch decode misses through ``locked_fetch``.``layer_tags`` names each layer's quant (see :func:`bank_layouts`); layers of different tags never share a pool.@@ -784,11 +788,11 @@ class OffloadMoeCache:def is_cpu_layer(self, layer_id: int) -> bool:"""Whether ``layer_id`` decodes on the CPU executor (vs the GPU offload path)."""- return layer_id in self.cpu_layer_ids+ return layer_id in self.cpu_layer_ids and layer_id not in self.locked_hybriddef is_unpinned_layer(self, layer_id: int) -> bool:"""Whether ``layer_id``'s host banks have no device address (LOCKED/PAGEABLE): the GPU slot-gather paths cannot serve it.- ``copy_missing`` takes the whole-layer pageable branch, which presumes materialize's position == expert id (never ``ensure_experts``'s LRU slot remap)."""+ ``copy_missing`` takes the whole-layer pageable branch, which presumes materialize's position == expert id; a locked_hybrid layer's decode fetches go through ``locked_fetch`` instead."""return layer_id in self._unpinned_layersdef alphas_for_slots(self, layer_id: int) -> tuple[torch.Tensor, torch.Tensor] | None:@@ -1096,7 +1100,12 @@ class OffloadMoeCache:self._pending_src_layer = layer_idself._pending_whole_layer = Falsefraction = self.hybrid_fetch_fraction[layer_id] if self.hybrid_fetch_fraction is not None else 0.0- return ensure_experts_hybrid(self, layer_id, expert_ids, weights, self.hybrid_max_fetch, fraction, clamp)+ max_fetch, cap = self.hybrid_max_fetch, None+ if layer_id in self.locked_hybrid:+ # a locked layer fetches through the staging, so at most its rows per step+ (max_fetch, fraction), cap = self.locked_fetch.plan[layer_id], self.locked_fetch.max_fetch+ self._pending_fetch_cap = cap+ return ensure_experts_hybrid(self, layer_id, expert_ids, weights, max_fetch, fraction, clamp, cap)def materialize_layer(self, layer_id: int) -> None:from freetoken.moe.offload_kernels import materialize_layer@@ -1156,6 +1165,29 @@ class OffloadMoeCache:"prefill_rows": self.prefill_total_rows,}+ def decode_stats_line(self) -> str | None:+ """Decode stats since the last call by layer group (pinned, locked with slots, CPU-only), as one log line; zeroes the counters."""+ if self.decode_target == "hybrid":+ rows = torch.stack([self.stat_steps_layer, self.stat_active_layer, self.stat_missing_layer, self.stat_fetched_layer], 1).tolist()+ else:+ # offload copies every miss+ cols = self.lru_stats.t()+ rows = torch.stack([cols[Stat.CALLS], cols[Stat.ACTIVE], cols[Stat.MISS], cols[Stat.MISS]], 1).tolist()+ self.reset_stats()+ groups = (("pinned", [l for l in range(self.num_layers) if l not in self.cpu_layer_ids]), ("locked", sorted(self.locked_hybrid)))+ parts = []+ for name, layers in groups:+ steps, active, missing, fetched = (sum(rows[l][k] for l in layers) for k in range(4))+ if steps and active:+ parts.append(+ f"{name} {_layers(len(layers))}: hit {1 - missing / active:.1%}, per layer-step "+ f"miss {missing / steps:.2f} fetch {fetched / steps:.2f} cpu {(missing - fetched) / steps:.2f}"+ )+ cpu_only = self.cpu_layer_ids - self.locked_hybrid+ if cpu_only:+ parts.append(f"cpu-only {_layers(len(cpu_only))}")+ return "MoE decode, " + "; ".join(parts) if parts else None+def decode_miss_stats_per_layer(self) -> dict:"""Per-MoE-layer realized decode stats for one (reset_stats-delimited) window.@@ -1224,13 +1256,16 @@ class OffloadMoeCache:assert layer_id is not None, "no staged misses (ensure_experts/materialize_layer first)"if layer_id in self._unpinned_layers:if not self._pending_whole_layer:+ if layer_id in self.locked_hybrid:+ self.locked_fetch.copy(self, layer_id)+ returnraise RuntimeError(f"layer {layer_id} is unpinned: its only copy is the whole-layer "f"pageable materialize (position == expert id); ensure_experts's "f"LRU slot remap cannot be honored without a device alias")# the only copy a non-pinned layer ever needs is the non-overlap prefill materialize, which schedules the whole layer into staging buffer 0 with position == expert id -- a plain synchronous pageable H2D copy- # never CUDA-graph captured: prefill is not captured, and decode never reaches this branch (it routes to the CPU executor)+ # never CUDA-graph captured: prefill is not captured, and decode of an unpinned layer routes to the CPU executor or, locked_hybrid, to the staged fetch abovefor (per_layer, _), view in zip(self.banks, self.stage_views(layer_id, 0)):view.copy_(per_layer[layer_id])return@@ -1269,6 +1304,10 @@ class OffloadMoeCache:)+def _layers(n: int) -> str:+ return f"{n} layer" if n == 1 else f"{n} layers"++def iter_offload_moe_layers(model) -> Iterator:from freetoken.layers import OffloadMoELayer, iter_moe_layersdiff --git a/python/freetoken/moe/offload_kernels.py b/python/freetoken/moe/offload_kernels.pyindex 8c4ad8f3..5b28c1be 100644--- a/python/freetoken/moe/offload_kernels.py+++ b/python/freetoken/moe/offload_kernels.py@@ -58,6 +58,7 @@ def ensure_experts_hybrid(max_fetch: int,fetch_fraction: float = 0.0,clamp: bool = False,+ fetch_cap: int | None = None,) -> HybridRoutes:"""Capped-fetch variant of ``ensure_experts`` (hybrid backend).@@ -71,13 +72,15 @@ def ensure_experts_hybrid(pre-cap miss count (stats).The same launch splits the routes: ``expert_ids`` becomes the GPU slots in place, and- ``clamp`` gives CPU routes slot 0 instead of -1 for GEMM kernels that read every slot."""+ ``clamp`` gives CPU routes slot 0 instead of -1 for GEMM kernels that read every slot.+ ``fetch_cap`` bounds the fetch count after ``max_fetch`` / ``fetch_fraction`` (a fixed-size staging buffer)."""assert expert_ids.is_cuda, "the hybrid ensure kernel runs on the GPU"# Q16 fixed point: the kernel caps in integers.frac_q16 = min(1 << 16, max(0, round(fetch_fraction * (1 << 16))))weights = weights.contiguous()routes = HybridRoutes(expert_ids, torch.empty_like(weights), torch.empty_like(expert_ids))- _ensure_experts_hybrid_gpu(cache, layer_id, expert_ids, weights, routes, clamp, max_fetch, frac_q16)+ cap = cache.num_experts if fetch_cap is None else fetch_cap+ _ensure_experts_hybrid_gpu(cache, layer_id, expert_ids, weights, routes, clamp, max_fetch, frac_q16, cap)return routes@@ -132,6 +135,7 @@ def _ensure_experts_hybrid_gpu(clamp: bool,max_fetch: int,frac_q16: int,+ fetch_cap: int,) -> None:id_of_slot, usage = cache.pool_state(layer_id)num_slots = id_of_slot.numel()@@ -158,6 +162,7 @@ def _ensure_experts_hybrid_gpu(expert_ids.numel(),int(max_fetch),int(frac_q16),+ int(fetch_cap),cache.num_experts,num_slots,BLOCK_E=block_e,@@ -268,7 +273,7 @@ def _materialize_layer_kernel([email protected](do_not_specialize=["layer_id", "num_active", "max_fetch", "fetch_frac_q16", "cache_size", "id_of_slot_ptr", "usage_ptr"])[email protected](do_not_specialize=["layer_id", "num_active", "max_fetch", "fetch_frac_q16", "fetch_cap", "cache_size", "id_of_slot_ptr", "usage_ptr"])def _ensure_experts_hybrid_kernel(expert_ids_ptr,slot_for_id_ptr,@@ -290,6 +295,7 @@ def _ensure_experts_hybrid_kernel(num_active,max_fetch,fetch_frac_q16,+ fetch_cap,num_experts: tl.constexpr,cache_size,BLOCK_E: tl.constexpr,@@ -342,7 +348,7 @@ def _ensure_experts_hybrid_kernel(max_fetch = tl.where(cost_lo <= cost_hi, lo, lo + 1)is_hit = is_active & (slot >= 0)# a pool smaller than num_experts may hold fewer free slots than misses- num_fetch = tl.minimum(tl.minimum(num_missing, max_fetch), cache_size - tl.sum(is_hit.to(tl.int32)))+ num_fetch = tl.minimum(tl.minimum(num_missing, tl.minimum(max_fetch, fetch_cap)), cache_size - tl.sum(is_hit.to(tl.int32)))tl.store(num_missing_full_ptr, num_missing.to(tl.int64))tl.store(num_indices_ptr, num_fetch.to(tl.int64))tl.store(usage_ptr + slot, step, mask=is_hit)diff --git a/python/freetoken/scheduler/scheduler.py b/python/freetoken/scheduler/scheduler.pyindex d057d3b9..220fa090 100644--- a/python/freetoken/scheduler/scheduler.py+++ b/python/freetoken/scheduler/scheduler.py@@ -143,14 +143,21 @@ class Scheduler(SchedulerIOMixin):self.config = configself._model_is_mrope = config.model_config.model_is_mropeself._warned_cut_image = False+ moe_cache = getattr(self.engine, "moe_offload_cache", None)self.status_reporter = SchedulerStatusReporter(log=logger.info_rank0,decode_log_interval=config.decode_log_interval,+ moe_stats=self._moe_stats_line if config.moe_collect_stats and moe_cache is not None else None,)# Initialize the I/O mixinsuper().__init__(config, self.engine.tp_cpu_group)+ def _moe_stats_line(self) -> str | None:+ # on the engine stream, so the read and the zeroing fall between decode replays+ with torch.cuda.stream(self.engine.stream):+ return self.engine.moe_offload_cache.decode_stats_line()+def run_when_idle(self) -> None:"""Called when the scheduler is idle to perform background tasks."""logger.info_rank0("Scheduler is idle, waiting for new reqs...")diff --git a/python/freetoken/scheduler/status.py b/python/freetoken/scheduler/status.pyindex ca706c50..76c20cd4 100644--- a/python/freetoken/scheduler/status.py+++ b/python/freetoken/scheduler/status.py@@ -12,6 +12,8 @@ class SchedulerStatusReporter:log: Callable[[str], None]clock: Callable[[], float] = time.perf_counterdecode_log_interval: int = 40+ # --moe-collect-stats: one more line per decode log line (the stats since the previous one), or None to skip+ moe_stats: Callable[[], str | None] | None = None_last_prefill_time: float = field(init=False)_last_decode_time: float = field(init=False)_decode_forward_count: int = field(default=0, init=False)@@ -122,6 +124,8 @@ class SchedulerStatusReporter:f"gen throughput (token/s): {gen_throughput:.2f}, "f"#queue-req: {queue_reqs}")+ if self.moe_stats is not None and (line := self.moe_stats()):+ self.log(line)def _usage_ratio(used: int, total: int) -> float:diff --git a/python/freetoken/server/args.py b/python/freetoken/server/args.pyindex 56d52ee7..d09534a8 100644--- a/python/freetoken/server/args.py+++ b/python/freetoken/server/args.py@@ -795,9 +795,11 @@ def parse_args("With --moe-strategy offload/hybrid: which MoE layers compute on the ""CPU executor instead of the GPU offload/PCIe path (where CUDA pinning ""is quota-capped, e.g. WSL, their banks are OS-locked instead of pinned). Explicit id list ('3,7,11'), a count ('8' = 8 "- "layers evenly strided), a fraction ('0.5'), or 'auto'. 'auto' is for Windows/WSL "- "only, where CUDA pinned memory is capped: it locks just enough head+tail layers "- "for the banks over the pin budget. Any value, 'auto' included, commits to CPU "+ "layers evenly strided), a fraction ('0.5'), or 'auto'. 'auto' is for hosts that cap CUDA "+ "pinned memory (Windows/WSL): it pins the banks layer by layer from the middle out until CUDA "+ "refuses one and OS-locks the rest. FREETOKEN_PIN_BUDGET_GB, FTW checkpoints and models the CPU executor "+ "cannot serve lock head+tail layers from a size estimate instead. Under hybrid, locked layers still decode from GPU slots "+ "(--moe-locked-fetch). Any value, 'auto' included, commits to CPU ""decode before the model is built, so the expert format must have a CPU executor ""path (bf16, fp8_block, nvfp4, mxfp4); do not pass it on Linux. Unset = every layer on the ""GPU; a boot whose banks exceed a known pin budget stops and asks for this flag."@@ -818,6 +820,29 @@ def parse_args(),)+ parser.add_argument(+ "--moe-locked-fetch",+ type=float,+ default=ServerArgs.moe_locked_fetch,+ help=(+ "For --moe-strategy hybrid with locked layers (banks over the pin quota, OS-locked instead of pinned): "+ "the share of a locked layer's misses fetched per decode step into GPU slots of its own pools, through "+ "pinned staging that CPU threads fill; the rest are computed on the CPU. -1 (default) = auto: per layer from "+ "the `ft bench bw` row of its kind, p / (2p + c) of the overlapped PCIe and CPU MoE rates (at most 1 expert "+ "per step without a row). 0 = locked layers decode on the CPU only."+ ),+ )++ parser.add_argument(+ "--moe-locked-max-fetch",+ type=int,+ default=ServerArgs.moe_locked_max_fetch,+ help=(+ "For --moe-locked-fetch: the most experts a locked layer fetches per decode step. Sizes the pinned "+ "staging: this many rows of the widest expert, per bank."+ ),+ )+parser.add_argument("--disable-moe-prefill-overlap",action="store_false",@@ -845,6 +870,18 @@ def parse_args(),)+ parser.add_argument(+ "--moe-collect-stats",+ action="store_true",+ default=ServerArgs.moe_collect_stats,+ help=(+ "With the offload-family strategies: after each decode log line, log the decode MoE stats since the "+ "previous one (hit rate, and per layer-step the misses, experts fetched to the GPU and experts computed "+ "on the CPU) for pinned layers, locked layers with GPU slots and CPU-only layers. Adds counter updates "+ "to every decode step."+ ),+ )+parser.add_argument("--moe-prefill-hit-d2d",action="store_true",diff --git a/setup.py b/setup.pyindex 070b8fd5..a836038b 100644--- a/setup.py+++ b/setup.py@@ -267,6 +267,8 @@ setup(name="freetoken.kernel._cpu_moe",sources=["python/freetoken/kernel/csrc/cpu_moe/cpu_moe_ext.cpp",+ "python/freetoken/kernel/csrc/cpu_moe/locked_fetch.cpp",+ "python/freetoken/kernel/csrc/cpu_moe/locked_fetch_io.cu",f"python/freetoken/kernel/csrc/cpu_moe/gpu_sync_{'hip' if IS_ROCM else 'cuda'}.cpp","python/freetoken/kernel/csrc/cpu_moe/gpu_sync_io.cu",*sorted(diff --git a/tests/moe/test_fused_copy.py b/tests/moe/test_fused_copy.pyindex 7a0c0864..54c42680 100644--- a/tests/moe/test_fused_copy.py+++ b/tests/moe/test_fused_copy.py@@ -5,6 +5,8 @@ zero-copy case), across banks of differing per-row sizes.from __future__ import annotations+import time+import pytestimport torch@@ -74,3 +76,34 @@ def test_fused_copy_matches_per_bank(num_indices):for b, (r, (_, c)) in enumerate(zip(ref, cache.banks)):assert torch.equal(r, c), f"bank {b} (feat={FEATS[b]}) fused != per-bank at num_indices={num_indices}"+++@CUDA+def test_fused_copy_identity_rows_and_wait_flag():+ # without src_indices row i copies source row i; with wait_flag the copy holds until the flag is set (the locked-layer staging)+ from freetoken.kernel.fast_index_copy import fast_index_copy_multi_jit+ from freetoken.kernel.pinned import alloc_pinned_tensor, device_ptr++ dev = torch.device("cuda")+ feats = [512, 256]+ src = [alloc_pinned_tensor(4 * f, dtype=torch.uint8) for f in feats]+ for t in src:+ t.random_(0, 256)+ dst = [torch.zeros(16 * f, dtype=torch.uint8, device=dev) for f in feats]+ dst_ptrs = torch.tensor([t.data_ptr() for t in dst], dtype=torch.int64, device=dev)+ src_ptrs = torch.tensor([device_ptr(t) for t in src], dtype=torch.int64, device=dev)+ feat_bytes = torch.tensor(feats, dtype=torch.int64, device=dev)+ slots = torch.tensor([5, 2, 9, 0], dtype=torch.int32, device=dev)+ num = torch.tensor([3], dtype=torch.int64, device=dev)+ flag = alloc_pinned_tensor(1, dtype=torch.int64)+ flag.fill_(0)++ fast_index_copy_multi_jit(dst_ptrs, src_ptrs, feat_bytes, slots, None, num, wait_flag=device_ptr(flag))+ time.sleep(0.2)+ assert not torch.cuda.current_stream().query(), "the copy ran before its flag was set"+ flag.fill_(1)+ torch.cuda.synchronize()+ for d, s, f in zip(dst, src, feats):+ for i, slot in enumerate([5, 2, 9]):+ assert torch.equal(d[slot * f:(slot + 1) * f].cpu(), s[i * f:(i + 1) * f])+ assert not d[0:f].any(), "num_indices bounds the rows"diff --git a/tests/moe/test_hybrid_fetch.py b/tests/moe/test_hybrid_fetch.pyindex 5f36ac7b..384a3524 100644--- a/tests/moe/test_hybrid_fetch.py+++ b/tests/moe/test_hybrid_fetch.py@@ -9,7 +9,7 @@ import osimport pytest-from freetoken.moe.bench_profile import PROFILE_VERSION, default_profile_path, fetch_fraction, layer_kind_key, load_layer_kinds+from freetoken.moe.bench_profile import PROFILE_VERSION, default_profile_path, fetch_fraction, layer_kind_key, load_layer_kinds, locked_fetch_fractiondef test_layer_kind_rows_give_fetch_fractions(tmp_path):@@ -32,6 +32,9 @@ def test_layer_kind_rows_give_fetch_fractions(tmp_path):assert fetch_fraction(rows[bf16]) == pytest.approx(0.4)# overlapped pair preferred: pcie_ov / (pcie_ov + cpu_ov)assert fetch_fraction(rows[gguf]) == pytest.approx(0.25)+ # a locked layer's fetch adds a memcpy at the CPU MoE rate before the gather: p / (2p + c), from the same pair+ assert locked_fetch_fraction(rows[bf16]) == pytest.approx(40 / 180)+ assert locked_fetch_fraction(rows[gguf]) == pytest.approx(30 / 150)# another thread count is another kindassert layer_kind_key("bf16", {}, 2048, 768, 8, 6) not in rows# a profile from different hardware is ignoreddiff --git a/tests/moe/test_offload.py b/tests/moe/test_offload.pyindex 5d7d0ab8..8833cdf9 100644--- a/tests/moe/test_offload.py+++ b/tests/moe/test_offload.py@@ -1030,6 +1030,101 @@ def test_locked_layer_copy_missing_rejects_ensure_experts_staging():cache.copy_missing()[email protected](not torch.cuda.is_available(), reason="needs CUDA")[email protected]("memops", [True, False])+def test_locked_hybrid_layer_fetches_through_staging_in_a_graph(monkeypatch, memops):+ # a locked layer's hybrid decode fetches at most max_fetch misses per step into its slots through the pinned staging; the rest go to the CPU+ # memops: the GPU waits on the doorbell with a stream memop, else the gather kernel spins on it+ from types import SimpleNamespace++ from freetoken.moe.locked_fetch import LockedLayerFetch++ monkeypatch.setenv("FREETOKEN_LOCKED_FETCH_THREADS", "2")+ torch.manual_seed(0)+ cache, sources = _make_split_cache(num_layers=2, locked=(1,), device="cuda")+ cache.decode_target = "hybrid"+ cache.locked_hybrid = frozenset({1})+ cache.locked_fetch = LockedLayerFetch(cache, [1], plan={1: (0, 1.0)}, max_fetch=2, executor=SimpleNamespace(core_ids=[], _coord_core=-1, _flag_sync=memops))++ top_k = 3+ ids = torch.zeros(top_k, dtype=torch.int32, device="cuda")+ weights = torch.rand(top_k, device="cuda")+ out = {}++ def step():+ slots, _, cpu_ids = cache.ensure_experts_hybrid(1, ids, weights)+ cache.copy_missing()+ out["slots"], out["cpu"] = slots, cpu_ids++ cache.collect_stats = True+ stream = torch.cuda.Stream()+ stream.wait_stream(torch.cuda.current_stream())+ with torch.cuda.stream(stream):+ step()+ torch.cuda.current_stream().wait_stream(stream)+ cache.reset()+ graph = torch.cuda.CUDAGraph()+ with torch.cuda.graph(graph):+ step()++ def replay(graph, steps):+ resident = set()+ for _ in range(steps):+ experts = torch.randperm(4)[:top_k].to(torch.int32)+ ids.copy_(experts)+ graph.replay()+ torch.cuda.synchronize()+ slots, cpu_ids = out["slots"].tolist(), out["cpu"].tolist()+ fetched = [e for e, s in zip(experts.tolist(), slots) if s >= 0 and e not in resident]+ assert len(fetched) <= 2+ for e, s, c in zip(experts.tolist(), slots, cpu_ids):+ assert (s >= 0) != (c >= 0)+ if s >= 0:+ for (src, _), view in zip(cache.banks, cache.pool_bank_views(1)):+ assert torch.equal(view[s].cpu(), src[1][e])+ resident = {e for e, s in enumerate(cache.slot_for_id[1].tolist()) if s >= 0}++ replay(graph, 12)+ # --moe-collect-stats: the locked layer reports as its own group, and each line clears the counters+ line = cache.decode_stats_line()+ assert "locked 1 layer: hit" in line and "pinned" not in line, line+ assert cache.decode_stats_line() is None+ # a runtime rebuild reallocates evict_slots and the slot caches; the fetch follows and a new capture still copies right+ del graph+ cache.rebuild(12)+ graph = torch.cuda.CUDAGraph()+ with torch.cuda.graph(graph):+ step()+ replay(graph, 6)+++def test_pin_probe_pins_middle_layers_until_refused(monkeypatch):+ # --moe-cpu-layers auto: deferred banks pin middle-out until the (simulated) quota refuses one; the rest lock, and the margin gives back the last pinned layer+ import freetoken.kernel.pinned as pinned+ import freetoken.moe.host_banks as hb++ monkeypatch.setattr(pinned, "host_register", lambda addr, nbytes: None)+ monkeypatch.setattr(pinned, "host_unregister", lambda addr: None)+ monkeypatch.setattr(hb, "_os_lock", lambda addr, nbytes: None)+ monkeypatch.setattr(hb, "_os_lock_failed", False)+ monkeypatch.setattr(hb, "_sim_pinned", 0)+ monkeypatch.setenv("FREETOKEN_SIMULATE_PIN_CAP_GB", str(3 * hb._BLK / 2**30))+ deferred = [hb.HostResidency.DEFERRED.value] * 6++ def probe(margin):+ banks = {"gate_up": [hb.HostBank((4,), torch.uint8) for _ in range(6)]}+ with hb.requested_residency(deferred) as plan:+ hb.pin_banks(banks)+ labels = hb.probe_pin(plan, 6, margin_bytes=margin)+ for bank in banks["gate_up"]:+ bank.unpin()+ return [i for i, r in enumerate(labels) if r == hb.HostResidency.LOCKED.value]++ # order 2, 3, 1, 4, 0, 5: the quota takes three, layer 4 is refused+ assert probe(0) == [0, 4, 5]+ assert probe(hb._BLK) == [0, 1, 4, 5]++def test_requested_residency_routes_layer_settles(monkeypatch):# the ambient plan installed by load_expert_banks must route each layer's banks by label at both slow-path settle points (PinPipeline layer sink, list-valued pin_banks) and record that it was consulted# without a plan everything pinsdiff --git a/tests/scheduler/test_scheduler_status.py b/tests/scheduler/test_scheduler_status.pyindex dcaa5740..be609f65 100644--- a/tests/scheduler/test_scheduler_status.py+++ b/tests/scheduler/test_scheduler_status.py@@ -159,3 +159,13 @@ def test_usage_ratio_guard():assert _usage_ratio(0, 0) == 0.0assert _usage_ratio(5, 0) == 0.0assert _usage_ratio(5, 10) == 0.5+++def test_moe_stats_line_follows_each_decode_line():+ logs: list[str] = []+ calls = []+ rep = SchedulerStatusReporter(log=logs.append, clock=lambda: 1.0, decode_log_interval=2, moe_stats=lambda: calls.append(1) or "MoE decode, x")+ for _ in range(4):+ rep.report_batch(_decode_batch(1), running_reqs=1, queue_reqs=0, kv_used_pages=1, kv_total_pages=10, page_size=1)+ assert len(calls) == 2+ assert [line.split(",")[0] for line in logs] == ["Decode batch", "MoE decode"] * 2